Updating c++/python TopCoder testers

The other day I talked about Greed and posted my c++ and python tester templates. The templates do some magic like putting ANSI color codes to make the results and status easier to read.

I have been using them for a while now which made me realize there was room for improvement. The whole stuff about editing test cases was over complicated. I want the tester to have easy-to add test cases. Test cases without a known result (so that you can test for timeout/runtime error but they don't give WA). Also, ever since the updates I made to KawigiEdit, I was used to being able to disable test cases, so that when running the code, only a few cases run.

Disabling test cases from just an editable file is not as practical as using KawigiEdit's test case editor. I had to come up with workarounds. The workarounds in the initial versions of the tester templates were not working great. Because you needed to specifically list the number of test cases and the test cases to run by default in some vector (And you had to update them manually whenever you added a test case).

I switched to making a function that contains all your test cases and takes a single integer argument (The test case number). If it ever returns 'm', it is a way of marking the end of the list of test cases. So the main method calls this function with 0, 1, 2, 3, ...., x until an x is found that returns m.

For disabled test cases, I make the function return 'd'. It is necessary to find out if test cases were disabled so that the tester doesn't tell you everything is correct when there is a disabled test case (Which could be the one test case that is wrong) . The tester should tell you everything is right only when all test cases in the set are correct or have unknown answer and ran correctly.

Anyway, this is the way it works now, for the SRM 591 div2 easy problem:

char run_testcase(int __no, bool __disabled) {
 int a;
 int b;
 int c;
 double __expected = 0.0;
 bool __unknownAnswer = false;
    
 switch (__no) {
  case 0: {
    a = 0;
    b = 1;
    c = 2;

    __expected = 0.0;
    break;
  }
  case 1: {
    a = 0;
    b = 2;
    c = 1;

    __expected = 1.5;
    break;
  }
  case 2: {
    a = 3;
    b = 2;
    c = 1;

    __expected = 0.0;
    break;
  }
  case 3: {
    a = 4;
    b = 4;
    c = 8;

    __expected = 2.0;
    break;
  }
  /*case 4: { 
    // Your custom testcase goes here:
    a = ;
    b = ;
    c = ;
     
    __unknownAnswer = true; 
    break;
  }*/

  default: return 'm'; //mark the end of tests
 }
 // To disable test cases: 
 if ( __disabled ) {
     return 'd';
 }
 return do_test(a, b, c, __expected, __no, __unknownAnswer);
}

This is the test case code automatically generated by Greed using the template. This problem takes three integer variables, a,b,c. So if you want to add a couple of cases, you edit this part of the file and add:

  case 4: { 
    a = 27;
    b = 383;
    c = 1000;
     
    __unknownAnswer = true; 
    break;
  }
  case 5: { 
    a = 112;
    b = 33;
    c = 67;
     
    __expected  = 56.5; 
    break;
  }

Test case 4 has an unknown answer, this means that no matter what it returns, as long as it didn't time out or seg fault, the tester will not consider the test case wrong. The test case will show a ? to describe the status and if all the other cases with known answer are correct, the whole solution will be considered correct, regardless of the answer. Test case 5 does have a known answer.

Disabling test cases is more complicated. The idea is to make the code return 'd' in case of a disabled case. Doing the following will work well:

  case 4: { 
    a = 27;
    b = 383;
    c = 1000;
     
    __unknownAnswer = true; 
    break;
  }
  case 5: { 
    a = 112;
    b = 33;
    c = 67;
     
    __expected  = 56.5; 
    return 'd'
    break;
  }

However, usually you want to disable all tests but one. At first I thought of doing something clever like making if (__no != 5) return 'd'. (Where __no is the identifier for the test case number). But remember that the function is supposed to return 'm' when a case number is not registered. This would break things if we return 'd' before the return 'm'. That caused some issues, but now I have this:

  default: return 'm'; //mark the end of tests
 }
 // To disable test cases: 
 if ( __disabled ) {
     return 'd'
 }
 return do_test(a, b, c, __expected, __no, __unknownAnswer);

The if statement with __disabled is the one that can be updated to have cooler conditions like (__no != 5) without breaking things (because the return 'm' already exists above).

Link

I am now using github to store these new templates. You can find the latest version of c++ and python templates for Greed with this functionality at: http://github.com/vexorian/topcoder-greed/tree/master/src/main/resources/Resource. The file names are ColorGCC11Test.cpp and ColorTest.py