cafe-grader-judge-scripts Commit

cafe-grader-judge-scripts

Commit `r31:d810d6c2e99e` public

parent | child

Description:

[grader] fixed: import_problem (error in erb calls), check_wrapper; better error handling -- will get 0 score for a particular test run that fails git-svn-id: http://theory.cpe.ku.ac.th/grader/judge/trunk/scripts@118 6386c4cd-e34a-4fa8-8920-d93eb39b512e

Commit status:

[Not Reviewed]

References:

default

Diff options:

| | | |

Comments:

0 Commit comments 0 Inline Comments

Unresolved TODOs:

There are no unresolved TODOs

Add another comment

r31:d810d6c2e99e - Fri, 14 Mar 2008 01:32:57 - 5 files changed: 28 inserted, 15 deleted

import_problem ¶ +3 -3

               #!/usr/bin/ruby
               # import_problem:
               #  * creates a directory for a problem in the current directory,
               #  * copy testdata in the old format and create standard testcase config file
               require 'erb'
               require 'fileutils'
               def input_filename(dir,i)
                 "#{dir}/input-#{i}.txt"
               end
               def answer_filename(dir,i)
                 "#{dir}/answer-#{i}.txt"
               end
               def copy_testcase(importing_test_dir,dir,i)
                 system("cp #{importing_test_dir}/#{i}.in #{input_filename(dir,i)}")
                 system("cp #{importing_test_dir}/#{i}.sol #{answer_filename(dir,i)}")
               end
               def process_options(options)
                 i = 4
                 while i<ARGV.length
                   if ARGV[i]=='-t'
                     options[:time_limit] = ARGV[i+1].to_i if ARGV.length>i+1
                     i += 1
                   end
                   if ARGV[i]=='-m'
                     options[:mem_limit] = ARGV[i+1].to_i if ARGV.length>i+1
                     i += 1
                   end
                   i += 1
                 end
               end
               SCRIPT_DIR = File.dirname(__FILE__)
               # print usage
               if (ARGV.length < 4) or (ARGV[3][0,1]=="-")
                 puts <<USAGE
               using: import_problem name dir num check [options]
                  where: name = problem_name [
                         dir = importing testcase directory
                         num = number of testcases
                         check = check script, which can be
                                  'integer', 'text' (for standard script),
                                  path_to_your_script, or
                                  'wrapper:(path_to_your_wrapped_script)'
                  options: -t time-limit (in seconds)
                           -m memory-limit (in megabytes)
               What it does:
                 * creates a directory for a problem in the current directory,
                 * copies testdata in the old format and create standard testcase config file
                 * copies a check script for grading
                 * creates a test_request template in the current directory + '/test_request'
               For wrapped checked script see comment in templates/check_wrapper for
                 information.
               USAGE
                 exit(127)
               end
               # processing arguments
               problem = ARGV[0]
               testcase_dir = ARGV[1]
               num_testcases = ARGV[2].to_i
               check_script = ARGV[3]
               options = {:time_limit => 1,  :mem_limit => 16}
               process_options(options)
               # start working
               puts "creating directories"
               system("mkdir #{problem}")
               system("mkdir #{problem}/script")
               system("mkdir #{problem}/test_cases")
               #system("cp #{GRADER_DIR}/std-script/* #{problem}/script")
               puts "copying testcases"
 .upto(num_testcases) do |i|
                 system("mkdir #{problem}/test_cases/#{i}")
                 copy_testcase("#{testcase_dir}","#{problem}/test_cases/#{i}",i)
               end
               # generating all_tests.cfg
               puts "generating testcase config file"
               template = File.open(SCRIPT_DIR + "/templates/all_tests.cfg.erb").read
               all_test_cfg = ERB.new(template)
               cfg_file = File.open("#{problem}/test_cases/all_tests.cfg","w")
               cfg_file.puts all_test_cfg.result
               cfg_file.close
               # copy check script
               if res = /^wrapper:(.*)$/.match(check_script)
                 # wrapper script
                 check_script_fname = res[1]
                 script_name = File.basename(check_script_fname)
-            -   check_wrapper_template = File.open(SCRIPT_DIR + "/templates/check_wrapper")
+            +   check_wrapper_template = File.open(SCRIPT_DIR + "/templates/check_wrapper").read
-            -   check_wrapper = ERB.new(template)
+            +   check_wrapper = ERB.new(check_wrapper_template)
                 check_file = File.open("#{problem}/script/check","w")
-            -   check_file.puts check_wrapper
+            +   check_file.puts check_wrapper.result
                 check_file.close
                 File.chmod(0755,"#{problem}/script/check")
                 system("cp #{check_script_fname} #{problem}/script/#{script_name}")
               else
                 if File.exists?(SCRIPT_DIR + "/templates/check.#{check_script}")
                   check_script_fname = SCRIPT_DIR + "/templates/check.#{check_script}"
                 else
                   check_script_fname = check_script
                 end
                 system("cp #{check_script_fname} #{problem}/script/check")
               end
               # generating test_request directory
               puts "generating test_request template"
               FileUtils.mkdir_p("test_request/#{problem}/script")
               FileUtils.mkdir_p("test_request/#{problem}/test_cases/1")
               template = File.open(SCRIPT_DIR + "/templates/test_request_all_tests.cfg.erb").read
               test_request_all_test_cfg = ERB.new(template)
               cfg_file = File.open("test_request/#{problem}/test_cases/all_tests.cfg","w")
               cfg_file.puts test_request_all_test_cfg.result
               cfg_file.close
               system("cp #{SCRIPT_DIR}/templates/check_empty test_request/#{problem}/script/check")
               system("cp #{SCRIPT_DIR}/templates/answer-1.txt test_request/#{problem}/test_cases/1")
               puts "done"

std-script/grade ¶ +10 -5

               #!/usr/bin/ruby
+            + CORRECT_MARK = 'P'
+            + INCORRECT_MARK = '-'
+            + TIMEOUT_MARK = 'T'
+            + RUN_ERROR_MARK = 'x'
+            +
               def log(str='')
                 if ENV['TALKATIVE']!=nil
                   puts str
                 end
                 if ENV['GRADER_LOGGING']!=nil
                   log_fname = ENV['GRADER_LOGGING']
                   fp = File.open(log_fname,"a")
                   fp.puts("grade: #{Time.new.strftime("%H:%M")} #{str}")
                   fp.close
                 end
               end
               def char_comment(comment)
                 if comment =~ /[iI]ncorrect/
-            -     '-'
+            +     INCORRECT_MARK
                 elsif comment =~ /[Cc]orrect/
-            -     'P'
+            +     CORRECT_MARK
                 elsif comment =~ /[Tt]ime/
-            -     'T'
+            +     TIMEOUT_MARK
                 else
-            -     'x'  # these are run time errors
+            +     RUN_ERROR_MARK  # these are run time errors
                 end
               end
               problem_home = ENV['PROBLEM_HOME']
               require "#{problem_home}/script/test_dsl.rb"
               load "#{problem_home}/test_cases/all_tests.cfg"
               problem = Problem.get_instance
               if problem.well_formed? == false
                 log "The problem specification is not well formed."
                 exit(127)
               end
               all_score = 0
               all_comment = ''
               (1..(problem.runs.length-1)).each do |k|
                 log "grade run #{k}"
                 run = problem.runs[k]
                 run_score = 0
                 run_comment = ''
                 run_comment_short = ''
                 run.tests.each do |test_num|
                   result_file_name = "#{test_num}/result"
                   if not File.exists?(result_file_name)
                     run_comment += "result file for test #{test_num} not found\n"
-            -       run_comment_short += 'x'
+            +       run_comment_short += RUN_ERROR_MARK
                     log "Cannot find the file #{test_num}/result!"
                   else
                     result_file = File.new(result_file_name, "r")
                     result_file_lines = result_file.readlines
                     run_score = run_score + result_file_lines[1].to_i
                     run_comment += result_file_lines[0]
                     run_comment_short += char_comment(result_file_lines[0])
                     result_file.close
                   end
                 end
                 run_result_file = File.new("result-#{k}", "w")
                 run_result_file.write run_score
                 run_result_file.write "\n"
                 run_result_file.close
                 run_comment_file = File.new("comment-#{k}", "w")
                 run_comment_file.write "#{run_comment}\n"
                 run_comment_file.close
                 all_score = all_score + run_score
                 all_comment += run_comment_short
               end
               result_file = File.new("result", "w")
               result_file.write all_score
               result_file.write "\n"
               result_file.close
               comment_file = File.new("comment", "w")
               comment_file.write "#{all_comment}\n"
               comment_file.close

std-script/judge ¶ +8 -4

               #!/usr/bin/ruby
               def log(str='')
                 if ENV['TALKATIVE']!=nil
                   puts str
                 end
                 if ENV['GRADER_LOGGING']!=nil
                   log_fname = ENV['GRADER_LOGGING']
                   fp = File.open(log_fname,"a")
                   fp.puts("judge: #{Time.new.strftime("%H:%M")} #{str}")
                   fp.close
                 end
               end
               problem_home = ENV['PROBLEM_HOME']
               def execute(command, error_message="")
                 if not system(command)
-            -     log "ERROR: #{error_message}"
+            +     msg = "ERROR: #{error_message}"
-            -     puts "ERROR: #{error_message}"
+            +     log msg
-            -     exit(127)
+            +     raise msg
                 end
               end
               # ARGV[0] --- language
               # ARGV[1] --- program source file
               # ARGV[2] --- test result directory
               # ARGV[3] --- sandbox directory
               if ARGV.length < 2 || ARGV.length > 4
                 puts "Usage: judge <language> <program-source> [<test-result-directory>] [<sandbox-directory>]"
                 puts "  <sandbox-directory> is defaulted to ./sandbox"
                 puts "  <test-result-directory> is defaulted to ./test-result"
                 puts "WARNING: The judge script will forcefully create the (implicitly and explicitly) specified directories and remove anything inside it."
                 exit(127)
               end
               language = ARGV[0]
               if language != "c" && language != "c++" && language != "pascal"
                 log "You specified a language that is not supported."
                 exit(127)
               end
               source_file = ARGV[1]
               if File.exist?(source_file) == false
                 log "The source file does not exist."
                 exit(127)
               end
               log "Making test result and sandbox directories..."
               current_dir = `pwd`
               current_dir.strip!
               if ARGV.length >= 3
                 test_result_dir = ARGV[2]
               else
                 test_result_dir = "#{current_dir}/test-result"
               end
               log "Test result directory: #{test_result_dir}"
               system("rm -Rf #{test_result_dir}")
               execute("mkdir #{test_result_dir}", "Cannot make directory #{test_result_dir}.")
               if ARGV.length >= 4
                 sandbox_dir = ARGV[3]
               else
                 sandbox_dir = "#{current_dir}/sandbox"
               end
               log "Sandbox directory: #{sandbox_dir}"
               system("rm -Rf #{sandbox_dir}")
               execute("mkdir #{sandbox_dir}", "Cannot make directory #{sandbox_dir}")
               # Compile
               log
               log "Compiling..."
               execute("cp #{source_file} #{sandbox_dir}", "Cannot copy the source file to #{sandbox_dir}")
               begin
                 Dir.chdir sandbox_dir
               rescue
                 log "ERROR: Cannot change directory to #{sandbox_dir}."
                 exit(127)
               end
               execute("#{problem_home}/script/compile #{language} #{source_file}", "Compilation error!")
               compile_message = `cat compiler_message`
               compile_message.strip!
               execute("mv compiler_message #{test_result_dir}", "Cannot move the compiler message to #{test_result_dir}.")
               if !FileTest.exist?("a.out")
                 log "Cannot compile the source code. See message in #{test_result_dir}/compile_message"
                 exit(127)
               else
                 execute("mv a.out #{test_result_dir}", "Cannot move the compiled program to #{test_result_dir}")
                 system("rm -Rf #{sandbox_dir}/*")
               end
               require "#{problem_home}/script/test_dsl.rb"
               load "#{problem_home}/test_cases/all_tests.cfg"
               problem = Problem.get_instance
               if problem.well_formed? == false
                 log "The problem specification is not well formed."
                 exit(127)
               end
               # Doing the testing.
               (1..(problem.num_tests)).each do |test_num|
                 log "Test number: #{test_num}"
                 execute("cp #{test_result_dir}/a.out #{sandbox_dir}", "Cannot copy the compiled program into #{sandbox_dir}")
-            -   execute("#{problem_home}/script/run #{language} #{test_num}", "Error occured during execution of the run script")
+            +   begin
+            +     execute("#{problem_home}/script/run #{language} #{test_num}", "Error occured during execution of the run script")
+            +   rescue
+            +     # do nothing
+            +   end
                 execute("mkdir #{test_result_dir}/#{test_num}", "Cannot create directory #{test_result_dir}/#{test_num}")
                 execute("mv #{sandbox_dir}/result #{test_result_dir}/#{test_num}", "Cannot copy the result file into #{test_result_dir}/#{test_num}")
                 execute("mv #{sandbox_dir}/comment #{test_result_dir}/#{test_num}", "Cannot copy the comment file into #{test_result_dir}/#{test_num}")
                 execute("mv #{sandbox_dir}/output.txt #{test_result_dir}/#{test_num}", "Cannot copy the output file into #{test_result_dir}/#{test_num}")
                 execute("rm -Rf #{sandbox_dir}/*", "Cannot clear #{sandbox_dir}")
               end
               # Grade
               log
               log "Grading..."
               begin
                 Dir.chdir test_result_dir
               rescue
                 log "ERROR: Cannot change directory to #{test_result_dir}."
                 exit(127)
               end
               execute("#{problem_home}/script/grade", "An error occured during grading!")
               log
               log "All done!"

std-script/run ¶ +1

               #!/usr/bin/ruby
               def log(str='')
                 if ENV['TALKATIVE']!=nil
                   puts str
                 end
                 if ENV['GRADER_LOGGING']!=nil
                   log_fname = ENV['GRADER_LOGGING']
                   fp = File.open(log_fname,"a")
                   fp.puts("run: #{Time.new.strftime("%H:%M")} #{str}")
                   fp.close
                 end
               end
               def extract_time(t)
                 if (result=/^(.*)r(.*)u(.*)s/.match(t))
                   {:real => result[1], :user => result[2], :sys => result[3]}
                 else
                   raise "Error reading running time"
                 end
               end
               def compile_box(source,bin)
                 system("gcc #{source} -o #{bin}")
               end
               if ARGV.length < 2 || ARGV.length > 3
                 puts "Usage: run <language> <test-num> [<program-name>]"
                 exit(127)
               end
               language = ARGV[0]
               test_num = ARGV[1].to_i
               if ARGV.length > 2
                 program_name = ARGV[2]
               else
                 program_name = "a.out"
               end
               problem_home = ENV['PROBLEM_HOME']
               require "#{problem_home}/script/test_dsl.rb"
               load "#{problem_home}/test_cases/all_tests.cfg"
               problem = Problem.get_instance
               if problem.well_formed? == false
                 log "The problem specification is not well formed."
                 exit(127)
               end
               # Check if the test number is okay.
               if test_num <= 0 || test_num > problem.num_tests
                 log "You have specified a wrong test number."
                 exit(127)
               end
               #####################################
               # Set the relavant file names here. #
               #####################################
               input_file_name = "#{problem_home}/test_cases/#{test_num}/input-#{test_num}.txt"
               #####################################
               time_limit = problem.get_time_limit test_num
               mem_limit = problem.get_mem_limit(test_num) * 1024
               # Copy the input file.
               #`cp #{problem_home}/test_cases/#{test_num}/#{input_file_name} .`
               time_output_format = "%Er%Uu%Ss"
               # check if box is there, if not, compile it!
               if !File.exists?("#{problem_home}/script/box")
                 log "WARNING: Compiling box: to increase efficiency, it should be compile manually"
                 compile_box("#{problem_home}/script/box.c",
                             "#{problem_home}/script/box")
               end
               # Run the program.
               run_command = "/usr/bin/time -f \"#{time_output_format}\" 2>run_result #{problem_home}/script/box -a 2 -f -t #{time_limit} -m #{mem_limit} -i #{input_file_name} -o output.txt #{program_name}"
               log "Running test #{test_num}..."
               log run_command
               log
               system(run_command)
               # Create the result file.
               result_file = File.new("result", "w")
               comment_file = File.new("comment", "w")
               # Check if the program actually produced any output.
               run_result_file = File.new("run_result", "r")
               run_result = run_result_file.readlines
               run_result_file.close
               time_elapsed = run_result[run_result.length-1]
               running_time = extract_time(time_elapsed)
               report = lambda{ |status, points, comment|
                 result_file.write status.strip
                 result_file.write "\n"
                 result_file.write points.to_s.strip
                 result_file.write "\n"
                 result_file.write time_elapsed.strip
                 result_file.write "\n"
                 result_file.close
                 `rm run_result`
                 # `rm output.txt`   --- keep the output
                 comment_file.write comment
                 comment_file.write "--run-result--\n"
                 run_result.each do |l|
                   comment_file.write l
                 end
                 comment_file.close
                 log "Done!"
                 exit(0)
               }
               if run_result[0][0,2] != "OK"
                 log "There was a runtime error."
                 report.call(run_result[0], 0, "No comment.\n")
               end
               if running_time[:user].to_f + running_time[:sys].to_f > time_limit
                 log "Time limit exceeded."
                 report.call("Time limit exceeded", 0, "No comment.\n")
               end
               # Run 'check' to evaluate the output.
               #puts "There was no runtime error. Proceed to checking the output."
               check_command = "#{problem_home}/script/check #{language} #{test_num}"
               log "Checking the output..."
               log check_command
               if not system(check_command)
                 log "Problem with check script"
+            +   report.call("Incorrect",0,"Check script error.\n")
                 exit(127)
               end
               check_file = File.new("check_result", "r")
               check_file_lines = check_file.readlines
               report.call(check_file_lines[0], check_file_lines[1], "No comment.\n")

templates/check_wrapper ¶ +6 -3

               #!/usr/bin/ruby
               #
               # This is a check script wrapper.  It read all required information
               # and call a real check script call REAL_CHECK_SCRIPT in directory
               # [problem_home]/script
               #
               REAL_CHECK_SCRIPT = "<%= script_name %>"
               # The REAL_CHECK_SCRIPT is called with:
               #
               #  (script) <lang> <test-num> <in-file> <out-file> <ans-file> <full-score>
               #
               # and REAL_CHECK_SCRIPT's output to standard out is redirected to
               # 'check_result' as required by normal check script.
               problem_home = ENV['PROBLEM_HOME']
               require "#{problem_home}/script/test_dsl.rb"
               if ARGV.length < 2
                 puts "Usage: check <language> <test-number> [<output-file>]"
                 exit(0)
               end
               language = ARGV[0]
               test_num = ARGV[1].to_i
               if ARGV.length >= 3
                 output_file_name = ARGV[2]
               else
                 output_file_name = "output.txt"
               end
               load "#{problem_home}/test_cases/all_tests.cfg"
               problem = Problem.get_instance
-            - answer_file_name = File.new("#{problem_home}/test_cases/#{test_num}/answer-#{test_num}.txt")
+            + answer_file_name = "#{problem_home}/test_cases/#{test_num}/answer-#{test_num}.txt"
-            - input_file_name = File.new("#{problem_home}/test_cases/#{test_num}/input-#{test_num}.txt")
+            + input_file_name = "#{problem_home}/test_cases/#{test_num}/input-#{test_num}.txt"
-            - score = Problem.get_score(test_num)
+            + score = problem.get_score(test_num)
               cmd = "#{problem_home}/script/#{REAL_CHECK_SCRIPT} " +
                 "#{language} #{test_num} #{input_file_name} #{output_file_name} " +
                 "#{answer_file_name} #{score} > check_result"
+            + #puts "wrapper-CMD: #{cmd}"
+            +
+            + system(cmd)

Write
Preview

You need to be logged in to leave comments. Login now

	Site-wide shortcuts
/	Use quick search box
g h	Goto home page
g g	Goto my private gists page
g G	Goto my public gists page
n r	New repository page
n g	New gist page

	Repositories
g s	Goto summary page
g c	Goto changelog page
g f	Goto files page
g F	Goto files page with file search activated
g p	Goto pull requests page
g o	Goto repository settings
g O	Goto repository permissions settings

Sign in to your account

Author

r31:d810d6c2e99e - Fri, 14 Mar 2008 01:32:57 - 5 files changed: 28 inserted, 15 deleted

Sign in to your account

Commit r31:d810d6c2e99e public

Author

r31:d810d6c2e99e - Fri, 14 Mar 2008 01:32:57 - 5 files changed: 28 inserted, 15 deleted

Commit `r31:d810d6c2e99e` public