cafe-grader-judge-scripts Commit

cafe-grader-judge-scripts

Commit `r66:c03ebf30fb10` public

parent | child

Description:

fixed test run scoring bug: now it takes the minimum of each test case's score git-svn-id: http://theory.cpe.ku.ac.th/grader/judge/trunk/scripts@273 6386c4cd-e34a-4fa8-8920-d93eb39b512e

Commit status:

[Not Reviewed]

References:

default

Diff options:

| | | |

Comments:

0 Commit comments 0 Inline Comments

Unresolved TODOs:

There are no unresolved TODOs

Add another comment

r66:c03ebf30fb10 - Wed, 20 Aug 2008 12:59:31 - 4 files changed: 17 inserted, 16 deleted

lib/test_request_helper.rb ¶ +2

               #
               # This part contains various test_request helpers for interfacing
               # with Grader::Engine.  There are TestRequestRoomMaker and
               # TestRequestReporter.
               module Grader
                 #
                 # A TestRequestRoomMaker is a helper object for Engine
                 # - finds grading room: in user_result_dir/(user)/test_request/ ...
                 # - prepare problem configuration for grading --- basically it copy
                 #   all config files, and copy user's input into the testcase
                 #   directory.  First, it finds the template from problem template
                 #   directory; if it can't find a template, it'll use the template
                 #   from default template.
                 class TestRequestRoomMaker
                   def initialize
                     @config = Grader::Configuration.get_instance
                   end
                   def produce_grading_room(test_request)
                     grading_room = grading_room_dir(test_request)
                     FileUtils.mkdir_p(grading_room)
                     #
                     # Also copy additional submitted file to this directory as well.
                     # The program would see this file only if it is copied
                     #    to the sandbox directory later.  The run script should do it.
                     #
+            +       if FileTest.exists?("#{test_request.input_file_name}.files")
                       cmd = "cp #{test_request.input_file_name}.files/* #{grading_room}"
                       system(cmd)
+            +       end
                     grading_room
                   end
                   def find_problem_home(test_request)
                     problem_name = test_request.problem_name
                     template_dir = "#{@config.test_request_problem_templates_dir}/" + problem_name
                     raise "Test Request: error template not found" if !File.exists?(template_dir)
                     problem_home = problem_home_dir(test_request)
                     FileUtils.mkdir_p(problem_home)
                     copy_problem_template(template_dir,problem_home)
                     link_input_file(test_request,problem_home)
                     problem_home
                   end
                   def save_source(test_request,source_name)
                     dir = self.produce_grading_room(test_request)
                     submission = test_request.submission
                     f = File.open("#{dir}/#{source_name}","w")
                     f.write(submission.source)
                     f.close
                   end
                   def clean_up(test_request)
                     problem_home = problem_home_dir(test_request)
                     remove_data_files(problem_home)
                   end
                   protected
                   def grading_room_dir(test_request)
                     problem_name = test_request.problem_name
                     user = test_request.user
                     grading_room = "#{@config.user_result_dir}" +
                       "/#{user.login}/test_request" +
                       "/#{problem_name}/#{test_request.id}"
                     grading_room
                   end
                   def problem_home_dir(test_request)
                     problem_name = test_request.problem_name
                     user = test_request.user
                     "#{@config.user_result_dir}" +
                       "/#{user.login}/test_request/#{problem_name}"
                   end
                   def copy_problem_template(template_dir,problem_home)
                     cmd = "cp -R #{template_dir}/* #{problem_home}"
                     system_and_raise_when_fail(cmd,"Test Request: cannot copy problem template")
                   end
                   def link_input_file(test_request,problem_home)
                     input_fname = "#{test_request.input_file_name}"
                     if !File.exists?(input_fname)
                       raise "Test Request: input file not found."
                     end
                     input_fname_problem_home = "#{problem_home}/test_cases/1/input-1.txt"
                     if File.exists?(input_fname_problem_home)
                       FileUtils.rm([input_fname_problem_home], :force => true)
                     end
                     cmd = "ln -s #{input_fname} #{input_fname_problem_home}"
                     system_and_raise_when_fail(cmd,"Test Request: cannot link input file")
                   end
                   def remove_data_files(problem_home)
                     if File.exists?("#{problem_home}/test_cases/1/input-1.txt")
                       cmd = "rm #{problem_home}/test_cases/1/*"
                       system_and_raise_when_fail(cmd,"Test Request: cannot remove data files")
                     end
                   end
                   def system_and_raise_when_fail(cmd,msg)
                     if !system(cmd)
                       raise msg
                     end
                   end
                 end
                 class TestRequestReporter
                   def initialize
                     @config = Grader::Configuration.get_instance
                   end
                   def report(test_request,test_result_dir)
                     save_result(test_request,read_result(test_result_dir))
                   end
                   def report_error(test_request, msg)
                     save_result(test_request, {:running_stat => {
                                     :msg => "#{msg}",
                                     :running_time => nil,
                                     :exit_status => "Some error occured. Program did not run",
                                     :memory_usage => nil
                                   }})
                   end
                   protected
                   def read_result(test_result_dir)
                     # TODO:
                     cmp_msg_fname = "#{test_result_dir}/compiler_message"
                     cmp_file = File.open(cmp_msg_fname)
                     cmp_msg = cmp_file.read
                     cmp_file.close
                     result_file_name = "#{test_result_dir}/1/result"
                     if File.exists?(result_file_name)
                       output_file_name = "#{test_result_dir}/1/output.txt"
                       results = File.open("#{test_result_dir}/1/result").readlines
                       stat = extract_running_stat(results)
                       return {
                         :output_file_name => output_file_name,
                         :running_stat => stat,
                         :comment => "",
                         :cmp_msg => cmp_msg}
                     else
                       return {
                         :running_stat => nil,
                         :comment => "Compilation error",
                         :cmp_msg => cmp_msg}
                     end
                   end
                   def extract_running_stat(results)
                     running_stat_line = results[-1]
                     # extract exit status line
                     run_stat = ""
                     if !(/[Cc]orrect/.match(results[0]))
                       run_stat = results[0].chomp
                     else
                       run_stat = 'Program exited normally'
                     end
                     # extract running time
                     if res = /r(.*)u(.*)s/.match(running_stat_line)
                       seconds = (res[1].to_f + res[2].to_f)
                       time_stat = "Time used: #{seconds} sec."
                     else
                       seconds = nil
                       time_stat = "Time used: n/a sec."
                     end
                     # extract memory usage
                     if res = /s(.*)m/.match(running_stat_line)
                       memory_used = res[1].to_i
                     else
                       memory_used = -1
                     end
                     return {
                       :msg => "#{run_stat}\n#{time_stat}",
                       :running_time => seconds,
                       :exit_status => run_stat,
                       :memory_usage => memory_used
                     }
                   end
                   def save_result(test_request,result)
                     if result[:output_file_name]!=nil
                       test_request.output_file_name = link_output_file(test_request,
                                                                        result[:output_file_name])
                     end
                     test_request.graded_at = Time.now
                     test_request.compiler_message = (result[:cmp_msg] or '')
                     test_request.grader_comment = (result[:comment] or '')
                     if result[:running_stat]!=nil
                       test_request.running_stat = (result[:running_stat][:msg] or '')
                       test_request.running_time = (result[:running_stat][:running_time] or nil)
                       test_request.exit_status = result[:running_stat][:exit_status]
                       test_request.memory_usage = result[:running_stat][:memory_usage]
                     else
                       test_request.running_stat = ''
                     end
                     test_request.save
                   end
                   protected
                   def link_output_file(test_request, fname)
                     target_file_name = random_output_file_name(test_request.user,
                                                                test_request.problem)
                     FileUtils.mkdir_p(File.dirname(target_file_name))
                     cmd = "ln -s #{fname} #{target_file_name}"
                     if !system(cmd)

std-script/grade ¶ +9 -11

               #!/usr/bin/ruby
               CORRECT_MARK = 'P'
               INCORRECT_MARK = '-'
               TIMEOUT_MARK = 'T'
               RUN_ERROR_MARK = 'x'
               def log(str='')
                 if ENV['TALKATIVE']!=nil
                   puts str
                 end
                 if ENV['GRADER_LOGGING']!=nil
                   log_fname = ENV['GRADER_LOGGING']
                   fp = File.open(log_fname,"a")
                   fp.puts("grade: #{Time.new.strftime("%H:%M")} #{str}")
                   fp.close
                 end
               end
               def char_comment(comment)
                 if comment =~ /[Ii]ncorrect/
                   INCORRECT_MARK
                 elsif comment =~ /[Cc]orrect/
                   CORRECT_MARK
                 elsif comment =~ /[Tt]ime/
                   TIMEOUT_MARK
                 elsif res = /^[Cc]omment:(.*)$/.match(comment)
                   res[1]
                 else
                   RUN_ERROR_MARK  # these are run time errors
                 end
               end
               problem_home = ENV['PROBLEM_HOME']
               require "#{problem_home}/script/test_dsl.rb"
               load "#{problem_home}/test_cases/all_tests.cfg"
               problem = Problem.get_instance
               if problem.well_formed? == false
                 log "The problem specification is not well formed."
                 exit(127)
               end
               all_score = 0
               all_comment = ''
               (1..(problem.runs.length-1)).each do |k|
                 log "grade run #{k}"
                 run = problem.runs[k]
-            -   run_score = 0
+            +   run_score = nil
                 run_comment = ''
                 run_comment_short = ''
                 run.tests.each do |test_num|
                   result_file_name = "#{test_num}/result"
                   if not File.exists?(result_file_name)
                     run_comment += "result file for test #{test_num} not found\n"
                     run_comment_short += RUN_ERROR_MARK
                     log "Cannot find the file #{test_num}/result!"
                   else
                     result_file = File.new(result_file_name, "r")
                     result_file_lines = result_file.readlines
                     if result_file_lines.length>=2
-            -         run_score = run_score + result_file_lines[1].to_i
+            +         current_run_score = result_file_lines[1].to_i
                       run_comment += result_file_lines[0]
                       run_comment_short += char_comment(result_file_lines[0].chomp)
                     else
+            +         current_run_score = 0
                       run_comment += "result file for test #{test_num} error\n"
                       run_comment_short += RUN_ERROR_MARK
                       log "Error in #{test_num}/result!"
                     end
+            +
+            +       # the score of this run should be the minimum of the score for
+            +       #   each test case
+            +       if (run_score==nil) or (run_score>current_run_score)
+            +         run_score = current_run_score
+            +       end
                     result_file.close
                   end
                 end
-            -   # find total score for this run
-            -   run_total_score = 0
-            -   problem = Problem.get_instance
-            -   run.tests.each { |test_num| run_total_score += problem.get_score(test_num) }
-            -   if run_total_score!=run_score   # fail in some test cases, fail the run
-            -     run_score = 0
-            -   end
                 run_result_file = File.new("result-#{k}", "w")
                 run_result_file.write run_score
                 run_result_file.write "\n"
                 run_result_file.close
                 run_comment_file = File.new("comment-#{k}", "w")
                 run_comment_file.write "#{run_comment}\n"
                 run_comment_file.close
                 all_score = all_score + run_score
                 # append comment for test run with many test cases
                 if run.tests.length > 1
                   run_comment_short = '[' + run_comment_short + ']'
                 end
                 all_comment += run_comment_short
               end
               result_file = File.new("result", "w")
               result_file.write all_score
               result_file.write "\n"
               result_file.close
               comment_file = File.new("comment", "w")
               comment_file.write "#{all_comment}\n"
               comment_file.close

templates/all_tests.cfg.erb ¶ +1 -1

               problem do
                 num_tests <%= num_testcases %>
                 full_score <%= num_testruns*10 %>
                 time_limit_each <%= options[:time_limit] %>
                 mem_limit_each <%= options[:mem_limit] %>
                 score_each 10
               <% tr_num = 0 %>
               <% testrun_info.each do |testrun| %>
                 <% tr_num += 1 %>
                 run <%= tr_num %> do
                   tests <%= (testrun.collect {|testcase| testcase[0]}).join(", ") %>
                   <% if testrun.length==1 %>
                   scores 10
                   <% else %>
-            -     scores 10 <% (testrun.length-1).times do %>,0 <% end %>
+            +     scores 10 <% (testrun.length-1).times do %>,10 <% end %>
                   <% end %>
                 end
               <% end %>
               end

test/data/ev/test_normal/test_cases/all_tests.cfg ¶ +5 -4

               problem do
                 num_tests 10
                 full_score 135
                 time_limit_each 1
                 mem_limit_each 11
                 score_each 10
                 run 1 do
                   tests 1, 2
-            -     scores 10, 20
+            +     scores 30, 30
                   time_limits 1, 2
                   mem_limits 5, 6
                 end
                 run 2 do
                   tests 3, 4, 5, 6, 7
-            -     score_each 10
+            +     score_each 50
                   time_limit_each 3
                   mem_limit_each 3
                 end
                 run 3 do
                   tests 8, 9, 10
                 end
                 test 8 do
-            -     score 30
+            +     score 55
                   time_limit 3
                   mem_limit 10
                 end
                 test 9 do
-            -     score 15
+            +     score 55
                 end
                 test 10 do
+            +     score 55
                   time_limit 1
                 end
               end

Write
Preview

You need to be logged in to leave comments. Login now

	Site-wide shortcuts
/	Use quick search box
g h	Goto home page
g g	Goto my private gists page
g G	Goto my public gists page
n r	New repository page
n g	New gist page

	Repositories
g s	Goto summary page
g c	Goto changelog page
g f	Goto files page
g F	Goto files page with file search activated
g p	Goto pull requests page
g o	Goto repository settings
g O	Goto repository permissions settings

Sign in to your account

Author

r66:c03ebf30fb10 - Wed, 20 Aug 2008 12:59:31 - 4 files changed: 17 inserted, 16 deleted

Sign in to your account

Commit r66:c03ebf30fb10 public

Author

r66:c03ebf30fb10 - Wed, 20 Aug 2008 12:59:31 - 4 files changed: 17 inserted, 16 deleted

Commit `r66:c03ebf30fb10` public