cafe-grader-judge-scripts Commit

cafe-grader-judge-scripts

Commit `r41:d4e37450ddbc` public

parent | child

Description:

[grader] extracted test_request info into new stat columns git-svn-id: http://theory.cpe.ku.ac.th/grader/judge/trunk/scripts@163 6386c4cd-e34a-4fa8-8920-d93eb39b512e

Commit status:

[Not Reviewed]

References:

default

Diff options:

| | | |

Comments:

0 Commit comments 0 Inline Comments

Unresolved TODOs:

There are no unresolved TODOs

Add another comment

r41:d4e37450ddbc - Thu, 03 Apr 2008 20:49:43 - 3 files changed: 37 inserted, 7 deleted

lib/test_request_helper.rb ¶ +20 -4

               #
               # This part contains various test_request helpers for interfacing
               # with Grader::Engine.  There are TestRequestRoomMaker and
               # TestRequestReporter.
               module Grader
                 #
                 # A TestRequestRoomMaker is a helper object for Engine
                 # - finds grading room: in user_result_dir/(user)/test_request/ ...
                 # - prepare problem configuration for grading --- basically it copy
                 #   all config files, and copy user's input into the testcase
                 #   directory.  First, it finds the template from problem template
                 #   directory; if it can't find a template, it'll use the template
                 #   from default template.
                 class TestRequestRoomMaker
                   def initialize
                     @config = Grader::Configuration.get_instance
                   end
                   def produce_grading_room(test_request)
                     grading_room = grading_room_dir(test_request)
                     FileUtils.mkdir_p(grading_room)
                     grading_room
                   end
                   def find_problem_home(test_request)
                     problem_name = test_request.problem_name
                     template_dir = "#{@config.test_request_problem_templates_dir}/" + problem_name
                     raise "Test Request: error template not found" if !File.exists?(template_dir)
                     problem_home = problem_home_dir(test_request)
                     FileUtils.mkdir_p(problem_home)
                     copy_problem_template(template_dir,problem_home)
                     link_input_file(test_request,problem_home)
                     problem_home
                   end
                   def save_source(test_request,source_name)
                     dir = self.produce_grading_room(test_request)
                     submission = test_request.submission
                     f = File.open("#{dir}/#{source_name}","w")
                     f.write(submission.source)
                     f.close
                   end
                   def clean_up(test_request)
                     problem_home = problem_home_dir(test_request)
                     remove_data_files(problem_home)
                   end
                   protected
                   def grading_room_dir(test_request)
                     problem_name = test_request.problem_name
                     user = test_request.user
                     "#{@config.user_result_dir}" +
                       "/#{user.login}/test_request" +
                       "/#{problem_name}/#{test_request.id}"
                   end
                   def problem_home_dir(test_request)
                     problem_name = test_request.problem_name
                     user = test_request.user
                     "#{@config.user_result_dir}" +
                       "/#{user.login}/test_request/#{problem_name}"
                   end
                   def copy_problem_template(template_dir,problem_home)
                     cmd = "cp -R #{template_dir}/* #{problem_home}"
                     system_and_raise_when_fail(cmd,"Test Request: cannot copy problem template")
                   end
                   def link_input_file(test_request,problem_home)
                     cmd = "ln -s #{test_request.input_file_name} #{problem_home}/test_cases/1/input-1.txt"
                     system_and_raise_when_fail(cmd,"Test Request: cannot link input file")
                   end
                   def remove_data_files(problem_home)
                     if File.exists?("#{problem_home}/test_cases/1/input-1.txt")
                       cmd = "rm #{problem_home}/test_cases/1/*"
                       system_and_raise_when_fail(cmd,"Test Request: cannot remove data files")
                     end
                   end
                   def system_and_raise_when_fail(cmd,msg)
                     if !system(cmd)
                       raise msg
                     end
                   end
                 end
                 class TestRequestReporter
                   def initialize
                     @config = Grader::Configuration.get_instance
                   end
                   def report(test_request,test_result_dir)
                     save_result(test_request,read_result(test_result_dir))
                   end
                   def report_error(test_request, msg)
-            -       save_result(test_request, {:running_stat => "#{msg}"})
+            +       save_result(test_request, {:running_stat => {:msg => "#{msg}"}})
                   end
                   protected
                   def read_result(test_result_dir)
                     # TODO:
                     cmp_msg_fname = "#{test_result_dir}/compiler_message"
                     cmp_file = File.open(cmp_msg_fname)
                     cmp_msg = cmp_file.read
                     cmp_file.close
                     result_file_name = "#{test_result_dir}/1/result"
                     if File.exists?(result_file_name)
                       output_file_name = "#{test_result_dir}/1/output.txt"
                       results = File.open("#{test_result_dir}/1/result").readlines
                       stat = format_running_stat(results)
                       return {
                         :output_file_name => output_file_name,
                         :running_stat => stat,
                         :comment => "",
                         :cmp_msg => cmp_msg}
                     else
                       return {
-            -           :running_stat => "",
+            +           :running_stat => nil,
                         :comment => "Compilation error",
                         :cmp_msg => cmp_msg}
                     end
                   end
                   def format_running_stat(results)
                     running_time_line = results[-1]
+            +       # extract exit status line
                     run_stat = ""
                     if !(/[Cc]orrect/.match(results[0]))
                       run_stat = results[0].chomp
+            +       else
+            +         run_stat = 'Program exited normally'
                     end
+            +       # extract running time
                     if res = /r(.*)u(.*)s/.match(running_time_line)
                       seconds = (res[1].to_f + res[2].to_f)
                       time_stat = "Time used: #{seconds} sec."
                     else
+            +         seconds = nil
                       time_stat = "Time used: n/a sec."
                     end
-            -       return "#{run_stat}\n#{time_stat}"
+            +       return {
+            +         :msg => "#{run_stat}\n#{time_stat}",
+            +         :running_time => seconds,
+            +         :exit_status => run_stat
+            +       }
                   end
                   def save_result(test_request,result)
                     if result[:output_file_name]!=nil
                       test_request.output_file_name = link_output_file(test_request,
                                                                        result[:output_file_name])
                     end
                     test_request.graded_at = Time.now
                     test_request.compiler_message = (result[:cmp_msg] or '')
                     test_request.grader_comment = (result[:comment] or '')
-            -       test_request.running_stat = (result[:running_stat] or '')
+            +       if result[:running_stat]!=nil
+            +         test_request.running_stat = (result[:running_stat][:msg] or '')
+            +         test_request.running_time = (result[:running_stat][:running_time] or nil)
+            +         test_request.exit_status = (result[:running_stat][:exit_status])
+            +         test_request.memory_usage = nil  # should be added later
+            +       else
+            +         test_request.running_stat = ''
+            +       end
                     test_request.save
                   end
                   protected
                   def link_output_file(test_request, fname)
                     target_file_name = random_output_file_name(test_request.user,
                                                                test_request.problem)
                     FileUtils.mkdir_p(File.dirname(target_file_name))
                     cmd = "ln -s #{fname} #{target_file_name}"
                     if !system(cmd)
                       raise "TestRequestReporter: cannot move output file"
                     end
                     return target_file_name
                   end
                   def random_output_file_name(user,problem)
                     problem_name = TestRequest.name_of(problem)
                     begin
                       tmpname =  "#{@config.test_request_output_base_dir}" +
                         "/#{user.login}/#{problem_name}/#{rand(10000)}"
                     end while File.exists?(tmpname)
                     tmpname
                   end
                 end
               end

test/data/test2_1-5sec.c ¶ +5 -3

               #include <stdio.h>
               #include <stdlib.h>
+            + #include <unistd.h>
               int main()
               {
                 int a,b;
                 int c=0;
                 scanf("%d %d",&a,&b);
                 printf("%d\n",a+b);
-            -   for(a=0; a<2; a++) {
+            +   sleep(1);
-            -     while(c<1550000000) {
+            +   c = 0;
+            +   while(c<1000000000) {
                   c++;
                   b+=c;
                 }
-            -   }
                 exit(0);
               }

test/engine_spec.rb ¶ +12

               require File.join(File.dirname(__FILE__),'spec_helper')
               require File.join(File.dirname(__FILE__),'engine_spec_helper')
               describe "A grader engine, when grading submissions" do
                 include GraderEngineHelperMethods
                 before(:each) do
                   @config = Grader::Configuration.get_instance
                   # this test is from Pong
                   @problem_test_normal = stub(Problem,
                                               :id => 1, :name => 'test_normal',
                                               :full_score => 135)
                   @user_user1 = stub(User,
                                      :id => 1, :login => 'user1')
                   @engine = Grader::Engine.new
                   init_sandbox
                 end
                 it "should grade normal submission" do
                   grader_should(:grade => "test1_correct.c",
                                 :on => @problem_test_normal,
                                 :and_report => {
                                   :score => 135,
                                   :comment => /^PASSED/})
                 end
                 it "should produce error message when submission cannot compile" do
                   grader_should(:grade => "test1_compile_error.c",
                                 :on => @problem_test_normal,
                                 :and_report => {
                                   :score => 0,
                                   :comment => 'FAILED: compilation error',
                                   :compiler_message => /[Ee]rror/})
                 end
                 it "should produce timeout error when submission runs forever" do
                   @problem_test_timeout = stub(Problem,
                                                :id => 1, :name => 'test_timeout',
                                                :full_score => 10)
                   grader_should(:grade => "test2_timeout.c",
                                 :on => @problem_test_timeout,
                                 :and_report => {
                                   :score => 0,
                                   :comment => 'FAILED: TT'})
                 end
                 it "should produce timeout error correctly when submission runs slower than expected in less than a second" do
                   @problem_test_timeout = stub(Problem,
                                                :id => 1, :name => 'test_timeout',
                                                :full_score => 20)
                   grader_should(:grade => "test2_1-5sec.c",
                                 :on => @problem_test_timeout,
                                 :and_report => {
                                   :score => 10,
                                   :comment => 'FAILED: TP'})
                 end
                 it "should produce runtime error when submission uses too much static memory" do
                   @problem_test_memory = stub(Problem,
                                               :id => 1, :name => 'test_memory',
                                               :full_score => 20)
                   grader_should(:grade => "add_too_much_memory_static.c",
                                 :on => @problem_test_memory,
                                 :and_report => {
                                   :score => 10,
                                   :comment => /FAILED: [^P]P/})
                 end
                 it "should not allow submission to allocate too much dynamic memory" do
                   @problem_test_memory = stub(Problem,
                                               :id => 1, :name => 'test_memory',
                                               :full_score => 20)
                   grader_should(:grade => "add_too_much_memory_dynamic.c",
                                 :on => @problem_test_memory,
                                 :and_report => {
                                   :score => 10,
                                   :comment => /FAILED: [^P]P/})
                 end
                 it "should score test runs correctly when submission fails in some test case" do
                   grader_should(:grade => "add_fail_test_case_1.c",
                                 :on => @problem_test_normal,
                                 :and_report => {
                                   :score => 105,
                                   :comment => /^FAILED:/})
                 end
                 it "should fail submission with non-zero exit status" do
                   grader_should(:grade => "add_nonzero_exit_status.c",
                                 :on => @problem_test_normal,
                                 :and_report => {
                                   :score => 0,
                                   :comment => /^FAILED:/})
                 end
                 it "should not allow malicious submission to see PROBLEM_HOME" do
                   problem_test_yesno =  stub(Problem,
                                               :id => 1, :name => 'test_yesno',
                                               :full_score => 10)
                   grader_should(:grade => "yesno_access_problem_home.c",
                                 :on => problem_test_yesno,
                                 :and_report => {
                                   :score => 0,
                                   :comment => /^FAILED:/})
                 end
                 it "should not allow malicious submission to open files" do
                   problem_test_yesno =  stub(Problem,
                                               :id => 1, :name => 'test_yesno',
                                               :full_score => 10)
                   grader_should(:grade => "yesno_open_file.c",
                                 :on => problem_test_yesno,
                                 :and_report => {
                                   :score => 0,
                                   :comment => /^FAILED:/})
                 end
                 def grader_should(args)
                   @user1 = stub(User,
                                 :id => 1, :login => 'user1')
                   submission =
                     create_submission_from_file(1, @user1, args[:on], args[:grade])
                   submission.should_receive(:graded_at=)
                   expected_score = args[:and_report][:score]
                   expected_comment = args[:and_report][:comment]
                   if args[:and_report][:compiler_message]!=nil
                     expected_compiler_message = args[:and_report][:compiler_message]
                   else
                     expected_compiler_message = ''
                   end
                   submission.should_receive(:points=).with(expected_score)
                   submission.should_receive(:grader_comment=).with(expected_comment)
                   submission.should_receive(:compiler_message=).with(expected_compiler_message)
                   submission.should_receive(:save)
                   @engine.grade(submission)
                 end
                 protected
                 def create_normal_submission_mock_from_file(source_fname)
                   create_submission_from_file(1, @user_user1, @problem_test_normal, source_fname)
                 end
               end
               describe "A grader engine, when grading test requests" do
                 include GraderEngineHelperMethods
                 before(:each) do
                   @config = Grader::Configuration.get_instance
                   @engine = Grader::Engine.new(Grader::TestRequestRoomMaker.new,
                                                Grader::TestRequestReporter.new)
                   init_sandbox
                 end
                 it "should report error if there is no problem template" do
                   problem = stub(Problem,
                                  :id => 1, :name => 'nothing')
                   grader_should(:grade => 'test1_correct.c',
                                 :on => problem,
                                 :with => 'in1.txt',
                                 :and_report => {
                                   :graded_at= => nil,
                                   :compiler_message= => '',
                                   :grader_comment= => '',
                                   :running_stat= => /template not found/,
+            +                     :running_time= => nil,
+            +                     :exit_status= => nil,
+            +                     :memory_usage= => nil,
                                   :save => nil})
                 end
                 it "should run test request and produce output file" do
                   problem = stub(Problem,
                                  :id => 1, :name => 'test_normal')
                   grader_should(:grade => 'test1_correct.c',
                                 :on => problem,
                                 :with => 'in1.txt',
                                 :and_report => {
                                   :graded_at= => nil,
                                   :compiler_message= => '',
                                   :grader_comment= => '',
                                   :running_stat= => /0.0 sec./,
                                   :output_file_name= => lambda { |fname|
                                     File.exists?(fname).should be_true
                                   },
+            +                     :running_time= => nil,
+            +                     :exit_status= => nil,
+            +                     :memory_usage= => nil,
                                   :save => nil})
                 end
                 it "should clean up problem directory after running test request" do
                   problem = stub(Problem,
                                  :id => 1, :name => 'test_normal')
                   grader_should(:grade => 'test1_correct.c',
                                 :on => problem,
                                 :with => 'in1.txt',
                                 :and_report => {
                                   :graded_at= => nil,
                                   :compiler_message= => '',
                                   :grader_comment= => '',
                                   :running_stat= => nil,
                                   :output_file_name= => nil,
+            +                     :running_time= => nil,
+            +                     :exit_status= => nil,
+            +                     :memory_usage= => nil,
                                   :save => nil})
                   File.exists?(@config.user_result_dir + "/test_request/test_normal/test_cases/1/input-1.txt").should be_false
                 end
                 it "should compile test request with error and report compilation error" do
                   problem = stub(Problem,
                                  :id => 1, :name => 'test_normal')
                   grader_should(:grade => 'test1_compile_error.c',
                                 :on => problem,
                                 :with => 'in1.txt',
                                 :and_report => {
                                   :graded_at= => nil,
                                   :compiler_message= => /.+/,
                                   :grader_comment= => /[Cc]ompil.*error/,
                                   :running_stat= => '',
                                   :save => nil})
                 end
                 it "should report exit status" do
                   problem = stub(Problem,
                                  :id => 1, :name => 'test_normal')
                   grader_should(:grade => 'add_nonzero_exit_status.c',
                                 :on => problem,
                                 :with => 'in1.txt',
                                 :and_report => {
                                   :graded_at= => nil,
                                   :compiler_message= => '',
                                   :grader_comment= => '',
                                   :running_stat= => /[Ee]xit.*status.*10.*0\.0 sec/m,
                                   :output_file_name= => lambda { |fname|
                                     File.exists?(fname).should be_true
                                   },
+            +                     :running_time= => nil,
+            +                     :exit_status= => /10/,
+            +                     :memory_usage= => nil,
                                   :save => nil})
                 end
                 protected
                 def grader_should(args)
                   @user1 = stub(User,
                                 :id => 1, :login => 'user1')
                   problem = args[:on]
                   input_file = @config.test_request_input_base_dir + "/" + args[:with]
                   submission =
                     create_submission_from_file(1, @user1, args[:on], args[:grade])
                   test_request = stub(TestRequest,
                                       :id => 1,
                                       :user => @user1,
                                       :problem => problem,
                                       :submission => submission,
                                       :input_file_name => input_file,
                                       :language => submission.language,
                                       :problem_name => problem.name)
                   expectations = args[:and_report]
                   expectations.each do |key,val|
                     if val==nil
                       test_request.should_receive(key)
                     elsif val.class == Proc
                       test_request.should_receive(key) { |fname|
                         val.call(fname)
                       }
                     else
                       test_request.should_receive(key).with(val)
                     end
                   end
                   @engine.grade(test_request)
                 end
               end

Write
Preview

You need to be logged in to leave comments. Login now

	Site-wide shortcuts
/	Use quick search box
g h	Goto home page
g g	Goto my private gists page
g G	Goto my public gists page
n r	New repository page
n g	New gist page

	Repositories
g s	Goto summary page
g c	Goto changelog page
g f	Goto files page
g F	Goto files page with file search activated
g p	Goto pull requests page
g o	Goto repository settings
g O	Goto repository permissions settings

Sign in to your account

Author

r41:d4e37450ddbc - Thu, 03 Apr 2008 20:49:43 - 3 files changed: 37 inserted, 7 deleted

Sign in to your account

Commit r41:d4e37450ddbc public

Author

r41:d4e37450ddbc - Thu, 03 Apr 2008 20:49:43 - 3 files changed: 37 inserted, 7 deleted

Commit `r41:d4e37450ddbc` public