cafe-grader-judge-scripts Commit

cafe-grader-judge-scripts

Commit `r210:3ae633914385` public

parent | child

Description:

Merge pull request #1 from jittat/master update from jittat/master

Commit status:

[Not Reviewed]

References:

merge default

Diff options:

| | | |

Comments:

0 Commit comments 0 Inline Comments

Unresolved TODOs:

There are no unresolved TODOs

Add another comment

r210:3ae633914385 - Fri, 06 Feb 2015 02:50:41 - 7 files changed: 14 inserted, 15 deleted

installer/install.sh ¶ +5 -6

               #!/bin/sh
               echo "This script will install and configure Cafe grader."
-            - echo "This will install Ruby 1.9.2 under rvm"
+            + RUBY_VERSION=2.1.2
+            + echo "This will install Ruby $RUBY_VERSION under RVM"
               echo "Installing required apts"
               sudo apt-get update
               sudo apt-get install mysql-server mysql-client \
                 g++ gcc apache2 libmysqlclient15-dev build-essential \
                 git-core openssl libreadline6 libreadline6-dev \
                 zlib1g zlib1g-dev libssl-dev libyaml-dev libsqlite3-dev \
                 sqlite3 libxml2-dev libxslt-dev autoconf libc6-dev \
                 ncurses-dev automake libtool bison subversion \
                 pkg-config curl nodejs unzip pyflakes ruby default-jdk
               echo "Installing RVM"
               curl -k -L https://get.rvm.io | bash -s stable
               source ~/.rvm/scripts/rvm
-            - echo "Installing Ruby 2.1.2 in RVM"
+            + echo "Installing Ruby $RUBY_VERSION in RVM"
-            - rvm install 2.1.2
+            + rvm install $RUBY_VERSION
-            - rvm use 2.1.2
+            + rvm use $RUBY_VERSION
               echo "Fetching Cafe Grader from Git repositories"
               echo "Fetching web interface"
               mkdir cafe_grader
               cd cafe_grader
               git clone -q git://github.com/jittat/cafe-grader-web.git web
               echo "Configuring rails app"
               cp web/config/application.rb.SAMPLE web/config/application.rb
               cp web/config/initializers/cafe_grader_config.rb.SAMPLE web/config/initializers/cafe_grader_config.rb
               #replace UTC in application.rb with the system timezone
               timezone='UTC'
               if [ -f '/etc/timezone' ]; then
                 timezone=\"`cat /etc/timezone`\"
               else
                 if [ -f '/etc/sysconfig/clock' ]; then
                   timezone=`grep -e '^TIMEZONE' /etc/sysconfig/clock | grep -o -e '\".*\"'`
                 fi
               fi
               replace="s!'UTC'!$timezone!g"
               sed -i $replace web/config/application.rb
               echo "At this point we will need MySQL user and database."
               echo "Have you created MySQL user and database for Cafe grader? (Y/N) "
               read ch
               if [ "$ch" = "n" -o "$ch" = "N" ]
               then
                 echo "Please open another terminal and create the user and database for Cafe grader."
                 echo "Don't forget to grant access to that database for the user."
                 echo "Please have username, password, and database name ready before continue."
                 echo
                 echo "The following are instructions:"
                 echo "1. Run mysql:"
                 echo
                 echo "      mysql -u root -p"
                 echo
                 echo "   if you have just installed mysql, the root password is the one that you have just entered"
                 echo "2. Create a new database, a new user, and grant access to grader database:"
                 echo
                 echo "      create user 'USERNAME'@'localhost' identified by 'PASSWORD';"
                 echo "      create database \`DATABASENEME\`;"
                 echo "      grant all on \`DATABASENAME\`.* to 'USERNAME'@'localhost';"
                 echo
                 echo "   Replace USERNAME, PASSWORD, and DATABASENAME accordingly."
                 echo
                 echo "Hit enter when ready..."
                 read dummy
               fi
               CAFE_PATH=`pwd`
               cd web
               echo "Please provide grader database:"
               read database
               echo "Please provide grader username:"
               read username
               echo "Please provide $username password:"
               read password
               echo "development:" > config/database.yml
               echo "  adapter: mysql2" >> config/database.yml
               echo "  encoding: utf8" >> config/database.yml
               echo "  reconnect: false" >> config/database.yml
               echo "  database: $database" >> config/database.yml
               echo "  pool: 5" >> config/database.yml
               echo "  username: $username" >> config/database.yml
               echo "  password: $password" >> config/database.yml
               echo "  host: localhost" >> config/database.yml
               echo "  socket: /var/run/mysqld/mysqld.sock" >> config/database.yml
               echo "" >> config/database.yml
               echo "production:" >> config/database.yml
               echo "  adapter: mysql2" >> config/database.yml
               echo "  encoding: utf8" >> config/database.yml
               echo "  reconnect: false" >> config/database.yml
               echo "  database: $database" >> config/database.yml
               echo "  pool: 5" >> config/database.yml
               echo "  username: $username" >> config/database.yml
               echo "  password: $password" >> config/database.yml
               echo "  host: localhost" >> config/database.yml
               echo "  socket: /var/run/mysqld/mysqld.sock" >> config/database.yml
               echo "Object.instance_eval{remove_const :GRADER_ROOT_DIR}" >> config/initializers/cafe_grader_config.rb
               echo "Object.instance_eval{remove_const :GRADING_RESULT_DIR}" >> config/initializers/cafe_grader_config.rb
               echo "GRADER_ROOT_DIR = '$CAFE_PATH/judge'" >> config/initializers/cafe_grader_config.rb
               echo "GRADING_RESULT_DIR = '$CAFE_PATH/judge/result'" >> config/initializers/cafe_grader_config.rb
               echo "Installing required gems"
               gem install bundler
               bundle install
               echo "Running rake tasks to initialize database"
               rake db:migrate
               rake db:seed
               echo "Running rake tasks to precompile the assets"
               rake assets:precompile
               echo "Intalling web interface complete..."
               echo
               echo "Fetching grader"
               cd ..
               mkdir judge
               cd judge
               git clone -q git://github.com/jittat/cafe-grader-judge-scripts.git scripts
               mkdir raw
               mkdir ev-exam
               mkdir ev
               mkdir result
               mkdir log
               echo "Configuring grader"
               cp scripts/config/env_exam.rb.SAMPLE scripts/config/env_exam.rb
               cp scripts/config/env_grading.rb.SAMPLE scripts/config/env_grading.rb
               # create new environment.rb file
               echo "RAILS_ROOT = '$CAFE_PATH/web'" > scripts/config/environment.rb
               echo "GRADER_ROOT = '$CAFE_PATH/judge/scripts'" >> scripts/config/environment.rb
               echo "require File.join(File.dirname(__FILE__),'../lib/boot')" >> scripts/config/environment.rb
               echo "require File.dirname(__FILE__) + \"/env_#{GRADER_ENV}.rb\"" >> scripts/config/environment.rb
               # compiling box
               MACHINE_TYPE=`uname -m`
               if [ ${MACHINE_TYPE} == 'x86_64' ]; then
                 gcc -std=c99 -o scripts/std-script/box scripts/std-script/box64-new.c
-            -   # 64-bit stuff here
               else
                 g++ -o scripts/std-script/box scripts/std-script/box.cc
-            -   # 32-bit stuff here
               fi
               cd ..
               echo "Now you are ready to run cafe grader...."
               echo
               echo "Try:"
               echo
               echo "  cd web"
               echo "  rails s"
               echo
               echo "and access web at http://localhost:3000/"
               echo "The root username is 'root', its password is 'ioionrails'."

new_problem ¶ +1 -1

-            - #!/usr/bin/ruby
+            + #!/usr/bin/env ruby
               # new_problem:
               #  * creates a directory for a problem in the current directory,
               #  * create standard testcase config file
               require 'erb'
               def process_options(options)
                 i = 2
                 while i<ARGV.length
                   if ARGV[i]=='-t'
                     options[:time_limit] = ARGV[i+1].to_i if ARGV.length>i+1
                     i += 1
                   end
                   if ARGV[i]=='-m'
                     options[:mem_limit] = ARGV[i+1].to_i if ARGV.length>i+1
                     i += 1
                   end
                   i += 1
                 end
               end
               puts "This script is out of dated, shall be fixed soon"
               puts "Right now, you can create raw_ev and import"
               exit(0)
               GRADER_DIR = File.dirname(__FILE__)
               # print usage
               if ARGV.length < 2
                 puts <<USAGE
               using: new_problem problem number_of_testcase [options]
               * creates a directory for a problem in the current directory,
               * create standard testcase config file
               * options: -t time-limit (in seconds)
                          -m mem-limit (in MB)
               USAGE
                 exit(127)
               end
               # processing arguments
               problem = ARGV[0]
               num_testcases = ARGV[1].to_i
               options = {:time_limit => 1,  :mem_limit => 16}
               process_options(options)
               # start working
               puts "creating directories"
               system("mkdir #{problem}")
               system("mkdir #{problem}/script")
               system("mkdir #{problem}/test_cases")
               puts "creating testcases directories"
 .upto(num_testcases) do |i|
                 system("mkdir #{problem}/test_cases/#{i}")
               end
               # generating all_tests.cfg
               puts "generating testcase config file"
               template = File.open(File.dirname(__FILE__) + "/templates/all_tests.cfg.erb").read
               all_test_cfg = ERB.new(template)
               cfg_file = File.open("#{problem}/test_cases/all_tests.cfg","w")
               cfg_file.puts all_test_cfg.result
               cfg_file.close
               puts "done"

templates/check_wrapper ¶ +1 -1

-            - #!/usr/bin/ruby
+            + #!/usr/bin/env ruby
               #
               # This is a check script wrapper.  It read all required information
               # and call a real check script call REAL_CHECK_SCRIPT in directory
               # [problem_home]/script
               #
               REAL_CHECK_SCRIPT = "<%= script_name %>"
               # The REAL_CHECK_SCRIPT is called with:
               #
               #  (script) <lang> <test-num> <in-file> <out-file> <ans-file> <full-score>
               #
               # and REAL_CHECK_SCRIPT's output to standard out is redirected to
               # 'check_result' as required by normal check script.
               problem_home = ENV['PROBLEM_HOME']
               require "#{problem_home}/script/test_dsl.rb"
               if ARGV.length < 2
                 puts "Usage: check <language> <test-number> [<output-file>]"
                 exit(0)
               end
               language = ARGV[0]
               test_num = ARGV[1].to_i
               if ARGV.length >= 3
                 output_file_name = ARGV[2]
               else
                 output_file_name = "output.txt"
               end
               load "#{problem_home}/test_cases/all_tests.cfg"
               problem = Problem.get_instance
               answer_file_name = "#{problem_home}/test_cases/#{test_num}/answer-#{test_num}.txt"
               input_file_name = "#{problem_home}/test_cases/#{test_num}/input-#{test_num}.txt"
               score = problem.get_score(test_num)
               cmd = "#{problem_home}/script/#{REAL_CHECK_SCRIPT} " +
                 "#{language} #{test_num} #{input_file_name} #{output_file_name} " +
                 "#{answer_file_name} #{score} > check_result"
               #puts "wrapper-CMD: #{cmd}"
               system(cmd)

test/data/ev/test_memory/script/check ¶ +1 -1

-            - #!/usr/bin/ruby
+            + #!/usr/bin/env ruby
               problem_home = ENV['PROBLEM_HOME']
               require "#{problem_home}/script/test_dsl.rb"
               if ARGV.length < 2
                 puts "Usage: check <language> <test-number> [<output-file>]"
                 exit(0)
               end
               language = ARGV[0]
               test_num = ARGV[1].to_i
               if ARGV.length >= 3
                 output_file_name = ARGV[2]
               else
                 output_file_name = "output.txt"
               end
               load "#{problem_home}/test_cases/all_tests.cfg"
               problem = Problem.get_instance
               output_file = File.new(output_file_name, "r")
               answer_file = File.new("#{problem_home}/test_cases/#{test_num}/answer-#{test_num}.txt")
               result_file = File.new("check_result", "w")
               output_file_content = output_file.read
               answer_file_content = answer_file.read
               report_correct = lambda {
                 result_file.write "Correct\n"
                 result_file.write problem.get_score(test_num)
                 result_file.write "\n"
                 result_file.close
                 exit(0)
               }
               report_wrong = lambda {
                 result_file.write "Incorrect\n"
                 result_file.write "0\n"
                 result_file.close
                 exit(0)
               }
               ##################
               # Your code here #
               ##################
               num_pattern = /^[0-9]*/
               if (output_file_content =~ num_pattern) == nil
                 report_wrong.call
               end
               output_i = output_file_content.to_i
               answer_i = answer_file_content.to_i
               if output_i == answer_i
                 report_correct.call
               else
                 report_wrong.call
               end

test/data/ev/test_normal/script/check ¶ +1 -1

-            - #!/usr/bin/ruby
+            + #!/usr/bin/env ruby
               problem_home = ENV['PROBLEM_HOME']
               require "#{problem_home}/script/test_dsl.rb"
               if ARGV.length < 2
                 puts "Usage: check <language> <test-number> [<output-file>]"
                 exit(0)
               end
               language = ARGV[0]
               test_num = ARGV[1].to_i
               if ARGV.length >= 3
                 output_file_name = ARGV[2]
               else
                 output_file_name = "output.txt"
               end
               load "#{problem_home}/test_cases/all_tests.cfg"
               problem = Problem.get_instance
               output_file = File.new(output_file_name, "r")
               answer_file = File.new("#{problem_home}/test_cases/#{test_num}/answer-#{test_num}.txt")
               result_file = File.new("check_result", "w")
               output_file_content = output_file.read
               answer_file_content = answer_file.read
               report_correct = lambda {
                 result_file.write "Correct\n"
                 result_file.write problem.get_score(test_num)
                 result_file.write "\n"
                 result_file.close
                 exit(0)
               }
               report_wrong = lambda {
                 result_file.write "Incorrect\n"
                 result_file.write "0\n"
                 result_file.close
                 exit(0)
               }
               ##################
               # Your code here #
               ##################
               num_pattern = /^[0-9]*/
               if (output_file_content =~ num_pattern) == nil
                 report_wrong.call
               end
               output_i = output_file_content.to_i
               answer_i = answer_file_content.to_i
               if output_i == answer_i
                 report_correct.call
               else
                 report_wrong.call
               end

test/data/ev/test_timeout/script/check ¶ +1 -1

-            - #!/usr/bin/ruby
+            + #!/usr/bin/env ruby
               problem_home = ENV['PROBLEM_HOME']
               require "#{problem_home}/script/test_dsl.rb"
               if ARGV.length < 2
                 puts "Usage: check <language> <test-number> [<output-file>]"
                 exit(0)
               end
               language = ARGV[0]
               test_num = ARGV[1].to_i
               if ARGV.length >= 3
                 output_file_name = ARGV[2]
               else
                 output_file_name = "output.txt"
               end
               load "#{problem_home}/test_cases/all_tests.cfg"
               problem = Problem.get_instance
               output_file = File.new(output_file_name, "r")
               answer_file = File.new("#{problem_home}/test_cases/#{test_num}/answer-#{test_num}.txt")
               result_file = File.new("check_result", "w")
               output_file_content = output_file.read
               answer_file_content = answer_file.read
               report_correct = lambda {
                 result_file.write "Correct\n"
                 result_file.write problem.get_score(test_num)
                 result_file.write "\n"
                 result_file.close
                 exit(0)
               }
               report_wrong = lambda {
                 result_file.write "Incorrect\n"
                 result_file.write "0\n"
                 result_file.close
                 exit(0)
               }
               ##################
               # Your code here #
               ##################
               num_pattern = /^[0-9]*/
               if (output_file_content =~ num_pattern) == nil
                 report_wrong.call
               end
               output_i = output_file_content.to_i
               answer_i = answer_file_content.to_i
               if output_i == answer_i
                 report_correct.call
               else
                 report_wrong.call
               end

test/engine_spec.rb ¶ +4 -4

                 end
                 it "should not allow malicious submission to see PROBLEM_HOME" do
                   problem_test_yesno =  stub(Problem,
                                               :id => 1, :name => 'test_yesno',
                                               :full_score => 10)
                   grader_should(:grade => "yesno_access_problem_home.c",
                                 :on => problem_test_yesno,
                                 :and_report => {
                                   :score => 0,
                                   :comment => /(-|x)/})
                 end
                 it "should not allow malicious submission to open files" do
                   problem_test_yesno =  stub(Problem,
                                               :id => 1, :name => 'test_yesno',
                                               :full_score => 10)
                   grader_should(:grade => "yesno_open_file.c",
                                 :on => problem_test_yesno,
                                 :and_report => {
                                   :score => 0,
                                   :comment => /(-|x)/})
                 end
                 def grader_should(args)
                   @user1 = stub(User,
                                 :id => 1, :login => 'user1')
                   submission =
                     create_submission_from_file(1, @user1, args[:on], args[:grade], args[:language])
                   submission.should_receive(:graded_at=)
                   expected_score = args[:and_report][:score]
                   expected_comment = args[:and_report][:comment]
                   if args[:and_report][:compiler_message]!=nil
                     expected_compiler_message = args[:and_report][:compiler_message]
                   else
                     expected_compiler_message = ''
                   end
                   submission.should_receive(:points=).with(expected_score)
                   submission.should_receive(:grader_comment=).with(expected_comment)
                   submission.should_receive(:compiler_message=).with(expected_compiler_message)
                   submission.should_receive(:save)
                   @engine.grade(submission)
                 end
                 protected
                 def create_normal_submission_mock_from_file(source_fname)
                   create_submission_from_file(1, @user_user1, @problem_test_normal, source_fname)
                 end
               end
               describe "A grader engine, when grading test requests" do
                 include GraderEngineHelperMethods
                 before(:each) do
                   @config = Grader::Configuration.get_instance
                   @engine = Grader::Engine.new(:room_maker => Grader::TestRequestRoomMaker.new,
                                                :reporter => Grader::TestRequestReporter.new)
                   init_sandbox
                 end
                 it "should report error if there is no problem template" do
                   problem = stub(Problem,
                                  :id => 1, :name => 'nothing')
                   grader_should(:grade => 'test1_correct.c',
                                 :on => problem,
                                 :with => 'in1.txt',
                                 :and_report => {
                                   :graded_at= => nil,
                                   :compiler_message= => '',
                                   :grader_comment= => '',
                                   :running_stat= => /template not found/,
                                   :running_time= => nil,
                                   :exit_status= => nil,
                                   :memory_usage= => nil,
                                   :save => nil})
                 end
                 it "should run test request and produce output file" do
                   problem = stub(Problem,
                                  :id => 1, :name => 'test_normal')
                   grader_should(:grade => 'test1_correct.c',
                                 :on => problem,
                                 :with => 'in1.txt',
                                 :and_report => {
                                   :graded_at= => nil,
                                   :compiler_message= => '',
                                   :grader_comment= => '',
                                   :running_stat= => /0.0\d* sec./,
                                   :output_file_name= => lambda { |fname|
-            -                       File.exists?(fname).should be_true
+            +                       File.exists?(fname).should be true
                                   },
                                   :running_time= => nil,
                                   :exit_status= => nil,
                                   :memory_usage= => nil,
                                   :save => nil})
                 end
                 it "should clean up problem directory after running test request" do
                   problem = stub(Problem,
                                  :id => 1, :name => 'test_normal')
                   grader_should(:grade => 'test1_correct.c',
                                 :on => problem,
                                 :with => 'in1.txt',
                                 :and_report => {
                                   :graded_at= => nil,
                                   :compiler_message= => '',
                                   :grader_comment= => '',
                                   :running_stat= => nil,
                                   :output_file_name= => nil,
                                   :running_time= => nil,
                                   :exit_status= => nil,
                                   :memory_usage= => nil,
                                   :save => nil})
-            -     File.exists?(@config.user_result_dir + "/test_request/test_normal/test_cases/1/input-1.txt").should be_false
+            +     File.exists?(@config.user_result_dir + "/test_request/test_normal/test_cases/1/input-1.txt").should be false
                 end
                 it "should compile test request with error and report compilation error" do
                   problem = stub(Problem,
                                  :id => 1, :name => 'test_normal')
                   grader_should(:grade => 'test1_compile_error.c',
                                 :on => problem,
                                 :with => 'in1.txt',
                                 :and_report => {
                                   :graded_at= => nil,
                                   :compiler_message= => /.+/,
                                   :grader_comment= => /[Cc]ompil.*error/,
                                   :running_stat= => '',
                                   :save => nil})
                 end
                 it "should report exit status" do
                   problem = stub(Problem,
                                  :id => 1, :name => 'test_normal')
                   grader_should(:grade => 'add_nonzero_exit_status.c',
                                 :on => problem,
                                 :with => 'in1.txt',
                                 :and_report => {
                                   :graded_at= => nil,
                                   :compiler_message= => '',
                                   :grader_comment= => '',
                                   :running_stat= => /[Ee]xit.*status.*10.*0\.0\d* sec/m,
                                   :output_file_name= => lambda { |fname|
-            -                       File.exists?(fname).should be_true
+            +                       File.exists?(fname).should be true
                                   },
                                   :running_time= => nil,
                                   :exit_status= => /10/,
                                   :memory_usage= => nil,
                                   :save => nil})
                 end
                 it "should produce running statistics for normal submission" do
                   problem = stub(Problem,
                                  :id => 1, :name => 'test_normal')
                   grader_should(:grade => 'test_run_stat.c',
                                 :on => problem,
                                 :with => 'in1.txt',
                                 :and_report => {
                                   :graded_at= => nil,
                                   :compiler_message= => '',
                                   :grader_comment= => '',
                                   :running_stat= => nil,
                                   :output_file_name= => lambda { |fname|
-            -                       File.exists?(fname).should be_true
+            +                       File.exists?(fname).should be true
                                   },
                                   :running_time= => lambda { |t|
                                     (t>=0.14) and (t<=0.16)
                                   },
                                   :exit_status= => nil,
                                   :memory_usage= => lambda { |s|
                                     (s>=500) and (s<=1000)
                                   },
                                   :save => nil})
                 end
                 protected
                 def grader_should(args)
                   @user1 = stub(User,
                                 :id => 1, :login => 'user1')
                   problem = args[:on]
                   input_file = @config.test_request_input_base_dir + "/" + args[:with]
                   submission =
                     create_submission_from_file(1, @user1, args[:on], args[:grade])
                   test_request = stub(TestRequest,
                                       :id => 1,
                                       :user => @user1,
                                       :problem => problem,
                                       :submission => submission,
                                       :input_file_name => input_file,
                                       :language => submission.language,
                                       :problem_name => problem.name)
                   expectations = args[:and_report]
                   expectations.each do |key,val|
                     if val==nil
                       test_request.should_receive(key)
                     elsif val.class == Proc
                       test_request.should_receive(key) { |fname|
                         val.call(fname)
                       }
                     else
                       test_request.should_receive(key).with(val)
                     end
                   end
                   @engine.grade(test_request)
                 end
               end

Write
Preview

You need to be logged in to leave comments. Login now

	Site-wide shortcuts
/	Use quick search box
g h	Goto home page
g g	Goto my private gists page
g G	Goto my public gists page
n r	New repository page
n g	New gist page

	Repositories
g s	Goto summary page
g c	Goto changelog page
g f	Goto files page
g F	Goto files page with file search activated
g p	Goto pull requests page
g o	Goto repository settings
g O	Goto repository permissions settings

Sign in to your account

Author

r210:3ae633914385 - Fri, 06 Feb 2015 02:50:41 - 7 files changed: 14 inserted, 15 deleted

Sign in to your account

Commit r210:3ae633914385 public

Author

r210:3ae633914385 - Fri, 06 Feb 2015 02:50:41 - 7 files changed: 14 inserted, 15 deleted

Commit `r210:3ae633914385` public