cafe-grader-judge-scripts Files

cafe-grader-judge-scripts

Location: cafe-grader-judge-scripts/grader - annotation

Commit Description:

better all sub progres report

Commit Description:

better all sub progres report

References:

r271:2f4b035474b5 tip default

File last commit:

r240:700050c502ba java

Show/Diff file:

Action:

History | Source | Raw | Download

Jittat Fakcharoenphol fixed grader to work with ruby 1.9	r137	#!/usr/bin/env ruby
jittat import original files...	r0
jittat [grader] process-wise stop file...	r24	def stop_grader(id)
		if id==:all
		File.open(File.dirname(__FILE__) + "/stop.all",'w').close
		else
		File.open(File.dirname(__FILE__) + "/stop.#{id}",'w').close
		end
jittat add grader stop, fix ARGV bug...	r6	end

		def check_stopfile
jittat [grader] process-wise stop file...	r24	FileTest.exist?(File.dirname(__FILE__) + "/stop.all") or
		FileTest.exist?(File.dirname(__FILE__) + "/stop.#{Process.pid}")
jittat add grader stop, fix ARGV bug...	r6	end

		def clear_stopfile
jittat [grader] process-wise stop file...	r24	if FileTest.exist?(File.dirname(__FILE__) + "/stop.#{Process.pid}")
Jittat Fakcharoenphol removed 'system' from grader, grader_id, import_problem	r99	File.delete(File.dirname(__FILE__) + "/stop.#{Process.pid}")
jittat [grader] process-wise stop file...	r24	end
jittat import original files...	r0	end
jittat add grader stop, fix ARGV bug...	r6
jittat Merged new-arch-branch changes 61:73 into the trunk...	r22	def config
		Grader::Configuration.get_instance
		end

jittat [grader] [MERGED] Merged new-arch-branch changes 74:105 into the trunk...	r23	def log_file_name
jittat [grader] raise when log directory does not exist...	r25	if !File.exists?(config.log_dir)
		raise "Log directory does not exist: #{config.log_dir}"
		end
jittat [grader] [MERGED] Merged new-arch-branch changes 74:105 into the trunk...	r23	config.log_dir +
		"/#{GRADER_ENV}_#{config.grader_mode}.#{Process.pid}"
		end

		def log(str)
jittat Merged new-arch-branch changes 61:73 into the trunk...	r22	if config.talkative
		puts str
		end
jittat [grader] [MERGED] Merged new-arch-branch changes 74:105 into the trunk...	r23	if config.logging
		fp = File.open(log_file_name,"a")
		fp.puts("GRADER: #{Time.new.strftime("%H:%M")} #{str}")
		fp.close
		end
		end

		def display_manual
		puts <<USAGE
		Grader.
		using: (1) grader
admin add options for grading all submissions of a specific problem	r163	(2) grader environment [mode] [options]
jittat [grader] process-wise stop file...	r24	(3) grader stop [all\|pids-list]
jittat [grader] [MERGED] Merged new-arch-branch changes 74:105 into the trunk...	r23	(4) grader --help
		(1) call grader with environment = 'exam', mode = 'queue'
Jittat Fakcharoenphol refactored grader, added autonew mode	r121	(2) possible modes are: 'queue', 'test_request', 'prob', 'sub', 'contest', and 'autonew'
admin add options for grading all submissions of a specific problem	r163	queue: repeatedly check the task queue and grade any available tasks

		prob: re-grade every user latest submission of the specific problem.
		the problem name must be specified by the next argument.

		additional options:
admin add --only-error to grader script	r240	--all-sub re-grade every submissions instead of just the latest submission of each user.
		--only-error re-grade only submissions that are "error during grading"
admin add options for grading all submissions of a specific problem	r163
		sub: re-grader the specified submission.
		The submission ID to be re-graded must be specified by the next argument.

admin add more logging	r174	options:
		--err-log log error to a file in the log dir

jittat [grader] [MERGED] Merged new-arch-branch changes 74:105 into the trunk...	r23	(3) create stop-file to stop running grader in queue mode
		(4) You are here.
		USAGE
jittat Merged new-arch-branch changes 61:73 into the trunk...	r22	end

Jittat Fakcharoenphol added --dry option to grader prob mode	r91	def process_options_and_stop_file
		# The list of options are:
		# - stop [all\|process ids]
		# -

		# Process 'help' option
		if (ARGV.length==1) and (/help/.match(ARGV[0]))
		display_manual
		exit(0)
		end

		# Process 'stop' option.
		if (ARGV.length >= 1) and (ARGV[0]=='stop')
		if ARGV.length==1
		puts "you should specify pid-list or 'all'"
		display_manual
		elsif (ARGV.length==2) and (ARGV[1]=='all')
		stop_grader(:all)
		puts "A global stop file ('stop.all') created."
		puts "You should remove it manually later."
		else
		(1..ARGV.length-1).each do \|i\|
		stop_grader(ARGV[i])
		end
		puts "stop file(s) created"
		end
		exit(0)
		end

		# Check stop file.
		if check_stopfile
		puts "Stop file exists. Terminated."
		clear_stopfile
		exit(0)
		end

		#default options
		options = {
		:mode => 'queue',
		:environment => 'exam',
		:dry_run => false,
		}

		# Process mode and environment option
		if ARGV.length >= 1
		options[:environment] = ARGV.shift
		if ARGV.length >=1
		options[:mode] = ARGV.shift
		end
admin add options for grading all submissions of a specific problem	r163	else
		puts 'no argument specified, using default mode and environment.'
Jittat Fakcharoenphol added --dry option to grader prob mode	r91	end

		options[:dry_run] = (ARGV.delete('--dry') != nil)
Jittat Fakcharoenphol refactored grader, added autonew mode	r121	if options[:dry_run] and (not ['prob','contest','autonew'].include? options[:mode])
Jittat Fakcharoenphol only grades users in the contest	r94	puts "Dry run currently works only for 'prob' or 'contest' modes."
Jittat Fakcharoenphol added grading report	r92	exit(0)
		end

		options[:report] = (ARGV.delete('--report') != nil)
Jittat Fakcharoenphol added after_save_hook to ResultCollector	r122	if options[:report] and (not ['prob','contest','autonew'].include? options[:mode])
Jittat Fakcharoenphol only grades users in the contest	r94	puts "Report currently works only for 'prob' or 'contest' modes."
Jittat Fakcharoenphol added --dry option to grader prob mode	r91	exit(0)
		end

admin add options for grading all submissions of a specific problem	r163	options[:all_sub] = (ARGV.delete('--all-sub') != nil)
admin add --only-error to grader script	r240	options[:only_err] = (ARGV.delete('--only-error') != nil)
admin add options for grading all submissions of a specific problem	r163
admin add more logging	r174	options[:err_log] = (ARGV.delete('--err-log') != nil)

Jittat Fakcharoenphol added --dry option to grader prob mode	r91	return options
jittat [grader] [MERGED] Merged new-arch-branch changes 74:105 into the trunk...	r23	end

Jittat Fakcharoenphol added grading report	r92	class ResultCollector
		def initialize
		@results = {}
		@problems = {}
		@users = {}
		end

Jittat Fakcharoenphol sends whole submission to result collecter in submission reporter, instead of just user and problem	r123	def after_save_hook(submission, grading_result)
Jittat Fakcharoenphol added after_save_hook to ResultCollector	r122	end

Jittat Fakcharoenphol sends whole submission to result collecter in submission reporter, instead of just user and problem	r123	def save(submission, grading_result)
		user = submission.user
		problem = submission.problem
Jittat Fakcharoenphol added grading report	r92	if not @problems.has_key? problem.id
		@problems[problem.id] = problem
		end
		if not @users.has_key? user.id
		@users[user.id] = user
		end
		@results[[user.id, problem.id]] = grading_result
Jittat Fakcharoenphol added after_save_hook to ResultCollector	r122
Jittat Fakcharoenphol sends whole submission to result collecter in submission reporter, instead of just user and problem	r123	after_save_hook(submission, grading_result)
Jittat Fakcharoenphol added grading report	r92	end

		def print_report_by_user
		puts "---------------------"
		puts " REPORT"
		puts "---------------------"

Jittat Fakcharoenphol only grades users in the contest	r94	print "login,email"
Jittat Fakcharoenphol added grading report	r92	@problems.each_value do \|problem\|
		print ",#{problem.name}"
		end
		print "\n"

		@users.each_value do \|user\|
Jittat Fakcharoenphol only grades users in the contest	r94	print "#{user.login},#{user.email}"
Jittat Fakcharoenphol added grading report	r92	@problems.each_value do \|problem\|
		if @results.has_key? [user.id, problem.id]
		print ",#{@results[[user.id,problem.id]][:points]}"
		else
		print ","
		end
		end
		print "\n"
		end
		end
		end

Jittat Fakcharoenphol refactored grader, added autonew mode	r121	def grader_general_loop(engine, grader_proc, options)
		runner = Grader::Runner.new(engine, grader_proc)
		while true

		if check_stopfile # created by calling grader stop
		clear_stopfile
		log "stopped (with stop file)"
		break
		end

		task = yield(runner)

		if task==nil
		sleep(1)
		end
		end
		end

		def grader_queue_loop(grader_proc, options)
		log "Grader: queue"
		engine = Grader::Engine.new
		grader_general_loop(engine, grader_proc, options) do \|runner\|
		runner.grade_oldest_task
		end
		end

		def grader_test_request_loop(grader_proc, options)
		log "Grader: test_request"
		engine = Grader::Engine.new(:room_maker => Grader::TestRequestRoomMaker.new,
		:reporter => Grader::TestRequestReporter.new)
		grader_general_loop(engine, grader_proc, options) do \|runner\|
		runner.grade_oldest_test_request
		end
		end

		def grader_autonew_loop(grader_proc, options)
		log "Grader: autonew"

Jittat Fakcharoenphol added after_save_hook to ResultCollector	r122	if options[:report]
		result_collector = ResultCollector.new
		else
		result_collector = nil
		end

Jittat Fakcharoenphol refactored grader, added autonew mode	r121	if options[:dry_run]
		puts "Running in dry mode"
		end

Jittat Fakcharoenphol added after_save_hook to ResultCollector	r122	prob_reporter = Grader::SubmissionReporter.new(:dry_run => options[:dry_run],
		:result_collector => result_collector)

Jittat Fakcharoenphol refactored grader, added autonew mode	r121	engine = Grader::Engine.new(:reporter => prob_reporter)
		runner = Grader::Runner.new(engine, grader_proc)

		grader_proc.report_active if grader_proc!=nil

		latest_submitted_at = nil
		graded_submission_ids = {}

		while true

		if check_stopfile # created by calling grader stop
		clear_stopfile
		log "stopped (with stop file)"
		break
		end

		if latest_submitted_at==nil
		submissions = Submission.all
		else
		submissions = Submission.all(:conditions => ["submitted_at >= :latest",
		{:latest => latest_submitted_at}])
		end

		graded_any = false

		if submissions.length != 0
		submissions.each do \|submission\|
Jittat Fakcharoenphol skips bad submissions in autonew mode	r133	if (submission.problem == nil) or (!submission.problem.available)
		next
		end
Jittat Fakcharoenphol refactored grader, added autonew mode	r121	if ! graded_submission_ids[submission.id]
		runner.grade_submission(submission)
		graded_submission_ids[submission.id] = true
		if (!latest_submitted_at or
		latest_submitted_at < submission.submitted_at)
		latest_submitted_at = submission.submitted_at
		end
		puts "graded: #{submission.id}"
		puts "latest: #{latest_submitted_at}"
		graded_any = true
		end
		end
		end

		if ! graded_any
		sleep(1)
		end
		end
		end

		def grader_grade_problems(grader_proc, options)
		if options[:report]
		result_collector = ResultCollector.new
		else
		result_collector = nil
		end

		if options[:dry_run]
		puts "Running in dry mode"
		end

		prob_reporter = Grader::SubmissionReporter.new(:dry_run => options[:dry_run],
		:result_collector => result_collector)
		engine = Grader::Engine.new(:reporter => prob_reporter)
		runner = Grader::Runner.new(engine, grader_proc)

		grader_proc.report_active if grader_proc!=nil

		ARGV.each do \|prob_name\|
		prob = Problem.find_by_name(prob_name)
		if prob==nil
		puts "cannot find problem: #{prob_name}"
		else
admin add options for grading all submissions of a specific problem	r163	runner.grade_problem(prob,options)
Jittat Fakcharoenphol refactored grader, added autonew mode	r121	end
		end

		if options[:report]
		result_collector.print_report_by_user
		end
		end

		def grader_grade_contests(grader_proc, options)
		# always use dry run when grading during contest
		dry_run = options[:dry_run] = true

		contest_name = ARGV.shift

		contest = Contest.find_by_name(contest_name)
		if contest==nil
		puts "cannot find contest: #{contest_name}"
		exit(0)
		end

		if options[:report]
		result_collector = ResultCollector.new
		else
		result_collector = nil
		end

		if options[:dry_run]
		puts "Running in dry mode"
		end

		prob_reporter = Grader::SubmissionReporter.new(:dry_run => dry_run,
		:result_collector => result_collector)
		engine = Grader::Engine.new(:reporter => prob_reporter)
		runner = Grader::Runner.new(engine, grader_proc)

		grader_proc.report_active if grader_proc!=nil

		contest.problems.each do \|problem\|
		puts "Grading: #{problem.name}"
		runner.grade_problem(problem,
		:user_conditions => lambda do \|u\|
		u.contest_finished? and
		u.contest_ids.include?(contest.id)
		end)
		end

		if options[:report]
		result_collector.print_report_by_user
		end
		end

		def grader_grade_submissions(grader_proc, options)
		engine = Grader::Engine.new
		runner = Grader::Runner.new(engine, grader_proc)

		grader_proc.report_active if grader_proc!=nil

		ARGV.each do \|sub_id\|
		puts "Grading #{sub_id}"
		begin
		submission = Submission.find(sub_id.to_i)
		rescue ActiveRecord::RecordNotFound
admin handle the case when problem id or submission id is null. Grader will simply skip such request. Add more report on console (for command line grading)	r150	puts "Submission #{sub_id} not found"
Jittat Fakcharoenphol refactored grader, added autonew mode	r121	submission = nil
		end

		if submission!=nil
		runner.grade_submission(submission)
		end
		end
		end

Jittat Fakcharoenphol added grading report	r92	#########################################
		# main program
		#########################################

Jittat Fakcharoenphol added --dry option to grader prob mode	r91	options = process_options_and_stop_file
		GRADER_ENV = options[:environment]
		grader_mode = options[:mode]
		dry_run = options[:dry_run]
jittat add grader stop, fix ARGV bug...	r6
		puts "environment: #{GRADER_ENV}"
admin add options for grading all submissions of a specific problem	r163	puts "grader mode: #{grader_mode}"
jittat Merged new-arch-branch changes 61:73 into the trunk...	r22	require File.join(File.dirname(__FILE__),'config/environment')
jittat import original files...	r0
jittat [grader] [MERGED] Merged new-arch-branch changes 74:105 into the trunk...	r23	# add grader_mode to config
		# this is needed because method log needs it. TODO: clean this up
		class << config
		attr_accessor :grader_mode
		end
		config.grader_mode = grader_mode

		# reading rails environment
		log 'Reading rails environment'
jittat import original files...	r0
jittat Merged new-arch-branch changes 61:73 into the trunk...	r22	RAILS_ENV = config.rails_env
		require RAILS_ROOT + '/config/environment'

jittat [grader] [MERGED] Merged new-arch-branch changes 74:105 into the trunk...	r23	# register grader process
jittat Merged new-arch-branch changes 61:73 into the trunk...	r22	if config.report_grader
		grader_proc = GraderProcess.register(config.grader_hostname,
jittat add task queue access control, grader status...	r20	Process.pid,
		grader_mode)
		else
		grader_proc = nil
		end

jittat [grader] [MERGED] Merged new-arch-branch changes 74:105 into the trunk...	r23	#set loggin environment
		ENV['GRADER_LOGGING'] = log_file_name
admin add more logging	r174	if options[:err_log]
		err_file_name = log_file_name + '.err'
		$stderr.reopen(err_file_name,"a")
		log "STDERR log to file [#{err_file_name}]"
admin fix bug	r176	warn "start logging for grader PID #{Process.pid} on #{Time.now.in_time_zone}"
admin add more logging	r174	end

jittat [grader] [MERGED] Merged new-arch-branch changes 74:105 into the trunk...	r23
jittat added termination reporting to grader (using at_exit)...	r70	# register exit handler to report inactive, and terminated
		at_exit do
		if grader_proc!=nil
		grader_proc.report_inactive
		grader_proc.terminate
		end
		end

jittat [grader] [MERGED] Merged new-arch-branch changes 74:105 into the trunk...	r23	#
		# MAIN LOOP
		#
jittat Merged new-arch-branch changes 61:73 into the trunk...	r22
jittat add warning for check...	r9	case grader_mode
Jittat Fakcharoenphol refactored grader, added autonew mode	r121	when "queue"
		grader_queue_loop(grader_proc, options)
jittat [grader] [MERGED] Merged new-arch-branch changes 74:105 into the trunk...	r23
Jittat Fakcharoenphol refactored grader, added autonew mode	r121	when "test_request"
		grader_test_request_loop(grader_proc, options)

jittat add warning for check...	r9	when "prob"
Jittat Fakcharoenphol refactored grader, added autonew mode	r121	grader_grade_problems(grader_proc, options)
Jittat Fakcharoenphol added grading report	r92
Jittat Fakcharoenphol added contest mode	r93	when "contest"
Jittat Fakcharoenphol refactored grader, added autonew mode	r121	grader_grade_contests(grader_proc, options)
Jittat Fakcharoenphol added contest mode	r93
jittat [grader] +grade submission...	r61	when "sub"
Jittat Fakcharoenphol refactored grader, added autonew mode	r121	grader_grade_submissions(grader_proc, options)
jittat [grader] +grade submission...	r61
Jittat Fakcharoenphol refactored grader, added autonew mode	r121	when "autonew"
		grader_autonew_loop(grader_proc, options)
Jittat Fakcharoenphol added contest mode	r93
jittat [grader] [MERGED] Merged new-arch-branch changes 74:105 into the trunk...	r23	else
		display_manual
		exit(0)
jittat import original files...	r0	end

	Site-wide shortcuts
/	Use quick search box
g h	Goto home page
g g	Goto my private gists page
g G	Goto my public gists page
n r	New repository page
n g	New gist page

	Repositories
g s	Goto summary page
g c	Goto changelog page
g f	Goto files page
g F	Goto files page with file search activated
g p	Goto pull requests page
g o	Goto repository settings
g O	Goto repository permissions settings

Sign in to your account

Last Author