harvester
- pilot.util.harvester.findfile(path, name)[source]
find the first instance of file in the directory tree
- Parameters:
path – directory tree to search
name – name of the file to search
- Returns:
the path to the first instance of the file
- pilot.util.harvester.get_event_status_file(args)[source]
Return the name of the event_status.dump file as defined in the pilot config file and from the pilot arguments.
- Parameters:
args – Pilot arguments object.
- Returns:
event staus file name.
- pilot.util.harvester.get_initial_work_report()[source]
Prepare the work report dictionary. Note: the work_report should also contain all fields defined in parse_jobreport_data().
- Returns:
work report dictionary.
- pilot.util.harvester.get_job_request_file_name()[source]
Return the name of the job request file as defined in the pilot config file.
- Returns:
job request file name.
- pilot.util.harvester.get_worker_attributes_file(args)[source]
Return the name of the worker attributes file as defined in the pilot config file and from the pilot arguments.
- Parameters:
args – Pilot arguments object.
- Returns:
worker attributes file name.
- pilot.util.harvester.is_harvester_mode(args)[source]
Determine if the pilot is running in Harvester mode. :param args: Pilot arguments object. :return: Boolean.
- pilot.util.harvester.kill_worker()[source]
Create (touch) a kill_worker file in the pilot launch directory. This file will let Harverster know that the pilot has finished.
- Returns:
- pilot.util.harvester.parse_job_definition_file(filename)[source]
This function parses the Harvester job definition file and re-packages the job definition dictionaries. The format of the Harvester job definition dictionary is: dict = { job_id: { key: value, .. }, .. } The function returns a list of these dictionaries each re-packaged as dict = { key: value } (where the job_id is now one of the key-value pairs: ‘jobid’: job_id)
- Parameters:
filename – file name (string).
- Returns:
list of job definition dictionaries.
- pilot.util.harvester.publish_job_report(job, args, job_report_file='jobReport.json')[source]
Copy job report file to make it accessible by Harvester. Shrink job report file.
- Parameters:
job – job object.
args – Pilot arguments object.
job_report_file – name of job report (string).
- Raises:
FileHandlingFailure – in case of IOError.
:return True or False
- pilot.util.harvester.publish_stageout_files(job, event_status_file)[source]
Publishing of work report to file. The work report dictionary should contain the fields defined in get_initial_work_report().
- Parameters:
args – Pilot arguments object.
job – job object.
name (event status file)
- Returns:
Boolean. status of writing the file information to a json
- pilot.util.harvester.publish_work_report(work_report=None, worker_attributes_file='worker_attributes.json')[source]
Publishing of work report to file. The work report dictionary should contain the fields defined in get_initial_work_report().
- Parameters:
work_report – work report dictionary.
worker_attributes_file
- Raises:
FileHandlingFailure – in case of IOError.
- Returns:
True or False
- pilot.util.harvester.remove_job_request_file()[source]
Remove an old job request file when it is no longer needed.
- Returns:
- pilot.util.harvester.request_new_jobs(njobs=1)[source]
Inform Harvester that the pilot is ready to process new jobs by creating a job request file with the desired number of jobs.
- Parameters:
njobs – Number of jobs. Default is 1 since on grids and clouds the pilot does not know how many jobs it can
process before it runs out of time. :return: