Registry.Registry

Registry for defining and executing computational jobs.

Usage

Registry.Registry(
    path,
    overwrite=False,
    readonly=False,
    backend=None,
)

Jobs can be registered with arbitrary parameters and subsequently executed using a configurable execution backend. The registry keeps track of completed and failed jobs and provides methods for inspecting their results.

Parameters

path: str: Path to registry folder in the file system.
overwrite: bool = False: Shall the registry folder be overwriten if it already exists? Default is False.
readonly: bool = False: Is the registry in read-only mode? In read-only mode jobs cannot be added anymore. Default is False.
backend: RunnerBackend | None = None: Instance of a runner backend. Default is an instance of the JoblibRunnerBackend.

Attributes

path: str: Path to registry folder in the file system.
overwrite: bool: Shall the registry folder be overwriten if it already exists?
job_collection: list[Job]: List of Jobs. Note that the first element is a dummy (always None) since jobs are numbered with natural numbers.
njobs: int: Number of jobs.
max_job_id: int: Highest sequential job number assigned so far.
readonly: bool: Is the registry in read-only mode? In read-only mode jobs cannot be added anymore.
design_path: str: path to experimental design file in the registry.
backend: RunnerBackend: Instance of a runner backend. Default is an instance of the JoblibRunnerBackend.

Methods

Name	Description
__repr__()	Return official string representation, which is aimed at programmers as they develop and maintain a program.
__str__()	Return informal readable string representation.
add_jobs()	Adds jobs via a full factorial design by default.
clear_logs()	Clears job logs.
get_done()	Calculate job IDs of done/finished jobs.
get_failed()	Calculate job IDs of failed jobs.
get_initialised()	Calculate job IDs of initialised jobs.
get_job()	Return the job with the respective job ID.
get_jobs()	Return jobs.
get_results()	Return job results.
get_running()	Calculate job IDs of running jobs.
load()	Load registry from file system.
load_design()	Read setup parameters from a CSV file.
pyrallex()	Run jobs in parallel.
reset_jobs()	Resets jobs.
set_backend()	Set runner backend.
show_logs()	Display job logs to stdout.
size()	Return the number of jobs.
tag()	Append tags to jobs.
touch()	Update job statuses.

repr()

Return official string representation, which is aimed at programmers as they develop and maintain a program.

Usage

Source

__repr__()

Returns

str: Technical string representation.

str()

Return informal readable string representation.

Usage

Source

__str__()

Returns

str: Human-readbale string representation.

add_jobs()

Adds jobs via a full factorial design by default.

Usage

Source

add_jobs(predicate=lambda _: True, **kwargs)

Optionally, arbitrary configurations can be excluded via a filter function.

Parameters

**kwargs: dict[str, any]: dictionary of keyword arguments.
predicate: Callable[[dict[str, any]], bool] = lambda _: True: Optional function that allows to ignore certain
and to return a Boolean: True if the parameter combination shall be kept and False if it is meant to be discarded.

Returns

Self: The callee itself (enables chaining).

clear_logs()

Clears job logs.

Usage

Source

clear_logs(jobids=None)

Parameters

jobids: list[int] = None: An optional list of job IDs. If None (the default) all job IDs are used.

Returns

Self: The callee itself (enables chaining).

get_done()

Calculate job IDs of done/finished jobs.

Usage

Source

get_done(jobids=None)

Parameters

jobids: list[int] = None: An optional list of job IDs. If None (the default) all job IDs are used.

Returns

list[int]: An integer list of job IDs.

get_failed()

Calculate job IDs of failed jobs.

Usage

Source

get_failed(jobids=None)

Parameters

jobids: list[int] = None: An optional list of job IDs. If None (the default) all job IDs are used.

Returns

list[int]: An integer list of job IDs.

get_initialised()

Calculate job IDs of initialised jobs.

Usage

Source

get_initialised(jobids=None)

Parameters

jobids: list[int] = None: An optional list of job IDs. If None (the default) all job IDs are used.

Returns

list[int]: An integer list of job IDs.

get_job()

Return the job with the respective job ID.

Usage

Source

get_job(jobid)

Parameters

jobid: int: The ID of the job starting at 1 (not 0).

Returns

Job: The respective ‘Job’ object.

get_jobs()

Return jobs.

Usage

Source

get_jobs(jobids=None, tags=None)

Parameters

jobids: list[int] = None: An optional list of job IDs. If None (the default) all job IDs are used.
tags: list[str] = None: An optional list of tags. Defaults to no tags at all.

Returns

list[Job]: A list of Job objects.

get_results()

Return job results.

Usage

Source

get_results(jobids, filter_none=True, simplify=False)

Parameters

jobids: list[int]: List of integer job IDs, e.g., of finished jobs.
filter_none: bool = True: If True jobs that did not return anything are skipped. Default is True.
simplify: bool = False: Should the result returned per job be simplified to a single dictionary? Default is False.

Returns

list[tuple[int, dict[str, any], dict[str, any]]] | list[dict[str, any]]: A list of 3-tuples (jobid, dictionary of parameters, dictionary of results) or a list of “merged” dictionaries if
the argument simplify is set to True.

get_running()

Calculate job IDs of running jobs.

Usage

Source

get_running(jobids=None)

Parameters

jobids: list[int] = None: An optional list of job IDs. If None (the default) all job IDs are used.

Returns

list[int]: An integer list of job IDs.

load()

Load registry from file system.

Usage

Source

load(path, readonly=True)

Parameters

path: str: Path to registry folder in the file system.
readonly: bool = True: Load in read-only mode? Default is True.

Returns

: A registry object.

load_design()

Read setup parameters from a CSV file.

Usage

Source

load_design(path, sep=",")

The file-format expected is a Comma Separated Values (CSV) file. The first line is interpreted as the header line with the experimental parameter names. Each following line contains the parameter values for one experiment.

Parameters

path: str: Path to the design file.
sep: str = ",": Seperator used in the CSV-file to separate columns. Default is ‘,’.

Returns

Self: The callee itself (enables chaining).

pyrallex()

Run jobs in parallel.

Usage

Source

pyrallex(runner, jobids=None, batchsize=None, shuffle=True, rerun=False)

Parameters

runner: Callable[[int, dict[any]], bool]: The actual runner function. It expects exactly two parameters:
jobids: list[int] | None = None: Optional list of job IDs. If None (the default), all job IDs are used.
batchsize: int | None = None: Size of job batches. If None (the default) the value is set via a heuristic as follows. The
shuffle: bool = True: Shall jobs be shuffled randomly prior to execution? Default is True. Setting this value to True is
rerun: bool = False: Shall already finished jobs be re-run? Default is False, i.e., finished jobs are skipped.

Returns

tuple[int, dict[str, any], dict[str, any]]: A list of 3-tuples (jobid, dictionary of parameters, dictionary of results).

reset_jobs()

Resets jobs.

Usage

Source

reset_jobs(jobids)

I.e., all results will be deleted and the job status will be reset to ‘initialised’.

Parameters

jobids: list[int]: A list of integer job IDs.

Returns

Self: The callee itself (enables chaining).

set_backend()

Set runner backend.

Usage

Source

set_backend(backend)

Parameters

backend: RunnerBackend: Instance of a sub-class of ‘RunnerBackend’.

Returns

Self: The callee itself (enables chaining).

show_logs()

Display job logs to stdout.

Usage

Source

show_logs(jobids=None)

The function goes through all log files and prints the content to stdout as a side effect.

    Args:
jobids: An optional list of job IDs. If ``None`` (the default) all job IDs are used.

Returns

Self: The callee itself (enables chaining).

size()

Return the number of jobs.

Usage

Source

size()

tag()

Append tags to jobs.

Usage

Source

tag(jobids, tags)

Parameters

jobids: list[int]: list of job IDs.
tags: list[str]: list of tags, i.e., simple strings.

Returns

Self: The callee itself (enables chaining).

touch()

Update job statuses.

Usage

Source

touch()

The function goes through re-loads all jobs from their pickle files. In particular the job status is loaded.

Returns

Self: The callee itself (enables chaining).