utopya.model_registry package

This submodule implements a registry of Utopia models, which is used to provide the required model information to the frontend and use it throughout.

utopya.model_registry.MODELS = <utopya.model_registry.registry.ModelRegistry object>

The model registry

Submodules

utopya.model_registry._registration module

Tools for model registration

utopya.model_registry._registration.register_models_from_list(*, registry: utopya.model_registry.registry.ModelRegistry, separator: str, model_names: str, executables: str, label: str, more_paths: dict = {}, source_dirs: str = None, exists_action: str = 'raise', set_as_default: bool = None, project_name: str = None, _log=<DantroLogger utopya.model_registry._registration (REMARK)>, **shared_bundle_kwargs)

Handles registration of multiple models where the model names, executables, and source directories are splittable lists of equal lengths.

Parameters:

• registry (utopya.model_registry.registry.ModelRegistry) – The model registry to store the models in

• separator (str) – Separation string to split model_names, executables, and source_dirs.

• model_names (str) – Splittable string of model names

• executables (str) – Splittable string of executables

• label (str) – Label under which to add the entries

• more_paths (dict, optional) – Additional paths that are to be parsed

• source_dirs (str, optional) – Splittable string of model source directories

• exists_action (str, optional) – Action to take upon existing label

• project_name (str, optional) – The associated project name

• _log (logging.Logger, optional) – A logger-like object

• **shared_bundle_kwargs – passed on to bundle creation

utopya.model_registry.entry module

This module implements the ModelRegistryEntry class, which contains a set of ModelInfoBundle objects. These can be combined in a labelled and unlabelled fashion.

class utopya.model_registry.entry.ModelRegistryEntry(model_name: str, *, registry_dir: str)

Bases: object

Holds model config bundles for a single model.

Instances of this class are associated with a file in the model registry directory. Upon change, they update that file.

__init__(model_name: str, *, registry_dir: str)

Initialize a model registry entry from a registry directory.

Expects a file named <model_name>.yml in the registry directory. If it does not exist, creates an empty one.

property model_name: str

Name of this model

property registry_dir: str

The associated registry directory

property registry_file_path: str

The absolute path to the registry file

property default_label: Optional[str]

The default label

property default_bundle: ModelInfoBundle

Returns the default bundle, if set; raises if not

__len__() → int

Returns number of registered bundles

__contains__(other: Union[ModelInfoBundle, str]) → bool

Checks if the given object or key is part of this registry entry.

__eq__(other) → bool

Check for equality by inspecting stored bundles and model name

__getitem__(key: str) → ModelInfoBundle

Return a bundle for the given label. If None, tries to return the single registered item.

item() → ModelInfoBundle

Retrieve a single bundle for this model, if not ambiguous.

This will work only in two cases:

• If a default label is set, returns the corresponding bundle

• If there is only a single bundle availble, returns that one

Returns:

The unambiguously selectable model info bundle

Return type:

ModelInfoBundle

Raises:

ModelRegistryError – In case the selection was ambiguous

keys() → Iterator[str]

Returns keys for item access, i.e.: all registered keys

values() → Generator[ModelInfoBundle, None, None]

Returns stored model config bundles, starting with labelled ones

items() → Generator[Tuple[Union[str, int], ModelInfoBundle], None, None]

Returns keys and registered bundles, starting with labelled ones

add_bundle(*, label: str, set_as_default: Optional[bool] = None, exists_action: str = 'raise', update_registry_file: bool = True, **bundle_kwargs) → ModelInfoBundle

Add a new configuration bundle to this registry entry.

This makes sure that the added bundle does not compare equal to an already existing bundle.

Parameters:

• label (str) – The label under which to add it.

• set_as_default (bool, optional) – Controls whether to set this bundle’s label as the default label for this entry.

• exists_action (str, optional) –

  What to do if the given label already exists:

  • raise: Do not add a new bundle and raise (default)

  • skip: Do not add a new bundle, instead return the exist

  • overwrite: Overwrite the existing bundle

  • validate: Make sure that the new bundle compares equal

    to the one that already exists.

• update_registry_file (bool, optional) – Whether to write changes directly to the registry file.

• **bundle_kwargs – Passed on to construct the ModelInfoBundle that is to be stored.

Raises:

• BundleExistsError – If label already exists and exists_action was set to raise.

• BundleValidationError – If validate was given but the bundle already stored under the given label does not compare equal to the to-be-added bundle.

pop(key: str, *, update_registry_file: bool = True) → ModelInfoBundle

Pop a configuration bundle from this entry.

clear(*, update_registry_file: bool = True)

Removes all configuration bundles from this entry.

set_default_label(label: str, *, update_registry_file: bool = True)

Sets the default label

Parameters:

• label (str) – The new label

• update_registry_file (bool, optional) – Whether to update the registry file.

_load_from_registry()

Load the YAML registry entry for this model from the associated registry file path.

_update_registry_file(*, overwrite_existing: bool = True) → str

Stores a YAML representation of this bundle in a file in the given directory and returns the full path.

The file is saved under <model_name>.yml, preserving the case of the model name. Before saving, this makes sure that no file exists in that directory whose lower-case version would compare equal to the lower-case version of this model.

classmethod _as_dict(obj) → dict

Return a copy of the dict representation of this object

classmethod to_yaml(representer, node)

Creates a YAML representation of the data stored in this entry. As the data is a combination of a dict and a sequence, instances of this class are also represented as such.

Parameters:

• representer (ruamel.yaml.representer) – The representer module

• node (ModelRegistryEntry) – The node to represent, i.e. an instance of this class.

Returns:

A YAML representation of the given instance of this class.

utopya.model_registry.info_bundle module

Implements the ModelInfoBundle class, which holds a single bundle of information about a model.

utopya.model_registry.info_bundle.TIME_FSTR = '%y%m%d-%H%M%S'

Format string to use for generating readable time stamps

class utopya.model_registry.info_bundle.ModelInfoBundle(*, model_name: str, paths: dict, metadata: Optional[dict] = None, project_name: Optional[str] = None, eval_after_run: Optional[bool] = None, registration_time: Optional[str] = None, missing_path_action: str = 'log', extract_model_info: bool = False)

Bases: object

A bundle of model information; behaves like a read-only dict

SRC_DIR_SEARCH_PATHS = {'base_plots': '{model_name:}_base_plots.yml', 'default_cfg': '{model_name:}_cfg.yml', 'default_plots': '{model_name:}_plots.yml', 'model_info': '{model_name:}_info.yml'}

Path keys that can be searched within the source directory. Values are format strings that are evaluated with additional information being available (model_name). The resulting paths are interpreted as paths relative to the source directory.

FSTR_SUFFIX = '_fstr'

A suffix used to detect format strings in the path parsing routine.

If a key ends with such a suffix, the corresponding value is assumed to be a format string and is evaluated. The resulting path is stored, dropping the suffix.

PATHS_SCHEMA = (('executable', <class 'str'>), ('model_info', <class 'str'>), ('source_dir', <class 'str'>), ('default_cfg', <class 'str'>), ('default_plots', <class 'str'>), ('base_plots', <class 'str'>), ('py_tests_dir', <class 'str'>), ('py_plots_dir', <class 'str'>), ('mv_model_cfg', <class 'str'>))

Schema to use for a bundle’s paths entry

METADATA_SCHEMA = (('version', <class 'str'>), ('long_name', <class 'str'>), ('description', <class 'str'>), ('long_description', <class 'str'>), ('license', <class 'str'>), ('author', <class 'str'>), ('email', <class 'str'>), ('website', <class 'str'>), ('utopya_compatibility', <class 'str'>), ('language', <class 'str'>), ('requirements', <class 'list'>), ('misc', <class 'dict'>))

Schema to use for a bundle’s metadata entry

__init__(*, model_name: str, paths: dict, metadata: Optional[dict] = None, project_name: Optional[str] = None, eval_after_run: Optional[bool] = None, registration_time: Optional[str] = None, missing_path_action: str = 'log', extract_model_info: bool = False)

Initialize a ModelInfoBundle

Parameters:

• model_name (str) – Name of the model this info bundle describes

• paths (dict) – A dictionary of paths

• metadata (dict, optional) – A dictionary of metadata entries

• project_name (str, optional) – The project this model is part of

• eval_after_run (bool, optional) – Whether a model run should be followed by an evaluation.

• registration_time (str, optional) – Timestamp of registration

• missing_path_action (str, optional) – Action upon a path in the paths dict that does not exist. Can be ignore, log, warn, raise.

• extract_model_info (bool, optional) – Whether to use the information from a potentially existing file at the model_info path to pre-populate this bundle. Any explicitly given arguments take precedence over the information from that file.

__eq__(other) → bool

Compares equality by comparing the stored configuration. Only if another ModelInfoBundle is compared against does the model name also take part in the comparison.

property model_name: str

property registration_time: str

Registration time string of this bundle

property as_dict: dict

Returns a deep copy of all bundle data. This does NOT include the model name and the registration time.

__getitem__(key: str)

Direct access to the full bundle data

property executable: Optional[str]

The path to the model executable

property paths: dict

Access to the paths information of the bundle

property metadata: dict

Access to the metadata information of the bundle

property project_name: str

Access to the Utopia project name information of the bundle

property eval_after_run: Optional[bool]

Whether a model run should be followed by the evaluation routine.

property project: Union[None, dict]

Load the project information corresponding to this project. Will be None if no project is associated.

property missing_paths: dict

Returns those paths where os.path.exists did not evaluate to True

_load_and_parse_model_info(path: str) → dict

Loads the model info file from the given path and parses it.

Parsing steps:

• Remove entries model_name or label that are not relevant at this point.

Parameters:

path (str) – Path to the model info YAML file

_parse_paths(*, missing_path_action: str, executable: Optional[str] = None, source_dir: Optional[str] = None, base_executable_dir: Optional[str] = None, base_source_dir: Optional[str] = None, model_info: Optional[str] = None, **more_paths) → dict

Given the path arguments, parse them into absolute paths. There are the following parsing steps:

1. Evaluate executable and source_dir, potentially using the base_* arguments to resolve relative paths.

2. If a model_info path is given, the directory of that file is used for any missing base_* argument.

3. If a source_dir was given, that directory is searched for further existing paths using SRC_DIR_SEARCH_PATHS.

4. Empty entries in more_paths are skipped

5. Remaining entries in more_paths that end with the suffix specified by the FSTR_SUFFIX class attribute are interpreted as format strings and resolved by providing information on the model_name and all available paths. Relative paths are interpreted as relative to the source directory.

6. For all paths, it is checked whether they exist. The missing_path_action argument determines what to do if not.

classmethod to_yaml(representer, node)

Creates a YAML representation of the data stored in this bundle.

Parameters:

• representer (ruamel.yaml.representer) – The representer module

• node (ModelInfoBundle) – The node to represent, i.e. an instance of this class.

Returns:

A YAML representation of the given instance of this class.

utopya.model_registry.registry module

This module implements the ModelRegistry, which combines ModelRegistryEntry objects and makes it possible to register new models.

class utopya.model_registry.registry.KeyOrderedDict(*args, key: Optional[Callable] = None, **kwds)

Bases: KeyOrderedDict

A key-ordered dict that expects string keys and sorts by the lower-case representation of keys.

DEFAULT_KEY_COMPARATOR(k)

class utopya.model_registry.registry.ModelRegistry(utopya_cfg_dir_path: Optional[str] = None)

Bases: object

The ModelRegistry class takes care of providing model information to the rest of the utopya package and, at the same time, maintains the model registry it is associated with.

It provides a dict-like interface to access the stored registry entries under their model name. Via register_model_info(), a model can be registered and information can be added to it.

Additionally, there are some functions that provide an overview over the registered models and the known information about them.

__init__(utopya_cfg_dir_path: Optional[str] = None)

Loads the utopya model registry from the configuration at the given path.

Parameters:

UTOPYA_CFG_DIR_path (str, optional) – The path to store the model registry folder in.

property registry_dir: str

The model registry directory path

property info_str: str

Returns a multi-line info string showing all registered models

property info_str_detailed: str

Returns a multi-line info string showing all registered models with additional details.

keys()

values()

items()

__contains__(model_name: str) → bool

Whether an entry for the given model name exists in the registry

__getitem__(model_name: str) → ModelRegistryEntry

Retrieve a deep copy of a model registration entry for the given model name.

register_model_info(model_name: str, **bundle_kwargs) → ModelRegistryEntry

Register information for a single model. This method also allows to create a new entry if a model does not exist.

However, it will raise an error if the model was already registered and neither the skip nor the remove options were explicitly specified.

Parameters:

• model_name (str) – The name of the model to register

• **bundle_kwargs – Passed on to ModelRegistryEntry.add_bundle

Returns:

The registry entry for this model.

Return type:

ModelRegistryEntry

remove_entry(model_name: str)

Removes a registry entry and deletes the associated registry file.

Parameters:

model_name (str) – The name of the model entry that is to be removed

Raises:

ValueError – On invalid (non-existing) model

_add_entry(model_name: str) → ModelRegistryEntry

Create a ModelRegistryEntry object for the given model, which loads the associated data from the registry directory, and store it here in the registry.

Parameters:

model_name (str) – Model name for which to add the utopya.model_registry.entry.ModelRegistryEntry object.

Raises:

ValueError – If the model already exists.

Returns:

The newly created entry

Return type:

ModelRegistryEntry

_load_from_registry_dir()

Load all available entries from the registry directory.

If called multiple times, will only load entries that are not already registered.

utopya.model_registry.utils module

Utility functions that work on the already initialized model registry.

utopya.model_registry.utils.get_info_bundle(*, model_name: Optional[str] = None, info_bundle: Optional[ModelInfoBundle] = None, bundle_label: Optional[Union[str, int]] = None) → ModelInfoBundle

Determine the model info bundle in cases where both a model name and an info bundle are allowed as arguments.

Parameters:

• model_name (str, optional) – The model name.

• info_bundle (ModelInfoBundle, optional) – The info bundle object. If given, will directly return this object again.

• bundle_label (Union[str, int], optional) – In cases where only the model name is given, the bundle_label can be used for item access, e.g. in cases where more than one bundle is available and access would be ambiguous.

Returns:

The selected info bundle item

Return type:

ModelInfoBundle

Raises:

ValueError – If neither or both model_name and info_bundle were None

utopya.model_registry.utils.load_model_cfg(**get_info_bundle_kwargs) → Tuple[dict, str, dict]

Loads the default model configuration file for the given model name, using the path specified in the info bundle.

Furthermore, extract_validation_objects() is called to extract any Parameter objects that require validation, replace them with their default values, and gather the Parameter class objects into a separate dict.

Parameters:

**get_info_bundle_kwargs – Used to retrieve the info bundle via get_info_bundle()

Returns:

The corresponding model configuration, the path

to the model configuration file, and the Parameter class objects requiring validation. Will be {}, None, {} if there is no default configuration available in the selected info bundle.

Return type:

Tuple[dict, str, dict]

Raises:

FileNotFoundError – On missing file