utopya.model_registry package#

This submodule implements a registry of Utopia models, which is used to provide the required model information to the frontend and use it throughout.

utopya.model_registry.MODELS = <utopya.model_registry.registry.ModelRegistry object>#: The model registry

Submodules#

utopya.model_registry._registration module#

Tools for model registration

utopya.model_registry._registration.register_models_from_list(*, registry: utopya.model_registry.registry.ModelRegistry, separator: str, model_names: str, executables: str, label: str, more_paths: dict = {}, source_dirs: str = None, exists_action: str = 'raise', set_as_default: bool = None, project_name: str = None, _log=<DantroLogger utopya.model_registry._registration (REMARK)>, **shared_bundle_kwargs)[source]#

Handles registration of multiple models where the model names, executables, and source directories are splittable lists of equal lengths.

Parameters

registry (utopya.model_registry.registry.ModelRegistry) – The model registry to store the models in
separator (str) – Separation string to split model_names, executables, and source_dirs.
model_names (str) – Splittable string of model names
executables (str) – Splittable string of executables
label (str) – Label under which to add the entries
more_paths (dict, optional) – Additional paths that are to be parsed
source_dirs (str, optional) – Splittable string of model source directories
exists_action (str, optional) – Action to take upon existing label
project_name (str, optional) – The associated project name
_log (logging.Logger, optional) – A logger-like object
**shared_bundle_kwargs – passed on to bundle creation

utopya.model_registry.entry module#

This module implements the ModelRegistryEntry class, which contains a set of ModelInfoBundle objects. These can be combined in a labelled and unlabelled fashion.

class utopya.model_registry.entry.ModelRegistryEntry(model_name: str, *, registry_dir: str)[source]#

Bases: object

Holds model config bundles for a single model.

Instances of this class are associated with a file in the model registry directory. Upon change, they update that file.

__init__(model_name: str, *, registry_dir: str)[source]#

Initialize a model registry entry from a registry directory.

Expects a file named <model_name>.yml in the registry directory. If it does not exist, creates an empty one.

property model_name: str#: Name of this model

property registry_dir: str#: The associated registry directory

property registry_file_path: str#: The absolute path to the registry file

property default_label: Optional[str]#: The default label

property default_bundle: utopya.model_registry.info_bundle.ModelInfoBundle#: Returns the default bundle, if set; raises if not

__len__() → int[source]#: Returns number of registered bundles

__contains__(other: Union[utopya.model_registry.info_bundle.ModelInfoBundle, str]) → bool[source]#: Checks if the given object or key is part of this registry entry.

__eq__(other) → bool[source]#: Check for equality by inspecting stored bundles and model name

__getitem__(key: str) → utopya.model_registry.info_bundle.ModelInfoBundle[source]#: Return a bundle for the given label. If None, tries to return the single registered item.

item() → utopya.model_registry.info_bundle.ModelInfoBundle[source]#

Retrieve a single bundle for this model, if not ambiguous.

This will work only in two cases:

If a default label is set, returns the corresponding bundle
If there is only a single bundle availble, returns that one

Returns: The unambiguously selectable model info bundle
Return type: ModelInfoBundle
Raises: ModelRegistryError – In case the selection was ambiguous

keys() → Iterator[str][source]#: Returns keys for item access, i.e.: all registered keys

values() → Generator[utopya.model_registry.info_bundle.ModelInfoBundle, None, None][source]#: Returns stored model config bundles, starting with labelled ones

items() → Generator[Tuple[Union[str, int], utopya.model_registry.info_bundle.ModelInfoBundle], None, None][source]#: Returns keys and registered bundles, starting with labelled ones

add_bundle(*, label: str, set_as_default: Optional[bool] = None, exists_action: str = 'raise', update_registry_file: bool = True, **bundle_kwargs) → utopya.model_registry.info_bundle.ModelInfoBundle[source]#

Add a new configuration bundle to this registry entry.

This makes sure that the added bundle does not compare equal to an already existing bundle.

Parameters

label (str) – The label under which to add it.
set_as_default (bool, optional) – Controls whether to set this bundle’s label as the default label for this entry.
exists_action (str, optional) –
What to do if the given label already exists:
- raise: Do not add a new bundle and raise (default)
- skip: Do not add a new bundle, instead return the exist
- overwrite: Overwrite the existing bundle
- validate: Make sure that the new bundle compares equal
  to the one that already exists.
update_registry_file (bool, optional) – Whether to write changes directly to the registry file.
**bundle_kwargs – Passed on to construct the ModelInfoBundle that is to be stored.

Raises

BundleExistsError – If label already exists and exists_action was set to raise.
BundleValidationError – If validate was given but the bundle already stored under the given label does not compare equal to the to-be-added bundle.

pop(key: str, *, update_registry_file: bool = True) → utopya.model_registry.info_bundle.ModelInfoBundle[source]#: Pop a configuration bundle from this entry.

clear(*, update_registry_file: bool = True)[source]#: Removes all configuration bundles from this entry.

set_default_label(label: str, *, update_registry_file: bool = True)[source]#

Sets the default label

Parameters

label (str) – The new label
update_registry_file (bool, optional) – Whether to update the registry file.

_load_from_registry()[source]#: Load the YAML registry entry for this model from the associated registry file path.

_update_registry_file(*, overwrite_existing: bool = True) → str[source]#

Stores a YAML representation of this bundle in a file in the given directory and returns the full path.

The file is saved under <model_name>.yml, preserving the case of the model name. Before saving, this makes sure that no file exists in that directory whose lower-case version would compare equal to the lower-case version of this model.

classmethod _as_dict(obj) → dict[source]#: Return a copy of the dict representation of this object

classmethod to_yaml(representer, node)[source]#

Creates a YAML representation of the data stored in this entry. As the data is a combination of a dict and a sequence, instances of this class are also represented as such.

Parameters

representer (ruamel.yaml.representer) – The representer module
node (ModelRegistryEntry) – The node to represent, i.e. an instance of this class.

Returns

A YAML representation of the given instance of this class.

utopya.model_registry.info_bundle module#

Implements the ModelInfoBundle class, which holds a single bundle of information about a model.

utopya.model_registry.info_bundle.TIME_FSTR = '%y%m%d-%H%M%S'#: Format string to use for generating readable time stamps

class utopya.model_registry.info_bundle.ModelInfoBundle(*, model_name: str, paths: dict, metadata: Optional[dict] = None, project_name: Optional[str] = None, eval_after_run: Optional[bool] = None, registration_time: Optional[str] = None, missing_path_action: str = 'log', extract_model_info: bool = False)[source]#

Bases: object

A bundle of model information; behaves like a read-only dict

SRC_DIR_SEARCH_PATHS = {'base_plots': '{model_name:}_base_plots.yml', 'default_cfg': '{model_name:}_cfg.yml', 'default_plots': '{model_name:}_plots.yml', 'model_info': '{model_name:}_info.yml'}#: Path keys that can be searched within the source directory. Values are format strings that are evaluated with additional information being available (model_name). The resulting paths are interpreted as paths relative to the source directory.

FSTR_SUFFIX = '_fstr'#

A suffix used to detect format strings in the path parsing routine.

If a key ends with such a suffix, the corresponding value is assumed to be a format string and is evaluated. The resulting path is stored, dropping the suffix.

PATHS_SCHEMA = (('executable', <class 'str'>), ('model_info', <class 'str'>), ('source_dir', <class 'str'>), ('default_cfg', <class 'str'>), ('default_plots', <class 'str'>), ('base_plots', <class 'str'>), ('py_tests_dir', <class 'str'>), ('py_plots_dir', <class 'str'>), ('mv_model_cfg', <class 'str'>))#: Schema to use for a bundle’s paths entry

METADATA_SCHEMA = (('version', <class 'str'>), ('long_name', <class 'str'>), ('description', <class 'str'>), ('long_description', <class 'str'>), ('license', <class 'str'>), ('author', <class 'str'>), ('email', <class 'str'>), ('website', <class 'str'>), ('utopya_compatibility', <class 'str'>), ('language', <class 'str'>), ('requirements', <class 'list'>), ('misc', <class 'dict'>))#: Schema to use for a bundle’s metadata entry

__init__(*, model_name: str, paths: dict, metadata: Optional[dict] = None, project_name: Optional[str] = None, eval_after_run: Optional[bool] = None, registration_time: Optional[str] = None, missing_path_action: str = 'log', extract_model_info: bool = False)[source]#

Initialize a ModelInfoBundle

Parameters

model_name (str) – Name of the model this info bundle describes
paths (dict) – A dictionary of paths
metadata (dict, optional) – A dictionary of metadata entries
project_name (str, optional) – The project this model is part of
eval_after_run (bool, optional) – Whether a model run should be followed by an evaluation.
registration_time (str, optional) – Timestamp of registration
missing_path_action (str, optional) – Action upon a path in the paths dict that does not exist. Can be ignore, log, warn, raise.
extract_model_info (bool, optional) – Whether to use the information from a potentially existing file at the model_info path to pre-populate this bundle. Any explicitly given arguments take precedence over the information from that file.

__eq__(other) → bool[source]#: Compares equality by comparing the stored configuration. Only if another ModelInfoBundle is compared against does the model name also take part in the comparison.

property model_name: str#

property registration_time: str#: Registration time string of this bundle

property as_dict: dict#: Returns a deep copy of all bundle data. This does NOT include the model name and the registration time.

__getitem__(key: str)[source]#: Direct access to the full bundle data

property executable: Optional[str]#: The path to the model executable

property paths: dict#: Access to the paths information of the bundle

property metadata: dict#: Access to the metadata information of the bundle

property project_name: str#: Access to the Utopia project name information of the bundle

property eval_after_run: Optional[bool]#: Whether a model run should be followed by the evaluation routine.

property project: Union[None, dict]#: Load the project information corresponding to this project. Will be None if no project is associated.

property missing_paths: dict#: Returns those paths where os.path.exists did not evaluate to True

_load_and_parse_model_info(path: str) → dict[source]#

Loads the model info file from the given path and parses it.

Parsing steps:

Remove entries model_name or label that are not relevant at this point.

Parameters: path (str) – Path to the model info YAML file

_parse_paths(*, missing_path_action: str, executable: Optional[str] = None, source_dir: Optional[str] = None, base_executable_dir: Optional[str] = None, base_source_dir: Optional[str] = None, model_info: Optional[str] = None, **more_paths) → dict[source]#

Given the path arguments, parse them into absolute paths. There are the following parsing steps:

Evaluate executable and source_dir, potentially using the base_* arguments to resolve relative paths.

If a model_info path is given, the directory of that file is used for any missing base_* argument.

If a source_dir was given, that directory is searched for further existing paths using SRC_DIR_SEARCH_PATHS.

Empty entries in more_paths are skipped

Remaining entries in more_paths that end with the suffix specified by the FSTR_SUFFIX class attribute are interpreted as format strings and resolved by providing information on the model_name and all available paths. Relative paths are interpreted as relative to the source directory.

For all paths, it is checked whether they exist. The missing_path_action argument determines what to do if not.

classmethod to_yaml(representer, node)[source]#

Creates a YAML representation of the data stored in this bundle.

Parameters

representer (ruamel.yaml.representer) – The representer module
node (ModelInfoBundle) – The node to represent, i.e. an instance of this class.

Returns

A YAML representation of the given instance of this class.

utopya.model_registry.registry module#

This module implements the ModelRegistry, which combines ModelRegistryEntry objects and makes it possible to register new models.

class utopya.model_registry.registry.KeyOrderedDict(*args, key: Optional[Callable] = None, **kwds)[source]#

Bases: dantro.utils.ordereddict.KeyOrderedDict

A key-ordered dict that expects string keys and sorts by the lower-case representation of keys.

DEFAULT_KEY_COMPARATOR(k)#

class utopya.model_registry.registry.ModelRegistry(utopya_cfg_dir_path: Optional[str] = None)[source]#

Bases: object

The ModelRegistry class takes care of providing model information to the rest of the utopya package and, at the same time, maintains the model registry it is associated with.

It provides a dict-like interface to access the stored registry entries under their model name. Via register_model_info(), a model can be registered and information can be added to it.

Additionally, there are some functions that provide an overview over the registered models and the known information about them.

__init__(utopya_cfg_dir_path: Optional[str] = None)[source]#

Loads the utopya model registry from the configuration at the given path.

Parameters: UTOPYA_CFG_DIR_path (str, optional) – The path to store the model registry folder in.

property registry_dir: str#: The model registry directory path

property info_str: str#: Returns a multi-line info string showing all registered models

property info_str_detailed: str#: Returns a multi-line info string showing all registered models with additional details.

keys()[source]#

values()[source]#

items()[source]#

__contains__(model_name: str) → bool[source]#: Whether an entry for the given model name exists in the registry

__getitem__(model_name: str) → utopya.model_registry.entry.ModelRegistryEntry[source]#: Retrieve a deep copy of a model registration entry for the given model name.

register_model_info(model_name: str, **bundle_kwargs) → utopya.model_registry.entry.ModelRegistryEntry[source]#

However, it will raise an error if the model was already registered and neither the skip nor the remove options were explicitly specified.

Parameters

model_name (str) – The name of the model to register
**bundle_kwargs – Passed on to ModelRegistryEntry.add_bundle

Returns

The registry entry for this model.

Return type

ModelRegistryEntry

remove_entry(model_name: str)[source]#

Removes a registry entry and deletes the associated registry file.

Parameters: model_name (str) – The name of the model entry that is to be removed
Raises: ValueError – On invalid (non-existing) model

_add_entry(model_name: str) → utopya.model_registry.entry.ModelRegistryEntry[source]#

Create a ModelRegistryEntry object for the given model, which loads the associated data from the registry directory, and store it here in the registry.

Parameters: model_name (str) – Model name for which to add the utopya.model_registry.entry.ModelRegistryEntry object.
Raises: ValueError – If the model already exists.
Returns: The newly created entry
Return type: ModelRegistryEntry

_load_from_registry_dir()[source]#

Load all available entries from the registry directory.

If called multiple times, will only load entries that are not already registered.

utopya.model_registry.utils module#

Utility functions that work on the already initialized model registry.

utopya.model_registry.utils.get_info_bundle(*, model_name: Optional[str] = None, info_bundle: Optional[utopya.model_registry.info_bundle.ModelInfoBundle] = None, bundle_label: Optional[Union[str, int]] = None) → utopya.model_registry.info_bundle.ModelInfoBundle[source]#

Determine the model info bundle in cases where both a model name and an info bundle are allowed as arguments.

Parameters

model_name (str, optional) – The model name.
info_bundle (ModelInfoBundle, optional) – The info bundle object. If given, will directly return this object again.
bundle_label (Union[str, int], optional) – In cases where only the model name is given, the bundle_label can be used for item access, e.g. in cases where more than one bundle is available and access would be ambiguous.

Returns

The selected info bundle item

Return type

ModelInfoBundle

Raises

ValueError – If neither or both model_name and info_bundle were None

utopya.model_registry.utils.load_model_cfg(**get_info_bundle_kwargs) → Tuple[dict, str, dict][source]#

Loads the default model configuration file for the given model name, using the path specified in the info bundle.

Furthermore, extract_validation_objects() is called to extract any Parameter objects that require validation, replace them with their default values, and gather the Parameter class objects into a separate dict.

Parameters

**get_info_bundle_kwargs – Used to retrieve the info bundle via get_info_bundle()

Returns

The corresponding model configuration, the path: to the model configuration file, and the Parameter class objects requiring validation. Will be {}, None, {} if there is no default configuration available in the selected info bundle.

Return type

Tuple[dict, str, dict]

Raises

FileNotFoundError – On missing file

utopya.model_registry package

Contents

utopya.model_registry package#

Submodules#

utopya.model_registry._registration module#

utopya.model_registry.entry module#

utopya.model_registry.info_bundle module#

utopya.model_registry.registry module#

utopya.model_registry.utils module#