utopya.model_registry package#
This submodule implements a registry of Utopia models, which is used to provide the required model information to the frontend and use it throughout.
- utopya.model_registry.MODELS = <utopya.model_registry.registry.ModelRegistry object>#
The model registry
Submodules#
utopya.model_registry._registration module#
Tools for model registration
- utopya.model_registry._registration.register_models_from_list(*, registry: utopya.model_registry.registry.ModelRegistry, separator: str, model_names: str, executables: str, label: str, more_paths: dict = {}, source_dirs: str = None, exists_action: str = 'raise', set_as_default: bool = None, project_name: str = None, _log=<DantroLogger utopya.model_registry._registration (REMARK)>, **shared_bundle_kwargs)[source]#
Handles registration of multiple models where the model names, executables, and source directories are splittable lists of equal lengths.
- Parameters
registry (utopya.model_registry.registry.ModelRegistry) – The model registry to store the models in
separator (str) – Separation string to split
model_names
,executables
, andsource_dirs
.model_names (str) – Splittable string of model names
executables (str) – Splittable string of executables
label (str) – Label under which to add the entries
more_paths (dict, optional) – Additional paths that are to be parsed
source_dirs (str, optional) – Splittable string of model source directories
exists_action (str, optional) – Action to take upon existing label
project_name (str, optional) – The associated project name
_log (logging.Logger, optional) – A logger-like object
**shared_bundle_kwargs – passed on to bundle creation
utopya.model_registry.entry module#
This module implements the ModelRegistryEntry class, which contains a set of ModelInfoBundle objects. These can be combined in a labelled and unlabelled fashion.
- class utopya.model_registry.entry.ModelRegistryEntry(model_name: str, *, registry_dir: str)[source]#
Bases:
object
Holds model config bundles for a single model.
Instances of this class are associated with a file in the model registry directory. Upon change, they update that file.
- __init__(model_name: str, *, registry_dir: str)[source]#
Initialize a model registry entry from a registry directory.
Expects a file named <model_name>.yml in the registry directory. If it does not exist, creates an empty one.
- property default_bundle: utopya.model_registry.info_bundle.ModelInfoBundle#
Returns the default bundle, if set; raises if not
- __contains__(other: Union[utopya.model_registry.info_bundle.ModelInfoBundle, str]) bool [source]#
Checks if the given object or key is part of this registry entry.
- __getitem__(key: str) utopya.model_registry.info_bundle.ModelInfoBundle [source]#
Return a bundle for the given label. If None, tries to return the single registered item.
- item() utopya.model_registry.info_bundle.ModelInfoBundle [source]#
Retrieve a single bundle for this model, if not ambiguous.
- This will work only in two cases:
If a default label is set, returns the corresponding bundle
If there is only a single bundle availble, returns that one
- Returns
The unambiguously selectable model info bundle
- Return type
- Raises
ModelRegistryError – In case the selection was ambiguous
- values() Generator[utopya.model_registry.info_bundle.ModelInfoBundle, None, None] [source]#
Returns stored model config bundles, starting with labelled ones
- items() Generator[Tuple[Union[str, int], utopya.model_registry.info_bundle.ModelInfoBundle], None, None] [source]#
Returns keys and registered bundles, starting with labelled ones
- add_bundle(*, label: str, set_as_default: Optional[bool] = None, exists_action: str = 'raise', update_registry_file: bool = True, **bundle_kwargs) utopya.model_registry.info_bundle.ModelInfoBundle [source]#
Add a new configuration bundle to this registry entry.
This makes sure that the added bundle does not compare equal to an already existing bundle.
- Parameters
label (str) – The label under which to add it.
set_as_default (bool, optional) – Controls whether to set this bundle’s
label
as the default label for this entry.exists_action (str, optional) –
What to do if the given
label
already exists:raise
: Do not add a new bundle and raise (default)skip
: Do not add a new bundle, instead return the existoverwrite
: Overwrite the existing bundlevalidate
: Make sure that the new bundle compares equalto the one that already exists.
update_registry_file (bool, optional) – Whether to write changes directly to the registry file.
**bundle_kwargs – Passed on to construct the
ModelInfoBundle
that is to be stored.
- Raises
BundleExistsError – If
label
already exists andexists_action
was set toraise
.BundleValidationError – If
validate
was given but the bundle already stored under the givenlabel
does not compare equal to the to-be-added bundle.
- pop(key: str, *, update_registry_file: bool = True) utopya.model_registry.info_bundle.ModelInfoBundle [source]#
Pop a configuration bundle from this entry.
- clear(*, update_registry_file: bool = True)[source]#
Removes all configuration bundles from this entry.
- _load_from_registry()[source]#
Load the YAML registry entry for this model from the associated registry file path.
- _update_registry_file(*, overwrite_existing: bool = True) str [source]#
Stores a YAML representation of this bundle in a file in the given directory and returns the full path.
The file is saved under
<model_name>.yml
, preserving the case of the model name. Before saving, this makes sure that no file exists in that directory whose lower-case version would compare equal to the lower-case version of this model.
- classmethod to_yaml(representer, node)[source]#
Creates a YAML representation of the data stored in this entry. As the data is a combination of a dict and a sequence, instances of this class are also represented as such.
- Parameters
representer (ruamel.yaml.representer) – The representer module
node (ModelRegistryEntry) – The node to represent, i.e. an instance of this class.
- Returns
A YAML representation of the given instance of this class.
utopya.model_registry.info_bundle module#
Implements the ModelInfoBundle class, which holds a single bundle of information about a model.
- utopya.model_registry.info_bundle.TIME_FSTR = '%y%m%d-%H%M%S'#
Format string to use for generating readable time stamps
- class utopya.model_registry.info_bundle.ModelInfoBundle(*, model_name: str, paths: dict, metadata: Optional[dict] = None, project_name: Optional[str] = None, eval_after_run: Optional[bool] = None, registration_time: Optional[str] = None, missing_path_action: str = 'log', extract_model_info: bool = False)[source]#
Bases:
object
A bundle of model information; behaves like a read-only dict
- SRC_DIR_SEARCH_PATHS = {'base_plots': '{model_name:}_base_plots.yml', 'default_cfg': '{model_name:}_cfg.yml', 'default_plots': '{model_name:}_plots.yml', 'model_info': '{model_name:}_info.yml'}#
Path keys that can be searched within the source directory. Values are format strings that are evaluated with additional information being available (
model_name
). The resulting paths are interpreted as paths relative to the source directory.
- FSTR_SUFFIX = '_fstr'#
A suffix used to detect format strings in the path parsing routine.
If a key ends with such a suffix, the corresponding value is assumed to be a format string and is evaluated. The resulting path is stored, dropping the suffix.
- PATHS_SCHEMA = (('executable', <class 'str'>), ('model_info', <class 'str'>), ('source_dir', <class 'str'>), ('default_cfg', <class 'str'>), ('default_plots', <class 'str'>), ('base_plots', <class 'str'>), ('py_tests_dir', <class 'str'>), ('py_plots_dir', <class 'str'>), ('mv_model_cfg', <class 'str'>))#
Schema to use for a bundle’s
paths
entry
- METADATA_SCHEMA = (('version', <class 'str'>), ('long_name', <class 'str'>), ('description', <class 'str'>), ('long_description', <class 'str'>), ('license', <class 'str'>), ('author', <class 'str'>), ('email', <class 'str'>), ('website', <class 'str'>), ('utopya_compatibility', <class 'str'>), ('language', <class 'str'>), ('requirements', <class 'list'>), ('misc', <class 'dict'>))#
Schema to use for a bundle’s
metadata
entry
- __init__(*, model_name: str, paths: dict, metadata: Optional[dict] = None, project_name: Optional[str] = None, eval_after_run: Optional[bool] = None, registration_time: Optional[str] = None, missing_path_action: str = 'log', extract_model_info: bool = False)[source]#
Initialize a ModelInfoBundle
- Parameters
model_name (str) – Name of the model this info bundle describes
paths (dict) – A dictionary of paths
metadata (dict, optional) – A dictionary of metadata entries
project_name (str, optional) – The project this model is part of
eval_after_run (bool, optional) – Whether a model run should be followed by an evaluation.
registration_time (str, optional) – Timestamp of registration
missing_path_action (str, optional) – Action upon a path in the
paths
dict that does not exist. Can beignore
,log
,warn
,raise
.extract_model_info (bool, optional) – Whether to use the information from a potentially existing file at the
model_info
path to pre-populate this bundle. Any explicitly given arguments take precedence over the information from that file.
- __eq__(other) bool [source]#
Compares equality by comparing the stored configuration. Only if another ModelInfoBundle is compared against does the model name also take part in the comparison.
- property as_dict: dict#
Returns a deep copy of all bundle data. This does NOT include the model name and the registration time.
- property eval_after_run: Optional[bool]#
Whether a model run should be followed by the evaluation routine.
- property project: Union[None, dict]#
Load the project information corresponding to this project. Will be None if no project is associated.
- _load_and_parse_model_info(path: str) dict [source]#
Loads the model info file from the given path and parses it.
- Parsing steps:
Remove entries
model_name
orlabel
that are not relevant at this point.
- Parameters
path (str) – Path to the model info YAML file
- _parse_paths(*, missing_path_action: str, executable: Optional[str] = None, source_dir: Optional[str] = None, base_executable_dir: Optional[str] = None, base_source_dir: Optional[str] = None, model_info: Optional[str] = None, **more_paths) dict [source]#
Given the path arguments, parse them into absolute paths. There are the following parsing steps:
Evaluate
executable
andsource_dir
, potentially using thebase_*
arguments to resolve relative paths.If a
model_info
path is given, the directory of that file is used for any missingbase_*
argument.If a
source_dir
was given, that directory is searched for further existing paths usingSRC_DIR_SEARCH_PATHS
.Empty entries in
more_paths
are skippedRemaining entries in
more_paths
that end with the suffix specified by theFSTR_SUFFIX
class attribute are interpreted as format strings and resolved by providing information on themodel_name
and all availablepaths
. Relative paths are interpreted as relative to the source directory.For all paths, it is checked whether they exist. The
missing_path_action
argument determines what to do if not.
- classmethod to_yaml(representer, node)[source]#
Creates a YAML representation of the data stored in this bundle.
- Parameters
representer (ruamel.yaml.representer) – The representer module
node (ModelInfoBundle) – The node to represent, i.e. an instance of this class.
- Returns
A YAML representation of the given instance of this class.
utopya.model_registry.registry module#
This module implements the ModelRegistry
, which combines
ModelRegistryEntry
objects and makes it possible to register new models.
- class utopya.model_registry.registry.KeyOrderedDict(*args, key: Optional[Callable] = None, **kwds)[source]#
Bases:
dantro.utils.ordereddict.KeyOrderedDict
A key-ordered dict that expects string keys and sorts by the lower-case representation of keys.
- DEFAULT_KEY_COMPARATOR(k)#
- class utopya.model_registry.registry.ModelRegistry(utopya_cfg_dir_path: Optional[str] = None)[source]#
Bases:
object
The ModelRegistry class takes care of providing model information to the rest of the utopya package and, at the same time, maintains the model registry it is associated with.
It provides a dict-like interface to access the stored registry entries under their model name. Via
register_model_info()
, a model can be registered and information can be added to it.Additionally, there are some functions that provide an overview over the registered models and the known information about them.
- __init__(utopya_cfg_dir_path: Optional[str] = None)[source]#
Loads the utopya model registry from the configuration at the given path.
- Parameters
UTOPYA_CFG_DIR_path (str, optional) – The path to store the model registry folder in.
- property info_str_detailed: str#
Returns a multi-line info string showing all registered models with additional details.
- __contains__(model_name: str) bool [source]#
Whether an entry for the given model name exists in the registry
- __getitem__(model_name: str) utopya.model_registry.entry.ModelRegistryEntry [source]#
Retrieve a deep copy of a model registration entry for the given model name.
- register_model_info(model_name: str, **bundle_kwargs) utopya.model_registry.entry.ModelRegistryEntry [source]#
Register information for a single model. This method also allows to create a new entry if a model does not exist.
However, it will raise an error if the model was already registered and neither the skip nor the remove options were explicitly specified.
- Parameters
model_name (str) – The name of the model to register
**bundle_kwargs – Passed on to
ModelRegistryEntry.add_bundle
- Returns
The registry entry for this model.
- Return type
- remove_entry(model_name: str)[source]#
Removes a registry entry and deletes the associated registry file.
- Parameters
model_name (str) – The name of the model entry that is to be removed
- Raises
ValueError – On invalid (non-existing) model
- _add_entry(model_name: str) utopya.model_registry.entry.ModelRegistryEntry [source]#
Create a
ModelRegistryEntry
object for the given model, which loads the associated data from the registry directory, and store it here in the registry.- Parameters
model_name (str) – Model name for which to add the
utopya.model_registry.entry.ModelRegistryEntry
object.- Raises
ValueError – If the model already exists.
- Returns
The newly created entry
- Return type
utopya.model_registry.utils module#
Utility functions that work on the already initialized model registry.
- utopya.model_registry.utils.get_info_bundle(*, model_name: Optional[str] = None, info_bundle: Optional[utopya.model_registry.info_bundle.ModelInfoBundle] = None, bundle_label: Optional[Union[str, int]] = None) utopya.model_registry.info_bundle.ModelInfoBundle [source]#
Determine the model info bundle in cases where both a model name and an info bundle are allowed as arguments.
- Parameters
model_name (str, optional) – The model name.
info_bundle (ModelInfoBundle, optional) – The info bundle object. If given, will directly return this object again.
bundle_label (Union[str, int], optional) – In cases where only the model name is given, the bundle_label can be used for item access, e.g. in cases where more than one bundle is available and access would be ambiguous.
- Returns
The selected info bundle item
- Return type
- Raises
ValueError – If neither or both model_name and info_bundle were None
- utopya.model_registry.utils.load_model_cfg(**get_info_bundle_kwargs) Tuple[dict, str, dict] [source]#
Loads the default model configuration file for the given model name, using the path specified in the info bundle.
Furthermore,
extract_validation_objects()
is called to extract any Parameter objects that require validation, replace them with their default values, and gather the Parameter class objects into a separate dict.- Parameters
**get_info_bundle_kwargs – Used to retrieve the info bundle via
get_info_bundle()
- Returns
- The corresponding model configuration, the path
to the model configuration file, and the Parameter class objects requiring validation. Will be
{}, None, {}
if there is no default configuration available in the selected info bundle.
- Return type
- Raises
FileNotFoundError – On missing file