utopya.eval package#
Implements a dantro-based simulation data evaluation pipeline.
Subpackages#
- utopya.eval.plots package
- Submodules
- utopya.eval.plots._attractor module
- utopya.eval.plots._graph module
- utopya.eval.plots._mpl module
- utopya.eval.plots._utils module
- utopya.eval.plots.abm module
- utopya.eval.plots.attractor module
- utopya.eval.plots.ca module
- utopya.eval.plots.distributions module
- utopya.eval.plots.graph module
- utopya.eval.plots.snsplot module
- utopya.eval.plots.time_series module
Submodules#
utopya.eval._plot_func_resolver module#
Implements a plot function resolver that takes model-specific information into account.
- class utopya.eval._plot_func_resolver.PlotFuncResolver(*, _model_info_bundle: utopya.model_registry.info_bundle.ModelInfoBundle, **kwargs)[source]#
Bases:
dantro.plot.utils.plot_func.PlotFuncResolver
A utopya-specific plot function resolver for
PlotManager
that takes information from the model info bundle into account.- BASE_PKG: str = 'utopya.eval.plots'#
Which package to use as base package for relative module imports
- __init__(*, _model_info_bundle: utopya.model_registry.info_bundle.ModelInfoBundle, **kwargs)[source]#
Initializes the plot function resolver and additionally stores the model info bundle.
- _get_custom_module_paths() Dict[str, str] [source]#
Aggregates a dict of module paths from which imports are attempted.
Uses model- or project-specific information, if available:
A model’s
py_plots_dir
A project’s
py_plots_dir
A project’s additional
py_modules
The framework’s additional
py_modules
The framework’s
py_plots_dir
utopya.eval.containers module#
Implements data container classes specialised on Utopia output data.
It is based on the dantro.base.BaseDataContainer
classes,
especially their numeric form, the
NumpyDataContainer
and the
XrDataContainer
.
- class utopya.eval.containers.NumpyDC(*, name: str, data: numpy.ndarray, **dc_kwargs)[source]#
Bases:
dantro.mixins.proxy_support.Hdf5ProxySupportMixin
,dantro.containers.numeric.NumpyDataContainer
This is the base class for numpy data containers used in Utopia.
It is based on the NumpyDataContainer provided by dantro and extends it with the Hdf5ProxySupportMixin, allowing to load the data from the Hdf5 file only once it becomes necessary.
- _abc_impl = <_abc._abc_data object>#
- class utopya.eval.containers.XarrayDC(*, name: str, data: Union[numpy.ndarray, xarray.DataArray], dims: Sequence[str] = None, coords: dict = None, extract_metadata: bool = True, apply_metadata: bool = True, **dc_kwargs)[source]#
Bases:
dantro.mixins.proxy_support.Hdf5ProxySupportMixin
,dantro.containers.xr.XrDataContainer
This is the base class for xarray data containers used in Utopia.
It is based on the XrDataContainer provided by dantro. As of now, it has no proxy support, but will gain it once available on dantro side.
- PROXY_RESOLVE_ASTYPE = None#
Which type to resolve the proxy to. None defaults to
numpy.ndarray
.
- PROXY_RETAIN = True#
Whether to retain a proxy after resolving it; allows reinstating proxy objects.
- PROXY_REINSTATE_FAIL_ACTION = 'log_warning'#
Which action to take if reinstating a proxy was not possible
- _XRC_DIMS_ATTR = 'dim_names'#
Define as class variable the name of the attribute that determines the dimensions of the
xarray.DataArray
.
- _XRC_DIM_NAME_PREFIX = 'dim_name__'#
Attributes prefixed with this string can be used to set names for specific dimensions. The prefix should be followed by an integer-parsable string, e.g.
dim_name__0
would be the dimension name for the 0th dim
- _XRC_COORDS_ATTR_PREFIX = 'coords__'#
Attributes prefixed with this string determine the coordinate values for a specific dimension. The prefix should be followed by the _name_ of the dimension, e.g.
coords__time
. The values are interpreted according to the default coordinate mode or, if given, thecoords_mode__*
attribute
- _XRC_COORDS_MODE_DEFAULT = 'values'#
The default mode by which coordinates are interpreted. See the base class,
dantro.containers.xr.XrDataContainer
for more information.- Available modes:
values
: the explicit values (iterable) to use for coordinatestrivial
: The trivial indices; ignores the coordinate argumentscalar
: makes sure only a single coordinate is providedrange
: python built-in range function argumentsarange
: np.arange argumentslinspace
: np.linspace argumentslogspace
: np.logspace argumentsstart_and_step
: the start and step values of an integer rangeexpression; the stop value is deduced by looking at the length of the corresponding dimension.
linked
: Load the coordinates from a linked object within thetree, specified by a relative path from the current object.
- _XRC_COORDS_MODE_ATTR_PREFIX = 'coords_mode__'#
Prefix for the coordinate mode if a custom mode is to be used. To, e.g., use mode
start_and_step
for time dimension, set thecoords_mode__time
attribute to valuestart_and_step
- _XRC_INHERIT_CONTAINER_ATTRIBUTES = True#
Whether to inherit the other container attributes
- _XRC_STRICT_ATTR_CHECKING = True#
Whether to use strict attribute checking; throws errors if there are container attributes available that match the prefix but don’t match a valid dimension name. Can be disabled for speed improvements.
- _abc_impl = <_abc._abc_data object>#
- class utopya.eval.containers.XarrayYamlDC(*, name: str, data: Union[numpy.ndarray, xarray.DataArray], dims: Sequence[str] = None, coords: dict = None, extract_metadata: bool = True, apply_metadata: bool = True, **dc_kwargs)[source]#
Bases:
utopya.eval.containers.XarrayDC
An
XarrayDC
specialization that assumes that each array entry is a YAML string, which is subsequently loaded. This can be done alongside the metadata application of theXarrayDC
.- _apply_metadata()[source]#
Whenever metadata is applied that is also a good point to resolve the YAML data…
- _abc_impl = <_abc._abc_data object>#
- class utopya.eval.containers.GridDC(*, name: str, data: Union[numpy.ndarray, xarray.core.dataarray.DataArray], **dc_kwargs)[source]#
Bases:
utopya.eval.containers.XarrayDC
This is the base class for all grid data used in Utopia.
It is based on the
XarrayDC
and reshapes the data to the grid shape. The last dimension is assumed to be the dimension that goes along the grid cell IDs.- _GDC_grid_shape_attr = 'grid_shape'#
The attribute to read the desired grid shape from
- _GDC_space_extent_attr = 'space_extent'#
The attribute to read the space extent from
- _GDC_index_order_attr = 'index_order'#
The attribute to read the index order from
- _GDC_grid_structure_attr = 'grid_structure'#
The attribute to read the desired grid structure from
- __init__(*, name: str, data: Union[numpy.ndarray, xarray.core.dataarray.DataArray], **dc_kwargs)[source]#
Initialize a GridDC which represents grid-like data.
Given the container attribute (see
_GDC_grid_shape_attr
), this container takes care to reshape the underlying data such that it represents that grid, even if it is saved in another shape.Note
Use this container if it is easier to store array data in a flat format (e.g. because there is no need to take care of slicing etc) but you still desire to work with it in its actual shape.
- Parameters
name (str) – The name of this container
data (Union[numpy.ndarray, xarray.DataArray]) – The not yet reshaped data. If this is 1D, it is assumed that there is no additional dimension. If it is 2D (or more), it is assumed to be
(..., cell ids)
.**kwargs – Further initialization kwargs, e.g.
attrs
…
- _abc_impl = <_abc._abc_data object>#
- _postprocess_proxy_resolution()[source]#
Invoked from
Hdf5ProxySupportMixin
after a proxy was resolved, this takes care to apply the reshaping operation onto the underlying data.
- _parse_sizes_from_metadata() Sequence[Tuple[str, int]] [source]#
Invoked from _format_shape when no metadata was applied but the dimension names are available. Should return data in the same form as
xr.DataArray.sizes.items()
does.
- property shape: tuple#
Returns shape, proxy-aware
This is an overload of the property in Hdf5ProxySupportMixin which takes care that not the actual underlying proxy data shape is returned but whatever the container’s shape is to be after reshaping.
- property ndim: int#
Returns ndim, proxy-aware
This is an overload of the property in Hdf5ProxySupportMixin which takes care that not the actual underlying proxy data ndim is returned but whatever the container’s ndim is to be after reshaping.
- _reshape_data() xarray.core.dataarray.DataArray [source]#
Looks at the current data shape and container attributes to reshape the data such that it represents a grid.
utopya.eval.data_ops module#
Custom data operations
- utopya.eval.data_ops.update_attrs(d, attrs: dict)[source]#
Updates the data attributes in
d
withattrs
.- Parameters
d (xarray.DataArray) – The data array to write the attributes to.
attrs (dict) – The attributes to use for updating
- Returns
A new data array like
d
with updated attributes.- Return type
- utopya.eval.data_ops.update_with_attrs_from(t, s)[source]#
Updates the data attributes in
t
with those froms
.- Parameters
t (xarray.DataArray) – The data array to write the attributes to.
s (xarray.DataArray) – The data array to copy the attributes from.
- Returns
A new data array for
t
with updated attributes froms
.- Return type
utopya.eval.datamanager module#
Implements a class that manages data written out by Utopia models.
It is based on dantro’s DataManager
class and the containers specialized
for Utopia data.
- utopya.eval.datamanager._condense_thresh_func(*, level: int, num_items: int, total_item_count: int) int [source]#
Dynamically computes the condensation threshold for the current element in a
DataManager
tree.
- class utopya.eval.datamanager.DataManager(data_dir: str, *, name: Optional[str] = None, load_cfg: Optional[Union[dict, str]] = None, out_dir: Union[str, bool] = '_output/{timestamp:}', out_dir_kwargs: Optional[dict] = None, create_groups: Optional[List[Union[str, dict]]] = None, condensed_tree_params: Optional[dict] = None, default_tree_cache_path: Optional[str] = None)[source]#
Bases:
dantro.data_loaders.AllAvailableLoadersMixin
,dantro.data_mngr.DataManager
This class manages the data that is written out by Utopia simulations.
It is based on the dantro
DataManager
class and adds the functionality for specific loader functions that are needed in Utopia: Hdf5 and Yaml. Furthermore, to enable file caching via the DAG framework, all available data loaders are included here.- _DATA_GROUP_CLASSES: Dict[str, type] = {'GraphGroup': <class 'utopya.eval.groups.GraphGroup'>, 'MultiverseGroup': <class 'utopya.eval.groups.MultiverseGroup'>}#
Known group types
- _HDF5_DSET_DEFAULT_CLS#
Tells the HDF5 loader which container class to use
alias of
utopya.eval.containers.XarrayDC
- _HDF5_GROUP_MAP: Dict[str, type] = {'graph': <class 'utopya.eval.groups.GraphGroup'>, 'network': <class 'utopya.eval.groups.GraphGroup'>, 'time_series': <class 'utopya.eval.groups.TimeSeriesGroup'>, 'time_series_heterogeneous': <class 'utopya.eval.groups.HeterogeneousTimeSeriesGroup'>}#
The mapping of different content values to a data group type
- _abc_impl = <_abc._abc_data object>#
- _HDF5_DSET_MAP: Dict[str, type] = {'array_of_yaml_strings': <class 'utopya.eval.containers.XarrayYamlDC'>, 'grid': <class 'utopya.eval.containers.GridDC'>, 'labelled_data': <class 'utopya.eval.containers.XarrayDC'>, 'unlabelled_data': <class 'utopya.eval.containers.NumpyDC'>}#
The mapping of different content values to a data container types
- _COND_TREE_MAX_LEVEL = 10#
Condensed tree representation maximum level
- _COND_TREE_CONDENSE_THRESH(**kws)#
Condensed tree representation threshold parameter
- _DEFAULT_TREE_CACHE_PATH = 'data/.tree_cache.d3'#
Where the tree cache file is to be stored; may be overwritten by a config entry
utopya.eval.groups module#
Implements specialized data group classes that are based on group types
provided in dantro.groups
.
Here, they are configured using their class variables.
- class utopya.eval.groups.UniverseGroup(*, name: str, containers: Optional[list] = None, attrs=None, parent: Optional[dantro.abc.AbstractDataGroup] = None)[source]#
Bases:
dantro.groups.psp.ParamSpaceStateGroup
This group represents the data of a single universe
- _abc_impl = <_abc._abc_data object>#
- class utopya.eval.groups.MultiverseGroup(*, name: str, pspace: Optional[paramspace.paramspace.ParamSpace] = None, containers: Optional[list] = None, **kwargs)[source]#
Bases:
dantro.groups.psp.ParamSpaceGroup
This group is meant to manage the
multiverse
group of the loaded data, i.e. the group where output of the individualUniverseGroup
objects is stored in.Its main aim is to provide easy access to universes. By default, universes are only identified by their ID, which is a zero-padded string. This group adds the ability to access them via integer indices.
Furthermore, via dantro, an easy data selector method is available, see
dantro.groups.psp.ParamSpaceGroup.select()
.- _NEW_GROUP_CLS#
alias of
utopya.eval.groups.UniverseGroup
- _abc_impl = <_abc._abc_data object>#
- class utopya.eval.groups.TimeSeriesGroup(*args, dims: Optional[Tuple[str]] = None, mode: Optional[str] = None, allow_deep_selection: Optional[bool] = None, **kwargs)[source]#
Bases:
dantro.groups.time_series.TimeSeriesGroup
This group is meant to manage time series data, with the container names being interpreted as the time coordinate.
- _NEW_CONTAINER_CLS#
alias of
utopya.eval.containers.XarrayDC
- _abc_impl = <_abc._abc_data object>#
- class utopya.eval.groups.HeterogeneousTimeSeriesGroup(*args, dims: Optional[Tuple[str]] = None, mode: Optional[str] = None, allow_deep_selection: Optional[bool] = None, **kwargs)[source]#
Bases:
dantro.groups.time_series.HeterogeneousTimeSeriesGroup
This group is meant to manage time series data, with the container names being interpreted as the time coordinate.
- _NEW_CONTAINER_CLS#
alias of
utopya.eval.containers.XarrayDC
- _abc_impl = <_abc._abc_data object>#
- class utopya.eval.groups.GraphGroup(*args, **kwargs)[source]#
Bases:
dantro.groups.graph.GraphGroup
This group is meant to manage graph data and create a NetworkX graph from it.
- _NEW_GROUP_CLS#
alias of
utopya.eval.groups.TimeSeriesGroup
- _ALLOWED_CONT_TYPES: Optional[tuple] = (<class 'utopya.eval.groups.TimeSeriesGroup'>, <class 'utopya.eval.containers.XarrayDC'>)#
The types that are allowed to be stored in this group. If None, all types derived from the dantro base classes are allowed. This applies to both containers and groups that are added to this group.
Hint
To add the type of the current object, add a string entry
self
to the tuple. This will be resolved totype(self)
at invocation.
- _GG_node_container = '_vertices'#
- _GG_edge_container = '_edges'#
- _GG_attr_directed = 'is_directed'#
- _GG_attr_parallel = 'allows_parallel'#
- _GG_attr_edge_container_is_transposed = 'edge_container_is_transposed'#
- _GG_attr_keep_dim = 'keep_dim'#
- _GG_WARN_UPON_BAD_ALIGN = True#
- _abc_impl = <_abc._abc_data object>#
utopya.eval.plotcreators module#
Implements utopya-specializations of dantro.plot.creators
- class utopya.eval.plotcreators.PyPlotCreator(name: str, *, style: Optional[dict] = None, **parent_kwargs)[source]#
Bases:
dantro.plot.creators.pyplot.PyPlotCreator
This is the Utopia-specific version of dantro’s
PyPlotCreator
.Its main purpose is to define common settings for plotting. By adding this extra layer, it allows for future extensibility as well.
- EXTENSIONS: Union[Tuple[str], str] = 'all'#
Which file extensions to support. A value of
all
leads to no checks being performed on the extension.
- DEFAULT_EXT = 'pdf'#
Default plot file extension
- PLOT_HELPER_CLS#
The PlotHelper class to use; here, the utopya-specific one
alias of
utopya.eval.plothelper.PlotHelper
- _abc_impl = <_abc._abc_data object>#
- class utopya.eval.plotcreators.UniversePlotCreator(*args, psgrp_path: Optional[str] = None, **kwargs)[source]#
Bases:
dantro.plot.creators.psp.UniversePlotCreator
,utopya.eval.plotcreators.PyPlotCreator
Makes plotting with data from a single universe more convenient
- _abc_impl = <_abc._abc_data object>#
- PSGRP_PATH: str = 'multiverse'#
The path within the data tree to arrive at the
MultiverseGroup
that this plot creator expects universes to be located in.
- class utopya.eval.plotcreators.MultiversePlotCreator(*args, psgrp_path: Optional[str] = None, **kwargs)[source]#
Bases:
dantro.plot.creators.psp.MultiversePlotCreator
,utopya.eval.plotcreators.PyPlotCreator
Makes plotting with data from all universes more convenient
- _abc_impl = <_abc._abc_data object>#
- PSGRP_PATH: str = 'multiverse'#
The path within the data tree to arrive at the
MultiverseGroup
.
utopya.eval.plothelper module#
Implements utopya’s PlotHelper specialization
- class utopya.eval.plothelper.PlotHelper(*, out_path: str, helper_defaults: Optional[dict] = None, update_helper_cfg: Optional[dict] = None, raise_on_error: bool = True, animation_enabled: bool = False)[source]#
Bases:
dantro.plot.plot_helper.PlotHelper
A specialization of dantro’s
PlotHelper
which is used for creatingmatplotlib.pyplot
-based plots usingPyPlotCreator
.This can be used to add additional helpers for use in utopya without requiring changes to dantro.
Note
The helpers implemented here should try to adhere to the interface exemplified by the dantro
PlotHelper
, with the aim that they can then be migrated into dantro in the long run.
utopya.eval.plotmanager module#
Implements a plotting framework based on dantro
In order to make the plotting framework specific to Utopia, this module derives both from the dantro PlotManager and some PlotCreator classes.
- class utopya.eval.plotmanager.PlotManager(*args, _model_info_bundle: Optional[utopya.model_registry.info_bundle.ModelInfoBundle] = None, **kwargs)[source]#
Bases:
dantro.plot_mngr.PlotManager
This is the Utopia-specific version of the dantro
PlotManager
.It registers the Utopia-specific plot creators and allows for custom interface specifications, e.g. by preloading custom modules.
- CREATORS: Dict[str, type] = {'base': <class 'dantro.plot.creators.base.BasePlotCreator'>, 'external': <class 'utopya.eval.plotcreators.PyPlotCreator'>, 'multiverse': <class 'utopya.eval.plotcreators.MultiversePlotCreator'>, 'pyplot': <class 'utopya.eval.plotcreators.PyPlotCreator'>, 'universe': <class 'utopya.eval.plotcreators.UniversePlotCreator'>}#
Supported plot creator classes
- MODEL_PLOTS_MODULE_NAME = 'model_plots'#
Name under which the model-specific plots are made importable
- PLOT_FUNC_RESOLVER#
The custom plot function resolver type to use.
- __init__(*args, _model_info_bundle: Optional[utopya.model_registry.info_bundle.ModelInfoBundle] = None, **kwargs)[source]#
Sets up a PlotManager.
This specialization of the
dantro.plot_mngr.PlotManager
additionally stores some utopya-specific metadata in form of aModelInfoBundle
that describes the model this PlotManager is used with. That information is then used to load some additional model-specific information once a creator is invoked.Furthermore, the
_preload_modules()
method takes care to make model-, project-, or framework-specific plot functions available.- Parameters
*args – Positional arguments passed to
PlotManager
._model_info_bundle (ModelInfoBundle, optional) – The internally-used argument to pass model information to the plot manager.
**kwargs – Keyword arguments passed on to
PlotManager
.
- property common_out_dir: str#
The common output directory of all plots that were created with this plot manager instance. This uses the plot output paths stored in the plot information dict, specifically the
target_dir
entry.If there was no plot information yet, the return value will be empty.
- plot_from_cfg(*args, plots_cfg: Optional[Union[dict, str]] = None, **kwargs)[source]#
Thin wrapper around parent method that shows which plot configuration file will be used.
- _get_plot_func_resolver(**init_kwargs) utopya.eval._plot_func_resolver.PlotFuncResolver [source]#
Instantiates the plot function resolver object.
Additionally attaches the model info bundle to the resolver, such that it can use that information for plot function lookup.
- _get_plot_creator(*args, **kwargs) dantro.plot.creators.base.BasePlotCreator [source]#
Sets up the BasePlotCreator and attaches a model information bundle to it such that this information is available downstream.
- _preload_modules()[source]#
Pre-loads the model-, project-, and framework-specific plot function modules. This allows to execute code (like registering model-specific dantro data operations) and have them available prior to the invocation of the creator and independently from the module that contains the plot function (which may be part of dantro, for instance).
utopya.eval.transform module#
Overloads and additions for dantro’s data transformation framework.
- utopya.eval.transform.register_operation(*args, skip_existing: bool = True, **kws) None [source]#
Register an operation with the dantro data operations database.
This invokes
register_operation()
, but hasskip_existing == True
as default in order to reduce number of arguments that need to be specified in utopya model plots, where duplicate module imports frequently cause existing entries.- Parameters
*args – Passed to
dantro.data_ops.db_tools.register_operation()
skip_existing (bool, optional) – Whether to skip (without an error) if an operation
**kws – Passed to
dantro.data_ops.db_tools.register_operation()
- utopya.eval.transform.is_operation(arg: Optional[Union[Callable, str]] = None, /, **kws)[source]#
Overload of dantro’s
is_operation
decorator, using utopya’s own registration function.Usage example:
from utopya.eval import is_operation @is_operation def my_operation(data, *args): pass @is_operation("op_with_custom_name") def my_other_operation(data, *args): pass @is_operation("my_operation", overwrite_existing=True) def yet_some_other_operation(data, *args): pass
- Parameters
arg (Union[str, Callable], optional) – The name that should be used in the operation registry. If not given, will use the name of the decorated function instead. If a callable, this refers to the
@is_operation
call syntax and will use that as a function.**kws – Passed to
register_operation()
.