smif.data_layer package

Subpackages

Submodules

smif.data_layer.abstract_config_store module

smif.data_layer.abstract_data_store module

A data store holds the bulk of model setup, intermediate and output data: - scenario variant and narrative data (including parameter defaults) - model interventions, initial conditions and state - conversion coefficients - results

class smif.data_layer.abstract_data_store.DataStore[source]

Bases: object

A DataStore must implement each of the abstract methods defined in this interface

abstractmethod available_results(modelrun_name)[source]

List available results from a model run

Returns:

Each tuple is (timestep, decision_iteration, model_name, output_name)

Return type:

list[tuple]

static dataframe_to_data_array(dataframe, spec, path)[source]
abstractmethod delete_results(model_run_name, model_name, output_name, timestep=None, decision_iteration=None)[source]

Delete results for a single timestep/iteration of a model output in a model run

Parameters:

  • model_run_name (str)

  • model_name (str)

  • output_name (str)

  • timestep (int, default=None)

  • decision_iteration (int, default=None)

classmethod filter_on_timesteps(dataframe, spec, path, timestep=None, timesteps=None)[source]

Filter dataframe by timestep

The ‘timestep’ dimension is treated as follows:

If a single timestep is specified, the spec MAY include ‘timestep’ as a dimension. If so, the returned DataArray’s spec will match the timestep requested. Otherwise, the DataArray will not include timestep as a dimension.

If multiple timesteps are specified, the returned DataArray’s spec will include a ‘timestep’ dimension to match the timesteps requested.

If timestep and timesteps are None, and the stored data has a timestep column, read all available timesteps. The returned DataArray’s spec ‘timestep’ dimension will match the timesteps requested. If the stored data does not have a timestep column, ignore and pass through unchanged.

abstractmethod read_coefficients(source_dim, destination_dim)[source]

Reads coefficients from the store

Coefficients are uniquely identified by their source/destination dimensions. This method and write_coefficients implement caching of conversion coefficients between a single pair of dimensions.

Parameters:

  • source_dim (str) – dimension name

  • destination_dim (str) – dimension name

Return type:

numpy.ndarray

Notes

To be called from Adaptor implementations.

abstractmethod read_initial_conditions(key) → List[Dict][source]

Read historical interventions for key

Parameters:

key (str)

Return type:

list[dict]

abstractmethod read_interventions(key)[source]

Read interventions data for key

Parameters:

key (str)

Returns:

A dict of intervention dictionaries containing intervention attributes keyed by intervention name

Return type:

dict[str, dict]

abstractmethod read_model_parameter_default(key, spec)[source]

Read data array

Parameters:
Returns:

data_array

Return type:

DataArray

abstractmethod read_narrative_variant_data(key, spec, timestep=None)[source]

Read data array

Parameters:

  • key (str)

  • spec (Spec)

  • timestep (int (optional)) – If None, read data for all timesteps

Returns:

data_array

Return type:

DataArray

abstractmethod read_results(modelrun_name, model_name, output_spec, timestep=None, decision_iteration=None) → DataArray[source]

Return results of a model from a model_run for a given output at a timestep and decision iteration

Parameters:

  • model_run_id (str)

  • model_name (str)

  • output_spec (Spec)

  • timestep (int, default=None)

  • decision_iteration (int, default=None)

Return type:

DataArray

abstractmethod read_scenario_variant_data(key, spec, timestep=None, timesteps=None) → DataArray[source]

Read scenario variant data array.

If a single timestep is specified, the spec MAY include ‘timestep’ as a dimension, which should match the timestep specified.

If multiple timesteps are specified, the spec MUST include ‘timestep’ as a dimension, which should match the timesteps specified.

If timestep and timesteps are None, read all available timesteps. Whether or not the spec includes ‘timestep’ as a dimension, the returned DataArray will include a ‘timestep’ dimension with all available timesteps included.

Parameters:

  • key (str)

  • spec (Spec)

  • timestep (int (optional)) – If set, read data for single timestep

  • timesteps (list[int] (optional)) – If set, read data for specified timesteps

Returns:

data_array

Return type:

DataArray

abstractmethod read_state(modelrun_name, timestep, decision_iteration=None) → List[Dict][source]

Read list of (name, build_year) for a given model_run, timestep, decision

Parameters:

  • model_run_name (str)

  • timestep (int)

  • decision_iteration (int, optional)

Return type:

list[dict]

abstractmethod scenario_variant_data_exists(key) → bool[source]

Test if scenario variant data exists

Parameters:

key (str)

Return type:

bool

abstractmethod write_coefficients(source_dim, destination_dim, data)[source]

Writes coefficients to the store

Coefficients are uniquely identified by their source/destination dimensions. This method and read_coefficients implement caching of conversion coefficients between a single pair of dimensions.

Parameters:

  • source_dim (str) – dimension name

  • destination_dim (str) – dimension name

  • data (numpy.ndarray)

Notes

To be called from Adaptor implementations.

abstractmethod write_initial_conditions(key, initial_conditions)[source]

Write historical interventions for key

Parameters:
abstractmethod write_interventions(key, interventions)[source]

Write interventions data for key

Parameters:
abstractmethod write_model_parameter_default(key, data_array)[source]

Read data array

Parameters:
Returns:

data_array

Return type:

DataArray

abstractmethod write_narrative_variant_data(key, data_array)[source]

Write data array

Parameters:
abstractmethod write_results(data, modelrun_name, model_name, timestep=None, decision_iteration=None)[source]

Write results of a model_name in model_run_name for a given output_name

Parameters:

  • data_array (DataArray)

  • model_run_id (str)

  • model_name (str)

  • timestep (int, optional)

  • decision_iteration (int, optional)

abstractmethod write_scenario_variant_data(key, data_array)[source]

Write data array

Parameters:
abstractmethod write_state(state: List[Dict], modelrun_name: str, timestep: int, decision_iteration=None)[source]

State is a list of decisions with name and build_year.

State is output from the DecisionManager

Parameters:

  • state (list[dict])

  • model_run_name (str)

  • timestep (int)

  • decision_iteration (int, optional)

smif.data_layer.abstract_metadata_store module

A metadata store holds shared metadata for smif model scenarios, inputs, parameters and outputs: - units - dimension definitions

class smif.data_layer.abstract_metadata_store.MetadataStore[source]

Bases: object

A MetaDataStore must implement each of the abstract methods defined in this interface

abstractmethod delete_dimension(dimension_name)[source]

Delete dimension

Parameters:

dimension_name (str)

abstractmethod read_dimension(dimension_name, skip_coords=False)[source]

Return dimension

Parameters:

  • dimension_name (str)

  • skip_coords (bool, default False) – If True, skip reading dimension elements (names and metadata)

Returns:

A dimension definition (including elements)

Return type:

Coords

abstractmethod read_dimensions(skip_coords=False)[source]

Read dimensions

Parameters:

skip_coords (bool, default False) – If True, skip reading dimension elements (names and metadata)

Return type:

list[Coords]

abstractmethod read_unit_definitions() → List[str][source]

Reads custom unit definitions

Returns:

Pint-compatible unit definitions

Return type:

list[str]

abstractmethod update_dimension(dimension_name, dimension)[source]

Update dimension

Parameters:

  • dimension_name (str)

  • dimension (Coords)

abstractmethod write_dimension(dimension)[source]

Write dimension to project configuration

Parameters:

dimension (Coords)

write_unit_definitions(definitions: List[str])[source]

Reads custom unit definitions

Parameters:

list[str] – Pint-compatible unit definitions

smif.data_layer.data_array module

DataArray provides a thin wrapper around multidimensional arrays and metadata

class smif.data_layer.data_array.DataArray(spec: ~smif.metadata.spec.Spec, data: <MagicMock id='140318177297328'>)[source]

Bases: object

DataArray provides access to input/parameter/results data, with conversions to common python data libraries (for example: numpy, pandas, xarray).

spec
Type:

smif.metadata.spec.Spec

data
Type:

numpy.ndarray

as_df() → <MagicMock id='140318174677952'>[source]

Access DataArray as a pandas.DataFrame

as_dict()[source]
as_ndarray() → <MagicMock id='140318174723152'>[source]

Access as a numpy.ndarray

as_xarray()[source]

Access DataArray as a xarray.DataArray

property coords

Coordinate labels for each dimension.

property description

A human-friendly description

dim_coords(dim)[source]

Coordinates for a given dimension

dim_elements(dim)[source]

Coordinate elements for a given dimension

dim_names(dim)[source]

Coordinate names for a given dimension

property dims

Names for each dimension

classmethod from_df(spec, dataframe)[source]

Create a DataArray from a pandas.DataFrame

classmethod from_xarray(spec, xr_data_array)[source]

Create a DataArray from a xarray.DataArray

property name

The name of the data that this spec describes.

property shape

The shape of the data array

property unit

The unit for all data points.

update(other)[source]

Update data values with any from other which are non-null

validate_as_full()[source]

Check that the data array contains no NaN values

smif.data_layer.data_array.find_duplicate_indices(dataframe)[source]

Find duplicate indices in a DataFrame

Return type:

list[dict]

smif.data_layer.data_array.show_null(dataframe) → <MagicMock id='140318174685296'>[source]

Shows missing data

Return type:

pandas.DataFrame

smif.data_layer.data_handle module

smif.data_layer.memory_interface module

smif.data_layer.model_loader module

smif.data_layer.results module

smif.data_layer.store module

smif.data_layer.validate module