smif.data_layer.store module¶
The store provides a common data interface to smif configuration, data and metadata.
raises: |
|
---|
Summary¶
Data:
Store |
Common interface to data store, composed of config, metadata and data store implementations. |
Reference¶
-
class
smif.data_layer.store.
Store
(config_store, metadata_store: smif.data_layer.abstract_metadata_store.MetadataStore, data_store: smif.data_layer.abstract_data_store.DataStore, model_base_folder='.')[source]¶ Bases:
object
Common interface to data store, composed of config, metadata and data store implementations.
Parameters: - config_store (ConfigStore) –
- metadata_store (MetadataStore) –
- data_store (DataStore) –
-
read_model_run
(model_run_name)[source]¶ Read a system-of-system model run
Parameters: model_run_name (str) – Returns: Return type: ModelRun
-
write_model_run
(model_run)[source]¶ Write system-of-system model run
Parameters: model_run (ModelRun) –
-
delete_model_run
(model_run_name)[source]¶ Delete a system-of-system model run
Parameters: model_run_name (str) –
-
read_sos_model
(sos_model_name)[source]¶ Read a specific system-of-system model
Parameters: sos_model_name (str) – Returns: Return type: SosModel
-
delete_sos_model
(sos_model_name)[source]¶ Delete a system-of-system model
Parameters: sos_model_name (str) –
-
read_model
(model_name, skip_coords=False)[source]¶ Read a model
Parameters: model_name (str) – Returns: Return type: Model
-
read_scenario
(scenario_name, skip_coords=False)[source]¶ Read a scenario
Parameters: scenario_name (str) – Returns: Return type: ScenarioModel
-
update_scenario
(scenario_name, scenario)[source]¶ Update scenario
Parameters: - scenario_name (str) –
- scenario (ScenarioModel) –
-
delete_scenario
(scenario_name)[source]¶ Delete scenario from project configuration
Parameters: scenario_name (str) –
-
prepare_scenario
(scenario_name, list_of_variants)[source]¶ Modify {scenario_name} config file to include multiple scenario variants.
Parameters:
-
prepare_model_runs
(model_run_name, scenario_name, first_var, last_var)[source]¶ Write multiple model run config files corresponding to multiple scenario variants of {scenario_name}, based on template {model_run_name}
Write batchfile containing each of the generated model runsParameters:
-
read_scenario_variants
(scenario_name)[source]¶ Read variants of a given scenario
Parameters: scenario_name (str) – Returns: Return type: list[dict]
-
read_scenario_variant
(scenario_name, variant_name)[source]¶ Read a scenario variant
Parameters: Returns: Return type:
-
write_scenario_variant
(scenario_name, variant)[source]¶ Write scenario to project configuration
Parameters:
-
update_scenario_variant
(scenario_name, variant_name, variant)[source]¶ Update scenario to project configuration
Parameters:
-
delete_scenario_variant
(scenario_name, variant_name)[source]¶ Delete scenario from project configuration
Parameters:
-
read_strategies
(model_run_name)[source]¶ Read strategies for a given model run
Parameters: model_run_name (str) – Returns: Return type: list[dict]
-
write_strategies
(model_run_name, strategies)[source]¶ Write strategies for a given model_run
Parameters:
-
read_unit_definitions
() → List[str][source]¶ Reads custom unit definitions
Returns: Pint-compatible unit definitions Return type: list[str]
-
write_unit_definitions
(definitions)[source]¶ Reads custom unit definitions
Parameters: list[str] – Pint-compatible unit definitions
-
read_dimension
(dimension_name, skip_coords=False)[source]¶ Return dimension
Parameters: dimension_name (str) – Returns: A dimension definition (including elements) Return type: Coords
-
write_dimension
(dimension)[source]¶ Write dimension to project configuration
Parameters: dimension (Coords) –
-
update_dimension
(dimension_name, dimension)[source]¶ Update dimension
Parameters: - dimension_name (str) –
- dimension (Coords) –
-
read_scenario_variant_data
(scenario_name: str, variant_name: str, variable: str, timestep: Optional[int] = None, timesteps: Optional[List[int]] = None, assert_exists: bool = False) → Union[smif.data_layer.data_array.DataArray, bool][source]¶ Read scenario data file
Parameters: Returns: data
Return type:
-
write_scenario_variant_data
(scenario_name, variant_name, data)[source]¶ Write scenario data file
Parameters:
-
read_narrative_variant_data
(sos_model_name, narrative_name, variant_name, parameter_name, timestep=None, assert_exists=False)[source]¶ Read narrative data file
Parameters: Returns: Return type:
-
write_narrative_variant_data
(sos_model_name, narrative_name, variant_name, data)[source]¶ Read narrative data file
Parameters:
-
read_model_parameter_default
(model_name, parameter_name, assert_exists=False)[source]¶ Read default data for a sector model parameter
Parameters: Returns: Return type:
-
write_model_parameter_default
(model_name, parameter_name, data)[source]¶ Write default data for a sector model parameter
Parameters:
-
read_interventions
(model_name)[source]¶ Read interventions data for model_name
Returns: A dict of intervention dictionaries containing intervention attributes keyed by intervention name Return type: dict[str, dict]
-
write_interventions
(model_name, interventions)[source]¶ Write interventions data for a model
Parameters: dict] (dict[str,) – A dict of intervention dictionaries containing intervention attributes keyed by intervention name
-
read_strategy_interventions
(strategy, assert_exists=False)[source]¶ Read interventions as defined in a model run strategy
-
read_initial_conditions
(model_name) → List[Dict][source]¶ Read historical interventions for model_name
Returns: A list of historical interventions, with keys ‘name’ and ‘build_year’ Return type: list[dict]
-
write_initial_conditions
(model_name, initial_conditions)[source]¶ Write historical interventions for a model
Parameters: list[dict] – A list of historical interventions, with keys ‘name’ and ‘build_year’
-
read_all_initial_conditions
(model_run_name) → List[Dict][source]¶ A list of all historical interventions
Returns: Return type: list[dict]
-
read_state
(model_run_name, timestep, decision_iteration=None) → List[Dict][source]¶ Read list of (name, build_year) for a given model_run, timestep, decision
Parameters: Returns: Return type:
-
write_state
(state, model_run_name, timestep, decision_iteration=None)[source]¶ State is a list of decisions with name and build_year.
State is output from the DecisionManager
Parameters:
-
read_coefficients
(source_dim: str, destination_dim: str) → <MagicMock id='140320895167848'>[source]¶ Reads coefficients from the store
Coefficients are uniquely identified by their source/destination dimensions. This method and write_coefficients implement caching of conversion coefficients between dimensions.
Parameters: Returns: Return type: Notes
To be called from
Adaptor
implementations.
-
write_coefficients
(source_dim: str, destination_dim: str, data: <MagicMock id='140320895225416'>)[source]¶ Writes coefficients to the store
Coefficients are uniquely identified by their source/destination dimensions. This method and read_coefficients implement caching of conversion coefficients between dimensions.
Parameters: - source_dim (str) – Dimension name
- destination_dim (str) – Dimension name
- data (numpy.ndarray) –
Notes
To be called from
Adaptor
implementations.
-
read_results
(model_run_name: str, model_name: str, output_spec: smif.metadata.spec.Spec, timestep: Optional[int] = None, decision_iteration: Optional[int] = None) → smif.data_layer.data_array.DataArray[source]¶ Return results of a model_name in model_run_name for a given output_name
Parameters: Returns: Return type:
-
write_results
(data_array, model_run_name, model_name, timestep=None, decision_iteration=None)[source]¶ Write results of a model_name in model_run_name for a given output_name
Parameters:
-
delete_results
(model_run_name, model_name, output_name, timestep=None, decision_iteration=None)[source]¶ Delete results for a single timestep/iteration of a model output in a model run
Parameters:
-
clear_results
(model_run_name)[source]¶ Clear all results from a single model run
Parameters: model_run_name (str) –
-
available_results
(model_run_name)[source]¶ List available results from a model run
Parameters: model_run_name (str) – Returns: Each tuple is (timestep, decision_iteration, model_name, output_name) Return type: list[tuple]
-
completed_jobs
(model_run_name)[source]¶ List completed jobs from a model run
Parameters: model_run_name (str) – Returns: Each tuple is (timestep, decision_iteration, model_name) Return type: list[tuple]
-
static
filter_complete_available_results
(available_results, expected_model_outputs)[source]¶ Filter available results from a model run to include only complete timestep/decision iteration combinations
Parameters: Returns: Each tuple is (timestep, decision_iteration, model_name)
Return type:
-
expected_model_outputs
(model_run_name)[source]¶ List expected model outputs from a model run
Parameters: model_run_name (str) – Returns: Each tuple is (model_name, output_name) Return type: list[tuple]
-
prepare_warm_start
(model_run_name)[source]¶ Copy the results from the previous model_run if available
The method allows a previous unsuccessful model_run to ‘warm start’ a new model run from a later timestep. Model results are recovered from the timestep that the previous model_run was run until, and the new model run runs from the returned timestep
Parameters: model_run_name (str) – Returns: Return type: int The timestep to which the data store was recovered Notes
Called from smif.controller.execute
-
canonical_available_results
(model_run_name)[source]¶ List the results that are available from a model run, collapsing all decision iterations.
This is the unique items from calling available_results, with all decision iterations set to 0.
This method is used to determine whether a model run is complete, given that it is impossible to know how many decision iterations to expect: we simply check that each expected timestep has been completed.
Parameters: model_run_name (str) – Returns: Return type: set Set of tuples representing available results
-
canonical_expected_results
(model_run_name)[source]¶ List the results that are expected from a model run, collapsing all decision iterations.
For a complete model run, this would coincide with the unique list returned from available_results, where all decision iterations are set to 0.
This method is used to determine whether a model run is complete, given that it is impossible to know how many decision iterations to expect: we simply check that each expected timestep has been completed.
Parameters: model_run_name (str) – Returns: Return type: set Set of tuples representing expected results
-
canonical_missing_results
(model_run_name)[source]¶ List the results that are missing from a model run, collapsing all decision iterations.
For a complete model run, this is what is left after removing canonical_available_results from canonical_expected_results.
Parameters: model_run_name (str) – Returns: Return type: set Set of tuples representing missing results
-
get_result_darray
(model_run_name, model_name, output_name, timesteps=None, decision_iterations=None, time_decision_tuples=None)[source]¶ Return data for multiple timesteps and decision iterations for a given output from a given sector model in a specific model run.
- You can specify either:
- a list of (timestep, decision) tuples
- in which case data for all of those tuples matching the available results will be returned
- or:
- a list of timesteps
- in which case data for all of those timesteps (and any decision iterations) matching the available results will be returned
- or:
- a list of decision iterations
- in which case data for all of those decision iterations (and any timesteps) matching the available results will be returned
- or:
- a list of timesteps and a list of decision iterations
- in which case data for the Cartesian product of those timesteps and those decision iterations matching the available results will be returned
- or:
- nothing
- in which case all available results will be returned
Then, for each tuple, the data array from the corresponding read_results call is stacked, and together with the new spec this information is returned as a new DataArray.
Parameters: Returns: Return type: DataArray with expanded spec and the data requested
-
get_results
(model_run_names: list, model_name: str, output_names: list, timesteps: list = None, decisions: list = None, time_decision_tuples: list = None)[source]¶ Return data for multiple timesteps and decision iterations for a given output from a given sector model for multiple model runs.
Parameters: - model_run_names (list[str]) – the requested model run names
- model_name (str) – the requested sector model name
- output_names (list[str]) – the requested output names (output specs must all match)
- timesteps (list[int]) – the requested timesteps
- decisions (list[int]) – the requested decision iterations
- time_decision_tuples (list[tuple]) – a list of requested (timestep, decision) tuples
Returns: Nested dictionary of DataArray objects, keyed on model run name and output name. Returned DataArrays include one extra (timestep, decision_iteration) dimension.
Return type: