smif.data_layer.datafile_interface module

File-backed data interface

Summary

Classes:

DatafileInterface

Read and write interface to YAML / CSV configuration files and intermediate CSV / native-binary data storage.

Reference

class smif.data_layer.datafile_interface.DatafileInterface(base_folder, storage_format='local_binary')

Bases: smif.data_layer.data_interface.DataInterface

Read and write interface to YAML / CSV configuration files and intermediate CSV / native-binary data storage.

Project.yml

Parameters:

• base_folder (str) – The path to the configuration and data files
• storage_format (str) – The format used to store intermediate data (local_csv, local_binary)

read_units_file_name()

Reads the unit definition file name from the store

Returns:Path to the units file Return type:str

read_sos_model_runs()

Read all system-of-system model runs

Returns:A list of sos_model_run dicts Return type:list

read_sos_model_run(sos_model_run_name)

Read a system-of-system model run

Parameters:sos_model_run_name (str) – A sos_model_run name Returns:sos_model_run – A sos_model_run dictionary Return type:dict

write_sos_model_run(sos_model_run)

Write system-of-system model run

Parameters:sos_model_run (dict) – A sos_model_run dictionary

update_sos_model_run(sos_model_run_name, sos_model_run)

Update system-of-system model run

Parameters:

• sos_model_run_name (str) – A sos_model_run name
• sos_model_run (dict) – A sos_model_run dictionary

delete_sos_model_run(sos_model_run_name)

Delete a system-of-system model run

Parameters:sos_model_run_name (str) – A sos_model_run name

read_sos_models()

Read all system-of-system models

Returns:A list of sos_model dicts Return type:list

read_sos_model(sos_model_name)

Read a specific system-of-system model

Parameters:sos_model_name (str) – A sos_model name Returns:sos_model – A sos_model dictionary Return type:dict

write_sos_model(sos_model)

Write system-of-system model

Parameters:sos_model (dict) – A sos_model dictionary

update_sos_model(sos_model_name, sos_model)

Update system-of-system model

Parameters:

• sos_model_name (str) – A sos_model name
• sos_model (dict) – A sos_model dictionary

delete_sos_model(sos_model_name)

Delete a system-of-system model

Parameters:sos_model_name (str) – A sos_model name

read_sector_models()

Read all sector models

sector_models.yml

Returns:A list of sector_model dicts Return type:list

read_sector_model(sector_model_name)

Read a sector model

Parameters:sector_model_name (str) – A sector_model name Returns:sector_model – A sector_model dictionary Return type:dict

write_sector_model(sector_model)

Write sector model

Parameters:sector_model (dict) – A sector_model dictionary

update_sector_model(sector_model_name, sector_model)

Update sector model

Parameters:

• sector_model_name (str) – A sector_model name
• sector_model (dict) – A sector_model dictionary

delete_sector_model(sector_model_name)

Delete a sector model

Parameters:sector_model_name (str) – A sector_model name

read_interventions(filename)

read_strategies(filename)

read_initial_conditions(filename)

read_state(modelrun_name, timestep, decision_iteration=None)

Read list of (name, build_year) for a given modelrun, timestep, decision

write_state(state, modelrun_name, timestep=None, decision_iteration=None)

State is a list of decision dicts with name and build_year keys,

State is output from the DecisionManager

Parameters:

• state (list) –
• modelrun_name (str) –
• timestep (int) –
• decision_iteration (int, default=None) –

read_region_definitions()

Read region_definitions from project configuration

Returns:A list of region_definition dicts Return type:list

read_region_definition_data(region_definition_name)

Read region_definition data file into a Fiona feature collection

The file format must be possible to parse with GDAL, and must contain an attribute “name” to use as an identifier for the region_definition.

Parameters:region_definition_name (str) – Name of the region_definition Returns:A list of data from the specified file in a fiona formatted dict Return type:list

read_region_names

Return the set of unique region names in region set region_definition_name

write_region_definition(region_definition)

Write region_definition to project configuration

Parameters:region_definition (dict) – A region_definition dict

Notes

Unused

update_region_definition(region_definition_name, region_definition)

Update region_definition to project configuration

Parameters:

• region_definition_name (str) – Name of the (original) entry
• region_definition (dict) – The updated region_definition dict

Notes

Unused

read_interval_definitions()

Read interval_definition sets from project configuration

Returns:A list of interval_definition set dicts Return type:list

read_interval_definition_data(interval_definition_name)

Read data for an interval definition

Parameters:interval_definition_name (str) – Returns:Interval definition data Return type:dict

Notes

Expects csv file to contain headings of id, start, end

read_interval_names(interval_definition_name)

Reads the unique set of interval names

Parameters:interval_definition_name (str) – Returns: Return type:list

write_interval_definition(interval_definition)

Write interval_definition to project configuration

Parameters:interval_definition (dict) – A interval_definition dict

Notes

Unused

update_interval_definition(interval_definition_name, interval_definition)

Update interval_definition to project configuration

Parameters:

• interval_definition_name (str) – Name of the (original) entry
• interval_definition (dict) – The updated interval_definition dict

read_scenario_set_scenario_definitions(scenario_set_name)

Read all scenarios from a certain scenario_set

Parameters:scenario_set_name (str) – Name of the scenario_set Returns:A list of scenarios within the specified scenario_set_name Return type:list

Notes

Unused

read_scenario_definition(scenario_name)

Read scenario definition data

Parameters:scenario_name (str) – Name of the scenario Returns:The scenario definition Return type:dict

read_scenario_sets()

Read scenario sets from project configuration

Returns:A list of scenario set dicts Return type:list

read_scenario_set(scenario_set_name)

Read a scenario_set

Parameters:scenario_set_name (str) – Name of the scenario_set Returns:Scenario set definition Return type:dict

write_scenario_set(scenario_set)

Write scenario_set to project configuration

Parameters:scenario_set (dict) – A scenario_set dict

update_scenario_set(scenario_set_name, scenario_set)

Update scenario_set to project configuration

Parameters:

• scenario_set_name (str) – Name of the (original) entry
• scenario_set (dict) – The updated scenario_set dict

delete_scenario_set(scenario_set_name)

Delete scenario_set from project configuration and all scenarios within scenario_set

Parameters:scenario_set_name (str) – A scenario_set name

read_scenarios()

Read scenarios from project configuration

Returns:A list of scenario dicts Return type:list

read_scenario(scenario_name)

Read a scenario

Parameters:scenario_name (str) – Name of the scenario Returns:A scenario dictionary Return type:dict

write_scenario(scenario)

Write scenario to project configuration

Parameters:scenario (dict) – A scenario dict

update_scenario(scenario_name, scenario)

Update scenario to project configuration

Parameters:

• scenario_name (str) – Name of the (original) entry
• scenario (dict) – The updated scenario dict

delete_scenario(scenario_name)

Delete scenario from project configuration

Parameters:scenario_name (str) – A scenario name

read_scenario_data(scenario_name, facet_name, spatial_resolution, temporal_resolution, timestep)

Read scenario data file

Parameters:

• scenario_name (str) – Name of the scenario
• facet_name (str) – Name of the scenario facet to read
• spatial_resolution (str) –
• temporal_resolution (str) –
• timestep (int) –

Returns:

data

Return type:

numpy.ndarray

Notes

Called from smif.data_layer.data_handle

read_narrative_sets()

Read narrative sets from project configuration

Returns:A list of narrative set dicts Return type:list

read_narrative_set(narrative_set_name)

Read all narratives from a certain narrative_set

Parameters:narrative_set_name (str) – Name of the narrative_set Returns:A narrative_set dictionary Return type:list

write_narrative_set(narrative_set)

Write narrative_set to project configuration

Parameters:narrative_set (dict) – A narrative_set dict

update_narrative_set(narrative_set_name, narrative_set)

Update narrative_set to project configuration

Parameters:

• narrative_set_name (str) – Name of the (original) entry
• narrative_set (dict) – The updated narrative_set dict

delete_narrative_set(narrative_set_name)

Delete narrative_set from project configuration

Parameters:narrative_set_name (str) – A narrative_set name

read_narratives()

Read narrative sets from project configuration

Returns:A list of narrative set dicts Return type:list

read_narrative(narrative_name)

Read all narratives from a certain narrative

Parameters:narrative_name (str) – Name of the narrative Returns:A narrative dictionary Return type:list

write_narrative(narrative)

Write narrative to project configuration

Parameters:narrative (dict) – A narrative dict

update_narrative(narrative_name, narrative)

Update narrative to project configuration

Parameters:

• narrative_name (str) – Name of the (original) entry
• narrative (dict) – The updated narrative dict

delete_narrative(narrative_name)

Delete narrative from project configuration

Parameters:narrative_name (str) – A narrative name

read_narrative_data(narrative_name)

Read narrative data file

Parameters:narrative_name (str) – Name of the narrative Returns:A list with dictionaries containing the contents of ‘narrative_name’ data file Return type:list

read_coefficients(source_name, destination_name)

Reads coefficients from the store

Coefficients are uniquely identified by their source/destination names. This method and write_coefficients implement caching of conversion coefficients between dimensions.

Parameters:

• source_name (str) – The name of a ResolutionSet
• destination_name (str) – The name of a ResolutionSet

Notes

Both source_name and destination_name should reference names of elements from the same ResolutionSet (e.g. both spatial or temporal resolutions)

Called from smif.convert.register

write_coefficients(source_name, destination_name, data)

Writes coefficients to the store

Coefficients are uniquely identified by their source/destination names.

This method and read_coefficients implement caching of conversion coefficients between dimensions.

source_name : str

The name of a ResolutionSet

destination_name : str

The name of a ResolutionSet

data : numpy.ndarray

Both source_name and destination_name should reference names of elements from the same ResolutionSet (e.g. both spatial or temporal resolutions)

Called from smif.convert.register

read_results(modelrun_id, model_name, output_name, spatial_resolution, temporal_resolution, timestep=None, modelset_iteration=None, decision_iteration=None)

Return results of a model_name in modelrun_name for a given output_name

Parameters:

• modelrun_id (str) –
• model_name (str) –
• output_name (str) –
• spatial_resolution (str) –
• temporal_resolution (str) –
• timestep (int, default=None) –
• modelset_iteration (int, default=None) –
• decision_iteration (int, default=None) –

Returns:

data

Return type:

numpy.ndarray

Notes

Called from smif.data_layer.data_handle

write_results(modelrun_id, model_name, output_name, data, spatial_resolution, temporal_resolution, timestep=None, modelset_iteration=None, decision_iteration=None)

Write results of a model_name in modelrun_name for a given output_name

Parameters:

• modelrun_id (str) –
• model_name (str) –
• output_name (str) –
• data (numpy.ndarray) –
• spatial_resolution (str) –
• temporal_resolution (str) –
• timestep (int, optional) –
• modelset_iteration (int, optional) –
• decision_iteration (int, optional) –

Notes

Called from smif.data_layer.data_handle

prepare_warm_start(modelrun_id)

Copy the results from the previous modelrun if available

The method allows a previous unsuccessful modelrun to ‘warm start’ a new model run from a later timestep. Model results are recovered from the timestep that the previous modelrun was run until, and the new model run runs from the returned timestep

Parameters:modelrun_id (str) – The name of the modelrun to recover Returns:The timestep to which the data store was recovered Return type:int

Notes

Called from smif.controller.execute