smif.data_layer package

Submodules

smif.data_layer.load module

Parse yaml config files, to construct sector models

smif.data_layer.load.dump(data, file_path)

Write plain data to a file as yaml

Parameters:

• file_path (str) – The path of the configuration file to write
• data – Data to write (should be lists, dicts and simple values)

smif.data_layer.load.load(file_path)

Parse yaml config file into plain data (lists, dicts and simple values)

Parameters:file_path (str) – The path of the configuration file to parse

smif.data_layer.sector_model_config module

Read and parse the config for sector models

class smif.data_layer.sector_model_config.SectorModelReader(initial_config=None)

Bases: object

Parses the configuration and input data for a sector model

Parameters:initial_config (dict) –

Sector model details, sufficient to read the full config from a set of files. Must contain the following fields:

”model_name”

The name of the sector model, for reference within the system-of-systems model

”model_path”

The path to the python module file that contains an implementation of SectorModel

”model_classname”

The name of the class that implements SectorModel

”model_config_dir”

The root path of model config/data to use, which must contain inputs.yaml, outputs.yaml, time_intervals.yaml and regions.shp/regions.geojson

”initial_conditions”

List of files containing initial conditions

”interventions”

List of files containing interventions

”parameters”

List of files containing parameter configuration

data

Expose all loaded config data

Returns:data –

Model configuration data, with the following fields:

”name”: The name of the sector model, for reference within the system-of-systems model

”path”: The path to the python module file that contains an implementation of SectorModel

”classname”: The name of the class that implements SectorModel

”inputs”: A list of the inputs that this model requires

”outputs”: A list of the outputs that this model provides

”time_intervals”: A list of time intervals within a year that are represented by the model, each with reference to the model’s internal identifier for timesteps

”regions”: A list of geographical regions used within the model, as objects with both geography and attributes

”initial_conditions”: A list of initial conditions required to set up the modelled system in the base year

”interventions”: A list of possible interventions that could be made in the modelled system

”parameters”: A list of model parameters

Return type:dict

load()

Load and check all config

load_initial_conditions()

Inital conditions are located in yaml files specified in sector model blocks in the sos model config

load_inputs()

Input spec is located in the data/<sectormodel>/inputs.yaml file

load_interventions()

Interventions are located in yaml files specified in sector model blocks in the sos model config

load_io_metadata(inputs_or_outputs)

Load inputs or outputs, allowing missing or empty file

load_outputs()

Output spec is located in data/<sectormodel>/output.yaml file

load_parameters()

Parameter configurations are located in yaml files specified in sector model blocks in the sos model config

smif.data_layer.sos_model_config module

Read and parse the config files for the system-of-systems model

class smif.data_layer.sos_model_config.SosModelReader(config_file_path)

Bases: object

Encapsulates the parsing of the system-of-systems configuration

Parameters:config_file_path (str) – A path to the master config file

data

Expose all model configuration data

Returns:Returns a dictionary with the following keys:

timesteps

the sequence of years

max_iterations

limit iterations for solving interdependencies

sector_model_config: list

The list of sector model configuration data

scenario_data: dict

A dictionary of scenario data, with the parameter name as the key and the data as the value

planning: list

A list of dicts of planning instructions

region_sets: dict

A dictionary of region set data, with the name as the key and the data as the value

interval_sets: dict

A dictionary of interval set data, with the name as the key and the data as the value

scenario_metadata: list of dicts

The spatial and temporal resolutions and units of scenario data

Return type:dict

load()

Load and check all config

load_convergence_absolute_tolerance()

Parse convergence_absolute_tolerance setting

load_convergence_max_iterations()

Parse convergence_max_iterations setting

load_convergence_relative_tolerance()

Parse convergence_relative_tolerance setting

load_dependencies()

Parse dependencies

load_planning()

Loads the set of build instructions for planning

Returns:A list of planning instructions loaded from the planning file Return type:list

load_regions()

Model regions are specified in data/<sectormodel>/regions.*

The file format must be possible to parse with GDAL, and must contain an attribute “name” to use as an identifier for the region.

load_scenario_data()

Load scenario data from list in sos model config

Working assumptions:

• scenario data is list of dicts, each like:

  {
      'parameter': 'parameter_name',
      'file': 'relative file path',
      'spatial_resolution': 'national',
      'temporal_resolution': 'annual',
      'units': 'kg'
  }
  

• data in file is list of dicts, each like:

  {
      'value': 100,
      # optional, depending on parameter type:
      'region': 'UK',
      'year': 2015
  }
  

Returns:A dictionary where keys are parameters names and values are the file contents, so a list of dicts Return type:dict

load_sector_model_data()

Parse list of sector models to run

Model details include: - model name - model config directory - SectorModel class name to call

load_sos_config()

Parse model master config

• configures run mode
• sets max iterations for solving interdependencies
• points to timesteps file
• points to shared data files
• points to sector models and sector model data files

load_time_intervals()

Within-year time intervals are specified in data/<sectormodel>/time_intervals.yaml

These specify the mapping of model timesteps to durations within a year (assume modelling 365 days: no extra day in leap years, no leap seconds)

Each time interval must have - start (period since beginning of year) - end (period since beginning of year) - id (label to use when passing between integration layer and sector model)

use ISO 8601[1]_ duration format to specify periods:

P[n]Y[n]M[n]DT[n]H[n]M[n]S

For example:

P1Y == 1 year
P3M == 3 months
PT168H == 168 hours

So to specify a period from the beginning of March to the end of May:

start: P2M
end: P5M
id: spring

References

[1]	https://en.wikipedia.org/wiki/ISO_8601#Durations

load_timesteps()

Parse model timesteps

smif.data_layer.validate module

Validate the correct format and presence of the config data for the system-of-systems model

exception smif.data_layer.validate.ValidationError

Bases: Exception

Custom exception to use for parsing validation.

smif.data_layer.validate.validate_dependency(dep)

Check a dependency specification

smif.data_layer.validate.validate_dependency_spec(input_spec, model_name)

Check the input specification for a single sector model

smif.data_layer.validate.validate_initial_condition(datum, file_path)

Check a single initial condition datum

smif.data_layer.validate.validate_initial_conditions(data, file_path)

Check a list of initial condition observations

smif.data_layer.validate.validate_interval_sets_config(interval_sets)

Check interval sets

smif.data_layer.validate.validate_interventions(data, path)

Validate the loaded data as required for model interventions

smif.data_layer.validate.validate_path_to_timesteps(timesteps)

Check timesteps is a path to timesteps file

smif.data_layer.validate.validate_planning_config(planning)

Check planning options

smif.data_layer.validate.validate_region_sets_config(region_sets)

Check regions sets

smif.data_layer.validate.validate_scenario(scenario)

Check a single scenario specification

smif.data_layer.validate.validate_scenario_data(data, file_path)

Check a list of scenario observations

smif.data_layer.validate.validate_scenario_data_config(scenario_data)

Check scenario data

smif.data_layer.validate.validate_scenario_datum(datum, file_path)

Check a single scenario datum

smif.data_layer.validate.validate_sector_model_initial_config(sector_model_config)

Check a single sector model initial configuration

smif.data_layer.validate.validate_sector_models_initial_config(sector_models)

Check list of sector models initial configuration

smif.data_layer.validate.validate_sos_model_config(data)

Check expected values for data loaded from master config file

smif.data_layer.validate.validate_time_interval(interval)

Check a single time interval

smif.data_layer.validate.validate_time_intervals(intervals, file_path)

Check time intervals

smif.data_layer.validate.validate_timesteps(timesteps, file_path)

Check timesteps is a list of integers

Module contents

Data access modules for loading system-of-systems model configuration

class smif.data_layer.DataInterface

Bases: object

read_interval_definition_data(interval_definition_name)

read_interval_definitions()

read_narrative_data(narrative_name)

read_narrative_set(narrative_set_name)

read_narrative_sets()

read_region_definition_data(region_definition_name)

read_region_definitions()

read_scenario_data(scenario_name)

read_scenario_set(scenario_set_name)

read_scenario_sets()

read_sector_model(sector_model_name)

read_sector_models()

read_sos_model_runs()

read_sos_models()

update_interval_definition(interval_definition)

update_narrative(narrative)

update_narrative_set(narrative_set)

update_region_definition(region_definition)

update_scenario(scenario)

update_scenario_set(scenario_set)

update_sector_model(sector_model_name, sector_model)

update_sos_model(sos_model_name, sos_model)

update_sos_model_run(sos_model_run_name, sos_model_run)

write_interval_definition(interval_definition)

write_narrative(narrative)

write_narrative_set(narrative_set)

write_region_definition(region_definition)

write_scenario(scenario)

write_scenario_set(scenario_set)

write_sector_model(sector_model)

write_sos_model(sos_model)

write_sos_model_run(sos_model_run)

class smif.data_layer.DatabaseInterface(config_path)

Bases: smif.data_layer.DataInterface

Read and write interface to Database

read_interval_data(interval_name)

read_intervals()

read_narrative_data(narrative_name)

read_narrative_set(narrative_set_name)

read_narrative_sets()

read_region_data(region_name)

read_regions()

read_scenario_data(scenario_name)

read_scenario_set(scenario_set_name)

read_scenario_sets()

read_sector_model(sector_model_name)

read_sector_models()

read_sos_model_runs()

read_sos_models()

read_units()

write_interval(interval)

write_narrative(narrative)

write_narrative_set(narrative_set)

write_region(region)

write_scenario(scenario)

write_scenario_set(scenario_set)

write_sector_model(sector_model)

write_sos_model(sos_model)

write_sos_model_run(sos_model_run)

write_unit(unit)

class smif.data_layer.DatafileInterface(base_folder)

Bases: smif.data_layer.DataInterface

Read and write interface to YAML / CSV configuration files

Project.yml

Parameters:base_folder (str) – The path to the configuration and data files

read_initial_conditions(filename)

Read the initial conditions from filename

Parameters:filename (str) – The name of the initial conditions yml file to read in

read_interval_definition_data(interval_definition_name)

Parameters:interval_definition_name (str) – Returns:Interval definition data Return type:dict

Notes

Expects csv file to contain headings of year, start, end

read_interval_definitions()

Read interval_definition sets from project configuration

Returns:A list of interval_definition set dicts Return type:list

read_interventions(filename)

Read the interventions from filename

Parameters:filename (str) – The name of the intervention yml file to read in

read_narrative_data(narrative_name)

Read narrative data file

Parameters:narrative_name (str) – Name of the narrative Returns:A list with dictionaries containing the contents of ‘narrative_name’ data file Return type:list

read_narrative_definition(narrative_name)

Read the narrative definition

Parameters:narrative_name (str) – Name of the narrative Returns: Return type:dict

read_narrative_set(narrative_set_name)

Read all narratives from a certain narrative_set

Parameters:narrative_set_name (str) – Name of the narrative_set Returns:A list of narratives within the specified ‘narrative_set_name’ Return type:list

read_narrative_sets()

Read narrative sets from project configuration

Returns:A list of narrative set dicts Return type:list

read_region_definition_data(region_definition_name)

Read region_definition data file into a Fiona feature collection

The file format must be possible to parse with GDAL, and must contain an attribute “name” to use as an identifier for the region_definition.

Parameters:region_definition_name (str) – Name of the region_definition Returns:A list of data from the specified file in a fiona formatted dict Return type:list

read_region_definitions()

Read region_definitions from project configuration

Returns:A list of region_definition dicts Return type:list

read_scenario_data(scenario_name)

Read scenario data file

Parameters:scenario_name (str) – Name of the scenario Returns:A dict of lists of dicts containing the contents of scenario_name data file(s) associated with the scenario parameters. The keys of the dict are the parameter names Return type:dict

read_scenario_definition(scenario_name)

Read scenario definition data

Parameters:scenario_name (str) – Name of the scenario Returns:The scenario definition Return type:dict

read_scenario_set(scenario_set_name)

Read all scenarios from a certain scenario_set

Parameters:scenario_set_name (str) – Name of the scenario_set Returns:A list of scenarios within the specified ‘scenario_set_name’ Return type:list

read_scenario_sets()

Read scenario sets from project configuration

Returns:A list of scenario set dicts Return type:list

read_sector_model(sector_model_name)

Read a sector model from a Yaml file

Raises an exception when the file does not exists

Parameters:sector_model_name (str) – Name of the sector_model (sector_model[‘name’])

read_sector_models()

Read all sector models from Yaml files

Returns:A list of sector_model dicts Return type:list

read_sos_model(sos_model_name)

Read a specific system-of-system model

Returns:A sos model configuration dictionary Return type:dict

read_sos_model_runs()

Read all system-of-system model runs from Yaml files

sos_model_runs.yml

Returns:A list of sos_model_run dicts Return type:list

read_sos_models()

Read all system-of-system models from Yaml files

Returns:A list of sos_models dicts Return type:list

update_interval_definition(interval_definition_name, interval_definition)

Update interval_definition to project configuration

Parameters:

• interval_definition_name (str) – Name of the (original) entry
• interval_definition (dict) – The updated interval_definition dict

update_narrative(narrative_name, narrative)

Update narrative to project configuration

Parameters:

• narrative_name (str) – Name of the (original) entry
• narrative (dict) – The updated narrative dict

update_narrative_set(narrative_set_name, narrative_set)

Update narrative_set to project configuration

Parameters:

• narrative_set_name (str) – Name of the (original) entry
• narrative_set (dict) – The updated narrative_set dict

update_region_definition(region_definition_name, region_definition)

Update region_definition to project configuration

Parameters:

• region_definition_name (str) – Name of the (original) entry
• region_definition (dict) – The updated region_definition dict

update_scenario(scenario_name, scenario)

Update scenario to project configuration

Parameters:

• scenario_name (str) – Name of the (original) entry
• scenario (dict) – The updated scenario dict

update_scenario_set(scenario_set_name, scenario_set)

Update scenario_set to project configuration

Parameters:

• scenario_set_name (str) – Name of the (original) entry
• scenario_set (dict) – The updated scenario_set dict

update_sector_model(sector_model_name, sector_model)

Update sector model in Yaml file

Parameters:

• sector_model_name (str) – A sector_model name
• sector_model (dict) – A sector_model dictionary

update_sos_model(sos_model_name, sos_model)

Update system-of-system model in Yaml file

Parameters:

• sos_model_name (str) – A sos_model name
• sos_model (dict) – A sos_model dictionary

update_sos_model_run(sos_model_run_name, sos_model_run)

Update system-of-system model run in Yaml file

Parameters:

• sos_model_run_name (str) – A sos_model_run name
• sos_model_run (dict) – A sos_model_run dictionary

write_interval_definition(interval_definition)

Write interval_definition to project configuration

Parameters:interval_definition (dict) – A interval_definition dict

write_narrative(narrative)

Write narrative to project configuration

Parameters:narrative (dict) – A narrative dict

write_narrative_set(narrative_set)

Write narrative_set to project configuration

Parameters:narrative_set (dict) – A narrative_set dict

write_region_definition(region_definition)

Write region_definition to project configuration

Parameters:region_definition (dict) – A region_definition dict

write_scenario(scenario)

Write scenario to project configuration

Parameters:scenario (dict) – A scenario dict

write_scenario_set(scenario_set)

Write scenario_set to project configuration

Parameters:scenario_set (dict) – A scenario_set dict

write_sector_model(sector_model)

Write sector model to a Yaml file

Existing configuration will be overwritten without warning

Parameters:sector_model (dict) – A sector_model dictionary

write_sos_model(sos_model)

Write system-of-system model to Yaml file

Existing configuration will be overwritten without warning

Parameters:sos_model (dict) – A sos_model dictionary

write_sos_model_run(sos_model_run)

Write system-of-system model run to Yaml file

Parameters:sos_model_run (dict) – A sos_model_run dictionary