smif.data_layer package¶

Submodules¶

smif.data_layer.load module¶

Parse yaml config files, to construct sector models

smif.data_layer.load.dump(data, file_path)[source]¶

Write plain data to a file as yaml

Parameters:	file_path (str) – The path of the configuration file to write data – Data to write (should be lists, dicts and simple values)

smif.data_layer.load.load(file_path)[source]¶

Parse yaml config file into plain data (lists, dicts and simple values)

Parameters:	file_path (str) – The path of the configuration file to parse

smif.data_layer.load.space_time_value_constructor(loader, node)[source]¶: Load ustom yaml representation of SpaceTimeValue

smif.data_layer.load.space_time_value_representer(dumper, data)[source]¶: Dump custom yaml representation of SpaceTimeValue

smif.data_layer.sector_model_config module¶

Read and parse the config for sector models

class smif.data_layer.sector_model_config.SectorModelReader(initial_config=None)[source]¶

Bases: object

Parses the configuration and input data for a sector model

Parameters:

initial_config (dict) –

Sector model details, sufficient to read the full config from a set of files. Must contain the following fields:

“model_name”

The name of the sector model, for reference within the system-of-systems model

“model_path”

The path to the python module file that contains an implementation of SectorModel

“model_classname”

The name of the class that implements SectorModel

“model_config_dir”

The root path of model config/data to use, which must contain inputs.yaml, outputs.yaml, time_intervals.yaml and regions.shp/regions.geojson

“initial_conditions”

List of files containing initial conditions

“interventions”

List of files containing interventions

data¶

Expose all loaded config data

Returns:

data – Model configuration data, with the following fields:

“name”: The name of the sector model, for reference within the system-of-systems model
“path”: The path to the python module file that contains an implementation of SectorModel

“classname”: The name of the class that implements SectorModel

“inputs”: A list of the inputs that this model requires

“outputs”: A list of the outputs that this model provides

“time_intervals”: A list of time intervals within a year that are represented by the model, each with reference to the model’s internal identifier for timesteps

“regions”: A list of geographical regions used within the model, as objects with both geography and attributes

“initial_conditions”: A list of initial conditions required to set up the modelled system in the base year

“interventions”: A list of possible interventions that could be made in the modelled system

Return type: dict

load()[source]¶: Load and check all config

load_initial_conditions()[source]¶: Inital conditions are located in yaml files specified in sector model blocks in the sos model config

load_inputs()[source]¶: Input spec is located in the data/<sectormodel>/inputs.yaml file

load_interventions()[source]¶: Interventions are located in yaml files specified in sector model blocks in the sos model config

load_io_metadata(inputs_or_outputs)[source]¶: Load inputs or outputs, allowing missing or empty file

load_outputs()[source]¶: Output spec is located in data/<sectormodel>/output.yaml file

smif.data_layer.sos_model_config module¶

Read and parse the config files for the system-of-systems model

class smif.data_layer.sos_model_config.SosModelReader(config_file_path)[source]¶

Bases: object

Encapsulates the parsing of the system-of-systems configuration

Parameters:	config_file_path (str) – A path to the master config file

data¶

Expose all model configuration data

Returns:

Returns a dictionary with the following keys:

timesteps: the sequence of years
max_iterations: limit iterations for solving interdependencies
sector_model_config: list: The list of sector model configuration data
scenario_data: dict: A dictionary of scenario data, with the parameter name as the key and the data as the value
planning: list: A list of dicts of planning instructions
region_sets: dict: A dictionary of region set data, with the name as the key and the data as the value
interval_sets: dict: A dictionary of interval set data, with the name as the key and the data as the value
resolution_mapping: dict: The mapping of spatial and temporal resolutions to inputs, outputs and scenarios

Return type: dict

load()[source]¶: Load and check all config

load_max_iterations()[source]¶: Parse max_iterations setting

load_planning()[source]¶

Loads the set of build instructions for planning

Returns:	A list of planning instructions loaded from the planning file
Return type:	list

load_regions()[source]¶

Model regions are specified in data/<sectormodel>/regions.*

The file format must be possible to parse with GDAL, and must contain an attribute “name” to use as an identifier for the region.

load_scenario_data()[source]¶

Load scenario data from list in sos model config

Working assumptions:

scenario data is list of dicts, each like:

{
    'parameter': 'parameter_name',
    'file': 'relative file path',
    'spatial_resolution': 'national'
    'temporal_resolution': 'annual'
}

data in file is list of dicts, each like:

{
    'value': 100,
    'units': 'kg',
    # optional, depending on parameter type:
    'region': 'UK',
    'year': 2015
}

Returns:	A dictionary where keys are parameters names and values are the file contents, so a list of dicts
Return type:	dict

load_sector_model_data()[source]¶

Parse list of sector models to run

Model details include: - model name - model config directory - SectorModel class name to call

load_sos_config()[source]¶

Parse model master config

configures run mode
sets max iterations for solving interdependencies
points to timesteps file
points to shared data files
points to sector models and sector model data files

load_time_intervals()[source]¶

Within-year time intervals are specified in data/<sectormodel>/time_intervals.yaml

These specify the mapping of model timesteps to durations within a year (assume modelling 365 days: no extra day in leap years, no leap seconds)

Each time interval must have - start (period since beginning of year) - end (period since beginning of year) - id (label to use when passing between integration layer and sector model)

use ISO 8601[1]_ duration format to specify periods:

P[n]Y[n]M[n]DT[n]H[n]M[n]S

For example:

P1Y == 1 year
P3M == 3 months
PT168H == 168 hours

So to specify a period from the beginning of March to the end of May:

start: P2M
end: P5M
id: spring

References

[1]	https://en.wikipedia.org/wiki/ISO_8601#Durations

load_timesteps()[source]¶: Parse model timesteps

smif.data_layer.validate module¶

Validate the correct format and presence of the config data for the system-of-systems model

exception smif.data_layer.validate.ValidationError[source]¶

Bases: Exception

Custom exception to use for parsing validation.

smif.data_layer.validate.validate_dependency(dep)[source]¶: Check a dependency specification

smif.data_layer.validate.validate_initial_condition(datum, file_path)[source]¶: Check a single initial condition datum

smif.data_layer.validate.validate_initial_conditions(data, file_path)[source]¶: Check a list of initial condition observations

smif.data_layer.validate.validate_input_spec(input_spec, model_name)[source]¶: Check the input specification for a single sector model

smif.data_layer.validate.validate_interval_sets_config(interval_sets)[source]¶: Check interval sets

smif.data_layer.validate.validate_interventions(data, path)[source]¶: Validate the loaded data as required for model interventions

smif.data_layer.validate.validate_output(dep)[source]¶: Check an output specification

smif.data_layer.validate.validate_output_spec(output_spec, model_name)[source]¶: Check the output specification for a single sector model

smif.data_layer.validate.validate_path_to_timesteps(timesteps)[source]¶: Check timesteps is a path to timesteps file

smif.data_layer.validate.validate_planning_config(planning)[source]¶: Check planning options

smif.data_layer.validate.validate_region_sets_config(region_sets)[source]¶: Check regions sets

smif.data_layer.validate.validate_scenario(scenario)[source]¶: Check a single scenario specification

smif.data_layer.validate.validate_scenario_data(data, file_path)[source]¶: Check a list of scenario observations

smif.data_layer.validate.validate_scenario_data_config(scenario_data)[source]¶: Check scenario data

smif.data_layer.validate.validate_scenario_datum(datum, file_path)[source]¶: Check a single scenario datum

smif.data_layer.validate.validate_sector_model_initial_config(sector_model_config)[source]¶: Check a single sector model initial configuration

smif.data_layer.validate.validate_sector_models_initial_config(sector_models)[source]¶: Check list of sector models initial configuration

smif.data_layer.validate.validate_sos_model_config(data)[source]¶: Check expected values for data loaded from master config file

smif.data_layer.validate.validate_time_interval(interval)[source]¶: Check a single time interval

smif.data_layer.validate.validate_time_intervals(intervals, file_path)[source]¶: Check time intervals

smif.data_layer.validate.validate_timesteps(timesteps, file_path)[source]¶: Check timesteps is a list of integers

Module contents¶

Data access modules for loading system-of-systems model configuration