smif package

Submodules

smif.decision module

The decision module handles the three planning levels

Currently, only pre-specified planning is implemented.

class smif.decision.Planning(planned_interventions=None)[source]

Bases: object

Holds the list of planned interventions, where a single planned intervention is an intervention with a build date after which it will be included in the modelled systems.

For example, a small pumping station might be built in Oxford in 2045:

{
    'name': 'small_pumping_station',
    'build_date': 2045
}
planned_interventions

list

A list of pre-specified planned interventions

names

Returns the set of assets defined in the planned interventions

timeperiods

Returns the set of build dates defined in the planned interventions

smif.intervention module

This module handles the collection of interventions and assets in a sector. The set of interventions describes the targets of possible physical (or non-physical) decisions which can be made in the sector.

An Asset is the general term for an existing component of an infrastructure system.

The difference between an Intervention and an Asset, is that the latter exists (it has been “built”), whereas the former describes the potential to build an Asset.

The set of assets defines the ‘state’ of the infrastructure system.

Notes

This module implements:

  • initialisation of the set of assets from model config (either a collection of yaml text files, or a database)
    • hold generic list of key/values
    • creation of new assets by decision logic (rule-based/optimisation solver)
    • maintain or derive set of possible assets
    • makes the distinction between known-ahead values and build-time values. Location and date are specified at build time, while cost and capacity are a function of time and location.
  • serialisation for passing to models
    • ease of access to full generic data structure
  • output list of assets for reporting
    • write out with legible or traceable keys and units for verification and understanding

Terminology

name:
A category of infrastructure intervention (e.g. power station, policy) which holds default attribute/value pairs. These names can be inherited by asset/intervention definitions to reduce the degree of duplicate data entry.
asset:
An instance of an intervention, which represents a single investment decisions which will take place, or has taken place. Historical interventions are defined as initial conditions, while future interventions are listed as pre-specified planning. Both historical and future interventions can make use of names to ease data entry. Assets must have location, build_date and name attributes defined.
intervention:
A potential asset or investment. Interventions are defined in the same way as for assets, cannot have a build_date defined.
class smif.intervention.Asset(name='', data=None, sector='')[source]

Bases: smif.intervention.Intervention

An instance of an intervention with a build date.

Used to represent pre-specified planning and existing infrastructure assets and interventions

Parameters:
  • name (str, default="") – The type of asset, which should be unique across all sectors
  • data (dict, default=None) – The dictionary of asset attributes
  • sector (str, default="") – The sector associated with the asset
build_date

The build date of this asset instance (if specified - asset types will not have build dates)

get_attributes()[source]

Ensures location is present and no build date is specified

class smif.intervention.AssetRegister[source]

Bases: smif.intervention.Register

Register each asset type

register(asset)[source]
class smif.intervention.Intervention(name='', data=None, sector='')[source]

Bases: smif.intervention.InterventionContainer

An potential investment to send to the logic-layer

An Intervention, is an investment which has a name (or name), other attributes (such as capital cost and economic lifetime), and location, but no build date.

The set of interventions are defined within each sector, and these are collected into an InterventionRegister when a smif.controller.SosModel is instantiated by the controller at runtime.

Parameters:
  • name (str, default="") – The type of asset, which should be unique across all sectors
  • data (dict, default=None) – The dictionary of asset attributes
  • sector (str, default="") – The sector associated with the asset
get_attributes()[source]

Ensures location is present and no build date is specified

location

The location of this asset instance (if specified - asset types may not have explicit locations)

class smif.intervention.InterventionContainer(name='', data=None, sector='')[source]

Bases: object

An container for asset types, interventions and assets.

An asset’s data is set up to be a flexible, plain data structure.

Parameters:
  • name (str, default="") – The type of asset, which should be unique across all sectors
  • data (dict, default=None) – The dictionary of asset attributes
  • sector (str, default="") – The sector associated with the asset
static deterministic_dict_to_str(data)[source]

Return a reproducible string representation of any dict

Parameters:data (dict) – An intervention attributes dictionary
Returns:A reproducible string representation
Return type:str
get_attributes()[source]

Override to return two lists, one containing required attributes, the other containing omitted attributes

Returns:Tuple of lists, one contained required attributes, the other which must be omitted
Return type:tuple
sector

The name of the sector model this asset is used in.

sha1sum()[source]

Compute the SHA1 hash of this asset’s data

Returns:
Return type:str
class smif.intervention.InterventionRegister[source]

Bases: smif.intervention.Register

The collection of Intervention objects

An InterventionRegister contains an immutable collection of sector specific assets and decision points which can be decided on by the Logic Layer

  • Reads in a collection of interventions defined in each sector model
  • Builds an ordered and immutable collection of interventions
  • Provides interfaces to
    • optimisation/rule-based planning logic
    • SectorModel class model wrappers

Key functions:

  • outputs a complete list of asset build possibilities (asset type at location) which are (potentially) constrained by the pre-specified planning instructions and existing infrastructure.
  • translate a binary vector of build instructions (e.g. from optimisation routine) into Asset objects with human-readable key-value pairs
  • translates an immutable collection of Asset objects into a binary vector to pass to the logic-layer.

Notes

Internal data structures

Intervention_types
is a 2D array of integers: each entry is an array representing an Intervention type, each integer indexes attribute_possible_values
attribute_keys
is a 1D array of strings
attribute_possible_values
is a 2D array of simple values, possibly (boolean, integer, float, string, tuple). Each entry is a list of possible values for the attribute at that index.

Invariants

  • there must be one name and one list of possible values per attribute
  • each Intervention type must list one value for each attribute, and that value must be a valid index into the possible_values array
  • each possible_values array should be all of a single type
__iter__()[source]

Iterate over the list of asset types held in the register

__len__()[source]

Returns the number of asset types stored in the register

attribute_index(key)[source]

Get the index of an attribute name

attribute_value_index(attr_idx, value)[source]

Get the index of a possible value for a given attribute index

get_intervention(name)[source]

Returns the named asset data

numeric_to_intervention(numeric_asset)[source]

Convert the numeric representation of an asset back to Asset (with legible key/value data)

Parameters:numeric_asset (list) – A list of integers of length self._attribute_keys
Returns:An Intervention object
Return type:Intervention

Examples

Given a (very minimal) possible state of a register:

>>> register = AssetRegister()
>>> register._names = [[1,1,1]]
>>> register._attribute_keys = ["name", "capacity", "sector"]
>>> register._attribute_possible_values = [
...     [None, "water_treatment_plant"],
...     [None, {"value": 5, "units": "ML/day"}],
...     [None, "water_supply"]
... ]

Calling this function would piece together the asset:

>>> asset = register.numeric_to_asset([1,1,1])
>>> print(asset)
Asset("water_treatment_plant", {"name": "water_treatment_plant",
"capacity": {"units": "ML/day", "value": 5}, "sector": "water_supply"})
register(intervention)[source]

Add a new intervention to the register

Parameters:intervention (Intervention) –
class smif.intervention.Register[source]

Bases: object

Holds interventions, pre-spec’d planning instructions & existing assets

  • register each asset type/intervention name
  • translate a set of assets representing an initial system into numeric representation
register(asset)[source]

Adds a new asset to the collection

smif.optimisation module

Solve the optimal planning problem for a system of systems model

smif.optimisation.define_basic_model(assets, availability_constraint, asset_costs)[source]

Define the binary integer planning problem

Parameters:
  • assets (list) – The list of assets
  • availability_constraint (dict) – A dictionary of binary constraints on whether you can build each asset in assets
  • asset_costs (dict) – The investment cost of each asset
  • asset_value (dict) – The value function approximation of each asset
Returns:

model – An abstract instance of the incomplete model with no objective function

Return type:

pyomo.environ.ConcreteModel

smif.optimisation.feature_vfa_model(assets, availability_constraint, asset_costs, feature_coefficients, asset_features)[source]

Define the value function approximation

Here we assume that the value function approximation is a function of the features of the state, rather than individual assets, or enumeration of all possible states

Parameters:
  • assets (list) – The list of assets
  • availability_constraint (dict) – A dictionary of binary constraints on whether you can build each asset in assets
  • asset_costs (dict) – The investment cost of each asset
  • features (list) – The set of features
  • feature_coefficients (dict) – The regression coefficients for each feature
  • asset_features (dict) – The mapping of features to assets
Returns:

model – A concrete instance of the model

Return type:

pyomo.environ.ConcreteModel

Notes

The use of features decomposes the optimisation problem into several sub-components. The first is to find the clusters of ‘bits’ in the state, which correctly predict minimal cost investments and operation of infrastructure assets.

These clusters can be used to define a feature, adding an entry to the feature set of features and a column to the asset_features matrix, where 1 indicates that the feature includes that asset.

Initially, feature selection could begin by regressing the results from a random sample of investment policies. This could highlight which patterns of investments seem to result in least cost systems. In addition, the attributes of the assets could provide a set of features which at least help categorise the assets (e.g. according to sector, size, location, and asset type).

The binary integer problem is posed as follows:

\min \sum_i^I c_i x_i + x_i \sum_f^{F}( e_{if} b_{f})

where

  • i is an element in the set of assets I
  • f is an element in the set of features F
  • c_i is the cost of asset i
  • x_i is the decision to invest in asset i
  • e_{if} is the mapping of feature f to asset i
  • b_{f} is the basis coefficient of asset i and feature f
smif.optimisation.formulate_model(asset_register, availability_constraint, feature_coefficients, asset_features)[source]
smif.optimisation.linear_vfa_model(assets, availability_constraint, asset_costs, asset_value)[source]

Define the value function approximation

Here we assume a linear relationship (in the asset)

Parameters:
  • assets (list) – The list of assets
  • availability_constraint (dict) – A dictionary of binary constraints on whether you can build each asset in assets
  • asset_costs (dict) – The investment cost of each asset
  • asset_value (dict) – The value function approximation of each asset
Returns:

model – A concrete instance of the model

Return type:

pyomo.environ.ConcreteModel

smif.optimisation.solve_model(model, state=None)[source]

Solves the model using glpk

Passing in a state as a list of asset names initialises the state of the model, fixing those decision variables.

Parameters:
  • model (pyomo.environ.ConcreteModel) – A concrete instance of the model
  • state (dict, optional, default=None) – A list of assets which were installed in a previous iteration or time period
  • returns (pyomo.core.Model.Instance) – An instance of the model populated with results
smif.optimisation.state_vfa_model(assets, availability_constraint, asset_costs, asset_value, states)[source]

Define the value function approximation

Here we assume that the value function approximation is a function of the state, rather than individual assets

Unfortunately, the number of states becomes very large, growing exponentially in the number of assets, and so representing the approximate value function like this is very inefficient as soon as the number of assets increases above 32 (about 4 GB).

Parameters:
  • assets (list) – The list of assets
  • availability_constraint (dict) – A dictionary of binary constraints on whether you can build each asset in assets
  • asset_costs (dict) – The investment cost of each asset
  • asset_value (dict) – The value function approximation of each asset
  • states (dict) – A dictionary where the keys are tuples of entries in assets and an index of states (2 ** len(assets)) and the value is a binary indicator showing the possible combinations
Returns:

model – A concrete instance of the model

Return type:

pyomo.environ.ConcreteModel

Notes

\hat{v}_t^n = \min_{a_t \in A_t^n} C_t^{INV}(S^n_t, a^n_t) +     {V}^n_t(S^n_t, a^n_t)

smif.parameters module

Encapsulates the input or output parameters of a sector model, for example:

- name: petrol_price
  spatial_resolution: GB
  temporal_resolution: annual
- name: diesel_price
  spatial_resolution: GB
  temporal_resolution: annual
- name: LPG_price
  spatial_resolution: GB
  temporal_resolution: annual
class smif.parameters.ModelParameters(parameters)[source]

Bases: object

A container for all the model inputs

Parameters:inputs (list) – A list of dicts of model parameter name, spatial resolution and temporal resolution
get_spatial_res(name)[source]

The spatial resolution for parameter name

Parameters:name (str) – The name of a model parameter
get_temporal_res(name)[source]

The temporal resolution for parameter name

Parameters:name (str) – The name of a model parameter
names
parameters

A list of the model parameters

Returns:
Return type:smif.parameters.ParameterList
spatial_resolutions

A list of the spatial resolutions

Returns:A list of the spatial resolutions associated with the model parameters
Return type:list
temporal_resolutions

A list of the temporal resolutions

Returns:A list of the temporal resolutions associated with the model parameters
Return type:list
class smif.parameters.Parameter(name, spatial_resolution, temporal_resolution)

Bases: tuple

__getnewargs__()

Return self as a plain tuple. Used by copy and pickle.

static __new__(_cls, name, spatial_resolution, temporal_resolution)

Create new instance of Parameter(name, spatial_resolution, temporal_resolution)

__repr__()

Return a nicely formatted representation string

name

Alias for field number 0

spatial_resolution

Alias for field number 1

temporal_resolution

Alias for field number 2

smif.sector_model module

This module acts as a bridge to the sector models from the controller

The SectorModel exposes several key methods for running wrapped sector models. To add a sector model to an instance of the framework, first implement SectorModel.

Key Functions

This class performs several key functions which ease the integration of sector models into the system-of-systems framework.

The user must implement the various abstract functions throughout the class to provide an interface to the sector model, which can be called upon by the framework. From the model’s perspective, SectorModel provides a bridge from the sector-specific problem representation to the general representation which allows reasoning across infrastructure systems.

The key functions include

  • converting input/outputs to/from geographies/temporal resolutions
  • converting control vectors from the decision layer of the framework, to asset Interventions specific to the sector model
  • returning scaler/vector values to the framework to enable measurements of performance, particularly for the purposes of optimisation and rule-based approaches
class smif.sector_model.SectorModel[source]

Bases: abc.ABC

A representation of the sector model with inputs and outputs

extract_obj(results)[source]

Implement this method to return a scalar value objective function

This method should take the results from the output of the simulate() method, process the results, and return a scalar value which can be used as a component of the objective function by the decision layer

Parameters:results (dict) – A nested dict of the results from the simulate() method
Returns:A scalar component generated from the simulation model results
Return type:float
initialise(initial_conditions)[source]

Implement this method to set up the model system

Parameters:initial_conditions (list) – A list of past Interventions, with build dates and locations as necessary to specify the infrastructure system to be modelled.
inputs

The inputs to the model

The inputs should be specified in a list. For example:

- name: eletricity_price
  spatial_resolution: GB
  temporal_resolution: annual
Parameters:value (list) – A list of dicts of inputs to the model. These includes parameters, assets and exogenous data
Returns:
Return type:smif.parameters.ModelInputs
intervention_names

The names of the interventions

Returns:A list of the names of the interventions
Return type:list
name

The name of the sector model

Returns:The name of the sector model
Return type:str
outputs

The outputs from the model

Parameters:value (list) – A list of dicts of outputs from the model. This may include results and metrics
Returns:
Return type:smif.parameters.ModelParameters
simulate(decisions, state, data)[source]

Implement this method to run the model

Parameters:
  • decisions (list) – A list of :py:class:Intervention to apply to the modelled system
  • state (list) – A list of :py:class:StateData to update the state of the modelled system
  • data (dict) – A dictionary of the format: data[parameter] = [SpaceTimeValue(region, interval, value, units), ...]
Returns:

A dictionary of the format: results[parameter] = [SpaceTimeValue(region, interval, value, units), ...]

Return type:

dict

Notes

In the results returned from the simulate() method:

interval
should reference an id from the interval set corresponding to the output parameter, as specified in model configuration
region
should reference a region name from the region set corresponding to the output parameter, as specified in model configuration
validate()[source]

Validate that this SectorModel has been set up with sufficient data to run

class smif.sector_model.SectorModelBuilder(name)[source]

Bases: object

Build the components that make up a sectormodel from the configuration

Parameters:name (str) – The name of the sector model
add_inputs(input_dict)[source]

Add inputs to the sector model

add_interventions(intervention_list)[source]

Add interventions to the sector model

Parameters:intervention_list (list) – A list of dicts of interventions
add_outputs(output_dict)[source]

Add outputs to the sector model

create_initial_system(initial_conditions)[source]

Set up model with initial system

finish()[source]

Validate and return the sector model

load_model(model_path, classname)[source]

Dynamically load model module

validate()[source]

Check and/or assert that the sector model is correctly set up - should raise errors if invalid

smif.sos_model module

This module coordinates the software components that make up the integration framework.

class smif.sos_model.ModelSet(models, sos_model)[source]

Bases: object

Wraps a set of interdependent models

Given a directed graph of dependencies between models, any cyclic dependencies are contained within the strongly-connected components of the graph.

A ModelSet corresponds to the set of models within a single strongly- connected component. If this is a set of one model, it can simply be run deterministically. Otherwise, this class provides the machinery necessary to find a solution to each of the interdependent models.

The current implementation first estimates the outputs for each model in the set, guaranteeing that each model will then be able to run, then begins iterating, running every model in the set at each iteration, monitoring the model outputs over the iterations, and stopping at timeout, divergence or convergence.

Notes

This calls back into SosModel quite extensively for state, data, decisions, regions and intervals.

converged(timestep)[source]

Check whether the results of a set of models have converged.

Returns:converged – True if the results have converged to within a tolerance
Return type:bool
Raises:DiverganceError – If the results appear to be diverging
guess_results(model, timestep)[source]

Dependency-free guess at a model’s result set.

Initially, guess zeroes, or the previous timestep’s results.

run(timestep)[source]

Runs a set of one or more models

class smif.sos_model.RunMode[source]

Bases: enum.Enum

Enumerates the operating modes of a SoS model

dynamic_optimisation = <RunMode.dynamic_optimisation: 3>
sequential_simulation = <RunMode.sequential_simulation: 1>
static_optimisation = <RunMode.static_optimisation: 2>
static_simulation = <RunMode.static_simulation: 0>
class smif.sos_model.SosModel[source]

Bases: object

Consists of the collection of timesteps and sector models

This is NISMOD - i.e. the system of system model which brings all of the sector models together. Sector models may be joined through dependencies.

This class is populated at runtime by the SosModelBuilder and called from smif.cli.run_model().

models

dict

This is a dictionary of smif.SectorModel

initial_conditions

list

List of interventions required to set up the initial system, with any state attributes provided here too

static add_data_series(list_a, list_b)[source]

Given two lists of SpaceTimeValues of identical spatial and temporal resolution, return a single list with matching values added together.

Notes

Assumes a data series is not sparse, i.e. has a value for every region/interval combination

determine_running_mode()[source]

Determines from the config in what mode to run the model

Returns:The mode in which to run the model
Return type:RunMode
get_data(model, timestep)[source]

Gets the data in the required format to pass to the simulate method

Returns:A nested dictionary of the format: data[parameter][region][time_interval] = {value, units}
Return type:dict

Notes

Note that the timestep is not passed to the SectorModel in the nested data dictionary. The current timestep is available in data['timestep'].

get_decisions(model, timestep)[source]

Gets the interventions that correspond to the decisions

Parameters:
get_state(model, timestep)[source]

Gets the state to pass to SectorModel.simulate

inputs

A dictionary of model names associated with an inputs

Returns:Keys are parameter names, value is a list of sector model names
Return type:dict
intervention_names

Names (id-like keys) of all known asset type

outputs

Model names associated with model outputs & scenarios

Returns:Keys are parameter names, value is a list of sector model names
Return type:dict
resolution_mapping

Returns the temporal and spatial mapping to an input, output or scenario parameter

Example

The data structure follows source->parameter->{temporal, spatial}:

{
    'scenario': {
        'raininess': {
            'temporal_resolution': 'annual',
            'spatial_resolution': 'LSOA'
        }
    }
}
results

Get nested dict of model results

Returns:Nested dictionary in the format results[int:year][str:model][str:parameter] => list of SpaceTimeValues
Return type:dict
run()[source]

Runs the system-of-system model

  1. Determine run mode
  2. Determine running order
  3. Run each sector model
  4. Return success or failure
run_sector_model(model_name)[source]

Runs the sector model

Parameters:model_name (str) – The name of the model, corresponding to the folder name in the models subfolder of the project folder
run_sector_model_timestep(model, timestep)[source]

Run the sector model for a specific timestep

Parameters:
scenario_data

Get nested dict of scenario data

Returns:Nested dictionary in the format data[year][param] = SpaceTimeValue(region, interval, value, unit)
Return type:dict
sector_models

The list of sector model names

Returns:A list of sector model names
Return type:list
set_data(model, timestep, results)[source]

Sets results output from model as data available to other/future models

Stores only latest estimated results (i.e. not holding on to iterations here while trying to solve interdependencies)

set_state(model, from_timestep, state)[source]

Sets state output from model ready for next timestep

timestep_after(timestep)[source]

Returns the timestep after a given timestep, or None

timestep_before(timestep)[source]

Returns the timestep previous to a given timestep, or None

timesteps

Returns the list of timesteps

Returns:A list of timesteps, distinct and sorted in ascending order
Return type:list
class smif.sos_model.SosModelBuilder[source]

Bases: object

Constructs a system-of-systems model

Builds a SosModel.

Examples

Call SosModelBuilder.construct() to populate a SosModel object and SosModelBuilder.finish() to return the validated and dependency-checked system-of-systems model.

>>> builder = SosModelBuilder()
>>> builder.construct(config_data)
>>> sos_model = builder.finish()
add_initial_conditions(model_name, initial_conditions)[source]

Adds initial conditions (state) for a model

add_interventions(model_name, interventions)[source]

Adds interventions for a model

add_model(model)[source]

Adds a sector model to the system-of-systems model

Parameters:model (smif.sector_model.SectorModel) – A sector model wrapper
add_model_data(model, model_data)[source]

Adds sector model data to the system-of-systems model which is convenient to have available at the higher level.

add_planning(planning)[source]

Loads the planning logic into the system of systems model

Pre-specified planning interventions are defined at the sector-model level, read in through the SectorModel class, but populate the intervention register in the controller.

Parameters:planning (list) – A list of planning instructions
add_resolution_mapping(resolution_mapping)[source]
Parameters:resolution_mapping (dict) – A dictionary containing information on the spatial and temporal resoultion of scenario data

Example

The data structure follows source->parameter->{temporal, spatial}:

{'scenario': {
 'raininess': {'temporal_resolution': 'annual',
               'spatial_resolution': 'LSOA'}}}
add_scenario_data(data)[source]

Load the scenario data into the system of systems model

Expect a dictionary, where each key maps a parameter name to a list of data, each observation with:

  • timestep
  • value
  • units
  • region (must use a region id from scenario regions)
  • interval (must use an id from scenario time intervals)

Add a dictionary of list of smif.SpaceTimeValue named tuples, for ease of iteration:

data[year][param] = SpaceTimeValue(region, interval, value, units)

Default region: “national” Default interval: “annual”

add_timesteps(timesteps)[source]

Set the timesteps of the system-of-systems model

Parameters:timesteps (list) – A list of timesteps
construct(config_data)[source]

Set up the whole SosModel

Parameters:config_data (dict) – A valid system-of-systems model configuration dictionary
finish()[source]

Returns a configured system-of-systems model ready for operation

Includes validation steps, e.g. to check dependencies

static intervention_state_from_data(intervention_data)[source]

Unpack an intervention from the initial system to extract StateData

load_interval_sets(interval_sets)[source]

Loads the time-interval sets into the system-of-system model

Parameters:interval_sets (list) – A dict, where key is the name of the interval set, and the value the data
load_models(model_data_list)[source]

Loads the sector models into the system-of-systems model

Parameters:
  • model_data_list (list) – A list of sector model config/data
  • assets (list) – A list of assets to pass to the sector model
load_region_sets(region_sets)[source]

Loads the region sets into the system-of-system model

Parameters:region_sets (list) – A dict, where key is the name of the region set, and the value the data
set_max_iterations(config_data)[source]

Module contents

smif

class smif.SpaceTimeValue(region, interval, value, units)[source]

Bases: object

A piece of model input/output data

Parameters:
  • region (str) – A valid (unique) region name which is registered in the region register
  • interval (str) – A valid (unique) interval name which is registered in the interval register
  • value (float) – The value
  • units (str) – The units associated with the value
class smif.StateData(target, data)[source]

Bases: object

A piece of state data

Parameters:
  • target – The id or name of the object described by this state
  • data – The state attribute/data to apply - could typically be a dict of attributes