smif.convert package

This module contains implementations of Adaptor to perform conversions between units and between different spatial or temporal dimensions where sufficient metadata is provided in the Spec definition.

These should be useful to link models in simple cases, where it may be reasonable to rely on strong assumptions about the underlying distributions of the variables to be converted.

class smif.convert.Adaptor(name)[source]

Bases: Model

Abstract Adaptor, to convert inputs/outputs between other Models

Override method generate_coefficients, which accepts two Spec definitions.

convert(data_array: ~smif.data_layer.data_array.DataArray, to_spec: ~smif.metadata.spec.Spec, coefficients: <MagicMock id='135493237135872'>)[source]

Convert a dataset between Spec definitions

Parameters:
Return type:

numpy.ndarray

static convert_with_coefficients(data: <MagicMock id='135493237303648'>, coefficients: <MagicMock id='135493237297072'>, axis: int)[source]

Unchecked conversion, given data, coefficients and axis

Parameters:
Return type:

numpy.ndarray

abstractmethod generate_coefficients(from_spec: Spec, to_spec: Spec) → <MagicMock id='135493237308064'>[source]

Generate coefficients for a pair of Spec definitions

Parameters:
Return type:

numpy.ndarray

get_coefficients(data_handle: DataHandle, from_spec: Spec, to_spec: Spec) → <MagicMock id='135493238208944'>[source]

Read coefficients, or generate and save if necessary

Parameters:
Return type:

numpy.ndarray

static get_convert_dims(from_spec, to_spec)[source]

Get dims for conversion from a pair of Spec, assuming only a single dimension will be converted.

Parameters:
Return type:

tuple(str)

simulate(data_handle: DataHandle)[source]

Convert from input to output based on matching variable names

class smif.convert.IntervalAdaptor(name)[source]

Bases: Adaptor

Convert intervals, assuming uniform distributions where necessary

generate_coefficients(from_spec, to_spec) → <MagicMock id='135493237403968'>[source]

Generate conversion coefficients for interval dimensions

Assumes that the Coordinates elements contain an ‘interval’ key whose value corresponds to Interval data, that is a {‘name’: interval_id, ‘interval’: list of interval extents}.

For example, intervals covering each hour of a period

{ 'name': 'first_hour', 'interval': [('PT0H', 'PT1H')] }
{ 'name': ''second_hour', 'interval': [('PT1H', 'PT2H')] }
...

Or intervals corresponding to repeating hours for each day of a period

{
    'name': midnight',
    'interval': [
        ('PT0H', 'PT1H'), ('PT24H', 'PT25H'), ('PT48H', 'PT49H'), ...
    ]
},
{
    'name': ''one_am',
    'interval': [
        ('PT1H', 'PT2H'), ('PT25H', 'PT26H'), ('PT49H', 'PT50H'), ...
    ]
}
...
class smif.convert.RegionAdaptor(name)[source]

Bases: Adaptor

Convert regions, assuming uniform distributions where necessary

generate_coefficients(from_spec, to_spec)[source]

Generate conversion coefficients for spatial dimensions

Assumes that the Coordinates elements contain a ‘feature’ key whose value corresponds to a GDAL vector feature represented as a dict, for example as returned by a fiona reader.

class smif.convert.UnitAdaptor(name)[source]

Bases: Adaptor

Scalar conversion of units

convert(data_array, to_spec, coefficients)[source]

Convert a dataset between Spec definitions

Parameters:
Return type:

numpy.ndarray

generate_coefficients(from_spec, to_spec)[source]

Generate coefficients for a pair of Spec definitions

Parameters:
Return type:

numpy.ndarray

get_coefficients(data_handle, from_spec, to_spec)[source]

Read coefficients, or generate and save if necessary

Parameters:
Return type:

numpy.ndarray

parse_unit(unit_string)[source]

Parse a unit string (abbreviation or full) into a Unit object

Parameters:

unit (str)

Returns:

quantity

Return type:

pint.Unit

simulate(data_handle: DataHandle)[source]

Register unit definitions in registry for model run

Submodules

smif.convert.adaptor module

Adaptor is a subclass of Model, to be used for converting data between units or dimensions.

The method to override is generate_coefficients, which accepts two Spec definitions.

class smif.convert.adaptor.Adaptor(name)[source]

Bases: Model

Abstract Adaptor, to convert inputs/outputs between other Models

Override method generate_coefficients, which accepts two Spec definitions.

convert(data_array: ~smif.data_layer.data_array.DataArray, to_spec: ~smif.metadata.spec.Spec, coefficients: <MagicMock id='135493237135872'>)[source]

Convert a dataset between Spec definitions

Parameters:
Return type:

numpy.ndarray

static convert_with_coefficients(data: <MagicMock id='135493237303648'>, coefficients: <MagicMock id='135493237297072'>, axis: int)[source]

Unchecked conversion, given data, coefficients and axis

Parameters:
Return type:

numpy.ndarray

abstractmethod generate_coefficients(from_spec: Spec, to_spec: Spec) → <MagicMock id='135493237308064'>[source]

Generate coefficients for a pair of Spec definitions

Parameters:
Return type:

numpy.ndarray

get_coefficients(data_handle: DataHandle, from_spec: Spec, to_spec: Spec) → <MagicMock id='135493238208944'>[source]

Read coefficients, or generate and save if necessary

Parameters:
Return type:

numpy.ndarray

static get_convert_dims(from_spec, to_spec)[source]

Get dims for conversion from a pair of Spec, assuming only a single dimension will be converted.

Parameters:
Return type:

tuple(str)

simulate(data_handle: DataHandle)[source]

Convert from input to output based on matching variable names

smif.convert.interval module

Handles conversion between the set of time intervals used in the SosModel

There are three main classes, which are currently rather intertwined. Interval represents an individual definition of a period within a year. This is specified using the ISO8601 period syntax and exposes methods which use the isodate library to parse this into an internal hourly representation of the period.

TimeIntervalRegister holds the definitions of time-interval sets specified for the sector models at the SosModel level. This class exposes one public method, add_interval_set() which allows the SosModel to add an interval definition from a model configuration to the register.

Quantities

Quantities are associated with a duration, period or interval. For example 120 GWh of electricity generated during each week of February.:

Week 1: 120 GW
Week 2: 120 GW
Week 3: 120 GW
Week 4: 120 GW

Other examples of quantities:

  • greenhouse gas emissions

  • demands for infrastructure services

  • materials use

  • counts of cars past a junction

  • costs of investments, operation and maintenance

Upscale: Divide

To convert to a higher temporal resolution, the values need to be apportioned across the new time scale. In the above example, the 120 GWh of electricity would be divided over the days of February to produce a daily time series of generation. For example:

1st Feb: 17 GWh
2nd Feb: 17 GWh
3rd Feb: 17 GWh
...

Downscale: Sum

To resample weekly values to a lower temporal resolution, the values would need to be accumulated. A monthly total would be:

Feb: 480 GWh

Remapping

Remapping quantities, as is required in the conversion from energy demand (hourly values over a year) to energy supply (hourly values for one week for each of four seasons) requires additional averaging operations. The quantities are averaged over the many-to-one relationship of hours to time-slices, so that the seasonal-hourly timeslices in the model approximate the hourly profiles found across the particular seasons in the year. For example:

hour 1: 20 GWh
hour 2: 15 GWh
hour 3: 10 GWh
...
hour 8592: 16 GWh
hour 8593: 12 GWh
hour 8594: 21 GWh
...
hour 8760: 43 GWh

To:

season 1 hour 1: 20+16+.../4 GWh # Denominator number hours in sample
season 1 hour 2: 15+12+.../4 GWh
season 1 hour 3: 10+21+.../4 GWh
...

Prices

Unlike quantities, prices are associated with a point in time. For example a spot price of £870/GWh. An average price can be associated with a duration, but even then, we are just assigning a price to any point in time within a range of times.

Upscale: Fill

Given a timeseries of monthly spot prices, converting these to a daily price can be done by a fill operation. E.g. copying the monthly price to each day.

From:

Feb: £870/GWh

To:

1st Feb: £870/GWh
2nd Feb: £870/GWh
...

Downscale: Average

On the other hand, going down scale, such as from daily prices to a monthly price requires use of an averaging function. From:

1st Feb: £870/GWh
2nd Feb: £870/GWh
...

To:

Feb: £870/GWh

Development Notes

  • We could use numpy.convolve() to compare time intervals as hourly arrays before adding them to the set of intervals

class smif.convert.interval.Interval(name, list_of_intervals, base_year=2010)[source]

Bases: object

A time interval

Parameters:

  • id (str) – The unique name of the Interval

  • list_of_intervals (str) – A list of tuples of valid ISO8601 duration definition string denoting the time elapsed from the beginning of the year to the (beginning, end) of the interval

  • base_year (int, default=2010) – The reference year used for conversion to a datetime tuple

Example

>>> a = Interval('id', ('PT0H', 'PT1H'))
>>> a.interval = ('PT1H', 'PT2H')
>>> repr(a)
"Interval('id', [('PT0H', 'PT1H'), ('PT1H', 'PT2H')], base_year=2010)"
>>> str(a)
"Interval 'id' starts at hour 0 and ends at hour 1"
property baseyear

The reference year

property bounds

Return a list of tuples of the intervals in terms of hours

Returns:

A list of tuples of the start and end hours of the year of the interval

Return type:

list

property end

The end hour of the interval(s)

Returns:

An integer or list of integers, representing the hour from the beginning of the year associated with the end of each of the intervals

Return type:

int or list

property interval

The list of intervals

Setter appends a tuple or list of intervals to the list of intervals

property name

The name (or id) of the interval(s)

Returns:

Name or ID

Return type:

str

property start

The start hour of the interval(s)

Returns:

A list of integers, representing the hour from the beginning of the year associated with the start of each of the intervals

Return type:

list

to_hourly_array()[source]

Converts a list of intervals to a boolean array of hours

Returns:

A boolean array

Return type:

numpy.ndarray

class smif.convert.interval.IntervalAdaptor(name)[source]

Bases: Adaptor

Convert intervals, assuming uniform distributions where necessary

generate_coefficients(from_spec, to_spec) → <MagicMock id='135493237403968'>[source]

Generate conversion coefficients for interval dimensions

Assumes that the Coordinates elements contain an ‘interval’ key whose value corresponds to Interval data, that is a {‘name’: interval_id, ‘interval’: list of interval extents}.

For example, intervals covering each hour of a period

{ 'name': 'first_hour', 'interval': [('PT0H', 'PT1H')] }
{ 'name': ''second_hour', 'interval': [('PT1H', 'PT2H')] }
...

Or intervals corresponding to repeating hours for each day of a period

{
    'name': midnight',
    'interval': [
        ('PT0H', 'PT1H'), ('PT24H', 'PT25H'), ('PT48H', 'PT49H'), ...
    ]
},
{
    'name': ''one_am',
    'interval': [
        ('PT1H', 'PT2H'), ('PT25H', 'PT26H'), ('PT49H', 'PT50H'), ...
    ]
}
...
class smif.convert.interval.IntervalSet(name, data, base_year=2010)[source]

Bases: ResolutionSet

A collection of intervals

Parameters:

  • name (str) – A unique identifier for the set of time intervals

  • data (list) – Time intervals required as a list of dicts, with required keys start, end and name

static check_interval_bounds_equal(bounds)[source]

Checks that each interval in the list of bounds is equal

Parameters:

bounds (list) – A list of tuples containing the start and end hours of an interval

Returns:

True if all intervals in list of bounds are equal in length

Return type:

bool

property coverage

The total coverage in hours of the year by the interval set

Return type:

float

property data

Returns the intervals as a list

Return type:

list

static get_bounds(entry)[source]

Implement this helper method to return bounds from an entry in the register

Parameters:

entry – An entry from a ResolutionSet

Returns:

The bounds of the entry

Return type:

bounds

get_entry_names()[source]

Returns the names of the intervals

get_proportion(from_index, to_interval)[source]

Find proportion of interval address by from_index in to_interval

Parameters:

  • from_index (int) – Index of source interval

  • to_interval (Interval) – The destination interval

Return type:

float

intersection(to_entry)[source]

Return the destination intervals that intersect with to_entry

Argument

to_entry : Interval

returns:

A list of Intervals that intersect with bounds

rtype:

list

Notes

Look at the columns of the intersection array and identify overlapping intervals

smif.convert.region module

Handles conversion between the sets of regions used in the SosModel

class smif.convert.region.NamedShape(name, shape)

Bases: tuple

name

Alias for field number 0

shape

Alias for field number 1

class smif.convert.region.RegionAdaptor(name)[source]

Bases: Adaptor

Convert regions, assuming uniform distributions where necessary

generate_coefficients(from_spec, to_spec)[source]

Generate conversion coefficients for spatial dimensions

Assumes that the Coordinates elements contain a ‘feature’ key whose value corresponds to a GDAL vector feature represented as a dict, for example as returned by a fiona reader.

class smif.convert.region.RegionSet(set_name, elements)[source]

Bases: ResolutionSet

Hold a set of regions, spatially indexed for ease of lookup when constructing conversion matrices.

Parameters:

  • set_name (str) – Name to use as identifier for this set of regions

  • elements (iterable) – Iterable (probably a list or a reader handle) of fiona feature records e.g. the ‘features’ entry of a GeoJSON collection

as_features()[source]

Get the regions as a list of feature dictionaries

Returns:

A list of GeoJSON-style dicts

Return type:

list

centroids_as_features()[source]

Get the region centroids as a list of feature dictionaries

Returns:

A list of GeoJSON-style dicts, with Point features corresponding to region centroids

Return type:

list

check_valid_shape(shape)[source]
property coverage
property data

Resolution set data

Return type:

list

static get_bounds(entry)[source]

Implement this helper method to return bounds from an entry in the register

Parameters:

entry – An entry from a ResolutionSet

Returns:

The bounds of the entry

Return type:

bounds

get_entry_names()[source]

Get the names of the entries in the ResolutionSet

Returns:

The set of names which identify each entry in the ResolutionSet

Return type:

set

get_proportion(from_idx, entry_b)[source]

Calculate the proportion of shape a that intersects with shape b

intersection(to_entry)[source]

Return the set of regions intersecting with the bounds of to_entry

smif.convert.register module

Register, ResolutionSet abstract classes to contain metadata and generate conversion coefficients.

NDimensionalRegister is used in smif.convert.interval.IntervalAdaptor and smif.convert.region.RegionAdaptor.

class smif.convert.register.LogMixin[source]

Bases: object

property logger
class smif.convert.register.NDimensionalRegister(axis=None)[source]

Bases: Register

Abstract class which holds N-Dimensional ResolutionSets

Parameters:

axis (int, default=None) – The axis over which operations on the data array are performed

generate_coefficients(from_set: ResolutionSet, to_set: ResolutionSet) → <MagicMock id='135493235440192'>[source]

Generate coefficients for converting between two :class:`ResolutionSet`s

Coefficients for converting a single dimension will always be 2D, of shape (len(from_set), len(to_set)).

Parameters:
Return type:

numpy.ndarray

get_coefficients(source: str, destination: str) → <MagicMock id='135493235436448'>[source]

Get coefficients representing intersection of sets

Parameters:

  • source (string) – The name of the source set

  • destination (string) – The name of the destination set

Return type:

numpy.ndarray

get_entry(name: str) → ResolutionSet[source]

Returns the ResolutionSet of name

Parameters:

name (str) – The unique identifier of a ResolutionSet in the register

Return type:

smif.convert.ResolutionSet

property names: List[str]

Names of registered region sets

Returns:

sets

Return type:

list[str]

register(resolution_set: ResolutionSet)[source]

Add a ResolutionSet to the register

Parameters:

resolution_set (smif.convert.ResolutionSet)

Raises:

ValueError – If a ResolutionSet of the same name already exists in the register

class smif.convert.register.Register(axis=None)[source]

Bases: LogMixin

Abstract class which holds the ResolutionSets

Parameters:

axis (int, default=None) – The axis over which operations on the data array are performed

convert(data: <MagicMock id='135493236577184'>, from_set_name: str, to_set_name: str) → <MagicMock id='135493237333824'>[source]

Convert a list of data points for a given set to another set

Parameters:
Return type:

numpy.ndarray

static convert_with_coefficients(data, coefficients: <MagicMock id='135493237329936'>, axis=None) → <MagicMock id='135493237326192'>[source]

Convert an array of data using given coefficients, along a given axis

Parameters:
Return type:

numpy.ndarray

abstractmethod get_coefficients(source: str, destination: str)[source]
abstract property names
abstractmethod register(resolution_set: ResolutionSet)[source]
store = None
class smif.convert.register.ResolutionSet[source]

Bases: object

Abstract class which holds the Resolution definitions

as_dict()[source]

Get a serialisable representation of the object

abstract property coverage
property data

Resolution set data

Return type:

list

abstractmethod static get_bounds(entry)[source]

Implement this helper method to return bounds from an entry in the register

Parameters:

entry – An entry from a ResolutionSet

Returns:

The bounds of the entry

Return type:

bounds

abstractmethod get_entry_names()[source]

Get the names of the entries in the ResolutionSet

Returns:

The set of names which identify each entry in the ResolutionSet

Return type:

set

abstractmethod get_proportion(entry_a, entry_b)[source]

Calculate the proportion of entry_a and entry_b

Parameters:

  • entry_a (string) – Name of an entry in ResolutionSet

  • entry_b (string) – Name of an entry in ResolutionSet

Returns:

The proportion of entry_a and entry_b

Return type:

float

abstractmethod intersection(bounds)[source]

Return the subset of entries intersecting with the bounds

smif.convert.unit module

Handles conversion between units used in the SosModel

class smif.convert.unit.UnitAdaptor(name)[source]

Bases: Adaptor

Scalar conversion of units

convert(data_array, to_spec, coefficients)[source]

Convert a dataset between Spec definitions

Parameters:
Return type:

numpy.ndarray

generate_coefficients(from_spec, to_spec)[source]

Generate coefficients for a pair of Spec definitions

Parameters:
Return type:

numpy.ndarray

get_coefficients(data_handle, from_spec, to_spec)[source]

Read coefficients, or generate and save if necessary

Parameters:
Return type:

numpy.ndarray

parse_unit(unit_string)[source]

Parse a unit string (abbreviation or full) into a Unit object

Parameters:

unit (str)

Returns:

quantity

Return type:

pint.Unit

simulate(data_handle: DataHandle)[source]

Register unit definitions in registry for model run