smif.controller package

Implements procedures and tools to run a system-of-systems model Provides methods for copying a sample project into a local directory and for or running a system-of-systems model.

Example

Copy a project folder:

>>> copy_project_folder('/projects/smif/')

Run a single system-of-systems model:

>>> execute_model_run('energy_supply_demand', store)

Use the SubProcessRunScheduler to run a system-of-systems model:

>>> my_scheduler = SubProcessRunScheduler()
>>> my_scheduler.add(
        'energy_supply_demand',
        {
            'directory': '/projects/smif'
        }
    )
>>> my_scheduler.status('energy_supply_demand')
{
    'message': 'running'
}
>>> my_scheduler.status('energy_supply_demand')
{
"err": "",
"output": "20102010\n2015\n2015\nModel run '20170918_energy_water' complete\n",
"status": "done"
}

class smif.controller.ModelRunner(warm_start=False)

Bases: object

The ModelRunner orchestrates the simulation of a SoSModel over decision iterations and timesteps as provided by a DecisionManager.

build_job_graph(model_run, bundle)

Build a job graph

Build and return the job graph for an entire bundle, including before_model_run jobs when the models were not yet initialised.

Constraints: - Bundle must have keys: ‘decision_iterations’ and ‘timesteps’ - Running a bundle runs each (decision iteration, timestep) pair specified by the

combinations of decision iterations and timesteps

• (decision iteration, timestep) pairs must be unique over an entire model run

• In a single bundle, timesteps must be a consecutive subset of the model horizon timesteps

The first timestep in each decision iteration of a bundle is either: - the first timestep in the model horizon and initialised from the model run starting

point with scenario data and initial-timestep interventions only

• or another timestep, picking up from where some previous (timestep, decision iteration) left off, which is explicitly included in the bundle.

If a bundle’s timesteps start from a timestep after the first one in the model horizon, the bundle must provide ‘decision_links’, and bundle must start from the very next timestep available in the model horizon.

Jobs need to be able to identify a point to pick up from, namely the (timestep, decision iteration) which identifies the immediately preceding simulation state.

E.g. request running first two timesteps:

{
    'decision_iterations': [0, 1],
    'timesteps': [0, 1]
}

Run first two timesteps again, with an updated decision:

{
    'decision_iterations': [1, 2],
    'timesteps': [0, 1]
}

Results meet decision requirements, so run next two timesteps, linking this bundle’s decision iterations to previous decision iterations:

{
    'decision_iterations': [3, 4],
    'timesteps': [2, 3],
    'decision_links': {3: 1, 4: 2}
}

Parameters:

model_run (smif.controller.modelrun.ModelRun bundle: dict)

Returns:

different operations and timesteps

Return type:

networkx.Graph A populated job graph with edges showing dependencies between

static filter_job_graph(modelrun_name, job_graph, complete_jobs)

solve_model(model_run, job_scheduler, store, dry_run=False)

Solve a ModelRun

This method steps through the model horizon, building a job graph and submitting this to the scheduler at each decision loop.

Parameters:

• model_run (smif.controller.modelrun.ModelRun)

• store (smif.data_layer.Store)

smif.controller.copy_project_folder(directory)

Creates folder structure in the target directory

Parameters:

directory – Location where the sample project should be copied to

smif.controller.execute_decision_step(model_run_id, decision, store)

Request the next set of decisions from the decision manager, including the bundle of timesteps and decision iterations to simulate next.

Parameters:

• model_run_id (str) – Modelrun id of overarching model run

• decision (int) – Decision to run - should be 0 to start, or n+1 where n is the max decision executed so far

• store (Store)

smif.controller.execute_model_before_step(model_run_id, model_name, store, dry_run=False)

Runs model initialisation

This method

smif.controller.execute_model_run(model_run_ids, store, warm=False, dry=False)

Runs the model run

Parameters:

modelrun_ids (list) – Modelrun ids that should be executed sequentially

smif.controller.execute_model_step(model_run_id, model_name, timestep, decision, store, dry_run=False)

Runs a single step of a model run

This method is designed to be the single place where smif actually calls wrapped models.

Loading model wrapper implementations (via ModelLoader) is also deferred to this method, so that the smif scheduler environment need not have all the python dependencies of all wrapped models.

For example, in a scheduled containerised environment, the model run can be configured and set running (smif run modelrun_name) in one environment, and individual models can run (smif run modelrun_name –model model_name –timestep 2050) in their own environments.

Parameters:

• model_run_id (str) – Modelrun id of overarching model run

• model_name (str) – Model to run

• timestep (int) – Timestep to run

• decision (int) – Decision to run

• store (Store)

Subpackages

• smif.controller.job package
  • SerialJobScheduler
    • SerialJobScheduler.add()
    • SerialJobScheduler.get_status()
    • SerialJobScheduler.kill()
  • Submodules
  • smif.controller.job.serial_job_scheduler module
    • SerialJobScheduler
      • SerialJobScheduler.add()
      • SerialJobScheduler.get_status()
      • SerialJobScheduler.kill()
• smif.controller.run package
  • DAFNIRunScheduler
    • DAFNIRunScheduler.add()
    • DAFNIRunScheduler.get_dict_from_json()
    • DAFNIRunScheduler.get_scheduler_type()
    • DAFNIRunScheduler.get_status()
    • DAFNIRunScheduler.get_yamls()
    • DAFNIRunScheduler.kill()
  • SubProcessRunScheduler
    • SubProcessRunScheduler.add()
    • SubProcessRunScheduler.get_scheduler_type()
    • SubProcessRunScheduler.get_status()
    • SubProcessRunScheduler.kill()
  • Submodules
  • smif.controller.run.dafni_run_scheduler module
    • DAFNIRunScheduler
      • DAFNIRunScheduler.add()
      • DAFNIRunScheduler.get_dict_from_json()
      • DAFNIRunScheduler.get_scheduler_type()
      • DAFNIRunScheduler.get_status()
      • DAFNIRunScheduler.get_yamls()
      • DAFNIRunScheduler.kill()
  • smif.controller.run.subprocess_run_scheduler module
    • SubProcessRunScheduler
      • SubProcessRunScheduler.add()
      • SubProcessRunScheduler.get_scheduler_type()
      • SubProcessRunScheduler.get_status()
      • SubProcessRunScheduler.kill()

Submodules

smif.controller.build module

smif.controller.build.build_model_run(model_run_config)

Builds the model run

Parameters:

model_run_config (dict) – A valid model run configuration dict with objects

Return type:

smif.controller.modelrun.ModelRun

smif.controller.build.get_model_run_definition(store, modelrun)

Builds the model run

Parameters:

• store (Store) – Path to the project directory

• modelrun (str) – Name of the model run to run

Returns:

The complete model_run configuration dictionary with contained ScenarioModel, SosModel and SectorModel objects

Return type:

dict

smif.controller.build.get_scenario_models(scenarios, handler)

Read in ScenarioModels

Parameters:

• scenarios (dict)

• handler (smif.data_layer.Store)

Return type:

list of ScenarioModel

smif.controller.build.get_sector_models(sector_model_names, handler)

Read and build SectorModels

Parameters:

• sector_model_names (list of str)

• handler (smif.data_layer.Store)

Return type:

list of SectorModel implementations

smif.controller.execute_run module

Execute a model run - discover and schedule all steps in order

smif.controller.execute_run.execute_model_run(model_run_ids, store, warm=False, dry=False)

Runs the model run

Parameters:

modelrun_ids (list) – Modelrun ids that should be executed sequentially

smif.controller.execute_step module

Execute a single step directly

smif.controller.execute_step.execute_decision_step(model_run_id, decision, store)

Request the next set of decisions from the decision manager, including the bundle of timesteps and decision iterations to simulate next.

Parameters:

• model_run_id (str) – Modelrun id of overarching model run

• decision (int) – Decision to run - should be 0 to start, or n+1 where n is the max decision executed so far

• store (Store)

smif.controller.execute_step.execute_model_before_step(model_run_id, model_name, store, dry_run=False)

Runs model initialisation

This method

smif.controller.execute_step.execute_model_step(model_run_id, model_name, timestep, decision, store, dry_run=False)

Runs a single step of a model run

This method is designed to be the single place where smif actually calls wrapped models.

Loading model wrapper implementations (via ModelLoader) is also deferred to this method, so that the smif scheduler environment need not have all the python dependencies of all wrapped models.

For example, in a scheduled containerised environment, the model run can be configured and set running (smif run modelrun_name) in one environment, and individual models can run (smif run modelrun_name –model model_name –timestep 2050) in their own environments.

Parameters:

• model_run_id (str) – Modelrun id of overarching model run

• model_name (str) – Model to run

• timestep (int) – Timestep to run

• decision (int) – Decision to run

• store (Store)

smif.controller.modelrun module

The Model Run collects scenarios, timesteps, narratives, and model collection into a package which can be built and passed to the ModelRunner to run.

The ModelRunner is responsible for running a ModelRun, including passing in the correct data to the model between timesteps and calling to the DecisionManager to obtain decisions.

ModeRun has attributes: - id - description - sosmodel - timesteps - scenarios - narratives - strategy - status

class smif.controller.modelrun.ModelRun

Bases: object

Collects timesteps, scenarios , narratives and a SosModel together

name

The unique name of the model run

Type:

str

timestamp

An ISO8601 compatible timestamp of model run creation time

Type:

datetime.datetime

description

A friendly description of the model run

Type:

str

sos_model

The contained SosModel

Type:

smif.model.sos_model.SosModel

scenarios

For each scenario set, a mapping to a valid scenario within that set

Type:

dict

narratives

A list of smif.parameters.Narrative objects

Type:

list

strategies

Type:

dict

status

Type:

str

logger

Type:

logging.Logger

results

Type:

dict

as_dict()

Serialises smif.controller.modelrun.ModelRun

Returns a dictionary definition of a ModelRun which is equivalent to that required by from_dict to construct a new model run

Return type:

dict

classmethod from_dict(config)

Create a smif.controller.modelrun.ModelRun from a dictionary

property model_horizon

Returns the list of timesteps

Returns:

A list of timesteps, distinct and sorted in ascending order

Return type:

list

run(store, job_scheduler, warm_start_timestep=None, dry_run=False)

Builds all the objects and passes them to the ModelRunner

The idea is that this will add ModelRuns to a queue for asychronous processing

validate()

Validate that this ModelRun has been set up with sufficient data to run

class smif.controller.modelrun.ModelRunner(warm_start=False)

Bases: object

The ModelRunner orchestrates the simulation of a SoSModel over decision iterations and timesteps as provided by a DecisionManager.

build_job_graph(model_run, bundle)

Build a job graph

Build and return the job graph for an entire bundle, including before_model_run jobs when the models were not yet initialised.

Constraints: - Bundle must have keys: ‘decision_iterations’ and ‘timesteps’ - Running a bundle runs each (decision iteration, timestep) pair specified by the

combinations of decision iterations and timesteps

• (decision iteration, timestep) pairs must be unique over an entire model run

• In a single bundle, timesteps must be a consecutive subset of the model horizon timesteps

The first timestep in each decision iteration of a bundle is either: - the first timestep in the model horizon and initialised from the model run starting

point with scenario data and initial-timestep interventions only

• or another timestep, picking up from where some previous (timestep, decision iteration) left off, which is explicitly included in the bundle.

If a bundle’s timesteps start from a timestep after the first one in the model horizon, the bundle must provide ‘decision_links’, and bundle must start from the very next timestep available in the model horizon.

Jobs need to be able to identify a point to pick up from, namely the (timestep, decision iteration) which identifies the immediately preceding simulation state.

E.g. request running first two timesteps:

{
    'decision_iterations': [0, 1],
    'timesteps': [0, 1]
}

Run first two timesteps again, with an updated decision:

{
    'decision_iterations': [1, 2],
    'timesteps': [0, 1]
}

Results meet decision requirements, so run next two timesteps, linking this bundle’s decision iterations to previous decision iterations:

{
    'decision_iterations': [3, 4],
    'timesteps': [2, 3],
    'decision_links': {3: 1, 4: 2}
}

Parameters:

model_run (smif.controller.modelrun.ModelRun bundle: dict)

Returns:

different operations and timesteps

Return type:

networkx.Graph A populated job graph with edges showing dependencies between

static filter_job_graph(modelrun_name, job_graph, complete_jobs)

solve_model(model_run, job_scheduler, store, dry_run=False)

Solve a ModelRun

This method steps through the model horizon, building a job graph and submitting this to the scheduler at each decision loop.

Parameters:

• model_run (smif.controller.modelrun.ModelRun)

• store (smif.data_layer.Store)

smif.controller.setup module

smif.controller.setup.copy_project_folder(directory)

Creates folder structure in the target directory

Parameters:

directory – Location where the sample project should be copied to