APIs and tools

ICP and real-time dispatch APIs

You can use the 'ICP Connection Data API' to get details about all installation control points in New Zealand and use the 'Real-Time Dispatch API' for up-to-the-minute data on electricity dispatch.

Datasets

We have data files covering a wide range of New Zealand electricity industry available for you to download and use for analysis, research and decision making.

These are divided into retail, wholesale, forward markets and environment.

GitHub

GitHub is an online code repository and change management facility of models and scripts. Analysts are welcome to use these codes or contribute to their development.

Matching trading periods to New Zealand time

The New Zealand wholesale electricity market runs on 30-minute trading periods. When daylight saving starts or ends and the clocks change, it can be tricky to map times to the correct trading period numbers.

New Zealand Standard Time (NZST) is currently defined in the Time Act 1974 as being 12 hours in advance of Co-ordinated Universal Time (UTC). Since 1974, New Zealand has also defined a Daylight Time (NZDT) as a one-hour advance on NZST. It is therefore 13 hours in advance of UTC.

From time to time, the New Zealand Government may modify the configuration of the daylight-saving regime. About daylight saving in NZ.

Daylight saving currently starts on the last Sunday in September, when 2.00am becomes 3.00am, and it ends on the first Sunday in April, when 3.00am becomes 2.00am.

You can use the following spreadsheet to easily match the time to the correct trading period.

vSPD (vectorised Scheduling, Pricing and Dispatch)

The vSPD (vectorised Scheduling, Pricing and Dispatch) model is a replica of SPD. It is written and solved using the GAMS software and is based on the published SPD formulation.

vSPD takes advantage of the computational efficiency of modern linear programming and mixed integer programming solvers by vectorising many SPD schedules prior to solving.

Vectorisation involves reformulating the sequential and independent problems as a single optimisation, requiring just a single solve operation. In the case of ex-post pricing, prior to the advent of real-time pricing on 1 November 2022, this meant that all 48 trading periods from a single day were solved as a single optimisation.

With real-time pricing, there are typically about 250 SPD schedules that give rise to dispatch prices, spanning the 48 trading periods each day. These comprise real-time dispatch and price-responsive schedules and are similarly vectorised to be solved as a single optimisation.

The input data for vSPD is provided in the form of GDX files. You can use Daily GDX files in vSPD and here are the files prior to the advent of real-time pricing.

vSPD output is generated as a collection of GDX and CSV files that report on the key model outputs such as generation, prices, branch flows and reserves. Some inputs to the model, for example load, are also included in the output reports.

JADE (hydrological modelling)

JADE is a modelling package that implements a multistage stochastic optimisation representing the New Zealand electricity generation sector, with a rich treatment of the hydrological aspects of the sector. The key outputs of the model include a water value surface for each stage or week of the modelled time horizon, typically a year, and corresponding marginal water values for each reservoir represented in the model.

One of the difficulties with planning and operational decision making in a hydro-dominated electricity system such as New Zealand's is the uncertainty and variability associated with inflows into hydro storage reservoirs. JADE is an ideal tool to aid decision making in the presence of such uncertainty.

Some high-level characteristics of JADE:

• The EPOC team at Auckland University created and maintain the JADE modelling package. Significant contributions over the years have come from A Philpott, G Pritchard, A Downward, O Dowson, and L Kapelevich
• JADE supersedes DOASA, another EPOC model that the Authority has used
• JADE is formulated using the JuMP package, an algebraic modelling language for mathematical optimisation written in the Julia programming language
• At the heart of JADE is the Julia package for stochastic dual dynamic programming by Oscar Dowson SDDP.jl
• JADE can be solved with open-source solvers, although a commercial solver requiring a paid license eg, Gurobi or Cplex, is recommended for large-scale models
• JADE is open source and available from GitHub.

JADE datasets

The Authority makes JADE input datasets available at least annually or whenever a significant change on the electricity system needs to be included in the modelling. We also publish expected water values and the input data required to compute on a weekly basis.

About SPD case files

SPD stands for Scheduling, Pricing and Dispatch and is the name of the model used to run the New Zealand wholesale electricity market. See Part 13 of the Code for the rules describing how scheduling, pricing, and dispatch works.

The Authority receives all SPD case files from the System Operator from which published prices are derived. An SPD case file contains all of the inputs required for that case and all of the outputs generated by SPD for that case.

View real-time dispatch case files - from 1 November 2022

View real-time dispatch case files before 1 November 2022 - before real-time pricing

Naming convention for SPD case files

A case file is a zipped collection of text files.

The naming format for an SPD case file is MSS_DDCCCYYYYMMHHHHNNN_0X.ZIP where:

• DD denotes the Coordinated Universal Time (UTC) day number for the trading day. This is not zero padded, so for the first nine days of the month this is a single digit
• CCC denotes the case type where:
  • 120 denotes the WDS – weekly dispatch schedule
  • 131 denotes the PRSL – long price-responsive schedule
  • 130 denotes the PRSS – short price-responsive schedule
  • 133 denotes the NRSL – long non-response schedule
  • 132 denotes the NRSS – short non-response schedule
  • 101 denotes the RTD – real-time dispatch schedule
  • 110 denotes the RTP – real-time pricing schedule (following the the introduction of real-time pricing on 1 November 2022, RTP cases ceased to exist)
  • 111 denotes the FP – final pricing schedule (following the the introduction of real-time pricing on 1 November 2022, FP cases ceased to exist)
• YYYY denotes the UTC year
• MM denotes the UTC month. This is zero padded so is always two digits
• HHHH denotes the UTC 24-hour time for the first trading period that the case is solved for
• NNN denotes three random digits and can be used to distinguish multiple instances of a particular case.

View our description of GDX files for use with vSPD.

GDX files for vSPD

The input data required to run vSPD is published in the form of a daily GDX file. Our entire collection of vSPD GDX files associated with final pricing SPD cases can be downloaded from our datasets or accessed directly from the underlying Azure storage account.

GDX stands for GAMS data exchange and represents a binary file format for use with GAMS. The files are a convenient way to store the input data to be used by a GAMS-based model such as vSPD, and to import the data at model run time.

Part 13 of the Code describes how final prices are to be determined and published. 

GDX files up to 31 October 2022

Final pricing GDX files are generated from SPD final pricing case files. The naming convention for the final pricing GDX files is FP_YYYYMMDDx_X.gdx where:

• FP denotes final pricing
• YYYY denotes year
• MM denotes month
• DD denotes day
• The lower case x, which is generally not present, denotes that SPD and vSPD have yielded different prices for at least one node in at least one trading period
• The upper case X denotes the status of final prices and may take on the values of P, I, F, or there may be no value at all, where:
  • P stands for provisional
  • I stands for interim
  • F stands for final
  • The absence of a value means that no status has yet been assigned by the pricing manager.

YYYYMMDD together denote the trading day, midnight to midnight, to which the file relates. 

Final pricing GDX files are published immediately after we have received and processed the SPD case file. Hence, a GDX file can appear in our datasets before any status (P, I or F) is assigned to it by the pricing manager.

The pricing status indicator may not appear on the GDX file until some time after the pricing manager has determined the status. It is worth noting that the status indication on the GDX file can change. This does not necessarily mean that a new file has been generated, rather we simply rename the file once the new status has been obtained.

Only one GDX file per day is published. If a subsequent final pricing case is generated, we process it to produce the new GDX file and remove the previous GDX file.  

View GDX files - from 1 November 2022

SPD and vSPD

SPD, and its mathematical replica, vSPD, are large linear programming problems and so may be degenerate.

Degeneracy in SPD means there can be a tie between two candidate prices at a node that give rise to an identical (optimal) objective function value. In other words, while the solution is optimal in both cases, it is arbitrary as to which price is selected to break the tie.

SPD and vSPD have not been aligned in terms of tie-breaking rules. The practical implication of degeneracy is that the model has at least one redundant constraint. This is an extremely common occurrence in large models.

Any instance of vSPD and SPD producing different prices is investigated. If it is not due to degeneracy, the cause is rectified.

The Authority receives all published SPD case files. Each SPD case file contains all of the inputs required for that case and all of the outputs generated by SPD for that case.