From meter readings to a dispatch schedule: the complete process

Integration failures usually occur in the middle of the process, not in the forecasting model or dispatch optimisation. Turning a raw smart meter reading into a generation dispatch decision involves several overlooked stages. They receive less attention because they seem less interesting than machine learning or optimisation, but understanding them is essential.

Step one: gather and aggregate data

Smart meters across distribution networks generally send readings to local data concentrators rather than directly to a central forecasting system. Depending on the hardware, each concentrator collects data from 50 to 200 meters. It then sends aggregated interval readings to a head-end system, the metering data management system, or MDMS, which validates and stores them before exposing them through an API or file export.

Latency across this chain can differ substantially. Some newer MDMS implementations deliver 15-minute interval data 5 to 10 minutes late, while older systems can package readings hourly. Communication problems can also leave gaps at individual meters. The MDMS fills those gaps using imputation rules whose quality varies.

The forecasting system gets its inputs from the MDMS rather than from the meters themselves. Consequently, MDMS imputation quality has a direct effect on the data used to train and run the model. Weak methods, such as repeating the previous interval to replace a missing reading, introduce patterns the model must disregard or may incorrectly interpret as genuine demand behaviour.

Step two: validate readings and detect anomalies

A 15-minute interval reading should pass several sanity checks before entering the forecasting feature pipeline. These checks cover values beyond physically plausible feeder-capacity limits, single-interval spikes far above the feeder's historical maximum, and consistency with a distribution transformer load reading when one exists.

Legacy pipelines often omit this stage or implement it only minimally. They send the raw MDMS export straight to the model, assuming the MDMS has completed quality assurance. However, MDMS rules target billing correctness rather than forecasting. A reading acceptable for billing can still be a forecasting outlier, and such values can distort the demand patterns learned during training in ways that are hard to identify later.

The eZap ingestion pipeline has a validation layer using feeder-specific thresholds calculated from that feeder's historical reading distribution. Failed readings are flagged and replaced by a model-estimated value instead of the anomalous observation. The system records each imputation for audit, allowing operators to inspect the corrected intervals and the size of every adjustment.

Step three: build forecasting features

Validated interval readings feed the forecasting model as engineered features rather than as raw values. For every forecast step, the time series becomes a feature vector containing lag values at t-24h, t-48h, t-168h, and other offsets, plus rolling means and standard deviations for the preceding 7 days at the same hour. External weather and calendar variables described in separate posts are joined as well.

For day-ahead forecasts, feature construction must use only information available when the forecast runs. Backtesting can easily violate this rule. If a lag uses a reading that became available only after the forecast was committed, the resulting test performance will look too good. Our pipeline applies an explicit "data availability horizon" so no feature can point to a timestamp later than the forecast generation time minus the documented data latency.

Step four: produce a probabilistic forecast

Using the feature vector, the model generates P10, P50, and P90 demand forecasts for each 15-minute interval in the horizon. A day-ahead forecast contains 96 intervals at four per hour over 24 hours for each feeder. eZap's full 48-hour day-ahead horizon contains 192 intervals per feeder.

The result is a structured time series made up of quantile values for each interval. We store and expose it through the API as a JSON array of objects containing a UTC timestamp, P10, P50, and P90 in MW, plus a quality flag showing whether any input feature used an imputed value. Operators can use those flags to trace a forecast step back to anomalous inputs and their treatment.

Step five: derive the dispatch schedule

The probabilistic forecast feeds the dispatch scheduler; it is not the final result. The scheduler treats P50 as expected demand and P90 as the risk envelope. It also considers contracted generation units, their capacity, ramp rates, and minimum commitment durations, then finds a least-cost schedule that meets expected demand while maintaining an acceptable probability of covering actual demand.

Dispatch produces a unit commitment schedule specifying which generation sources operate, their MW output, and the intervals involved. For a small DSO with three or four contracted reserve units, the optimisation remains manageable. Larger operators with more complicated portfolios face tighter constraints, especially when minimum run times or ramp rates link one interval to the next.

The operator's commercial team uses the dispatch schedule to notify generation providers and the balancing market. It must be created and sent before the applicable gate closure deadline. For day-ahead operations, that is typically 12:00 to 14:00 on the preceding day. The entire journey from raw meter readings to submission must fit inside that period while leaving time for operator review.

Where the pipeline's middle stages break down

Data latency and imputation quality are the pipeline's two most frequent failure points. If integration does not document the data availability lag, forecasts may rely on older readings than expected and lose accuracy quietly. An MDMS rule that systematically fills communication outages with zeros, such as during low-demand overnight periods, teaches the model to expect artificially low overnight demand and pushes P10 downward, affecting reserve sizing.

Neither problem is usually dramatic, and neither makes a typical day's forecast visibly wrong. Instead, each produces a persistent small error that may be blamed on the forecasting model because the less visible pipeline is overlooked. Careful pipeline design and thorough integration tests prevent these issues, while routine comparison of forecast accuracy with data-quality metrics helps detect them.