Back to Blog

Improve your ML with EQ

Hugo Birkelund
Archived blog post. This blog post has been transferred from our previous blogging platform. Links and images may not work as intended.

Machine learning (ML) and artificial intelligence (AI) present some fascinating modeling approaches. However, even with ML and hyper-fast computing, the old saying "garbage in, garbage out" or simply GIGO holds true. EQ provides structured and curated data for modeling. Our database is a treasure trove for model developers.

EQ = self-service

Help yourself to data. EQ’s business is to provide the best possible curated historical data and aligned forecasts, empowering you to excel in your job with minimum waste of time and energy. If you don't already have an account, get your access here.

Looking for something that is not there

Throwing in more data will not help you (much) if what you are looking for is not there. I have tried to illustrate the problem with two pictures. In the left picture, I have simply removed information and added noise, analog to a what you might see in a dataset.

 

Thanks to Fiona for guarding my Parrot Zik headset.

You can easily observe the problem in a picture. However, in data analysis, you can, at best, anticipate that something is missing. It could be too big times steps, poor spatial resolution, misaligned input used for model fitting and forecasting, or good old fashioned out-layers and missing values. More data might add to the problem and certainly waste your time.

A checklist

A checklist before you hit enter "estimate model" might save you a lot of time:

  1. Check your input data. At EQ, we do this as a profession. For instance, we provide curated synthetic data which we believe is a far superior alternative to actuals generated by TSOs.
  2. Avoid settling for raw weather data. Failing to align the output generated by weather models with real-life observations creates nasty errors and gut-wrenching frustrations. I know, from personal experience.
  3. Consider adding contextual information. Capacity changes, holiday effects, REMIT data, and more input add depth to the analysis. Failing to add context might contaminate your model and drain it for explanatory power.
  4. Look out for "known" non-linearity. A simple trick is, e.g. to use transformed data. Regardless of how advanced your model is, this simple trick might limit the data requirement tremendously, and improve the forecasts properties.

Take vitamins, not medicine

EQ provides a vast set of curated historical weather-driven fundamentals and indices. We believe they are perfect input for modeling and forecasting. Further, we provide all data in 15 minutes resolution, which is likely to become the power market’s preferred resolution. Using the correct data granularity from the get-go hugely simplifies modeling.

Contact me to discuss the best solutions for your team. We may have your data requirement covered, faster and simpler than you can imagine.

...I almost forgot

You do not have to download EQ’s data. Connect your developer tool or models directly into EQ’s timeseries database. It saves you time and simplifies your life.

EQ_Cut_Down_on_Data_managment_2

EQ generates approximately 4,5 million data points per price areas every day. It makes sense to leave the data in our custody.

More from the Blog

What's the price effect of Cobra?

Hugo Birkelund

The 700 MW Cobra cable, connecting the Netherlands and DK1 (Jylland) is scheduled to start commercial operation on Monday. In this short blog, we investigate the effects it will have on the price formation.

Read Story

Happy birthday 2019

Hugo Birkelund

This week we celebrate our 2nd anniversary as Energy Quantified or simply “EQ” – a Montel company. Two years on, we provide “all you need to be in power” for all price areas in Europe.

Read Story

SEE hydrology and an exciting August?

Eylert Ellefsen

August can become an exciting month for traders in the SEE region. Recent precipitation forecasts have remained very dry. If these continue, there will be a further reduction in hydropower production and an additional upward pressure on spot-prices in the SEE-region. This is the conclusion gleaned from EQ’s new Pan European hydro model, which will be launched soon.

Read Story

Large differences in Alpine hydro balances

Eylert Ellefsen

EQ's new hydrology models indicate that we enter August with large differences in hydrological balance across the Alpine area. Switzerland and Austria now have reservoir fillings of, respectively, 12% and 8% above their seasonal normal, whereas France and Italy are on their seasonal normal. High snow reservoirs and delayed melting during the spring, was followed by a heatwave end of June and start of July. This sparked melting, which in turn led to high inflow levels and ultimately rapidly increasing hydro reservoir levels in Switzerland and Austria.

Read Story

River temperatures, we can have you covered

Hugo Birkelund

As rising river temperatures may affect the output at thermal power plants due to cooling water restrictions, we invite you to suggest a river location for which we can provide forecasts and historical data.

Read Story

Can it rain from the ground up?

Eylert Ellefsen

Of course not! However, during the last seven days of June 3,5 TWh worth of hydropower resources simply vanished in thin air in the CWE area. That is what our new model for energy precipitation in Germany, France, Austria, Switzerland and Italy, tells us.

Read Story

Precipitation energy SEE area

Eylert Ellefsen

As part of our ongoing hydrology project, EQ now provides energy precipitation forecasts, seasonal norms and historical data for countries in South-East Europe, covering around 50 TWh of annual hydropower production – or nearly 30% of total power supply in the region.

Read Story

REMIT untangled

Hugo Birkelund

How do REMIT messages stack up? Energy Quantified (EQ) now provides quantified REMIT and UMM messages for all your areas. A notoriously complicated, but important data feed is boiled down to easy-to-handle numeric time series.

Read Story

Mr Hydrology joins EQ

Hugo Birkelund

We are pleased to announce that Eylert Ellefsen joins Energy Quantified (EQ) as a senior analyst. Many will recognise Eylert as the person who pioneered the development of hydro models in Europe.

Read Story

Video: Spot Market Sensitivities

Hugo Birkelund

Have you ever wondered how price sensitive your spot market is? Which hours that could rise sharply or dive into the negative?

Read Story

Ready to try Energy Quantified?

No payment or credit card required.
Would you rather like a personal demo? Book one here with Hugo