In various forms, they are part of nearly all the systems that permeate our world. Whether in finance, transportation, or the energy sector, there is a database behind every application. Depending on the field of application, one model may be more suitable than another. In all of the industries mentioned above, however, it can be crucial to know how a specific variable behaves over an extended period of time. It doesn’t matter whether we’re talking about stock prices, telemetry data, or power frequencies—in every case, values must be measured at continuous intervals.
Time-series databases are particularly well-suited for this purpose, as they are also capable of identifying specific trends. Whenever series of measurements, observations, or states at specific points in time are required, it makes sense to use time-series databases. The advantages of a time-series database include not only the organization of data by time, but also the high number of write operations and queries, as well as access to parallel data sources. In addition, it offers great flexibility in defining and typing data and includes functions for automatically deleting and compressing it.
In our energy projects in particular, we have repeatedly explored the advantages of time-series databases such as InfluxDB. Together with market-leading energy providers, we have developed applications for managing and forecasting energy shortages and surpluses in virtual power plants. Our customers supply electricity from renewable energy sources on the balancing power market. Energy providers are required to submit forecasts of their feed-in and withdrawal to maintain the standard frequency of the German power grid at 50 hertz. Since renewable energy production depends on various factors, fluctuations often occur that must be balanced out. To analyze these fluctuations, we have worked with our clients on applications using InfluxDB and Telegraf.
InfluxDB
InfluxDB was developed by InfluxData specifically for time-series data and is designed to handle enormous volumes of time-stamped data. Individual data points in a time series are automatically identified in InfluxDB by a timestamp. Tailored for high write and query volumes, the database is a reliable tool for continuous operation. InfluxDB offers efficient solutions, particularly for data management, with its features for aggregating data (Continuous Query) and deleting it (Retention Policy). Before we delve into that, however, we’d first like to clarify a few terms that are essential for understanding how time-series data is organized and stored in InfluxDB.
Structure
(using InfluxDB v1.8 as an example)
The top level is the database. It serves as a container for one or more time series, which are referred to as "measurements" in InfluxDB. Measurements are strings, and their names generally determine what data is recorded (e.g., temperature, speed). They contain tags, fields, and the time column. All data in InfluxDB is automatically and mandatorily assigned a time column, which in turn contains the timestamp indicating the time of the measurement, including the date and time.
Like the time column, fields are mandatory in an InfluxDB data structure. They form the key-value pair that records metadata (field keys) and the actual data value (field values). Field keys are strings and can, for example, represent energy production, while field values denote the actual measured values associated with the timestamp.
Unlike tags, fields are not automatically indexed by InfluxDB. Since each index must be updated, this has a negative impact on write operations. Therefore, it is recommended to index only those fields that are frequently used for queries and aggregations.
Tags are the best way to describe the data points. Like fields, they are key-value pairs and consist of a tag key and a tag value. They are not a mandatory part of the InfluxDB data structure, but they are indexed, which is why they are primarily suitable for storing data for queries.
Another important component for better understanding the InfluxDB structure—which is also referred to as the schema—is data points. They represent individual data records and consist of a measurement, a tag set, a field set, and a timestamp. Data points are made unique by the timestamp and the series to which they belong. A series is a collection of data points that share the same measurement, tag set, and field key.
To illustrate this structure in a practical way, we have developed a brief use case for an energy producer with a biogas plant, using fictional data.
(using InfluxDB v1.8 as an example)
In this example, the Measurement (engine) collects data on the generators. Under the tag key (engine_id), the tag value displays all generators that produce electricity. Since tag keys are ideal for querying metadata, we’ve added two more. One tag key (malo_id) identifies the market location, so we know for which market location the respective generator is producing electricity, and the other tag key (unit) specifies the unit in which energy production is measured. The actual measured values are found in the field values. Here, too, we have defined three Field Keys. The first Field Key (power_production) measures energy production, the second Field Key (rpm) measures the generator’s rotational speed, and the last Field Key (max_delta) determines how many additional kWh the generator could still supply up to its maximum energy output.
Retention Policy and Continuous Query
InfluxDB provides two valuable tools for managing and organizing data—especially large volumes of data—namely the retention policy (data deletion) and continuous queries (data aggregation).
Retention policies are unique to each database and specify how long data should be retained. To do this, InfluxDB compares the server's timestamp with the data's timestamp and deletes any data that is older than the time specified by the retention policy.
Continuous queries are queries that run automatically and regularly and store the results in a new measurement. To run a continuous query, it must include a function in the SELECT clause and a GROUP BY() time clause. InfluxQL offers a wide range of functions, which are categorized under the terms "aggregate," "select," "transform," and "predict.".
The following chart illustrates the effects of a retention policy and a continuous query. The time range of the data points to be deleted by the retention policy is continuously extended. For the continuous query, the mean() aggregate function is used to condense the data points, combining more and more data into a single data point.
However, to collect data or send it to InfluxDB, you need a dedicated application. Telegraf, for example, is a suitable option—it’s software that was developed specifically for this purpose by InfluxData.
Telegraph
Telegraf is a plugin-driven server agent for collecting and sending metrics and events from databases, systems, and IoT sensors. The application is written in Go and is compiled into a single binary file with no external dependencies. The plugin system supports a wide range of options for retrieving data (third-party APIs, StatsD, Kafka, MQTT, etc.) and outputting data (InfluxDB, OpenTSDB, Kafka, MQTT, etc.).
In closing, we would like to return to our example of the biogas plant and illustrate how the data source, Telegraf, database (InfluxDB), and output work together.
Architecture
It all starts with the energy generation or consumption system, which provides the data for InfluxDB. Using specialized hardware (sensors, tachometers) and software, measurements (energy production, motor speed, deviation from maximum energy production) are recorded and transmitted to an MQTT broker. Telegraf is programmed so that the application retrieves the data and then forwards it to InfluxDB. For example, through a portal, the data can be displayed to the energy producer in various formats, such as charts. This does not always require a dedicated application; tools such as Grafana are also capable of visually representing the data.
InfluxDB has a wide range of applications. Given its structure and features, which are designed to handle large numbers of time series, it’s worth trying out InfluxDB—or another time-series database—for one project or another.
