An AGILE New Platform for Data Streaming

NREL software engineers developed a new platform named AGILE, or Autonomous Grids – Identification, Learning, and Estimation, thanks to funding from the Department of Energy's Office of Electricity. At AGILE’s core is a data bridge for researchers who need streaming access to data stored in various repositories. It solves a problem when integrating tools with unique connections, or Application Programming Interfaces (APIs), into a single common API. It does this by connecting over the Transmission Control Protocol (TCP). This TCP transport layer creates a common interface between the data repository, data sharing, and data analysis components of the most widely used analysis platforms.

Advanced sensor and measurement systems make it easier to visualize the dynamic interactions playing out on modern power grids, making them essential for energy system operators and for researchers analyzing grid performance. Thanks to technological advancements, these systems record more data points at increasingly higher resolution and fidelity. The resulting enlarged datasets are particularly valuable to help plan for integrating renewable energy sources onto the grid while improving energy reliability and resilience. Yet, extracting actionable information from this fire hose of data presents a challenge.

Features

AGILE transforms information from a fire hose of data into a manageable stream. Some of its key features include:

• Built-in functionality: Researchers using MATLAB®, Python, or test hardware, can easily plug into this tool without modifying the software’s configuration or pulling in data libraries.
• Compatibility: Data stored in InfluxDB and Kafka are now available for users to subscribe to or stream on any application that can make a TCP connection.
• Real-time data streaming: Select specific time windows and simulate the data as if it were being reported in real time. This makes for a much more detailed simulation of what is happening in a live power system compared to the batched data in more traditional analysis.
• Flexibility: The tool is built on top of JavaScript and the Node.js runtime and is deployed in a containerized fashion. Drawing on native library files allows the application to go through rapid iterations during testing and high performance.

Image

In the AGILE Streaming Data Platform, data repository, data sharing, and data analysis components are connected by the AGILE data bridge.

Software Architecture

Tools like MATLAB® do not natively support subscribing to Kafka or querying InfluxDB to access the data stored within. There are a few libraries available to interface with these tools, but they are not as complete. Plus, using them requires modifying the baseline configuration of the MATLAB® software. The national laboratory researchers chose to design a TCP functionality for that reason, which serves as the lowest common denominator supported out of the box by most analysis software, programming languages, and even some hardware devices. Building a custom tool also allowed them to abstract most common use cases for interacting with data into simple configuration and control messages. The complicated stream management logic can be developed independently of the analysis algorithms.

The AGILE data bridge uses the concepts of sources, sinks, and processors to manage where data comes from, where it goes, and how it is presented. Messages sent over TCP control all aspects of the application. They can query and simulate specific sets of data, publish, and subscribe to external data tools, and stop and reset the application when required. Messages follow a common format consisting of a JSON object followed by a line break. All messages include an action field and may include other fields based on the action.

The AGILE data bridge is built on top of JavaScript and the Node.js runtime and is deployed in a containerized fashion in a Kubernetes environment. It makes use of built-in Node.js stream constructs and JSON processing capability to the maximum extent possible. This allows for flexibility, rapid iteration of the application during testing, and high performance since the critical portions of the code make use of native libraries.

AGILE in Action

AGILE was first implemented in performing system identification at NREL’s facility for Advanced Research on Integrated Energy Systems (ARIES). ARIES is equipped with advanced sensor systems and megawatt-scale power system hardware including utility-scale solar, battery storage, and wind energy. The AGILE software tool helped stream data to a data center 25 miles away where researchers had established a Kafka cluster for analysis. The heterogeneous advanced sensors sampled a broad spectrum of dynamic phenomena from microseconds to seconds. Researchers used the sensing data to design novel data-driven methods tailored to reveal nonlinear dynamics without requiring first-principles models. For example, they designed and tested a novel forecasting-aided state estimator that considers the dynamic response of inverter-based resources.

Image

In a demonstration of AGILE Data Bridge Control and Data Flow, the sent and received messages are shown on the left and the state of the data bridge is shown on the right.

Visit the National Renewable Energy Laboratory’s AGILE webpage to learn more about the application.