Managing High-Frequency Data for Increased Regulatory Demands

by | Jul 11, 2016

This article is included in these additional categories:

Large plants and facilities face increasingly stringent regulatory pressure to manage larger amounts of data at a faster frequency than ever before. Companies all over the world are moving to higher frequency data collection and calculations. Decades ago, facilities could meet regulatory requirements with monthly, quarterly or even yearly estimates based on summary input data collected at those same frequencies. Now they are driven to collect such data every few seconds, summarize it every few minutes, and calculate emissions in near real time.

One notable example is the HRVOC (highly reactive volatile organic compounds) rule in Texas. Recognizing that HRVOCs play a significant role in ground-level ozone formation, the state started requiring companies to make efforts to track and reduce HRVOC emissions. Starting in 2014, the program included mandatory participation in an HRVOC cap-and-trade program for many chemical plants and oil refineries highly concentrated in the Houston-Galveston area. Through this cap-and-trade program, the state limits the total tons of HRVOCs that can be emitted collectively by all the plants in that area, and each plant has its own specific cap that it cannot exceed. This has pushed these plants to track HRVOC emissions at a frequency and level of detail needed to accurately inventory their HRVOC emissions so they know whether they are within their assigned caps, or if they need to purchase expensive emissions credits. These plants need to collect HRVOC data from monitoring devices every few seconds, and calculate emissions based on this data every few minutes. For just one of the emission sources at a plant, like a flare, there could be 25 different pollutant calculations executed every five minutes. This means millions of calculations a year for a single emission source, of which there are hundreds within a typical plant.

Driven by more stringent regulations, demanding sustainability reporting, and pressure from shareholders, companies are rethinking their systems for managing high-frequency data. They are taking a closer look at ways to remove manual processing, and collect and utilize huge amounts of accurate data at a faster frequency than they ever have before.

Avoid custom integration. In the past, it was sufficient for an in-house IT person to create a custom routine or custom integration to retrieve data once a month from a production system, for example. That meant receiving a data dump from other systems into a file. Doing that type of custom integration wasn’t hard to maintain.

Now that monitoring data needs to be collected and used in calculations at such high frequency, integration is crucial. The data being collected every few minutes has to be integrated from a variety of systems: data historians, operating systems, process historians like PI or IP21, lab information management systems, production accounting systems, continuous emissions monitoring systems, and many others.

What we’re seeing is that air engineers have to have near-real-time integration between the many plant systems and their environmental calculation system, and data needs to be retrieved every few seconds. The software versions for these different systems are changing constantly, too. Keeping the connections to these systems working when versions change is just too much time and effort for an in-house IT person.

The IT departments in these companies are also getting smaller. The hosting of servers and IT is being outsourced to third parties, so the personnel that previously would have helped the air engineer are not there anymore. Thus, environmental calculation systems must integrate with other systems out-of-the-box. While a company could customize an old system to work, that usually ends up being more expensive to implement and maintain over time. Instead, companies are turning to off-the-shelf solutions.

Match system performance to expectations. Without the right technology to do calculations in an automated way, the result is error-laden data. When companies are buying and trading emissions in mandatory cap-and-trade programs, having errors can result in non-compliance and significant costs.

The software system for environmental data management and air emissions calculations has to be able to handle many years’ worth of values and calculations across all emissions sources, which is likely tens or even hundreds of millions of data points per year. Recently I met with a chemical company that has had the same off-the-shelf system for about a decade. When they use this system to process one source of emissions for a year, the calculations take days. The system was not originally designed to perform calculations at such a high frequency. In order to increase system flexibility and easily adapt to changing requirements, they are better served with off-the-shelf technology that is more advanced than what they first purchased 10 years ago.

Integration needs to happen in near-real-time. Part of the current challenge with integration is the sheer quantity of data coming in from different systems. An air or environmental engineer needs to get that data in near-real-time, making integration much more complicated. The tool they use has to calculate emissions tied to all these systems every few minutes.

Years ago, it was fine not to have true real-time integration. A summary of data exported into a spreadsheet from another system once a month was enough. Now each data value must come across into their system in real time. That has also put the onus on having high-performing and proven off-the-shelf integration.

Automatically check data for quality. High-frequency data is coming in from monitoring devices that can fail. They break, they lose power, they might not be calibrated correctly, delivering unusable and invalid values. All the data has to be reviewed and checked for quality.

When engineers only had to do these reports once a month or once a year, they could eyeball the data to catch missing or unexpected values. With data at such high frequencies and quantities, that is not humanly possible. The software system that brings in the data and performs the calculations has to be able to quality-check the data in an automated way. Errors it must check for include missing values, values that are out of the expected range, and values that deviate too much from other recently collected values. The software has to have rules that will check all the data and uncover these issues.

An engineer still needs to know about the issues and decide what to do about them, but that person can’t find them all without help. The software must proactively alert the engineer about these issues. The engineer might then override a missing value, enter a new one, or make manual adjustments. They might have to talk to someone in operations and ask about the cause of the erroneous reading. Also, for certain types of data errors where a repeatable logic can be defined by the engineer to fix or substitute the value, the system itself must have the ability to be configured to perform data repairs and substitutions automatically, alleviating the burden on the engineer to fix all data errors manually.

For example, a customer in Hong Kong was trying to manually deal with millions of values and calculations using spreadsheets. They were missing many errors in their data, and the local government started coming down hard on them with fines and penalties due to their inaccurate emissions reporting. Then they made the move from manual spreadsheets to an automated software system for collecting, checking, and processing their high-frequency data. Now the millions of data points they collect each year are managed accurately and efficiently.

Based on all of these requirements, companies are rethinking their systems for managing environmental performance data. Systems that were adequate in the past are now being severely tested by the tremendous volume and high frequency of data needed today. As regulations continue to push the limits of current systems, companies must look for solutions that meet the latest needs.

Neal Rosen manages the global Operational Excellence & Risk Management (OE&RM) Solution Engineering team at IHS. He and his team are a group of OE&RM subject matter experts with over 100 years of experience in solving business problems in the OE&RM area across large business verticals like oil and gas, chemicals, and metals and mining. Mr. Rosen has a degree in chemical engineering and over 20 years of experience in OE&RM data management and software solutions. 

Additional articles you will be interested in.

Stay Informed

Get E+E Leader Articles delivered via Newsletter right to your inbox!

This field is for validation purposes and should be left unchanged.
Share This