Every day, organizations that operate sensor networks capture an enormous volume of information about their processes, their equipment, and their environment. Temperature, pressure, humidity, speed, consumption. The measurement is continuous and silent. And much of that data is never looked at again. The industry has a name for the phenomenon: dark data. Gartner coined the term to describe the information organizations collect, process, and store during ordinary activity, and then never use for analysis, decisions, or any other purpose. Its defining trait is invisibility. The data exists, takes up space, consumes infrastructure, and at the same time remains unmonitored, unclassified, and ungoverned.
How much data stays in the dark
The figures describe a large-scale problem. IBM estimates that close to 90% of collected industrial data goes unused. In environments with sensors and telemetry generating continuous streams, different sources agree that up to 90% of the data produced is never analyzed. An industrial Internet of Things analysis raises the estimate further, holding that more than 99% of the data from connected industrial systems goes unanalyzed and unused.
Dark data is not a marginal matter. It typically represents more than half of a company's total data.
Organizations know it
What is notable is that the problem does not go unnoticed by those who suffer it. A Splunk survey, conducted by TRUE Global Intelligence across more than 1,300 technology and business leaders, found that over 60% of organizations believe at least half of their data is dark data, and a third estimate that 75% or more is never used. In the same poll, 77% of respondents said that finding and using that data should be a priority.
The cost of storing what you never look at
Storing data that is never used costs money. A Veritas study calculates that 52% of the average company's data storage budget is spent on dark data. The organization pays to keep information that returns no decision.
A failure that is not technological
The most revealing point of recent research is the diagnosis of the cause. Underused data is not a failure of capture technology. The sensors work, the measurement arrives, the storage exists. The failure is in the database — in the way the organization structures, governs, and prepares the information so it can become analysis and decision.
The investment in instrumentation is already made. The data is already captured. What remains outstanding is the layer that transforms it into something useful. And while that layer is missing, half of the organization's information stays in the dark.