a definitive guide
to data historians

In your manufacturing plant, the data and institutional knowledge you need to make good decisions is scattered everywhere. It lives isolated inside separate PLCs, proprietary SCADA platforms, localized databases, and the memory of your senior operators.

Chasing down trustworthy numbers or tracking down the right people to talk to just to figure out why a line went down is exhausting. It is a frustrating way to run a facility, and it makes you feel like you are managing your operations completely blind.  We know that pain. We spent years working the plant floor ourselves, fighting the exact same disconnected data silos.  

Historically, companies turned to traditional manufacturing data historian software to solve this. These systems were designed to ingest rapid streams of time-series data from machinery and store them for long-term review. But legacy historians brought a heavy trade-off: massive financial costs, rigid configurations, and data compression techniques that alter your data's historical truth.  A modern data historian must do things differently. It should serve as an open, accessible foundation for your entire operation, stripping away complexity instead of adding to it.  

To understand how to build a modern infrastructure, we have to look closely at how traditional legacy systems handle data collection and storage. Systems like AspenTech IP21, AVEVA PI System (formerly OSIsoft), and Rockwell FactoryTalk Historian were engineered decades ago when corporate servers faced major hardware boundaries.

Lossy Compression & The Swinging-Door Algorithm
Legacy architectures prioritize disk space optimization over raw accuracy. They achieve small storage footprints by using lossy compression routines, with the Swinging-Door Compression algorithm being the industry default. When sensor data flows into a traditional historian, the software does not record every point. Instead, it creates a virtual "door" that pivots around a baseline data point. As long as incoming sequential numbers fall within an arbitrary technical tolerance window (the compression deviation), the historian suppresses those values and deletes them from memory. When a sensor value finally breaks outside that tolerance line, the door snaps shut, a new coordinate is saved, and the process repeats.

When a user pulls up a trend line from five years ago, the legacy software draws a straight line between those surviving points. The database is not showing you what your physical machinery actually did; it is showing you a geometric approximation.

Monolithic tag structures and the "Tag Tax"
Legacy storage systems utilize highly structured, specialized flat-file configurations or proprietary relational hybrids to index data. Because these systems assign specific, hardware-locked memory boundaries to each parameter, vendors introduce a rigid administrative model: the per-tag license.

Under this model, a factory must purchase license tiers based on the exact quantity of data streams (tags) they intend to track (e.g., a 5,000-tag license vs. a 100,000-tag license). If an engineering team wants to track a secondary asset, like ambient temperature near a critical pump, they must audit their remaining tag allotment. This licensing strategy treats operational parameters as a scarce expense, creating data silos by forcing operations teams to purposefully ignore minor assets just to avoid triggering a higher software billing tier.

Raw Data Ingestion Without Compromise
Traditional industrial historians were built for an era when server hard drives were small and expensive. To save disk space, they relied on lossy compression algorithms, swinging-door compression being the most common. These algorithms intentionally throw away high-frequency data points that fall within an arbitrary tolerance window. The historian essentially "guesses" the line between points to save storage space.

When you are trying to run modern predictive analytics or train machine learning models, those missing data points matter. Lookback audits become inaccurate. A modern historian must preserve 100% data integrity. Instead of throwing values away, efficiency should happen through intelligent storage design. By recording data strictly on an update-on-change basis (delta logging), you eliminate the waste of writing the exact same value every single second. Most sensor data stays flat for long stretches; recording only the changes captures massive efficiency without sacrificing a single raw number. If connectivity is broken, the system immediately writes a null value alongside an exact timestamp, ensuring an honest, unalterable audit trail.

Timebase Historian abandons the core technical assumptions of legacy architectures, swapping lossy compression and restrictive indexing for a lightweight, high-throughput NoSQL model.

Lossless Storage via Delta-on-Change
Timebase preserves 100% data integrity without causing storage bloat. It eliminates lossy filtering entirely, ensuring that every captured value matches physical reality. Instead of drawing geometric approximations across missing values, Timebase optimizes files using a strict Delta-on-Change execution path.

The system continually monitors incoming telemetry. If a machine component polls at 100Hz but holds a steady temperature of 82.0°C for three hours, Timebase suppresses the redundant data writes. The moment the sensor registers 82.1°C, the new value is instantly committed to disk along with its exact timestamp.

For complex strings and alphanumeric statuses, Timebase builds an active index of known value state combinations. Instead of repeatedly writing long text blocks into the historical data file, it logs a short binary key pointing to the master value pairing. This technique maximizes storage space efficiency while keeping the underlying data completely raw, unmanipulated, and accurate.

Complete Network Loss Accountability
In industrial computing, an empty data block can skew critical operational analysis. If a machine stops reporting data due to a severed network line, a standard IT database might simply show a flat line, making it look as though the machine was idling perfectly.

Timebase addresses network drops immediately. The millisecond a data collector loses contact with its parent system or a physical sensor drops offline, the engine writes an explicit null marker directly into the log sequence, coupled with the precise timestamp of the connection break. This clear indicator prevents analytics tools from confusing a network outage with normal asset performance, ensuring an unalterable, honest audit trail.

Flat Files and Isolated Metadata Indexing
Traditional historians often encounter read latency bottlenecks because they mix tag configuration data (names, descriptors, engineering units) directly into the time-series files. As you scale to hundreds of thousands of tags over several years, reading a historical trend forces the database to sift through gigabytes of duplicated structural text.Timebase solves this by splitting identity from data:

Because tag names are never duplicated within the time-series files, file bloat drops significantly. This backend layout allows the engine to maintain a consistent write speed of 150,000 updates per second, while enabling read requests to execute at up to ten times that speed, all regardless of whether you are querying data from ten minutes or ten years ago.

The way data is filed on the back end dictates how fast you can read it years down the road. Legacy systems often slow to a crawl when executing long-term historical queries because their databases become bloated with duplicated metadata.

When your data is unified cleanly, your daily work changes. Everyone across the organization operates from the same source of truth. Plant floor operators can see exact real-time trends; quality teams can audit precise historical batches without gaps; and management can make clear, confident choices based on sensor reality. You move from reacting to historical headaches to making better, faster decisions for the future of your business.