Case Study: Designing a High-Throughput Time-Series Integration

When the Right Architecture Means Reconsidering the First Choice..

An architecture can look right at first glance.

The interface was sleek, documented and easy to integrate with. Described as a swiss army knife integration with buffering seemed like a clean way to connect an industrial data source to a historian.

The initial design made sense.

Then the requirements became the real test.

The integration had to handle significant data volumes, remain reliable under interruptions, and—most importantly—maintain extremely high data completeness. For industrial time-series data, losing a small percentage of measurements is not necessarily a minor technical issue. It can affect analysis, reporting and downstream processes.

Performance testing exposed a constraint in the selected API path. Adding concurrency helped, but the scaling behaviour remained less attractive than expected. The buffering component also introduced its own limitations.

At that point, the question changed.

It was no longer:

How do we optimize the architecture we already designed?

It became:

Does this architecture actually satisfy the requirements better than the available alternatives?

Reconsidering the obvious choice

An alternative was the manufacturer’s proprietary adapter.

At first glance, a proprietary adapter can appear less attractive. It introduces vendor dependency and may feel less flexible than building an integration around a generic API.

But architecture is not a competition between “modern” and “proprietary”.

The relevant question is whether the solution meets the actual requirements.

In this case, the manufacturer’s adapter had characteristics that aligned better with the problem:

• better scaling behaviour;
• established handling of buffering;
• integration patterns already familiar within the operational environment;
• and a stronger fit with the existing approach to maintaining data completeness.

The trade-off therefore became clearer.

Initial preference
API + custom integration
        │
        ▼
Measure against requirements
        │
        ├── Throughput: constraint
        ├── Buffering: constraint
        └── Data completeness: critical
        │
        ▼
Re-evaluate alternatives
        │
        ▼
Manufacturer adapter
        │
        ▼
Better fit with requirements

The outcome was not driven by which technology looked cleaner on a diagram.

It was driven by which architecture gave the strongest overall fit with the requirements and operational realities.

Requirements should be able to kill your favourite design

This is an important part of architecture work.

It is easy to become attached to an early solution because it is technically elegant, familiar, or already partially implemented.

An architect has to be willing to challenge that choice when evidence shows a mismatch.

That does not mean constantly redesigning systems.

It means defining the critical requirements early enough that architectural alternatives can be compared against them objectively.

In this case, throughput was measurable.

Data completeness was a much stronger architectural driver.

The ability to recover from interruptions and preserve data was more important than simply having a modern REST interface.

That changes the weighting of the decision.

The proprietary option was not automatically the better option

Choosing the manufacturer’s adapter does not make proprietary technology universally preferable.

It introduced its own trade-offs, including greater dependence on the vendor’s technology and potentially less control over the integration layer.

The point was not that proprietary was “better”.

The point was that the requirements changed the architectural evaluation.

A design that looks preferable before testing can become less suitable once real performance, resilience and operational requirements are understood.

The architect’s role

This is where the HLD adds real value.

The HLD should make the important requirements and architectural drivers explicit, document the alternatives considered, and provide a defensible rationale for the selected direction.

That also makes architecture-board review much more useful.

The discussion can move away from:

“I prefer this technology.”

toward:

“These are the requirements, these are the constraints we measured, these are the alternatives, and this is why the selected option fits the required outcome.”

The design can then be iterated based on feedback rather than defended simply because it was the original proposal.

Key insight: A good architect is not the person who gets the first design right. It is the person who is willing to change the design when the requirements and evidence show that another option is a better fit.