Replacing File-Based Integration with an API Architecture

An industrial data flow was built around specialized equipment and a file-based integration pattern. Measurements were collected upstream, exported through an intermediary system, and transferred as files to a historian.

The architecture worked, but the integration boundary was effectively a folder and a file format.

That created several practical problems: delivery timing was implicit, retries were difficult to reason about, duplicate files were possible, and failures could occur at multiple points without a clear contract between producer and consumer.

The architectural direction was to replace the file-transfer dependency with an API-based integration:

specialized equipment → source system → REST API → AWS integration → historian

Lizard sits on the receiving side as the historian; it is not the integration mechanism itself.

The important change was not simply moving from FTP to REST. The goal was to make the responsibilities explicit.

The source system owns the data and exposes it through a defined API. The AWS integration layer handles transport, authentication, processing and resilience. Object storage can provide durable buffering where required, while serverless components can process or forward data without introducing a permanently running integration service.

This also makes failure handling much more explicit. An API request can have a defined response, retry strategy and error state. Data can be persisted before processing, allowing temporary failures downstream without requiring the source system to resend everything manually.

The main architectural question became:

Where should the integration responsibility live?

Keeping the connection directly inside the historian would tightly couple the historian to the external system. Putting the integration logic in the source system would make the source responsible for downstream concerns it does not own.

A separate integration boundary provides cleaner separation:

producer owns production of data integration layer owns transport and transformation historian owns historical storage and use

That makes the architecture easier to operate, monitor and change later.

Key insight: Replacing FTP with REST is not the architecture. The real improvement is turning an implicit file-transfer dependency into an explicit, observable integration contract with clear ownership.