Overview

The IData Pipeline is a cloud-based data pipeline that can easily transform and load your data into a data lake, Lakehouse and/or data warehouse, instantly ready for consumption downstream.

There is no coding required, the Pipeline runs entirely on JSON configurations for each dataset registered. Once a dataset is registered via our API, representative data files can be pushed to the Pipeline for processing. Based upon the JSON configuration, data can be run through data quality checks, deduplicated, transformed and loaded into an object-store data lake, Lakehouse, or into a data warehouse such as Snowflake or Redshift.

The product also supports pulling data into the Pipeline from database tables including from Postgres, MS-SQL Server, and MySQL databases.

The Pipeline employs open-source Apache Iceberg technology. Apache Iceberg is an open source project that enables building a Lakehouse architecture on top of data lakes. Apache Iceberg provides ACID transactions, scalable metadata handling, and unifies streaming and batch data processing on top of existing data lakes, such as S3.

Additionally, the Pipeline supports Change Data Capture (CDC) technology for Microsoft SQL-Server. Database changes can continually flow into the Pipeline via SNS and delivered via FIFO to the selected SQS queues.