Skip to content

Plain-language guides to iot development

Date

Monday, November 16, 2026

Posted by

Date

Building an IoT Data Pipeline from Sensor to Insight

iot data pipeline

Your sensors are talking, but the readings arrive late, messy or not at all. An iot data pipeline fixes that by moving each reading through four stages: broker, processor, database and dashboard.

This guide sits inside the wider topic of iot development, so read it alongside the other build topics.

Item Value
Sensors in a batch-versus-streaming example 10,000 sensors
Data those sensors produce 600,000 data points per minute
Worst-case delay of an hourly batch pipeline up to 60 minutes
Output of 5,000 sensors at 1-second intervals 432 million data points per day
Full-resolution retention in a sample storage policy 7 days
1-minute average retention in the same policy 90 days

The figures come from RisingWave and iFactory examples and can change, so check the sources before you rely on them.

What an IoT data pipeline does from sensor to insight

A pipeline is a chain of small, replaceable stages. One published EMQX community walkthrough starts with Node-RED pulling sensor data from OPC-UA servers, then passes it on to a broker, a Python client, a time-series database and Grafana.

Whatever tools you pick, the stages stay the same:

  1. Collect: read the sensor or the industrial protocol.
  2. Transport: publish readings to a broker.
  3. Process: clean, filter and aggregate.
  4. Store: write to a database built for timestamps.
  5. Serve: show dashboards and fire alerts.

Why bother with streaming? RisingWave describes a factory floor with 10,000 sensors, which adds up to 600,000 data points per minute. An hourly batch job could leave an overheating motor undetected for up to 60 minutes.

Collect and transport readings with MQTT

For most devices, MQTT is the transport of choice. A system design write-up notes it uses a publish-subscribe model over TCP with a 2-byte header overhead, plus persistent sessions that queue messages for offline devices.

Devices publish to a topic, and anything that needs the data subscribes to it. That keeps the sensor ignorant of who reads its data, which is what lets you swap the later stages without touching firmware.

You can run the broker yourself or rent one. A low-level design guide names Mosquitto and EMQX as self-hosted options, while an AWS walkthrough uses IoT Core as the managed broker that devices connect to.

iot data pipeline Converging strands
Thin metallic strands converge toward a point against a dark background.

Security belongs at this stage. The same design guide has each broker authenticate devices through mutual TLS, checking device certificates against a device registry.

At higher volume, many teams place a log between the broker and the processors. The system design write-up says Kafka serves as a buffer and replay log, which helps when a downstream stage falls behind.

Process and clean data close to the source

Raw readings are noisy, so decide early what to keep. The EMQX walkthrough calls this edge analytics, and its author says sending only filtered data reduces network congestion.

The same source lists three typical jobs for an edge processor:

  • Anomaly detection, such as spotting out-of-range values.
  • Simple event rules, such as raising an alert when temperature passes a threshold.
  • Summarising data, such as averages over a window of time.

Real networks add messier problems. The low-level design guide warns that devices often resend duplicate messages, payload formats vary by firmware version, and out-of-order delivery is common.

That guide also suggests a per-device rate limiter, so one malfunctioning device cannot eat the capacity everyone else needs. Dropped messages should be logged as events rather than vanish silently.

Firmware changes the payload your devices send, so plan iot ota firmware updates together with your schema changes.

Where should you store sensor data?

Use a time-series database, not a general SQL server. iFactory calculates that 5,000 sensors reporting at 1-second intervals produce 432 million data points per day, and says relational databases don’t support automatic downsampling or retention-based deletion.

Several engines fit this job. The same source positions InfluxDB for pure time-series workloads, TimescaleDB where SQL compatibility and joins matter, and ClickHouse for heavy analytical queries.

Storage also needs a lifecycle. One sample policy from iFactory keeps 1-second resolution for 7 days, 1-minute averages for 90 days, hourly aggregates for 2 years and daily summaries forever.

Treat those numbers as an example, not a rule. Your retention depends on how far back your dashboards, reports and compliance checks need to look.

Turn stored data into dashboards and alerts

Dashboards are the part people actually see. In the EMQX example, Grafana connects to QuestDB to visualise sensor data in real time, while PowerBI reads aggregated data from DuckDB for higher-level reports.

If you prefer SQL over custom code, a streaming database can compute aggregates for you. RisingWave uses materialized views that update incrementally as each new reading arrives, and its TUMBLE function groups readings into fixed, non-overlapping intervals.

Alerting needs one more safeguard: deduplication. The system design write-up suggests firing an alert once when a condition is met and sending a resolution alert when it clears, so people are not paged repeatedly for one fault.

Build it in small steps

You do not need the full stack on day one. A small build from iot prototyping lets you prove the data path before you invest in scale.

  1. Publish readings from one sensor to a broker.
  2. Subscribe with a short script and print what arrives.
  3. Write the readings to a time-series database.
  4. Chart them in a dashboard.
  5. Add one threshold alert, then more devices.

Managed services can shorten the path. The AWS example notes that the IoT Rules Engine routes messages to different AWS services based on SQL-like queries.

Frequently asked questions

Do you need Kafka in an IoT data pipeline?

Show answer

Not at the start. A broker and one subscriber can carry a small project, and you can add a durable log later if processors fall behind, as the design write-up describes.

Should analysis happen at the edge or in the cloud?

Show answer

Use the edge for quick checks and filtering, and the cloud for long-term history and cross-site reports. The EMQX walkthrough does both, keeping real-time views separate from aggregated business reporting.

Why not just use PostgreSQL or MySQL?

Show answer

Plain relational setups lack built-in downsampling and automatic expiry of old rows, which sensor data needs. iFactory points to time-series engines, and TimescaleDB is one that keeps SQL.

Is MQTT the only protocol you can use?

Show answer

No. The system design write-up lists CoAP for extremely resource-constrained devices and HTTP/REST for devices with enough resources, where reliability matters more than efficiency.