Learn Kusto - Reference Architecture

23 Apr, 2024 |

Brian is the person behind the dcode.bi site. He is keen on helping others be better at what they do around data and intelligence.

Learn kusto - Reference Architecture

Kusto cannot stand alone

When using Kusto and Azure Data Explorer the service cannot stand alone. It needs some data to give it value.

This blogpost tries to give some inout on the two IoT services som Azure that seamlessly integrates with the Azure Data Explorer (there is more, but to narrow it down :smile:).

Azure IoT Hub vs Event Hub

When working with Azure Data Explorer and loading data to the storage engine, you might have some streaming devices or services that should land in the engine.

Azure provides two out-of-the-box services:

  1. Azure IoT Hub
  2. Azure Event Hub

At first glance it seems like teh two services are doing the exact same thing - sending events through to other services in Azure. But there are some differences.

Azure IoT Hub

Azure IoT Hub is the cloud gateway that connects IoT devices to gather data and drive business insights and automation. In addition, IoT Hub includes features that enrich the relationship between your devices and your backend systems. In other words, the IoT hub can use bi-directional communication with your devices, if you need to send messages to the device and manage policies. This could be a message back to the device to update it over a download link or to directly update a message type or policy on the device. This also leverages the governance around your devices to ex. keep them at the same patch level etc.

Cloud-to-device communication also enables you to send cloud intelligence to your edge devices with Azure IoT Edge. The unique device-level identity provided by IoT Hub helps better secure your IoT solution from potential attacks.

The IoT hub also integrates directly with the IoT Edge from Azure giving almost endless possibilities for edge computing.

Azure Event Hub

Azure Event Hubs is the big data streaming service of Azure.

It’s designed for high throughput data streaming scenarios where customers may send billions of requests per day, and uses a partitioned consumer model to scale out your stream. Event Hubs is integrated into the big data and analytics services of Azure, including Azure Data Explorer, Databricks, Stream Analytics, ADLS, and HDInsight.

With features like Event Hubs Capture and Auto-Inflate, this service is designed to support your big data apps and solutions. Additionally, IoT Hub uses Event Hubs for its telemetry flow path, so your IoT solution also benefits from the tremendous power of Event Hubs.

What to choose when

If you have a lot of IoT devices for which you wish to keep a good governance and management in a centralized manner, then the IoT hub is the service to use. If you on the other hand are using IoT devices with high throughput, need easy integration with other services in Azure and don’t need a centralized management, then the Event Hub is a better match.

Reference architecture

Microsoft does a very good job in helping the end user to design their IoT footprint in Azure. Reference architectures like the one below, can be found on the Microsoft Learn documentation.

IoT reference architecture

IoT reference architecture

The central service of streaming

When working with IoT and EoT devices in Azure - one of the central elements in the data analytics and visualization is the Azure Data Explorer service. With this service you have state of the art technology to store and analyse data in real time - also leveraging the reporting functionality directly in Power BI with ultra fast Direct Query connections to the Azure Explorer service.

Azure Data Explorer reference architecture

Azure Data Explorer service

comments powered by Disqus