There is a growing need to process and analyze real-time data from a wide variety of sources so that organizations can respond quickly and effectively to changes. With its Real-Time Intelligence capabilities, Microsoft Fabric offers a solution to address these challenges. Two essential components within Fabric are Evenstream and Eventhouse. In this blog, we’ll take a closer look at them.
Eventhouse is a solution within Microsoft Fabric specifically designed for real-time analysis and exploration of large amounts of data. An Eventhouse serves as a workspace where multiple databases can be managed, which optimizes management. The database type within Eventhouse is a KQL database. A KQL database in Fabric is a log-based analytical database that runs on the Kusto Engine. It is used to quickly process and analyze large volumes of data using Kusto Query Language (KQL).
Fabric-Eventstream: The Link to Real-Time Data
Fabric Eventstreams enable you to integrate, transform, and forward real-time events in Fabric to various destinations without the need for code. Fabric event streams can be viewed as a data router that can continuously receive, process, and forward events. In addition, Fabric event streams support Apache Kafka endpoints. Apache Kafka is a popular open-source event streaming solution used for real-time data processing. In Microsoft Fabric, Kafka endpoints can be used within Fabric event streams to send and receive real-time data.
How does it work?
We'll start by creating a new Eventhouse in Ms Fabric. To do this, you'll need a Fabric Capacity license.
After entering a name, an Eventhouse is created in the Fabric workspace, and an empty KQL database is also automatically created.
Now we can create an Eventstream.
After we've named the Evenstream, we'll see the screen below and can select a data source.
In this example, I select “External Sources” and, on the next screen, "Events for Fabric Workspace Items." This ensures a constant stream of logs whenever Fabric items are viewed, created, or modified in the relevant workspace.
On the next screen, all I have to do is add my target to the event stream, where I select the Eventhouse I created earlier and the corresponding KQL database. As soon as I publish this stream and enter the table name I want to use, it becomes active. Optionally, I also had the option to add any transformations.
When we reopen the KQL database, we can see that our table is updated almost immediately with the Fabric logs.
Real-time dashboard or Power BI report?
Using the KQL database, you can create a real-time dashboard (not to be confused with a Power BI dashboard). This type of dashboard is similar to the dashboards you can create in Azure Data Explorer, and each tile corresponds to a specific KQL query. It also offers the ability to set up automatic refreshes and define alerts.
Is it possible to create a Power BI report based on the KQL database? Absolutely—I can also choose to analyze the data from my KQL database using Power BI. So which tool should you choose? Opt for a real-time dashboard when you need immediate insights and instant responses to data changes. Choose Power BI when you need comprehensive data visualization, advanced analytics, and interactive reports.
If you choose Power BI, a semantic model is automatically created in the background, which I can use to build my report. When the semantic model is opened in Tabular Editor, we see that the source is AzureDataExplorer, with a link to the KQL database I created. The partition is DirectQuery, which makes sense, of course, if we want to view real-time data. In Tabular Editor, I can then further customize the model, such as by adding a measure.
OneLake Synchronization
The KQL database provides the ability to make the data available in OneLake. This replication can be configured to be virtually real-time. The advantage is that you then have the data in a uniform Data Lake table structure and are not dependent on the specific Kusto Query Language. In addition, OneLake is easier to integrate with other Fabric engines such as Power BI, Lakehouse, and Notebooks.
Conclusion
Real-Time Event Streaming is a valuable feature in Microsoft Fabric when an immediate response is required in response to data changes that meet specific triggers. Consider, for example, IoT devices that collect real-time data from sensors. Another example is financial transactions monitored in real time via SQL CDC, where potentially suspicious transactions trigger further analysis. Or, if you’re an e-commerce company that wants to analyze customer behavior in real time to make personalized offers, a REST API that collects customer interactions and purchases in real time could serve as the source for an event stream.
