Event Hubs Availability and Consistency
Event Hubs is a streaming and event ingestion service on the Microsoft Azure Platform capable of receiving and processing millions of events per second. The service will allow you to process and analyze the massive amounts of data produced by your connected devices and applications.
In the tip adding an Event Hub, we discussed that you could specify the number of partitions. With partitions, you can improve the availability and parallelization for a single Event Hub. By specifying more than one partition, means that within the data center (Azure Region) where you provisioned the namespace and created the Event Hub, a partition resides on another server. Hence, with partitions, you increase availability. Moreover, with more partitions, you enable you to have more readers of your events – meaning more aggregated throughput.
Note that with partitions you need to think about the number during setup as you cannot change it afterwards. Furthermore, you might to considering the ordering too, in case you increase the number from one partition as some scenario’s in which you’ll use Event Hubs care about the order of incoming events. The CAP Theorem applies for Event Hubs once you multiple partitions with regards, to partition tolerance (continue processing even if one or more partitions fail), availability, and consistency.
When devices asynchronously send events to a partitioned event hub – the events will be distributed between the partitions automatically. Something that you do not have to worry about, and in case you build an Event producer, you need to use the SendAsync method. Secondly, when dealing with consistency, you need partition key in your events to enforce the reading order. You can choose the aggregate events first and use a sequence number for each, before processing them.
Consuming of events in the above diagram also depicts consumer groups, which is a view of an entire Event Hub. In receiving the data, each application (consumer) will be tied to a consumer group, and partitions can only be accessed through a consumer group. By default, an Event Hub can have up to 20 consumer groups (standard tier). And there can up to 5 concurrent readers on a partition per consumer group.
To conclude, when adding an Event Hub in your namespace, you have to think about partitioning keeping availability, and consistency in mind.