Subscribe to queues when working with Event Grid

When working in an event-driven solution in Azure, you’re probably using Azure Event Grid with Custom Topics to notify your system whenever something happens. I know I am!

Event Grid is a great central system built to handle many events coming from Azure and your cloud solution. To receive the events, you need to set up a subscription to a topic. Don’t confuse these with Service Bus Topics, which is something completely different.

Subscriptions can be set up for Storage Queues, Service Bus Queues/Topics, Azure Functions, Webhooks, Event Hub, and via an Azure Relay Hybrid Connection.

When handling some event from within your solution, it is often necessary to execute some business logic. Therefore, you may be inclined to create a subscription to an Azure Function and handle the required logic in this Azure Function.

I completely understand this mindset, and it is something I’ve also done in the past. But after having deployed this in Production, our team quickly discovered why this isn’t a good idea.

If an event is published to Event Grid, it will send this event (push) to every subscriber of the said event. This is great until your event handler can’t handle the event. In the case of an Azure Function, it might be down for some reason due to a transient error, or maybe an unhandled exception occurs in your Function. Either way, the event might not get delivered, and if you’ve configured a dead letter for the subscription, it’ll be dropped over there (in a Storage container).

Getting these dead letter events out of the Storage Account is a burden and has cost us quite a bit of time to get everything working again.

We needed a more reliable service and seeing Storage Queues are very cheap and have a very high uptime, we opted for this. Setting up a subscription to Storage Queues is like setting one up for an Azure Function.

Now that all events are being pushed towards the Storage Queue, an Azure Function can pick up the events from the queue and handle them accordingly.

When the Azure Function runtime isn’t running correctly for some reason, events aren’t lost but stored on the queue. If there’s an (unhandled) exception in our Azure Function, the event will be moved to the poison queue for further investigation.

I can tell you from experience; it’s a lot easier to handle messages/events on the poison queue compared to the events stored in a blob container.

Are there downsides to this setup?

Sure, there are, but the maintainability & consistency of your platform is much better. The biggest downside, in my opinion, is the additional service you have in this setup. But keep in mind, you can set up multiple Storage Queues in a single Storage Account. I think it’s a good idea to have a single Storage Account inside your ‘domain’ dedicated to hosting these Event Grid subscription queues.

You might wonder: Why not use Service Bus Queues/Topics instead?

My answer would be: You probably don’t need such a heavy-weight service for handling just events. Service Bus offers many great features, but you probably don’t need any of them for event handling. If you do, please use it instead.

The pattern remains the same, just a different implementation of the event handler.