Tracking Azure Blob Storage Changes with Blob Change Feed
    • Dark
      Light
    • PDF

    Tracking Azure Blob Storage Changes with Blob Change Feed

    • Dark
      Light
    • PDF

    Article Summary

    #ServerlessTips - Azure Storage Account
    Author: Rahul Rai Microsoft MVP

    Blob change feed is a feature of Azure Blob Storage that provides transaction logs of all the changes that occur to the blobs and the blob metadata in your storage account. It allows you to track and react to changes in your blob data, such as adding, deleting, or modifying blobs. Blob change feed is helpful for data replication, auditing, backup, and disaster recovery scenarios.

    The Blob Change Feed provides a unified log of all the changes in Blob Storage, including additions, deletions, and modifications to blobs. Developers can use this information to build applications that can react to changes in Blob Storage, such as updating search indexes, triggering workflows, or generating notifications.

    The change feed is stored as blobs in a container named "$blobchangefeed" in the storage account. The retention period of these files can be controlled based on requirements. The change events are appended as records in the Apache Avro format, which provides rich data structures and is widely used in the Hadoop ecosystem, Stream Analytics, and Azure Data Factory. The logs can be processed incrementally or in full, and multiple client applications can read the change feed independently in parallel. Analytics applications such as Apache Drill or Apache Spark can consume the logs directly as Avro files, enabling low-cost and high-bandwidth processing without custom transformations.

    These instructions will help you enable the change feed on your Azure storage account:

    1. In the storage account, select the "Data Protection" option under "Data Management.”
    2. Select "Enable blob change feed" under "Tracking.” Optionally, you can specify the duration of the change feed.
    3. Click "Save" to turn on the feature.

    download 191

    Fig1.Enable change feed

    You can read the change feed records using the blob change feed processor library, available for .NET, Java, Python, and JavaScript. The library provides a convenient way to stream and process the change feed records in a scalable and reliable manner.

    You will need to install the following NuGet packages to process the change feed in your .NET applications:
    dotnet add package Azure.Storage.Blobs --prerelease
    dotnet add package Azure.Storage.Blobs.ChangeFeed --prerelease

    The following code snippet shows how to read the change feed records in .NET recursively:

    using Azure; 
    using Azure.Storage.Blobs; 
    using Azure.Storage.Blobs.ChangeFeed; 
    
    // Get a new blob service client. 
    var blobServiceClient = new BlobServiceClient("<connection-string>"); 
    
    // Get a new change feed client. 
    var changeFeedClient = blobServiceClient.GetChangeFeedClient(); 
    
    string? cursor = null; 
    while (true) 
    
    { 
    
        IAsyncEnumerator<Page<BlobChangeFeedEvent>> enumerator = changeFeedClient 
            .GetChangesAsync(cursor) 
            .AsPages() 
            .GetAsyncEnumerator(); 
    
     
    
        while (true) 
        { 
            var result = await enumerator.MoveNextAsync(); 
            
            if (result) 
            { 
                foreach (var changeFeedEvent in enumerator.Current.Values) 
                { 
                    var subject = changeFeedEvent.Subject; 
                    var eventType = changeFeedEvent.EventType.ToString(); 
    
                    Console.WriteLine("Subject: " + subject + "\n" + "Event Type: " + eventType + "\n"); 
                } 
    
                // Update cursor to resume reading the next changes.  
                cursor = enumerator.Current.ContinuationToken; 
            } 
            else 
            { 
                break; 
            } 
        } 
    
        // Changes are published to change feed every 60 seconds. 
        await Task.Delay(TimeSpan.FromSeconds(60)); 
    } 
    

    An example of the output from the demo application can be seen in the screenshot below:

    download 201

    Fig2.Output from the application

    Check the Microsoft documentation website for more information about Azure Blob storage's change feed support.

    MicrosoftTeams-image 503


    Was this article helpful?