How to monitor Azure Data Factory?

In this article, we will see how to monitor the Azure Data Factory pipelines and metrics.

Prerequisites

Azure Subscription.

Azure Data Factory.

Scheduling the Pipeline Execution

1. Open the Azure Portal using your Internet Browser and search for your Azure Data Factory service.

2. In the Azure Data Factory, click Launch Studio to open the Azure Data Factory Studio:

3. In the Azure Data Factory Studio, click in the Monitor icon to start monitoring your Azure Data Factory:

4. In the Monitor window, you can check the execution status for the running pipelines and the execution result for the completed pipelines. This applies for both the pipelines that are triggered using a predefined schedule, or the pipelines that are executed using Debug option:

5. The Pipeline monitor window allows you also to filter for specific pipeline name, pipeline status or execution time, with the ability to export the execution logs as a CSV file:

6. In the Trigger runs monitor window, you can monitor and filter based on the trigger type, name or execution result too, for the pipelines that are executed based on a predefined trigger:

7. The rich Monitor window in Azure Data Factor provides you also with the ability to monitor the Integration Runtimes that are created in your Azure Data Factory:

8. As any Azure service, you will find a built-in monitoring dashboard in the Azure Data Factory Portal Overview page, that provides you with an overview insight about the service status and usage. This dashboard is built over the Azure Monitor service.

For Azure Data Factory, it provides us with number of pipelines, activities and triggers executed on a specific time. In addition, it will provide us with an overview of the Integration Runtime resources usage as below:

9. If you plan to perform a proactive monitoring for your Azure Data Factory pipelines, you can create alerts to receive a notification when a pipeline or activity failed, completed or succeeded:

10. If you click on the Metrics option under the Monitor window, it will move you to Azure Monitor service, that allows you to choose the custom metrics that you will use to monitor Azure Data Factory: