How to Debug a Pipeline in Azure Data Factory?

In this article, we will see how to debug the pipelines in Azure Data Factory.

The main purpose of using the debug option in Azure Data Factory, is to make it easier for us to test the functionality or the performance of the pipeline activities gradually during the development phase, before publishing the changes to the data factory, especially when the pipeline structure is complex and not easy to test it as one block.

Prerequisites

Azure Subscription.

Azure Data Factory.

You went through the previous articles and created at least one of the pipelines.

Debug a pipeline

1. Open the Azure Portal using your Internet Browser and search for your Azure Data Factory service.

2. In the Azure Data Factory, click Launch Studio to open the Azure Data Factory Studio:

3. In the Azure Data Factory Studio, click in the pencil icon to start debugging the pipeline:

4. In the Author window, click on one of the Pipelines that we created in the previous demos.

5. On the pipeline design window, click on the Debug button, taking into consideration that when we debug the pipeline, it will be executed completely and the ETL process will be performed:

6. When the pipeline debug starts, you can monitor the execution progress using the output window:

In the output window, you can find the pipeline Run ID that distinguishes the pipeline execution from other executions for the same pipeline and other pipelines, and the activity Run ID that distinguishes the activity execution from other executions for the same activity and other activities.

7. When the pipeline execution completed successfully, you will be able to see the Input information for the specified activity:

The output information for that activity:

And full details about the copy activity:

8. If any issue occurred during the pipeline activity execution, you can get a detailed error message that helps in troubleshooting the cause of the execution failure:

9. Azure Data Factory provides us also with the ability to check the resources consumed by debugging the pipeline, that helps in expecting the cost and working to reduce it by tuning the pipeline: