Run Command on Virtual machines using Azure Data Factory and Azure management API

If you are using Azure data factory and at some step, you want to run a script on your virtual machine Windows/Linux on Azure or Arc Enabled.

You can use Azure management APIs to achieve the same by invoking the Azure Run command feature.

What we are trying to achieve here is post-processing file a with ADF to password protect the source file on the server, overall architecture will look something like that :

Step 1

Create a web activity on Azure Data factory which will point to the Virtual machine management API and also the script inside the virtual machine (more info: https://docs.microsoft.com/en-us/rest/api/compute/virtual-machines :

• Create a Webactivity
• URL : https://management.azure.com/subscriptions/<subscrtiption-id>/resourceGroups/<resource-group-name>/providers/Microsoft.Compute/virtualMachines/<vm-name>/runCommand?api-version=2021-07-01
• Method : POST
• Body :

{             

“commandId”:”RunShellScript”,

“script”:[“bash <your-script-path>”]

}

• Authentication : Managed Identity
• Resource : https://management.azure.com/

Step 2

Provide access to the Managed Identity for the ADF on the Virtual machine :

–

• Go to Identity and access management (IAM)
• Assign access
• Select Managed Identity – > Select ADF Instance Name
• Access level Contributor

Step 3

Place the Python/Bash script for file encryption on the server.

Step 4

Assign Managed Identity access for the VM on Azure Key Vault

• Go to Identity and access management (IAM)
• Assign access
• Select Managed Identity – > Select Virtual machine Name
• Access level Contributor
• Add VM in KV Access policies

And you are set to call your script from the ADF instance and fetch the password from Azure Key Vault.