Azure Data lake Gen 2 · Azure Data Platform · Azure Databricks · Azure Services · Data Engineering · Python

Re-runnable Mounting and Unmounting between ADLS Gen 2 and Azure Databricks

Introduction

In this blogpost, we will see the code of re-runnable mount and unmount point creations to access data files from Azure Data Lake Storage(ADLS) Gen 2 in Azure Databricks and for the scratch like things related to Resource Group, Key vault, ADLS Gen 2, Azure Databricks and App Registrations in Azure Active Directory and IAM Access Control – check below two links.

Prerequisite

Please Check above two links for knowing the creation of service principal as well as how to check the related required accesses whether we have or not, in simple way and reach GitHub for Azure Databricks Key vault creating secret Scope.

Re-runnable Mount point creation

Provide the below input parameter values

  • adlsContainerName
  • adlsStorageAccountName
  • adlsFolderName, that we want to test the accesses
  • MountPointMain that we can provide the naming that we desired to keep
  • ScopeName, the name of the scope provided in Azure Databricks while linking Key Vault
  • ServicePrincipalID, the respective Key Vault Name
  • ServicePrincipalKey, the respective Key Vault Name
  • TenantID, the respective Key Vault Name
For creating required StoragePointMain and Endpoint

By using the above inputs, we can dynamically create StoragePointMain as well as endpoint aka Directory.

Re-runnable Mount Point Creation

By using above simple if else config, we can easily able to create re-runnable and touchless mounting between ADLS Gen 2 and Azure Databricks.

To Test the Mount point creation

To test connection as well as listing files in folder
To test and cross-check at data level

Re-runnable UnMounting

We would be in the same session, so local variables above set via inputs would already be holding the required values, here MountPointMain. so there is no separate inputs required for unmounting.

Re-runnable Unmounting

If you are interest to re-use or practice the above python script/code, visit Arulmouzhi/AzureStuffs Github repos.

Recent Related Posts

Conclusion

Thus, in this blogpost, we saw how easily we can able to create/use re-runnable code and do both mounting as well as unmounting via Service Principal and tested the accesses at the files, folders and data level between ADLS gen 2 using re-runnable python script in notebook of Azure Databricks.

Follow Blog and Show your Support for many more interesting upcoming Posts!

Advertisement

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s