Re-runnable Mounting and Unmounting between ADLS Gen 2 and Azure Databricks

Introduction In this blogpost, we will see the code of re-runnable mount and unmount point creations to access data files from Azure Data Lake Storage(ADLS) Gen 2 in Azure Databricks and for the scratch like things related to Resource Group, Key vault, ADLS Gen 2, Azure Databricks and App Registrations in Azure Active Directory and…

PART 2 – File Movement from SharePoint to Azure Data Lake Storage Gen 2

Introduction This a Part-2 of the series File Movement from SharePoint to Azure Data Lake Storage Gen 2 In this blogpost, we will see the how to do movement of files at folder level in automatic way from SharePoint to ADLS Gen 2 via Azure Logic Apps. The use case we are going to see here is, moving…

Checking ADLS Gen 2 accesses in Azure Databricks via Service principal

Introduction In this blogpost, we will see how easily we can test the Service Principal accesses like below for files and folders at ADLS gen 2 in dynamic way (within some secs) via re-runnable python script in notebook of Azure Databricks. We can also utilize the below code to test the working of Key Vaults…

Pandas Basic Commands

Introduction In this blogpost, we will see some of the Python's pandas library basic commands and its operations. For running below commands, here, I have used Azure Databricks Notebook with python language. Via magic commands(%python), we can use the same below commands under other language connected notebooks as well. Basic pandas commands 1. Importing pandas library…

File Movement from SharePoint to Azure Data Lake Storage Gen 2

Introduction In this blogpost, we will see the how to do file movements in automatic way from SharePoint to ADLS Gen 2 via Azure Logic Apps. The use case, we are going to see here is when a person/someone uploads file in SharePoint(Source), we need to automatically move those files in the required Storage Account-ADLS Gen 2(Sink/Target)…

Service Principal Creation for Accessing ADLS Gen 2 in Azure Databricks

Service Principal Creation in Azure to access ADLS Gen 2 in Azure Databricks

T-SQL Queries Vs Spark SQL Queries

Introduction In this blog post, we will see some of the Transact SQL (T-SQL) Queries and its equivalents in Spark SQL with examples. some T-SQL vs Spark SQL Keyword Differences T-SQL Queries The below input as well as output taken by running T-SQL Queries in SSMS via Azure Synapse Analytics/Dedicated SQL Pool, the same we…

Question and Answer #5

Question – Out of below 2 servers, We need to find which one is 'Microsoft Azure Synapse Analytics'? Other might be 'SQL Database'.1️⃣ "s1.database.windows.net"2️⃣ "s2.database.windows.net" How to find? 💡 For "Dedicated SQL pool (formerly SQL DW)" , the endpoint would be same as SQL Database ".database.windows.net" [Still pricing tier is different: cDWU vs DTU/vCore]Under "Azure Synapse Workspace"…

Lift and Shift SSIS Package in ADF

In this blogpost, we are going to see how the SSIS package runs in Azure Data Factory at high level with an example. Here, we will create Sample SSIS Package first and then create ADF instance with Azure-SSIS Integration Runtime and using Execute SSIS package activity within Pipeline. Prerequisite Active Azure Subscription. If you don’t have,…

Bringing Folder Structure via Azure Data Factory

In this blogpost, we would see how to create Folder Structure in dynamic way as the process via Azure Data Factory with some sample files those are having different file formats. Here, We are going to make those file formats as dynamic folders and within those, we would also going to make them as year-->month-->date folder…