Azure Data lake Gen 2 · Azure Data Platform · Azure Logic Apps · Azure Services · Data Engineering · Real Scenario Logic's · SharePoint

PART 2 – File Movement from SharePoint to Azure Data Lake Storage Gen 2

Introduction

This a Part-2 of the series File Movement from SharePoint to Azure Data Lake Storage Gen 2

In this blogpost, we will see the how to do movement of files at folder level in automatic way from SharePoint to ADLS Gen 2 via Azure Logic Apps. The use case we are going to see here is, moving all the files within the Folder from SharePoint(Source) to the required Storage Account-ADLS Gen 2(Sink/Target) /container/folder/ with the same file formats and naming’s, at some user defined update frequency/regular interval etc., There may be N possible ways to do the same, here we will see how to do this easily within 4 steps via Azure Logic Apps from Azure Portal.

Prerequisite

  • Specific SharePoint Folder/File level Upload/Read/Edit/Manage kind of Accesses.
  • Active Azure Subscription. If you don’t have, create a free account.
  • Subscription Level Contributor access or Owner access with basic understandings of using Azure and its components and knowing to create Simple blank Azure Logic App. we can check our Access in Subscriptions–>Access Control (IAM)–>View my Access. (like below)

SharePoint

SharePoint with Sample Path

ADLS Gen 2

ADLS Gen 2 in MS-Azure Storage Explorer with Sample Path

Azure Logic Apps

Create a Simple Blank Logic App and follow the below detailed step by step flows.

Azure Logic App with 4 steps (Brief here & Detailed Steps below mentioned)

Detailed Step by Step Flows:

1. When a HTTP request is received (or) Recurrence kind of Schedule

Type and choose ‘Request
Choose Triggers and choose the above highlighted triggerWhen a HTTP request is received
Select Add new parameter and select Method as GET. Then Click as Save. After Save, URL will be generated! (HTTP POST URL). To view this http post url, after save, refresh the page.

The above trigger named ‘When a HTTP request is received‘ is used for user defined click or with any other orchestrators like ADF kind of purposes as Web Activity with hereby generated HTTP POST URL. The Other Alternative for this & for regular time update frequency, we can use Recurrence kind of Schedule.

Type and choose ‘Schedule
Choose Triggers and choose the above highlighted triggerRecurrence
In Recurrence, we can able to set the Interval as well as Frequency like second, minute, hour, day, week, month. We can also choose other optional parameters like Start time and Time Zone at Add new parameter.

2. List Folder

Type and choose ‘SharePoint
Choose Actions and choose the above highlighted actionList Folder
Under Parameters TAB, provide Site Address, File Identifier where the folder with files reside; create/change Connection if required

3. Get file content

Choose the SharePoint Connector again for this operation just like how we done it for List Folders action.

Choose Actions and choose the above highlighted actionGet file content
Under Parameters TAB, provide Site Address, File Identifier (Id in suggesting expression list) & Keep the rest as default

once we select Id in suggesting expression list for File Identifier, the LA designer is itself knows that we may actually having multiple files in the folder, so it would pack up Get file content within a For each loop.

4. Create blob (V2) [within For each loop]

Type and choose Azure Blob Storage
Create or Change Connection if required

Default Authentication Type is Access key. If so, we need to provide both Azure Storage Account name as well as Azure Storage Account’s respective Access Key. We can also use Azure AD Integrated option as Authentication type.

Under Parameters TAB, provide Storage account name, Folder path, Blob name (DisplayName in suggesting expression list) and Blob Content (File Content in suggesting expression list) & Keep the rest as default

Unit Testing

Unit Testing by Placing File in SharePoint’s sample path and Checking the moved file in ADLS Gen 2’s sample path and Cross-Check the Run History in Azure Logic App, here mentioned 1 file as sample where as multiple files inside a folder is also tested-works fine!

Additional Notes – which we will see in upcoming blogposts

We can also ‘Add an action‘s like below, to make it much more effective as well as automated:

  1. Adding ‘Create a pipeline run‘ to run the ADF Pipeline automatically(Once ADLS receives file from SharePoint).
  2. Adding ‘Send an email (V2)‘ to automate mailings that we will receive once the above steps are completed.

Summary

Thus, in this blogpost we saw regarding the PART-2 of file movement in automatic way from SharePoint (Source) to ADLS Gen 2 (Target) with the help of Azure Logic Apps in easy and simple steps, just 4 steps.

If you really like this blogpost, Please do Like, Share & Follow Blog and Show your Support for many more interesting upcoming Posts!

Advertisement

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s