Category Archives: Azure
Create Azure Logic Apps
Login to https://portal.azure.com/ Search for logic app in search bar in top header. Click on Add. After clicking on Add, a new form opens. Select Subscription, here I selected Pay-As-You-Go. Select Resource group, here I created new Resource group with name AzureLogicAppLearning. Enter Logic App name in my case it is SendEmailOnAccountCreation. Select the location, here I selected Southeast Asia. Now, click Review + Create. After, clicking Create Deployment of app starts. Post deployment completion you will be redirected to App overview page.
Share Story :
Setting Up Notifications in Azure API management Service
`Read this Blog to understand how you can configure API Management notification settings for Users to receive notifications for new subscriptions
Share Story :
How to read files from SharePoint and store in Azure Blob Storage
This blog will help you resolve common errors that one might come across while integration between SharPoint and Azure Blob and will also guide you through how it is achieved.
Share Story :
How to Customize the API management new Developer Portal
Learn how you can customize the new Azure API management Developer Portal.
Share Story :
How to copy data from rest API to Azure data lake via Azure Data Factory
In this blog I am going to explain you how to copy data from rest API to Azure data lake using Azure Data Factory. Step 1: Create an azure storage account via azure portal using your credentials. Step 2: Once your storage creation done, create a container called “restapidata” that will going to use to store rest API data. Step 3: If you have rest API endpoint then its fine otherwise search dummy rest api example in google and copy one of the URL. Here I have taken employee endpoint (http://dummy.restapiexample.com/api/v1/employees) Step 4: Create an Azure Data Factory service in azure portal and create a pipeline. Step 5: Create a link service for rest API. Step 6: Create a link service for Azure data storage. Step 7: Create a dataset for rest API. Step 8: Create a dataset for Azure data lake storage. Step 9: Drag a copy activity in the pipeline and set a general property. Step 10: Set a source property of copy activity. You can also click on preview data to check whether data is populated or not. Step 11: Set a sink property of copy activity. Step 11: Set a mapping property of copy activity. Step 12: Debug the pipeline and check the output. Also click on details to check record proceed. I hope this will help you.
Share Story :
How to capture the detail level logs when Azure Data Factory Pipeline fails or success
In this blog I am going to explain you how to store detail level logs when Azure Data Factory Pipeline fails or success. First you need to create ADF_LogsRecording table using following SQL script. CREATE TABLE [dbo].[ADF_LogsRecording]( [LogsRecordingID] [int] IDENTITY(1,1) NOT NULL, [DataFactoryName] [nvarchar](200) NULL, [PipelineName] [nvarchar](200) NULL, [PipelineId] [nvarchar](200) NULL, [PipelineStartTime] [datetime] NULL, [ErrorCode] [nvarchar](1000) NULL, [ErrorDescription] [nvarchar](max) NULL, [ErrorLogTime] [datetime] NULL, [ActivityID] [nvarchar](100) NULL, [ActivityName] [nvarchar](200) NULL, [ActivityStartTime] [datetime] NULL, [ActivityEndTime] [datetime] NULL, [ActivityDuration] [time](7) NULL, [ActivityStatus] [nvarchar](100) NULL, [Itemcode] [nvarchar](50) NULL, [FrgnName] [nvarchar](100) NULL, [U_COR_BU_TXTS] [nvarchar](max) NULL, [U_COR_BU_TXTQ] [nvarchar](max) NULL, [CreatedOn] [datetime] NULL, CONSTRAINT [PK_ADF_LogsRecording] PRIMARY KEY CLUSTERED ( [LogsRecordingID] ASC )WITH (STATISTICS_NORECOMPUTE = OFF, IGNORE_DUP_KEY = OFF) ON [PRIMARY] ) ON [PRIMARY] TEXTIMAGE_ON [PRIMARY] GO ALTER TABLE [dbo].[ADF_LogsRecording] ADD CONSTRAINT [DF_CreatedOn] DEFAULT (getdate()) FOR [CreatedOn] GO Also create a store procedure suppose you are going to use in ADF Pipeline. CREATE PROCEDURE [dbo].[sp_ADF_LogsRecording_Insert] ( @DataFactoryName nvarchar(200), @PipelineName nvarchar(200), @PipelineId nvarchar(200), @PipelineStartTime datetime, @ErrorCode nvarchar(1000), @ErrorDescription nvarchar(max), @ErrorLogTime datetime, @ActivityID nvarchar(100), @ActivityName nvarchar(200), @ActivityStartTime datetime, @ActivityEndTime datetime, @ActivityDuration time, @ActivityStatus nvarchar(200), @Itemcode nvarchar(100), @FrgnName nvarchar(200), @U_COR_BU_TXTS nvarchar(max), @U_COR_BU_TXTQ nvarchar(max) ) AS BEGIN INSERT INTO ADF_LogsRecording ( DataFactoryName, PipelineName, PipelineId, PipelineStartTime, ErrorCode, ErrorDescription, ErrorLogTime, ActivityID, ActivityName, ActivityStartTime, ActivityEndTime, ActivityDuration, ActivityStatus, Itemcode, FrgnName, U_COR_BU_TXTS, U_COR_BU_TXTQ ) VALUES ( @DataFactoryName, @PipelineName, @PipelineId, @PipelineStartTime, @ErrorCode, @ErrorDescription, @ErrorLogTime, @ActivityID, @ActivityName, @ActivityStartTime, @ActivityEndTime, @ActivityDuration, @ActivityStatus, @Itemcode, @FrgnName, @U_COR_BU_TXTS, @U_COR_BU_TXTQ ) END We are also creating SQL store procedure to get all record suppose you want to either insert or update in destination. CREATE procedure [dbo].[SP_GETItem] AS BEGIN SELECT itemcode,FrgnName,U_COR_BU_TXTS,U_COR_BU_TXTQ FROM OITM WITH (NOLOCK) END GO CREATE procedure [dbo].[sp_GetItemByItemCode] ( @ItemCode nvarchar(200) ) AS BEGIN SELECT ItemCode,FrgnName,U_COR_BU_TXTS,U_COR_BU_TXTQ FROM OITM where ItemCode=@ItemCode END GO Now create a pipeline Step 1: Use Lookup activity to get all item. Step 2: ForEach activity should use to loop each itemcode wise, configure the settings of foreach activity Step 3: Add a Copy activity inside ForEach activity and set source properties. Set Sink properties as per below screenshot. Step 4: Add Store procedure activity for both Success and Failure of Copy activity. Step 5: Add store procedure activity for success and configure properties as per below screenshot. Step 6: Add store procedure activity for failure and configure properties as per below screenshot. Now when we run the pipeline, based on failure or success of pipeline record level logs were stored in table. I hope this will help you.
Share Story :
Integration of CSV file from ADLS to Azure SQL DB using ADF
Go through the blog to understand how to integrate a CSV file stored in your ADLS to Azure SQL DB.
Share Story :
Azure Logic App for creating an API for SQL DB.
Read this blog to understand how one can create logic App using Azure portal to create an API for SQL DB.
Share Story :
Triggering A Logic App
Read this blog to understand about Triggers in Logic App. Learn how you can work with Simple Timer Invoke and how to trigger Logic App containing a start Block as ‘When an HTTP request is received’
Share Story :
Develop D365 Finance, SCM and Retail Reports using Azure Data Lake Gen2
The BYOD feature for the Dynamics 365 for finance and Operations was release in 2016, it’s provide feature for the customer to bring their own Azure SQL Database, but the drawback through this was Entity Store only accessible for the inbuilt data warehouse only which means it is not accessible to outside D365. The new feature for the Data Lake Gen2 makes the Entity Store get Sync with Data Lake. Following are the steps to setup the environment. Let’s create the Storage Account for the Data Lake Gen2 Click on the Create Resource and search for Storage accounts and go to the Storage Account and click on Add. Choose the Subscription and Resource group, resource group is the container that holds the resource. Here, we are creating the new resource for our Data Lake. Make sure you select the Location same as your Power BI dataset environment and set the other option as shown below. Once storage account gets created, we can access it from the resource group open the storage account Now let’s take a copy of the connection string since it will be required later, the connection string can be access by going to setting under the storage account. Let’s create the Key vault resource to store the secret and create the secret for the connection string. Secret can be created by going to the key vault by going to the Key under the Settings tab, click on the generate the key. We are creating the secret for the connection string that we have copied earlier, set the value for the connection string as the key that we have copied and click on the create. Once we have created the app secret for the connection the next step is to authorize the user and resource to the request. Here we are going to register app for the authorization of the D365FO environment. To register the app, go to the Azure Active Directory and Select the app registration and click on the new registration. Fill the application name of your choice and redirect URI is set to WEB and select the D365 environment URI as shown below. Once we register the app the next step is to grant the API permission as a part of consent process. Grant all the permission that application should require. From Azure Key Vault select the Permission as user_impersonation which provide full access to Azure Key Vault service. The next step is to create client secret and make sure to note down the value of the app generated since we are going to use that in the D365FO environment. Next is to add the D365FO in access policy list and select the Key and Secret permission to Get and List from the drop down. Once we have added D365FO in access policy list the next step is to add the Application ID and Application Secret of Azure Key Vault in Data connection in D365FO environment, which can be access by going to Module > System administration > Setup > System parameters > Data connections tab. Note:- Make sure if data connection tab is not visible to you which means you are missing some configuration setting during environment setup, to make the data connection tab available we need to check the Value of CDSAMaster in SQL Studio Management Studio. Open SSMS go to the AxDW database and check the value in SYSFLIGHTING for CDSAMaster if not present then insert the value in table. SELECT * FROM SYSFLIGHTING /*To check the Flightname values*/ INSERT INTO SYSFLIGHTING VALUES (‘CDSAMaster’,1,12719367,5637144576,5637144589,1) /* CDSAMaster FlightName 1 Enabled 12719367 FlightserviceID 5637144576 Partition 5637144589 RecID 1 Recversion */ Before Update After Updating Once Data connections is added fill the Application ID, Application Secret, DNS name and connection secret name as shown below and make sure to enable the Data Lake Integration. Test the Azure Key Vault and Azure Storage. Note:- If you are getting error 401 which means Azure user in which data lake is hosted has no access to D365FO environment in that case you need to import the user in environment and assign role as Administrator. Create Reports using Azure Data Lake Gen2 Once done with above steps the next step is to configure the storage account, following are the requirement for the Power BI. The storage account must be created in the same AAD tenant as your Power BI tenant. The storage account must be created in the same region as your Power BI tenant. The storage account must have the Hierarchical Name Space feature enabled (Make sure to enable this at time of storage account creation) Power BI service must be granted a Reader role on the storage account. You must have a Global Administrator account, this account is required to connect and configure Power BI to store the dataflow definition, and data, in your Azure Data Lake Storage Gen2 account As we have created storage account previously lets, grant the Reader role to the storage account, in Azure portal Go to the Storage account > Access control > Role Assignments then click on Add role. Once we assign role to the Storage Account the next step is to Assign the directory level permission to the Storage that we have created. Here we are granting Power BI permission to the file system. We need to get the collection of IDs of the Power BI. Which can get by navigating to the AAD > Enterprise Application > All Application, copy the Object IDs of the Power BI Premium, Power BI Service and Power Query Online. For each Power BI Object that we have collected in previous steps grant the below access for each of the object. Once we granted the access to the storage the next step is to connect your Azure Data Lake Storage Gen 2 to Power BI Go to your Power BI service > Click on Admin portal navigate to Dataflow settings then Select the Connect your Azure Data Lake Storage Gen2 button. The following … Continue reading Develop D365 Finance, SCM and Retail Reports using Azure Data Lake Gen2