What is Azure Databricks?

Azure Databricks is a data analytics platform that provides powerful computing capability, and the power comes from the Apache Spark cluster. In addition, Azure Databricks provides a collaborative platform for data engineers to share the clusters and workspaces, which yields higher productivity. Azure Databricks plays a major role in Azure Synapse, Data Lake, Azure Data Factory, etc., in the modern data warehouse architecture and integrates well with these resources.

Data engineers and data architects work together with data and develop the data pipeline for data ingestion with data processing. All data engineers work in a sandbox environment, and when they have verified the data ingestion process, the data pipeline is ready to be moved to Dev/Staging and Production.

Manually moving the data pipeline to staging/production environments via Azure portal will potentially introduce the difference in environments and add a tedious task to repeat manual processes in multiple environments. Automated deployment with service principal credentials is the only solution to move all your work to higher environments. There will be no privilege to configure via the Azure portal as a user. As data engineers complete the data pipeline, Cloud automation engineers will use IaC (Infrastructure as Code) to deploy all Azure resources and configure them via the automation pipeline. That includes all data related to Azure resources and Azure Databricks.

Data engineers work in Databricks with their user account, and it works very well integrating Azure Databricks with Azure key vault using key vault secret scope. All the secrets are persisted in key vault, and Databricks can get the secret value directly via linked service. Databricks uses user credentials to go against Keyvault to get the secret values. This does not work with service principal (SPN) access from Azure Databricks to the key vault. This functionality is requested but not yet there as per this GitHub issue.

JOIN OUR TEAM
Passionate about data? Check out our open data careers and apply to join our quickly growing team today!

Let’s Look at a Scenario

The data team has given automation engineers two requirements:

  • Deploy an Azure Databricks, a cluster, a dbc archive file which contains multiple notebooks in a single compressed file (for more information on dbc file, read here), secret scope, and trigger a post-deployment script.
  • Create a key vault secret scope local to Azure Databricks so the data ingestion process will have secret scope local to Databricks.

Azure Databricks is an Azure native resource, but any configurations within that workspace is not native to Azure. Azure Databricks can be deployed with Hashicorp Terraform code. For Databricks workspace-related artifacts, the Databricks provider needs to be added. For creating a cluster, use this implementation. If you are only uploading a single notebook file for creating a notebook, then use Terraform implementation like this. If not, there is an example below to use Databricks CLI to upload multiple notebook files as a single dbc archive file. The link to my GitHub repo for complete code is at the end of this blog post.

Terraform implementation

terraform {
  required_providers {
    azurerm = "~> 2.78.0"
    azuread = "~> 1.6.0"
    databricks = {
      source = "databrickslabs/databricks"
      version = "0.3.7"
    }
  }

  backend "azurerm" {
    resource_group_name  = "tf_backend_rg"
    storage_account_name = "tfbkndsapoc"
    container_name       = "tfstcont"
    key                  = "data-pipe.tfstate"
  }
}

provider "azurerm" {
  features {}
}

provider "azuread" {
}

data "azurerm_client_config" "current" {
}

// Create Resource Group
resource "azurerm_resource_group" "rgroup" {
  name     = var.resource_group_name
  location = var.location
}

// Create Databricks
resource "azurerm_databricks_workspace" "databricks" {
  name                          = var.databricks_name
  location                      = azurerm_resource_group.rgroup.location
  resource_group_name           = azurerm_resource_group.rgroup.name
  sku                           = "premium"
}

// Databricks Provider
provider "databricks" {
  azure_workspace_resource_id = azurerm_databricks_workspace.databricks.id
  azure_client_id             = var.client_id
  azure_client_secret         = var.client_secret
  azure_tenant_id             = var.tenant_id
}

resource "databricks_cluster" "databricks_cluster" {
  depends_on              = [azurerm_databricks_workspace.databricks]
  cluster_name            = var.databricks_cluster_name
  spark_version           = "8.2.x-scala2.12"
  node_type_id            = "Standard_DS3_v2"
  driver_node_type_id     = "Standard_DS3_v2"
  autotermination_minutes = 15
  num_workers             = 5
  spark_env_vars          = {
    "PYSPARK_PYTHON" : "/databricks/python3/bin/python3"
  }
  spark_conf = {
    "spark.databricks.cluster.profile" : "serverless",
    "spark.databricks.repl.allowedLanguages": "sql,python,r"
  }
  custom_tags = {
    "ResourceClass" = "Serverless"
  }
}

GitHub Actions workflow with Databricks CLI implementation

deploydatabricksartifacts:
    needs: [terraform]
    name: 'Databricks Artifacts Deployment'
    runs-on: ubuntu-latest

    steps:
    - uses: actions/checkout@v2.3.4
   
    - name: Set up Python 3.0
      uses: actions/setup-python@v2
      with:
        python-version: 3.0

    - name: Install dependencies
      run: |
        python -m pip install --upgrade pip

    - name: Download Databricks CLI
      id: databricks_cli
      shell: pwsh
      run: |
        pip install databricks-cli
        pip install databricks-cli --upgrade

    - name: Azure Login
      uses: azure/login@v1
      with:
        creds: ${{ secrets.AZURE_CREDENTIALS }}
   
    - name: Databricks management
      id: api_call_databricks_manage
      shell: bash
      run: |
        # Set DataBricks AAD token env
        export DATABRICKS_AAD_TOKEN=$(curl -X GET -d "grant_type=client_credentials&client_id=${{ env.ARM_CLIENT_ID }}&resource=2ff814a6-3304-4ab8-85cb-cd0e6f879c1d&client_secret=${{ env.ARM_CLIENT_SECRET }}" https://login.microsoftonline.com/${{ env.ARM_TENANT_ID }}/oauth2/token | jq -r ".access_token")

        # Log into Databricks with SPN
        databricks_workspace_url="https://${{ steps.get_databricks_url.outputs.DATABRICKS_URL }}/?o=${{ steps.get_databricks_url.outputs.DATABRICKS_ID }}"
        databricks configure --aad-token --host $databricks_workspace_url

        # Check if workspace notebook already exists
        export DB_WKSP=$(databricks workspace ls /${{ env.TF_VAR_databricks_notebook_name }})
        if [[ "$DB_WKSP" != *"RESOURCE_DOES_NOT_EXIST"* ]];
        then
          databricks workspace delete /${{ env.TF_VAR_databricks_notebook_name }} -r
        fi

        # Import DBC archive to Databricks Workspace
        databricks workspace import Databricks/${{ env.databricks_dbc_name }} /${{ env.TF_VAR_databricks_notebook_name }} -f DBC -l PYTHON

While the above example shows how to leverage Databricks CLI to do automation operations within Databricks, Terraform also provides richer capabilities with Databricks providers. Here is an example of how to add ‘service principal’ to Databricks ‘admins’ group in workspace using Terraform. This is essential for Databricks API to work when connecting as a service principal.

Databricks Creating Cluster
Databricks cluster deployed via Terraform
Jobs Deployed via Terraform
No Jobs have been deployed via Terraform
Databricks CLI
 Job deployed using Databricks CLI in GitHub Actions workflow
Deployment with Databricks
Job triggered via Databricks CLI in GitHub Actions workflow

Not just Terraform and Databricks CLI, but also Databricks API provides similar options to access Databricks artifacts and manage them. For example, to access the clusters in the Databricks:

  • To access clusters, first, authenticate if you are a workspace user via automation or using service principal.
  • If your service principal is already part of the workspaces admins group, use this API to get the clusters list.
  • If the service principal (SPN) is not part of the workspace, use this API that uses access and management tokens.
  • If you would rather add the service principal to Databricks admins workspace group, use this API (same as Terraform option above to add the SPN).

The secret scope in Databricks can be created using Terraform or using Databricks CLI or using Databricks API!

Databricks with other Azure resources have pretty good documentation, and for automating deployments, these options are essential: learn and use the best option that suits the needs!

Here is the link to my GitHub repo for complete code on using Terraform, Databricks CLI in GitHub Actions! In addition, you can find a bonus learning how to deploy synapse, ADLS, etc., as part of modern data warehouse deployment, which I will cover in my next blog post.

Until then, happy automating!

Before you start deep dive for implementing DevSecOps in this blog post, please review the fundamentals of DevSecOps in my first blog post. It will help understand the ‘Sec’ in DevSecOps and get up to speed on various security tools for implementing DevSecOps in your CI/CD pipeline. Although there are many code repositories tools with CI/CD built-in, this blog walks through GitHub and its security scanning tools for DevSecOps implementations.

This blog post provides a GitHub repo for you to fork and try it on your own. The repo has an out of the box .NET core application, docker file to containerize the application. GitHub actions workflow YML code to build and deploy the containerized application to Azure. Further steps below will help understand introducing code scanning and container scanning tools. While I have used CodeQL for code scanning and Anchore for container scanning, can easily be replaced with other security scanning tools.

This repo/blog does not have all DevSecOps pipeline security features but integrates code scanning and container scanning tools. It shows how to get started on a GitHub Actions workflow and add these tools for security scanning. DevSecOps fundamentals are to understand and integrate these security tools in your pipeline. Source code is the single source of truth, and adding CI/CD pipeline automation is the first step. The security tools integrated into the pipeline scan your source code, scan the dependency packages/software, and will be added to your code.

GitHub Actions CI/CD

As we saw in the previous blog post, we will use the below high-level DevSecOps CI/CD pipeline workflow. I am not covering ‘artifactory’ functionality in this blog post and covering it in the next blog post!

Pipeline Workflow

Pic: Simple DevSecOps pipeline workflow.

A .NET core application is then added to the GitHub repo. The docker file is added to containerize the .NET Core application using docker build. A GitHub actions workflow is added to run the CI/CD pipeline. The actions will contain code for build/deploy and integrating security tools within the pipeline. The GitHub Actions workflow in the repo has the following steps:

  1. Starts the workflow manually which helps with debugging/testing. It can be changed to run when a branch is pushed or to main, and when a Pull Request is submitted.
  2. Azure related variables are declared as workflow environment variables
  3. Build the .NET code
  4. Parallelly CodeQL scans the code. CodeQL action can be added to the main pipeline YML.
  5. Anchore container scanning before the container deployed to ACR
  6. Log in to Azure
  7. Create an Azure Container Registry (ACR) and log into ACR.
  8. Docker build and push the application container to ACR
  9. GitHub container scan, scans the container in ACR
  10. Create a new Azure Container Instance to host the application running in a container
  11. Log out from Azure after completing the pipeline workflow

Now log into the Application is successfully running in Azure. Log in & verify your application running.

To learn more about GitHub Actions workflow and how to leverage your projects, follow the links:

CodeQL

Code scanning is essential even before security scanning is applied in the pipeline, as this is the first defense line. GitHub blog, GitHub code scanning is a developer-first, GitHub-native approach to find security vulnerabilities before they reach production easily. We’re thrilled to announce the general availability of code scanning. You can enable it on your public repository today!

The code scanning is completed using CodeQL, which is GitHub native approach to scan the code to identify the security vulnerabilities while the code is still being built. Other third-party code scanning workflows can also be added, as you see from the below screenshots. To set CodeQL code scanning, click on ‘Security’ from your repo. Then select ‘Code Scanning’ to add CodeQL workflow.

Add Code Scanning to GitHub repo

Pic: Add Code Scanning to your code in GitHub repo.Add CodeQL workflow

Pic: Add CodeQL workflow to your code. Third-party code scanning can be added as well.

Results of CodeQL Code Scanning

Pic: CodeQL Code scanning results.

CodeQL code scanning workflow runs parallel to the main workflow. It can be changed to run at the beginning of the main workflow also. CodeQL identifies what type of language is in the repo and runs the scanning accordingly. As this repo contains C# and JavaScript, code scanning is done for those languages. If there are significant errors/vulnerabilities found, then the code scanning workflow will fail. But for general warnings, alerts are available for review and fix the warnings.

View Code Scanning Alerts

Pic: View the code scanning alerts.

Results of Code Scanning

Pic: Code scanning results.

CodeQL scanning warnings can be analyzed and decide whether it is needed to fix the errors/warning and if it is safe to ignore. As this is the out of box .NET application, there are not many alerts/warnings. But if its production code and especially any migration-related codes might have more security vulnerabilities that need a fix before proceeding in the CI pipeline.

GitHub Container Scanning

This container scanning is native GitHub action to scan Docker containers in the CI pipeline. This identifies any vulnerabilities before the container is published to an application instance. More information can be found here, and this action uses Trivy and Dockle for running container scans on these images. They follow CIS Container Benchmarks for a baseline to secure the container images.

GitHub container Scan

Pic: GitHub Container Scan is added to the main workflow.

Having this action in the repo and running the main workflow failed due to these vulnerabilities findings. Any of the vulnerability findings can be ignored if it is in acceptable criteria. They can be added to ‘allowedlist.yaml’ file. The container action will then ignore those vulnerabilities. For more information, read here. Container scanning in GitHub is robust in scanning for vulnerabilities before the containers are deployed to the customer environment. But a closer review of these vulnerabilities needs to be done so appropriate scanning is performed in the repo. An analysis will yield better evaluations to find if the findings are indeed from code or from any dependencies. The container scan provides scantizer results like this.

Finding Vulnerabilities

Pic: Vulnerability findings.

CIS Container baseline violations warning & info

Pic: CIS Container baseline violations warning & info.

Added Vulnerabilities to ignore list

Pic: Added vulnerabilities to ignore list.

Anchore Container Scanning

Anchore is an open-source container scanning tool added to the GitHub Actions pipeline. More than one container scanning actions can be added to a repo workflow—more information on how Anchore container scanning works.

Anchore Container Scanning Action added to main pipeline workflow

Pic: Anchore container scanning action added to main pipeline workflow.

Anchore container scan also identified the same vulnerabilities

Pic: Anchore container scan also identified the same vulnerabilities as GitHub container scan.

According to NVD (National Vulnerability Database) in NIST, CVE (Common Vulnerabilities Exposures)
CVE defines vulnerability as:
“A weakness in the computational logic (e.g., code) found in software and hardware components that, when exploited, results in a negative impact to confidentiality, integrity, or availability. Mitigation of the vulnerabilities in this context typically involves coding changes, but could also include specification changes or even specification deprecations (e.g., removal of affected protocols or functionality in their entirety).”
All vulnerabilities in the NVD have been assigned a CVE identifier and thus, abide by this definition.

I took one of the CVE identified in both GitHub & Anchore container scan actions, CVE-2019-3843. According to NVD database in NIST, this is related to systems service-specific running in the container instances. As this repo is out of the box .NET core application, no additional code has been added. Further analysis must be done to understand the CVE depth to mitigate or override.

Current Description Title Link

Pic: https://nvd.nist.gov/vuln/detail/CVE-2019-3843#vulnCurrentDescriptionTitle

The vulnerabilities could be coming from the dependency container image used for building the docker image for the application. Hence it is essential to make sure dependencies are scanned as well.

Application Container Deployed in Azure

If all scanning goes well, then the pipeline is ready to deploy the application to Azure. Here the Azure CLI is used in the pipeline to deploy the containerized application to Azure Container Instance (ACI)!

Resource group for Azure resources

Pic: Resource group for Azure resources.

Azure Resources ACR & ACI

Pic: Azure resources (ACR & ACI).

Image of Container in ACR

Pic: Container in ACR.

Container deployed to Azure Container Instance (ACI)

Pic: Container deployed to Azure Container Instance (ACI)

Application up and running in Azure

Pic: Application up and running in Azure.

I hope the DevSecOps implementation walk-through with GitHub repo to try it out on your own adds value in understanding ‘Sec’ in DevSecOps and how to protect the codebase from vulnerabilities before deployed into production. While the blog post walked through GitHub CI/CD, all the other DevOps tools such as Azure DevOps, GitLab, Jenkins, etc. provide similar security tools implementations in their CI/CD pipeline. So, make sure to integrate security tools and shift the security to the left in whichever DevOps tool your organization decides to implement for your CI/CD pipelines. Follow up back here for the next blog post in this series for artifactory, running security tools from containers, the importance of containers, Kubernetes, and how to consume hardened container images from IronBank offered by DoD DevSecOps!

DevOps implements a Continuous Integration/Continuous Delivery (CI/CD) process. When multiple team members work in the same codebase, anyone’s update could break the integrated code. So, Continuous Integration is to trigger a build pipeline whenever a code update is pushed. The build pipeline will fail if the newly updated code is incompatible with the existing codebase if there are any conflicts. The codebase might work well within a single developer environment, but in a build pipeline where all configurations and dependencies are expected to be in place can fail. Continuous Delivery speeds up the deployment process. The release pipeline helps to deploy the same code base to multiple environments based on configurations. This helps to support code to be deployed in all environments without many manual changes.

Having an approval process helps peer code reviews, identifies potential issues, and any security flaws ahead of time. The current production applications are very distributed and complex. Whether it is an on-premise or cloud-based solution, missing a dependency or proper configurations could cost significant risk in deployments. DevOps helps to maintain the same code base for repeatable deployment in many environments with just configuration changes. DevOps avoids manually building the deployment packages and handing over to the operations team who would not have insights on what is being deployed. If an error occurs during deployment or post-deployment, then the development team jumps in at that time, which is time-consuming. This will cost in production timeline and end up with some unhappy customers also!
DevOps ImagePicture credit: DoD DevOps

Popular DevOps Tools

Follow here to learn more about DevOps practices from other AIS bloggers!

Why not just “DevOps”?

DevOps is fundamental for any organization’s build and deployment process with seamless CI/CD integration. Then, what is ‘DevSecOps’ and why is ‘Sec’ added between Dev and Ops. The ‘Sec’ in DevSecOps is ‘Security.‘ Though it’s added in between, security implementation should start from Development and continue in Operations. As development and deployment packages add many dependencies from both internal and external, this could introduce vulnerabilities. It could cost severe issues in production if not identified earlier in the build pipeline. Code scans help identify possible weaknesses in code implementations. But for any cybersecurity-related vulnerabilities, only specific tools at different stages of the pipeline must be used to identify as early as possible. Adding security scanning earlier in the pipeline and automating are essential for DevSecOps.

DevSecOps Software Lifecycle

Picture Credit: DoD DevSecOps

DevSecOps is not a tool or pattern but a practice and can be enhanced by adding appropriate tools. It is a process in securing the build and deployment by using several security tools by shifting security to the left. These security tools help to identify vulnerabilities that the code could have introduced, recommend possible solutions to fix those issues, and in some instances, the tools can mitigate some of those issues as well. This is to use the ‘fail fast’ method to identify vulnerabilities earlier in the build pipeline. As more applications moved into the cloud, it is highly imperative to follow Cloud Native Computing Foundation (CNCF) certified tools and implement security benchmarks that provided CIS benchmarks. DevSecOps avoids manual changes once the code is in the pipeline, deployed, and deployed. The codebase will be a single source of truth and should not be manipulated at any point.

Adding scanning tools for security and vulnerabilities helps to mitigate any flaws introduced in code and operations. Many open-source tools provide these functionalities. Enabling logging, continuous monitoring, alerting processes, and any self-fix for faster remediation are key for ongoing business operations. Containerizing with hardened container images from DoD Iron Bank helps to protect application container images. Hardened images can be kept up to date from reliable providers. Containers provide cloud-agnostic and no vendor lock-in solutions.

All the security tools in the DevSecOps pipeline must be deployed and running for pipeline scanning in the customer environment. A request will be sent to those security tools from the pipeline code via API request or trigger command-line interface (CLI) commands. Those tools then respond with their findings, statistics, and provide pass/fail criteria. If a tool identifies any vulnerability findings in the scan, then the pipeline will fail.

Deploying the security tools as SaaS services will require permission from the security team. Not all are approved to run in highly secured cloud environments. Those tools all need to be Authority to Operate (ATO) to deploy and configure. Whereas getting the hardened container images for those tools is a safer and secure approach to deploy those tools in the cloud. As the containers are already hardened, which means scanned, secured, and ready to go with all dependencies, they will provide continuous ATO. The hardened container images can be downloaded from DoD Iron Bank, and almost all tool providers provide container images. Many of these providers have different downloads, whether as a software download or a container image. When downloading as a software image, additional tasks to be done to ensure all the dependencies are appropriately configured or should pre-exist. Simultaneously, downloading as hardened container images comes with dependencies and are pre-scanned. The tools can be deployed into Kubernetes in your cloud environment to provide scalable functionality.

Below is a sample DevSecOps pipeline implementation with recommended security tools, as depicted in the picture below:

  • Source code pull request is approved by reviewers
  • The build pipeline kicks off and code scan is run after a successful initial build
    • If any code vulnerabilities are identified, then the pipeline fails
  • Build pipeline continues with DAST and PEN testing
    • If any vulnerabilities are identified, then the pipeline fails
  • Build artifacts are added to private repository either as packages or container
    • Repository scan is performed using repository scanning tools and vulnerabilities are reported
  • Release pipeline picks up artifacts from private repositories and deploys to Azure (or cloud of your choice)
    • Kubernetes is a highly recommended deployment for orchestration, but deployment can be an application of your choice such as Function App, App Service, Azure Container Instances, etc.
  • Security has been applied throughout the pipeline process and will continue once the application is deployed. Both native security tools such as Azure Monitor, Azure Security Center, Azure Policies, etc., and third-party tools such as Twistlock, Qualys, etc. Can be used to monitor the health of your production environment.DevSecOps Diagram

Let’s look at a few of the recommended tools to support the security validations in the DevSecOps process.

Build tools/CLI

A developer can write their code in their favorite editor such as Visual Studio, VS Code, and run/execute to test their applications. The code editor also generates debug/release packages generating binaries using the build tool that comes with the editor. The application works seamlessly from the developer environment as the dependencies and correct configurations exist. For the build to work in the pipeline, the build tool must be available to build the code. Based on the code language, the build tool varies, and they must be available in the pipeline.

Some of the build tools are:

  • DotNet Build
  • MSBuild
  • Maven
  • Gradle

Static Application Security Testing (SAST)

A code scan is one of the essential steps in securing the codebase. Automated testing helps identify failures, but these specific code scan tools help identify security flaws and vulnerabilities. The application does not need to be running for code scan tools as it scans only the codebase and not any dependencies.

Some of the Code scanning tools are:

  • SonarQube
  • Fortify
  • Anchore
  • JFrog Xray
  • OpenSCAP
  • HBSS
  • OWASP dependency check

Dynamic Application Security Testing (DAST)

DAST scans the application while its running or a container image that is hosted in private repositories. Container scanning before deploying helps resolve many security vulnerabilities.

Some of the DAST scanning tools are:

Penetration (Pen) Testing

Provides Web Applications scanner to help to find security vulnerabilities. Read here to learn about, “Top 10 Web Application Security Risks”

PEN testing tools:

  • OWASP ZAP

Deploy Code & IaC (Infrastructure as Code)

IaC is paramount in DevOps to avoid any manual work in customer environments and help with immutable infrastructure.

Popular IaC tools are:

  • Azure ARM Templates
  • Terraform
  • HELM
  • Private Repositories

In DevSecOps, a private repository is recommended to host the build dependencies, reference container images, container images for tools, and the built packages or application container images. This is to keep all the artifacts together in one centralized location, and the release pipeline can continue with deployments from there.
Some of the private repositories are:
JFrog
Docker Hub
Azure Container Registry (ACR)

Private Repository Scanning

As the pipeline requires security scanning, the repositories require scanning also. These tools scan for vulnerabilities in all packages and container artifacts stored in the repository. A scan report is being sent/notified for any issues.

Some artifact scanning tools are:

  • XRay
  • SonaType
  • Azure Monitor
  • Azure Security Center

Deploy

As the recommendation to deploy the security tools with container orchestration, the same recommendation goes to deployed applications. Containers provide high security with limited ways to be affected by attackers. Sidecar containers protect by continually monitoring applications with a container security stack built-in. Applications are scalable on a demand basis using Kubernetes and tools such as Kubectl; HELM packages are used to deploy and manage K8S clusters. ArgoCD is a declarative tool specifically for Kubernetes deployment in CI/CD pipeline.

Deployments to Azure could be:

  • Azure function app
  • Azure App Service
  • Azure Container Instance
  • Azure Kubernetes Service (AKS)
  • Open Shift in Azure
  • Monitoring/Alerting

Monitoring/Alerting

As the applications deployed and running in a cloud environment, it must be continuously monitored for attacks and identify any security vulnerabilities. For containers, these tools act as sidecar containers to regularly protect main containers from attacks, and some mitigate the issue. All these tools have built-in alert/notify operations team for immediate actions.

Monitoring/alerting tools:

  • Azure Monitor
  • Azure Security Center
  • Twistlock
  • Qualys
  • Aqua Security

So, all powered up with learning DevSecOps! Follow up back here for the next blog post in container-based deployments and containers scanning in the DevSecOps pipeline!

References for continuing your DevSecOps Journey

Have you been in a situation where an Azure Resource Manager (ARM) template with Azure Blueprint is the only option available for Infrastructure as Code (IaC) deployment? Have you realized that certain operations during deployment are not possible with an ARM template? Have you been in a situation that PowerShell or Azure CLI can get the job done, but there is no way to inject that script into the ARM template? Maybe you have felt like ARM-disabled at this time. If you answered yes to at least one of the situations, this blog is for you. Let’s dive into a cool feature, ‘Deployment Scripts in ARM Templates,’ which helped me to overcome these hurdles!

Azure managed disks by default have Server-Side Encryption (SSE) with Platform Managed Key (PMK), identified as SSE + PMK. I had the requirement to encrypt VM’s (Windows or Linux) with either SSE or Azure Disk Encryption (ADE) with Customer Managed Key (CMK). CMK provides an additional security layer as the customer can manage the keys, and they can rotate the keys periodically. Both types of encryptions require an RSA-based Key Vault key. While you can implement an SSE and ADE with CMK can using PowerShell and Azure CLI, I only had an ARM template deployment option. An ARM template has the functionality to create key vault secrets but cannot create key vault keys. For SSE encryption, a ‘disk encryption set‘ needs to be created. In the automated deployment, the key vault key and disk encryption set must exist for the virtual machine deployment to consume the key vault key to encrypt the VM and OS/Data disks.

The following picture shows default encryption on a VM managed disk – SSE with PMK (by default):

VM Disk

Choosing encryption types is based on customer requirements. Azure offers SSE + PMK as a default feature, which provides encryption at rest. Either SSE + CMK or ADE + CMK can be applied on top of default encryption. For more information to understand SSE and ADE, please read this great blog post. It explains fundamental differences between these types of encryptions when to choose one over the other, and the caveats to watch out after the encryption is applied.

Microsoft explains the difference between SSE and ADE by stating that Azure Disk Encryption leverages the BitLocker feature of Windows to encrypt managed disks with Customer-Managed Keys within the guest VM. Server-Side Encryption with Customer-Managed Keys improves on ADE by enabling you to use any OS type and images for your VMs by encrypting data in the Storage service.

The following screenshot shows Managed disk SSE + CMK encryption implemented via ARM template at the time of VM creation using Disk Encryption Set:

SSE + CMK encryption

The following screenshot shows Managed disk ADE + CMK encryption implemented via ARM Custom Extension:

ADE + CMK

The following screenshot shows ARM Custom Extension for ADE + CMK encryption:

ARM Custom Extension

The following screenshot shows how Key Vault secret is used by writing ‘Encrypted BEK’ for ADE + CMK encryption:

Key Vault secret

While I learned more about disk encryption, I was still contemplating disk encryption options in the ARM template. Microsoft ARM template team announced this cool new feature, deployment scripts! This was announced at the MS Build Event, which you can view here for more information. Deployment scripts are a new resource in ARM that can run PowerShell and Azure CLI scripts within the ARM template! The feature came in time to resolve ARM deployment impediments. The implementation creates a storage account to copy the script from the ARM template into file share and run an Azure container instance to execute the script. A user-managed identity is created and given permission to the resource group and added to key vault access policies. The script is executed as this user-managed identity. At this time, no system assigned identity is supported. The deployment scripts resource is available in both Azure public and government, wherever Azure container instances are available. Both the storage account and container instances are deleted after successful deployment scripts resource deployment.

The following screenshot is the deployment scripts ARM template resource:

Deployment Scripts ARM template

The following screenshot shows a container instance and storage account resources used by deployment scripts. They will be deleted after successful deployment script execution. The user managed identity is created in an ARM to execute the deployment script. The deployment script creates Key Vault, Keys in key vault, and Disk Encryption Set.

Container Instance

Additional features in deployment scripts resource:

  • The scripts can be either embedded inline or referenced from a location accessible from deployment.
  • The output from deployment scripts resource can be consumed/referenced by other resources in the deployment
  • An existing storage account can be referenced, and that storage account will be used instead of creating a temporary storage account.
  • Any task done via PowerShell and Azure CLI can be done in the deployment scripts, such as disk encryption, the storage account can be encrypted with CMK via PowerShell in deployment scripts.

Security considerations:

  • Deployment scripts create Standard_LRS storage account SKU, and if a Geo-Redundant security policy is enabled then the policy might fail and report. As the storage account gets deleted after the deployment scripts are done, the policy will not be non-compliant.
  • The storage account access should be public when using the existing storage account for the deployment script. This is to allow the script to be copied to file share and by permission for the container, instance to execute the script.
  • The Key Vault firewall needs to be turned off for the deployment scripts to be able to perform operations in key vault key and certificates. But the firewall can be enabled back in the PowerShell after all the work is done.

The following screenshot shows ARM Template Key vault resource, no firewall restriction:

ARM Template

In the following screenshot, I enable firewall access to the key vault in PowerShell script after the tasks are done:

Enable Firewall access

ARM template for SSE + CMK disk encryption:

SSE + CMK is applied when the disk-encryption-type parameter is set to ‘SSE’. If it is ‘ADE’ then no SSE is applied.

SSE + CMK is applied

ARM Template for ADE + CMK Disk Encryption using VM Extension

ARM Template for ADE

In the code example, the newly created Azure key vault key is saved to secrets to be able to access from an ARM template. ARM does not have the functionality to access key vault keys but can access secrets using the template reference parameter, like below.

Template Reference parameter

Instead of using as secrets, another approach to write the key vault key (KeK ID) as deployment scripts output and use that output in the VM ARM template.

The following screenshot shows how to write output from deployment scripts resource in keyvault.json:

Write up output

The following screenshot shows how to consume the output written in deployment scripts resource in vm.json for KeK URL:

Consume the output

For full code access to try using ‘deployment scripts’ in an ARM template to create Key vault keys, disk encryption set, and to encrypt the VM disks with SSE or ADE, please follow the link to get the code.

Once downloaded, follow the steps to test in your environment, Azure Cloud or Azure Government. Update the parameters before deploying:

Connect-AzAccount -Tenant "000000-000-0000-0000" -SubscriptionId "000000-000-0000-0000" -EnvironmentName AzureCloud

# Create Resource Group
$rg = New-AzResourceGroup -Name "deploy-script-rg" -Location "eastus2"

# Create Key Vault with ‘deployment scripts’
New-AzResourceGroupDeployment -ResourceGroupName $rg.ResourceGroupName -TemplateFile "C:\deploymentScriptsARM\keyvault.json" -TemplateParameterFile "C:\deploymentScriptsARM\keyvault.parameters.json"

# Create Virtual Machine with disk encryption
New-AzResourceGroupDeployment -ResourceGroupName $rg.ResourceGroupName -TemplateFile "C:\deploymentScriptsARM\vm.json" -TemplateParameterFile "C:\deploymentScriptsARM\vm.parameters.json"

Additional ARM Templates references:

Happy ARM Templating for IaC!!!

As promised in the previous blog post, here is a detailed explanation of how to connect to APIs secured in the Azure AD from SharePoint Framework (SPFx) web parts. Please read part I of this blog for a thorough understanding of the SharePoint Framework, comparing it with other models, and its constraints/disadvantages before diving in further.  

Connecting to APIs is essential functionality in today’s communication as it extends the versatile communication with outside data repositories. SharePoint web parts can render data not only from SharePoint lists and libraries but also from external repositories. The data can be owned by anyone outside the organization. The external repositories can be connected for data retrieval from SharePoint via API (Application Programming Interface) call. The external repositories can be on different platforms, domains, etc. SPFx comes with many namespaces to leverage the communication between the SPFx web part from SharePoint online with other repositories via API call.

Types of API Communications from SPFx

  • Connect to SharePoint APIs to access data residing in SharePoint (SPHttpClient with OData) 
  • This is access data residing in SharePoint lists/libraries 
  • Connect to Microsoft Graph (MSGraphClient through MSGraphClientFactory) 
  • This is access users and other user-related info from Azure Active Directory (AAD) 
  • Connect to enterprise APIs secured in Azure AD (enterprise APIs using AadHttpClient & aadHttpClientFactory) 
  • Connect to Azure API secured in Azure AD from the SharePoint Framework web part in single-tenant implementation, where both Azure and SharePoint online are under the same tenant. This blog covers the details of implementing this functionality. 
  • Connect to Azure API secured in Azure AD from the SharePoint Framework web part in multi-tenant implementation, where both Azure and SharePoint online are in different tenant’s 
  • Connect to anonymous APIs (using HttpClient to connect to public APIs for weather etc.) 
  • Anonymous API’s are used to access any weather and other publicly available API’s

Connect to Azure AD Secured APIs

Microsoft Web API Permissions

Figure Credit: Microsoft

Pre-Requisites

If your environment is already set up, ensure you have the latest version of Yeoman SharePoint generator by entering: 

npm update – g @microsoft/generator-sharepoint@latest

Steps to Develop, Deploy, and Test SPFx Connecting to Function API Secured in Azure AD

Once all the pre-requisites are met, follow the steps below to develop, deploy, and test the SharePoint Framework connecting to Azure API secured in an Azure active directory. 

1. Create an Azure function (HttpTrigger) returning mock data

Create Azure Function

Figure: Create a new Azure function

New Function Created

Figure: New Azure function is created

2. Create a HttpTrigger & add C# code to return the list of orders

Create Http Trigger

Figure: C# Azure Function code to return the list of orders

DOWNLOAD THE AZURE FUNCTION CODE HERE

3. Secure the azure function by enabling the authentication/authorization via Azure Active Directory (AAD) and create an app in AAD. Verify azure function works when called from the browser.

Secure Azure Function Screenshot

Figure: Configure authentication/authorization for the Azure function

4. Enable ‘App Service Authentication’

Authenticating App Services Screenshot

Figure: Selection Azure Active Directory for authentication/authorization

5. Active Directory authentication is set up & the API is secured in an Azure AD
Register Azure Function

Fig: Registered Azure function in Azure AD

6. Enable CORS (Cross-Origin Resource Sharing). Even though Azure & SharePoint Online are in the same tenant, they are in different domains.

Cross Origin Resource Sharing

Figure: Configure CORS

7. Add the SharePoint tenant URL.

Add SharePoint for CORS

    Figure: Add the SharePoint for CORS to authenticate the SharePoint site in Azure

8. Azure function API is secured in Azure AD & the application ID will be used in the SPFx web part.


Azure function installed on AD

Figure: Azure function is registered in the Azure AD as an Enterprise application

9. SharePoint online tenant/admin center in O365.

Available sites in SharePoint admin center

Figure: Available sites in the SharePoint admin center

10. Create an SPFx web part project to render the data by connecting to API secured in Azure AAD. Use Yeomen to generate a web part. Use this link for more information on generating web parts.

Yeoman generator to generate SPFx web part

Figure: Yeoman generator to generate SPFx web part

11. Add web API permission requests in config/package-solution.json file.

src\webparts\[webpartname]\config\package-solution.json – add two web api permission requests

 “webApiPermissionRequests”: [
{
“resource”: “contoso-api”,
“scope”: “user_impersonation”
},

{
“resource”: “Windows Azure Active Directory”,
“scope”: “User.Read”
},
]

SPFx web part configuration file

Figure: API permissions in SPFx web part configuration file

12. Import namespaces for enterprise API communication.

import AadHttpClient to connect with API in src\webparts\[webpartname]\[webpartname]WebPart.ts

import { AadHttpClient, HttpClientResponse } from ‘@microsoft/sp-http’;

src\webparts\[webpartname]\[webpartname]WebPart.ts

namespaces for enterprise API communication

DOWNLOAD THE WEB PART CODE HERE

13. Build, package, and upload the package to the SharePoint App Catalog.

gulp bundle –ship && gulp package-solution –ship

gulp clean (for redeploying after updates)

14. Add the SPFx package to the tenant app catalog in your Office 365 tenant. SPFx deploys API related file to the SharePoint admin center.

SPFx deploys API related file to the SharePoint

15. Approve requested API permissions.

From the SharePoint admin center in Office 365/SharePoint, approve the API from API Management page

Once the API is approved, the SPFx web part can be added to the SharePoint site page


API permissions are available in SharePoint online admin center

Figure: API permissions are available in SharePoint online admin center

16. Create a new site page from the developers’ site and add the SPFx web part.

Add the web part to SharePoint page

Figure: Add the web part to the SharePoint page

17. If all goes well, your web part will be rendered with data that is served from the API call!
Rendered with Data

SPFx – Connect to APIs Gotchas

  • Connecting to API secured in Azure AD did not work via SPFx AadHttpClient & aadHttpClientFactory in SharePoint 2019 on-premise. The this.context.aadHttpClientFactory did not work. I choose a web part that is part of “SharePoint 2019 and SharePoint Online” when creating the SPFx web part via the Yeomen generator. Choose only ‘SharePoint Online’ to use AadHttpClient & aadHttpClientFactory
  • Microsoft example code did not work as it is. The azure function needed a slight tweak. Additional permission must be added “webApiPermissionRequests” in the config\package-solution.json

{

“resource”: “Windows Azure Active Directory”,

“scope”: “User.Read”

}

  • Single-tenant vs multiple tenant access
    • First, I set up azure function and secure it in AAD API in a personal MSDN subscription tenant and tried to connect from Developer (Free) O365/SharePoint Online SPFx web part. But the API permission in the SharePoint admin center could not get approved. The permission to access API from a different tenant did not work with the way Azure function API is configured in AAD.
    • To overcome that issue set up an Azure tenant under developer O365 and with the same credential. The API was able to get approved. More Info…
    • To overcome this, need to configure AAD API authentication to multi-tenant. More info…
  • gulp clean – important when re-deploying, otherwise new updates will not get updated
  • Debugging is quite important for troubleshooting 

TRY IT YOURSELF! DOWNLOAD THE CODE TO GET STARTED.

Further ahead:

Accurately identifying and authenticating users is an essential requirement for any modern application. As modern applications continue to migrate beyond the physical boundaries of the data center and into the cloud, balancing the ability to leverage trusted identity stores with the need for enhanced flexibility to support this migration can be tricky. Additionally, evolving requirements like allowing multiple partners, authenticating across devices, or supporting new identity sources push application teams to embrace modern authentication protocols.

Microsoft states that federated identity is the ability to “Delegate authentication to an external identity provider. This can simplify development, minimize the requirement for user administration, and improve the user experience of the application.”

As organizations expand their user base to allow authentication of multiple users/partners/collaborators in their systems, the need for federated identity is imperative.

The Benefits of Federated Authentication

Federated authentication allows organizations to reliably outsource their authentication mechanism. It helps them focus on actually providing their service instead of spending time and effort on authentication infrastructure. An organization/service that provides authentication to their sub-systems are called Identity Providers. They provide federated identity authentication to the service provider/relying party. By using a common identity provider, relying applications can easily access other applications and web sites using single sign on (SSO).

SSO provides quick accessibility for users to multiple web sites without needing to manage individual passwords. Relying party applications communicate with a service provider, which then communicates with the identity provider to get user claims (claims authentication).

For example, an application registered in Azure Active Directory (AAD) relies on it as the identity provider. Users accessing an application registered in AAD will be prompted for their credentials and upon authentication from AAD, the access tokens are sent to the application. The valid claims token authenticates the user and the application does any further authentication. So here the application doesn’t need to have additional mechanisms for authentication thanks to the federated authentication from AAD. The authentication process can be combined with multi-factor authentication as well.

Glossary

Abbreviation Description
STS Security Token Service
IdP Identity Provider
SP Service Provider
POC Proof of Concept
SAML Security Assertion Markup Language
RP Relying party (same as service provider) that calls the Identity Provider to get tokens
AAD Azure Active Directory
ADDS Active Directory Domain Services
ADFS Active Directory Federation Services
OWIN Open Web Interface for .NET
SSO Single sign on
MFA Multi factor authentication

OpenId Connect/OAuth 2.0 & SAML

SAML and OpenID/OAuth are the two main types of Identity Providers that modern applications implement and consume as a service to authenticate their users. They both provide a framework for implementing SSO/federated authentication. OpenID is an open standard for authentication and combines with OAuth for authorization. SAML is also open standard and provides both authentication and authorization.  OpenID is JSON; OAuth2 can be either JSON or SAML2 whereas SAML is XML based. OpenID/OAuth are best suited for consumer applications like mobile apps, while SAML is preferred for enterprise-wide SSO implementation.

Microsoft Azure Cloud Identity Providers

The Microsoft Azure cloud provides numerous authentication methods for cloud-hosted and “hybrid” on-premises applications. This includes options for either OpenID/OAuth or SAML authentication. Some of the identity solutions are Azure Active Directory (AAD), Azure B2C, Azure B2B, Azure Pass through authentication, Active Directory Federation Service (ADFS), migrate on-premises ADFS applications to Azure, Azure AD Connect with federation and SAML as IdP.

The following third-party identity providers implement the SAML 2.0 standard: Azure Active Directory (AAD), Okta, OneLogin, PingOne, and Shibboleth.

A Deep Dive Implementation

This blog post will walk through an example I recently worked on using federated authentication with the SAML protocol. I was able to dive deep into identity and authentication with an assigned proof of concept (POC) to create a claims-aware application within an ASP.NET Azure Web Application using the federated authentication and SAML protocol. I used OWIN middleware to connect to Identity Provider.

The scope of POC was not to develop an Identity Provider/STS (Security Token Service) but to develop a Service Provider/Relying Party (RP) which sends a SAML request and receives SAML tokens/assertions. The SAML tokens are used by the calling application to authorize the user into the application.

Given the scope, I used stub Identity Provider so that the authentication implementation could be plugged into a production application and communicate with other Enterprise SAML Identity Providers.

The Approach

For an application to be claims aware, it needs to obtain a claim token from an Identity Provider. The claim contained in the token is then used for additional authorization in the application. Claim tokens are issued by an Identity Provider after authenticating the user. The login page for the application (where the user signs in) can be a Service Provider (Relying Party) or just an ASP.NET UI application that communicates with the Service Provider via a separate implementation.

Figure 1: Overall architecture – Identity Provider Implementation

Figure 1: Overall architecture – Identity Provider Implementation

The Implementation

An ASP.NET MVC application was implemented as SAML Service provider with OWIN middleware to initiate the connection with the SAML Identity Provider.

First, the communication is initiated with a SAML request from service provider. The identity provider validates the SAML request, verifies and authenticates the user, and sends back the SAML tokens/assertions. The claims returned to service provider are then sent back to the client application. Finally, the client application can authorize the user after reviewing the claims returned from the SAML identity provider, based on roles or other more refined permissions.

SustainSys is an open-source solution and its SAML2 libraries add SAML2P support to ASP.NET web sites and serve as the SAML2 Service Provider (SP).  For the proof of concept effort, I used a stub SAML identity provider SustainSys Saml2 to test the SAML service provider. SustainSys also has sample implementations of a service provider from stub.

Implementation steps:

  • Start with an ASP.NET MVC application.
  • Add NuGet packages for OWIN middleware and SustainSys SAML2 libraries to the project (Figure 2).
  • Modify the Startup.cs (partial classes) to build the SAML request; set all authentication types such as cookies, default sign-in, and SAMLl2 (Listing 2).
  • In both methods CreateSaml2Options and CreateSPOptions SAML requests are built with both private and public certificates, federation SAML Identity Provider URL, etc.
  • The service provider establishes the connection to identity on start up and is ready to listen to client requests.
  • Cookie authentication is set, default authentication type is “Application,” and set the SAML authentication request by forming the SAML request.
  • When the SAML request options are set, instantiate Identity Provider with its URL and options. Set the Federation to true. Service Provider is instantiated with SAML request options with the SAML identity provider. Upon sign in by the user, OWIN middleware will issue a challenge to the Identity Provider and get the SAML response, claim/assertion back to the service provider.
  • OWIN Middleware issues a challenge to SAML Identity Provider with the callback method (ExternalLoginCallback(…)). Identity provider returns that callback method after authenticating the user (Listing 3).
  • AuthenticateSync will have claims returned from the Identity Provider and the user is authenticated at this point. The application can use the claims to authorize the user to the application.
  • No additional web configuration is needed for SAML Identity Provider communication, but the application config values can be persisted in web.config.

Figure 2: OWIN Middleware NuGet Packages

Figure 2: OWIN Middleware NuGet Packages

Listing 1:  Startup.cs (Partial)

using Microsoft.Owin;
using Owin;

[assembly: OwinStartup(typeof(Claims_MVC_SAML_OWIN_SustainSys.Startup))]

namespace Claims_MVC_SAML_OWIN_SustainSys
{
    public partial class Startup
    {
        public void Configuration(IAppBuilder app)
        {
            ConfigureAuth(app);
        }
    }
}

Listing 2: Startup.cs (Partial)

using Microsoft.Owin;
using Microsoft.Owin.Security;
using Microsoft.Owin.Security.Cookies;
using Owin;
using Sustainsys.Saml2;
using Sustainsys.Saml2.Configuration;
using Sustainsys.Saml2.Metadata;
using Sustainsys.Saml2.Owin;
using Sustainsys.Saml2.WebSso;
using System;
using System.Configuration;
using System.Globalization;
using System.IdentityModel.Metadata;
using System.Security.Cryptography.X509Certificates;
using System.Web.Hosting;

namespace Claims_MVC_SAML_OWIN_SustainSys
{
    public partial class Startup
    {
        public void ConfigureAuth(IAppBuilder app)
        {            
            // Enable Application Sign In Cookie
            var cookieOptions = new CookieAuthenticationOptions
                {
                    LoginPath = new PathString("/Account/Login"),
                AuthenticationType = "Application",
                AuthenticationMode = AuthenticationMode.Passive
            };

            app.UseCookieAuthentication(cookieOptions);

            app.SetDefaultSignInAsAuthenticationType(cookieOptions.AuthenticationType);

            app.UseSaml2Authentication(CreateSaml2Options());
        }

        private static Saml2AuthenticationOptions CreateSaml2Options()
        {
            string samlIdpUrl = ConfigurationManager.AppSettings["SAML_IDP_URL"];
            string x509FileNamePath = ConfigurationManager.AppSettings["x509_File_Path"];

            var spOptions = CreateSPOptions();
            var Saml2Options = new Saml2AuthenticationOptions(false)
            {
                SPOptions = spOptions
            };

            var idp = new IdentityProvider(new EntityId(samlIdpUrl + "Metadata"), spOptions)
            {
                AllowUnsolicitedAuthnResponse = true,
                Binding = Saml2BindingType.HttpRedirect,
                SingleSignOnServiceUrl = new Uri(samlIdpUrl)
            };

            idp.SigningKeys.AddConfiguredKey(
                new X509Certificate2(HostingEnvironment.MapPath(x509FileNamePath)));

            Saml2Options.IdentityProviders.Add(idp);
            new Federation(samlIdpUrl + "Federation", true, Saml2Options);

            return Saml2Options;
        }

        private static SPOptions CreateSPOptions()
        {
            string entityID = ConfigurationManager.AppSettings["Entity_ID"];
            string serviceProviderReturnUrl = ConfigurationManager.AppSettings["ServiceProvider_Return_URL"];
            string pfxFilePath = ConfigurationManager.AppSettings["Private_Key_File_Path"];
            string samlIdpOrgName = ConfigurationManager.AppSettings["SAML_IDP_Org_Name"];
            string samlIdpOrgDisplayName = ConfigurationManager.AppSettings["SAML_IDP_Org_Display_Name"];

            var swedish = CultureInfo.GetCultureInfo("sv-se");
            var organization = new Organization();
            organization.Names.Add(new LocalizedName(samlIdpOrgName, swedish));
            organization.DisplayNames.Add(new LocalizedName(samlIdpOrgDisplayName, swedish));
            organization.Urls.Add(new LocalizedUri(new Uri("http://www.Sustainsys.se"), swedish));

            var spOptions = new SPOptions
            {
                EntityId = new EntityId(entityID),
                ReturnUrl = new Uri(serviceProviderReturnUrl),
                Organization = organization
            };
        
            var attributeConsumingService = new AttributeConsumingService("Saml2")
            {
                IsDefault = true,
            };

            attributeConsumingService.RequestedAttributes.Add(
                new RequestedAttribute("urn:someName")
                {
                    FriendlyName = "Some Name",
                    IsRequired = true,
                    NameFormat = RequestedAttribute.AttributeNameFormatUri
                });

            attributeConsumingService.RequestedAttributes.Add(
                new RequestedAttribute("Minimal"));

            spOptions.AttributeConsumingServices.Add(attributeConsumingService);

            spOptions.ServiceCertificates.Add(new X509Certificate2(
                AppDomain.CurrentDomain.SetupInformation.ApplicationBase + pfxFilePath));

            return spOptions;
        }
    }
}

Listing 3: AccountController.cs

using Claims_MVC_SAML_OWIN_SustainSys.Models;
using Microsoft.Owin.Security;
using System.Security.Claims;
using System.Text;
using System.Web;
using System.Web.Mvc;

namespace Claims_MVC_SAML_OWIN_SustainSys.Controllers
{
    [Authorize]
    public class AccountController : Controller
    {
        public AccountController()
        {
        }

        [AllowAnonymous]
        public ActionResult Login(string returnUrl)
        {
            ViewBag.ReturnUrl = returnUrl;
            return View();
        }

        //
        // POST: /Account/ExternalLogin
        [HttpPost]
        [AllowAnonymous]
        [ValidateAntiForgeryToken]
        public ActionResult ExternalLogin(string provider, string returnUrl)
        {
            // Request a redirect to the external login provider
            return new ChallengeResult(provider, Url.Action("ExternalLoginCallback", "Account", new { ReturnUrl = returnUrl }));
        }

        // GET: /Account/ExternalLoginCallback
        [AllowAnonymous]
        public ActionResult ExternalLoginCallback(string returnUrl)
        {
            var loginInfo = AuthenticationManager.AuthenticateAsync("Application").Result;
            if (loginInfo == null)
            {
                return RedirectToAction("/Login");
            }

            //Loop through to get claims for logged in user
            StringBuilder sb = new StringBuilder();
            foreach (Claim cl in loginInfo.Identity.Claims)
            {
                sb.AppendLine("Issuer: " + cl.Issuer);
                sb.AppendLine("Subject: " + cl.Subject.Name);
                sb.AppendLine("Type: " + cl.Type);
                sb.AppendLine("Value: " + cl.Value);
                sb.AppendLine();
            }
            ViewBag.CurrentUserClaims = sb.ToString();
            
            //ASP.NET ClaimsPrincipal is empty as Identity returned from AuthenticateAsync should be cast to IPrincipal
            //var identity = (ClaimsPrincipal)Thread.CurrentPrincipal;
            //var claims = identity.Claims;
            //string nameClaimValue = User.Identity.Name;
            //IEnumerable<Claim> claimss = ClaimsPrincipal.Current.Claims;
          
            return View("Login", new ExternalLoginConfirmationViewModel { Email = loginInfo.Identity.Name });
        }

        // Used for XSRF protection when adding external logins
        private const string XsrfKey = "XsrfId";

        private IAuthenticationManager AuthenticationManager
        {
            get
            {
                return HttpContext.GetOwinContext().Authentication;
            }
        }
        internal class ChallengeResult : HttpUnauthorizedResult
        {
            public ChallengeResult(string provider, string redirectUri)
                : this(provider, redirectUri, null)
            {
            }

            public ChallengeResult(string provider, string redirectUri, string userId)
            {
                LoginProvider = provider;
                RedirectUri = redirectUri;
                UserId = userId;
            }

            public string LoginProvider { get; set; }
            public string RedirectUri { get; set; }
            public string UserId { get; set; }

            public override void ExecuteResult(ControllerContext context)
            {
                var properties = new AuthenticationProperties { RedirectUri = RedirectUri };
                if (UserId != null)
                {
                    properties.Dictionary[XsrfKey] = UserId;
                }
                context.HttpContext.GetOwinContext().Authentication.Challenge(properties, LoginProvider);
            }
        }
    }
}

Listing 4: Web.Config

<?xml version="1.0" encoding="utf-8"?>
<!--
  For more information on how to configure your ASP.NET application, please visit
  https://go.microsoft.com/fwlink/?LinkId=301880
  -->
<configuration>
  <appSettings>
    <add key="webpages:Version" value="3.0.0.0" />
    <add key="webpages:Enabled" value="false" />
    <add key="ClientValidationEnabled" value="true" />
    <add key="UnobtrusiveJavaScriptEnabled" value="true" />
    <add key="SAML_IDP_URL" value="http://localhost:52071/" />
    <add key="x509_File_Path" value="~/App_Data/stubidp.sustainsys.com.cer"/>
    <add key="Private_Key_File_Path" value="/App_Data/Sustainsys.Saml2.Tests.pfx"/>
    <add key="Entity_ID" value="http://localhost:57234/Saml2"/>
    <add key="ServiceProvider_Return_URL" value="http://localhost:57234/Account/ExternalLoginCallback"/>
    <add key="SAML_IDP_Org_Name" value="Sustainsys"/>
    <add key="SAML_IDP_Org_Display_Name" value="Sustainsys AB"/>
  </appSettings>

Claims returned from the identity provider to service provider:

Claims returned from the identity provider to service provider

Additional References