The cloud-oriented domain presents numerous styles of building automated solutions for various personal & business use cases. And there is no such thing as a correct or incorrect solution to these problems. It all depends on your requirement and any restraints in building the solution.

The introduction of serverless solutions proved a perfect fit for scenarios where you must perform event-driven tasks cost-effectively and flexibly.

Azure Functions

Azure Functions are a lightweight serverless compute-on-demand, Platform-as-a-Service (PaaS) offering from Microsoft where you can run & deploy your code directly without being bothered about the underlying infrastructure to host your application. They scale automatically as the load increases & you just need to pay for the execution time of the functions.

Azure Functions Execution Mode: In-process vs. Out of Process

Initially, .NET Azure Functions only supported a strongly coupled mode of execution known as In-process mode, where your function code runs in the same .NET process as the host (Azure Functions runtime). In this mode, your code should run on the same framework version which is being used by the runtime. The tight coupling provided performance benefits such as faster cold start times but had shortcomings, such as the inability to create/use your Host instance to register your Middleware, Logger, etc.

Starting .NET 5, Microsoft introduced a new isolated-process (aka out-of-process) mode for running your code in Azure Functions. The main benefit of isolated mode is that it decouples your function code from the Azure Functions runtime, thus letting the users utilize any supported version of .NET even if it’s different from the runtime version.

Isolated mode removed the limitations of in-process execution mode, as it provided the user with the following:

  • Full control over how you configure & run your code inside Azure Functions
  • Ability to utilize features such as implementing custom Middleware, Logging, etc.
  • Encountering fewer conflicts between the code assemblies & the assemblies used by the host process.

Isolated-process Function Apps certainly offer numerous advantages compared to in-process Function Apps. And as per the roadmap shared by Microsoft for Azure Functions, isolated-process functions are indeed the future & eventually will be the only choice from .NET 7 onwards.

Isolated Process

This means that if you choose to go with isolated mode, the upgrade process for your codebase would be easier than in-process mode.

When writing this blog, our codebase is running on the .NET 6 framework. So, for our solution, we decided to implement Azure Functions (version 4.x) based on the isolated-process model.

Azure Functions Isolated Mode: Timer Triggers

To set up an Azure Functions project to run in isolated-process mode, select the Azure Functions project template & select the “.NET 6.0 Isolated (LTS)” type in the dropdown:

Setup Azure Functions

The requirement was to implement an automated solution scheduled to run at certain time intervals, so we decided to work with timer trigger functions as they let you run your code on a specified schedule.

Timer trigger functions use a CRON expression which describes a schedule on which the function is supposed to execute. A CRON expression contains six space-separated values represented as:

{second} {minute} {hour} {day} {month} {day-of-week}

In the above expression, each field can hold a specific (valid) numeric value, a range of values (2-4), a set of values (2,4,6), an interval value (e.g., /5 means every fifth of that unit) or all values (represented by an asterisk *).

The value present in the “Schedule” column in the above screenshot is an example of a CRON expression. The value says:

10 */5 * 15-20 * *

This CRON expression translates to “Run the Timer Trigger function at the 10th second of every 5th minute of every hour between day 15 and 20 in every month”.

After creating a .NET isolated function project, the Solution Explorer will contain the following files:

.NET isolation function project

The critical point to notice in an isolated functions project is that it contains a Program.cs file, which provides direct access to the Host instance, giving you full control on setting any code configurations & dependencies. There is no separate Startup class available/required here.

Another difference is the local.settings.json file, where the FUNCTIONS_WORKER_RUNTIME application setting is set to “dotnet-isolated”. The value is set to “dotnet” for an in-process Azure Function.

Note: If you are using your own appsettings.json files to store configuration values for various environments, you need to add the FUNCTIONS_WORKER_RUNTIME variable in the parent appsettings file as well so that it gets inherited & applied to all your appsettings.json files.

Functions Worker Runtime

FREE WHITEPAPER DOWNLOAD
Learn more about cost optimization techniques and realize impactful cost savings in Azure with our free white paper download.

Using Middleware with Azure Functions

For our use case, we implemented our middleware to perform exception handling & log every exception that occurs in our Function App. To do that, we created a class called.

“ExceptionHandlingMiddleware.cs”. Now, for this class to act like a middleware, it must inherit the IFunctionsWorkerMiddleware interface & implement it. At this point, your code should look like this:

public class ExceptionHandlingMiddleware : IFunctionsWorkerMiddleware
{
    public async Task Invoke(FunctionContext context, FunctionExecutionDelegate next)
    {
        await next(context);
    }
}

Let’s add some code to our middleware class to grab those exceptions and log them to the respective logger sources.

public sealed class ExceptionHandlingMiddleware : IFunctionsWorkerMiddleware
{
   
   public async Task Invoke(FunctionContext context, FunctionExecutionDelegate next)
   {
       try
       {
           await next(context);
       }
       catch (Exception e)
       {
           var log = context.GetLogger<ExceptionHandlingMiddleware>();
           log.Log(LogLevel.Error, eventId: 101, e, $"An exception occurred while calling the timer-triggered function. Exception message: {e.Message}. Original exception type: {e.GetType().Name}.");
       }
   }
}

Our custom middleware is set to handle any exceptions occurring during the execution of our function app. To use it, we need to register it in our Program.cs file.

public class Program
{
    public static void Main()
    {
        var host = new HostBuilder()
            .ConfigureFunctionsWorkerDefaults(
                builder =>
                {
                    builder.UseMiddleware<ExceptionHandlingMiddleware>();
                }
            )
            .Build();

        return host.Run();
    }
}

If an exception occurs in your Function App, it will get caught and logged to your console window, and if you have Application Insights configured, it will be logged in the respective Azure App Insights resource.

5-Minute Serverless First Series

Watch videos seven and eight in our Serverless First series, where we demonstrate how to automate scheduled tasks by implementing Azure Functions in Middleware.

SERVERLESS FIRST-SERIES
Check out our video series around Serverless and the impact that it is providing to companies around writing code, scaling, and more!

Conclusion

Isolated-process mode of Azure Functions provides the users complete control over the code, from deciding how to start the App to controlling our functions’ configuration. It can significantly reduce execution costs compared to other automation solutions, such as Azure Runbooks (the initial solution in our use-case before introducing Azure Functions), not to mention other benefits such as auto-scaling, ability to perform code-based regression testing, availability of multiple input/output options (compared to just JSON input for Runbooks) & various upgradeable tiers to host them.

Considering the benefits mentioned above, I guess it is safe to say that Azure Functions will play a key role in helping people solve complex problems in the cloud automation domain in the future.

What is Azure Databricks?

Azure Databricks is a data analytics platform that provides powerful computing capability, and the power comes from the Apache Spark cluster. In addition, Azure Databricks provides a collaborative platform for data engineers to share the clusters and workspaces, which yields higher productivity. Azure Databricks plays a major role in Azure Synapse, Data Lake, Azure Data Factory, etc., in the modern data warehouse architecture and integrates well with these resources.

Data engineers and data architects work together with data and develop the data pipeline for data ingestion with data processing. All data engineers work in a sandbox environment, and when they have verified the data ingestion process, the data pipeline is ready to be moved to Dev/Staging and Production.

Manually moving the data pipeline to staging/production environments via Azure portal will potentially introduce the difference in environments and add a tedious task to repeat manual processes in multiple environments. Automated deployment with service principal credentials is the only solution to move all your work to higher environments. There will be no privilege to configure via the Azure portal as a user. As data engineers complete the data pipeline, Cloud automation engineers will use IaC (Infrastructure as Code) to deploy all Azure resources and configure them via the automation pipeline. That includes all data related to Azure resources and Azure Databricks.

Data engineers work in Databricks with their user account, and it works very well integrating Azure Databricks with Azure key vault using key vault secret scope. All the secrets are persisted in key vault, and Databricks can get the secret value directly via linked service. Databricks uses user credentials to go against Keyvault to get the secret values. This does not work with service principal (SPN) access from Azure Databricks to the key vault. This functionality is requested but not yet there as per this GitHub issue.

JOIN OUR TEAM
Passionate about data? Check out our open data careers and apply to join our quickly growing team today!

Let’s Look at a Scenario

The data team has given automation engineers two requirements:

  • Deploy an Azure Databricks, a cluster, a dbc archive file which contains multiple notebooks in a single compressed file (for more information on dbc file, read here), secret scope, and trigger a post-deployment script.
  • Create a key vault secret scope local to Azure Databricks so the data ingestion process will have secret scope local to Databricks.

Azure Databricks is an Azure native resource, but any configurations within that workspace is not native to Azure. Azure Databricks can be deployed with Hashicorp Terraform code. For Databricks workspace-related artifacts, the Databricks provider needs to be added. For creating a cluster, use this implementation. If you are only uploading a single notebook file for creating a notebook, then use Terraform implementation like this. If not, there is an example below to use Databricks CLI to upload multiple notebook files as a single dbc archive file. The link to my GitHub repo for complete code is at the end of this blog post.

Terraform implementation

terraform {
  required_providers {
    azurerm = "~> 2.78.0"
    azuread = "~> 1.6.0"
    databricks = {
      source = "databrickslabs/databricks"
      version = "0.3.7"
    }
  }

  backend "azurerm" {
    resource_group_name  = "tf_backend_rg"
    storage_account_name = "tfbkndsapoc"
    container_name       = "tfstcont"
    key                  = "data-pipe.tfstate"
  }
}

provider "azurerm" {
  features {}
}

provider "azuread" {
}

data "azurerm_client_config" "current" {
}

// Create Resource Group
resource "azurerm_resource_group" "rgroup" {
  name     = var.resource_group_name
  location = var.location
}

// Create Databricks
resource "azurerm_databricks_workspace" "databricks" {
  name                          = var.databricks_name
  location                      = azurerm_resource_group.rgroup.location
  resource_group_name           = azurerm_resource_group.rgroup.name
  sku                           = "premium"
}

// Databricks Provider
provider "databricks" {
  azure_workspace_resource_id = azurerm_databricks_workspace.databricks.id
  azure_client_id             = var.client_id
  azure_client_secret         = var.client_secret
  azure_tenant_id             = var.tenant_id
}

resource "databricks_cluster" "databricks_cluster" {
  depends_on              = [azurerm_databricks_workspace.databricks]
  cluster_name            = var.databricks_cluster_name
  spark_version           = "8.2.x-scala2.12"
  node_type_id            = "Standard_DS3_v2"
  driver_node_type_id     = "Standard_DS3_v2"
  autotermination_minutes = 15
  num_workers             = 5
  spark_env_vars          = {
    "PYSPARK_PYTHON" : "/databricks/python3/bin/python3"
  }
  spark_conf = {
    "spark.databricks.cluster.profile" : "serverless",
    "spark.databricks.repl.allowedLanguages": "sql,python,r"
  }
  custom_tags = {
    "ResourceClass" = "Serverless"
  }
}

GitHub Actions workflow with Databricks CLI implementation

deploydatabricksartifacts:
    needs: [terraform]
    name: 'Databricks Artifacts Deployment'
    runs-on: ubuntu-latest

    steps:
    - uses: actions/checkout@v2.3.4
   
    - name: Set up Python 3.0
      uses: actions/setup-python@v2
      with:
        python-version: 3.0

    - name: Install dependencies
      run: |
        python -m pip install --upgrade pip

    - name: Download Databricks CLI
      id: databricks_cli
      shell: pwsh
      run: |
        pip install databricks-cli
        pip install databricks-cli --upgrade

    - name: Azure Login
      uses: azure/login@v1
      with:
        creds: ${{ secrets.AZURE_CREDENTIALS }}
   
    - name: Databricks management
      id: api_call_databricks_manage
      shell: bash
      run: |
        # Set DataBricks AAD token env
        export DATABRICKS_AAD_TOKEN=$(curl -X GET -d "grant_type=client_credentials&client_id=${{ env.ARM_CLIENT_ID }}&resource=2ff814a6-3304-4ab8-85cb-cd0e6f879c1d&client_secret=${{ env.ARM_CLIENT_SECRET }}" https://login.microsoftonline.com/${{ env.ARM_TENANT_ID }}/oauth2/token | jq -r ".access_token")

        # Log into Databricks with SPN
        databricks_workspace_url="https://${{ steps.get_databricks_url.outputs.DATABRICKS_URL }}/?o=${{ steps.get_databricks_url.outputs.DATABRICKS_ID }}"
        databricks configure --aad-token --host $databricks_workspace_url

        # Check if workspace notebook already exists
        export DB_WKSP=$(databricks workspace ls /${{ env.TF_VAR_databricks_notebook_name }})
        if [[ "$DB_WKSP" != *"RESOURCE_DOES_NOT_EXIST"* ]];
        then
          databricks workspace delete /${{ env.TF_VAR_databricks_notebook_name }} -r
        fi

        # Import DBC archive to Databricks Workspace
        databricks workspace import Databricks/${{ env.databricks_dbc_name }} /${{ env.TF_VAR_databricks_notebook_name }} -f DBC -l PYTHON

While the above example shows how to leverage Databricks CLI to do automation operations within Databricks, Terraform also provides richer capabilities with Databricks providers. Here is an example of how to add ‘service principal’ to Databricks ‘admins’ group in workspace using Terraform. This is essential for Databricks API to work when connecting as a service principal.

Databricks Creating Cluster
Databricks cluster deployed via Terraform
Jobs Deployed via Terraform
No Jobs have been deployed via Terraform
Databricks CLI
 Job deployed using Databricks CLI in GitHub Actions workflow
Deployment with Databricks
Job triggered via Databricks CLI in GitHub Actions workflow

Not just Terraform and Databricks CLI, but also Databricks API provides similar options to access Databricks artifacts and manage them. For example, to access the clusters in the Databricks:

  • To access clusters, first, authenticate if you are a workspace user via automation or using service principal.
  • If your service principal is already part of the workspaces admins group, use this API to get the clusters list.
  • If the service principal (SPN) is not part of the workspace, use this API that uses access and management tokens.
  • If you would rather add the service principal to Databricks admins workspace group, use this API (same as Terraform option above to add the SPN).

The secret scope in Databricks can be created using Terraform or using Databricks CLI or using Databricks API!

Databricks with other Azure resources have pretty good documentation, and for automating deployments, these options are essential: learn and use the best option that suits the needs!

Here is the link to my GitHub repo for complete code on using Terraform, Databricks CLI in GitHub Actions! In addition, you can find a bonus learning how to deploy synapse, ADLS, etc., as part of modern data warehouse deployment, which I will cover in my next blog post.

Until then, happy automating!

Introduction

PowerShell DSC is possibly one of the most potent configuration managers for Windows Operating Systems. Even with the increase in popularity of Ansible and SaltStack, Powershell DSC remains supreme when enforcing the desired state on a Windows VM. Ansible itself has included the win_dsc module, which allows Ansible to run PowerShell DSC. In this blog post, we will dive deeper into one of PowerShell DSC’s most powerful aspects, 3rd Party Resources, and how they interact with Azure Automation.

3rd Party Resources are PowerShell DSC Modules that are created by the community. Any PowerShell community member can create modules, and there are tons of modules out there to choose from. Modules are kept in repositories, the most well known and default PowerShell repository being the PowerShell Gallery run by Microsoft. This is a common repository for PowerShell modules that are deployed to the Gallery by the community. PowerShell Modules in the PSGallery can be downloaded and installed by the PowerShellGet Module.

As developers and infrastructure engineers, there are many different reasons to script various services you are creating. Often, instead of developing behavior or scripts from scratch, it is much easier to leverage the work that others have done to expedite a task’s completion. 3rd Party Modules allow for easily repeatable code that can become production-ready through collaboration.

Often, DSC Configuration can become complicated. Engineers can be asked to do many things, from creating an Azure AD Domain, configuring OMS Solutions associated with a VM, and even interactions with non-native Azure products, such as Splunk.

These may all seem very daunting, but don’t fret! Members of the PowerShell community have dealt with these problems and many others, and often you will find third party modules to help do the work for you.

Here is an example of a Third Party Resource, named ActiveDirectoryDsc, which will help in the promotion, configuration, and management of Active Directory

Azure Automation is a robust PaaS offering from Azure that allows for a cloud-based DSC pull server. Within Azure Automation, it is possible to add both custom modules that the user develops and third-party modules available in any hosted source.
⚠ It should be known that organizations in locked-down environments can manage their Repository of PowerShell Modules, which have been vetted by the respective InfoSec team. It is possible to deploy your Artifact repo using the Azure DevOps product shown here. It allows an internal team to deploy its versions of packages, and you can use that as your URI references.
⚠ There are a few ways to upload modules to the Azure Portal natively. You can upload manually through the portal as shown here in this picture:

Uploading modules to the Azure Portal

However, being DevOps Engineers, we want to automate this process as much as possible. One way to do this is via ARM Templates, like the ones we used in the previous module.
Below is an example of how to add a 3rd party module to your Azure Automation Account via ARM Templates:

{
"name": "[concat(parameters('automationAccountName'), '/', parameters('name'))]",
"type": "Microsoft.Automation/automationAccounts/modules",
"apiVersion": "2015-10-31",
"location": "[resourceGroup().location]",
"properties": {
"isGlobal": false,
"sizeInBytes": 0,
"contentLink": {
"uri": "uri to package"
}
}
}

If you are deploying from the PowerShellGallery, your Uri would look something like this:

"uri": "[concat('https://www.powershellgallery.com/api/v2/package/', parameters('name'), '/', parameters('version'))]"

Alternatively, you can script the import of modules using the New-AzAutomationModule module in a Powershell Script.

Oddly enough, there is sometimes some difficulty understanding the correct ContentUri to use in both the ARM and Powershell case. Finding the correct one can be done by navigating the right module in the Powershell Gallery, and adding /api/v2 to the URL, and replacing packages (plural) with package (singular).

Add the /api/v2 to a URL

Conclusion

3rd Party Modules are a great way for developers to speed up development and productivity. If you are inclined to help in the development of these modules, head over to GitHub and contribute!

Azure Automation provides credential assets for securely passing credentials between the automation account and a Desired State Configuration (DSC). Credentials can be created directly through the portal or through PowerShell and easily accessed in the DSC configuration. However, there a few disadvantages with storing credentials in an automation account vs. storing credentials in a KeyVault:

  • More fine-grained permissions can be set on a KeyVault – for example, custom access policies can be set for different principals.
  • KeyVault secret values can be viewed in the portal (assuming you have access). Passwords in an automation account credential cannot.
  • In most scenarios, a KeyVault is the “single source of truth”, where all secure assets in a tenant are stored. If you need to access credentials in an ARM template and a DSC configuration, they must be in a KeyVault for use in the ARM template.

In this example, we will walk through a very simple workstation configuration that pulls a username and password from a KeyVault, then passes those parameters into a DSC resource to create a new local user on a target VM.

Prerequisites

  • A Key Vault already provisioned
  • An Automation Account already provisioned
  • The Az.Accounts and Az.KeyVault modules imported into the Automation Account

Permissions

The Automation Connection, or more specifically, the service principal of the automation connection, needs to have at least “Get” selected under Secret permissions in the KeyVault access policies.

Creating the DSC file

These are the minimum set of parameters necessary to extract a username and password from a KeyVault:

param
(
    [Parameter(Mandatory)]
    [string] $keyVaultName,

    [Parameter(Mandatory)]
    [string] $usernameSecretName,

    [Parameter(Mandatory)]
    [string] $passwordSecretName,

    [Parameter(Mandatory)]
    [string] $automationConnectionName
)

The first 3 parameters’ purpose should be self-explanatory. The final parameter, automationConnectionName, is used to establish a connection to Azure. Even though this code is executing in the context of an Automation Account, it is not connected to Azure in the same way as if we had connected using Login-AzAccount or Connect-AzAccount. There are special cmdlets available when running in an automation account that we can use to establish a “full” connection:

$automationConnection = Get-AutomationConnection -Name $connectionName

Note that we are calling Get-AutomationConnection, NOT Get-AzAutomationConnection. The latter command only works when you have already established a connection to Azure. Get-AutomationConnection is one of those special cmdlets available when running in an Automation Account. Conversely, Get-AutomationConnection will not work if the DSC is executing outside the context of an Automation Account. For more information on connections in Azure Automation, refer to https://docs.microsoft.com/en-us/azure/automation/automation-connections

Get-AutomationConnection returns an object containing all the necessary properties for us to establish a “full” connection to Azure using the Connect-AzAccount cmdlet:

Connect-AzAccount -Tenant $automationConnection.TenantID -ApplicationId $automationConnection.ApplicationID -CertificateThumbprint $automationConnection.CertificateThumbprint 

Note that for those of you that aren’t running this in the Azure public cloud (such as Azure Government or Azure Germany), you’ll also need to add an environment switch to point to the correct cloud environment (such as -Environment AzureUSGovernment)

At this point, we can run the az cmdlets to extract the secrets from the KeyVault:

$username = (Get-AzKeyVaultSecret -VaultName $keyVaultName -Name $usernameSecretName).SecretValueText

$password = (Get-AzKeyVaultSecret -VaultName $keyVaultName -Name $passwordSecretName).SecretValue

Full Example Configuration

configuration Workstation
{
	param
	(
		[Parameter(Mandatory)]
		[string] $keyVaultName,

		[Parameter(Mandatory)]
		[string] $usernameSecretName,

		[Parameter(Mandatory)]
		[string] $passwordSecretName,

		[Parameter(Mandatory)]
		[string] $automationConnectionName
	)

	Import-DscResource -ModuleName PSDesiredStateConfiguration

	$automationConnection = Get-AutomationConnection -Name $automationConnectionName
	Connect-AzAccount -Tenant $automationConnection.TenantID -ApplicationId $automationConnection.ApplicationID -CertificateThumbprint $automationConnection.CertificateThumbprint

	$username = (Get-AzKeyVaultSecret -VaultName $keyVaultName -Name $usernameSecretName).SecretValueText

	$password = (Get-AzKeyVaultSecret -VaultName $keyVaultName -Name $passwordSecretName).SecretValue

	$credentials = New-Object System.Management.Automation.PSCredential ($username, $password)

	Node SampleWorkstation
	{
		User NonAdminUser
		{
			UserName = $username
			Password = $credentials
		}
	}
}

Final Thoughts

Remember the nature of DSC compilation – all variables are resolved at compile-time and stored in the resulting MOF file that is stored in the automation account. The compiled MOF is what is actually downloaded and executed on the target Node/VM. This means that if you change one of the secret values in the KeyVault, the MOF will still contain the old values until you recompile the DSC.

I thought Per Werngren made some important observations in his recent article for Redmond Channel Partner Magazine. His main point: System Integrators (SIs) need to evolve their business models or risk disintermediation. As workloads are migrated to AWS and Azure, automation replaces the need for people to perform those tasks. This automation enables governance and compliance to standards, while also setting the stage for better downstream, fully-automated management, monitoring and operations. This, of course, further reduces the need for people performing in those roles,

Meanwhile, the new generation of intelligent PaaS services for predictive analytics, artificial intelligence, machine learning, etc. are also replacing jobs once done by hand. These new tools allow us to build better and more intelligent applications.

FREE HALF DAY SESSION: APP MODERNIZATION APPROACHES & BEST PRACTICES
Transform your business into a modern enterprise that engages customers, supports innovation, and has a competitive advantage, all while cutting costs with cloud-based app modernization.

Despite all this potential for automation, we still regularly see organizations allowing contractors to move workloads manually. It’s simply in a staffing contractor’s best interest to have people do this, despite it being a time-consuming and error-prone process. But why would an SI recommend automation and reduce their long-term revenue? Read More…

Microsoft has over a thousand Virtual Machine images available in the Microsoft Azure Marketplace. If your organization has their own on-premises “Gold Image” that’s been tailored, hardened, and adapted to meet specific organizational requirements (compliance, business, security, etc.), you can bring those images into your Azure subscription for reuse, automation, and/or manageability.

I recently had the opportunity to take a client’s virtualized Windows Server 2008 R2 “Gold Image” in .OVA format (VMware ), extract the contents using 7-Zip, run the Microsoft Virtual Machine Converter to create a VHD, prepare and upload the VHD, and create a Managed Image that was then deployed using PowerShell and an Azure Resource Manager Template.

It’s actually quite simple! Here’s how… Read More…

Please read Part One and Part Two

Part 3: Azure Automation, Azure RunBooks, and Octopus Deploy

With just PowerShell and an Azure ARM template, we can kick off a deployment in just a few minutes. But there are still some manual steps involved – you still need to login to your Azure subscription, enter a command to create a new resource group, and enter another command to kick off a deployment. With the help of an Azure automation account and a platform called Octopus Deploy, we can automate this process even further to a point where it takes as little as three clicks to deploy your whole infrastructure! Read More…