Azure DevOps Services for Government: Information Storage

June 29, 2020
This is the second of two blog posts that will share solutions to common Azure DevOps Services concerns:

Storing Service Connection Credentials

To deploy Azure infrastructure from an Azure DevOps pipeline, your pipeline agent needs permission to create resources in your subscription. These permissions are granted in Azure DevOps with a Service Connection. You have two options when creating this service connection:

  1. Service Principal Authentication
  2. Managed Identity Authentication

If you choose to use a Service Principal, you must store a secret or certificate of an Azure Active Directory (AAD) managed application in Azure DevOps. Since this secret is equivalent to a service account password, some customers may be uncomfortable with storing this in Azure DevOps. In the alternative option, Managed Identity Authentication, you configure your pipeline agent as a system-assigned managed identity and give that identity the necessary permissions to your Azure resources. In this configuration, the certificate needed to authenticate is stored on the pipeline agent within your private network.

Storing Sensitive Information in the Backlog

It is important for the User Stories in the backlog to be well defined so that everyone involved in the development process understands how to define “done”. In some environments, the requirements themselves could contain information that is not appropriate to store in Azure DevOps.

You will need to ensure your team has a strong understanding of what information should not be captured in the backlog. For example, your organization may not allow the team to describe system security vulnerabilities, but the team needs to know what the vulnerability is to fix it. In this case, the team may decide to define a process in which the backlog contains placeholder work items that point to a location that can safely store the details of the work that are considered too sensitive to store within the backlog itself. To reinforce this process, the team could create a custom work item type with a limited set of fields.

New Security Vulnerability

This is an example of how a tool alone, such as Azure DevOps, is not enough to fully adopt a DevOps. You must also address the process and people.

Storing Sensitive Information in Code

Depending on your organization and the nature of your project, the content of the source code could also become an area of concern. Most developers are aware of the techniques used to securely reference sensitive values at build or runtime so that these secrets are not visible in the source code.

A common example is using a pointer to an Azure Key Vault Secrets in your code that is used to pull the secret down at runtime. Key Vault makes it easy to keep secrets out of your code in both application development and in infrastructure as code development by using the secret reference function in Azure Resource Manager (ARM) Template or retrieving them at compile time in your Desired State Configurations (DSC).

Features in some code scanning tools, such as Security Hotspot detection in SonarQube, can be used to detect sensitive values before they’ve been accidentally checked in, but like backlog management do not rely entirely on tools to get this done. Train your team to think about security throughout the entire development lifecycle and develop a process that can detect and address mistakes early.

Storing Deployment Logs

Even if using a private pipeline agent that is running on your internal network, deployment logs are sent back to Azure DevOps. You must be aware of any logs that contain information that your organization’s policies do not allow to be stored in Azure DevOps. It is difficult to be certain of the contents of the logs as they are not typically included in any type of quality assurance or security testing, and because they will constantly evolve as your application changes.

The solution is to filter the logs before they leave your internal network. While the pipeline agent has no configuration option to do this, you can design your pipeline to execute deployment jobs within a process you control. David Laughlin describes two ways to achieve this in his post Controlling Sensitive Output from Azure Pipelines Deployment Scripts.

Conclusion

The Azure DevOps product is becoming increasingly valuable to government development teams as they adopt DevSecOps practices. You have complete control over Azure DevOps Server at the cost of additional administrative overhead. With Azure DevOps Services, Microsoft takes much of the administrative burden off your team, but you must consider your organization’s compliance requirements when deciding how the tool is being used to power your development teams. I hope the solutions I have presented in this series help your team decide which to choose.

Azure DevOps Services for Government: Access Control

March 3, 2020
Azure DevOps provides a suite of tools that your development team needs to plan, create, and ship products. It comes in two flavors:

  • Azure DevOps Services – the SaaS option hosted by Microsoft.
  • Azure DevOps Server – the IaaS option hosted by you.

When comparing the two to decide which option enables your team to deliver the most value in the least amount of time, Azure DevOps Services is the clear winner, but velocity alone is not the only consideration for most government teams. The services you use must also be compliant with standardized government-wide security, authorization, and monitoring requirements.

Azure DevOps Services are hosted by Microsoft in Azure regions. At the time of this writing, you do not yet have the option to host Azure DevOps in an Azure Government region so you must use one of the available Azure Commercial regions. As a public service, it has also has not yet achieved compliance with any FedRAMP or DoD CC SRG audit scopes. This may sound like a non-starter, but as it states on the FedRAMP website, it depends on your context and how you use the product.

Depending on the services being offered, the third-party vendor does not necessarily have to be FedRAMP compliant, but there are security controls you must make sure they adhere to. If there is a connection to the third-party vendor, they should be listed in the System Security Plan in the Interconnection Table.

This is the first of two blog posts that will share solutions to common Azure DevOps Services concerns:

  1. In “Azure DevOps Services for Government: Access Control” (this post), I will cover common access control concerns.
  2. In “Azure DevOps Services for Government: Information Storage”, I will address common concerns with storing information in the commercial data centers that host Azure DevOps Services.

Managing User Access

All Azure DevOps Organizations support cloud authentication through either Microsoft accounts (MSA) or Azure Active Directory (AAD) accounts. If you’re using an MSA backed Azure DevOps organization, users will create their own accounts and will self-manage security settings, like multi-factor authentication. It is common for government projects to require a more centralized oversight of account management and policies.

The solution is to back your Azure DevOps Services Organization with an AAD tenant. An AAD backed Azure DevOps organization meets the following common authentication and user management requirements:

  • Administrators control the lifecycle of user accounts, not the users themselves. Administrators can centrally create, disable, or delete user accounts.
  • With AAD Conditional Access, administrators can create policies that allow or deny access to Azure DevOps based on conditions such as user IP location.
  • AAD can be configured to federate with an internal identity provider, which could be used to enable CAC authentication.

Controlling Production Deployment Pipelines

The value added by Continuous Delivery (CD) pipelines includes increasing team velocity and reducing the risks associated with introducing a change. To fully realize these benefits, you’ll want to design your pipeline to begin at the source code and end in production so that you can eliminate bottlenecks and ensure a predictable deployment outcome across your test, staging, and production environments.
It is common for teams to grant limited production access to privileged administrators, so some customers raise valid concerns with the idea of a production deployment pipeline when they realize that an action triggered by a non-privileged developer could initiate a process that ultimately changes production resources (ex: deployment pipeline triggered from a code change).

The solution is to properly configure your Azure DevOps pipeline Environments. Each environment in Azure DevOps represents a target environment of a deployment pipeline. Environment management is separate from pipeline configuration which creates a separation of duties:

  • Team members who define what a deployment needs to do to deploy an application.
  • Team members who control the flow of changes into environments.

Example

A developer team member has configured a pipeline that deploys a Human Resource application. The pipeline is called “HR Service”. You can see in the YAML code below, the developer intends to run the Deploy.ps1 scripts on the Production pipeline Environment.

Production Environment

If we review the Production environment configuration, we can see that an approval check has been configured. When the deployment reaches the stage that will attempt to run against the production environment, a member of the “Privileged Administrators” AAD group will be notified that deployment is awaiting their approval.

Approvals and Checks

Only the Privileged Administrators group has been given access to administer the pipeline environment, so the team member awaiting approval would not be able to bypass the approval step by disabling it in the environment configuration or in the pipelines YAML definition.

Security Production

By layering environment configuration with other strategies you’ll establish the boundaries needed to protect your environment but will also empower your development team to work autonomously and without unnecessary bottlenecks. Other strategies to layer include:

  • Governance enforces with Azure Policies
  • Deployment gates based on environment monitoring
  • Automated quality and security scans into your pipeline

It is also important for all team members involved with a release to be aware of what is changing so that you are not just automating a release over the wall of confusion. This is an example of how a product alone, such as Azure DevOps, is not enough to fully adopt DevOps. You must also address the process and people.

Deployment Pipeline Access to Private Infrastructure

How will Azure DevOps, a service on a commercial Azure region, be able to “see” my private infrastructure hosted in Azure Government? That’s usually one of the first questions I hear when discussing production deployment pipelines. Microsoft has a straight-forward answer to this scenario: self-hosted pipeline agents. Azure DevOps will have no line of sight to your infrastructure. The high-level process to follow when deploying an agent into your environment looks like this:

  • Deploy a virtual machine or container into your private network.
  • Apply any baseline configuration to the machine, such as those defined in the DISA’s Security Technical Implementation Guides (STIG).
  • Install the pipeline agent software on the machine and register the agent with an agent pool in Azure DevOps.
  • Authorize pipelines to use the agent pool to run deployments.

With this configuration, a pipeline job queued in Azure DevOps will now be retrieved by the pipeline agent over 443, pulled into the private network, and then executed.

Azure DevOps Configuration

Conclusion

In this post, I’ve introduced you to several practices you can use to develop applications faster without sacrificing security. Stay tuned for the next post in this series where we will discuss common concerns from Government clients around the storage of data within Azure DevOps.

Advancements in SharePoint Framework and DevOps for Office 365

December 20, 2019
In this post, I will show you how DevOps practices can add value to a variety of Office 365 development scenarios. The practices we will discuss are Infrastructure as Code, Continuous Integration, and Continuous Delivery. The advances in DevOps and SharePoint Framework (SPFx) have allowed us to make advancements in the way that we develop software and have improved our efficiency.

Infrastructure as Code (IaC)

Practicing IaC means that the infrastructure your applications depend on is created and maintained by code that is source controlled, tested, and deployed to production much like software. When discussing IaC, we’re typically talking about provisioning resources to a cloud provider. In our case, the “infrastructure” is Office 365 – a SaaS product with extensive customization and configuration options.  

While you could manage your O365 tenant with PowerShell, the code-centric and template-based PnP Provisioning Framework aligns better with this practice because: 

  1. Using the frameworks declarative XML syntax, you describe what you want to exist rather than writing code to manage how it gets created.
  2. It is easier for developers to run idempotent deployments to enact the desired state of your Office 365 tenant.  

While originally developed to support SharePoint Online and on-premise deployments, you can see in its latest schema that it has expanded to support Microsoft Teams, OneDrive, and Active Directory.  

Continuous Integration (CI) 

The practice of continuously integrating first means that your team has established the habit of frequently merging small batches of changes into a central code repository. Upon that merge, we automatically build and test the code to quickly identify bugs and quality issues.  

SharePoint Framework is a commonly used tool used to extend the capabilities of SharePoint Online and on-premise.  Much like the Provisioning Framework, SharePoint Framework is expanding to support other Office 365 services. You can currently use it to develop for Microsoft Teams and will soon be able to use it to develop Office Add-Ins.

Azure DevOps is a one-stop-shop service that provides everything you need throughout the software development lifecycle. For example, your team can version control your project’s source code in Repos. After merging changes, use Pipelines to trigger a CI process that runs the build and test tasks of your SharePoint Framework solution.  

Continuous Delivery (CD)

Continuous Delivery, the practice of running automated deployments through a sequence of environments, starts after a completed CI process. Azure DevOps Pipelines will again be the tool of choice to execute the deployment procedures against each environment.

Example Solution

A solution demonstrating how to use the technologies and practices described above is available on Applied Information Science’s GitHub account. The result is a pipeline capable of receiving frequent changes to O365 configuration and SPFx applications from 1 or many developers, verifying the quality of the change, and deploying it to a series of environments.

Dev Tenant Diagram

I encourage you to explore the source code using the following summary as a guide. You’ll find the solution organized into three areas – SPFx, Provisioning, and Pipeline.

SPFx

A simple hello world web part was created using the yeoman generator. Jest was added to test the code. Npm and gulp scripts are used to build and package the source code which produces an sppkg file.

Provisioning

The PnP Provisioning Template XML file found here defines the desired state of the target tenant. The following is the desired state:

  1. Install the SPFx App into the tenant App Catalog.
  2. Create a site collection that will host our web parts page.
  3. Install the SPFx App to the Site Collection.
  4. Create a page that will host our web part.
  5. Add the web part to the page.

By using parameters for the tenant URL and site owner, the same template can be deployed to multiple environments. A PowerShell build script bundles the template and all required files, such as the SPFx sppkg file, into a single pnp file ready for deployment.

Pipeline

A multi-stage YAML pipeline defined in the Pipeline folder of the example solution runs the following process:

  1. Build, test, and package the SPFx and Provisioning Template source code.
  2. Deploy the prerequisite SharePoint infrastructure to the tenant if it does not already exist.
  3. Install and configure the SPFx web part.
  4. Repeat #2 and #3 for all environments.

Build Process Diagram

Secret variables, such as the username and password used to connect to the tenant, are only referenced in the pipeline. The values are set and encrypted in the Azure DevOps pipeline editor.

Variables Diagram with Passwords

Conclusion

In the not-too-distant past, it was high effort to write unit tests for a SharePoint solution, and most deployments were manual. In this post, I have shown you how advancements in the platform and tooling have changed this. The mentality, practices, and tools brought by DevOps can improve the pace and quality of any software development and infrastructure management project, including projects building upon Office 365.