This past Thanksgiving marked the first anniversary of going live with a SharePoint environment that AIS migrated from on-prem to Microsoft Azure IL5. Since then, our client has experienced 100% uptime during business hours and reduced deployment timelines, from weeks to minutes.

Challenge: Improve Performance, Speed Up Deployments

AIS set out to help a DoD agency that had experienced ongoing service issues with their existing provider while operating their on-prem SharePoint farm. During the year before the migration, the DoD customer experienced three service outages during business hours, which halted the ability to perform mission-critical activities. Additionally, their existing enterprise service provider required a lead-in time of 1-2 weeks to deploy any code changes or new capabilities into the environment. AIS was tasked with building a cloud solution to maximize uptime and accommodate rapid deployments to better serve the fast tempo required by our DoD customer.

Solution: Hybrid IaaS/Paas in Azure IL5

To provide a solution tailored to maximize uptime and accommodate rapid deployments, AIS architected a DoD first: a hybrid IaaS/PaaS environment in Azure Government IL5 that utilized the DISA Cloud Access Point to integrate with NIPRNet. We leveraged a suite of technologies to employ DevSecOps methodologies, allowing the solution to remain scalable while adhering to industry best practices. By implementing an automated code scanning solution, we reduced deployment lead-in time from weeks to minutes. Our infrastructure as code (IaC) development also drastically reduced the time required to build a new environment from several days to under one hour.

Looking Ahead: Cost-Sharing, Scale Across the DoD

AIS has worked with our DoD customers to offer these cloud services to neighboring agencies to benefit from cost-sharing. In doing so, we have passed on lessons learned and processes that we have developed to share our success across the DoD enterprise. As we grow, we continue to integrate evolving best practices to remain at the DoD DevSecOps initiative’s forefront.

AIS FIRST TO DEPLOY AZURE IL6 EVNVIRONMENT AND ACHIEVE ATO!

Before you start deep dive for implementing DevSecOps in this blog post, please review the fundamentals of DevSecOps in my first blog post. It will help understand the ‘Sec’ in DevSecOps and get up to speed on various security tools for implementing DevSecOps in your CI/CD pipeline. Although there are many code repositories tools with CI/CD built-in, this blog walks through GitHub and its security scanning tools for DevSecOps implementations.

This blog post provides a GitHub repo for you to fork and try it on your own. The repo has an out of the box .NET core application, docker file to containerize the application. GitHub actions workflow YML code to build and deploy the containerized application to Azure. Further steps below will help understand introducing code scanning and container scanning tools. While I have used CodeQL for code scanning and Anchore for container scanning, can easily be replaced with other security scanning tools.

This repo/blog does not have all DevSecOps pipeline security features but integrates code scanning and container scanning tools. It shows how to get started on a GitHub Actions workflow and add these tools for security scanning. DevSecOps fundamentals are to understand and integrate these security tools in your pipeline. Source code is the single source of truth, and adding CI/CD pipeline automation is the first step. The security tools integrated into the pipeline scan your source code, scan the dependency packages/software, and will be added to your code.

GitHub Actions CI/CD

As we saw in the previous blog post, we will use the below high-level DevSecOps CI/CD pipeline workflow. I am not covering ‘artifactory’ functionality in this blog post and covering it in the next blog post!

Pipeline Workflow

Pic: Simple DevSecOps pipeline workflow.

A .NET core application is then added to the GitHub repo. The docker file is added to containerize the .NET Core application using docker build. A GitHub actions workflow is added to run the CI/CD pipeline. The actions will contain code for build/deploy and integrating security tools within the pipeline. The GitHub Actions workflow in the repo has the following steps:

  1. Starts the workflow manually which helps with debugging/testing. It can be changed to run when a branch is pushed or to main, and when a Pull Request is submitted.
  2. Azure related variables are declared as workflow environment variables
  3. Build the .NET code
  4. Parallelly CodeQL scans the code. CodeQL action can be added to the main pipeline YML.
  5. Anchore container scanning before the container deployed to ACR
  6. Log in to Azure
  7. Create an Azure Container Registry (ACR) and log into ACR.
  8. Docker build and push the application container to ACR
  9. GitHub container scan, scans the container in ACR
  10. Create a new Azure Container Instance to host the application running in a container
  11. Log out from Azure after completing the pipeline workflow

Now log into the Application is successfully running in Azure. Log in & verify your application running.

To learn more about GitHub Actions workflow and how to leverage your projects, follow the links:

CodeQL

Code scanning is essential even before security scanning is applied in the pipeline, as this is the first defense line. GitHub blog, GitHub code scanning is a developer-first, GitHub-native approach to find security vulnerabilities before they reach production easily. We’re thrilled to announce the general availability of code scanning. You can enable it on your public repository today!

The code scanning is completed using CodeQL, which is GitHub native approach to scan the code to identify the security vulnerabilities while the code is still being built. Other third-party code scanning workflows can also be added, as you see from the below screenshots. To set CodeQL code scanning, click on ‘Security’ from your repo. Then select ‘Code Scanning’ to add CodeQL workflow.

Add Code Scanning to GitHub repo

Pic: Add Code Scanning to your code in GitHub repo.Add CodeQL workflow

Pic: Add CodeQL workflow to your code. Third-party code scanning can be added as well.

Results of CodeQL Code Scanning

Pic: CodeQL Code scanning results.

CodeQL code scanning workflow runs parallel to the main workflow. It can be changed to run at the beginning of the main workflow also. CodeQL identifies what type of language is in the repo and runs the scanning accordingly. As this repo contains C# and JavaScript, code scanning is done for those languages. If there are significant errors/vulnerabilities found, then the code scanning workflow will fail. But for general warnings, alerts are available for review and fix the warnings.

View Code Scanning Alerts

Pic: View the code scanning alerts.

Results of Code Scanning

Pic: Code scanning results.

CodeQL scanning warnings can be analyzed and decide whether it is needed to fix the errors/warning and if it is safe to ignore. As this is the out of box .NET application, there are not many alerts/warnings. But if its production code and especially any migration-related codes might have more security vulnerabilities that need a fix before proceeding in the CI pipeline.

GitHub Container Scanning

This container scanning is native GitHub action to scan Docker containers in the CI pipeline. This identifies any vulnerabilities before the container is published to an application instance. More information can be found here, and this action uses Trivy and Dockle for running container scans on these images. They follow CIS Container Benchmarks for a baseline to secure the container images.

GitHub container Scan

Pic: GitHub Container Scan is added to the main workflow.

Having this action in the repo and running the main workflow failed due to these vulnerabilities findings. Any of the vulnerability findings can be ignored if it is in acceptable criteria. They can be added to ‘allowedlist.yaml’ file. The container action will then ignore those vulnerabilities. For more information, read here. Container scanning in GitHub is robust in scanning for vulnerabilities before the containers are deployed to the customer environment. But a closer review of these vulnerabilities needs to be done so appropriate scanning is performed in the repo. An analysis will yield better evaluations to find if the findings are indeed from code or from any dependencies. The container scan provides scantizer results like this.

Finding Vulnerabilities

Pic: Vulnerability findings.

CIS Container baseline violations warning & info

Pic: CIS Container baseline violations warning & info.

Added Vulnerabilities to ignore list

Pic: Added vulnerabilities to ignore list.

Anchore Container Scanning

Anchore is an open-source container scanning tool added to the GitHub Actions pipeline. More than one container scanning actions can be added to a repo workflow—more information on how Anchore container scanning works.

Anchore Container Scanning Action added to main pipeline workflow

Pic: Anchore container scanning action added to main pipeline workflow.

Anchore container scan also identified the same vulnerabilities

Pic: Anchore container scan also identified the same vulnerabilities as GitHub container scan.

According to NVD (National Vulnerability Database) in NIST, CVE (Common Vulnerabilities Exposures)
CVE defines vulnerability as:
“A weakness in the computational logic (e.g., code) found in software and hardware components that, when exploited, results in a negative impact to confidentiality, integrity, or availability. Mitigation of the vulnerabilities in this context typically involves coding changes, but could also include specification changes or even specification deprecations (e.g., removal of affected protocols or functionality in their entirety).”
All vulnerabilities in the NVD have been assigned a CVE identifier and thus, abide by this definition.

I took one of the CVE identified in both GitHub & Anchore container scan actions, CVE-2019-3843. According to NVD database in NIST, this is related to systems service-specific running in the container instances. As this repo is out of the box .NET core application, no additional code has been added. Further analysis must be done to understand the CVE depth to mitigate or override.

Current Description Title Link

Pic: https://nvd.nist.gov/vuln/detail/CVE-2019-3843#vulnCurrentDescriptionTitle

The vulnerabilities could be coming from the dependency container image used for building the docker image for the application. Hence it is essential to make sure dependencies are scanned as well.

Application Container Deployed in Azure

If all scanning goes well, then the pipeline is ready to deploy the application to Azure. Here the Azure CLI is used in the pipeline to deploy the containerized application to Azure Container Instance (ACI)!

Resource group for Azure resources

Pic: Resource group for Azure resources.

Azure Resources ACR & ACI

Pic: Azure resources (ACR & ACI).

Image of Container in ACR

Pic: Container in ACR.

Container deployed to Azure Container Instance (ACI)

Pic: Container deployed to Azure Container Instance (ACI)

Application up and running in Azure

Pic: Application up and running in Azure.

I hope the DevSecOps implementation walk-through with GitHub repo to try it out on your own adds value in understanding ‘Sec’ in DevSecOps and how to protect the codebase from vulnerabilities before deployed into production. While the blog post walked through GitHub CI/CD, all the other DevOps tools such as Azure DevOps, GitLab, Jenkins, etc. provide similar security tools implementations in their CI/CD pipeline. So, make sure to integrate security tools and shift the security to the left in whichever DevOps tool your organization decides to implement for your CI/CD pipelines. Follow up back here for the next blog post in this series for artifactory, running security tools from containers, the importance of containers, Kubernetes, and how to consume hardened container images from IronBank offered by DoD DevSecOps!

DevOps implements a Continuous Integration/Continuous Delivery (CI/CD) process. When multiple team members work in the same codebase, anyone’s update could break the integrated code. So, Continuous Integration is to trigger a build pipeline whenever a code update is pushed. The build pipeline will fail if the newly updated code is incompatible with the existing codebase if there are any conflicts. The codebase might work well within a single developer environment, but in a build pipeline where all configurations and dependencies are expected to be in place can fail. Continuous Delivery speeds up the deployment process. The release pipeline helps to deploy the same code base to multiple environments based on configurations. This helps to support code to be deployed in all environments without many manual changes.

Having an approval process helps peer code reviews, identifies potential issues, and any security flaws ahead of time. The current production applications are very distributed and complex. Whether it is an on-premise or cloud-based solution, missing a dependency or proper configurations could cost significant risk in deployments. DevOps helps to maintain the same code base for repeatable deployment in many environments with just configuration changes. DevOps avoids manually building the deployment packages and handing over to the operations team who would not have insights on what is being deployed. If an error occurs during deployment or post-deployment, then the development team jumps in at that time, which is time-consuming. This will cost in production timeline and end up with some unhappy customers also!
DevOps ImagePicture credit: DoD DevOps

Popular DevOps Tools

Follow here to learn more about DevOps practices from other AIS bloggers!

Why not just “DevOps”?

DevOps is fundamental for any organization’s build and deployment process with seamless CI/CD integration. Then, what is ‘DevSecOps’ and why is ‘Sec’ added between Dev and Ops. The ‘Sec’ in DevSecOps is ‘Security.‘ Though it’s added in between, security implementation should start from Development and continue in Operations. As development and deployment packages add many dependencies from both internal and external, this could introduce vulnerabilities. It could cost severe issues in production if not identified earlier in the build pipeline. Code scans help identify possible weaknesses in code implementations. But for any cybersecurity-related vulnerabilities, only specific tools at different stages of the pipeline must be used to identify as early as possible. Adding security scanning earlier in the pipeline and automating are essential for DevSecOps.

DevSecOps Software Lifecycle

Picture Credit: DoD DevSecOps

DevSecOps is not a tool or pattern but a practice and can be enhanced by adding appropriate tools. It is a process in securing the build and deployment by using several security tools by shifting security to the left. These security tools help to identify vulnerabilities that the code could have introduced, recommend possible solutions to fix those issues, and in some instances, the tools can mitigate some of those issues as well. This is to use the ‘fail fast’ method to identify vulnerabilities earlier in the build pipeline. As more applications moved into the cloud, it is highly imperative to follow Cloud Native Computing Foundation (CNCF) certified tools and implement security benchmarks that provided CIS benchmarks. DevSecOps avoids manual changes once the code is in the pipeline, deployed, and deployed. The codebase will be a single source of truth and should not be manipulated at any point.

Adding scanning tools for security and vulnerabilities helps to mitigate any flaws introduced in code and operations. Many open-source tools provide these functionalities. Enabling logging, continuous monitoring, alerting processes, and any self-fix for faster remediation are key for ongoing business operations. Containerizing with hardened container images from DoD Iron Bank helps to protect application container images. Hardened images can be kept up to date from reliable providers. Containers provide cloud-agnostic and no vendor lock-in solutions.

All the security tools in the DevSecOps pipeline must be deployed and running for pipeline scanning in the customer environment. A request will be sent to those security tools from the pipeline code via API request or trigger command-line interface (CLI) commands. Those tools then respond with their findings, statistics, and provide pass/fail criteria. If a tool identifies any vulnerability findings in the scan, then the pipeline will fail.

Deploying the security tools as SaaS services will require permission from the security team. Not all are approved to run in highly secured cloud environments. Those tools all need to be Authority to Operate (ATO) to deploy and configure. Whereas getting the hardened container images for those tools is a safer and secure approach to deploy those tools in the cloud. As the containers are already hardened, which means scanned, secured, and ready to go with all dependencies, they will provide continuous ATO. The hardened container images can be downloaded from DoD Iron Bank, and almost all tool providers provide container images. Many of these providers have different downloads, whether as a software download or a container image. When downloading as a software image, additional tasks to be done to ensure all the dependencies are appropriately configured or should pre-exist. Simultaneously, downloading as hardened container images comes with dependencies and are pre-scanned. The tools can be deployed into Kubernetes in your cloud environment to provide scalable functionality.

Below is a sample DevSecOps pipeline implementation with recommended security tools, as depicted in the picture below:

  • Source code pull request is approved by reviewers
  • The build pipeline kicks off and code scan is run after a successful initial build
    • If any code vulnerabilities are identified, then the pipeline fails
  • Build pipeline continues with DAST and PEN testing
    • If any vulnerabilities are identified, then the pipeline fails
  • Build artifacts are added to private repository either as packages or container
    • Repository scan is performed using repository scanning tools and vulnerabilities are reported
  • Release pipeline picks up artifacts from private repositories and deploys to Azure (or cloud of your choice)
    • Kubernetes is a highly recommended deployment for orchestration, but deployment can be an application of your choice such as Function App, App Service, Azure Container Instances, etc.
  • Security has been applied throughout the pipeline process and will continue once the application is deployed. Both native security tools such as Azure Monitor, Azure Security Center, Azure Policies, etc., and third-party tools such as Twistlock, Qualys, etc. Can be used to monitor the health of your production environment.DevSecOps Diagram

Let’s look at a few of the recommended tools to support the security validations in the DevSecOps process.

Build tools/CLI

A developer can write their code in their favorite editor such as Visual Studio, VS Code, and run/execute to test their applications. The code editor also generates debug/release packages generating binaries using the build tool that comes with the editor. The application works seamlessly from the developer environment as the dependencies and correct configurations exist. For the build to work in the pipeline, the build tool must be available to build the code. Based on the code language, the build tool varies, and they must be available in the pipeline.

Some of the build tools are:

  • DotNet Build
  • MSBuild
  • Maven
  • Gradle

Static Application Security Testing (SAST)

A code scan is one of the essential steps in securing the codebase. Automated testing helps identify failures, but these specific code scan tools help identify security flaws and vulnerabilities. The application does not need to be running for code scan tools as it scans only the codebase and not any dependencies.

Some of the Code scanning tools are:

  • SonarQube
  • Fortify
  • Anchore
  • JFrog Xray
  • OpenSCAP
  • HBSS
  • OWASP dependency check

Dynamic Application Security Testing (DAST)

DAST scans the application while its running or a container image that is hosted in private repositories. Container scanning before deploying helps resolve many security vulnerabilities.

Some of the DAST scanning tools are:

Penetration (Pen) Testing

Provides Web Applications scanner to help to find security vulnerabilities. Read here to learn about, “Top 10 Web Application Security Risks”

PEN testing tools:

  • OWASP ZAP

Deploy Code & IaC (Infrastructure as Code)

IaC is paramount in DevOps to avoid any manual work in customer environments and help with immutable infrastructure.

Popular IaC tools are:

  • Azure ARM Templates
  • Terraform
  • HELM
  • Private Repositories

In DevSecOps, a private repository is recommended to host the build dependencies, reference container images, container images for tools, and the built packages or application container images. This is to keep all the artifacts together in one centralized location, and the release pipeline can continue with deployments from there.
Some of the private repositories are:
JFrog
Docker Hub
Azure Container Registry (ACR)

Private Repository Scanning

As the pipeline requires security scanning, the repositories require scanning also. These tools scan for vulnerabilities in all packages and container artifacts stored in the repository. A scan report is being sent/notified for any issues.

Some artifact scanning tools are:

  • XRay
  • SonaType
  • Azure Monitor
  • Azure Security Center

Deploy

As the recommendation to deploy the security tools with container orchestration, the same recommendation goes to deployed applications. Containers provide high security with limited ways to be affected by attackers. Sidecar containers protect by continually monitoring applications with a container security stack built-in. Applications are scalable on a demand basis using Kubernetes and tools such as Kubectl; HELM packages are used to deploy and manage K8S clusters. ArgoCD is a declarative tool specifically for Kubernetes deployment in CI/CD pipeline.

Deployments to Azure could be:

  • Azure function app
  • Azure App Service
  • Azure Container Instance
  • Azure Kubernetes Service (AKS)
  • Open Shift in Azure
  • Monitoring/Alerting

Monitoring/Alerting

As the applications deployed and running in a cloud environment, it must be continuously monitored for attacks and identify any security vulnerabilities. For containers, these tools act as sidecar containers to regularly protect main containers from attacks, and some mitigate the issue. All these tools have built-in alert/notify operations team for immediate actions.

Monitoring/alerting tools:

  • Azure Monitor
  • Azure Security Center
  • Twistlock
  • Qualys
  • Aqua Security

So, all powered up with learning DevSecOps! Follow up back here for the next blog post in container-based deployments and containers scanning in the DevSecOps pipeline!

References for continuing your DevSecOps Journey