In January, AIS’ Steve Suing posted a great article titled “What to Know When Purchasing a COTS Product and Moving to the Cloud.” In his post, Steve discusses some things to consider such as security, automation, and integration. Springboarding off of those topics for this blog, I am going to discuss another major consideration: legacy data.

Many COTS implementations are done to modernize a business function that is currently being performed in a legacy application. There are some obvious considerations here. For example, when modernization a legacy app, you might ask:

  • How much of the business needs are met by purely out of the out-of-the-box functionality?
  • How much effort will be necessary to meet the rest of your requirements in the form of configuration and customization of the product or through changing your business process?

However, legacy data is a critical consideration that should not be overlooked during this process.

Dealing with Data When Retiring a Legacy System

If one of the motivators for moving to a new system is to realize cost savings by shutting down legacy systems, that means the legacy data likely needs to be migrated. If there is a considerable amount of data and length of legacy history, the effort involved in bringing that data into the new product should be carefully considered. This might seem trivial, with most products at least proclaiming to have a multitude of ways to connect to other systems but recent experience has shown me that the cost of not properly verifying the level of effort can far exceed the savings provided by a COTS product. These types of questions can help avoid a nightmare scenario:

  1. How will the legacy data be brought into the product?
  2. How much transformation of data is necessary, and can it be done with tools that already exist?
  3. Does the product team have significant experience of similarly legacy systems (complexity, amount of data, industry space, etc.) moving to the product?

The first and second conversations usually start by sitting down and talking to the product team and having them outline the common processes that take place to bring legacy data into the product. During these conversations, be sure both sides have technical folks in the room. This will ensure the conversations have the necessary depth to uncover how the process works.

Once your team has a good understanding of the migration methods recommended by the product team, start exploring some actual use cases related to your data. After exploring some of the common scenarios to get a handle on the process, jump quickly into some of the toughest use cases. Sure, the product might be able to bring in 80% of the data with a simple extract, transform, load (ETL), but what about that other 20%? Often legacy systems were created in a time long, long ago, and sometimes functionality will have been squished into them in ways that make the data messy. Also, consider how the business processes have changed over time and how the migration will address legacy data created as rules changed over the history of the system. Don’t be afraid to look at that messy data and ask the tough questions about it. This is key.

Your Stakeholders During Legacy Data Migration

Be sure those involved are not too invested in the product to decrease confirmation bias, the tendency to search for, interpret, and/or focus on information that will confirm that this is your silver bullet. This way they will ask the hard questions. Bring in a good mix of technical and business thought leaders. For instance, challenge the DBA’s and SME’s to work together to come up with things that will simply not work. Be sure to set up an environment in which people are rewarded for coming up with blockers and not demotivated by being seen as difficult. Remember every blocker you come up with in this early phase, could be the key to making better decisions, and likely have a major downstream impact.

The product of these sessions should be a list of tough use cases. Now it’s time to bring the product team again. Throw the use cases up on a whiteboard and see how the product team works through them. On your side of the table, be sure to include people that will be responsible for the effort of bringing the new system to life. With skin in the game, these people are much less likely to allow problems to be glossed over and drive a truly realistic conversation. Because this involves the tough questions, the exercise will likely take multiple sessions. Resist any pressure to get this done quickly, keeping in mind that a poor decision now, can have ripple efforts that will last for years.

After some of these sessions, both sides should have a good understanding of the legacy system, the target product, and some of the complexities of meshing the two. With that in place, ask the product team for examples of similar legacy systems they have tackled. This question should be asked up front, but it really cannot be answered intelligently until at least some of the edge cases of the legacy system are well understood. While the product team might not be able to share details due to non-disclosure agreements, they should be able to speak in specific enough terms to demonstrate experience. Including those involved in the edge case discovery sessions is a must. With the familiarity gained during those sessions, those are the people that are going to be most likely to make a good call on whether the experience being presented by the product team relates closely enough to your needs.

Using Lessons Learned to Inform Future Data Migration Projects

Have I seen this process work? Honestly, the answer is no. My hope is that the information I have presented, based on lessons learned from projects past helps others avoid some of the pitfalls I have faced and the ripple effects that could lead to huge cost overruns. It’s easy to lose sight of the importance of asking hard questions up front and continuing to do so in light of the pressure to get started. If you feel you are giving in to such pressure, just reach out to future you and ask them for some input.

And a final thought. As everyone knows, for complex business processes, COTS products don’t always perfectly align to your legacy system. There are two ways to find alignment. Customizing the system is the obvious route, although it often makes more sense to change business processes to align with a product you are investing in rather than trying to force a product to do what it was not designed for. If you find that this is not possible, perhaps you should be looking at another product. Keep in mind that if a product cannot be found that fits your business processes closely enough, it might make more financial sense to consider creating your own system from the ground up.

Azure DevOps provides a suite of tools that your development team needs to plan, create, and ship products. It comes in two flavors:

  • Azure DevOps Services – the SaaS option hosted by Microsoft.
  • Azure DevOps Server – the IaaS option hosted by you.

When comparing the two to decide which option enables your team to deliver the most value in the least amount of time, Azure DevOps Services is the clear winner, but velocity alone is not the only consideration for most government teams. The services you use must also be compliant with standardized government-wide security, authorization, and monitoring requirements.

Azure DevOps Services are hosted by Microsoft in Azure regions. At the time of this writing, you do not yet have the option to host Azure DevOps in an Azure Government region so you must use one of the available Azure Commercial regions. As a public service, it has also has not yet achieved compliance with any FedRAMP or DoD CC SRG audit scopes. This may sound like a non-starter, but as it states on the FedRAMP website, it depends on your context and how you use the product.

Depending on the services being offered, the third-party vendor does not necessarily have to be FedRAMP compliant, but there are security controls you must make sure they adhere to. If there is a connection to the third-party vendor, they should be listed in the System Security Plan in the Interconnection Table.

This is the first of two blog posts that will share solutions to common Azure DevOps Services concerns:

  1. In “Azure DevOps Services for Government: Access Control” (this post), I will cover common access control concerns.
  2. In “Azure DevOps Services for Government: Information Storage”, I will address common concerns with storing information in the commercial data centers that host Azure DevOps Services.

Managing User Access

All Azure DevOps Organizations support cloud authentication through either Microsoft accounts (MSA) or Azure Active Directory (AAD) accounts. If you’re using an MSA backed Azure DevOps organization, users will create their own accounts and will self-manage security settings, like multi-factor authentication. It is common for government projects to require a more centralized oversight of account management and policies.

The solution is to back your Azure DevOps Services Organization with an AAD tenant. An AAD backed Azure DevOps organization meets the following common authentication and user management requirements:

  • Administrators control the lifecycle of user accounts, not the users themselves. Administrators can centrally create, disable, or delete user accounts.
  • With AAD Conditional Access, administrators can create policies that allow or deny access to Azure DevOps based on conditions such as user IP location.
  • AAD can be configured to federate with an internal identity provider, which could be used to enable CAC authentication.

Controlling Production Deployment Pipelines

The value added by Continuous Delivery (CD) pipelines includes increasing team velocity and reducing the risks associated with introducing a change. To fully realize these benefits, you’ll want to design your pipeline to begin at the source code and end in production so that you can eliminate bottlenecks and ensure a predictable deployment outcome across your test, staging, and production environments.
It is common for teams to grant limited production access to privileged administrators, so some customers raise valid concerns with the idea of a production deployment pipeline when they realize that an action triggered by a non-privileged developer could initiate a process that ultimately changes production resources (ex: deployment pipeline triggered from a code change).

The solution is to properly configure your Azure DevOps pipeline Environments. Each environment in Azure DevOps represents a target environment of a deployment pipeline. Environment management is separate from pipeline configuration which creates a separation of duties:

  • Team members who define what a deployment needs to do to deploy an application.
  • Team members who control the flow of changes into environments.

Example

A developer team member has configured a pipeline that deploys a Human Resource application. The pipeline is called “HR Service”. You can see in the YAML code below, the developer intends to run the Deploy.ps1 scripts on the Production pipeline Environment.

Production Environment

If we review the Production environment configuration, we can see that an approval check has been configured. When the deployment reaches the stage that will attempt to run against the production environment, a member of the “Privileged Administrators” AAD group will be notified that deployment is awaiting their approval.

Approvals and Checks

Only the Privileged Administrators group has been given access to administer the pipeline environment, so the team member awaiting approval would not be able to bypass the approval step by disabling it in the environment configuration or in the pipelines YAML definition.

Security Production

By layering environment configuration with other strategies you’ll establish the boundaries needed to protect your environment but will also empower your development team to work autonomously and without unnecessary bottlenecks. Other strategies to layer include:

  • Governance enforces with Azure Policies
  • Deployment gates based on environment monitoring
  • Automated quality and security scans into your pipeline

It is also important for all team members involved with a release to be aware of what is changing so that you are not just automating a release over the wall of confusion. This is an example of how a product alone, such as Azure DevOps, is not enough to fully adopt DevOps. You must also address the process and people.

Deployment Pipeline Access to Private Infrastructure

How will Azure DevOps, a service on a commercial Azure region, be able to “see” my private infrastructure hosted in Azure Government? That’s usually one of the first questions I hear when discussing production deployment pipelines. Microsoft has a straight-forward answer to this scenario: self-hosted pipeline agents. Azure DevOps will have no line of sight to your infrastructure. The high-level process to follow when deploying an agent into your environment looks like this:

  • Deploy a virtual machine or container into your private network.
  • Apply any baseline configuration to the machine, such as those defined in the DISA’s Security Technical Implementation Guides (STIG).
  • Install the pipeline agent software on the machine and register the agent with an agent pool in Azure DevOps.
  • Authorize pipelines to use the agent pool to run deployments.

With this configuration, a pipeline job queued in Azure DevOps will now be retrieved by the pipeline agent over 443, pulled into the private network, and then executed.

Azure DevOps Configuration

Conclusion

In this post, I’ve introduced you to several practices you can use to develop applications faster without sacrificing security. Stay tuned for the next post in this series where we will discuss common concerns from Government clients around the storage of data within Azure DevOps.

In Part 1 of this blog series, we outlined the case for a Service Catalog driven approach for Enterprise DevOps. At AIS, after having worked with several of our Enterprise clients with their DevOps journey, we harvested our learnings in the form of a fully managed service catalog – AIS Service Catalog. AIS Service Catalog is a Microsoft Azure focused SaaS offering that is designed to jumpstart your DevOps efforts by bringing you the benefits of the service catalog driven approach.

ASC

A Service Catalog for Microsoft Azure

In Part 1 of this blog post series, we identified four key features of a Service Catalog that are fundamental to establishing DevOps in an enterprise. Let us briefly talk about how AIS Service Catalog realizes these features using Microsoft Azure specific building blocks. Read More…

Yesterday, Microsoft announced the general availability of its offering of Infrastructure as a Service (IaaS). They join an already-crowded market of IaaS providers, but this offering gives all companies the ability to offload workloads that have traditionally run in a company data center to the cloud. Welcome, Microsoft — the water is fine.

This announcement also represents a major chunk of Microsoft’s family of Azure offerings…and in my opinion, a stepping stone many companies simply must take in moving out of the traditional data center and into the cloud.  The following diagram shows the stepping stones out of the traditional data center:

Read More…