AIS is working with a large organization that wants to discover relationships between data and the business by iteratively integrating data from many sources into Azure Data Lake. The data will be analyzed by different groups within the organization to discover new factors that might affect the business. Data will then be published to the appropriate consumers using PowerBI.

In the initial phase, data lake ingests data from some of the Operational Systems. Eventually, data will be captured not only from all the organization’s systems but also from streaming data from IoT devices.  

Azure Data Lake 

Azure Data Lake allows us to store a vast amount of data of various types and structures. Data can be analyzed and transformed by Data Scientists and Data Engineers. 

The challenge with any data lake system is preventing it from becoming a data swamp. To establish an inventory of what is in a data lake, we capture the metadata such as origin, size, and content type during ingestion. We also have the Interface Control Document (ICD) from the Operational Systems that describe the data definition of the source data. 

Logical Zones

The data in the data lake is segregated into logical zones to allow logical and physical separation to keep the environment secure, organized and agile. As the data progress through the zones various transformation is performed. 

  • Landing Zone is a place where the original data files are stored untouched. No data is deleted from this zone, and access to this zone is limited.  
  • Raw Zone is a place where data quality validation is applied based on the rules defined in source ICDAny data filed validation moves to Error Zone. 
  • Curated Zone is a place where we store the cleansed and transformed data and ready for consumption. The transformation is done for different audiences, and within the Zone, folders will be created for each specialized change.  
  • Error Zone is a place where we store data that filed validation. A notification is sent to the registered data curators upon arriving new data.  
  • Metadata Zone is a place where we keep track of metadata of the source and the transformed data.Metadata Zone Organization

The source systems have security requirements that prevent access to sensitive data. When the folders are created, permissions are given to security groups in Azure Active Directory. The same security rules are applied to the subsequent folders.

Now that the data is in the data lake, we allow each consuming group to create their own transformation rules. The transformed data is then moved to the curated zone ready to be loaded to the Azure Data Warehouse.

Azure Data Factory

Azure Data Factory orchestrated the movement and transformation of data, as shown in the diagram below. When a file is dropped in the Landing Zone, the Azure Data Factory pipeline that consists of activities to Unzip, Validate, Transform, and Load the data into Data Warehouse.

The unzipping is performed by a custom code Azure Function activity rather than the copy activity’s decompress functionality. The out of box functionality of Azure Data Factory can be used to uncompressed only GZip, Deflate and BZip2 files but not Tar, Rar, 7Zip, Lzip.

The basic validation rules, such as data range, valid values, and reference data, are described in the ICD. A custom Azure Function activity was created to validate the incoming data.

Data is transformed using Spark activity in Azure Data Factory for each consuming user. Each consumer has a folder under the Curated Zone.

Data Processing Example

Tables in the Azure Data Warehouse were created based on the Curated zone by executing the Generate Azure Function activity to create data definition language (DDL). The script modifies the destination table if there is a new field added.

Finally, the data is copied to the destination tables to be used by end-users and warehouse designers.

In each step, we captured business, operational, and technical metadata to help us descript the data in the lake. The metadata information can be uploaded to a metadata management system in the future.

When Microsoft introduced pipelines as part of their Azure DevOps cloud service offering, we received the tools to add continuous integration (CI) and continuous delivery (CD) practices to our development processes. An Azure DevOps pipeline can be created in two ways: 1) The current generally available “classic” pipeline tooling, and 2) the new multi-stage YAML pipeline feature which is currently in preview.

Classic Pipelines

Classic pipelines achieve CI through Azure DevOps build pipelines. A build pipeline executes before a developer integrates code changes into a code base. The pipeline does things like execute a build task, run the unit tests and/or run static code analysis. It then either accepts or rejects the new changes based on the outcome of these tasks.

CD is achieved through Azure DevOps release pipelines. After the build pipeline has produced a build artifact, a release pipeline will publish the artifact to various environments for manual functional testing, user experience testing and quality assurance. When testers have thoroughly tested the deployed artifacts, the release pipeline can then push the artifacts to a production environment.

As powerful as these classic CI/CD pipelines are, they do have their drawbacks. Firstly, the tooling for creating build and release pipelines does not provide a unified experience. CI pipelines provide an intuitive GUI to create and visualize the integration steps…

Classic build pipeline editor

… and also allow you to define those very same steps in YAML:

YAML build pipeline editor

Release pipelines also provide a GUI to create and visualize the pipeline steps. The problem is that the interface is different from that of the build pipeline and does not allow YAML definitions:

Classic release pipeline editor

Multi-stage Pipelines

To resolve these discrepancies Microsoft introduced multi-stage pipelines. Currently in preview, these pipelines allow an engineer to define a build, release or a combined build and release pipeline as a single YAML document. Besides the obvious benefits gained through a unified development experience, there are many other good reasons to choose YAML over classic pipelines for both your builds and releases.

Since you commit YAML definitions directly to source control, you get the same benefits source control has been providing developers for decades. Here are the top 10 reasons (in no particular order) you should choose YAML for your next Azure DevOps pipeline:

1. History

Want to see what your pipeline looked like last month before you moved your connection strings to Azure KeyVault? No problem! Source control allows you to see every change ever make to your pipeline since the beginning of time.

2. Diff

Have you ever discovered an issue with your build but not known exactly when it started failing or why? Having the ability to compare the failing definition with the last known working definition can greatly reduce the recovery time.

3. Blame

Similarly, it can be useful to see who committed the bug that caused the failure and who approved the pull request. You can pull these team members into discussions on how best to fix the issue while ensuring that the original objectives are met.

4. Work Items

Having the ability to see what was changed is one thing but seeing why it was changed is another. By attaching a user story or task to each pipeline commit, you don’t need to remember the thought process that went into a particular change.

5. Rollback

If you discover that the pipeline change you committed last night caused a bad QA environment configuration, simply rollback to the last known working version. You’ll have your QA environment back up in minutes.

6. Everything As Code

Having your application, infrastructure and now build and release pipelines as code in the same source control repository gives you a complete snapshot of your system at any point in the past. By getting an older version of your repo, you can easily spin up an identical environment, execute the exact same pipelines and deploy the same code exactly as it was. This is an extremely powerful capability!

7. Reuse and Sharing

Sharing or duplicating a pipeline (or part thereof) is as simple as copy and paste. It’s just text so you can even email it to a colleague if desired.

8. Multiple Engineers

Modern CI/CD pipelines can be large and complex, and more than one engineer might modify the same YAML file, causing a conflict. Source control platforms solved this problem long ago and provide easy to use tools for merging conflicting changes. For better or worse, YAML definitions allow multiple engineers to work on the same file at the same time.

9. Peer Reviews

If application code peer reviews are important, so are pipeline peer reviews. The ability to submit a pull request before bringing in new changes allows team members to weigh in and provides an added level of assurance that the changes will perform as desired.

10. Branching

Have a crazy idea you want to try out? Create a new branch for it and trigger a pipeline execution from that branch. If your idea doesn’t pan out, simply delete the branch. No harm done.

Though still in preview, the introduction of fully text-based pipeline definitions that can be committed to source control provides benefits that cannot be achieved with classic GUI-based definitions, especially for larger organizations. Be sure to consider YAML for your next Azure DevOps pipeline implementation.

This blog is a follow on about Azure Cognitive Services, Microsoft’s offering for enabling artificial intelligence (AI) applications in daily life. The offering is a collection of AI services with capabilities around speech, vision, search, language, and decision.

In Azure Cognitive Services Personalizer: Part One, we discussed the core concepts and architecture of Azure Personalizer Service, Feature Engineering, its relevance, and its importance.

In this blog, Part Two, we will go over a couple of use cases in which Azure Personalizer Service is implemented. We will look at features used, reward calculation, and their test run result. Stay tuned for Part Three, where we will list out recommendations and capacities for implementing solutions using Azure Personalizer Service.

Use Cases and Results

Two use cases implemented using Personalizer involves the ranking of content for each user of a business application.

Use Case 1: Dropdown Options

Different users of an application with manager privileges would see a list of reports that they can run. Before Personalizer was implemented, the list of dozens of reports was displayed in alphabetical order, requiring most of the managers to scroll through the lengthy list to find the report they needed. This created a poor user experience for daily users of the reporting system, making for a good use case for Personalizer. The tooling learned from the user behavior and began to rank frequently run reports on the top of the dropdown list. Frequently run reports would be different for different users, and would change over time for each manager as they get assigned to different projects. This is exactly the situation where Personalizer’s reward score-based learning models come into play.

Context Features

In our use case of dropdown options, the context features JSON is as below with sample data

{
    "contextFeatures": [
        { 
            "user": {
                "id":"user-2"
            }
        },
        {
            "scenario": {
                "type": "Report",
                "name": "SummaryReport",
                "day": "weekend",
                "timezone": "est"
            }
        },
        {
            "device": {
                "mobile":false,
                "Windows":true,
                "screensize": [1680,1050]
            }
        }
    ]
}

Actions (Items) Features

Actions were defined as the following JSON object (with sample data) for this use case

{
    "actions": [
    {
        "id": "Project-1",
        "features": [
          {
              "clientName": "Client-1",
              "projectManagerName": "Manager-2"
          },
          {

                "userLastLoggedDaysAgo": 5
          },
          {
              "billable": true,
              "common": false
          }
        ]
    },
    {
         "id": "Project-2",
         "features": [
          {
              "clientName": "Client-2",
              "projectManagerName": "Manager-1"
          },
          {

              "userLastLoggedDaysAgo": 3
           },
           {
              "billable": true,
              "common": true
           }
        ]
    }
  ]
}

Reward Score Calculation

Reward score was calculated based on the actual report selected (from the dropdown list) by the user from the ranked list of reports displayed with the following calculation:

  • If the user selected the 1st report from the ranked list, then reward score of 1
  • If the user selected the 2nd report from the ranked list, then reward score of 0.5
  • If the user selected the 3rd report from the ranked list, then reward score of 0
  • If the user selected the 4th report from the ranked list, then reward score of – 0.5
  • If the user selected the 5th report or above from the ranked list, then reward score of -1

Results

View of the alphabetically ordered report names in the dropdown before personalization:

alphabetically ordered report names in the dropdown before personalization

View of the Personalizer ranked report names in the dropdown for the given user:

Azure Personalizer ranked report names based on frequency

Use Case 2: Projects in Timesheet

Every employee in the company logs a daily timesheet listing all of the projects the user is assigned to. It also lists other projects, such as overhead. Depending upon the employee project allocations, his or her timesheet table could have few to a couple of dozen active projects listed. Even though the employee is assigned to several projects, particularly at lead and manager levels, they don’t log time in more than 2 to 3 projects for a few weeks to months.

Before personalization, the projects in the timesheet table were listed in alphabetical order, again resulting in a poor user experience. Even more troublesome, frequent user errors caused the accidental logging of time in the incorrect row. Personalizer was a good fit for this use case as well, allowing the system to rank projects in the timesheet table based on time logging patterns for each user.

Context Features

For the Timesheet use case, context features JSON object is defined as below (with sample data):

{
    "contextFeatures": [
        { 
            "user": {
                "loginid":"user-1",
                "managerid":"manager-1"
		  
            }
        },
        {
            "scenario": {
                "type": "Timesheet",
                "day": "weekday",
                "timezone": "ist"
            }
        },
        {
            "device": {
                "mobile":true,
                "Windows":true,
                "screensize": [1680,1050]
            }
        }
     ]
}

Actions (Items) Features

For the timesheet use case, the Actions JSON object structure (with sample data) is as under:

{
    "actions": [
    {
        "id": "Project-1",
        "features": [
          {
              "clientName": "Client-1",
              "userAssignedForWeeks": "4-8"
          },
          {

              "TimeLoggedOnProjectDaysAgo": 3
          },
          {
              "billable": true,
              "common": false
          }
        ]
    },
    {
         "id": "Project-2",
         "features": [
          {
              "clientName": "Client-2",
              "userAssignedForWeeks": "8-16"
          },
          {

              " TimeLoggedOnProjectDaysAgo": 2
           },
           {
              "billable": true,
              "common": true
           }
        ]
    }
  ]
}

Reward Score Calculation

The reward score for this use case was calculated based on the proximity between the ranking of projects in timesheet returned by the Personalizer and the actual projects that the user would log time as follows:

  • Time logged in the 1st row of the ranked timesheet table, then reward score of 1
  • Time logged in the 2nd row of the ranked timesheet table, then reward score of 0.6
  • Time logged in the 3rd row of the ranked timesheet table, then reward score of 0.4
  • Time logged in the 4th row of the ranked timesheet table, then reward score of 0.2
  • Time logged in the 5th row of the ranked timesheet table, then reward score of 0
  • Time logged in the 6th row of the ranked timesheet table, then reward score of -0.5
  • Time logged in the 7th row or above of the ranked timesheet table, then reward score of -1

The above approach to reward score calculation considers that most of the time users would not need to fill out their timesheet for more than 5 projects at a given time. Hence, when a user logs time against multiple projects, the score can be added up and then capped between 1 to -1 while calling Personalizer Rewards API.

Results

View of the timesheet table having project names alphabetically ordered before personalization:

project names alphabetically ordered before Azure personalization

View of the timesheet table where project names are ordered based on ranking returned by Personalization Service:

timesheet table ordered by Azure Personalization Service

Testing

In order to verify the results of implementing the Personalizer in our selected use cases, unit tests were effective. This method was helpful in two important aspects:

  1. Injecting the large number of user interactions (learning loops)
  2. In simulating the user behavior towards a specific pattern

This provided an easy way to verify how Personalizer reflects the current and changing trends injected via Unit Tests in the user behavior by using reward scores and exploration capability. This also enabled us to test different configuration settings provided by Personalizer Service.

Test Run 1

This 1st test run simulated different user choices with different explorations settings. The test results show the number of learning loops that started reflecting the user preference from intermittent to a consistent point.

Unit Test Scenario
Learning Loops, Results and Exploration Setting
User selection of Project-A Personalizer Service started ranking Project-A at the top intermittently after 10 – 20 learning loops and ranked it consistently at the top after 100 learning loops with exploration set to 0%
User selection of Project-B Personalizer Service started reflecting the change in user preference (from Project-A to Project-B) by ranking Project-B at the top intermittently after 100 learning loops and ranked it consistently at the top after 1200 learning loops with exploration set to 0%
User selection of Project-C

 

Personalizer Service started reflecting the change in user preference (from Project-B to Project-C) by ranking Project-C at the top intermittently after 10 – 20 learning loops and ranked it almost consistently at the top after 150 learning loops with exploration set to 50%

 

Personalizer adjusted with the new user preference quicker when exploration was utilized.

 

User selection of Project-D

 

Personalizer Service started reflecting the change in user preference (from Project-C to Project-D) by ranking Project-D at the top intermittently after 10 – 20 learning loops and ranked it almost consistently at the top after 120 learning loops with exploration set to 50%

 

Test Run 2

In this 2nd test run, the impact of having and removing sparse features (little effective features) is observed.

Unit Test Scenario
Learning Loops, Results and Exploration Setting
User selection of Project-E Personalizer Service started reflecting the change in user preference (from Project-D to Project-E) by ranking Project-E at the top intermittently after 10 – 20 learning loops and ranked it almost consistently at the top after 150 learning loops with exploration set to 20%
User selection of Project-F Personalizer Service started reflecting the change in user preference (from Project-E to Project-F) by ranking Project-F at the top intermittently after 10 – 20 learning loops and ranked it almost consistently at the top after 250 learning loops with exploration set to 20%
User selection of Project-G Two less effective features (sparse features) of type datetime were removed. Personalizer Service started reflecting the change in user preference (from Project-F to Project-G) by ranking Project-G at the top intermittently after 5 – 10 learning loops and ranked it almost consistently at the top after only 20 learning loops with exploration set to 20%

 

User selection of Project-H

 

Two datetime sparse features were added back. Personalizer Service started reflecting the change in user preference (from Project-G to Project-H) by ranking Project-H at the top intermittently after 10 – 20 learning loops and ranked it almost consistently at the top after 500 learning loops with exploration set to 20%

 

Thanks for reading! In the next part of this blog post, we will look at the best practices and recommendations for implementing Personalizer solutions. We will also touch upon the capacities and limits of the Personalizer service at present.

I recently had the privilege and opportunity to attend this year’s DEF CON conference, one of the world’s largest and most notable hacker conventions, held annually in Las Vegas. Deciding what talks and sessions to attend can be a logistics nightmare for a conference that has anywhere between 20,000 – 30,000 people in attendance, but I pinpointed the ones that I felt would be beneficial for myself and AIS.

During the conference, Tanya Janca, a cloud advocate for Microsoft, and Teri Radichel from 2nd Sight Lab did a presentation on “DIY Azure Security Assessment” that dove into how to verify the security of your Azure environments. More specifically they went into detail on using Azure Security Center, and setting scope, policies, and threat protection. With this post, I want to share what I took away from the talk I found most helpful.

Security in Azure

Security is a huge part of deploying any implementation in Azure and ensuring fail-safes are in place to stop attacks before they occur. I will break down the topics I took away that can help you better understand and perform your own security assessment in Azure along with looking for vulnerabilities and gaps.

The first step in securing your Azure environment is to find the scope at which you are trying to assess and protect. This could also include things external to Azure, such as hybrid solutions with on-premises. These items include the following:

  • Data Protection
  • Application Security
  • Network Security
  • Access Controls
  • Cloud Security Controls
  • Cloud Provider Security
  • Governance
  • Architecture

Second, is using the tools and features within Azure in order to accomplish this objective. Tanya and Teri started out by listing a few key features that every Azure implementation should use. This includes:

  • Turning on Multi-Factor Authentication (MFA)
  • Identity and Access Management (IAM)
    • Roles in Azure AD
    • Policies for access
    • Service accounts
      • Least privilege
    • Account Structure and Governance
      • Management Groups
      • Subscriptions
      • Resource Groups

A key item I took away from this section was allowing access at the “least privileged” level using service accounts, meaning only the required permissions should be granted when needed using accounts that are not for administrative use. Along with tightening access, it’s also important to understand at what level to manage this governance. Granting access at a management group level will cast a wider and more manageable net. A more defined level, such as a subscription level, could help with segregation of duties but this is heavily based on the current landscape of your groups and subscription model.

The Center for Internet Security (CIS)

So maybe now you have an understanding of what scope you want to assess the security of your Azure environment at, but do not know where to start. This is where The Center for Internet Security (CIS) can come into play. CIS is crowd-sourced security for best practices and threat prevention which includes members such as corporations, governments, and academic institutions. It was initially intended for on-premises use. However, as the cloud has grown so has the need for increased security. CIS can help you decide what best practices you should follow based on known threat vectors; these include 20 critical controls broken down into the following 3 sections:

Basic Center for Internet Security Controls

Examples of these CIS control practices could be:

  • Inventory and Control of Hardware Assets by utilizing a software inventory tool
  • Controlled Use of Administrative Privileges by setting up alerts and logs

An additional feature is the CIS Benchmark which has recommendations for best practices in various platforms and services, such as Microsoft SQL or IIS. Plus it’s free! Another cool feature that CIS offers is within the Azure Marketplace. They have pre-defined system images that are already hardened for these best practices.

CIS Offers in Azure Marketplace

The figure below shows an example benchmark for control practice that gives you the recommendation to “Restrict access to Azure AD administration portal.” This will then output audits that show what steps need to be taken to be within the scope of that best practice.

Control Practice to Restrict access to Azure AD administration portal

Azure Security Center (ASC)

In this next section, I detail the features of Azure Security Center (ASC) that I took away from this presentation and how to get started using ASC. The figure below is of the dashboard. As you can see, there are a lot of options inside the ASC dashboard, including sections such as Policy & Compliance and Resource Security Hygiene. The settings inside of those can dive deeper into resources all the way down to the VM or application level.

Azure Security Center Dashboard

Making sure you have ASC turned on should be your first step when implementing the features within it. The visuals you get in ASC are very helpful, including things like subscription coverage and your security score. Policy management is also a feature with ASC to use pre-defined and custom rules to keep your environment within the desired compliance levels.

Cloud Networking

Your network design in Azure plays a crucial role in securing against incoming attacks, including more than just closing ports. When you build a network with security in mind you not only limit your attack surface but also make spotting vulnerabilities easier; all while making it harder for attackers to infiltrate your systems. Using Network Security Groups (NSGs) and routes can also help by allowing only the required ports. You can also utilize Network Watcher to test these effective security rules. Other best practices include not making RDP, SSH, and SQL accessible from the internet. At a higher-level, below are some more networking features and options to secure Azure including:

  • Azure Firewall
    • Protecting storage accounts
    • Using logging
    • Monitored
  • VPN/Express Route
    • Encryption between on-premises and Azure
  • Bastion Host
    • Access to host using jump box feature
    • Heavy logging
  • Advanced Threat Protection
    • Alerts of threats in low, medium and high severity
    • Unusually activities such as large amounts of storage files copied
  • Just in Time (JIT)
    • Access host only when needed in a configured time frame.
    • Select IP Ranges and ports
  • Azure WAF (Web Application Firewall)
    • Layer 7 firewall for applications
    • Utilize logging and monitoring

An additional design factor to consider is the layout of your network architecture. Keeping all your resources divided into tiers can be a great security practice to minimize risk to each component. An example would be utilizing a three-tier design. This design divides a web application into three-tier (VNets). In the figure below you can see a separate web tier, app tier, and data tier. This is much more secure because the front-end web tier can still access the app tier but cannot directly talk to the data tier which helps to minimize risk to your data.

Three Tier Network Architecture: web tier, app tier, and data tier

Logging and Monitoring

Getting the best data and analytics to properly monitor and log your data is an important part of assessing your Azure environment. For those in security roles, liability is an important factor in the ‘chain of custody’. When handling security incidents, extensive logging is required to ensure you understand the full scope of the incident. This includes having logging and monitoring turned on for the following recommended items:

  • IDS/IPS
  • DLP
  • DSN
  • Firewall/WAF
  • Load Balancers
  • CDN

The next possible way to gather even more analytics is the use of a SEIM (Security Information and Event Management) like Azure Sentinel. This just adds another layer of protection to collect, detect, investigate, and respond to threats from on-premises to multi-cloud vendors. An important note of this is to make sure you tune your SEIM, so you are detecting the threats accurately and not diluting the alerts with false positives.

Advanced Data Security

The final point I want to dive into is Advanced Data Security. The protection of data in any organization should be at the top of their list of priorities. Beginning by classifying your data is an important first step to know the sensitivity of your data. This is where Data Discovery & Classification can help in labeling the sensitivity of your data. Next is utilizing the vulnerability assessment scanning which helps assess the risk level of your databases and minimize leaks. Overall, these cloud-native tools are just another great way to help secure your Azure environment.

Conclusion

In closing, Azure has a plethora of tools at your disposal within the Azure Security Center to do your own security assessment and protect yourself, your company, and your clients from future attacks. The ASC can become your hub to define and maintain a compliant security posture for your enterprise. Tanya and Teri go into great detail the steps to take and even supply a checklist you can follow yourself to assess an Azure environment.

Checklist

  1. Set scope & only test what’s in scope
  2. Verify account structure, identity, and access control
  3. Set Azure policies
  4. Turn on Azure Security Center for all subs
  5. Use cloud-native security features – threat protection and adaptive controls, file integrity monitoring, JIT, etc.
  6. Follow networking best practices, NSGs, routes, access to compute and storage, network watcher, Azure Firewall, Express Route and Bastion host
  7. Always be on top of alerts and logs for Azure WAF and Sentinel
  8. VA everything, especially SQL databases
  9. Encryption, for your disk and data (in transit and rest)
  10. Monitor all that can be monitored
  11. Follow the Azure Security Center recommendations
  12. Then call a Penetration Tester

I hope you found this post to be helpful and make you, your company, and your clients’ experience on Azure more secure. For the full presentation, including a demo on Azure Security Center, check out this link. 

Lift n Shift Approach to Cloud Transformation

What does it mean to Rehost an application?

Rehosting is an approach to migrating business applications hosted in on-premises data center environments to the cloud by moving the application “as-is,” with little to no changes to the business functions performed by the application. It’s a faster, less resource-intensive migration approach that gets your apps into the cloud without much code modification. It is often a good first step to cloud transformation.

Organizations with applications that were initially developed for an on-premises environment commonly look to rehosting to take advantage of cloud computing benefits. These benefits may include increased availability and networking speeds, reduced technical debt, and a pay-per-usage cost structure. When defining your cloud migration strategy, it’s essential to analyze all migration approaches, such as re-platforming, refactoring, replacing, and retiring.

CHECK OUT OUR WHITEPAPER & LEARN ABOUT CLOUD-BASED APP MODERNIZATION APPROACHES

Read More…

As we think about services that Azure can offer, we often think about apps (e.g., App Services, AKS, and Service Fabric) and data (e.g., Azure Storage, Data Bricks, and Azure Data Lake). It turns out that you can also leverage Azure networking, also known as as a Network-as-a-Service (NaaS). Network is a basic building block for all app and data services in Azure. But, by NaaS, I am explicitly talking about leveraging Azure networking in a *standalone* manner. Allow me to explain what I mean: In the picture below, you will see a representation of the Azure global footprint. It is so vast that it includes 100K+ miles of fiber and subsea cables, and 130 edge locations connecting over 50 regions worldwide. Think of NaaS as a way to tap into Azure’s global infrastructure to improve network performance and resilience of your applications, regardless of whether the apps are hosted in Azure or not.

Azure's Global Footprint

Let’s discuss two specific Azure networking Services that offer NaaS capabilities. Note, there are other services like Azure Firewall – think firewall as a service – that can fall in the NaaS category. However, I am limiting my discussion to two services – Azure Front Door Service and Azure Virtual WAN Service. In my opinion, these services closely align with a focus on leveraging Azure network infrastructure and services in a standalone manner.

Azure Front Door Service Icon

Azure Front Door Service

Azure Front Door service allows you to define global routing for your applications that optimize performance and resilience. Front Door is a layer 7 (HTTP/HTTPS) service. Please refer to the diagram

below for a high-level view of how Front Door works – you can advertise your application’s URL using the anycast protocol. This way, traffic directed towards your application will get picked up by the “closest” Azure Front Door service and routed to your application hosted in Azure on-premises – for applications hosted outside of Azure, the traffic will traverse the Azure network to the point of exit closest to the location of the app.

The primary benefit of using the Azure Front Door service is to improve the network performance by routing over the Azure backbone (instead of the long-haul public internet). It turns out there are several secondary benefits to highlight: You can increase the reliability of your application by having Front Door provide instant failover to a backup location. The Azure Front Door service uses smart health probes to check for the health of your application. Additionally, Front Door offers SSL termination and certificate management, application security via Web Application Firewall, and URL based routing.

Routing Azure Front Door

Azure Virtual WAN

Azure Virtual WAN

Azure Virtual WAN offers branch connectivity to, and through, Azure. In essence, think of Azure regions as hubs, that along with the Azure network backbone can help you establish branch-to-VNet and branch-to-branch connectivity.

You are probably wondering how Virtual WAN relates to existing cloud connectivity options like point-to-site, site-to-site, and express route. Azure WAN brings together the above connectivity options into a single operational interface.

Azure Virtual Wan Branch Connectivity

The following diagram illustrates a client’s virtual network overlay over the Azure backbone. The Azure WAN virtual hub is located in the Western Europe region. The virtual hub is a managed virtual network, and in turn, enables connectivity to VNets in Western Europe (VNetA and VNetB) and an on-premises branch office (testsite1) connected via site-to-site VPN tunnel over IPSec. An important thing to note is that the site-to-site connection is hooked to the virtual hub and *not* directly to the VNet (as is the case with a virtual network gateway).

Virtual Network Overlay Azure Backbone

Finally, you can work with one of many Azure WAN partners to automate the site-to-site connection including setting up the branch device (VPN/SD-WAN – software defined wide area network) that automates the connectivity setup with Azure.

I just returned from Microsoft BUILD 2019 where I presented a session on Azure Kubernetes Services (AKS) and Cosmos. Thanks to everyone who attended. We had excellent attendance – the room was full! I like to think that the audience was there for the speaker 😊 but I’m sure the audience interest is a clear reflection of how popular AKS and Cosmos DB are becoming.

For those looking for a 2-minute overview, here it is:

In a nutshell, the focus was to discuss the combining Cloud-Native Service (like AKS) and a Managed Database

Microsoft Build Session Architecting Cloud-Native Apps with AKS and Cosmos DB Slide Deck

We started with a discussion of Cloud-Native Apps, along with a quick introduction to AKS and Cosmos. We quickly transitioned into stateful app considerations and talked about new stateful capabilities in Kubernetes including PV, PVC, Stateful Sets, CSI, and Operators. While these capabilities represent significant progress, they don’t match up with external services like Cosmos DB.

Microsoft Build Session Architecting Cloud-Native Apps with AKS and Cosmos DB Slide Deck

Microsoft Build Session Architecting Cloud-Native Apps with AKS and Cosmos DB Slide Deck Cloud Native Tooling

One option is to use Open Service Broker – It allows Kubernetes hosted services to talk to external services using cloud-native tooling like svcat (Service Catalog).

Microsoft Build Session Architecting Cloud-Native Apps with AKS and Cosmos DB Slide Deck svcat

Microsoft Build Session Architecting Cloud-Native Apps with AKS and Cosmos DB Slide Deck SRE

External services like Cosmos DB can go beyond cluster SRE and offer “turn-key” SRE in essence – Specifically, geo-replication, API-based scaling, and even multi-master writes (eliminating the need to failover).

Microsoft Build Session Architecting Cloud-Native Apps with AKS and Cosmos DB Slide Deck Mutli Master Support

Microsoft Build Session Architecting Cloud-Native Apps with AKS and Cosmos DB Slide Deck Configure Regions

Microsoft Build Session Architecting Cloud-Native Apps with AKS and Cosmos DB Slide Deck Portability

Since the Open Service Broker is an open specification, your app remains mostly portable even when you move to one cloud provider to another. OpenService Broker does not deal with syntactic differences, say connection string prefix difference between cloud providers.  One way to handle these differences is to use Helm.

Learn more about my BUILD session:

Here you can find the complete recording of the session and slide deck: https://mybuild.techcommunity.microsoft.com/sessions/77138?source=sessions#top-anchor

Additionally, you can find the code for the sample I used here: https://github.com/vlele/build2019 

WORK WITH THE BRIGHTEST LEADERS IN SOFTWARE DEVELOPMENT

As developers, we spend a lot of time developing APIs. Sometimes it’s to expose data that we’ve transformed or to ingest data from other sources. Coincidentally, more and more companies are jumping into the realm of API Management—Microsoft, Google, MuleSoft and Kong all have products now that provide this functionality. With this much investment from the big players in the tech industry, API management is obviously a priority. Now, why would anyone want to use an API Management tool?

The answer is simple: It allows you to create an API Gateway that you can load all your APIs into, providing a single source to query and curate. API Management makes life as an admin, a developer, and a consumer easier by providing everything for you in one package.

Azure API Management

Azure API Management logoWhat does Azure API Management provide? Azure API Management (APIM) is a cloud-based PaaS offering available in both commercial Azure and Azure Government. APIM provides a one-stop-shop for API authority, with the ability to create products, enforce policies, and utilize a robust developer portal.

Not only can API Management integrate seamlessly with your existing Azure infrastructure, but it can also manage APIs that exist on-prem and in other clouds. APIM is also available in both the IL4 and IL5 environments in Azure Government, which allows for extensibility and management for those working in the public sector.

APIM leverages a few key concepts to provide its functionality to you as a developer, including:

  • Products
  • Policies
  • Developer Portal

From providing security to leveraging rate-limiting and abstraction, Azure API Management does it all for API consolidation and governance in Azure. Any API can be ingested, and it gets even easier when APIs follow the OpenAPI Format.

What Are Products?

Products are a layer of abstraction provided inside APIM. Products allow you to create subsets of APIs that are already ingested into the solution—allowing you to overlap the use of APIs while restricting the use of individual collections of APIs. This level of compartmentalization allows you to not only separate your APIs into logical buckets but also enforce rules on these products separately, providing one more layer of control.

Product access is very similar to Azure RBAC—with different groups created inside of the APIM instance. These groups are yet another way for APIM admins to encapsulate and protect their APIs, allowing them to add users already associated to the APIM instance into separate subsets. Users can also be members of multiple groups, so admins can make sure the right people have access to the right APIs stored in their APIM instance.

What Are Policies?

Policies are APIM’s way of enforcing certain restrictions and providing a more granular level of control. There is an entire breadth of policies available in APIM, which range from simply disallowing usage of the API after calling it five times, to authentication, logging, caching, and transformation of requests or responses from JSON to XML and vice versa. Policies are perhaps the most powerful function of APIM and drive the control that everyone wants and need. Policies are written in XML and can be easily edited within the APIM XML Editor. Policies can also leverage C# 7 Syntax, which brings the power of the .NET Framework to your APIM governance.

What Is the Developer Portal?

The Azure API Management Developer Portal is an improved version of the Swagger documentation that’s generated when you use the OpenAPI spec. The Developer Portal provides an area for developers to readily see APIs, products, and associated applications. The Portal also provides sample request bodies (no more guessing API request structures!) and responses, along with code samples in many different languages.

Finally, the portal also allows you to try API calls with customized request bodies and headers, so you have the ability to see exactly what kind of call you want to make. Along with all that functionality, you can also download your own copy of the OpenAPI Spec for your API after it’s been ingested into your instance.

Why Should I Use APIM?

Every business should be using some form of API Management. You’ll be providing yourself a level of control previously not available. By deploying an API Gateway, that extra layer of abstraction allows for much tighter control of your APIs. Once an API has been ingested, APIM provides many additional functionalities.

First, you can match APIs to products, providing a greater level of compartmentalization. Second, you can add different groups to each product, with groups being subsets of users (i.e. Back-end Devs, Billing Devs, etc.). Third, you automatically generate a robust developer portal, which provides all of the functionality of the Swagger portal, but with added features, such as code snippets.  Finally, APIM also has complete integration with Application Insights in commercial Azure, providing access to a world-class logging and visualization tool.

Azure API Management brings power to the user, and no API should be left out.

There are many times when you want to move to Azure but don’t have the liberty of re-platforming the database due to the dependency of applications. Don’t despair, there are still ways to move your workload. Azure has support for Oracle! We previously showed a way to almost run an Oracle Database as a Service on Azure, but if you have a line of business (LOB) application or an application that requires support for high availability, it is also possible:  Presenting Azure support for Oracle disaster recovery including Oracle Data Guard.

Oracle Support on Azure

Oracle supports running Oracle DB 12.1 Standard and Enterprise editions in Azure on virtual machine (VM) images based on Oracle Linux.   Oracle has guaranteed license mobility from on-premises to Azure. “These images are considered “Bring Your Own License” and as such you will only be charged for compute, storage, and networking costs incurred by running a VM. It is assumed you are properly licensed to use Oracle software and that you have a current support agreement in place with Oracle.

Licensing Details

Microsoft Azure is an ‘Authorized Cloud Environment.’   Under this program, “count two vCPUs as equivalent to one Oracle Processor license if hyper-threading is enabled, and one vCPU as equivalent to one Oracle Processor license if hyper-threading is not enabled.” Please note that unlimited license agreements (ULAs) may also be used.

High Availability

Any conversation about this topic requires us to look at the recovery time objective (RTO) and recovery point objective (RPO) of an application. In our application, this is how long the database can be down as well as the maximum amount of data loss that can be tolerated. Oracle defines well-known reference architectures based on RTO & RPO objectives.

Assuming we require comprehensive high availability (HA) and disaster recovery (DR), real-time failover and zero/near-zero data loss, an implementation using Oracle Data Guard will be pursued.

Oracle Data Guard

Data Guard is one offering from Oracle that ensures high availability, data protection, and disaster recovery for enterprise data. It uses a standby database (exact replica) to survive outages of any kind and data corruptions. This technology is available in Oracle Enterprise Edition.   Oracle also provides, as a separately licensed product, Oracle Active Data Guard; this technology allows for real-time data protection and disaster recovery and expands capabilities.

A few key features from Oracle regarding Oracle Data Guard are:

  • Fast redo transport for best recovery point objective (RPO), fast apply performance for best recovery time objective (RTO)
  • Fast failover to a standby database to maintain availability should the primary database fail for any reason
  • Automatic or automated (depending upon configuration) re-synchronization of a failed primary database, quickly converting it to a synchronized standby database after a failover occurs.
  • Reduction of planned downtime by utilizing a standby database to perform maintenance in a rolling fashion

Oracle Real Application Clusters (RAC) was also considered, but it is not supported on any cloud environment other than Oracle’s.

Oracle Data Guard on Azure

Microsoft recommends — for the best performance of Oracle DB production workloads on Azure — to be sure to properly size the VM image and use Managed Disks that are backed by Premium Storage.

Azure provides the M-Series virtual machines that are ideal for extremely large databases or other applications that benefit from high vCPU counts and large amounts of memory.

Azure also provides a Managed Disk offering called Ultra SSD that can scale performance up to 160,000 IOPS. IOPS are “an input/output performance measurement used to characterize computer storage devices.” Database performance is often constrained by the performance of the underlying storage; therefore, utilizing the Ultra SSD offering, you can be sure to minimize this concern.

Get Your Oracle Resources into the Cloud, and Fast! Request an Oracle Migration Assessment from AIS Today.

Another key concept to enable this high availability scenario is to leverage Azure availability sets for the virtual machines. Availability sets are a fundamental technology that makes our scenario possible.

  • Each virtual machine in your availability set is assigned an update domain and a fault domain by the underlying Azure platform.
  • Update domains allow for a virtual machine to recover before maintenance on another virtual machine is initiated on a different update domain.
  • Fault domains define the group of virtual machines that share a common power source and network switch.
  • Managed disks provide better reliability for availability sets by ensuring that the disks of VMs in an availability set are sufficiently isolated from each other to avoid single points of failure.

Availability zones, an alternative to availability sets, expand the level of control you have to maintain the availability of the database VMs. With Availability Zones, Azure offers a 99.99% VM uptime Service Level Agreement (SLA).

Lastly, Azure Site Recovery (ASR) provides an additional level of disaster recovery via its ability to orchestrate replication, perform disaster recovery testing, and run failovers and failback; ASR is fully compatible with Oracle Data Guard on Azure.

A logical architecture using a virtual network (VNET) and subnets (isolation & projection), Oracle DB VMs, premium storage, and Oracle Data Guard is presented below.

Diagram of VM architecture

AIS recently participated in a joint proof of concept (POC) where this architecture was deployed with M128 VMs and premium storage and achieved the following goals:

  • ~35TB Database (Oracle Table Limit)
  • ~30-45 second failover under user load of 5000- 15,000 concurrent users with Zero Data Loss
  • Active Data Guard Running in Max Availability (SYNC, Successful Failover)
  • Can handle both planned and unplanned failovers
  • Achieve 120,000 IOPS
  • Can be further secured using Network Security Groups (NSG) and Application Security Groups (ASG)

Looking to Retire Your Oracle Hardware and Migrate to the Cloud? Contact AIS Today to Discuss Your Options.

Let’s say you are trying to move an Oracle database to Azure, but don’t want to go down the route of creating an Oracle Database in an Azure VM for obvious reasons: You don’t want to be responsible for maintaining the VM’s availability, hotfixes patching, etc. At the same time, let’s say you do want to take advantage of a fully-managed persistence service that offers local and geo-redundancy and the ability to create snapshots to protect against accidental deletes or data corruption.

It turns out that the latest advancements in Azure Container Instances (ACI), combined with the ability to deploy them in a VNET, can get you close.

Get Your Oracle Resources into the Cloud, and Fast! Request an Oracle Migration Assessment from AIS Today.

Let’s start by reviewing the architecture.

Architecture diagram

We can host an Oracle DB container image inside Azure Container Instances (ACI). ACI is a container-as-a-service offering that removes the need to manage the underlying virtual machines. It also eliminates the need for setting up our orchestrator. Additionally, ACI-hosted containers (Linux only for now) are placed in a delegated subnet. This allows the Azure container instance to be available from inside a VNET without the need to open a public endpoint.

Finally, the data files (persistent) aspect of the database resides in Azure Files, which removes the need to manage our durable storage since Azure Files takes care of the local and geo-redundancy. Additionally, Azure Files can take snapshots, allowing us a point-in-time restore ability.

(Azure Files also support Virtual Network service endpoints that allow for locking down access to the resources within the VNET.)

ACI also offers fast start times, plus policy-based automatic restarting of the container upon failure.

Here are the three steps to get this setup working:

Step One

Create the ACI hosting Oracle Database Server 12.2.01 that mounts an Azure File share and is connected to a delegated subnet.

az container create -g <> --name <> --image registry-1.docker.io/store/oracle/database-enterprise:12.2.0.1 --registry-username <> --registry-password <> --ports 1521 5500 --memory 8 --cpu 2 --azure-file-volume-account-name <> --azure-file-volume-account-key <> --azure-file-volume-share-name <> --azure-file-volume-mount-path /ORCL --vnet <> --vnet-address-prefix <> --subnet <> --subnet-address-prefix <>

Step Two

This step is a workaround that could be eliminated if we had access to the Docker file used to create this image.  We are essentially copying /oradata files containing the control files, data files, etc. to the Azure file share.

mkdir -p /u02/app/oracle/oradata/ORCL; cp -r /u02/app/oracle/oradata/ORCLCDB/. /u02/app/oracle/oradata/ORCL/

Step Three

Connect to the Oracle database from the VNET.

Since the Oracle DB container is created in a VNET, a private IP address is assigned to the container.  We can use this IP to connect to it from inside the VNET.

That’s it! We now have an Oracle database without the need to maintain the underlying VM or data volume.

Let’s Talk Pricing

Azure Container Instances bill per second at the “container” group level. “Container group” resources like vCPU/ Memory are shared across multiple containers sharing the same host

The current pricing per second is listed below:

Container group duration:
Memory: $0.0000015 per GB-s
vCPU: $0.0000135 per vCPU-s

The setup we defined above (8GB memory and two vCPUs) will cost ~$100/month based on the following pricing calculation:

Memory duration:

Number of container groups * memory duration (seconds) * GB * price per GB-s * number of days

1 container group * 86,400 seconds * 8 GB * $0.0000015 per GB-s * 30 days = $31.1

vCPU duration:

Number of container groups * vCPU duration (seconds) * vCPU(s) * price per vCPU-s * number of days

1 container groups * 86,400 seconds * 1 vCPU * $0.0000135 per vCPU-s * 30 days = $69.98

Total billing:

Memory duration (seconds) + vCPU duration (seconds) = total cost

$31.1 + $69.98 = $101 per month

Almost! But not completely there!

As the title suggests, the approach mentioned above gets us close to our objective of “Oracle DB as a service on Azure” but we are not all the way there. I would be remiss not to mention some of the challenges that remain.

Our setup is resilient to failure (e.g., policy-based restart) but this setup does not offer us high availability. For that, you will have to rely on setting up something like the Oracle Data Guard on Azure.

ACI supports horizontal scaling and as such the vertical scaling options are limited to current ACI limits (16GB and four vCPU).

ACI VNET Integration capability has some networking limits around outbound NSGs and public peering that you need to aware of.

I’d like to thank Manish Agarwal and his team for help with this setup.

Looking to Retire Your Oracle Hardware and Migrate to the Cloud? Contact AIS Today to Discuss Your Options.