Category Archives: IaaS

Azure VM Serial Console Access Preview

One of the biggest challenges in any public cloud IaaS platform is enabling the direct serial-based access so that engineers can fix boot issues. However Microsoft Azure Compute team has overcome this challenge and has introduced serial console access (preview feature) to Azure VMs recently.

This allows access to a text-based console for Linux and Windows virtual machines. Furthermore this serial connection  to COM1 serial port of the VM provides access to the VM settings that are not related to its network/operating system state.

Following prerequisites must be met in order to use this feature.

If you want to disable this feature all you have to do is to disable the boot diagnostics setting for the desired VM.

This can be found under VM setiings blade > Support + Troubleshooting > Serial console (Preview)

Enabling Serial Console access in Windows VMs

  • RDP into your Azure VM running Windows OS.
  • Execute the following commands as an administrator in command prompt.
bcdedit /ems {current} on

bcdedit /emssettings EMSPORT:1 EMSBAUDRATE:115200
  • Once you reboot the VM serial console access will be enabled.

This preview is available in all Azure regions.

Isolated Azure VMs | New Sizes

Microsoft has announced two new IaaS VM sizes called E64i_v3 and E64is_v3 earlier today. These VM SKUs have isolated hardware and can be dedicated to a single customer.Microsoft recommends these two SKUs for workloads that needs a  a high degree of isolation from other customers for compliance and regulatory requirements.
Performance & Pricing Matters

The performance and pricing of E64i_v3 and E64is_v3 are same as their predecessors cousins E64_v3 and E64s_v3. Also the two new SKUs are available in all regions supported by E64_v3 and E64s_v3 sizes. As you may have already guessed, the simple  letter ‘i’ in the VM size name denotes that they are isolated.

Here are some facts about the new sizes.

  • E64i_v3 and E64is_v3 are hardware bound sizes unlike   E64_v3 and E64s_v3.
  • E64i_v3 and E64is_v3 operate on Intel® Xeon® Processor E5-2673 v4 2.3GHz hardware only and will be available until at least December 2021.
  • These sizes will have 12 months in advance after December 2021 to be decommissioned and after that MSFT will  offer updated isolated sizes with their next gen hardware.
  • On May 1st, 2018 these sizes will be made available as one-year Reserved VM Instances.
What are my other options?

Here is the full list of isolate VM sizes that is available in Azure Public cloud as of today.

 

Protecting Azure IaaS VMs with Managed Disks with Site Recovery

Microsoft Azure Site Recovery team has just announced the capability to protect Azure IaaS VMs with managed disks using ASR as a public preview in all ASR enabled regions. This was an important steps as lot of customers were expecting to protect their VM workloads with managed disks, compared to the previous unmanaged disk scenario.

A2A Managed Disk Architecture

The challenge here was how do you replicate disk without requiring a VM object. To overcome this hurdle, ASR uses one storage account, in source region which will cache the disks to the target region. This enables ASR to create a replica managed disk in the target region for each VM protected in the primary region and this replica disk will be the data store for the source disk in the primary region. One important thing to note is that the initial replication between the source and target happens externally using a snapshot at the VM level, so are delta syncs.

Things to Remember

  • VM protection can be enabled via VM settings blade or in the recovery services vault settings.
  • If you have VMs with unmanaged disks that are currently protected by ASR, you need to disable protection and convert the VMs to managed disks first as conversion from unmanaged to managed disks while protected by ASR is not supported.
  • There is an option for you to selected the type of the replica disks, to standard or premium. See below screenshot.
  • Only one cache storage account is needed to so store the data changes from source to primary regions, but you can leverage multiple cache accounts per VM as well.

VM Inception | Nested Virtualization in Azure

I bet that most of you have watched the movie “Inception”, where a group of people are building a dream within a dream within a dream. Before Windows Server 2016 you couldn’t deploy a VM within a VM in Hyper-V. Lot of people are/were encouraged to use VMware as it supported this capability called “Nested Virtualization”. But with the release of Windows Server 2016 & Hyper-V server 2016 this functionality has been introduced. This is specially useful when you don’t have lot of hardware to run your lab environments or want to deploy a PoC system without burning thousands of dollars.

Microsoft announced the support for nested virtualization Azure IaaS VMs using the newly announced  Dv3 and Ev3 VM sizes. This capability allows you to create nested VMs in an Azure VM and also run Hyper-V containers in Azure by using nested VM hosts. Now let’s have look on how this is implemented in Azure Azure Compute fabric.

Image Courtesy Build 2017

As you can see in the above diagram, on top of the Azure hardware layer, Microsoft has deployed the Windows Server 2016 Hyper-V hypervisor. Microsoft then adds vCPU on top of that to expose the Azure IaaS VMs that you would normally get. With nested virtualization, you can enable Hyper-V inside those Azure IaaS VMs running Windows Server 2016. You can then run any number of  Hyper-V 2016 supported guest operating systems inside these nested VM hosts.

Following references from MSFT provides more information on how you can get started with nested virtualization in Azure.

 

 

 

 

 

VM Management Failure in Azure

When deploying a new VM from the Azure Portal, I have come across a bizarre error. The VM deployment has been interrupted and it was in a failed state. I have tried deploying the VM several times to the same resource group but without any luck. 

The Provisioning State was failed in my case. Surprisingly the recommendation to remediate this issue seemed little odd. I was trying to deploy a single Windows VM of DS2 v2 SKU, nothing out of the ordinary. 

After examining the deployment properties, I have noticed that the issue was  with the Virtual Machine resource in Azure Resource Manager. But the level of detail for the error was still the same. 

Although the article Understand common error messages when you manage Windows virtual machines in Azure lists common Windows VM management errors in Azure, it doesn’t contain any troubleshooting steps on how to recover.

Allocation failed. Please try reducing the VM size or number of VMs, retry later, or try deploying to a different Availability Set or different Azure location.

As a last resort, I tried starting the failed VM manually from the portal. Well that worked anyway. I tried deploying couple of VMs thereafter in to the same resource group which did not reported any errors.

Have you faced the similar error before? For me the mystery still remains.

502 Bad Gateway error | Azure Application Gateway Troubleshooting

I was setting up an Azure Application Gateway for a project couple days back. The intended workload was setting up git on nginix.  But when I tried to reach the git URL I noticed that it was failing with 502 Bad Gateway error.

Initial Troubleshooting Steps

  • We tried to access backend servers from Application Gateway IP 10.62.124.6, backend server IPs are 10.62.66.4 and 10.62.66.4. Application Gateway configured for SSL Offload.
  • We were able to access the backend servers directly on port 80, but when accessed via Application Gateway this issue occurs.
  • Rebooted the Application Gateway and the backend servers. and configured custom probe as well. But the issue was with the Request time out value which is by default configured for 30 seconds.
  • This means that when user request is received on Application Gateway, it forwards it to the backend pool and waits for 30 seconds and if it fails to get a response back then within that period users will receive a 502 error.
  • The issue has been temporarily  resolved after the time out period on the Backend HTTP settings has been changed to 120 seconds.

Real Deal

Increasing the timeout values were only a temporary fix as we were unable to find out a permanent fix. I have reached out to Microsoft Support and they wanted us to run below diagnostics.

  • Execute the below cmdlet and share the results.

$getgw = Get-AzureRmApplicationGateway -Name <application gateway name> -ResourceGroupName <resource group name>

  • Collect simultaneous Network traces:
    1. Start network captures on On-premises machine(Source client machine) and Azure VMs (Backend servers)
      • Windows: Netsh
        1. Command to start the trace:  Netsh trace start capture=yes report=yes maxsize=4096 tracefile=C:\Nettrace.etl
        2. Command to stop the trace:  Netsh trace stop 
      • Linux: TCPdump
        1. TCP DUMP command: sudo tcpdump -i eth0 -s 0 -X -w vmtrace.cap 
    2. Reproduce the behavior.
    3. Stop network captures.

Analysis

The network traces collected on Client machine and Destination servers while the issue was reproduced indicates that,  during the time period the trace was collected, for every HTTP Get Request (default probing) from the Application Gateway instances on the backend servers, the servers responded “Status: Forbidden” HTTP Response.

This has resulted in Application Gateway marking the backend servers as unhealthy as the expected response is HTTP 200 OK.
 
The Application Gateway “gitapp” configured for 2 instances (Internal instance IPs: 10.62.124.4, 10.62.124.5)
 
Trace collected on backend server 10.62.66.4
 
12:50:18 PM 1/3/2017    10.62.124.4         10.62.66.4            HTTP      HTTP:Request, GET /
12:50:18 PM 1/3/2017    10.62.66.4            10.62.124.4         HTTP      HTTP:Response, HTTP/1.1, Status: Forbidden, URL: /
12:50:45 PM 1/3/2017    10.62.124.5         10.62.66.4            HTTP      HTTP:Request, GET /
12:50:45 PM 1/3/2017    10.62.66.4            10.62.124.5         HTTP      HTTP:Response, HTTP/1.1, Status: Forbidden, URL: /
 
Trace collected on backend server 10.62.66.5
 
12:50:48 PM 1/3/2017    10.62.124.4         10.62.66.5            HTTP      HTTP:Request, GET /
12:50:48 PM 1/3/2017    10.62.66.5            10.62.124.4         HTTP      HTTP:Response, HTTP/1.1, Status: Forbidden, URL: /
12:50:45 PM 1/3/2017    10.62.124.5         10.62.66.5            HTTP      HTTP:Request, GET /
12:50:45 PM 1/3/2017    10.62.66.5            10.62.124.5         HTTP      HTTP:Response, HTTP/1.1, Status: Forbidden, URL: /

Rootcause

Due to the security feature ‘rack_attack’ enabled on the backend servers, it has blacklisted the application gateway instance IP’s and therefore the servers were not responding to the Application Gateway, causing it to mark the backend servers as Unhealthy.

Fix

Once this feature was disabled on the backend web servers (niginx) , the issue was resolved and we could successfully access the web application using Application Gateway.

 

SOS for Azure VMs with Set-AzureRmVM

“Save Our Souls” is the International distress call for help in maritime operations. Over the years SOS has become more common term to imply a call for help in a disastrous situation. In Microsoft Azure sometime you may have faced such situations especially with IaaS VMs. For an example RDP not working in a Windows VM or SSH ceased to function in a Linux VM. When all hope is lost you may contact Azure Support or try to restart the VM (from Azure Portal) or resize the VM as a last resort.

Now going into all of the above troubles is no longer required to rescue your Azure IaaS VMs. The latest Microsoft Azure PowerShell cmdlet improvements allows you to redeploy your virtual machine when you invoke a redeploy operation through Azure PowerShell.

Important

  • Below cmdlet works only with Azure Resource Manager based VMs.
  • Latest version of Azure PowerShell needs to be installed in the management PC from which you are invoking the redeploy operation.
  • Dynamic IP addresses will be changed after completing the redeploy operation.
  • Data on local disks (ephemeral storage) will be lost.

Following is the syntax for the updated Set-AzureRmVM cmdlet. Note that the -Redeploy switch is used to invoke a redeploy operation.

SetAzureRmVM Redeploy ResourceGroupName $rgname Name $vmname

The VM status changes from Running > Updating > Starting > Running during the operation. The final Running status means that VM has been successfully redeployed.

For a complete reference of the Set-AzureRmVM cmdlet please refer here.

Microsoft Azure new SQL IaaS configuration experience

Happy New Year to all of my blog readers.

2016 is going to be an exciting year as we wait for the newest releases of Azure Stack, Windows Server & System Center from Microsoft. In my new year post I’m going to share some happy news for all Azure Ninjas out there working on IaaS.

If someone asks me what Microsoft Product makes my Azure deployment most complex I’d answer SQL server. The reason is SQL server being a awesome product needs an extra DBA care. When you are provisioning a SQL server VM in Azure you need to think about IOPs, Connectivity, Backups, Security first hand and how to provide the same level of experience as of a  on-premises data center.

Microsoft Azure team understood the pain us system administrators face when it comes to SQL server configuration and came up with a set of new configuration options for SQL VMs in Azure Resource Manager Deployment Model. In order to use the new configuration experience you need to create a VM in new Resource Manager deployment model and it supports any version of SQL server that Azure marketplace offers.

Simplified Connectivity

SQL IaaS in Azure (1)

In the classic model in order to configure SQL server connectivity from on-premises using SQL Server Management Studio (SSMS) you had to first Remote Desktop to the VM, open the SQL Server port in Windows firewall, enable SQL Server Authentication, and to allow inbound connectivity had to create a Public Azure endpoint for the VM. The new experience allows you to do all of that in the portal itself during the provisioning time and you can select whether this SQL server can only be contact from the VM itself or within the Virtual Network.

Automated Patching & Backup

SQL IaaS in Azure (3)

Another pain that IT Pros encounter with SQL server is patching. The new automated patching capability allows the administrators to define a maintenance window at their convenience during the VM provisioning itself. So if your customers need to take off the burden of patches for their SQL VM instances in Azure this is a life saver.

SQL IaaS in Azure (4)

What about backup then? The new Automated backup feature allows administrators to automatically backup all databases in SQL Server and it is not enabled by default  as different workloads can have  different backup requirements. You can retain these backups up to 30 days and even encrypt them.

Storage Optimizations

Be it on-premises or cloud the most important thing in SQL server instance is storage. Previously in Azure classic deployment model we had to attach the required number of data disks to provide the IOPs and throughput manually and stripe the SQL files across the disks or create a Storage Pool to divide the IOPs or throughput across them.  The new deployment model has all of these included in to provisioning by enabling us to specify the required IOPs, throughput and VHD size within the allowable limit of the VM instance size. The cool thing is when you tweak these settings Azure automatically changes the number of data disks using  Windows storage spaces. So you no longer have to worry about calculations.

Also you can select between any of the below three storage optimization method for your SQL VM depending on your workloads.

  • General is the default setting and supports most workloads.
  • Transactional processing optimizes the storage for traditional database OLTP workloads.
  • Data warehousing optimizes the storage for analytic and reporting workloads.

SQL IaaS in Azure (2)

For the automation geeks you can use the Azure Resource Manager templates to make it even more automated for larger deployments. Considering the amount of effort and time taken previously for SQL IaaS VM configuration in Azure the new deployment experience offers much more hassle free one time setup for SQL workloads.

Azure Service Update | Static Public IP addresses for Azure VMs are now available

As you may already aware earlier this month Microsoft has announced the general availability of the new Azure portal with Azure Resource Manager Deployment (ARM) model. Most of the Azure services that are available in the Classic Deployment model are now available in ARM model except a very few. However Microsoft introduces regular enhancements to the fabric providing a much smoother cloud experience to the customers.

In the recent service update Microsoft has enabled the capability for a Static public IP addresses to be assigned directly to a virtual machine (VM) created using the ARM  deployment model. Previously we were only able to assign a  dynamic public IP address to the network adapter of the VM.

There is a difference in classic deployment model where a static public IP address can only be assigned to cloud services. which is called as a reserved IP. But you can assign a Instance Level Dynamic PIP to a VM in classic model and that hasn’t been changed with this service update.

If you are new to Azure Networking following references will be much valuable to decide how you want to plan networking & connectivity in the cloud.

Reserving existing dynamic VIPs for Azure Cloud Services

When you create a cloud service in Azure the Public Virtual IP address (VIP) assigned to it is by default dynamic. What this means is that once you stop the cloud service from Azure Portal the next time you start it you are assigned a different public IP. As an example if you de-allocate the underlying VMs in a cloud service this will happen definitely.

Microsoft understands that the customers require static Public IP addresses for their workloads due to the fact that Internet enabled DNS always rely on a static IP. When you create a new cloud service you can assign a static IP reservation for that. Previously it was only possible for new cloud services not existing deployments.

All Azure subscriptions are authorized to use 20 reserved IPs.However if you require more you can ask them an extension.

Let’s see how we can reserve an existing VIP assigned to a cloud service reserved.

Convert a dynamic VIP of to Reserved VIP

Take a look at below Azure PowerShell cmdlet.

New-AzureReservedIP -ReservedIPName “jcbvip01” -Location “Southeast Asia” -ServiceName “jcbvipcs”

Notice the -ServiceName parameter. This enables to provide an existing cloud service name where it will automatically reserve the exiting VIP.

Removing a Reserved IP from an existing deployment

You can use below cmdlet to remove the static VIP association.

Remove-AzureReservedIPAssociation -ReservedIPName “jcbvip01” -ServiceName “jcbvipcs”

Reassigning the VIP to another deployment

Here you simply provide the new cloud service name.

Set-AzureReservedIPAssociation -ReservedIPName “jcbvip01” -ServiceName “nabvipcs”

One important thing is that reassigning should occur within the same Azure region as these IP addresses are location specific. You cannot assign the same IP reservation to a service that is running in a different Azure region.