Category Archives: IaaS

VM Inception | Nested Virtualization in Azure

I bet that most of you have watched the movie “Inception”, where a group of people are building a dream within a dream within a dream. Before Windows Server 2016 you couldn’t deploy a VM within a VM in Hyper-V. Lot of people are/were encouraged to use VMware as it supported this capability called “Nested Virtualization”. But with the release of Windows Server 2016 & Hyper-V server 2016 this functionality has been introduced. This is specially useful when you don’t have lot of hardware to run your lab environments or want to deploy a PoC system without burning thousands of dollars.

Microsoft announced the support for nested virtualization Azure IaaS VMs using the newly announced  Dv3 and Ev3 VM sizes. This capability allows you to create nested VMs in an Azure VM and also run Hyper-V containers in Azure by using nested VM hosts. Now let’s have look on how this is implemented in Azure Azure Compute fabric.

Image Courtesy Build 2017

As you can see in the above diagram, on top of the Azure hardware layer, Microsoft has deployed the Windows Server 2016 Hyper-V hypervisor. Microsoft then adds vCPU on top of that to expose the Azure IaaS VMs that you would normally get. With nested virtualization, you can enable Hyper-V inside those Azure IaaS VMs running Windows Server 2016. You can then run any number of  Hyper-V 2016 supported guest operating systems inside these nested VM hosts.

Following references from MSFT provides more information on how you can get started with nested virtualization in Azure.

 

 

 

 

 

502 Bad Gateway error | Azure Application Gateway Troubleshooting

I was setting up an Azure Application Gateway for a project couple days back. The intended workload was setting up git on nginix.  But when I tried to reach the git URL I noticed that it was failing with 502 Bad Gateway error.

Initial Troubleshooting Steps

  • We tried to access backend servers from Application Gateway IP 10.62.124.6, backend server IPs are 10.62.66.4 and 10.62.66.4. Application Gateway configured for SSL Offload.
  • We were able to access the backend servers directly on port 80, but when accessed via Application Gateway this issue occurs.
  • Rebooted the Application Gateway and the backend servers. and configured custom probe as well. But the issue was with the Request time out value which is by default configured for 30 seconds.
  • This means that when user request is received on Application Gateway, it forwards it to the backend pool and waits for 30 seconds and if it fails to get a response back then within that period users will receive a 502 error.
  • The issue has been temporarily  resolved after the time out period on the Backend HTTP settings has been changed to 120 seconds.

Real Deal

Increasing the timeout values were only a temporary fix as we were unable to find out a permanent fix. I have reached out to Microsoft Support and they wanted us to run below diagnostics.

  • Execute the below cmdlet and share the results.

$getgw = Get-AzureRmApplicationGateway -Name <application gateway name> -ResourceGroupName <resource group name>

  • Collect simultaneous Network traces:
    1. Start network captures on On-premises machine(Source client machine) and Azure VMs (Backend servers)
      • Windows: Netsh
        1. Command to start the trace:  Netsh trace start capture=yes report=yes maxsize=4096 tracefile=C:\Nettrace.etl
        2. Command to stop the trace:  Netsh trace stop 
      • Linux: TCPdump
        1. TCP DUMP command: sudo tcpdump -i eth0 -s 0 -X -w vmtrace.cap 
    2. Reproduce the behavior.
    3. Stop network captures.

Analysis

The network traces collected on Client machine and Destination servers while the issue was reproduced indicates that,  during the time period the trace was collected, for every HTTP Get Request (default probing) from the Application Gateway instances on the backend servers, the servers responded “Status: Forbidden” HTTP Response.

This has resulted in Application Gateway marking the backend servers as unhealthy as the expected response is HTTP 200 OK.
 
The Application Gateway “gitapp” configured for 2 instances (Internal instance IPs: 10.62.124.4, 10.62.124.5)
 
Trace collected on backend server 10.62.66.4
 
12:50:18 PM 1/3/2017    10.62.124.4         10.62.66.4            HTTP      HTTP:Request, GET /
12:50:18 PM 1/3/2017    10.62.66.4            10.62.124.4         HTTP      HTTP:Response, HTTP/1.1, Status: Forbidden, URL: /
12:50:45 PM 1/3/2017    10.62.124.5         10.62.66.4            HTTP      HTTP:Request, GET /
12:50:45 PM 1/3/2017    10.62.66.4            10.62.124.5         HTTP      HTTP:Response, HTTP/1.1, Status: Forbidden, URL: /
 
Trace collected on backend server 10.62.66.5
 
12:50:48 PM 1/3/2017    10.62.124.4         10.62.66.5            HTTP      HTTP:Request, GET /
12:50:48 PM 1/3/2017    10.62.66.5            10.62.124.4         HTTP      HTTP:Response, HTTP/1.1, Status: Forbidden, URL: /
12:50:45 PM 1/3/2017    10.62.124.5         10.62.66.5            HTTP      HTTP:Request, GET /
12:50:45 PM 1/3/2017    10.62.66.5            10.62.124.5         HTTP      HTTP:Response, HTTP/1.1, Status: Forbidden, URL: /

Rootcause

Due to the security feature ‘rack_attack’ enabled on the backend servers, it has blacklisted the application gateway instance IP’s and therefore the servers were not responding to the Application Gateway, causing it to mark the backend servers as Unhealthy.

Fix

Once this feature was disabled on the backend web servers (niginx) , the issue was resolved and we could successfully access the web application using Application Gateway.

 

SOS for Azure VMs with Set-AzureRmVM

“Save Our Souls” is the International distress call for help in maritime operations. Over the years SOS has become more common term to imply a call for help in a disastrous situation. In Microsoft Azure sometime you may have faced such situations especially with IaaS VMs. For an example RDP not working in a Windows VM or SSH ceased to function in a Linux VM. When all hope is lost you may contact Azure Support or try to restart the VM (from Azure Portal) or resize the VM as a last resort.

Now going into all of the above troubles is no longer required to rescue your Azure IaaS VMs. The latest Microsoft Azure PowerShell cmdlet improvements allows you to redeploy your virtual machine when you invoke a redeploy operation through Azure PowerShell.

Important

  • Below cmdlet works only with Azure Resource Manager based VMs.
  • Latest version of Azure PowerShell needs to be installed in the management PC from which you are invoking the redeploy operation.
  • Dynamic IP addresses will be changed after completing the redeploy operation.
  • Data on local disks (ephemeral storage) will be lost.

Following is the syntax for the updated Set-AzureRmVM cmdlet. Note that the -Redeploy switch is used to invoke a redeploy operation.

SetAzureRmVM Redeploy ResourceGroupName $rgname Name $vmname

The VM status changes from Running > Updating > Starting > Running during the operation. The final Running status means that VM has been successfully redeployed.

For a complete reference of the Set-AzureRmVM cmdlet please refer here.

Microsoft Azure new SQL IaaS configuration experience

Happy New Year to all of my blog readers.

2016 is going to be an exciting year as we wait for the newest releases of Azure Stack, Windows Server & System Center from Microsoft. In my new year post I’m going to share some happy news for all Azure Ninjas out there working on IaaS.

If someone asks me what Microsoft Product makes my Azure deployment most complex I’d answer SQL server. The reason is SQL server being a awesome product needs an extra DBA care. When you are provisioning a SQL server VM in Azure you need to think about IOPs, Connectivity, Backups, Security first hand and how to provide the same level of experience as of a  on-premises data center.

Microsoft Azure team understood the pain us system administrators face when it comes to SQL server configuration and came up with a set of new configuration options for SQL VMs in Azure Resource Manager Deployment Model. In order to use the new configuration experience you need to create a VM in new Resource Manager deployment model and it supports any version of SQL server that Azure marketplace offers.

Simplified Connectivity

SQL IaaS in Azure (1)

In the classic model in order to configure SQL server connectivity from on-premises using SQL Server Management Studio (SSMS) you had to first Remote Desktop to the VM, open the SQL Server port in Windows firewall, enable SQL Server Authentication, and to allow inbound connectivity had to create a Public Azure endpoint for the VM. The new experience allows you to do all of that in the portal itself during the provisioning time and you can select whether this SQL server can only be contact from the VM itself or within the Virtual Network.

Automated Patching & Backup

SQL IaaS in Azure (3)

Another pain that IT Pros encounter with SQL server is patching. The new automated patching capability allows the administrators to define a maintenance window at their convenience during the VM provisioning itself. So if your customers need to take off the burden of patches for their SQL VM instances in Azure this is a life saver.

SQL IaaS in Azure (4)

What about backup then? The new Automated backup feature allows administrators to automatically backup all databases in SQL Server and it is not enabled by default  as different workloads can have  different backup requirements. You can retain these backups up to 30 days and even encrypt them.

Storage Optimizations

Be it on-premises or cloud the most important thing in SQL server instance is storage. Previously in Azure classic deployment model we had to attach the required number of data disks to provide the IOPs and throughput manually and stripe the SQL files across the disks or create a Storage Pool to divide the IOPs or throughput across them.  The new deployment model has all of these included in to provisioning by enabling us to specify the required IOPs, throughput and VHD size within the allowable limit of the VM instance size. The cool thing is when you tweak these settings Azure automatically changes the number of data disks using  Windows storage spaces. So you no longer have to worry about calculations.

Also you can select between any of the below three storage optimization method for your SQL VM depending on your workloads.

  • General is the default setting and supports most workloads.
  • Transactional processing optimizes the storage for traditional database OLTP workloads.
  • Data warehousing optimizes the storage for analytic and reporting workloads.

SQL IaaS in Azure (2)

For the automation geeks you can use the Azure Resource Manager templates to make it even more automated for larger deployments. Considering the amount of effort and time taken previously for SQL IaaS VM configuration in Azure the new deployment experience offers much more hassle free one time setup for SQL workloads.

Azure Service Update | Static Public IP addresses for Azure VMs are now available

As you may already aware earlier this month Microsoft has announced the general availability of the new Azure portal with Azure Resource Manager Deployment (ARM) model. Most of the Azure services that are available in the Classic Deployment model are now available in ARM model except a very few. However Microsoft introduces regular enhancements to the fabric providing a much smoother cloud experience to the customers.

In the recent service update Microsoft has enabled the capability for a Static public IP addresses to be assigned directly to a virtual machine (VM) created using the ARM  deployment model. Previously we were only able to assign a  dynamic public IP address to the network adapter of the VM.

There is a difference in classic deployment model where a static public IP address can only be assigned to cloud services. which is called as a reserved IP. But you can assign a Instance Level Dynamic PIP to a VM in classic model and that hasn’t been changed with this service update.

If you are new to Azure Networking following references will be much valuable to decide how you want to plan networking & connectivity in the cloud.

Reserving existing dynamic VIPs for Azure Cloud Services

When you create a cloud service in Azure the Public Virtual IP address (VIP) assigned to it is by default dynamic. What this means is that once you stop the cloud service from Azure Portal the next time you start it you are assigned a different public IP. As an example if you de-allocate the underlying VMs in a cloud service this will happen definitely.

Microsoft understands that the customers require static Public IP addresses for their workloads due to the fact that Internet enabled DNS always rely on a static IP. When you create a new cloud service you can assign a static IP reservation for that. Previously it was only possible for new cloud services not existing deployments.

All Azure subscriptions are authorized to use 20 reserved IPs.However if you require more you can ask them an extension.

Let’s see how we can reserve an existing VIP assigned to a cloud service reserved.

Convert a dynamic VIP of to Reserved VIP

Take a look at below Azure PowerShell cmdlet.

New-AzureReservedIP -ReservedIPName “jcbvip01” -Location “Southeast Asia” -ServiceName “jcbvipcs”

Notice the -ServiceName parameter. This enables to provide an existing cloud service name where it will automatically reserve the exiting VIP.

Removing a Reserved IP from an existing deployment

You can use below cmdlet to remove the static VIP association.

Remove-AzureReservedIPAssociation -ReservedIPName “jcbvip01” -ServiceName “jcbvipcs”

Reassigning the VIP to another deployment

Here you simply provide the new cloud service name.

Set-AzureReservedIPAssociation -ReservedIPName “jcbvip01” -ServiceName “nabvipcs”

One important thing is that reassigning should occur within the same Azure region as these IP addresses are location specific. You cannot assign the same IP reservation to a service that is running in a different Azure region.

Azure VM Reboot Logs

How many of us really pay attention to e-mails sent by Microsoft Azure regarding planned maintenance? Does anyone actually monitor what happens to our VMs during a planned maintenance window schedule by Azure Team? If the answer is NO let’s see how we can get some insight on IaaS VM availability during a scheduled maintenance.

As there are periodic updates that need to be performed in Azure data centers globally, Microsoft will notify you of any scheduled maintenance windows. Usually we are not facing any service interruption during such unless Microsoft has specifically mentioned that there will be a service unavailability. Also sometimes these updates may restart the Azure VMs if required and up to now you as a customer didn’t have any visibility other than the regular email communique.

If you play attention to any cloud service (hosting VMs) > Dashboard section you may notice that there is a new feature called View Reboot Logs is available as per below. Note that this is the only place you get such an option as of now.

Azure VM Reboot Logs 1You can select a data range and analyze what are the planned reboots that has happened. Remember this is only applicable for planned maintenance not unexpected downtime.

Azure VM Reboot Logs 2

For shell lovers below is a sample PowerShell cmdlets that you need to run to get a more liner output.

Get-AzureDeploymentEvent -ServiceName <cloudservicename> -StartTime <start time>-EndTime <end time>

The input for start and end times should be similar to the format of get-date cmdlets output.

As per Microsoft they will be adding more events to this feature where you will have complete visibility over your VM availability. For me this is yet another win for my customers to see really understand the value of Microsoft Cloud platform.

Introducing Linux Integration Services 4.0 Preview for Microsoft Azure

In Hyper-V platform integration services or rather device drivers for emulated hardware plays a vital role. The purpose of these services are to enhance the functionality of VMs to get the maximum performance in par with an actual physical server. Microsoft has announced the availability of LIS 4.0 for Azure VMs recently as a early preview.

This preview version of LIS supports CentOS 6.0-6.6, 7.0-7.1 64 bit editions running on Azure VMs and has introduced below additional functionality to Azure Linux IaaS VMs.

  • CentOS version 6.6 through 7.1 is now supported.
  • Dynamic Memory – Hot Add for above CentOS releases which allows you to dynamically increase the amount of memory that is available to a running VM.

You can download and install the LIS Package from this official Microsoft Link. For a list of features offered by Integration Services for Linux & FreeBSD refer here.

Upgrading Linux Integration Services on Azure Linux VMs

Following procedure needs to be performed as a super user or a user in suborders list.

  • Verify the Linux version first by running below command.

# cat /etc/centos-release

  • SCP (secure copy) the lis4.tar.gz file to the target VM. You can use putty for this.
  • Extracted the tar file by executing

# tar xvzf lis4.tar.gz

  • Traverse to the appropriate release version inside the lis4 directory where XX is the version obtained earlier.

# cd lis4/CentOs<XX>

  • Execute the upgrade script and reboot the VM.

# ./upgrade.sh

# reboot

 

 

Azure VM Extensions for Linux VMs

Whenever you are creating a Linux VM (or a Windows VM) in Azure from the gallery you may have noticed that in the last screen you get the option to install Azure Agent depending on the platform. This Azure Virtual Machine Agent can be is used to install, configure, manage and run Azure VM Extensions which extends the functionality of the VMs that they are installed on.

VM Agent

VM Extensions

VM Extensions can,

  • Modify security and identity features, such as resetting account values and using anti malware
  • Start, stop, or configure monitoring and diagnostics
  • Reset or install connectivity features, such as RDP and SSH
  • Diagnose, monitor, and manage your VMs

What we are focusing today is VM Extenstions for Linux VMs.Below are some of the cool VM Extensions that you can use in Linux VMs.

CustomScript: Run any script on a Linux VM

  • Download files and run scripts from an Azure storage account
  • Not limited to a specific scripting language

VMAccess

  • Reset the password of the original provisioned user or create new user
  • (Re)set SSH key for user
  • Ensure SSH firewall port (22) is open
  • Restore the SSH server configuration to a working default

OSPatching

  • Extension enables scheduled/automatic updates for a Linux VM
  • Coordinate patch schedule across multiple VMs
  • Portal integration

VMAccess extension in particular is a great tool if you ever lost access to your Linux VM. In Linux you always reach the VM via SSH and this extension is capable of altering the SSH configuration in case of a disaster.

Below are some great resources you can use if you are interested in leveraging VM extensions for Azure Linux VMs.

Resources

  1. Regaining control of your Linux VM with VMAccess Extension
  2. List of Azure VM Extensions
  3. Automated OS patching for Linux VMs

Trick or Treat | Protecting large VHDs with Azure Site Recovery

One of the most annoying problems with ASR is that it cannot protect VMs with VHDs which has capacity greater 1 TB. Now we know the OS drive limitation has been already addressed by Microsoft (refer my previous blog post) but still 1 TB cap is there for data disks. This  is a limitation in Azure itself as of now. Let’s see a workaround that we can leverage to overcome this barrier.

Solution

This solution involves creating new striped disk which consists of the creation of a new striped disk drive consisting multiple smaller VHD images less than 1 TB each. Here we are copying the data from the old VHD to the new striped volume and the remove the old VHD.

Prerequisites

    • VM should be in shutdown state.
    • Required number of 1 TB VHDs should be added to the VM that can accommodate the size of the VHD which is greater than 1 TB. Keep in mind these VHDs are dynamic not fixed.

Procedure

  • Start the VM and stop any application services that are running i.e SQL
  • Go to Computer Management > Disk management tab. If prompted to initialize the new VHDs click OK and proceed.
  • Right-click one of the new unallocated volumes, and then click Create striped volume.
  • Select all the new volumes that are displayed in the wizard to create a striped volume.
  • Assign a temporary drive letter (i.e, F:) to the new drive and format the new drive to NTFS
  • Now you can copy the data between the two drives. Use below robocopy command to do so.
robocopy E:\ F:\ /mir
  • Change the drive letter of the new disk to the drive letter of the old disk. (Swap the drive letters)
  • In a PowerShell window (as administrator) run diskpart.
  • Type SAN POLICY=OnlineAll.
  • Shut down the VM and remove the old VHD image from the VM.
  • Start the VM and the services that you’ve stopped earlier, and try to protect the VM with Site Recovery now.