Tag Archives: Azure

Deploying Dependency Agent for Service Map with Azure VM Extension

OMS Service Map solution automatically discovers application components, processes and services on Windows and Linux systems and maps the communication between them. In order to use this solution, you need to deploy the Service Map Dependency Agent for the respective OS. As you can see in the below diagram, this agent does not transmit any data by itself, but rather transmits data to the OMS agent which will then publish the data to OMS. 

Image Courtesy Microsoft Docs

Microsoft has announced the release of a new Azure VM extension that will allow you to automatically deploy the dependency agent for Service Maps. Since Service Maps supports both Windows & Linux, there are two variants of this extension. One thing you do have to remember is that the dependency agent still depends on (ah the irony) the OMS agent and your Azure VMs need to have the OMS agent installed and configured prior deploying the dependency agent. 

Installing the dependency agent with Azure VM extension (PowerShell)

Following is a PowerShell code snippet (from Microsoft) that will allow you to install the Service Map on all VMs in an Azure resource group. 

$version = "9.1"

$ExtPublisher = "Microsoft.Azure.Monitoring.DependencyAgent"

$OsExtensionMap = @{ "Windows" = "DependencyAgentWindows"; "Linux" = "DependencyAgentLinux" }

$rmgroup = "<Your Resource Group Here>"

Get-AzureRmVM -ResourceGroupName $rmgroup |

ForEach-Object {

""

$name = $_.Name

$os = $_.StorageProfile.OsDisk.OsType

$location = $_.Location

$vmRmGroup = $_.ResourceGroupName

"${name}: ${os} (${location})"

Date -Format o

$ext = $OsExtensionMap.($os.ToString())

$result = Set-AzureRmVMExtension -ResourceGroupName $vmRmGroup -VMName $name -Location $location `

-Publisher $ExtPublisher -ExtensionType $ext -Name "DependencyAgent" -TypeHandlerVersion $version

$result.IsSuccessStatusCode

}
Installing the dependency agent with Azure VM extension (ARM Template)

However Microsoft didn’t publish any official reference on to deploy the Azure VM extension for deploying the dependency agent using ARM templates (as of yet). My colleague MVP Stanislav Zhelyazkov has published a blog post on how to do this with ARM templates. You can find his reference template from GitHub.

Note

This VM extension is currently available in the West US region and Microsoft is rolling it out all Azure regions in the next couple of days.

VM Inception | Nested Virtualization in Azure

I bet that most of you have watched the movie “Inception”, where a group of people are building a dream within a dream within a dream. Before Windows Server 2016 you couldn’t deploy a VM within a VM in Hyper-V. Lot of people are/were encouraged to use VMware as it supported this capability called “Nested Virtualization”. But with the release of Windows Server 2016 & Hyper-V server 2016 this functionality has been introduced. This is specially useful when you don’t have lot of hardware to run your lab environments or want to deploy a PoC system without burning thousands of dollars.

Microsoft announced the support for nested virtualization Azure IaaS VMs using the newly announced  Dv3 and Ev3 VM sizes. This capability allows you to create nested VMs in an Azure VM and also run Hyper-V containers in Azure by using nested VM hosts. Now let’s have look on how this is implemented in Azure Azure Compute fabric.

Image Courtesy Build 2017

As you can see in the above diagram, on top of the Azure hardware layer, Microsoft has deployed the Windows Server 2016 Hyper-V hypervisor. Microsoft then adds vCPU on top of that to expose the Azure IaaS VMs that you would normally get. With nested virtualization, you can enable Hyper-V inside those Azure IaaS VMs running Windows Server 2016. You can then run any number of  Hyper-V 2016 supported guest operating systems inside these nested VM hosts.

Following references from MSFT provides more information on how you can get started with nested virtualization in Azure.

 

 

 

 

 

VM Management Failure in Azure

When deploying a new VM from the Azure Portal, I have come across a bizarre error. The VM deployment has been interrupted and it was in a failed state. I have tried deploying the VM several times to the same resource group but without any luck. 

The Provisioning State was failed in my case. Surprisingly the recommendation to remediate this issue seemed little odd. I was trying to deploy a single Windows VM of DS2 v2 SKU, nothing out of the ordinary. 

After examining the deployment properties, I have noticed that the issue was  with the Virtual Machine resource in Azure Resource Manager. But the level of detail for the error was still the same. 

Although the article Understand common error messages when you manage Windows virtual machines in Azure lists common Windows VM management errors in Azure, it doesn’t contain any troubleshooting steps on how to recover.

Allocation failed. Please try reducing the VM size or number of VMs, retry later, or try deploying to a different Availability Set or different Azure location.

As a last resort, I tried starting the failed VM manually from the portal. Well that worked anyway. I tried deploying couple of VMs thereafter in to the same resource group which did not reported any errors.

Have you faced the similar error before? For me the mystery still remains.

502 Bad Gateway error | Azure Application Gateway Troubleshooting

I was setting up an Azure Application Gateway for a project couple days back. The intended workload was setting up git on nginix.  But when I tried to reach the git URL I noticed that it was failing with 502 Bad Gateway error.

Initial Troubleshooting Steps

  • We tried to access backend servers from Application Gateway IP 10.62.124.6, backend server IPs are 10.62.66.4 and 10.62.66.4. Application Gateway configured for SSL Offload.
  • We were able to access the backend servers directly on port 80, but when accessed via Application Gateway this issue occurs.
  • Rebooted the Application Gateway and the backend servers. and configured custom probe as well. But the issue was with the Request time out value which is by default configured for 30 seconds.
  • This means that when user request is received on Application Gateway, it forwards it to the backend pool and waits for 30 seconds and if it fails to get a response back then within that period users will receive a 502 error.
  • The issue has been temporarily  resolved after the time out period on the Backend HTTP settings has been changed to 120 seconds.

Real Deal

Increasing the timeout values were only a temporary fix as we were unable to find out a permanent fix. I have reached out to Microsoft Support and they wanted us to run below diagnostics.

  • Execute the below cmdlet and share the results.

$getgw = Get-AzureRmApplicationGateway -Name <application gateway name> -ResourceGroupName <resource group name>

  • Collect simultaneous Network traces:
    1. Start network captures on On-premises machine(Source client machine) and Azure VMs (Backend servers)
      • Windows: Netsh
        1. Command to start the trace:  Netsh trace start capture=yes report=yes maxsize=4096 tracefile=C:\Nettrace.etl
        2. Command to stop the trace:  Netsh trace stop 
      • Linux: TCPdump
        1. TCP DUMP command: sudo tcpdump -i eth0 -s 0 -X -w vmtrace.cap 
    2. Reproduce the behavior.
    3. Stop network captures.

Analysis

The network traces collected on Client machine and Destination servers while the issue was reproduced indicates that,  during the time period the trace was collected, for every HTTP Get Request (default probing) from the Application Gateway instances on the backend servers, the servers responded “Status: Forbidden” HTTP Response.

This has resulted in Application Gateway marking the backend servers as unhealthy as the expected response is HTTP 200 OK.
 
The Application Gateway “gitapp” configured for 2 instances (Internal instance IPs: 10.62.124.4, 10.62.124.5)
 
Trace collected on backend server 10.62.66.4
 
12:50:18 PM 1/3/2017    10.62.124.4         10.62.66.4            HTTP      HTTP:Request, GET /
12:50:18 PM 1/3/2017    10.62.66.4            10.62.124.4         HTTP      HTTP:Response, HTTP/1.1, Status: Forbidden, URL: /
12:50:45 PM 1/3/2017    10.62.124.5         10.62.66.4            HTTP      HTTP:Request, GET /
12:50:45 PM 1/3/2017    10.62.66.4            10.62.124.5         HTTP      HTTP:Response, HTTP/1.1, Status: Forbidden, URL: /
 
Trace collected on backend server 10.62.66.5
 
12:50:48 PM 1/3/2017    10.62.124.4         10.62.66.5            HTTP      HTTP:Request, GET /
12:50:48 PM 1/3/2017    10.62.66.5            10.62.124.4         HTTP      HTTP:Response, HTTP/1.1, Status: Forbidden, URL: /
12:50:45 PM 1/3/2017    10.62.124.5         10.62.66.5            HTTP      HTTP:Request, GET /
12:50:45 PM 1/3/2017    10.62.66.5            10.62.124.5         HTTP      HTTP:Response, HTTP/1.1, Status: Forbidden, URL: /

Rootcause

Due to the security feature ‘rack_attack’ enabled on the backend servers, it has blacklisted the application gateway instance IP’s and therefore the servers were not responding to the Application Gateway, causing it to mark the backend servers as Unhealthy.

Fix

Once this feature was disabled on the backend web servers (niginx) , the issue was resolved and we could successfully access the web application using Application Gateway.

 

SQL RP Installation Failure in Azure Stack TP1 | Fix It

Myself & my good friend CDM MVP Nirmal Thewarathanthri have been experimenting with Azure Stack for a while now. Although we tried more than 30 times to install SQL Resource Provider in our Azure Stack lab it was never quite successful. The biggest problem is cleaning up the Azure Stack environment every time after a failure as sometimes we had to do a fresh install from scratch.

The Epic Failure

Following were the symptoms of this issue.

  • The SQL VM installs just fine.
  • Deployment always fails at DSC configuration in the SQL VM.
  • The URL of the ARM template for SQL VM seems no longer valid as you can see.

Here is the full description of the error that we encountered.

VERBOSE: 8:54:27 AM – Resource Microsoft.Compute/virtualMachines/extensions ‘sqlrp/InstallSqlServer’ provisioning
status is running
New-AzureRmResourceGroupDeployment : 10:29:12 AM – Resource Microsoft.Compute/virtualMachines/extensions
‘sqlrp/InstallSqlServer’ failed with message ‘The resource operation completed with terminal provisioning state
‘Failed’.’
At D:\SQLRP\AzureStack.SqlRP.Deployment.5.11.61.0\Content\Deployment\SqlRPTemplateDeployment.ps1:207 char:5
+     New-AzureRmResourceGroupDeployment -Name “newSqlRPTemplateDeploym …
+     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (:) [New-AzureRmResourceGroupDeployment], Exception
    + FullyQualifiedErrorId : Microsoft.Azure.Commands.Resources.NewAzureResourceGroupDeploymentCommand

New-AzureRmResourceGroupDeployment : 10:29:12 AM – An internal execution error occurred.
At D:\SQLRP\AzureStack.SqlRP.Deployment.5.11.61.0\Content\Deployment\SqlRPTemplateDeployment.ps1:207 char:5
+     New-AzureRmResourceGroupDeployment -Name “newSqlRPTemplateDeploym …
+     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (:) [New-AzureRmResourceGroupDeployment], Exception
    + FullyQualifiedErrorId : Microsoft.Azure.Commands.Resources.NewAzureResourceGroupDeploymentCommand

The issue in this case was the unstable Internet connection we had. The ARM template for SQL RP downloads the SQL 2014 ISO first. In our case timeout in the download has stopped the entire process. Once the VM was created SLQ Server 2014 wasn’t installed in it.

To solve this issue we followed below procedure in a fresh installation of MAS TP1. You can try this out in an existing installation with failed SQL RP deployment but there’s no guarantee that you will be able to cleanup the existing resource group. If you have executed the SQL RP installation only once clean up may work but if you have tried it multiple times there’s a high chance of failing to cleanup the existing resource group/s.

  1. Download the SQL image from here.
  2. Open the default PIR image. This is available in the MAS TP1 host  \\sofs\Share\CRP\PlatformImage\WindowsServer2012R2DatacenterEval\WindowsServer2012R2DatacenterEval.vhd
  3. Once you mount the VHD (simply double click to mount), create new Folder called  SQL2014 on the PIR image under C:\ drive
  4. Copy all files from the downloaded ISO into the folder SQL2014
  5. Start the deployment script. If you are trying this on an existing failed deployment, then  re-run the deployment after cleaning up the existing resource group/s for SQL RP.

Once all the deployment tasks are completed you can see a successfully deployed SQL Resource Provider in the portal as below.

SQL RP Success (1)

You can refer the MSFT guide on how to add a SQL resource provider in MAS TP1 deployment here for more information.

Exporting your Azure Resource Groups to ARM Templates | Part 2

In my previous post I showed you how we can export Azure resource groups into ARM templates using the Azure Portal. For those of us who are not GUI fans (including myself) Azure PowerShell and Azure CLI provide cmdlets/commands to leverage the export feature for cloning, redeploying and automating Azure resource group deployments.

Azure PowerShell

With the latest Azure PowerShell you can execute below cmdlet to export a running resource group to an ARM template.

Export-AzureRmResourceGroup -ResourceGroupName <RG name> -Path <template path>

To export resource groups from a previous deployment you may use the below cmdlet syntax.

Save-AzureRmResourceGroupDeploymentTemplate -DeploymentName <Deployment Name> -ResourceGroupName <RG Name>-Path <template path>

Azure CLI

You can use the following syntax to export a running resource group to an ARM template.

azure group export <name> [template path]

Use below command syntax to export to an ARM template from a previously deployed Resource Group

group deployment template download [options] <resource-group> <name> [directory]

 

Exporting your Azure Resource Groups to ARM Templates | Part 1

Have you ever wanted to clone your resource group  deployment in Azure to another subscription or perhaps redeploy again without manual interaction with GUI? Now you can export your resource groups as ARM templates and redeploy wherever you want without having further barriers. Let’s explore how to use this feature in Azure.

Export from an existing Resource Group Deployment

When you select a resource group you can see the Export Template option in Settings.

Export RG to ARM (1)

Export RG to ARM (1)

Export from a previous deployment

In your resource group select the particular deployment slot and you will have the option to export that particular slot with parameters submitted for that specific instance of deployment.

Export RG to ARM (5)

Saving and Redeploying to a new resource group

Alternatively you have the option to Save the template and it will be saved under Browse > Templates in the Azure Portal.

Export RG to ARM (6)

Export RG to ARM (4)

Selecting the Deploy button will allow you to start a new deployment.

Export RG to ARM (3)

Keep in mind that currently not all the resource types are supported in with export feature. For an example you may encounter failure s when you try to export resources such as WebApps, Service Bus, Stream Analytics etc… Following is such an error which happened when we tried to export a resource group with Service Bus resources.

The schema of resource type ‘Microsoft.ServiceBus/namespaces’ is not available. Resources of this type will not be exported to the template. (Code: ResourceTypeSchemaNotFound)

This has been reported to Microsoft and this post will be updated once Microsoft provide a list of supported resource types/add more and more supported resource types to this feature. Right now I can confirm that IaaS resources are fully supported in this feature.

In this next post let’s see how we can leverage Azure PowerShell or Azure CLI to export resource groups into ARM templates.

Why you should update DPM 2012 R2 to UR9?

Microsoft has released the Update rollup 9 for Data Protection Manager 2012 R2 a little over one month back. This UR contains a number of significant improvements to the current DPM version thereby enabling more capabilities for your enterprise backup strategy.

Here are four reasons that you should consider to apply this UR.

No need for consistency check for file server backups in case of a DPM Filter corruption

When your production file servers encounter an unexpected downtime, DPM file tracking filter gets corrupted and results in an inconsistent replica. In UR9 DPM leverages USN journal to track the changes in files, thereby running a consistent check to repair the damaged filters is no longer needed. The repair operation will be displayed as a synchronization job in DPM which will also sync the replica to latest. Running a consistency check is really painful especially when the replica is huge.

Say Goodbye to Production Server Restart

One of the biggest headaches while upgrading the DPM agent is the requirement to restart the protected servers unlike SCOM or SCCM agents. Finally Microsoft has got rid of the restart requirement. Microsoft has eliminated all the causes for restarting servers while upgrading DPM agents except the filter driver update. Any backup products that perform incremental backups use a filter driver, and whenever there is an update to the filter driver a reboot is needed. If you are already on UR6 or above you can easily upgrade your DPM agents without the restart requirement.  So unless any future UR doesn’t contain a filter driver update no restart required at all.

Cache Space for Online Backup has been reduced

In previous versions of Microsoft Azure Backup Agent local disk cache space requirement was 15% of the data source size for backup to Azure which is a big issue if your data source is  over 10 TB.  This has been reduced to 5% now.

Number of Recovery Points for Online Backups has been increased

For organizations with strict industry compliance requirements the need to have longer retention policies is a mandatory requirement. DPM now supports 9999 recovery points for a online backup in Azure where previously it was 366. This enables more flexible and consistent recovery policies for cloud backup.

You can refer the full KB article and download the binaries for UR9 package from here.

 

Creating a SQL Database V12 Server in Azure

Few days ago my friend Business Solutions (Dynamics NAV) MVP Tharanga Chandrasekara came up with an interesting question. Creating a logical server for SQL Azure DB (PaaS) in the old Azure Service Management Portal is possible but why can’t we do that in the new Azure Resource Manager Portal. To find out what is happening I tried exploring the SQL PaaS option in the ARM portal.

When we create a new SQL database in the ARM portal we can create a logical server along with it as below.

SQL V12 1

But somehow when we checked two days back there was no Add button under SQL servers. We have tried the same thing in several Azure Subscriptions but there was no luck.

SQL V12 3

But today I checked again the same thing in one of my subscriptions and could see the Add button and could create a server without any problem.

SQL V12 2

Nothing out of the ordinary was mentioned in any forum as well and Tharanga has posted a question in User Voice. We were hoping the PG can shed some light into this. Whether it was a glitch on certain subscriptions or actually missing feature until now.

This led me to explore how to do this in ARM using PowerShell.

  • First you need to install the new Azure PowerShell module to start with. You can refer this to understand how to do so.
  • Then you can execute below cmdlets in Azure Powershell to login to your Azure Subscription and choose the exact subscription (if you have many Azure subscriptions under one account)

Add-AzureRmAccount
Select-AzureRmSubscription -SubscriptionId <Subscription ID>

  • Not all resources in ARM are available in all regions so it is always better to check whether the V12 database servers are available in the region you were planning to deploy.

(Get-AzureRmLocation | where-object {$_.Name -eq “Microsoft.Sql/servers” }).Locations

  • Next step is to create  a resource group in your desired region. I chose East US.

New-AzureRmResourceGroup -Name “jcbv12sql-RG” -Location “East US”

  • Then you can create the SQL V12 server and add firewall rules to allow any connections from outside Azure.

New-AzureRmSqlServer -ResourceGroupName “jcbv12sql-RG” -ServerName “jcbv12svr01” -Location “East US” -ServerVersion “12.0”

New-AzureRmSqlServerFirewallRule -ResourceGroupName “jcbv12sql-RG” -ServerName “jcbv12svr01” -FirewallRuleName “exrule1” -StartIpAddress “<First IP Address>” -EndIpAddress “<Last IP Address>”

Microsoft Azure new SQL IaaS configuration experience

Happy New Year to all of my blog readers.

2016 is going to be an exciting year as we wait for the newest releases of Azure Stack, Windows Server & System Center from Microsoft. In my new year post I’m going to share some happy news for all Azure Ninjas out there working on IaaS.

If someone asks me what Microsoft Product makes my Azure deployment most complex I’d answer SQL server. The reason is SQL server being a awesome product needs an extra DBA care. When you are provisioning a SQL server VM in Azure you need to think about IOPs, Connectivity, Backups, Security first hand and how to provide the same level of experience as of a  on-premises data center.

Microsoft Azure team understood the pain us system administrators face when it comes to SQL server configuration and came up with a set of new configuration options for SQL VMs in Azure Resource Manager Deployment Model. In order to use the new configuration experience you need to create a VM in new Resource Manager deployment model and it supports any version of SQL server that Azure marketplace offers.

Simplified Connectivity

SQL IaaS in Azure (1)

In the classic model in order to configure SQL server connectivity from on-premises using SQL Server Management Studio (SSMS) you had to first Remote Desktop to the VM, open the SQL Server port in Windows firewall, enable SQL Server Authentication, and to allow inbound connectivity had to create a Public Azure endpoint for the VM. The new experience allows you to do all of that in the portal itself during the provisioning time and you can select whether this SQL server can only be contact from the VM itself or within the Virtual Network.

Automated Patching & Backup

SQL IaaS in Azure (3)

Another pain that IT Pros encounter with SQL server is patching. The new automated patching capability allows the administrators to define a maintenance window at their convenience during the VM provisioning itself. So if your customers need to take off the burden of patches for their SQL VM instances in Azure this is a life saver.

SQL IaaS in Azure (4)

What about backup then? The new Automated backup feature allows administrators to automatically backup all databases in SQL Server and it is not enabled by default  as different workloads can have  different backup requirements. You can retain these backups up to 30 days and even encrypt them.

Storage Optimizations

Be it on-premises or cloud the most important thing in SQL server instance is storage. Previously in Azure classic deployment model we had to attach the required number of data disks to provide the IOPs and throughput manually and stripe the SQL files across the disks or create a Storage Pool to divide the IOPs or throughput across them.  The new deployment model has all of these included in to provisioning by enabling us to specify the required IOPs, throughput and VHD size within the allowable limit of the VM instance size. The cool thing is when you tweak these settings Azure automatically changes the number of data disks using  Windows storage spaces. So you no longer have to worry about calculations.

Also you can select between any of the below three storage optimization method for your SQL VM depending on your workloads.

  • General is the default setting and supports most workloads.
  • Transactional processing optimizes the storage for traditional database OLTP workloads.
  • Data warehousing optimizes the storage for analytic and reporting workloads.

SQL IaaS in Azure (2)

For the automation geeks you can use the Azure Resource Manager templates to make it even more automated for larger deployments. Considering the amount of effort and time taken previously for SQL IaaS VM configuration in Azure the new deployment experience offers much more hassle free one time setup for SQL workloads.