Tag Archives: SCVMM

Hotfix 1 for SCVMM 2016 Update Rollup 1

Microsoft has published a new Hotfix 3208888 for those who are running SCVMM 2016 Update Rollup 1. This includes a fix for the issue where when you use VMM to live migrate a VM from one host that’s running one version of Windows Server 2016 to another host that’s running a different version of Windows Server 2016, the placement page assigns a zero rating to the target host.(i.e Datacenter edition to Standard edition). This issue happens only when you try to live migrate between two version of Windows Server 2016 but not when you are trying a live migration between hosts that are running 2012 R2 and 2016.

This leads to block the live migration with below error message:

Unable to migrate or clone the virtual machine VM_name because the version of virtualization software on the host does not match the version of virtual machine’s virtualization software on source version_number. To be migrated or cloned, the virtual machine must be stopped and should not contain any saved state.

Installing KB3208888

Note the this is applicable only to those who are running SCVMM 2016 Update Rollup 1.

  • Download the KB package from here.
  • Use an elevated Command Prompt to install the KB manually.

msiexec.exe /update kb3208888_vmmserver_amd64.msp


Salvaging SCVMM 2012 R2 with an existing database

Recently I was working in a SCVMM 2012 R2 deployment project and came across a DEFCON 1 situation. I was ready to uninstall everything and re-deploy VMM from scratch but I wanted to minimize the post installation configuration tasks after that. I came across a great post by the SCVMM team on how to achieve this with two SQL stored procedures and here is how I managed to save couple hours of the deployment time with that.

Backup First

Although you seriously know what you are doing, it is wise to back up the VMM database first. Just in case if you manage to screw up the entire database you can always restore from a copy.

The Process

The entire process is clearly explained in this TechNet article so I’m going to skip the lecture. But there are few things that I followed based on my gut feeling to make it work in a single attempt.

  • Stopped the VMM Server service before executing the first stored procedure and backing up the VMM database.
  • I restored the backup VMM  database using a dummy name and tested the stored procedures first to see there are any exceptions thrown during execution. Luckily it was successful. This is optional but it doesn’t hurt to try.
  • After installing the secondary VMM server I made sure to install the relevant UR version that was installed previously in the old VMM instance. This is critical or otherwise the database will not be useful at all.

Now if you have a Highly Available VMM environment things might look a little scary (VMM service fails most of the times) but the article explains how you can safely use the existing database by stopping at starting the VMM service manually before and after proceeding with the setup.



Library Server Failure in SCVMM 2012 R2

Few days back I was working with my colleague Law Wen Feng on a SCVMM Managed Hyper-V Cluster. The idea was to update the environment from SCVMM 2012 R2 UR 2 to UR 7. We noticed a strange issue where the Library Server (VMM Server itself) was complaining about a refresh failure. It seemed like the VMM agent was no longer functioning properly in the VMM Management Server.

WinRM Issue  (1)

As a poor man’s alternative we removed the library server from VMM. Then we tried to re-add the same VMM server as a library server which resulted in bizarre output. Nevertheless the VMM rejected another file share in a different server which we were hoping to add an alternative.

WinRM Issue  (2)

The error reads as the VMM Agent was no longer functional on the target server. But it was indeed running without any issue.

WinRM Issue  (3)

WinRM Issue  (4)

I’ve reached out to my fellow MVP colleagues Krisitan Nese, Stanislav Zhelyazkov & Daniel Neuman for some suggestions. They suggested that we do re-associate the VMM Agent with VMM Server. Yes it sound like chicken and egg situation. But this is no ordinary Hyper-V host but the VMM server itself.

Register-SCVMMManagedComputer cmdlet can be used to re-associate a managed computer on which VMM agent software is installed with a different VMM management server. But here we chose to add it to the same VMM server.

WinRM Issue  (5)Now it was complaining about WinRM was no longer functional. For those who are familiar WinRM is necessary component that is needed for you to remotely manage Windows Server. By default during the installation SCVMM takes care of enabling and running the WinRM service. Rebuilding the VMM server with retain DB option was not an option as we were middle of preparing demo lab and as I always believe needed to get to the bottom of it.

The evil WinRM GPO

We checked the GPO settings for the domain and found out WinRM was forced to all computers in our domain by a GPO. We moved the VMM server to a test OU and then disabled inheritance for that particular GPO and guess what, after a gpupdate /force in the VMM server we were able to add the library server back again.

WinRM Issue  (6)

Is that All? No it is not.

But I suspected it couldn’t be the only solution or the issue. So some digging into the default WinRM behavior in SCVMM I noticed that infact there was an actual configuration item that has been missed in the GPO itself.

According to Microsoft, there are some consideration for WinRM when you adda Hyper-V host to a VMM environment. Following has been extracted from above TechNet Article the highlighted section focuses on configuring WinRM listeners for both IPv4 & IPv6.

If you use Group Policy to configure Windows Remote Management (WinRM) settings, understand the following before you add a Hyper-V host to VMM management:

  • VMM supports only the configuration of WinRM Service settings through Group Policy, and only on hosts that are in a trusted Active Directory domain. Specifically, VMM supports the configuration of the Allow automatic configuration of listeners, Turn On Compatibility HTTP Listener, and Turn on Compatibility HTTPS Listener Group Policy settings. VMM does not support configuration of the other WinRM Service policy settings.
  • If you enable the Allow automatic configuration of listeners policy setting, you must configure it to allow messages from any IP address. To verify this configuration, view the policy setting and make sure that the IPv4 filter and IPv6 filter (depending on whether you use IPv6) are set to *.
  • VMM does not support the configuration of WinRM Client settings through Group Policy. If you configure WinRM Client Group Policy settings, these policy settings may override client properties that VMM requires for the VMM agent to work correctly.

I had a look at the Allow Automatic Configuration of Listeners policy setting under Computer Configuration\Administrative Templates\Windows Components\Windows Remote Management node in the GPO and the IPv6 filter was set to null, we changed that to accept from any IP address by putting an asterisk (*). Of course IPv6 was enabled in all Hyper-V hosts and the VMM Server by default.

WinRM Issue  (7)

Now it was about time to move back the VMM Server to it’s original OU with the GPO applied and execute a gpupdate /force. Surprisingly it did the trick. We were able to re-add the library server (in VMM) and couple of other file share as library shares without any issue.

WinRM Issue  (8)

Amazing isn’t it? We may never gaze upon TechNet for such trivial issues when they happen but it was worth all the trouble without rebuilding the VMM server. I must thank all who helped by sharing their ideas to sort this issue out. That is what I love about the community. When all is lost somewhere far away in the world, there will always be good people to help you out.

Static IP configuration is missing in E2A Azure Site Recovery

Azure Site recovery is a great cost effective platform to host your DR sites. These days most of my time is spent on this technology and I’m experimenting on new things everyday. Troubleshooting ASR is not so easy as the information available is relatively low in some cases.

In one of my ASR deployments I have noticed below issue.

As per Microsoft Documentation for a SCVMM to ASR scenario we can enable the protected VM in ASR to have a predefined IP address from a mapped virtual network. The guidelines read as “If the network adapter of source virtual machine is configured to use static IP then the user can provide the IP for the target virtual machine. User can use this capability to retain the ip of the source virtual machine after a failover. If no IP is provided any available IP would be given to network adapter at the time of failover. In case the target IP provided by user is already used by some other virtual machine that is already running in Azure then the failover would fail.”

Now I have enabled replication on one VM and checked the configuration section and guess what the only option available was DHCP.

ASR DHCP 1Solution

ASR sees what VMM can see. In this case the on premise logical network didn’t have any static IP pool assigned to it. When I checked the VM properties in VMM I noticed it is also reflecting IP as DHCP.

ASR DHCP VM 2Below are the steps I’ve performed to overcome this issue.

  1. Create a static IP Pool for my logical network. As I didn’t use network virtualization I didn’t need to create a static IP pool for my VM network. You can follow this guide to create a static IP pool in a logical network.
  2. This static IP pool should be of the same range that you used for your VMs. If you click the Connection details button as in above screen you get get the actual IP address assigned in the OS level and determine the range.
  3. Next step is to refresh virtual machines. Once you refresh a VM and and check the Network adapter properties in SCVMM it will now display the IP as static.
  4. I have already replicated one VM. For that I had to disconnect it from the on-premise network (Connectivity > Not Connected in above screen) and connect it again to the same VM network. Then I did a VM refresh and et viola now I can see the static IP option in the ASR portal.


Debugging VMM Issues with logman

Sometimes Microsoft support will ask you to provide the VMM debug trace logs if you encounter issues with your VMM deployment. Today I’m going to simply explain the process of collecting debug logs for VMM and prasing them to text files.

  • First of all create a flder to store your VMM log files. I prefer to save them on C:\VMMLogs path.
  • Delete any existing VMM logs if present. In order to do this open up a PowerShell window as an administrator on your VMM server and type logman delete VMM and press enter. There will be warnings such as “Data Collector Set was not found” and you can safetly ignore same.
  • Create a VMM trace. You can use the following command to that.

logman create trace VMM -v mmddhhmm -o $env:SystemDrive\VMMlogs\VMMLog_$env:computername.ETL -cnf 01:00:00 -p Microsoft-VirtualMachineManager-Debug -nb 10 250 -bs 16 -max 512

  • Start the VMM trace by entering logman start vmm in the same PowerShell window.
  • Now you can reproduce the VMM issue that you have faced (i.e a job failure)
  • Immediately after reproducing the iussue you need to stop the VMM strace by entering logman stop vmm
  • The log files you created will be of ETL file format.The ETL is a log file created by Microsoft Tracelog, a program that creates logs using the events from the kernel in Microsoft operating systems and are machine readable. So next step is to convert same to text format.
  • You can convert the collected ETL log by entering Netsh trace convert <Path to file name>

I find these logs very useful specially when the errors in Windows Event Viewer are too generic. In fact debug trace can provide more information if you are encountering bizarre issues in your VMM deployment.

Replication Failure in Azure Site Recovery

Azure Site Recovery is a great product for those who want to setup their DR environment with a minimal cost. It is based on Hyper-V replica technology for Hyper-V workloads and supports replication VMware & Physical server workloads to DR as well. Today I’m going to discuss a common issue one can encounter when enabling ASR replication to the cloud.

I’ve been working on an ASR setup during couple months and encountered strange issue when I enabled replication in protected VMs.

The enable protection job fails with below error.

Job ID: f9f84765-b18c-4002-96a4-d420dfb76ea6-2015-05-14 10:00:29Z

Start Time: 5/14/2015 3:30:29 PM

Duration: 10 MINUTES

Protection couldn’t be enabled for the virtual machine. (Error code: 70094)

Provider error: Unable to complete the request. Operation on the <Hyper-V Node>  timed out.

Try the operation again. (Provider error code: 2924)

Possible causes: Protection can’t be enabled with the virtual machine in its current state. Check the Provider errors for more information.

Recommendation: Fix any issues in the Event Viewer logs (Applications and Service Logs – MicrosoftAzureRecoveryServices) on the Hyper-V host server. If this virtual machine is enabled for replication on the Hyper-V host, disable this setting. Then try to enable protection again.

UTC Time: Thu May 14 2015 10:15:59 GMT+0530 (Sri Lanka Standard Time)

Browser: Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.135 Safari/537.36

Language: en-us

Portal Version: 5.4.00298.11 (rd_auxportal_stable.150511-1702)

PageRequestId: a04f08ed-8932-43f2-95bc-2faab60ed958

Email Address: xxxxxx@outlook.com (MSA)

Subscriptions: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxx

In the particular Hyper-V host following error has been logged in Event logs.

Enable replication failed for virtual machine ‘XXXXXX’ due to a network communication failure. (Virtual Machine ID 807780f6-bb7c-48d5-937d-4857a654dec3, Data Source ID 2256321007502018113, Task ID 8c1a5d7d-0693-4d6b-9243-37cc5e96a7d6)

This ASR setup was a on-premise to Cloud scenario with a single SCVMM server.

After spending a good number of troubleshooting hours I finally figured out what went wrong. The Hyper-V Hosts themselves need Internet connectivity to replicate the VMs to ASR. If you cannot enable direct Internet connectivity on the Hyper-V hosts you should do so via a proxy setup. You can change the proxy settings in ASR Provider in Hyper-V Host.

ASR replication requires traffic to be sent over port 443 (SSL) and in my case only the SCVMM server was configured with Internet access. If you are using a proxy server you may need to consider allowing below for successful replication.

  • *.hypervrecoverymanager.windowsazure.com
  • *.accesscontrol.windows.net
  • *.backup.windowsazure.com
  • *.blob.core.windows.net
  • *.store.core.windows.net
  • Allow the IP addresses in Azure Datacenter IP Ranges and HTTPS (443) protocol. Also your IP address whitelist should contain that of your primary region and  West US IP address ranges.

Protect your Private Cloud with 5Nine Cloud Security

When it comes to virtualization lot of people start asking questions about how they can secure their environment against security threats. Installing an AV solution inside individual VMs looks like the correct answer but what will happen in case of a network related security threat? Let’s explore the best answer for these issues in Hyper-V context.

5nine Cloud Security is an agentless security solution for Hyper-V which uses the extensible Hyper-V switch capabilities. This solution is capable of providing VM isolation, compliance and antivirus features.

5Nine also offers firewall, AV & IDS functions out of the box. The most important thing about this is it is an agent;less solution where you do not install any agent inside VMs to achieve these goals.

For hosters using Windows Azure Pack 5Nine offers Azure Pack extension which allows them to bring true IDS capabilities to their tenants. As the number of tenants increase security becomes the number one concern of any hoster. Not only that the 5Nine Cloud Security SCVMM plugin let you to deploy all these features via SCVMM if you are only focused about managing your own environment through SCVMM, making it easier to integrate both solutions.

All these features come at an attractive price $199/2 CPUs per host. If you are interested you can visit www.5nine.com for more information. Below is a short demonstration of what 5Nine Cloud Security can do to protect your Hyper-V Hosts, Private Cloud or Service Provider Cloud.

In a future post I’m going to discuss how to configure 5Nine Cloud Security to protect your Microsoft virtualization solution.

Security Alert – Virtual Machine Manager Elevation of Privilege Vulnerability

Microsoft has recently identified an exploit in SCVMM that could allow user privilege elevation. Any hacker who leverages this vulnerability have to first login using Active Directory credentials and could gain administrative privileges and thereby can control VMs managed by a particular VMM Server. Basically this is a result of incorrect user role validation within VMM.

This affects Microsoft System Center Virtual Machine Manager 2012 R2 Update Rollup 4 (VMM Server update 2992024)

Microsoft has immediately released a patch for this issue. You can download the KB3023195 for VMM 2012 R2 Update Rollup 5 from here.

Another important thing to remember is if you have Administrator console installed on your VMM server, you need to install Admin Console Update for UR5 (KB3023914) which can be downloaded from here. When you are installing you’ll have to install UR5 for VMM followed by UR5 for administrator console for proper remediation of this threat.

Tech Update | AWS System Manager for SCVMM

Wouldn’t it be painful just to use the web browser to manage your resources in Amazon EC2 cloud? How about managing them from VMM? That’s not going to be a problem anymore according to amazon.

Recently Amazon introduced AWS System Manager an add-on that can be installed on SCVMM 2012 SP1 onwards. This will let you to manage your EC2 Windows Instances from VMM console.

Basically you can start,stop, restart your AMIs from this tool. If you require remote access you can even RDP into same.

You can download this tool from here. Most importantly it’s FREE.

Azure Site Recovery | Things you should keep in mind

Hi folks, we have successfully deployed an ASR solution throughout this series and today we are going to look at some FAQs people have. First lets take a look at how to clean the demo environment that we setup for ASR.

Cleanup your ASR Deployment.

There are actually two ways to do this.

  1. Remove the VMM server from registered servers in ASR Vault. This will disassociate the VMM server with your ASR vault. But yet again you will need to remove storage account and ASR vault manually from Azure portal if you don’t require them any longer. Although you may need to remove obsolete registry entries of the VMM server as described in here.
  2. Use the Cleanup script from TechNet. This is useful when you no longer have access to the Azure account. It’s quite simple actually. All you have to do is run the PowerSehll Script in the VMM server that has been registered with ASR. Actually this script will remove the registration information and cloud pairing information  of the protected clouds of the  VMM Server from Azure. Assuming you are not the administrator of the Azure account using this script is much more safer.

Also you can disable the protection of VM separately. Refer this article from TechNet in order to achieve this.

Lets take a look on some common FAQs related to ASR

Q. I have a existing VMware environment. Can I leverage ASR for DR in my environment?

A. Yes & No. For VMware workloads Microsoft has a separate product called Microsoft Migration Accelerator. You can use this to move your VMs to the cloud from AWS or VMware. In order to provide replication Inmage Scout by Microsoft is the best tool.

Q. What are the system requirements for ASR?

A. Note that ASR doesn’t support VHDX file format yet as it is not available is Azure still. Also there are number of compliant Linux distros that are supported as Guest OS.

  • An Azure Subscription
  • Management certificate
  • System Center Virtual Machine Manager 2012 R2
  • Windows Server 2012 R2 Hyper-V – used as VM host
  • Gen 1, fixed disk .vhd VMs in Hyper-V
  • Guest OS Windows Server 2008 or later

Q. I have full System Center suite deployed in my environment. How can I leverage that with ASR?

A. As all our system center products support Microsoft Azure, this depends on how you want use them with Azure. For an example you can use Orchestrator runbooks for automatic fail over to Azure, SCOM to generate alerts during fail over window etc…

Q. How can I get the pricing information for ASR?

A. Visit this link to learn more about the product and pricing

Azure Site Recovery is a emerging and continuously evolving product. With the announcement of vNext of System Center & Windows Server platform we can expect lot of new & exciting features with Azure pretty soon.