Category Archives: SCVMM

Hotfix 1 for SCVMM 2016 Update Rollup 1

Microsoft has published a new Hotfix 3208888 for those who are running SCVMM 2016 Update Rollup 1. This includes a fix for the issue where when you use VMM to live migrate a VM from one host that’s running one version of Windows Server 2016 to another host that’s running a different version of Windows Server 2016, the placement page assigns a zero rating to the target host.(i.e Datacenter edition to Standard edition). This issue happens only when you try to live migrate between two version of Windows Server 2016 but not when you are trying a live migration between hosts that are running 2012 R2 and 2016.

This leads to block the live migration with below error message:

Unable to migrate or clone the virtual machine VM_name because the version of virtualization software on the host does not match the version of virtual machine’s virtualization software on source version_number. To be migrated or cloned, the virtual machine must be stopped and should not contain any saved state.

Installing KB3208888

Note the this is applicable only to those who are running SCVMM 2016 Update Rollup 1.

  • Download the KB package from here.
  • Use an elevated Command Prompt to install the KB manually.

msiexec.exe /update kb3208888_vmmserver_amd64.msp

 

Salvaging SCVMM 2012 R2 with an existing database

Recently I was working in a SCVMM 2012 R2 deployment project and came across a DEFCON 1 situation. I was ready to uninstall everything and re-deploy VMM from scratch but I wanted to minimize the post installation configuration tasks after that. I came across a great post by the SCVMM team on how to achieve this with two SQL stored procedures and here is how I managed to save couple hours of the deployment time with that.

Backup First

Although you seriously know what you are doing, it is wise to back up the VMM database first. Just in case if you manage to screw up the entire database you can always restore from a copy.

The Process

The entire process is clearly explained in this TechNet article so I’m going to skip the lecture. But there are few things that I followed based on my gut feeling to make it work in a single attempt.

  • Stopped the VMM Server service before executing the first stored procedure and backing up the VMM database.
  • I restored the backup VMM  database using a dummy name and tested the stored procedures first to see there are any exceptions thrown during execution. Luckily it was successful. This is optional but it doesn’t hurt to try.
  • After installing the secondary VMM server I made sure to install the relevant UR version that was installed previously in the old VMM instance. This is critical or otherwise the database will not be useful at all.

Now if you have a Highly Available VMM environment things might look a little scary (VMM service fails most of the times) but the article explains how you can safely use the existing database by stopping at starting the VMM service manually before and after proceeding with the setup.

 

 

Introducing Technical Preview 4 | Windows Server 2016 & System Center 2016

With dawn of the year 2016 almost upon us, Microsoft has released another build for it’s upcoming Windows Server & System Center 2016 suite of products. This Technical Preview 4 contains much new advancements and fixes based on customer feedback on the product clearly making it’s way as the cloud OS for next generation of computing.

Nano Server gets a new touch

Nano server, a headless installation option like server core which is going to be one of the installation option for Windows Server 2016 has improved a lot since last preview. In this release IIS & DNS server roles can be installed in Nano server in addition to existing Hyper-V & Scale-out File Server features.

Introducing Hyper-V Containers

Providing additional layer of isolation for Windows Containers, Hyper-V containers can be now deployed as virtual sandboxes to host application workloads. This technology utilizes the nested virtualization capability introduced in Windows Server TP4. Also you can use both docker & PowerShell to create, deploy and manage Windows Containers.

System Center 2016 Improvements

Another milestone is the System Center 2016 TP4 release with some awesome features for private cloud management. Now you can use the SCOM agent to monitor your Nano Servers in TP4. SCCM 2016 TP4 has introduced some new functionality to improve Windows 10 deployment experience via SCCM.

  • Mobile Device management (MDM): enhanced feature parity with Intune standalone – Many of the  MDM feature that are supported via Intune standalone (cloud only) are also enabled for Configuration Manager integrated with Intune (hybrid) in this release.

  • Integration with Windows Update for Business – Now you can view the list of devices that are controlled by Windows Update for Business.

  • Certificate provisioning for Windows 10 devices managed via on-premises mobile device management

You can download Windows Server 2016 Technical Preview 4 & System Center 2016 Technical Preview 4 evaluation bits from here.

Library Server Failure in SCVMM 2012 R2

Few days back I was working with my colleague Law Wen Feng on a SCVMM Managed Hyper-V Cluster. The idea was to update the environment from SCVMM 2012 R2 UR 2 to UR 7. We noticed a strange issue where the Library Server (VMM Server itself) was complaining about a refresh failure. It seemed like the VMM agent was no longer functioning properly in the VMM Management Server.

WinRM Issue  (1)

As a poor man’s alternative we removed the library server from VMM. Then we tried to re-add the same VMM server as a library server which resulted in bizarre output. Nevertheless the VMM rejected another file share in a different server which we were hoping to add an alternative.

WinRM Issue  (2)

The error reads as the VMM Agent was no longer functional on the target server. But it was indeed running without any issue.

WinRM Issue  (3)

WinRM Issue  (4)

I’ve reached out to my fellow MVP colleagues Krisitan Nese, Stanislav Zhelyazkov & Daniel Neuman for some suggestions. They suggested that we do re-associate the VMM Agent with VMM Server. Yes it sound like chicken and egg situation. But this is no ordinary Hyper-V host but the VMM server itself.

Register-SCVMMManagedComputer cmdlet can be used to re-associate a managed computer on which VMM agent software is installed with a different VMM management server. But here we chose to add it to the same VMM server.

WinRM Issue  (5)Now it was complaining about WinRM was no longer functional. For those who are familiar WinRM is necessary component that is needed for you to remotely manage Windows Server. By default during the installation SCVMM takes care of enabling and running the WinRM service. Rebuilding the VMM server with retain DB option was not an option as we were middle of preparing demo lab and as I always believe needed to get to the bottom of it.

The evil WinRM GPO

We checked the GPO settings for the domain and found out WinRM was forced to all computers in our domain by a GPO. We moved the VMM server to a test OU and then disabled inheritance for that particular GPO and guess what, after a gpupdate /force in the VMM server we were able to add the library server back again.

WinRM Issue  (6)

Is that All? No it is not.

But I suspected it couldn’t be the only solution or the issue. So some digging into the default WinRM behavior in SCVMM I noticed that infact there was an actual configuration item that has been missed in the GPO itself.

According to Microsoft, there are some consideration for WinRM when you adda Hyper-V host to a VMM environment. Following has been extracted from above TechNet Article the highlighted section focuses on configuring WinRM listeners for both IPv4 & IPv6.

If you use Group Policy to configure Windows Remote Management (WinRM) settings, understand the following before you add a Hyper-V host to VMM management:

  • VMM supports only the configuration of WinRM Service settings through Group Policy, and only on hosts that are in a trusted Active Directory domain. Specifically, VMM supports the configuration of the Allow automatic configuration of listeners, Turn On Compatibility HTTP Listener, and Turn on Compatibility HTTPS Listener Group Policy settings. VMM does not support configuration of the other WinRM Service policy settings.
  • If you enable the Allow automatic configuration of listeners policy setting, you must configure it to allow messages from any IP address. To verify this configuration, view the policy setting and make sure that the IPv4 filter and IPv6 filter (depending on whether you use IPv6) are set to *.
  • VMM does not support the configuration of WinRM Client settings through Group Policy. If you configure WinRM Client Group Policy settings, these policy settings may override client properties that VMM requires for the VMM agent to work correctly.

I had a look at the Allow Automatic Configuration of Listeners policy setting under Computer Configuration\Administrative Templates\Windows Components\Windows Remote Management node in the GPO and the IPv6 filter was set to null, we changed that to accept from any IP address by putting an asterisk (*). Of course IPv6 was enabled in all Hyper-V hosts and the VMM Server by default.

WinRM Issue  (7)

Now it was about time to move back the VMM Server to it’s original OU with the GPO applied and execute a gpupdate /force. Surprisingly it did the trick. We were able to re-add the library server (in VMM) and couple of other file share as library shares without any issue.

WinRM Issue  (8)

Amazing isn’t it? We may never gaze upon TechNet for such trivial issues when they happen but it was worth all the trouble without rebuilding the VMM server. I must thank all who helped by sharing their ideas to sort this issue out. That is what I love about the community. When all is lost somewhere far away in the world, there will always be good people to help you out.

Debugging VMM Issues with logman

Sometimes Microsoft support will ask you to provide the VMM debug trace logs if you encounter issues with your VMM deployment. Today I’m going to simply explain the process of collecting debug logs for VMM and prasing them to text files.

  • First of all create a flder to store your VMM log files. I prefer to save them on C:\VMMLogs path.
  • Delete any existing VMM logs if present. In order to do this open up a PowerShell window as an administrator on your VMM server and type logman delete VMM and press enter. There will be warnings such as “Data Collector Set was not found” and you can safetly ignore same.
  • Create a VMM trace. You can use the following command to that.

logman create trace VMM -v mmddhhmm -o $env:SystemDrive\VMMlogs\VMMLog_$env:computername.ETL -cnf 01:00:00 -p Microsoft-VirtualMachineManager-Debug -nb 10 250 -bs 16 -max 512

  • Start the VMM trace by entering logman start vmm in the same PowerShell window.
  • Now you can reproduce the VMM issue that you have faced (i.e a job failure)
  • Immediately after reproducing the iussue you need to stop the VMM strace by entering logman stop vmm
  • The log files you created will be of ETL file format.The ETL is a log file created by Microsoft Tracelog, a program that creates logs using the events from the kernel in Microsoft operating systems and are machine readable. So next step is to convert same to text format.
  • You can convert the collected ETL log by entering Netsh trace convert <Path to file name>

I find these logs very useful specially when the errors in Windows Event Viewer are too generic. In fact debug trace can provide more information if you are encountering bizarre issues in your VMM deployment.

Replication Failure in Azure Site Recovery

Azure Site Recovery is a great product for those who want to setup their DR environment with a minimal cost. It is based on Hyper-V replica technology for Hyper-V workloads and supports replication VMware & Physical server workloads to DR as well. Today I’m going to discuss a common issue one can encounter when enabling ASR replication to the cloud.

I’ve been working on an ASR setup during couple months and encountered strange issue when I enabled replication in protected VMs.

The enable protection job fails with below error.

Job ID: f9f84765-b18c-4002-96a4-d420dfb76ea6-2015-05-14 10:00:29Z

Start Time: 5/14/2015 3:30:29 PM

Duration: 10 MINUTES

Protection couldn’t be enabled for the virtual machine. (Error code: 70094)

Provider error: Unable to complete the request. Operation on the <Hyper-V Node>  timed out.

Try the operation again. (Provider error code: 2924)

Possible causes: Protection can’t be enabled with the virtual machine in its current state. Check the Provider errors for more information.

Recommendation: Fix any issues in the Event Viewer logs (Applications and Service Logs – MicrosoftAzureRecoveryServices) on the Hyper-V host server. If this virtual machine is enabled for replication on the Hyper-V host, disable this setting. Then try to enable protection again.

UTC Time: Thu May 14 2015 10:15:59 GMT+0530 (Sri Lanka Standard Time)

Browser: Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.135 Safari/537.36

Language: en-us

Portal Version: 5.4.00298.11 (rd_auxportal_stable.150511-1702)

PageRequestId: a04f08ed-8932-43f2-95bc-2faab60ed958

Email Address: xxxxxx@outlook.com (MSA)

Subscriptions: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxx

In the particular Hyper-V host following error has been logged in Event logs.

Enable replication failed for virtual machine ‘XXXXXX’ due to a network communication failure. (Virtual Machine ID 807780f6-bb7c-48d5-937d-4857a654dec3, Data Source ID 2256321007502018113, Task ID 8c1a5d7d-0693-4d6b-9243-37cc5e96a7d6)

This ASR setup was a on-premise to Cloud scenario with a single SCVMM server.

After spending a good number of troubleshooting hours I finally figured out what went wrong. The Hyper-V Hosts themselves need Internet connectivity to replicate the VMs to ASR. If you cannot enable direct Internet connectivity on the Hyper-V hosts you should do so via a proxy setup. You can change the proxy settings in ASR Provider in Hyper-V Host.

ASR replication requires traffic to be sent over port 443 (SSL) and in my case only the SCVMM server was configured with Internet access. If you are using a proxy server you may need to consider allowing below for successful replication.

  • *.hypervrecoverymanager.windowsazure.com
  • *.accesscontrol.windows.net
  • *.backup.windowsazure.com
  • *.blob.core.windows.net
  • *.store.core.windows.net
  • Allow the IP addresses in Azure Datacenter IP Ranges and HTTPS (443) protocol. Also your IP address whitelist should contain that of your primary region and  West US IP address ranges.

System Center Technical Preview 2 Released

During Microsoft Ignite, System Center Team has announced the availability of System Center Technical Preview 2 recently. Preview 2 will be ultimately renamed o System Center 2016 when it will be released next year and as of now has some great enhancements over the current version.

Improved Linux Management Capabilities – Preview 2 has Desired State Configuration (DSC), Native SSH support and improved LAMP server monitoring support for your Linux workloads.

Software defined Datacenter Management – The System Center vNext supports mixed mode cluster upgrades, enhanced Scale-Out File Server (SOFS) management, and deployment of software-defined networking (SDN).

New Workload Monitoring – This version is capable of monitoring Azure & Office 365 and SQL & Exchange server monitoring has improved monitoring scenarios.

You can download the installation files from here. Also you can download pre-configured VHD files for each system center component from below.

System Center Technical Preview 2 Virtual Machine Manager VHD

System Center Technical Preview 2 Data Protection Manager VHD

System Center Technical Preview 2 Orchestrator VHD

System Center Technical Preview 2 Operations Manager VHD

System Center Technical Preview 2 Service Manager VHD

Protect your Private Cloud with 5Nine Cloud Security

When it comes to virtualization lot of people start asking questions about how they can secure their environment against security threats. Installing an AV solution inside individual VMs looks like the correct answer but what will happen in case of a network related security threat? Let’s explore the best answer for these issues in Hyper-V context.

5nine Cloud Security is an agentless security solution for Hyper-V which uses the extensible Hyper-V switch capabilities. This solution is capable of providing VM isolation, compliance and antivirus features.

5Nine also offers firewall, AV & IDS functions out of the box. The most important thing about this is it is an agent;less solution where you do not install any agent inside VMs to achieve these goals.

For hosters using Windows Azure Pack 5Nine offers Azure Pack extension which allows them to bring true IDS capabilities to their tenants. As the number of tenants increase security becomes the number one concern of any hoster. Not only that the 5Nine Cloud Security SCVMM plugin let you to deploy all these features via SCVMM if you are only focused about managing your own environment through SCVMM, making it easier to integrate both solutions.

All these features come at an attractive price $199/2 CPUs per host. If you are interested you can visit www.5nine.com for more information. Below is a short demonstration of what 5Nine Cloud Security can do to protect your Hyper-V Hosts, Private Cloud or Service Provider Cloud.

In a future post I’m going to discuss how to configure 5Nine Cloud Security to protect your Microsoft virtualization solution.

Security Alert – Virtual Machine Manager Elevation of Privilege Vulnerability

Microsoft has recently identified an exploit in SCVMM that could allow user privilege elevation. Any hacker who leverages this vulnerability have to first login using Active Directory credentials and could gain administrative privileges and thereby can control VMs managed by a particular VMM Server. Basically this is a result of incorrect user role validation within VMM.

This affects Microsoft System Center Virtual Machine Manager 2012 R2 Update Rollup 4 (VMM Server update 2992024)

Microsoft has immediately released a patch for this issue. You can download the KB3023195 for VMM 2012 R2 Update Rollup 5 from here.

Another important thing to remember is if you have Administrator console installed on your VMM server, you need to install Admin Console Update for UR5 (KB3023914) which can be downloaded from here. When you are installing you’ll have to install UR5 for VMM followed by UR5 for administrator console for proper remediation of this threat.

Tech Update | AWS System Manager for SCVMM

Wouldn’t it be painful just to use the web browser to manage your resources in Amazon EC2 cloud? How about managing them from VMM? That’s not going to be a problem anymore according to amazon.

Recently Amazon introduced AWS System Manager an add-on that can be installed on SCVMM 2012 SP1 onwards. This will let you to manage your EC2 Windows Instances from VMM console.

Basically you can start,stop, restart your AMIs from this tool. If you require remote access you can even RDP into same.

You can download this tool from here. Most importantly it’s FREE.