Azure VM Serial Console Access Preview

One of the biggest challenges in any public cloud IaaS platform is enabling the direct serial-based access so that engineers can fix boot issues. However Microsoft Azure Compute team has overcome this challenge and has introduced serial console access (preview feature) to Azure VMs recently.

This allows access to a text-based console for Linux and Windows virtual machines. Furthermore this serial connection  to COM1 serial port of the VM provides access to the VM settings that are not related to its network/operating system state.

Following prerequisites must be met in order to use this feature.

If you want to disable this feature all you have to do is to disable the boot diagnostics setting for the desired VM.

This can be found under VM setiings blade > Support + Troubleshooting > Serial console (Preview)

Enabling Serial Console access in Windows VMs

  • RDP into your Azure VM running Windows OS.
  • Execute the following commands as an administrator in command prompt.
bcdedit /ems {current} on

bcdedit /emssettings EMSPORT:1 EMSBAUDRATE:115200
  • Once you reboot the VM serial console access will be enabled.

This preview is available in all Azure regions.

Isolated Azure VMs | New Sizes

Microsoft has announced two new IaaS VM sizes called E64i_v3 and E64is_v3 earlier today. These VM SKUs have isolated hardware and can be dedicated to a single customer.Microsoft recommends these two SKUs for workloads that needs a  a high degree of isolation from other customers for compliance and regulatory requirements.
Performance & Pricing Matters

The performance and pricing of E64i_v3 and E64is_v3 are same as their predecessors cousins E64_v3 and E64s_v3. Also the two new SKUs are available in all regions supported by E64_v3 and E64s_v3 sizes. As you may have already guessed, the simple  letter ‘i’ in the VM size name denotes that they are isolated.

Here are some facts about the new sizes.

  • E64i_v3 and E64is_v3 are hardware bound sizes unlike   E64_v3 and E64s_v3.
  • E64i_v3 and E64is_v3 operate on Intel® Xeon® Processor E5-2673 v4 2.3GHz hardware only and will be available until at least December 2021.
  • These sizes will have 12 months in advance after December 2021 to be decommissioned and after that MSFT will  offer updated isolated sizes with their next gen hardware.
  • On May 1st, 2018 these sizes will be made available as one-year Reserved VM Instances.
What are my other options?

Here is the full list of isolate VM sizes that is available in Azure Public cloud as of today.

 

Azure Standard Load Balancer | NSGs are a must

Azure Standard Load Balancer is a new Load Balancer SKU that provide 10x performance and more features than the Basic SKU. One of the coolest features is the ability to use two Standard Load Balancers as a public and internal Load Balancer at the same time. This allows a VM NIC to be connected to the backend pool of one public and one internal Load Balancer resource  where both are of Standard SKU.

I’m my test lab I tried to replicate the Internal + External loadbalancing scenario and found that only the Internal Load Balancer has been working. I’ve tried every possible change I could do including:

  • replicating the same inbound rules as the Internal LB,
  • deleting and re-creating the LB,
  • changing the health probes countless time,
  • changing the health probe settings,

and none of the tweaks I did worked for the External Azure Standard LB.

The Culprit

This article explains the security model of the new Azure Standard Load Balancer.  I had a through look in the documentation and found out the following section.

Standard Load Balancer is fully onboarded to the virtual network. The virtual network is a private, closed network. Because Standard Load Balancers and Standard public IP addresses are designed to allow this virtual network to be accessed from outside of the virtual network, these resources now default to closed unless you open them. This means Network Security Groups (NSGs) are now used to explicitly permit and whitelist allowed traffic. You can create your entire virtual data center and decide through NSG what and when it should be available. If you do not have an NSG on a subnet or NIC of your virtual machine resource, we will not permit traffic to reach this resource.

As you can see if we don’t assign an NSG rule to explicitly allow inbound traffic from the Internet to reach the LB backends (VMs) connected to the Azure Standard Load Balancer. Let’s examine a practical example.

In my example the backend VMs are connected to an Azure Standard LB that resides in the DMZ subnet of a VNet. I have created subnet specific NSGs so therefore the rule I’m going to create is assigned to the DMZ NSG.

As you can see all the inbound traffic originating from the Internet which are destined to ports 443,8443,8444 on DMZ subnet 10.150.4.0/24  have been whitelisted in this rule.  Also note that the External Azure Standard LB has three loadbalancing rules corresponsing to the same ports.

It is a simple solution yet can cause a huge trouble in your environment. In my case not having the NSG rules assigned has resulted in a Traffic Manager endpoint (of course the endpoint was an External Azure Standard LB) to be  in a degraded state during a DR drill. Therefore it is quite important to make sure that you have created the appropriate NSG rules if you are using the Standard SKU for your external Azure LBs.

 

Process Server fails to communicate after ASR Update Rollup 22

Recently I have been working on an ASR implementation for a customer, where it was required to upgrade the MARS version from 9.8 to 9.13. Microsoft has set a deadline of 28th February for this update as after that enabling replication using older version of MARS wouldn’t work. Here is what you see when you are prompted to perform this upgrade.

Since we were running an incompatible version of MARS (Microsoft claims that you need to have a n-4 version of DRA in order to successfully perform this update) we had to perform a step upgrade from 9.8 to 9.10 and then to 9.13. This link provides to reference to that.

The Issue

Both process servers in the environment have been successfully upgraded to 9.13 as a step upgrade. But as soon as the latest version was installed, both the config server and the secondary process server lost communication with the ASR vault and refresh server connection task has been failing for no reason.

We have raised this with Microsoft support and the support engineer assigned to our case found out one alarming issue under ASR operational log under Windows Event viewer in the config server.

 

System.IO.FileNotFoundException: Could not load file or assembly 'Microsoft.IdentityModel, Version=3.5.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35' or one of its dependencies. The system cannot find the file specified.  File name: 'Microsoft.IdentityModel, Version=3.5.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35'    

at SrsRestApiClientLib.SrsCreds.InitializeServiceUrl()    

at SrsRestApiClientLib.SrsCreds..ctor(String resourceId, String siteId, String draId, X509Certificate2 cert, AcsConfiguration acsConfig, Boolean retryFetch, String apiVersion, Proxy webProxy, String msiVersion, String vmmVersion)    

at SrsRestApiClientLib.ClientHelper.InitializeDra(String resourceId, String siteId, String draId, X509Certificate2 cert, AcsConfiguration acsConfig, String apiVersion, Proxy webProxy, String msiVersion, String vmmVersion)    

at Dra.SrsCommunication.SrsCommunicationClient.InitializeSrsCommunication()    

at Dra.SrsCommunication.SrsCommunicationClient..ctor(IDraFabricAdapter fabricAdapter)    

at Dra.Dra.Initialize(IDraFabricAdapter fabricAdapter)    

at Dra.DraFactory.CreateInstance(IDraFabricAdapter fabricAdapter)    

at Microsoft.VirtualManager.Engine.DRA.Core.Core.Initialize()   

WRN: Assembly binding logging is turned OFF.  To enable assembly bind failure logging, set the registry value [HKLM\Software\Microsoft\Fusion!EnableLog] (DWORD) to 1.  Note: There is some performance penalty associated with assembly bind failure logging.  To turn this feature off, remove the registry value [HKLM\Software\Microsoft\Fusion!EnableLog].  ,{00000000-0000-0000-0000-000000000000}



======================

The Culprit

As you may have already notice there is an exception logged stating that the DRA cannot find ‘Microsoft.IdentityModel, Version=3.5.0.0’, which is indeed a part of Windows Identity Foundation 3.5 role in Windows Server. When we checked the installed roles in the config server this role wasn’t present and we have enabled it again to see whether it would have made any difference. Viola!, the process servers reestablished the communication to the vault soon after this role has been installed.

Aftermath

ASR engineering team has confirm that the DRA has a pre-requisite to check whether the config server has Microsoft .NET framework 4.5 installed. Furthermore with .NET 4.5, WIF role is fully integrated into the .NET Framework, meaning that it should be automatically installed alongside .NET 4.5 should have been present in this case. They have reproduced the issue and verified that in upgrade from 9.10 to 9.13, no issues are observed as there are no options available to disable WIF along with .NET Framework 4.5 installation (DRA runs ion a minimum of .NET 4.5 and the latest requirement supports Recently .NET framework 4.6.2).

My best bet is that this has happened when we performed the upgrade from 9.8 to 9.10 as we haven’t reproduced that possibility. The config server haven’t had any major change in its configuration so the only suspect is the 9.8 to 9.10 upgrade. Well I’m exploring that possibility right now and will be publishing a another post if my theory is confirmed. But until then, it still remains a mystery.