Category Archives: Storage

Data corruption issue with NTFS sparse files in Windows Server 2016

Microsoft has released a new patch KB4025334 which prevents a critical data corruption issue with NTFS sparse files in Windows Server 2016.  This patch will prevent possible data corruptions that could occur when using Data Deduplication in Windows Server 2016. However this update also a remedy prevent this issue in all applications and Windows components that leverage sparse files on NTFS in Windows Server 2016.

Although this is an optional update, Microsoft recommends to install this KB to avoid any corruptions in Data deduplication although this KB doesn’t provide  a way to recover from existing data corruptions. The reason being is that NTFS incorrectly removes in-use clusters from the file and there is no way to identify what clusters were incorrectly removed afterwards. Furthermore this update will become a mandatory patch in the “Patch Tuesday” release cycle in August 2017.

Since this issue is hard to notice, you won’t be able detected that by monitoring the weekly Dedup integrity scrubbing job. To overcome this challenge this KB also includes an update to chkdsk which will allow you to identify which files are already corrupted.

Identifying corrupted NTFS sparse files with chkdsk in KB4025334

  • First, install KB4025334 on affected servers and restart same. Keep in mind that if your servers are in a failover cluster this patch needs to be applied for all the servers in your cluster.
  • Execute chkdsk in read-only mode which is the default mode for chkdsk.
  • For any possibly corrupted files, chkdsk will provide an output similar to below. Here 20000000000f3 is the file id and make a note of all the file ids of the output.
The total allocated size in attribute record (128, "") of file 20000000000f3 is incorrect.
  • Then you can use fsutil to query the corrupted files by their ids as per below example.
D:\afftectedfolder> fsutil file queryfilenamebyid D:\ 0x20000000000f3
  • Once you run above command, you should get a similar output like below. D:/affectedfolder/TEST.0 is the corrupted file in this case.
A random link name to this file is [file://%3f/D:/affectedfolder/TEST.0]\\?\D:\affectedfolder\TEST.0

Storage Spaces Direct | Deploying S2D in Azure

This post explores how to build a Storage Space Direct lab in Azure. Bear in mind that S2D in Azure is not a supported scenario for production workloads as of yet.

Following are the high level steps that needs to be followed in order to create provision a S2D lab in Azure. For this lab, I’m using DS1 V2 VMs with Windows Server 2016 Datacenter edition for all the roles and two P20 512 GB Premium SSD disks in each storage node.

Create a VNET

In my Azure tenant I have created a VNET called s2d-vnet with 10.0.0.0/24 address space with a single subnet as below.

1-s2d-create-vnet

Create a Domain Controller

I have deployed a domain controller called jcb-dc in a new windows active directory jcb.com with DNS role installed. Once DNS role has been installed, I have changed the DNS server IP address in the s2d-vnet to my domain controller’s IP address. You may wonder what is the second DNS IP address. It is actually the default Azure DNS IP address added as a redundant DNS server in case if we lose connectivity to the domain controller. This will provide Internet name resolution to the VMs in case domain controller is no longer functional.

1-s2d-vnet-dns

Create the Cluster Nodes

Here I have deployed 3 Windows Server VMs jcb-node1, jcb-node2 and jcb-node3 and joined them to the jcb.com domain. All 3 nodes are deployed in a single availability set.

Configure Failover Clustering

Now we have to configure the Failover Cluster. I’m installing the Failover Clustering role in all 3 nodes using below PowerShell snippet.

$nodes = (“jcb-node01”, “jcb-node02”, “jcb-node03”)

icm $nodes {Install-WindowsFeature Failover-Clustering -IncludeAllSubFeature -IncludeManagementTools}

3-s2d-install-fc

Then I’m going to create the Failover Cluster by executing below snippet in any of the three nodes. This will create a Failover Cluster called JCB-CLU.

$nodes = (“jcb-node01”, “jcb-node02”, “jcb-node03”)

New-Cluster -Name JCB-CLU -Node $nodes –StaticAddress 10.0.0.10

4-s2d-create-fc

Deploying S2D

When I execute Enable-ClusterS2D cmdlet, it will enable Storage Paces Direct and start creating a storage pool automatically as below.

5-s2d-enable-1

5-s2d-enable-2

12-s2d-csv

You can see that the storage pool has been created.

7-s2d-pool-fcm

8-s2d-pool

Creating a Volume

Now we can create a volume in our new S2D setup.

New-Volume -StoragePoolFriendlyName S2D* -FriendlyName JCBVDisk01 -FileSystem CSVFS_REFS -Size 800GB

9-s2d-create-volume

Implementing Scale-out File Server Role

Now we can proceed with SOFS role installation followed by adding SOFS cluster role.

icm $nodes {Install-WindowsFeature FS-FileServer}

Add-ClusterScaleOutFileServerRole -Name jcb-sofs

10-s2d-sofs-install

11-s2d-sofs-enable

Finally I have created an SMB share called Janaka in the newly created CSV.
13-s2d-smb-share

Automating S2D Deployment in Azure with ARM Templates

If you want to automate the entire deployment of the S2D lab you can use below ARM template by Keith Mayer which will create a 2-node S2D Cluster.

Create a Storage Spaces Direct (S2D) Scale-Out File Server (SOFS) Cluster with Windows Server 2016 on an existing VNET

This template requires you to have active VNET and a domain controller deployed first which you can automate using below ARM template. 

Create a 2 new Windows VMs, create a new AD Forest, Domain and 2 DCs in an availability set

We will discuss how to use DISKSPD & VMFLET to perform load and stress testing in a S2D deployment in our next post.

Storage Spaces Direct | Architecture

In my last post I’ve explained the basics of Storage Spaces Direct in Windows Server 2016. This post explores the internals of S2D and it’s architecture in much simple context.

S2D Architecture & Design

(Image Courtesy) Microsoft Technet

S2D is designed to provide nearly 600K IOPS (read) & 1 Tbps of throughput at it’s ultimate configuration with RDMA adapters & NVMe SSD drives. S2D is all about Software Defined Storage and let’s dissect the pieces that makes up the S2D paradigm one by one.

Physical Disks – You can deploy S2D just inside 2 servers up to 16 servers on from 2 to 16 servers with locally-attached SATA, SAS, or NVMe drives. Keep in mind that each server should at least have 2 SSDs, and at least 4 additional drives which can be SAS or SATA HDD. These commodity SATA and SAS devices should be leverage a host-bus adapter (HBA) and SAS expander. 

Software Storage Bus – Think this as the Fiber Channel and Shared SAS cabling in your SAN solution. Software Storage Bus spans across the storage cluster to establish a software-defined storage fabric and allows all the servers can see all the local drives in each and every host in the cluster.

Failover Cluster & Networking – For server communication, S2D leverages the native clustering feature in Windows Server ans uses SMB3, including SMB Direct and SMB Multichannel, over Ethernet. Microsoft recommends to use 10+ GbE (Mellanox) network cards and switches with remote-direct memory access (RDMA), either iWARP or RoCE.

Storage Pool & Storage Spaces – With the recommendation of one pool per cluster Storage Pools consists of the drives that forms the S2D and it is created by discovering and adding all eligible drives automatically to the Storage Pool. Storage Spaces are your software-defined RAID based on Storage Pools. With S2D the data can have tolerance up to two simultaneous drive or server failures along with chassis and rack fault tolerance as well.

Storage Bus Layer Cache – The duty of the Software Storage Bus  is to dynamically bind the fastest drives present  to slower drives (i.e SSD to HDD) which provides server-side read/write caching to accelerate IO and to boost throughput.

Resilient File System (ReFS) & Cluster Shared Volumes – ReFS is a file system that has been built to enhance server virtualization experience in Windows Server. With Acclerated VHDX Operations feature in ReFS it improves the creation, expansion, and checkpoint merging in Virtual Disks significantly. Cluster Shared Volumes consolidate all the ReFS volumes into a single namespace which you can access from any server so it becomes shared storage.

Scale-Out File Server (SOFS) – If your S2D deployment is a Converged solution it is required to implement SOFS which provides remote file access using the SMB3 protocol to clients. i.e Hyper-V Computer Cluster. In a Hyper Converged S2D solution both storage and compute reside in the same cluster thus there is no need to introduce SOFS.

In my next post I’m going to explore how we can deploy S2D in Azure. This will be a Converged setup as Azure doesn’t allow nested virtualization. 

Storage Spaces Direct | Introduction

What is Storage Spaces Direct?

Storage Spaces Direct (S2D) is a new storage feature in Windows Server 2016 which allows you to leverage the locally attached disk drives of the servers in your datacentre to build highly available, highly scalable software-defined storage solutions. S2D helps you save your investments on expensive SAN or NAS solutions by allowing you to use your existing NVMe, SSD or SAS drives combined together to provide high performing and simple storage solutions for your datacentre workloads.

S2D Deployment Choices

There are two deployment options available with S2D.

Converged

In a Converged or disaggreagted S2D architecture, Scale-out File Server/s (SoFS) built on top of S2D  provides shared storage on  SMB3 file shares. Like your traditional NAS systems this separates the storage layer from compute and this option is ideal for large scale enterprise deployments such as Hyper-V VMs hosted by a service provider. 

(Image Courtesy) Microsoft TechNet

Hyper Converged

With Hyper Converged S2D deployments, both compute and storage layers reside in same server/s and this allows to further reduce the hardware cost and ideal for SMEs. 

(Image Courtesy) Microsoft TechNet

S2D is the successor of Storage Spaces introduced in Windows Server 2012 and it is the underlying storage system for Microsoft Azure & Azure Stack. In my next post I will explain about the S2D architecture and key components of an S2D solution in much detail.

Following video explains the core concepts of S2D and it’s  internals and use cases.

Azure Cool Blob Storage | What, Why & How?

What is Azure Cool Blob Storage?

Few days back Microsoft Azure storage team added a new variant of  a storage offering called Cool Blobs. Like Amazon S3, Azure blob storage is a low cost object storage offering for Azure which enables you store your backup, media content such as images and videos, scientific data, compliance and archival data.

Why Cool Blob Storage?

Cool Blob Storage is ideal of infrequent accessed object data, that is data accessed less than once a month. Based  on the frequency of access, you can select between Hot or Cool access tiers for a storage account now. Cool Blob Storage provides following benefits for you as an end user.

  • Cost effective: Data stored at cool access tier comes at a lower price point as low as $0.01 per GB in some regions, where data you store in a hot storage tier start at $0.024 in some regions.
  • Compatibility: This is  100% API compatible with exiting Azure Blob storage and you can use this new type of storage accounts right away in your exiting applications.
  • Performance: Both Hot and Cool tiers have the same performance in terms of latency and throughput.
  • Availability:The data write SLA for Hot access tier is 99.99% where it is 99% for Cool tier. Also the read SLA is 99.99% for Hot tier where it is 99.9 for the Cold tier by leveraging the Read Access-Geo Redundant Storage, storage replica option in Azure.
  • Durability: Unlike Amazon S3 which guarantees you have Nine 11s (99.999999999%) of durability, Microsoft guarantees that your data will never be lost.  The AWS S3 SLA really interprets as “If you store 10,000 objects with us, on average we may lose one of them every 10 million years or so. This storage is designed in such a way that we can sustain the concurrent loss of data in two separate storage facilities.” Both Hot and Cool storage tiers in Azure provide the same high durability that Azure is currently offering which is 0% data loss.
  • Scalability and Security: The same scalability and security options in Azure Storage is provided in the new Blob storage accounts tiers as well.

How to deploy?

Let’s explore how you can create a new blob storage account with hot or cold access tiers in Azure GUI. Notice that this is only possible with ARM storage accounts not with classic storage. Also as of now this feature is only supported in storage accounts with standard performance.Blob Storage 1Changing the access tier is easy and takes only a click of a button.

Blob Storage 2

FAQs

Can I store my VM’s in cool/hot storage? No. Azure IaaS VM disks require page blobs and this is offered only in block blobs.
Can I convert my existing storage account to a Blob storage account? No. You need to create a new storage account or migrate data from an existing storage account to a new account.
Is this available in the classic model? No. This only supports ARM based deployments.
Can I have both hot/cool tiers in a single storage account? Not at this time. The access tier attribute is set at an account level and applies to all objects in that account.
Will I be charged for changing the access tier of my blob storage account? Changing the access tier at an account level will apply to all objects stored in the account. If you are changing from from hot to cool there won’t be any charge but changing from cool to hot will incur a per GB cost for reading all the data in the storage account.

 

 

Introducing ExpressRoute | A safe passage to Windows Azure

Today I got an interesting e-mail from my boss which has caught my attention.  Microsoft has partnered with Equinix, AT&T and Level3 to deliver a new service offered by Windows Azure called ExpressRouteExpressRoute offers private, reliable and low latency connections between customers’ data centers and Azure. This is good news for those who willing to maximize the efficiency of their on-premise infrastructure while moving into the  hybrid cloud. Currently this is available only in US region as a preview feature.

With this new service, you have multiple ways of connecting to Azure.

  1. Aggregate your traffic over VPN connections at Equinix datacenters, or add Azure services to your MPLS VPN provided by AT&T.
  2. Level 3 offers traffic aggregation over a single hand-off or enable multiple locations to connect to Azure services as an extension of your MPLS VPN.

If you fear that your connections to the Microsoft Cloud over Internet is not secure and reliable you can use direct connections to Azure with this feature. Well folks isn’t it exciting news from Microsoft indeed?

References:

Windows Azure Hyper-V Recovery Manager

As a SysAdmin you are responsible for the protection of your data center. We are all comfy with traditional backups and DR methods with off-site backup infrastructure. But what happens if your secondary site caught up in a fire?  The worst case is you are running the whole data center in a private cloud. This gives us a good example of why we should backup our VMs to the cloud.

Windows Hyper-V Recovery Manager protects applications by coordinating the replication of virtual machines at a secondary location. It provides Hyper-V Replica asynchronous virtual machine replication capabilities in Windows Server and the Virtual Machine Manager component of system center and combines them with the power of Windows Azure to provide site-to-site protection of your virtual machines and private clouds. There are three key functions provided by this feature.

Automated Protection

Protection of VMs can be automated once configured. This integrates with Hyper-V replica and SCVMM technologies and delivers on going replication of VMs. Also all the work load data remains in your network which means they neither moved into or passes through Windows Azure.

Continuous Health Monitoring

Once automated protection is in place the Hyper-V Replica Manager monitors the health of VMs real-time with SCVMM. Keep in mind that only SCVMM servers can communicate directly with Azure.

Orchestrated Recovery

In an event of a failure, VMs can be recovered in an orderly fashion. You can predefine which VMs you need to recover first. You can create customized recovery workflow, store them in Azure and even test them before deploying.

How it works Windows Azure Recover Manager

If you have,

  • a secondary site
  • a SCVMM server
  • Unprotected workloads

Then Azure Recovery Manager is the best solution for your data center protection.

Resources

  1. Hyper-V Recovery Manager
  2. Configure Windows Azure Hyper-V Recovery Manager

Take your office to home with Work Folders

Windows Server 2012 R2 release has introduced the concept of Work Folders which is similar to the functionality of Dropbox or SkyDrive to corporate servers. Work folders is a file replication service which enables you to access your corporate files even when you are roaming or at home from your own device.

This feature is only supported in Windows 8.1/8.1 RT clienst yet but Microsoft is planning to introduce it to Windows 7, iPad and Android (may be) pretty much soon. The operation is pretty much simple. Work folders keep copies of files both in server and client, syncs the files when connected to the server. But keep in mind this feature doesn’t support web access or sharing like Dropbox which I think is clear, because you don’t wanna see your sensitive data in wrong hands.

This feature include the following functionality.

  • Data encryption capability and remote data wipe with Windows Intune
  • Security policies for PCs and devices (i.e encrypt lock folders and use a lock screen password)
  • High availability is possible with Failover clustering
  • Files can be accessed offline and will be synced with the central file server when the device is connected to the corporate network or Internet depending on the scenario

OK. What about the limitations and scope considerations on this one?

  • Work folders must reside in local storage of file servers.
  • Cannot sync arbitrary file shares. Users sync to their own folder on the file serve.r (e.g. you can’t sync sales file share to your device)
  • Doesn’t provide sharing & collaboration capabilities. Microsoft recommends using SkyDrive Pro if you need document collaboration features.

If you plan to deploy work folders in your environment, here is the comprehensive TechNet article on how to do it. You can refer the storage team blog article here if you need more insight.

See below video from BJTechNews on how to create work folders in Windows Server 2012 R2.