How to Migrate and Troubleshoot File Servers in Azure

June 2, 2020
One of our clients has been pushing big to migrate all the application infrastructure to Azure. Some of the applications have been using on-prem file servers and we have been looking at different options available to migrate these file shares to Azure. We looked at Azure Blob storage, Azure Files and Azure Disks to find out the most fitting solution for us that would offer high performance, permissions at the folder level, and long-term backup retention.

Although Azure Blob storage is great for massive-scale storage, it was easy to dismiss it as our applications were using the native file system and didn’t have to support any streaming and random-access scenarios. Azure Files offered fully managed file shares in the cloud. Unlike Azure Blob storage, Azure Files offer SMB access to Azure file shares. By using SMB, we could mount an Azure file share directly on Windows, Linux, or macOS, either on-premises or in cloud VMs, without writing any code or attaching any special drivers to the file system. It provided windows like file-locking but there were some limitations with Azure Files e.g., we don’t have folder/file level control over the permission and the only way to accomplish that was to create shared access signature on folders where we could specify read-only or write-only permissions, which didn’t work for us. Azure Files Backup provided an easy way to schedule backup and recovery. However, the important limitation was that backups could retain files only for a maximum of 180 days.

Azure Disk fulfilled all our requirements. Although running a file server with Azure Disk as back-end storage was much more expensive than Azure File share but it was the most high-performance file storage option in Azure, which was very important in our scenario as files were used in real-time under heavy load. For compliance and regulatory reasons, all files needed to be backed up, which could be easily done by Azure Virtual Machine Backup without any additional maintenance. The only limitation of Azure Virtual Machine Backup was that it only supports disks that are less than 4 TB. So, in the future, if a need arises for additional storage that meant having multiple disks in a striped volume. Also, after implementing the file server in Azure VM, we could still get the best of both Azure Files and Data Disk by using Azure File Sync. Having a file server and Azure file share in the Sync group would ensure minimal duplication and set the volume of free space. So finally, we decided to deploy a file server in Azure Windows VM with premium SSD.

Troubleshooting

After we deployed the file server in Azure Virtual Machine everything worked like a charm and we had found an ideal solution in Azure for our file servers. However, after some time we used to encounter intermittent issues in the VM, CPU usage would near 100 percent, shares would become inaccessible and VM would hang at OS level bringing everything to a halt.

CPU Average

Generally, to troubleshoot issues in Azure VM, we can connect to a VM using the below tools:

  • Remote CMD
  • Remote PowerShell
  • Remote Registry
  • Remote services console

However, the issue we were encountering hung the OS per se so we could not use the above tools to troubleshoot the VM. There was not even any event log generated that would indicate the possible guest OS hung situation. So, the only option left was to generate the memory dump to find the root cause of the issue. Now, I will explain how to configure the Azure VM for a crash dump and how to trigger the NMI (Non-Maskable Interrupt) crash dumps from the serial console.

Serial Console is a console for Azure VMs that can be accessed from the Azure portal for VMs that have been deployed using the resource management deployment model. It connects directly to COM1 serial port of the VM. From Serial Console we can start a CMD/PowerShell session or send an NMI to VM. NMI creates a signal that VM cannot ignore so it is used as a mechanism to debug or troubleshoot systems that are not responding.

To enable Serial Console, we need to RDP into the VM and run the below commands.

bcdedit   /ems {default} on
bcdedit   /emssettings EMSPORT:1 EMSBAUDRATE:115200

We also need to execute the below commands to enable boot loader prompts.

bcdedit    /set {bootmgr} displaybootmenu yes
bcdedit    /set {bootmgr} timeout 7
bcdedit    /set {bootmgr} bootems yes

When the VM receives an NMI, its response is controlled by the VM configuration. Run the below commands in Windows CMD to configure it to crash and create a memory dump file when receiving an NMI.

REG ADD "HKLM\SYSTEM\CurrentControlSet\Control\CrashControl" /v DumpFile /t REG_EXPAND_SZ /d "%SystemRoot%\MEMORY.DMP" /f
REG ADD "HKLM\SYSTEM\CurrentControlSet\Control\CrashControl" /v NMICrashDump /t REG_DWORD /d 1 /f
REG ADD "HKLM\SYSTEM\CurrentControlSet\Control\CrashControl" /v CrashDumpEnabled /t REG_DWORD /d 1 /f

Before we can use Serial Console to send NMI, we also need to enable Boot diagnostics of VM in Azure portal. Now we are all set to generate the crash dump. Whenever the VM hangs all we need to do is open the Serial Console in the Azure portal, wait till it displays SAC prompt, and hit the Send Non-Maskable Interrupt (NMI) from the menu, as shown below.

Serial Console in Azure Portal

In our case, we have been using FireEye service for security purposes. After we analyzed the memory dump it was found the FeKern.sys (FireEye) is stuck waiting for a spinlock and the driver exhausting CPU % time. FeKern.sys is a file system filter driver that intercepts the requests and in this case, was not able to handle the load. There can be varied reasons for unresponsive VMs and NMI crash dumps can be of great help to troubleshoot and resolve the issues.

Happy Computing!
Sajad Deyargaroo

Work Successfully from Home

April 21, 2020
Even before COVID-19, work from home/anywhere was a growing trend across many industries. Being mutually beneficial –  employers could recruit and retain top talent due to the convenience of remote work, flexibility around time, domestic responsibilities, and geographic flexibility – which is all very appealing to employees, not to mention the lack of a commute. For companies, it not only lowers organizational costs but also boosts employee productivity, according to the recent research at Harvard Business School. Now, the coronavirus pandemic has given a big push to work from home and it has caused almost all firms to deploy the work from home practices for employees, and quickly. Companies with a distributed workforce are already in a better position to keep operations running. For employees who have become accustomed to working from home, there is nothing new. However, those who are doing it for the first time may feel disconnected, distracted, and uncertain about the future of their job. There are many potential challenges employees may face while working from home requiring ground rules to stay motivated, organized, and productive.

Working From Home

Your Work Items

When you work from home there is no clear delineation between work and home. When work and home are in the same place, it’s hard to make the shift from professional to personal. You may start work from sun up to sundown or get distracted because you feel you have too much time. You may find yourselves working more hours and logging in work time in the evening and weekends. As Parkinson’s law states, work expands to fill the time available for its completion. This results in anxiousness and you may become frustrated with yourself.

Set up a routine, it is not necessary to follow nine-to-five, you may be in control and have flexible working hours, which may be good in many cases e.g., some people are not made for the morning even if some are physically there at 9 AM they don’t start being productive until later in the day. Studies have shown by allowing for this, productivity is greatly increased. It’s a company investment. No matter what your working hours are, set reasonable limits to work hours and determine how to meet work requirements and still preserve personal time.

Create a daily ritual e.g., begin your morning with exercise, read a newspaper, or actually enjoy your tea or coffee. Prepare for the day! You may also like to get dressed before starting work. Many find it very useful as it helps to switch from personal to professional. Similarly, after long work hours when you switch back to personal life you may take a shower, as a signal that your workday is over.

Your Work Area

Unfortunately, home life comes with distractions. It’s easy to become distracted by the TV, what’s happening on social media or visitors popping over! If you have children or pets at home setting up limits can be especially tough. In addition, there are always other household errands that need doing, but you are best to put these interruptions off until after your structured working hours as if you were not at home.

Ideally, you will work in a room by yourself behind a closed door, but not everyone has a designated home office. However, it’s critical to have a separate space that you use just for work and not for other activities. Working from home can greatly affect your mind, but creating a workspace of some kind allows you to simulate your brain to recognize it. When you walk into that space, you subconsciously know I am turning on work. When you end for the day, you can say, “I have finished working and now into my evening.” Desk clutter and digital clutter can also be distracting. Having a clean space will result in a clean mind. Ensure family members understand that although you are at home, you are working. You must set working boundaries, if possible. If you have frequent phone or video meetings, you may need a closed-door policy. If you do allow children in your office, make some simple (but fun) rules to follow like knock before entering or never enter when you are on the phone or to remember to use quiet voices.

Your Goals

One of the hardest aspects of working from home is concentrating on the task at hand. Although you set up a workspace and time schedule, you need to set goals for what you will accomplish during that time. You should plan your work before you start the day. Create a task list and keep it in front of you, cut those completed tasks, it will give you a sense of accomplishment. Prioritize your tasks and schedule time slots including extra time in between to cater for contingencies like unforeseen work calls and new tasks, etc. When you begin a task, try to continue it until the end. Do not switch between tasks or get pulled away by your need to respond to a new email. Multi-tasking is a productivity killer, but we all have to multi-task sometimes, the trick is to do it effectively in a limited way.

Your Communication Tools

Today we can work effectively from home, if not more because of the communication tools. There are tons of tools to create more productivity around your communication with your team like email, group chat/call tools like Microsoft Teams, Google Hangouts, etc. Good communication is an essential part of any successful home working arrangement. Besides textual exchanges, we should use multiple forms of communication to avoid any misunderstanding. We need to understand that phone calls, emails, and meetings are still part of our workday just as they are in the company offices. Without open communication, collaboration becomes difficult.

Your Time with your Loved Ones

Many of us love to work in solitude but after some time we may start to feel lonely. If you don’t have anything to look forward to, it may be difficult to keep yourself motivated. So, it is important that you schedule (virtual) time with family and friends. Reward yourself with a book, time with your family, or loved ones.

With freedom comes added responsibilities, it is not easier to work from home, just different location. Happy WFH!

Applied Microsoft Azure: Mobile Services

December 10, 2014
Welcome to part nine of the blog series based on Vishwas Lele’s PluralSight course: Applied Azure. Previously, we’ve discussed Big Compute, Azure Web Sites, Azure Worker Roles, Identity and Access with Azure Active Directory, Azure Service Bus and MongoDB, HIPPA Compliant Apps in Azure and Offloading SharePoint Customizations to Azure and “Big Data” with Windows Azure HDInsight.

Motivation

Windows Azure Mobile Services is a powerful building block in the Windows Azure platform. It brings together a set of services that enables you to create a versatile backend API very quickly. Moreover, it is supported by all major platforms such as Windows 8, Windows Phone, Android, iOS and HTML5. Read More…