“Big Data” with Windows Azure HDInsight

September 8, 2014
Welcome to part seven of our blog series based on my latest PluralSight course: Applied Azure. Previously, we’ve discussed Azure Web Sites, Azure Worker Roles, Identity and Access with Azure Active Directory, Azure Service Bus and MongoDB, HIPPA Compliant Apps in Azure and Offloading SharePoint Customizations to Azure.

Motivation

No lengthy commentary is needed to communicate the growing importance of big data technologies. Look no further than the rounds of funding [1][2][3] that companies like Cloudera, Hortonworks and MapR have attracted in recent months. It is widely expected that the market for Hadoop will likely grow to $20 Billion by 2018.

The key motivations for the growth of big data technologies includes:

  • The growing need to process ever increasing volumes of data. This growth in data is not limited to web scale companies alone. Businesses of all sizes are seeing this growth.
  • Not all data conforms to a well-defined structure/schema, so there is a need to supplement (if not replace) the traditional data processing and analysis tools such as EDWs.
  • Ability to take advantage of deep compute analytics using massively parallel, commodity based clusters. We will see examples of deep compute analysis a little bit later but this is a growing area of deriving knowledge from the data.
  • Overall simplicity (from the standpoint of the analyst/ developer authoring the query) that hides the non-trivial complexity of the underlying infrastructure.
  • Price-performance benefit accorded due to the commodity based clusters and fault tolerance.
  • The ability to tap into fast paced innovation taking place within the “Hadoop” ecosystem. Consider that Map Reduce, which has been the underpinning of Hadoop ecosystem for years, is being replaced by projects such as Yarn in recent months. Read More…

Offloading SharePoint Customization to Microsoft Azure

August 6, 2014
Welcome to part six of our blog series based on my latest PluralSight course: Applied Azure. Previously, we’ve discussed, HIPAA Compliant Apps with Windows Azure Trust Center,  Azure Web Sites, Azure Worker Roles, Identity and Access with Azure Active Directory and Azure Service Bus and MongoDB.

Motivation

Question: “How does an admin protect their SharePoint farm from poorly written custom code?” Answer: “Force custom code to run in the SharePoint sandbox mode.” Not quite! Turns out that running in a sandbox mode (as the name suggests, it is a restricted execution mode within SharePoint) is not very productive because of the performance penalty and very limited capabilities available to code running in it. A better approach is to move the code “outside” of SharePoint and into a “private” execution environment (so that the errant developers can shoot themselves in the foot, but not everyone else). Read More…

Cloud Patterns with Vishwas Lele

June 17, 2014
Recently, I sat down with hosts Carl Franklin and Richard Campbell of .NET Rocks! to chat about the architectural patterns of cloud development. If you’re not familiar with .NET Rocks! it is a weekly online talk show for anyone interested in programming on the Microsoft .NET platform. The shows range from introductory information to hardcore geekiness.

During this discussion I talk about how the cloud influences application design, focused on more asynchronous, scalable and flexible messaging focused architecture. While the patterns could be applied to any cloud technology, Microsoft Azure is particularly well-suited to these architectural patterns, providing services that cover each pattern approach for optimal results.

Click here to listen to “Cloud Patterns with Vishwas Lele.”

HIPAA Compliant Apps with Windows Azure Trust Center

June 3, 2014
Welcome to part five of our blog series based on my latest PluralSight course: Applied Azure. Previously, we’ve discussed Azure Web Sites, Azure Worker Roles, Identity and Access with Azure Active Directory and Azure Service Bus and MongoDB.

Motivation

Let’s face it, security, privacy and compliance are the key concerns when it comes to adopting any public cloud platform. To alleviate such concerns, Windows Azure team has setup a Windows Azure Trust Center website to provide the latest updates on these topics. Windows Azure complies with several international, country and industry-specific compliance requirements including ISO 27001, FedRAMP, PCI-DSS and HIPAA. In this blog post we are going to focus on building HIPAA compliant applications on the Windows Azure platform.  Read More…

Applied Microsoft Azure: Loosely Coupled Apps with Azure Service Bus and Mongo DB

May 4, 2014
Welcome to part four of our blog series based on my latest PluralSight course: Applied Azure. Previously, we’ve discussed Azure Web Sites, Azure Worker Roles and Identity and Access with Azure Active Directory.

Motivation

Hopefully we can all agree that “loose coupling” is a good thing when it comes to software design. The goal is to build software systems that are made up of parts that have the least possible dependence on each other. The benefits are obvious, we can change, remove or add parts with minimal impact to the rest of the application.

While the design principle of “loose coupling” is applicable all the way from overall system architecture to the lowest level class design, in this blog post, we will focus on coupling between high level sub-systems of an application. We will also focus on message exchange patterns (such as publish-subscribe, temporal decoupling and load balancing) and persistence patterns such (schema less and NoSQL document based store). My objective is to demonstrate how these well-known patterns can be realized using Microsoft Azure Platform based services such as Microsoft Azure Service Bus and MongoDB. Read More…

Applied Windows Azure: Identity & Access with Windows Azure Active Directory

March 11, 2014
Welcome to part three of a blog series based on my latest PluralSight course: Applied Windows Azure. Previously, we’ve discussed Azure Web Sites and Azure Worker Roles.

Motivation

Windows Azure Active Directory (WAAD) is another important building block offered as part of the Windows Azure platform. You can think of WAAD as a repository for your organization’s directory data in the cloud. Directory objects include users and groups along with their identity and access information.

By externalizing the directory data into a common location (WAAD), it is possible to provide a single sign-on and sign-out experience for enterprise applications, as well as SaaS offerings.

While WAAD is a cloud-based service, you can use it for on-premises in addition to cloud-based applications. Read More…

Applied Windows Azure: Compute-Intensive Apps With Azure Worker Roles

March 4, 2014
Welcome to part two of a blog series based on my latest PluralSight course: Applied Windows Azure. If you missed part one, you can read it here.

Motivation

Azure Worker Roles are executing units that can be used to offload long-running, compute-intensive tasks. You can also think of them as “managed” VMs that execute custom tasks for you. I refer to VMs as “managed” because you don’t have to worry about OS, patches, fault-tolerant setup, etc. Worker roles are backed by a 99.95 uptime SLA. Furthermore, it is possible to dynamically scale worker roles based on load (both horizontally by adding more worker role instances, and vertically by provisioning larger VMs). Read More…

Applied Windows Azure: Readymade Execution Units with Azure Web Sites

February 24, 2014
Welcome to the first of an ongoing blog series based on my latest PluralSight course, Applied Windows Azure.

“Applied Windows Azure” as the name suggests, is about utilizing various building blocks to develop practical, meaningful and cost-effective applications that run on Windows Azure.

This course is broken up into 10 modules (listed below). Within each module, after motivating the use of the relevant building Azure building blocks, I cover the core concepts, key mechanisms and design tradeoffs. Of course, no course can be complete without looking at some code – so  I’ll walk through important parts of an application built specifically for each module.  All source code is provided as part of this course.

Finally, the focus is on breadth learning — the ability to operate across the Windows Azure building blocks in a coherent and productive way.  My hope is that the viewer walks away from this course with an idea to apply these Windows Azure building blocks/patterns to a challenge specific to their own domain.  I encourage the viewers and readers to reach out to me at @vlele or comment below anytime to continue this discussion. The value of a course of this type is greatly enhanced by a two-way conversation.

List of Course Modules:

  • Readymade Execution Units with Azure Web Sites
  • Compute Intensive Apps with Azure Worker Roles
  • Identity & Access with Windows Azure Active Directory
  • HIPAA Compliant Apps with Windows Azure Trust Center
  • Loosely Coupled Apps with Azure Service Bus and Mongo DB
  • Offloading SharePoint Customization to Windows Azure
  • Mobile Services as a backend for any app (not just mobile apps)
  • Integration with Windows Azure BizTalk Services
  • “Big Compute” with Azure HPC Services for Excel
  • “Big Data” with Windows Azure HDInsight

I am going to assume some basic understanding of Windows Azure building blocks. If you are new to Azure, there is some excellent material on MSDN and Windows Azure Toolkit to get you started.

Here’s how each blog post for this series will be structured: For each scenario, we start out with the key motivation for the building block, followed by a discussion of scenarios where this building block may be applicable. Next, we will cover the core concepts in detail by walking through a diagram or some sort. Finally, we will look at some of the key design considerations and tradeoffs associated Azure Web Sites. Read More…

Introducing Our “Intelligent” Twitter Bot @AzureUpdates

February 10, 2014
Let’s face it, keeping up with the latest on Windows Azure is hard. Whether it is a new feature announcement, a white paper, a code sample or just another attempt at “cloud washing,” it’s difficult to keep up with the latest, no matter how adept you are at mining the various social media channels.

This is why we built this “intelligent” twitter bot ( @AzureUpdates) as a weekend project. @AzureUpdates is designed to keep you to up to date with all things #WindowsAzure, in and around the Twitterverse.

Here’s how it works:

1) The bot looks for new tweets with the hashtag #WindowsAzure or #Azure *AND* have a (retweet_count + favorite_count) of five or more. Furthermore, we count retweets/favorites only if they are coming from a twitter user with a follower_count of 50 or more.

This way we know that we have a “tweet worth retweeting.” (Okay, fine, I admit I’ve been watching too many “ideas worth spreading”-themed TED talks lately.)

2) Since the #Azure hashtag is used by folks who are not just talking about Windows Azure (overzealous Azure developers may have likely forgotten that “Azure” also stands for a variation of the color blue), we apply a supervised machine learning algorithm based on Naïve Bayes classifier to filter out the tweets that are not referring to Windows Azure. (I am making it sound like I know a lot of machine learning when in fact I don’t. I only just completed a coursera course on machine learning. But if you are interested in this topic, I highly recommend it.)

3)  Now, some sneaky folks don’t retweet; they copy the tweet as a quote, modify it ever so slightly, and then tweet it. Now, we don’t want our twitterbot to be tweeting the same tweet again (remember our objective is to reduce the signal/noise ratio). Fortunately it’s not a tough problem to solve, at least for tweets that contain a hyperlink. Since we log every (re)tweet, we have an indexed inventory of all the “reduced”  hyperlinks. So if someone changes the text but points to the same “reduced” hyperlink, we will simply skip that tweet.

4) Okay, we know what you’re thinking. “I could use my super fancy twitter client, set up a search for the hashtag, language, set engagement counts for retweets, replies and favorite, and filter out unneeded tweets” Well, you certainly can, but who has time for that? Hopefully the sum of all the little “intelligent” features we described above is greater than the individual parts.

That’s it. Hope you follow @AzureUpdates.  Before we go, let me will give you one more reason to follow @AzureUpdates.  It is built on all-around favorite #WindowsAzure #MobileServices!

Thanks to Mohit Chabra from AIS for helping with the development of this bot.