The “High Availability problem” has been one of the few stumbling blocks in Hadoop’s plans to become the premier data storage and processing platform. The single point of failure offered by Hadoop’s single Name Node approach was greatly publicised as a critical flaw in the framework. In fact, statistics from the Yahoo cluster (one of the largest and most highly utilised Hadoop clusters ever built) show that NameNode failure was so rare that it could be considered negligible and, even in the event of a failure the results were not quite as catastrophic as they were first made out to be (Hadoop World 2011: HDFS NameNode High Availability - Aaron Myers & Sanjay Radia). A minor issue this may have been but still an issue that needed to be fixed to preserve Hadoop’s reputation.

The HA issue has now been solved in the 3 main distributions of Hadoop (considered by this article to be MapR’s M series, Cloudera’s CDH series and HortonWorks HDP) however the approaches, and eventual solutions to the issue could not be more varied. This article intends to explain the different approaches taken by each of these distributions and evaluate the advantages of each.

Cloudera CDH4

Cloudera’s approach to NameNode high availability (HA) is the simplest of the 3. The newest Cloudera distributions include a second NameNode that is launched at the same time as the primary NameNode and, in the event of a failure can be promoted to primary. This “Hot Standby” methodology initially involved a manual step to transition the backup node to primary but Cloudera now recommends that the Apache Zookeeper project be used to provide automatic monitoring and switch over should an event occur.

This approach is both simple and effective but requires that the two nodes (primary and standby) be synchronised. This is currently done through an NFS mount on both nodes and this adds complexity as it is highly recommended that the mount itself be highly available.

It should be noted that in order to provide HA in this way, Cloudera’s distribution is based upon the very latest Apache Hadoop version (2.0). There has been considerable concern over this as some people are of the opinion that the new code has not been fully tested yet and is not ready for commercial release.

HortonWorks HDP

Unlike Cloudera, HortonWorks have not moved to the Hadoop version 2.0 codebase yet and have instead taken an innovative approach to HA based around the version 1.0 codebase. The HortonWorks approach leverages features built into VMware’s VSphere virtualisation offerings.

In HDP the NameNode runs atop a virtual machine that is itself highly available. Using VSphere features an event is identifying by the virtual machine host and, if necessary, a new virtual machine is created to take its place. One particular benefit to this the extreme resilience to hardware failure. Should a hardware failure occur, VSphere will search all hardware associated with the virtual machine host and attempt to restart the machine on the first suitable hardware that can be found. In the case of multiple failures the virtual machine host will provide every effort to restart the NameNode as opposed to the fixed single attempt made by Cloudera’s approach.

The obvious disadvantage to this system however is the reliance on 3rd party management tools. There is a whole extra layer added to the Hadoop cluster that must be managed and supported (at potentially large cost).

The choice of VMWare specifically also poses issues for the cloud. It is expected that cloud based instances of Hadoop will rapidly grow in popularity and virtualisation is not usually supported by cloud providers as their instances are already virtual. This removes any HA functionality for the HortonWorks distribution in the cloud unless specific provisions have been made. It should be noted that Microsoft’s Hadoop on Azure service is thought to be based around the HortonWorks distribution so they must have either made special provisions or foregone HA.

MapR M5

MapR distributions are unique in that they are designed from the ground up with no single point of failure. In MapR the NameNode is itself distributed and therefore able to be very highly resilient. A key bonus to this approach is that both Cloudera’s and HortonWorks’ approaches to HA involve downtime whilst the NameNode is brought back up. As the NameNode is distributed and replicated in MapR, a complete version of it remains held in the cluster at all times (barring the most extreme of failure cases).

This is made possible by proprietary extensions to the Hadoop framework that have been developed by MapR. In recent times MapR has been criticised for diluting the open source nature of Hadoop with these closed extensions but it is a common business model employed in many open source projects.

In conclusion, the solutions outlined above are so varied that the decision as to which Hadoop distribution (and accordingly which HA methodology) is suitable for a particular deployment is likely to depend more on external factors than considerations related to the distribution itself. Perhaps the most critical consideration (given the dynamic nature of the Hadoop project) is whether a choice of HA strategy here may forego future developments in Hadoop. Only Cloudera’s distribution is based on Hadoop version 2.0 and it is thought that soon, if not already, major developments will be applied to this version of the framework only.
Vincent Carbonare