hadoop-mini-clusters

hadoop-mini-clusters provides an easy way to test Hadoop projects directly in your IDE, without the need for a full blown development cluster or container orchestration. It allows the user to debug with the full power of the IDE. It provides a consistent API around the existing Mini Clusters across the ecosystem, eliminating the tedious task of learning the nuances of each project's approach.

Modules:

The project structure changed with 0.1.0. Each mini cluster now resides in a module of its own. See the module names below.

Modules Included:

• hadoop-mini-clusters-hdfs - Mini HDFS Cluster
• hadoop-mini-clusters-yarn - Mini YARN Cluster (no MR)
• hadoop-mini-clusters-mapreduce - Mini MapReduce Cluster
• hadoop-mini-clusters-hbase - Mini HBase Cluster
• hadoop-mini-clusters-zookeeper - Curator based Local Cluster
• hadoop-mini-clusters-hiveserver2 - Local HiveServer2 instance
• hadoop-mini-clusters-hivemetastore - Derby backed HiveMetaStore
• hadoop-mini-clusters-storm - Storm LocalCluster
• hadoop-mini-clusters-kafka - Local Kafka Broker
• hadoop-mini-clusters-oozie - Local Oozie Server - Thanks again Vladimir
• hadoop-mini-clusters-mongodb - I know... not Hadoop
• hadoop-mini-clusters-activemq - Thanks Vladimir Zlatkin!
• hadoop-mini-clusters-hyperscaledb - For testing various databases

Tests:

Tests are included to show how to configure and use each of the mini clusters. See the *IntegrationTest classes.

Windows Support:

Support for Windows Developer workstations has been added. Note that only Window 8 and above are supported. Most of the project will work on Windows 7 except MapReduce/YARN, due to changes in Windows. See: Access Denied Windows 7

Más información en: https://github.com/sakserv/hadoop-mini-clusters