Cloning for Clusters

After completing SSH and Base Node configuration, I recommend cloning your virtual machine.

One of the clones will be designated the Master (namenode) and the rest of the clones will be designated as Slaves (datanode). I recommend retaining the original VM as a rollback environment. The advantage to cloning at this point is that each clone will be able to talk to any other clone without further setup.

We have a single VM instance with a public key, private key, and authorized_keys file.  The VM is able to SSH back into itself without requiring a password.

If we clone the VM at this point, we'll get a situation like this:

The clone will automatically be able to SSH into the original VM, and vice versa.  Both the clone and the original have identical public and private keys.

Likewise, if we create an additional clone, we have this:

and so on.

Update /etc/hosts (again)

If you've been following along in this series, you already updated this file with the master host.

The hosts file is a computer file used by an operating system to map hostnames to IP addresses. The hosts file is a plain text file, and is conventionally named hosts.

It will be helpful, for the purpose of subsequent configuration, if we modify the hosts file on this VM instance to include the name of each slave and associated IP address.

I currently have two slaves in my cluster, and my hosts file now looks like this:

127.0.0.1     localhost
127.0.1.1     CVB
192.168.1.43     master
192.168.1.45     slave1
192.168.1.46     slave2
# The following lines are desirable for IPv6 capable hosts
::1   ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters

The bold text above was added just now.  The line above that (for the master node) was added when I configured the base VM node.

I prefer to create this file on one node with all the relevant information, and then copy it to each clone that will form part of the cluster.  While not strictly necessary, I find it convenient when each node in the cluster has equivalent information.

Here's a convenient way of copying your hosts file to Slave nodes:

sudo scp /etc/hosts craigtrim@slave1:~
ssh slave1
sudo mv ~/hosts /etc/hosts

You'll need to do this for each designated Slave in the cluster.

The Slaves

The conf/slaves file lists the hosts, one per line, where the Hadoop slave daemons (DataNodes and TaskTrackers) will be run. Just like the masters conf file, list one IP address (or host name) per line.

sudo gedit $HADOOP_HOME/conf/slaves

My file is this simple:

slave1
slave2

How do I know if the slaves file was used correctly?
When running the start-dfs file below, you'll see something like this

 master: STARTUP_MSG:  build = https://git-wip-us.apache.org/repos/asf/hadoop.git -r 2e18d179e4a8065b6a9f29cf2de9451891265cce; compiled by 'jenkins' on 2014-09-05T23:11Z  
 master: STARTUP_MSG:  java = 1.8.0_25  
 master: ************************************************************/  
 192.168.1.35: starting datanode, logging to /usr/lib/apache/hadoop/2.5.1/logs/hadoop-craigtrim-datanode-CVB.out  
 192.168.1.39: starting datanode, logging to /usr/lib/apache/hadoop/2.5.1/logs/hadoop-craigtrim-datanode-CVB.out  
 192.168.1.38: starting datanode, logging to /usr/lib/apache/hadoop/2.5.1/logs/hadoop-craigtrim-datanode-CVB.out  
 192.168.1.41: starting datanode, logging to /usr/lib/apache/hadoop/2.5.1/logs/hadoop-craigtrim-datanode-CVB.out  
 192.168.1.37: starting datanode, logging to /usr/lib/apache/hadoop/2.5.1/logs/hadoop-craigtrim-datanode-CVB.out  
 192.168.1.40: starting datanode, logging to /usr/lib/apache/hadoop/2.5.1/logs/hadoop-craigtrim-datanode-CVB.out  
 192.168.1.36: starting datanode, logging to /usr/lib/apache/hadoop/2.5.1/logs/hadoop-craigtrim-datanode-CVB.out  
 192.168.1.34: starting datanode, logging to /usr/lib/apache/hadoop/2.5.1/logs/hadoop-craigtrim-datanode-CVB.out  

Note each line starting with an IP address.  Seeing this in the log file will prove that the conf/slaves file is being read correctly and at the right time.

Best Practices

When initially creating the cluster, I recommend creating a single master and slave node. Once you configure these successfully, and are able to start the cluster, then you can create multiple clones of the slave node. Create as many clones as you like. The only change you'll need to make to add them to the cluster is to update the slaves file in the master machine's configuration.

Copying the Keys (Optional)

If you've already cloned your VMs as described above, you can skip this step.

If you're not into cloning, you'll need to move the public key from the Master (NameNode) to each Slave (DataNode). If you're not sure whether you need this step or not, try to SSH into each designated Slave (DataNode) from your designated Master (NameNode).

If you are able to SSH without being prompted for a password, you can safely skip this step.

We'll want to move the public key at this location

~/.ssh/id_dsa.pub

To each slave node in the cluster.

We can do this using the scp command like this:

scp ~/.ssh/id_dsa.pub user@master:~/.ssh  

In the slave node, you can go to the .ssh directory and list the contents; make sure this public key exists.

If everything has been set up correctly, you should be able to create a secure channel of communication from your master node to your slave node, like this:

ssh master 

If the configuration has been successful, you’ll be able to set up an SSH channel without having to enter a password.

This is because the public key that belongs to the master has been copied to the slave. Note that it will not be possible to SSH from the slave to the master without a password, since we haven’t performed this operation in reverse.

Configuring a Hadoop Development Environment

This article is part of the Hadoop Masterpage.


When starting with Hadoop, it’s helpful to think of installation and configuration as having two facets:

1. for deployment
1. single-node
2. clustered
2. for development

It is possible that if one is planning to use a cloud-based solution (e.g. IBM's Softlayer) for deployment of MapReduce algorithms, only development configuration and installation is necessary. 

Working with Hadoop for deployment is becoming increasingly uncommon as cloud-based solutions grow in popularity. Given the myriad of hadoop and linux distributions, and the number and variety of configuration scenarios, it’s far easier to configure a version of hadoop on a laptop, and simply use a tool like Maven to automatically generate a JAR file which can then be uploaded to a remote site and deployed in a cloud environment.

It is this second option – installing and configuring Maven for local development – that this tutorial will cover.

Apache Maven

This tutorial assumes familiarity with Apache Maven for building a project. My article Installing Maven on Ubuntu covers installation of Maven (version 3.2.3) on Linux (Ubuntu 14.04).

Initializing the Java Project

Under my home directory I’ve created a workspaces directory, and a test-workspace sub-directory under that:

 mkdir –p ~/workspaces/hadoop/  
 cd ~/workspaces/hadoop/  

Within this test-workspace, I’m going to create my project to work with Hadoop:

 mvn archetype:generate -DgroupId=dev.hadoop.sandbox -DartifactId=sandbox -DarchetypeArtifactId=maven-archetype-quickstart -DinteractiveMode=false  

This will create a project (artifactId = hadoop-dev) with an initial package structure (groupId = dev.hadoop).

Operational Output

When this command is executed for the first time, various files necessary to Maven will be downloaded.  Depending on your internet connection, this may take a while.

Ultimately, you should end up with something like this:

 craigtrim@CVB:~/workspace/hadoop$ mvn archetype:generate -DgroupId=dev.hadoop.sandbox -DartifactId=sandbox -DarchetypeArtifactId=maven-archetype-quickstart -DinteractiveMode=false  
 [INFO] Scanning for projects...  
 [INFO]                                       
 [INFO] ------------------------------------------------------------------------  
 [INFO] Building Maven Stub Project (No POM) 1  
 [INFO] ------------------------------------------------------------------------  
 [INFO]   
 [INFO] >>> maven-archetype-plugin:2.2:generate (default-cli) > generate-sources @ standalone-pom >>>  
 [INFO]   
 [INFO] <<< maven-archetype-plugin:2.2:generate (default-cli) < generate-sources @ standalone-pom <<<  
 [INFO]   
 [INFO] --- maven-archetype-plugin:2.2:generate (default-cli) @ standalone-pom ---  
 [INFO] Generating project in Batch mode  
 [INFO] ----------------------------------------------------------------------------  
 [INFO] Using following parameters for creating project from Old (1.x) Archetype: maven-archetype-quickstart:1.0  
 [INFO] ----------------------------------------------------------------------------  
 [INFO] Parameter: basedir, Value: /home/craigtrim/workspace/hadoop  
 [INFO] Parameter: package, Value: dev.hadoop.sandbox  
 [INFO] Parameter: groupId, Value: dev.hadoop.sandbox  
 [INFO] Parameter: artifactId, Value: sandbox  
 [INFO] Parameter: packageName, Value: dev.hadoop.sandbox  
 [INFO] Parameter: version, Value: 1.0-SNAPSHOT  
 [INFO] project created from Old (1.x) Archetype in dir: /home/craigtrim/workspace/hadoop/sandbox  
 [INFO] ------------------------------------------------------------------------  
 [INFO] BUILD SUCCESS  
 [INFO] ------------------------------------------------------------------------  
 [INFO] Total time: 1.993 s  
 [INFO] Finished at: 2014-11-24T15:57:47-08:00  
 [INFO] Final Memory: 13M/241M  
 [INFO] ------------------------------------------------------------------------  
 craigtrim@CVB:~/workspace/hadoop$   

Look for the line in your terminal output that reads "BUILD SUCCESS".

Updating the POM

Next, navigate into the project

 cd sandbox  

and edit the pom.xml file with the text editor of your choice.

 sudo gedit pom.xml  

You will want to add the Maven Dependencies for Hadoop.

My final POM looks like this:

 <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"  
      xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">  
      <modelVersion>4.0.0</modelVersion>  
      <groupId>dev.hadoop.sandbox</groupId>  
      <artifactId>sandbox</artifactId>  
      <packaging>jar</packaging>  
      <version>1.0-SNAPSHOT</version>  
      <name>sandbox</name>  
      <url>http://maven.apache.org</url>  
      <dependencies>  
           <dependency>  
                <groupId>junit</groupId>  
                <artifactId>junit</artifactId>  
                <version>3.8.1</version>  
                <scope>test</scope>  
           </dependency>  
           <dependency>  
                <groupId>org.apache.hadoop</groupId>  
                <artifactId>hadoop-core</artifactId>  
                <version>1.2.1</version>  
           </dependency>  
           <dependency>  
                <groupId>org.apache.hadoop</groupId>  
                <artifactId>hadoop-client</artifactId>  
                <version>2.5.2</version>  
           </dependency>  
           <dependency>  
                <groupId>org.apache.hadoop</groupId>  
                <artifactId>hadoop-hdfs</artifactId>  
                <version>2.5.2</version>  
           </dependency>  
           <dependency>  
                <groupId>org.apache.hadoop</groupId>  
                <artifactId>hadoop-common</artifactId>  
                <version>2.5.2</version>  
           </dependency>  
      </dependencies>  
 </project>  

Package the Project

Now we're going to execute what's known as a a project phase.  A phase is a step in the build lifecycle.

Navigate into the project

 cd /home/craigtrim/workspace/hadoop/sandbox  

and run this command:

 mvn package  

This is the "package command" and will instruct Maven to take the compiled code and package it in its distributable format, such as a JAR.

This command will test the JAR file:

 java -cp target/sandbox-1.0-SNAPSHOT.jar dev.hadoop.sandbox.App  

Operational Output

Successful output on my workstation looks like this:

 craigtrim@CVB:~$ cd ~/workspace/hadoop/  
 craigtrim@CVB:~/workspace/hadoop$ mvn package  
 [INFO] Scanning for projects...  
 [INFO] ------------------------------------------------------------------------  
 [INFO] BUILD FAILURE  
 [INFO] ------------------------------------------------------------------------  
 [INFO] Total time: 0.130 s  
 [INFO] Finished at: 2014-11-24T16:02:56-08:00  
 [INFO] Final Memory: 5M/241M  
 [INFO] ------------------------------------------------------------------------  
 [ERROR] The goal you specified requires a project to execute but there is no POM in this directory (/home/craigtrim/workspace/hadoop). Please verify you invoked Maven from the correct directory. -> [Help 1]  
 [ERROR]   
 [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.  
 [ERROR] Re-run Maven using the -X switch to enable full debug logging.  
 [ERROR]   
 [ERROR] For more information about the errors and possible solutions, please read the following articles:  
 [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MissingProjectException  
 craigtrim@CVB:~/workspace/hadoop$ clear  
 craigtrim@CVB:~/workspace/hadoop$ cd sandbox/  
 craigtrim@CVB:~/workspace/hadoop/sandbox$ mvn package  
 [INFO] Scanning for projects...  
 [INFO]                                       
 [INFO] ------------------------------------------------------------------------  
 [INFO] Building sandbox 1.0-SNAPSHOT  
 [INFO] ------------------------------------------------------------------------  
 [INFO]   
 [INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ sandbox ---  
 [WARNING] Using platform encoding (UTF-8 actually) to copy filtered resources, i.e. build is platform dependent!  
 [INFO] skip non existing resourceDirectory /home/craigtrim/workspace/hadoop/sandbox/src/main/resources  
 [INFO]   
 [INFO] --- maven-compiler-plugin:3.1:compile (default-compile) @ sandbox ---  
 [INFO] Changes detected - recompiling the module!  
 [WARNING] File encoding has not been set, using platform encoding UTF-8, i.e. build is platform dependent!  
 [INFO] Compiling 1 source file to /home/craigtrim/workspace/hadoop/sandbox/target/classes  
 [INFO]   
 [INFO] --- maven-resources-plugin:2.6:testResources (default-testResources) @ sandbox ---  
 [WARNING] Using platform encoding (UTF-8 actually) to copy filtered resources, i.e. build is platform dependent!  
 [INFO] skip non existing resourceDirectory /home/craigtrim/workspace/hadoop/sandbox/src/test/resources  
 [INFO]   
 [INFO] --- maven-compiler-plugin:3.1:testCompile (default-testCompile) @ sandbox ---  
 [INFO] Changes detected - recompiling the module!  
 [WARNING] File encoding has not been set, using platform encoding UTF-8, i.e. build is platform dependent!  
 [INFO] Compiling 1 source file to /home/craigtrim/workspace/hadoop/sandbox/target/test-classes  
 [INFO]   
 [INFO] --- maven-surefire-plugin:2.12.4:test (default-test) @ sandbox ---  
 [INFO] Surefire report directory: /home/craigtrim/workspace/hadoop/sandbox/target/surefire-reports  
 -------------------------------------------------------  
  T E S T S  
 -------------------------------------------------------  
 Running dev.hadoop.sandbox.AppTest  
 Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.011 sec  
 Results :  
 Tests run: 1, Failures: 0, Errors: 0, Skipped: 0  
 [INFO]   
 [INFO] --- maven-jar-plugin:2.4:jar (default-jar) @ sandbox ---  
 [INFO] Building jar: /home/craigtrim/workspace/hadoop/sandbox/target/sandbox-1.0-SNAPSHOT.jar  
 [INFO] ------------------------------------------------------------------------  
 [INFO] BUILD SUCCESS  
 [INFO] ------------------------------------------------------------------------  
 [INFO] Total time: 2.993 s  
 [INFO] Finished at: 2014-11-24T16:03:13-08:00  
 [INFO] Final Memory: 25M/319M  
 [INFO] ------------------------------------------------------------------------  
 craigtrim@CVB:~/workspace/hadoop/sandbox$   

Keep in mind, I've instructed Maven to download Hadoop JARs in the past.

The output I have shared above occurs when running this command on subsequent iterations. The first time a new dependency is added to a POM file, Maven will seek out all the direct and indirect dependencies, and download them to your local repository.  Check toward the end of this article for repository navigation and dependency visualization.

Preparing for Eclipse

Once the packaging phase has completed, we need to prepare the project for use within Eclipse.

All the Eclipse standard meta-data required to support this project within the IDE can be generated using this command:

 mvn eclipse:eclipse  

from within the project (same location as the mvn package) command.

Operational Output

Successful output on my workstation looks like this:

 craigtrim@CVB:~/workspace/hadoop/sandbox$ mvn eclipse:eclipse  
 [INFO] Scanning for projects...  
 [INFO]                                       
 [INFO] ------------------------------------------------------------------------  
 [INFO] Building sandbox 1.0-SNAPSHOT  
 [INFO] ------------------------------------------------------------------------  
 [INFO]   
 [INFO] >>> maven-eclipse-plugin:2.9:eclipse (default-cli) > generate-resources @ sandbox >>>  
 [INFO]   
 [INFO] <<< maven-eclipse-plugin:2.9:eclipse (default-cli) < generate-resources @ sandbox <<<  
 [INFO]   
 [INFO] --- maven-eclipse-plugin:2.9:eclipse (default-cli) @ sandbox ---  
 [INFO] Using Eclipse Workspace: /home/craigtrim/workspace/hadoop  
 [INFO] Adding default classpath container: org.eclipse.jdt.launching.JRE_CONTAINER  
 [INFO] Not writing settings - defaults suffice  
 [INFO] Wrote Eclipse project for "sandbox" to /home/craigtrim/workspace/hadoop/sandbox.  
 [INFO]   
 [INFO] ------------------------------------------------------------------------  
 [INFO] BUILD SUCCESS  
 [INFO] ------------------------------------------------------------------------  
 [INFO] Total time: 2.819 s  
 [INFO] Finished at: 2014-11-24T16:32:20-08:00  
 [INFO] Final Memory: 14M/431M  
 [INFO] ------------------------------------------------------------------------  
 craigtrim@CVB:~/workspace/hadoop/sandbox$  

Running Eclipse

Open Eclipse at the workspace root:

Once inside, you'll need to import the "sandbox" project using

 File > Import > Existing Projects into Workspace  

Build Phases

While this article is not strictly a Maven tutorial, it's helpful to understand what happens when "mvn package" is executed on the command line.

The package command will execute the following build phases that preceded it:

 validate - validate the project is correct and all necessary information is available  
 compile - compile the source code of the project  
 test - test the compiled source code using a suitable unit testing framework. These tests should not require the code be packaged or deployed  
 package - take the compiled code and package it in its distributable format, such as a JAR.  
 integration-test - process and deploy the package if necessary into an environment where integration tests can be run  
 verify - run any checks to verify the package is valid and meets quality criteria  
 install - install the package into the local repository, for use as a dependency in other projects locally  
 deploy - done in an integration or release environment, copies the final package to the remote repository for sharing with other developers and projects.  

Repository Navigation

In Ubuntu, I can view the local Maven repository by opening a file browser and navigating to my Home directory. Once there, I use the CTRL+H command to show hidden files and folders. The .m2 folder should appear and look something like this:

The folder above contains many of the more direct dependencies required by the four dependencies I've added to the POM file.

Dependency Visualization

To configure Eclipse for Hadoop development, we had to specify four dependencies in the Maven POM file.

It turns out, there's exactly 84 JAR files (at the time of this article) required to support this configuration. Maven prevents us from having to track down each of these 84 dependencies, most of them indirect (to the nth degree!) by downloading them for us, and resolving any conflicts automagically.

To see what's really going on, there's a handy tree visualization command that we can use.

Run this in the same directory as your project:

 mvn dependency:tree  

Operational Output

Successful output on my workstation looks like this:

  org.test.hadoop:hadoop:jar:1.0-SNAPSHOT   
  +- junit:junit:jar:3.8.1:test   
  +- org.apache.hadoop:hadoop-core:jar:1.2.1:compile   
  | +- commons-cli:commons-cli:jar:1.2:compile   
  | +- xmlenc:xmlenc:jar:0.52:compile   
  | +- com.sun.jersey:jersey-core:jar:1.8:compile   
  | +- com.sun.jersey:jersey-json:jar:1.8:compile   
  | | +- org.codehaus.jettison:jettison:jar:1.1:compile   
  | | | \- stax:stax-api:jar:1.0.1:compile   
  | | +- com.sun.xml.bind:jaxb-impl:jar:2.2.3-1:compile   
  | | | \- javax.xml.bind:jaxb-api:jar:2.2.2:compile   
  | | |  +- javax.xml.stream:stax-api:jar:1.0-2:compile   
  | | |  \- javax.activation:activation:jar:1.1:compile   
  | | +- org.codehaus.jackson:jackson-jaxrs:jar:1.7.1:compile   
  | | \- org.codehaus.jackson:jackson-xc:jar:1.7.1:compile   
  | +- com.sun.jersey:jersey-server:jar:1.8:compile   
  | | \- asm:asm:jar:3.1:compile   
  | +- commons-io:commons-io:jar:2.1:compile   
  | +- commons-httpclient:commons-httpclient:jar:3.0.1:compile   
  | +- commons-codec:commons-codec:jar:1.4:compile   
  | +- org.apache.commons:commons-math:jar:2.1:compile   
  | +- commons-configuration:commons-configuration:jar:1.6:compile   
  | | +- commons-digester:commons-digester:jar:1.8:compile   
  | | | \- commons-beanutils:commons-beanutils:jar:1.7.0:compile   
  | | \- commons-beanutils:commons-beanutils-core:jar:1.8.0:compile   
  | +- commons-net:commons-net:jar:1.4.1:compile   
  | +- org.mortbay.jetty:jetty:jar:6.1.26:compile   
  | | \- org.mortbay.jetty:servlet-api:jar:2.5-20081211:compile   
  | +- org.mortbay.jetty:jetty-util:jar:6.1.26:compile   
  | +- tomcat:jasper-runtime:jar:5.5.12:compile   
  | +- tomcat:jasper-compiler:jar:5.5.12:compile   
  | +- org.mortbay.jetty:jsp-api-2.1:jar:6.1.14:compile   
  | | \- org.mortbay.jetty:servlet-api-2.5:jar:6.1.14:compile   
  | +- org.mortbay.jetty:jsp-2.1:jar:6.1.14:compile   
  | | \- ant:ant:jar:1.6.5:compile   
  | +- commons-el:commons-el:jar:1.0:compile   
  | +- net.java.dev.jets3t:jets3t:jar:0.6.1:compile   
  | +- hsqldb:hsqldb:jar:1.8.0.10:compile   
  | +- oro:oro:jar:2.0.8:compile   
  | +- org.eclipse.jdt:core:jar:3.1.1:compile   
  | \- org.codehaus.jackson:jackson-mapper-asl:jar:1.8.8:compile   
  +- org.apache.hadoop:hadoop-client:jar:2.5.2:compile   
  | +- org.apache.hadoop:hadoop-mapreduce-client-app:jar:2.5.2:compile   
  | | +- org.apache.hadoop:hadoop-mapreduce-client-common:jar:2.5.2:compile   
  | | | +- org.apache.hadoop:hadoop-yarn-client:jar:2.5.2:compile   
  | | | | \- com.sun.jersey:jersey-client:jar:1.9:compile   
  | | | \- org.apache.hadoop:hadoop-yarn-server-common:jar:2.5.2:compile   
  | | \- org.apache.hadoop:hadoop-mapreduce-client-shuffle:jar:2.5.2:compile   
  | |  \- org.fusesource.leveldbjni:leveldbjni-all:jar:1.8:compile   
  | +- org.apache.hadoop:hadoop-yarn-api:jar:2.5.2:compile   
  | +- org.apache.hadoop:hadoop-mapreduce-client-core:jar:2.5.2:compile   
  | | \- org.apache.hadoop:hadoop-yarn-common:jar:2.5.2:compile   
  | +- org.apache.hadoop:hadoop-mapreduce-client-jobclient:jar:2.5.2:compile   
  | \- org.apache.hadoop:hadoop-annotations:jar:2.5.2:compile   
  +- org.apache.hadoop:hadoop-hdfs:jar:2.5.2:compile   
  | +- com.google.guava:guava:jar:11.0.2:compile   
  | +- commons-lang:commons-lang:jar:2.6:compile   
  | +- commons-logging:commons-logging:jar:1.1.3:compile   
  | +- commons-daemon:commons-daemon:jar:1.0.13:compile   
  | +- javax.servlet.jsp:jsp-api:jar:2.1:compile   
  | +- log4j:log4j:jar:1.2.17:compile   
  | +- com.google.protobuf:protobuf-java:jar:2.5.0:compile   
  | +- javax.servlet:servlet-api:jar:2.5:compile   
  | +- org.codehaus.jackson:jackson-core-asl:jar:1.9.13:compile   
  | \- io.netty:netty:jar:3.6.2.Final:compile   
  \- org.apache.hadoop:hadoop-common:jar:2.5.2:compile   
   +- org.apache.commons:commons-math3:jar:3.1.1:compile   
   +- commons-collections:commons-collections:jar:3.2.1:compile   
   +- org.slf4j:slf4j-api:jar:1.7.5:compile   
   +- org.slf4j:slf4j-log4j12:jar:1.7.5:compile   
   +- org.apache.avro:avro:jar:1.7.4:compile   
   | +- com.thoughtworks.paranamer:paranamer:jar:2.3:compile   
   | \- org.xerial.snappy:snappy-java:jar:1.0.4.1:compile   
   +- org.apache.hadoop:hadoop-auth:jar:2.5.2:compile   
   | +- org.apache.httpcomponents:httpclient:jar:4.2.5:compile   
   | | \- org.apache.httpcomponents:httpcore:jar:4.2.4:compile   
   | \- org.apache.directory.server:apacheds-kerberos-codec:jar:2.0.0-M15:compile   
   |  +- org.apache.directory.server:apacheds-i18n:jar:2.0.0-M15:compile   
   |  +- org.apache.directory.api:api-asn1-api:jar:1.0.0-M20:compile   
   |  \- org.apache.directory.api:api-util:jar:1.0.0-M20:compile   
   +- com.jcraft:jsch:jar:0.1.42:compile   
   +- com.google.code.findbugs:jsr305:jar:1.3.9:compile   
   +- org.apache.zookeeper:zookeeper:jar:3.4.6:compile   
   \- org.apache.commons:commons-compress:jar:1.4.1:compile   
   \- org.tukaani:xz:jar:1.0:compile   

The effort to establish all of those dependencies manually would be immense.  Manual dependency resolution is possible, but extremely difficult and error prone.