When working with the main Hadoop services, it is not necessary to work with the console at all time (event though this is the most powerful way of doing so). Most Hadoop distributions also come with a User Interface. The user interface is called “Apache Hue” and is a web-based interface running on top of a distribution. Apache Hue integrates major Hadoop projects in the UI such as Hive, Pig and HCatalog. The nice thing about Apache Hue is that it makes the management of your Hadoop installation pretty easy with a great web-based UI.
The following screenshot shows Apache Hue on the Cloudera distribution.
Apache Sqoop is in charge of moving large datasets between different storage systems such as relational databases to Hadoop. Sqoop supports a large number of connectors such as JDBC to work with different data sources. Sqoop makes it easy to import existing data into Hadoop.
Sqoop supports the following databases:
HSQLDB starting version 1.8
MySQL starting version 5.0
Oracle starting version 10.2
Sqoop provides several possibilities to import and export data from and to Hadoop. The service also provides several mechanisms to validate data.
Most IT departments produce a large amount of log data. This occurs especially when server systems are monitored, but it is also necessary for device monitoring. Apache Flume comes into play when this log data needs to be analyzed.
Flume is all about data collection and aggregation. The architecture is built with a flexible architecture that is based on streaming data flows. The service allows you to extend the data model. Key elements of Flume are:
Event. An event is data that is transported from one place to another place.
Flow. A flow consists of several events that are transported between several places.
Client. A client is the start of a transport. There are several clients available. A frequently used client for example is the Log4j appender.
Agent. An Agent is an independent process that provides components to flume.
Source. This is an interface implementation that is capable of transporting events. A sample of that is an Avro source.
Channels. If a source receives an event, this event is passed on to several channels. A channel is a storage that can handle the event, e.g. JDBC.
Sink. A sink takes an event from the channel and transports it to the next process.
The following figure illustrates the typical workflow for Apache Flume with its components.
Apache Mahout is the service on Hadoop that is in charge of what is often called “data science”. Mahout is all about learning algorithms, pattern recognition and alike. An interesting fact about Mahout is that under the hood MapReduce was replaced by Spark.
Mahout is in charge of the following tasks:
Machine Learning. Learning from existing data and.
Recommendation Mining. This is what we often see at websites. Remember the “You bought X, you might be interested in Y”? This is exactly what Mahout can do for you.
Cluster data. Mahout can cluster documents and data that has some similarities.
Classification. Learn from existing classifications.
A Mahout program is written in Java. The next listing shows how the recommendation builder works.
DataModel model = new FileDataModel(new File(“/home/var/mydata.xml”));
RecommenderEvaluator eval = new AverageAbsoluteDifferenceRecommenderEvaluator();
RecommenderBuilder builder = new MyRecommenderBuilder();
Double res = eval.evaluate(builder, null, model, 0.9, 1.0);
Both Apache Graph and Tez are focused on Graph processing. Apache Giraph is a very popular tool for graph processing. A famous use-case for Giraph is the social graph at Facebook. Facebook uses Giraph to analyze how one might know a person in order to find out what other persons could be friends. Graph processing works on the travelling Sales-Person problem, trying to answer the question on what is the shortest way to get to the customers.
Apache Tez is focused on improving the performance when working with graphs. This makes the development ways easier and reduces the number of MapReduce jobs that are executed underneath it significantly. Apache Tez highly increases the performance against typical MapReduce queries and optimizes the resource management.
The following figure demonstrates graph processing with and without Tez.
S4 is another near-real-time project for Hadoop. S4 is built with a decentralized architecture in mind, focusing on a scaleable and event-oriented architecture. S4 is a long-running process that analyzes streaming data.
S4 is built with Java and with flexibility in mind. This is done via dependency injection, which makes the platform very easy to extend and change. S4 heavily relies on Loose-coupling and dynamic association via the Publish/Subscribe pattern. This makes it easy for S4 to integrate sub-systems into larger systems and updating services on sub-systems can be done independently.
S4 is built to be highly fault-tolerant. Mechanisms built into S4 allow fail-over and recovery.