WordCount - Hadoop 的 HelloWorld

Thu, 20 Jun 2013 18:14:54 +0800

按照官方教程，http://hadoop.apache.org/docs/stable/mapred_tutorial.html，开始着手写一个 Hadoop 的 HelloWorld 程序，叫做 WordCount

首先准备好代码如下：

package org.myorg;
	
import java.io.IOException;
import java.util.*;

import org.apache.hadoop.fs.Path;
import org.apache.hadoop.conf.*;
import org.apache.hadoop.io.*;
import org.apache.hadoop.mapred.*;
import org.apache.hadoop.util.*;

public class WordCount {

    public static class Map extends MapReduceBase implements Mapper<LongWritable, Text, Text, IntWritable> {
      private final static IntWritable one = new IntWritable(1);
      private Text word = new Text();

      public void map(LongWritable key, Text value, OutputCollector<Text, IntWritable> output, Reporter reporter) throws IOException {
        String line = value.toString();
        StringTokenizer tokenizer = new StringTokenizer(line);
        while (tokenizer.hasMoreTokens()) {
          word.set(tokenizer.nextToken());
          output.collect(word, one);
        }
      }
    }

    public static class Reduce extends MapReduceBase implements Reducer<Text, IntWritable, Text, IntWritable> {
      public void reduce(Text key, Iterator<IntWritable> values, OutputCollector<Text, IntWritable> output, Reporter reporter) throws IOException {
        int sum = 0;
        while (values.hasNext()) {
          sum += values.next().get();
        }
        output.collect(key, new IntWritable(sum));
      }
    }

    public static void main(String[] args) throws Exception {
      JobConf conf = new JobConf(WordCount.class);
      conf.setJobName("wordcount");

      conf.setOutputKeyClass(Text.class);
      conf.setOutputValueClass(IntWritable.class);

      conf.setMapperClass(Map.class);
      conf.setCombinerClass(Reduce.class);
      conf.setReducerClass(Reduce.class);

      conf.setInputFormat(TextInputFormat.class);
      conf.setOutputFormat(TextOutputFormat.class);

      FileInputFormat.setInputPaths(conf, new Path(args[0]));
      FileOutputFormat.setOutputPath(conf, new Path(args[1]));

      JobClient.runJob(conf);
    }
}

然后进行编译打包

$ mkdir wordcount_classes 
$ javac -classpath ${HADOOP_HOME}/hadoop-${HADOOP_VERSION}-core.jar -d wordcount_classes WordCount.java 
$ jar -cvf /usr/joe/wordcount.jar -C wordcount_classes/ .

编译完了可以看到有这三个文件

root@cluster-1:~/workspace/hadoop-1.1.2/wordcount_classes/org/myorg# ll
total 20
drwxr-xr-x 2 root root 4096 2013-06-20 18:00 ./
drwxr-xr-x 3 root root 4096 2013-06-20 18:00 ../
-rw-r--r-- 1 root root 1546 2013-06-20 18:00 WordCount.class
-rw-r--r-- 1 root root 1938 2013-06-20 18:00 WordCount$Map.class
-rw-r--r-- 1 root root 1611 2013-06-20 18:00 WordCount$Reduce.class

打包完得到 wordcount.jar

如果之前有按照官方例程做那个 example 的话，需要先清空文件

root@cluster-1:~/workspace/hadoop-1.1.2# bin/hadoop fs -rmr input
Deleted hdfs://cluster-1:9000/user/root/input
root@cluster-1:~/workspace/hadoop-1.1.2# bin/hadoop fs -rmr output
Deleted hdfs://cluster-1:9000/user/root/output

然后根据教程准备好输入的文件

Sample text-files as input:

$ bin/hadoop dfs -ls /usr/joe/wordcount/input/ /usr/joe/wordcount/input/file01 /usr/joe/wordcount/input/file02

$ bin/hadoop dfs -cat /usr/joe/wordcount/input/file01 Hello World Bye World

$ bin/hadoop dfs -cat /usr/joe/wordcount/input/file02 Hello Hadoop Goodbye Hadoop

然后开始运行

$ bin/hadoop jar wordcount.jar org.myorg.WordCount /user/root/input /user/root/output

得到结果

bin/hadoop fs -cat output/*
Bye	1
Goodbye	1
Hadoop	2
Hello	2
World	2

Wordcount on ZRJ | 学习笔记

WordCount - Hadoop 的 HelloWorld