Hadoop on ZRJ | 学习笔记

hadoop yarn hdfs 概念的理解

Fri, 22 Apr 2016 00:13:58 +0800

有几个说的不错的文章，看这里，https://www.ibm.com/developerworks/cn/opensource/os-cn-hadoop-yarn/，来自 ibm，从理论到例子，讲解了 yarn 的产生和变化

http://www.lai18.com/content/1103036.html，对计算框架的历史演变有一个概述

http://www.cnblogs.com/LeftNotEasy/archive/2012/02/18/why-yarn.html，辅助理解 yarn 的一些产生背景

=================

http://www.cnblogs.com/esingchan/p/3917094.html，一一个 wordcount 为例，说明了老 mapreduce 的过程，详细说明了 map sort partition combine shuffle merge reduce 的阶段和概念

Hadoop HBase 碎碎念

Fri, 25 Mar 2016 22:08:12 +0800

一些小笔记会持续追加到这里

2016-3-25 22:09:24

重启或添加节点： $bin/hadoop-daemon.sh start datanode

-————————-

2016-3-25 22:09:31

Had the same problem with 2.6.0, and shamouda’s answer solved it (I was not using dfs.hosts at all so that could not be the answer. I did add

<property>
  <name>dfs.namenode.datanode.registration.ip-hostname-check</name>
  <value>false</value>
</property>

to hdfs-site.xml and that was enough to fix the issue.

-———————-

2016-3-25 22:11:55

$bin/hbase-daemon.sh stop regionserver $bin/hbase-daemon.sh start regionserver

-———————–

2016-3-27 16:54:22

disable ‘kline_minute’ describe ‘kline_minute’ alter ‘kline_minute’ , NAME=>‘kline’, COMPRESSION=>‘gz’ describe ‘kline_minute’ enable ‘kline_minute’ major_compact ‘kline_minute’

启用压缩

http://www.searchtb.com/2011/01/understanding-hbase.html，这里说明了文件的存储方式 http://stackoverflow.com/questions/18656483/data-size-increases-in-hbase，这里有一个计算的例子

-———————–

2016-3-27 16:56:14

重启一个regionserver

bin/graceful_stop.sh –restart –reload –debugnodename

这个操作是平滑的重启regionserver进程，对服务不会有影响，他会先将需要重启的regionserver上面的所有region迁移到其它的服务器，然后重启，最后又会将之前的region迁移回来，但我们修改一个配置时，可以用这种方式重启每一台机子，这个命令会关闭balancer，所以最后我们要在hbase shell里面执行一下balance_switch true，对于hbase regionserver重启，不要直接kill进程，这样会造成在zookeeper.session.timeout这个时间长的中断，也不要通过bin/hbase-daemon.sh stop regionserver去重启，如果运气不太好，-ROOT-或者.META.表在上面的话，所有的请求会全部失败。

关闭下线一台regionserver

bin/graceful_stop.sh –stop nodename

和上面一样，系统会在关闭之前迁移所有region，然后stop进程，同样最后我们要手工balance_switch true，开启master的region均衡。

hadoop 2.5.2 集群安装

Fri, 18 Mar 2016 16:40:52 +0800

之前写过一个，http://zrj.me/archives/888，不过现在是 2.5.2，东西不同了，安装方式有所变化。

说起来，主要以下几点：

安装 JDK
打通 SSH
停掉防火墙
配置 core-site.xml, maprd-site.xml, hdfs-site.xml，新增了一个 yarn-site.xml

参考这里的文章，http://blog.csdn.net/tang9140/article/details/42869531，http://blog.csdn.net/greensurfer/article/details/39450369，配置文件使用如下：

core-site.xml

<configuration>  
    <property>  
        <name>hadoop.tmp.dir</name>  
        <value>/home/hadoop/tmp</value>  
        <description>Abase for other temporary directories.</description>  
    </property>  
    <property>  
        <name>fs.defaultFS</name>  
        <value>hdfs://10.0.1.100:9000</value>  
    </property>  
    <property>  
        <name>io.file.buffer.size</name>  
        <value>4096</value>  
    </property>  
</configuration>

hdfs-site.xml

<configuration>  
    <property>  
        <name>dfs.nameservices</name>  
        <value>hadoop-cluster1</value>  
    </property>  
    <property>  
        <name>dfs.namenode.secondary.http-address</name>  
        <value>10.0.1.100:50090</value>  
    </property>  
    <property>  
        <name>dfs.namenode.name.dir</name>  
        <value>/home/hadoop/dfs/name</value>  
    </property>  
    <property>  
        <name>dfs.datanode.data.dir</name>  
        <value>/home/hadoop/dfs/data</value>  
    </property>  
    <property>  
        <name>dfs.replication</name>  
        <value>2</value>  
    </property>  
    <property>  
        <name>dfs.webhdfs.enabled</name>  
        <value>true</value>  
    </property>  
</configuration>

mapred-site.xml

<configuration>  
    <property>  
        <name>mapreduce.framework.name</name>  
        <value>yarn</value>  
    </property>  
    <property>  
        <name>mapreduce.jobtracker.http.address</name>  
        <value>10.0.1.100:50030</value>  
    </property>  
    <property>  
        <name>mapreduce.jobhistory.address</name>  
        <value>10.0.1.100:10020</value>  
    </property>  
    <property>  
        <name>mapreduce.jobhistory.webapp.address</name>  
        <value>10.0.1.100:19888</value>  
    </property>  
</configuration>

yarn-site.xml

<configuration>  
  
<!-- Site specific YARN configuration properties -->  
    <property>  
        <name>yarn.nodemanager.aux-services</name>  
        <value>mapreduce_shuffle</value>  
    </property>  
    <property>  
        <name>yarn.resourcemanager.address</name>  
        <value>10.0.1.100:8032</value>  
    </property>  
    <property>  
        <name>yarn.resourcemanager.scheduler.address</name>  
        <value>10.0.1.100:8030</value>  
    </property>  
    <property>  
        <name>yarn.resourcemanager.resource-tracker.address</name>  
        <value>10.0.1.100:8031</value>  
    </property>  
    <property>  
        <name>yarn.resourcemanager.admin.address</name>  
        <value>10.0.1.100:8033</value>  
    </property>  
    <property>  
        <name>yarn.resourcemanager.webapp.address</name>  
        <value>10.0.1.100:8088</value>  
    </property>  
</configuration>

最后配置 slaves 文件，没有了 master 文件

然后格式化 namenode，输出如下

zrj@FWSVRLPT01:~/hadoop-2.5.2$ ./bin/hdfs namenode -format
16/03/18 16:22:31 INFO namenode.NameNode: STARTUP_MSG: 
/************************************************************
STARTUP_MSG: Starting NameNode
STARTUP_MSG:   host = FWSVRLPT01/10.16.18.206
STARTUP_MSG:   args = [-format]
STARTUP_MSG:   version = 2.5.2
STARTUP_MSG:   classpath = /home/zrj/hadoop-2.5.2/etc/hadoop:/home/zrj/hadoop-2.5.2/share/hadoop/common/lib/commons-configuration-1.6.jar:/home/zrj/hadoop-2.5.2/share/hadoop/common/lib/jasper-compiler-5.5.23.jar:/home/zrj/hadoop-2.5.2/share/hadoop/common/lib/activation-1.1.jar:/home/zrj/hadoop-2.5.2/share/hadoop/common/lib/jackson-jaxrs-1.9.13.jar:/home/zrj/hadoop-2.5.2/share/hadoop/common/lib/jsp-api-2.1.jar:/home/zrj/hadoop-2.5.2/share/hadoop/common/lib/jaxb-impl-2.2.3-1.jar:/home/zrj/hadoop-2.5.2/share/hadoop/common/lib/apacheds-kerberos-codec-2.0.0-M15.jar:/home/zrj/hadoop-2.5.2/share/hadoop/common/lib/commons-io-2.4.jar:/home/zrj/hadoop-2.5.2/share/hadoop/common/lib/paranamer-2.3.jar:/home/zrj/hadoop-2.5.2/share/hadoop/common/lib/httpclient-4.2.5.jar:/home/zrj/hadoop-2.5.2/share/hadoop/common/lib/log4j-1.2.17.jar:/home/zrj/hadoop-2.5.2/share/hadoop/common/lib/jets3t-0.9.0.jar:/home/zrj/hadoop-2.5.2/share/hadoop/common/lib/zookeeper-3.4.6.jar:/home/zrj/hadoop-2.5.2/share/hadoop/common/lib/jsr305-1.3.9.jar:/home/zrj/hadoop-2.5.2/share/hadoop/common/lib/hadoop-auth-2.5.2.jar:/home/zrj/hadoop-2.5.2/share/hadoop/common/lib/java-xmlbuilder-0.4.jar:/home/zrj/hadoop-2.5.2/share/hadoop/common/lib/commons-el-1.0.jar:/home/zrj/hadoop-2.5.2/share/hadoop/common/lib/jettison-1.1.jar:/home/zrj/hadoop-2.5.2/share/hadoop/common/lib/jersey-server-1.9.jar:/home/zrj/hadoop-2.5.2/share/hadoop/common/lib/jackson-mapper-asl-1.9.13.jar:/home/zrj/hadoop-2.5.2/share/hadoop/common/lib/avro-1.7.4.jar:/home/zrj/hadoop-2.5.2/share/hadoop/common/lib/commons-codec-1.4.jar:/home/zrj/hadoop-2.5.2/share/hadoop/common/lib/commons-cli-1.2.jar:/home/zrj/hadoop-2.5.2/share/hadoop/common/lib/commons-net-3.1.jar:/home/zrj/hadoop-2.5.2/share/hadoop/common/lib/jetty-util-6.1.26.jar:/home/zrj/hadoop-2.5.2/share/hadoop/common/lib/protobuf-java-2.5.0.jar:/home/zrj/hadoop-2.5.2/share/hadoop/common/lib/netty-3.6.2.Final.jar:/home/zrj/hadoop-2.5.2/share/hadoop/common/lib/commons-digester-1.8.jar:/home/zrj/hadoop-2.5.2/share/hadoop/common/lib/hadoop-annotations-2.5.2.jar:/home/zrj/hadoop-2.5.2/share/hadoop/common/lib/guava-11.0.2.jar:/home/zrj/hadoop-2.5.2/share/hadoop/common/lib/commons-compress-1.4.1.jar:/home/zrj/hadoop-2.5.2/share/hadoop/common/lib/jsch-0.1.42.jar:/home/zrj/hadoop-2.5.2/share/hadoop/common/lib/commons-beanutils-1.7.0.jar:/home/zrj/hadoop-2.5.2/share/hadoop/common/lib/jersey-core-1.9.jar:/home/zrj/hadoop-2.5.2/share/hadoop/common/lib/api-util-1.0.0-M20.jar:/home/zrj/hadoop-2.5.2/share/hadoop/common/lib/api-asn1-api-1.0.0-M20.jar:/home/zrj/hadoop-2.5.2/share/hadoop/common/lib/xz-1.0.jar:/home/zrj/hadoop-2.5.2/share/hadoop/common/lib/commons-httpclient-3.1.jar:/home/zrj/hadoop-2.5.2/share/hadoop/common/lib/commons-beanutils-core-1.8.0.jar:/home/zrj/hadoop-2.5.2/share/hadoop/common/lib/stax-api-1.0-2.jar:/home/zrj/hadoop-2.5.2/share/hadoop/common/lib/asm-3.2.jar:/home/zrj/hadoop-2.5.2/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar:/home/zrj/hadoop-2.5.2/share/hadoop/common/lib/jackson-xc-1.9.13.jar:/home/zrj/hadoop-2.5.2/share/hadoop/common/lib/commons-logging-1.1.3.jar:/home/zrj/hadoop-2.5.2/share/hadoop/common/lib/jersey-json-1.9.jar:/home/zrj/hadoop-2.5.2/share/hadoop/common/lib/slf4j-api-1.7.5.jar:/home/zrj/hadoop-2.5.2/share/hadoop/common/lib/commons-collections-3.2.1.jar:/home/zrj/hadoop-2.5.2/share/hadoop/common/lib/commons-math3-3.1.1.jar:/home/zrj/hadoop-2.5.2/share/hadoop/common/lib/jetty-6.1.26.jar:/home/zrj/hadoop-2.5.2/share/hadoop/common/lib/snappy-java-1.0.4.1.jar:/home/zrj/hadoop-2.5.2/share/hadoop/common/lib/hamcrest-core-1.3.jar:/home/zrj/hadoop-2.5.2/share/hadoop/common/lib/jaxb-api-2.2.2.jar:/home/zrj/hadoop-2.5.2/share/hadoop/common/lib/commons-lang-2.6.jar:/home/zrj/hadoop-2.5.2/share/hadoop/common/lib/junit-4.11.jar:/home/zrj/hadoop-2.5.2/share/hadoop/common/lib/jackson-core-asl-1.9.13.jar:/home/zrj/hadoop-2.5.2/share/hadoop/common/lib/mockito-all-1.8.5.jar:/home/zrj/hadoop-2.5.2/share/hadoop/common/lib/servlet-api-2.5.jar:/home/zrj/hadoop-2.5.2/share/hadoop/common/lib/httpcore-4.2.5.jar:/home/zrj/hadoop-2.5.2/share/hadoop/common/lib/apacheds-i18n-2.0.0-M15.jar:/home/zrj/hadoop-2.5.2/share/hadoop/common/lib/xmlenc-0.52.jar:/home/zrj/hadoop-2.5.2/share/hadoop/common/lib/jasper-runtime-5.5.23.jar:/home/zrj/hadoop-2.5.2/share/hadoop/common/hadoop-common-2.5.2-tests.jar:/home/zrj/hadoop-2.5.2/share/hadoop/common/hadoop-nfs-2.5.2.jar:/home/zrj/hadoop-2.5.2/share/hadoop/common/hadoop-common-2.5.2.jar:/home/zrj/hadoop-2.5.2/share/hadoop/hdfs:/home/zrj/hadoop-2.5.2/share/hadoop/hdfs/lib/jsp-api-2.1.jar:/home/zrj/hadoop-2.5.2/share/hadoop/hdfs/lib/commons-io-2.4.jar:/home/zrj/hadoop-2.5.2/share/hadoop/hdfs/lib/log4j-1.2.17.jar:/home/zrj/hadoop-2.5.2/share/hadoop/hdfs/lib/commons-daemon-1.0.13.jar:/home/zrj/hadoop-2.5.2/share/hadoop/hdfs/lib/jsr305-1.3.9.jar:/home/zrj/hadoop-2.5.2/share/hadoop/hdfs/lib/commons-el-1.0.jar:/home/zrj/hadoop-2.5.2/share/hadoop/hdfs/lib/jersey-server-1.9.jar:/home/zrj/hadoop-2.5.2/share/hadoop/hdfs/lib/jackson-mapper-asl-1.9.13.jar:/home/zrj/hadoop-2.5.2/share/hadoop/hdfs/lib/commons-codec-1.4.jar:/home/zrj/hadoop-2.5.2/share/hadoop/hdfs/lib/commons-cli-1.2.jar:/home/zrj/hadoop-2.5.2/share/hadoop/hdfs/lib/jetty-util-6.1.26.jar:/home/zrj/hadoop-2.5.2/share/hadoop/hdfs/lib/protobuf-java-2.5.0.jar:/home/zrj/hadoop-2.5.2/share/hadoop/hdfs/lib/netty-3.6.2.Final.jar:/home/zrj/hadoop-2.5.2/share/hadoop/hdfs/lib/guava-11.0.2.jar:/home/zrj/hadoop-2.5.2/share/hadoop/hdfs/lib/jersey-core-1.9.jar:/home/zrj/hadoop-2.5.2/share/hadoop/hdfs/lib/asm-3.2.jar:/home/zrj/hadoop-2.5.2/share/hadoop/hdfs/lib/commons-logging-1.1.3.jar:/home/zrj/hadoop-2.5.2/share/hadoop/hdfs/lib/jetty-6.1.26.jar:/home/zrj/hadoop-2.5.2/share/hadoop/hdfs/lib/commons-lang-2.6.jar:/home/zrj/hadoop-2.5.2/share/hadoop/hdfs/lib/jackson-core-asl-1.9.13.jar:/home/zrj/hadoop-2.5.2/share/hadoop/hdfs/lib/servlet-api-2.5.jar:/home/zrj/hadoop-2.5.2/share/hadoop/hdfs/lib/xmlenc-0.52.jar:/home/zrj/hadoop-2.5.2/share/hadoop/hdfs/lib/jasper-runtime-5.5.23.jar:/home/zrj/hadoop-2.5.2/share/hadoop/hdfs/hadoop-hdfs-2.5.2-tests.jar:/home/zrj/hadoop-2.5.2/share/hadoop/hdfs/hadoop-hdfs-2.5.2.jar:/home/zrj/hadoop-2.5.2/share/hadoop/hdfs/hadoop-hdfs-nfs-2.5.2.jar:/home/zrj/hadoop-2.5.2/share/hadoop/yarn/lib/jline-0.9.94.jar:/home/zrj/hadoop-2.5.2/share/hadoop/yarn/lib/activation-1.1.jar:/home/zrj/hadoop-2.5.2/share/hadoop/yarn/lib/jackson-jaxrs-1.9.13.jar:/home/zrj/hadoop-2.5.2/share/hadoop/yarn/lib/jaxb-impl-2.2.3-1.jar:/home/zrj/hadoop-2.5.2/share/hadoop/yarn/lib/commons-io-2.4.jar:/home/zrj/hadoop-2.5.2/share/hadoop/yarn/lib/log4j-1.2.17.jar:/home/zrj/hadoop-2.5.2/share/hadoop/yarn/lib/zookeeper-3.4.6.jar:/home/zrj/hadoop-2.5.2/share/hadoop/yarn/lib/jsr305-1.3.9.jar:/home/zrj/hadoop-2.5.2/share/hadoop/yarn/lib/jettison-1.1.jar:/home/zrj/hadoop-2.5.2/share/hadoop/yarn/lib/jersey-server-1.9.jar:/home/zrj/hadoop-2.5.2/share/hadoop/yarn/lib/jackson-mapper-asl-1.9.13.jar:/home/zrj/hadoop-2.5.2/share/hadoop/yarn/lib/commons-codec-1.4.jar:/home/zrj/hadoop-2.5.2/share/hadoop/yarn/lib/commons-cli-1.2.jar:/home/zrj/hadoop-2.5.2/share/hadoop/yarn/lib/jetty-util-6.1.26.jar:/home/zrj/hadoop-2.5.2/share/hadoop/yarn/lib/protobuf-java-2.5.0.jar:/home/zrj/hadoop-2.5.2/share/hadoop/yarn/lib/jersey-client-1.9.jar:/home/zrj/hadoop-2.5.2/share/hadoop/yarn/lib/aopalliance-1.0.jar:/home/zrj/hadoop-2.5.2/share/hadoop/yarn/lib/netty-3.6.2.Final.jar:/home/zrj/hadoop-2.5.2/share/hadoop/yarn/lib/guava-11.0.2.jar:/home/zrj/hadoop-2.5.2/share/hadoop/yarn/lib/commons-compress-1.4.1.jar:/home/zrj/hadoop-2.5.2/share/hadoop/yarn/lib/jersey-core-1.9.jar:/home/zrj/hadoop-2.5.2/share/hadoop/yarn/lib/jersey-guice-1.9.jar:/home/zrj/hadoop-2.5.2/share/hadoop/yarn/lib/guice-3.0.jar:/home/zrj/hadoop-2.5.2/share/hadoop/yarn/lib/xz-1.0.jar:/home/zrj/hadoop-2.5.2/share/hadoop/yarn/lib/commons-httpclient-3.1.jar:/home/zrj/hadoop-2.5.2/share/hadoop/yarn/lib/stax-api-1.0-2.jar:/home/zrj/hadoop-2.5.2/share/hadoop/yarn/lib/asm-3.2.jar:/home/zrj/hadoop-2.5.2/share/hadoop/yarn/lib/jackson-xc-1.9.13.jar:/home/zrj/hadoop-2.5.2/share/hadoop/yarn/lib/commons-logging-1.1.3.jar:/home/zrj/hadoop-2.5.2/share/hadoop/yarn/lib/jersey-json-1.9.jar:/home/zrj/hadoop-2.5.2/share/hadoop/yarn/lib/guice-servlet-3.0.jar:/home/zrj/hadoop-2.5.2/share/hadoop/yarn/lib/commons-collections-3.2.1.jar:/home/zrj/hadoop-2.5.2/share/hadoop/yarn/lib/jetty-6.1.26.jar:/home/zrj/hadoop-2.5.2/share/hadoop/yarn/lib/jaxb-api-2.2.2.jar:/home/zrj/hadoop-2.5.2/share/hadoop/yarn/lib/commons-lang-2.6.jar:/home/zrj/hadoop-2.5.2/share/hadoop/yarn/lib/leveldbjni-all-1.8.jar:/home/zrj/hadoop-2.5.2/share/hadoop/yarn/lib/jackson-core-asl-1.9.13.jar:/home/zrj/hadoop-2.5.2/share/hadoop/yarn/lib/servlet-api-2.5.jar:/home/zrj/hadoop-2.5.2/share/hadoop/yarn/lib/javax.inject-1.jar:/home/zrj/hadoop-2.5.2/share/hadoop/yarn/hadoop-yarn-server-tests-2.5.2.jar:/home/zrj/hadoop-2.5.2/share/hadoop/yarn/hadoop-yarn-applications-distributedshell-2.5.2.jar:/home/zrj/hadoop-2.5.2/share/hadoop/yarn/hadoop-yarn-api-2.5.2.jar:/home/zrj/hadoop-2.5.2/share/hadoop/yarn/hadoop-yarn-server-common-2.5.2.jar:/home/zrj/hadoop-2.5.2/share/hadoop/yarn/hadoop-yarn-server-web-proxy-2.5.2.jar:/home/zrj/hadoop-2.5.2/share/hadoop/yarn/hadoop-yarn-server-nodemanager-2.5.2.jar:/home/zrj/hadoop-2.5.2/share/hadoop/yarn/hadoop-yarn-applications-unmanaged-am-launcher-2.5.2.jar:/home/zrj/hadoop-2.5.2/share/hadoop/yarn/hadoop-yarn-server-resourcemanager-2.5.2.jar:/home/zrj/hadoop-2.5.2/share/hadoop/yarn/hadoop-yarn-client-2.5.2.jar:/home/zrj/hadoop-2.5.2/share/hadoop/yarn/hadoop-yarn-server-applicationhistoryservice-2.5.2.jar:/home/zrj/hadoop-2.5.2/share/hadoop/yarn/hadoop-yarn-common-2.5.2.jar:/home/zrj/hadoop-2.5.2/share/hadoop/mapreduce/lib/commons-io-2.4.jar:/home/zrj/hadoop-2.5.2/share/hadoop/mapreduce/lib/paranamer-2.3.jar:/home/zrj/hadoop-2.5.2/share/hadoop/mapreduce/lib/log4j-1.2.17.jar:/home/zrj/hadoop-2.5.2/share/hadoop/mapreduce/lib/jersey-server-1.9.jar:/home/zrj/hadoop-2.5.2/share/hadoop/mapreduce/lib/jackson-mapper-asl-1.9.13.jar:/home/zrj/hadoop-2.5.2/share/hadoop/mapreduce/lib/avro-1.7.4.jar:/home/zrj/hadoop-2.5.2/share/hadoop/mapreduce/lib/protobuf-java-2.5.0.jar:/home/zrj/hadoop-2.5.2/share/hadoop/mapreduce/lib/aopalliance-1.0.jar:/home/zrj/hadoop-2.5.2/share/hadoop/mapreduce/lib/netty-3.6.2.Final.jar:/home/zrj/hadoop-2.5.2/share/hadoop/mapreduce/lib/hadoop-annotations-2.5.2.jar:/home/zrj/hadoop-2.5.2/share/hadoop/mapreduce/lib/commons-compress-1.4.1.jar:/home/zrj/hadoop-2.5.2/share/hadoop/mapreduce/lib/jersey-core-1.9.jar:/home/zrj/hadoop-2.5.2/share/hadoop/mapreduce/lib/jersey-guice-1.9.jar:/home/zrj/hadoop-2.5.2/share/hadoop/mapreduce/lib/guice-3.0.jar:/home/zrj/hadoop-2.5.2/share/hadoop/mapreduce/lib/xz-1.0.jar:/home/zrj/hadoop-2.5.2/share/hadoop/mapreduce/lib/asm-3.2.jar:/home/zrj/hadoop-2.5.2/share/hadoop/mapreduce/lib/guice-servlet-3.0.jar:/home/zrj/hadoop-2.5.2/share/hadoop/mapreduce/lib/snappy-java-1.0.4.1.jar:/home/zrj/hadoop-2.5.2/share/hadoop/mapreduce/lib/hamcrest-core-1.3.jar:/home/zrj/hadoop-2.5.2/share/hadoop/mapreduce/lib/junit-4.11.jar:/home/zrj/hadoop-2.5.2/share/hadoop/mapreduce/lib/leveldbjni-all-1.8.jar:/home/zrj/hadoop-2.5.2/share/hadoop/mapreduce/lib/jackson-core-asl-1.9.13.jar:/home/zrj/hadoop-2.5.2/share/hadoop/mapreduce/lib/javax.inject-1.jar:/home/zrj/hadoop-2.5.2/share/hadoop/mapreduce/hadoop-mapreduce-client-shuffle-2.5.2.jar:/home/zrj/hadoop-2.5.2/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.5.2.jar:/home/zrj/hadoop-2.5.2/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.5.2-tests.jar:/home/zrj/hadoop-2.5.2/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.5.2.jar:/home/zrj/hadoop-2.5.2/share/hadoop/mapreduce/hadoop-mapreduce-client-hs-2.5.2.jar:/home/zrj/hadoop-2.5.2/share/hadoop/mapreduce/hadoop-mapreduce-client-app-2.5.2.jar:/home/zrj/hadoop-2.5.2/share/hadoop/mapreduce/hadoop-mapreduce-client-hs-plugins-2.5.2.jar:/home/zrj/hadoop-2.5.2/share/hadoop/mapreduce/hadoop-mapreduce-client-common-2.5.2.jar:/home/zrj/hadoop-2.5.2/share/hadoop/mapreduce/hadoop-mapreduce-client-core-2.5.2.jar:/home/zrj/hadoop-2.5.2/contrib/capacity-scheduler/*.jar
STARTUP_MSG:   build = https://git-wip-us.apache.org/repos/asf/hadoop.git -r cc72e9b000545b86b75a61f4835eb86d57bfafc0; compiled by 'jenkins' on 2014-11-14T23:45Z
STARTUP_MSG:   java = 1.7.0_72
************************************************************/
16/03/18 16:22:31 INFO namenode.NameNode: registered UNIX signal handlers for [TERM, HUP, INT]
16/03/18 16:22:31 INFO namenode.NameNode: createNameNode [-format]
Formatting using clusterid: CID-98edc835-228b-479b-b912-be8db8cc32ad
16/03/18 16:22:31 INFO namenode.FSNamesystem: fsLock is fair:true
16/03/18 16:22:31 INFO blockmanagement.DatanodeManager: dfs.block.invalidate.limit=1000
16/03/18 16:22:31 INFO blockmanagement.DatanodeManager: dfs.namenode.datanode.registration.ip-hostname-check=true
16/03/18 16:22:31 INFO blockmanagement.BlockManager: dfs.namenode.startup.delay.block.deletion.sec is set to 000:00:00:00.000
16/03/18 16:22:31 INFO blockmanagement.BlockManager: The block deletion will start around 2016 Mar 18 16:22:31
16/03/18 16:22:31 INFO util.GSet: Computing capacity for map BlocksMap
16/03/18 16:22:31 INFO util.GSet: VM type       = 64-bit
16/03/18 16:22:31 INFO util.GSet: 2.0% max memory 889 MB = 17.8 MB
16/03/18 16:22:31 INFO util.GSet: capacity      = 2^21 = 2097152 entries
16/03/18 16:22:31 INFO blockmanagement.BlockManager: dfs.block.access.token.enable=false
16/03/18 16:22:31 INFO blockmanagement.BlockManager: defaultReplication         = 2
16/03/18 16:22:31 INFO blockmanagement.BlockManager: maxReplication             = 512
16/03/18 16:22:31 INFO blockmanagement.BlockManager: minReplication             = 1
16/03/18 16:22:31 INFO blockmanagement.BlockManager: maxReplicationStreams      = 2
16/03/18 16:22:31 INFO blockmanagement.BlockManager: shouldCheckForEnoughRacks  = false
16/03/18 16:22:31 INFO blockmanagement.BlockManager: replicationRecheckInterval = 3000
16/03/18 16:22:31 INFO blockmanagement.BlockManager: encryptDataTransfer        = false
16/03/18 16:22:31 INFO blockmanagement.BlockManager: maxNumBlocksToLog          = 1000
16/03/18 16:22:31 INFO namenode.FSNamesystem: fsOwner             = zrj (auth:SIMPLE)
16/03/18 16:22:31 INFO namenode.FSNamesystem: supergroup          = supergroup
16/03/18 16:22:31 INFO namenode.FSNamesystem: isPermissionEnabled = true
16/03/18 16:22:31 INFO namenode.FSNamesystem: HA Enabled: false
16/03/18 16:22:31 INFO namenode.FSNamesystem: Append Enabled: true
16/03/18 16:22:31 INFO util.GSet: Computing capacity for map INodeMap
16/03/18 16:22:31 INFO util.GSet: VM type       = 64-bit
16/03/18 16:22:31 INFO util.GSet: 1.0% max memory 889 MB = 8.9 MB
16/03/18 16:22:31 INFO util.GSet: capacity      = 2^20 = 1048576 entries
16/03/18 16:22:31 INFO namenode.NameNode: Caching file names occuring more than 10 times
16/03/18 16:22:31 INFO util.GSet: Computing capacity for map cachedBlocks
16/03/18 16:22:31 INFO util.GSet: VM type       = 64-bit
16/03/18 16:22:31 INFO util.GSet: 0.25% max memory 889 MB = 2.2 MB
16/03/18 16:22:31 INFO util.GSet: capacity      = 2^18 = 262144 entries
16/03/18 16:22:31 INFO namenode.FSNamesystem: dfs.namenode.safemode.threshold-pct = 0.9990000128746033
16/03/18 16:22:31 INFO namenode.FSNamesystem: dfs.namenode.safemode.min.datanodes = 0
16/03/18 16:22:31 INFO namenode.FSNamesystem: dfs.namenode.safemode.extension     = 30000
16/03/18 16:22:31 INFO namenode.FSNamesystem: Retry cache on namenode is enabled
16/03/18 16:22:31 INFO namenode.FSNamesystem: Retry cache will use 0.03 of total heap and retry cache entry expiry time is 600000 millis
16/03/18 16:22:31 INFO util.GSet: Computing capacity for map NameNodeRetryCache
16/03/18 16:22:31 INFO util.GSet: VM type       = 64-bit
16/03/18 16:22:31 INFO util.GSet: 0.029999999329447746% max memory 889 MB = 273.1 KB
16/03/18 16:22:31 INFO util.GSet: capacity      = 2^15 = 32768 entries
16/03/18 16:22:31 INFO namenode.NNConf: ACLs enabled? false
16/03/18 16:22:31 INFO namenode.NNConf: XAttrs enabled? true
16/03/18 16:22:31 INFO namenode.NNConf: Maximum size of an xattr: 16384
16/03/18 16:22:31 INFO namenode.FSImage: Allocated new BlockPoolId: BP-667685565-10.16.18.206-1458289351864
16/03/18 16:22:31 INFO common.Storage: Storage directory /home/zrj/hadoop-2.5.2/dfs/name has been successfully formatted.
16/03/18 16:22:32 INFO namenode.NNStorageRetentionManager: Going to retain 1 images with txid >= 0
16/03/18 16:22:32 INFO util.ExitUtil: Exiting with status 0
16/03/18 16:22:32 INFO namenode.NameNode: SHUTDOWN_MSG: 
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at FWSVRLPT01/10.16.18.206
************************************************************/

记得要分别在文件hadoop-env.sh和yarn-env.sh中添加JAVA_HOME配置

启动 dfs

zrj@FWSVRLPT01:~/hadoop-2.5.2$ ./sbin/start-dfs.sh 
Starting namenodes on [FWSVRLPT01]
FWSVRLPT01: starting namenode, logging to /home/zrj/hadoop-2.5.2/logs/hadoop-zrj-namenode-FWSVRLPT01.out
FWSVRLPT02: starting datanode, logging to /home/zrj/hadoop-2.5.2/logs/hadoop-zrj-datanode-FWSVRLPT02.out
qtdata: starting datanode, logging to /home/zrj/hadoop-2.5.2/logs/hadoop-zrj-datanode-qtdata.out
Starting secondary namenodes [FWSVRLPT01]
FWSVRLPT01: starting secondarynamenode, logging to /home/zrj/hadoop-2.5.2/logs/hadoop-zrj-secondarynamenode-FWSVRLPT01.out

启动 yarn

zrj@FWSVRLPT01:~/hadoop-2.5.2$ ./sbin/start-yarn.sh 
starting yarn daemons
starting resourcemanager, logging to /home/zrj/hadoop-2.5.2/logs/yarn-zrj-resourcemanager-FWSVRLPT01.out
FWSVRLPT02: starting nodemanager, logging to /home/zrj/hadoop-2.5.2/logs/yarn-zrj-nodemanager-FWSVRLPT02.out
qtdata: starting nodemanager, logging to /home/zrj/hadoop-2.5.2/logs/yarn-zrj-nodemanager-qtdata.out

通过浏览器访问 http://10.0.1.100:50070/ http://10.0.1.100:8088/

WordCount - Hadoop 的 HelloWorld

Thu, 20 Jun 2013 18:14:54 +0800

按照官方教程，http://hadoop.apache.org/docs/stable/mapred_tutorial.html，开始着手写一个 Hadoop 的 HelloWorld 程序，叫做 WordCount

首先准备好代码如下：

package org.myorg;
	
import java.io.IOException;
import java.util.*;

import org.apache.hadoop.fs.Path;
import org.apache.hadoop.conf.*;
import org.apache.hadoop.io.*;
import org.apache.hadoop.mapred.*;
import org.apache.hadoop.util.*;

public class WordCount {

    public static class Map extends MapReduceBase implements Mapper<LongWritable, Text, Text, IntWritable> {
      private final static IntWritable one = new IntWritable(1);
      private Text word = new Text();

      public void map(LongWritable key, Text value, OutputCollector<Text, IntWritable> output, Reporter reporter) throws IOException {
        String line = value.toString();
        StringTokenizer tokenizer = new StringTokenizer(line);
        while (tokenizer.hasMoreTokens()) {
          word.set(tokenizer.nextToken());
          output.collect(word, one);
        }
      }
    }

    public static class Reduce extends MapReduceBase implements Reducer<Text, IntWritable, Text, IntWritable> {
      public void reduce(Text key, Iterator<IntWritable> values, OutputCollector<Text, IntWritable> output, Reporter reporter) throws IOException {
        int sum = 0;
        while (values.hasNext()) {
          sum += values.next().get();
        }
        output.collect(key, new IntWritable(sum));
      }
    }

    public static void main(String[] args) throws Exception {
      JobConf conf = new JobConf(WordCount.class);
      conf.setJobName("wordcount");

      conf.setOutputKeyClass(Text.class);
      conf.setOutputValueClass(IntWritable.class);

      conf.setMapperClass(Map.class);
      conf.setCombinerClass(Reduce.class);
      conf.setReducerClass(Reduce.class);

      conf.setInputFormat(TextInputFormat.class);
      conf.setOutputFormat(TextOutputFormat.class);

      FileInputFormat.setInputPaths(conf, new Path(args[0]));
      FileOutputFormat.setOutputPath(conf, new Path(args[1]));

      JobClient.runJob(conf);
    }
}

然后进行编译打包

$ mkdir wordcount_classes 
$ javac -classpath ${HADOOP_HOME}/hadoop-${HADOOP_VERSION}-core.jar -d wordcount_classes WordCount.java 
$ jar -cvf /usr/joe/wordcount.jar -C wordcount_classes/ .

编译完了可以看到有这三个文件

root@cluster-1:~/workspace/hadoop-1.1.2/wordcount_classes/org/myorg# ll
total 20
drwxr-xr-x 2 root root 4096 2013-06-20 18:00 ./
drwxr-xr-x 3 root root 4096 2013-06-20 18:00 ../
-rw-r--r-- 1 root root 1546 2013-06-20 18:00 WordCount.class
-rw-r--r-- 1 root root 1938 2013-06-20 18:00 WordCount$Map.class
-rw-r--r-- 1 root root 1611 2013-06-20 18:00 WordCount$Reduce.class

打包完得到 wordcount.jar

如果之前有按照官方例程做那个 example 的话，需要先清空文件

root@cluster-1:~/workspace/hadoop-1.1.2# bin/hadoop fs -rmr input
Deleted hdfs://cluster-1:9000/user/root/input
root@cluster-1:~/workspace/hadoop-1.1.2# bin/hadoop fs -rmr output
Deleted hdfs://cluster-1:9000/user/root/output

然后根据教程准备好输入的文件

Sample text-files as input:

$ bin/hadoop dfs -ls /usr/joe/wordcount/input/ /usr/joe/wordcount/input/file01 /usr/joe/wordcount/input/file02

$ bin/hadoop dfs -cat /usr/joe/wordcount/input/file01 Hello World Bye World

$ bin/hadoop dfs -cat /usr/joe/wordcount/input/file02 Hello Hadoop Goodbye Hadoop

然后开始运行

$ bin/hadoop jar wordcount.jar org.myorg.WordCount /user/root/input /user/root/output

得到结果

bin/hadoop fs -cat output/*
Bye	1
Goodbye	1
Hadoop	2
Hello	2
World	2

Hadoop 1.1.2 安装（集群版）

Wed, 19 Jun 2013 11:10:29 +0800

首先还是安装必须的 jdk，看到这里，http://os.51cto.com/art/201003/189114.htm

终端下进入你存放jdk-6u12-linux-i586.bin，例如我的位置是:/home/liangshihong

$ sudo -s ./jdk-6u12-linux-i586.bin

一路回车，直到询问是否安装，输入yes回车

ok，安装完毕，下面配置环境变量

配置classpath，修改所有用户的环境变量

$ sudo gedit /etc/profile

在文件最后添加

#set java environment

JAVA_HOME=/home/liangshihong/jdk1.6.0_12

export JRE_HOME=/home/liangshihong/jdk1.6.0_12/jre

export CLASSPATH=.:$JAVA_HOME/lib:$JRE_HOME/lib:$CLASSPATH

export PATH=$JAVA_HOME/bin:$JRE_HOME/bin:$PATH

重新启动计算机，用命令测试jdk的版本

java -version

显示如下信息：成功安装

java version “1.6.0_12”

Java(TM) SE Runtime Environment (build 1.6.0_12-b04)

Java HotSpot(TM) Server VM (build 11.2-b01, mixed mode)

liangshihong@liangshihong-Imagine:~$

另外需要修改 /etc/hosts

10.241.32.32 cluster-1
10.241.158.17 cluster-2
10.241.158.171 cluster-3

然后需要互相打通集群内的访问，由于我们的目标是，master 需要 ssh 要下面各个 node，而 node 之间不需要互相连接，对于这个需求，可以这么来实现

在 master 上面使用

$ ssh-keygen -t dsa -P '' -f ~/.ssh/id_dsa
$ cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys

来生成自己的公钥，然后把这个 auth 文件复制到下面的每个 node 就可以了

scp /root/.ssh/authorized_keys cluster-2:~/.ssh/
scp /root/.ssh/authorized_keys cluster-3:~/.ssh/

插入一下 tips，在修改完毕 profile 文件后，执行source /etc/profile 来使其生效。

测试一下，应该就可以实现各个机器之间的无密码访问了，然后开始配置 Hadoop，主要看到这个教程，http://blog.csdn.net/hguisu/article/details/7237395

3. 集群配置（所有节点相同） 3.1配置文件：conf/core-site.xml

fs.default.name hdfs://node1:49000 hadoop.tmp.dir /home/hadoop/hadoop_home/var

1）fs.default.name是NameNode的URI。hdfs://主机名:端口/ 2）hadoop.tmp.dir ：Hadoop的默认临时路径，这个最好配置，如果在新增节点或者其他情况下莫名其妙的DataNode启动不了，就删除此文件中的tmp目录即可。不过如果删除了NameNode机器的此目录，那么就需要重新执行NameNode格式化的命令。 3.2配置文件：conf/mapred-site.xml

mapred.job.tracker node1:49001 mapred.local.dir /home/hadoop/hadoop_home/var

1）mapred.job.tracker是JobTracker的主机（或者IP）和端口。主机:端口。

3.3配置文件：conf/hdfs-site.xml

dfs.name.dir /home/hadoop/name1, /home/hadoop/name2 #hadoop的name目录路径

dfs.data.dir /home/hadoop/data1, /home/hadoop/data2

dfs.replication 2

1） dfs.name.dir是NameNode持久存储名字空间及事务日志的本地文件系统路径。当这个值是一个逗号分割的目录列表时，nametable数据将会被复制到所有目录中做冗余备份。 2） dfs.data.dir是DataNode存放块数据的本地文件系统路径，逗号分割的列表。当这个值是逗号分割的目录列表时，数据将被存储在所有目录下，通常分布在不同设备上。 3）dfs.replication是数据需要备份的数量，默认是3，如果此数大于集群的机器数会出错。注意：此处的name1、name2、data1、data2目录不能预先创建，hadoop格式化时会自动创建，如果预先创建反而会有问题。

3.4配置masters和slaves主从结点配置conf/masters和conf/slaves来设置主从结点，注意最好使用主机名，并且保证机器之间通过主机名可以互相访问，每个主机名一行。

vi masters：输入：

node1

vi slaves：

输入： node2 node3

配置结束，把配置好的hadoop文件夹拷贝到其他集群的机器中，并且保证上面的配置对于其他机器而言正确，例如：如果其他机器的Java安装路径不一样，要修改conf/hadoop-env.sh

$ scp -r /home/hadoop/hadoop-0.20.203 root@node2: /home/hadoop/

对于我，我的 conf/core-site.xml 如下

<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<!-- Put site-specific property overrides in this file. -->

<configuration>
     <property>
         <name>fs.default.name</name>
         <value>hdfs://cluster-1:9000</value>
     </property>
</configuration>

conf/mapred-site.xml 如下

<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<!-- Put site-specific property overrides in this file. -->

<configuration>
     <property>
         <name>mapred.job.tracker</name>
         <value>cluster-1:9001</value>
     </property>
</configuration>

conf/hdfs-site.xml

<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<!-- Put site-specific property overrides in this file. -->

<configuration>
     <property>
         <name>dfs.replication</name>
         <value>2</value>
     </property>
</configuration>

接下来配置主从节点，修改 conf/masters

cluster-1

和 conf/slaves

cluster-2
cluster-3

然后把配置文件拷贝到其他机器

scp conf/core-site.xml conf/mapred-site.xml conf/hdfs-site.xml conf/masters conf/slaves cluster-2:~/workspace/hadoop-1.1.2/conf
scp conf/core-site.xml conf/mapred-site.xml conf/hdfs-site.xml conf/masters conf/slaves cluster-3:~/workspace/hadoop-1.1.2/conf

格式化 namenode

root@cluster-1:~/workspace/hadoop-1.1.2# bin/hadoop namenode -format
13/06/19 10:48:55 INFO namenode.NameNode: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting NameNode
STARTUP_MSG:   host = cluster-1/10.241.32.32
STARTUP_MSG:   args = [-format]
STARTUP_MSG:   version = 1.1.2
STARTUP_MSG:   build = https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.1 -r 1440782; compiled by 'hortonfo' on Thu Jan 31 02:03:24 UTC 2013
************************************************************/
13/06/19 10:48:56 INFO util.GSet: VM type       = 64-bit
13/06/19 10:48:56 INFO util.GSet: 2% max memory = 19.33375 MB
13/06/19 10:48:56 INFO util.GSet: capacity      = 2^21 = 2097152 entries
13/06/19 10:48:56 INFO util.GSet: recommended=2097152, actual=2097152
13/06/19 10:48:56 INFO namenode.FSNamesystem: fsOwner=root
13/06/19 10:48:56 INFO namenode.FSNamesystem: supergroup=supergroup
13/06/19 10:48:56 INFO namenode.FSNamesystem: isPermissionEnabled=true
13/06/19 10:48:56 INFO namenode.FSNamesystem: dfs.block.invalidate.limit=100
13/06/19 10:48:56 INFO namenode.FSNamesystem: isAccessTokenEnabled=false accessKeyUpdateInterval=0 min(s), accessTokenLifetime=0 min(s)
13/06/19 10:48:56 INFO namenode.NameNode: Caching file names occuring more than 10 times
13/06/19 10:48:57 INFO common.Storage: Image file of size 110 saved in 0 seconds.
13/06/19 10:48:57 INFO namenode.FSEditLog: closing edit log: position=4, editlog=/tmp/hadoop-root/dfs/name/current/edits
13/06/19 10:48:57 INFO namenode.FSEditLog: close success: truncate to 4, editlog=/tmp/hadoop-root/dfs/name/current/edits
13/06/19 10:48:57 INFO common.Storage: Storage directory /tmp/hadoop-root/dfs/name has been successfully formatted.
13/06/19 10:48:57 INFO namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at cluster-1/10.241.32.32
************************************************************/

如果格式化失败，可以尝试这个命令

root@cluster-1:~/workspace/hadoop-1.1.2# rm -rf /tmp/hadoop-root*

格式化之后，启动集群，在这里，由于我们的 namenode 和 jobtracker 在同一台机器，所以可以这样来启动

root@cluster-1:~/workspace/hadoop-1.1.2# bin/start-all.sh
starting namenode, logging to /root/workspace/hadoop-1.1.2/libexec/../logs/hadoop-root-namenode-cluster-1.out
cluster-2: starting datanode, logging to /root/workspace/hadoop-1.1.2/libexec/../logs/hadoop-root-datanode-cluster-2.out
cluster-3: starting datanode, logging to /root/workspace/hadoop-1.1.2/libexec/../logs/hadoop-root-datanode-cluster-3.out
cluster-1: starting secondarynamenode, logging to /root/workspace/hadoop-1.1.2/libexec/../logs/hadoop-root-secondarynamenode-cluster-1.out
starting jobtracker, logging to /root/workspace/hadoop-1.1.2/libexec/../logs/hadoop-root-jobtracker-cluster-1.out
cluster-2: starting tasktracker, logging to /root/workspace/hadoop-1.1.2/libexec/../logs/hadoop-root-tasktracker-cluster-2.out
cluster-3: starting tasktracker, logging to /root/workspace/hadoop-1.1.2/libexec/../logs/hadoop-root-tasktracker-cluster-3.out

由于在配置 conf/hdfs-site.xml 的时候，我们没有配置 dfs.data.dir，所以，datanode 中，数据的存放位置是 /tmp，我们可以在 cluster-2 和 clutster-3 中看到

root@cluster-2:/tmp# ll
total 32
drwxrwxrwt  6 root root 4096 2013-06-19 10:57 ./
drwxr-xr-x 21 root root 4096 2013-06-18 17:03 ../
drwxr-xr-x  4 root root 4096 2013-06-19 10:52 hadoop-root/
-rw-r--r--  1 root root    5 2013-06-19 10:57 hadoop-root-datanode.pid
-rw-r--r--  1 root root    5 2013-06-19 10:57 hadoop-root-tasktracker.pid

root@cluster-3:/tmp# ll
total 32
drwxrwxrwt  6 root root 4096 2013-06-19 10:57 ./
drwxr-xr-x 21 root root 4096 2013-06-18 17:18 ../
drwxr-xr-x  4 root root 4096 2013-06-19 10:52 hadoop-root/
-rw-r--r--  1 root root    5 2013-06-19 10:57 hadoop-root-datanode.pid
-rw-r--r--  1 root root    5 2013-06-19 10:57 hadoop-root-tasktracker.pid

同样可以通过 web 来看到集群的情况

然后运行官方教程的例程，http://hadoop.apache.org/docs/stable/single_node_setup.html

Copy the input files into the distributed filesystem: $ bin/hadoop fs -put conf input

Run some of the examples provided: $ bin/hadoop jar hadoop-examples-*.jar grep input output ‘dfs[a-z.]+’

Examine the output files:

Copy the output files from the distributed filesystem to the local filesytem and examine them: $ bin/hadoop fs -get output output $ cat output/*

or

View the output files on the distributed filesystem: $ bin/hadoop fs -cat output/*

首先拷贝文件

root@cluster-1:~/workspace/hadoop-1.1.2# bin/hadoop fs -put conf input
root@cluster-1:~/workspace/hadoop-1.1.2# bin/hadoop fs -ls input
Found 16 items
-rw-r--r--   2 root supergroup       7457 2013-06-19 11:14 /user/root/input/capacity-scheduler.xml
-rw-r--r--   2 root supergroup        535 2013-06-19 11:14 /user/root/input/configuration.xsl
-rw-r--r--   2 root supergroup        294 2013-06-19 11:14 /user/root/input/core-site.xml
-rw-r--r--   2 root supergroup        327 2013-06-19 11:14 /user/root/input/fair-scheduler.xml
-rw-r--r--   2 root supergroup       2240 2013-06-19 11:14 /user/root/input/hadoop-env.sh
-rw-r--r--   2 root supergroup       1488 2013-06-19 11:14 /user/root/input/hadoop-metrics2.properties
-rw-r--r--   2 root supergroup       4644 2013-06-19 11:14 /user/root/input/hadoop-policy.xml
-rw-r--r--   2 root supergroup        274 2013-06-19 11:14 /user/root/input/hdfs-site.xml
-rw-r--r--   2 root supergroup       4441 2013-06-19 11:14 /user/root/input/log4j.properties
-rw-r--r--   2 root supergroup       2033 2013-06-19 11:14 /user/root/input/mapred-queue-acls.xml
-rw-r--r--   2 root supergroup        290 2013-06-19 11:14 /user/root/input/mapred-site.xml
-rw-r--r--   2 root supergroup         10 2013-06-19 11:14 /user/root/input/masters
-rw-r--r--   2 root supergroup         20 2013-06-19 11:14 /user/root/input/slaves
-rw-r--r--   2 root supergroup       1243 2013-06-19 11:14 /user/root/input/ssl-client.xml.example
-rw-r--r--   2 root supergroup       1195 2013-06-19 11:14 /user/root/input/ssl-server.xml.example
-rw-r--r--   2 root supergroup        382 2013-06-19 11:14 /user/root/input/taskcontroller.cfg

然后运行例程

root@cluster-1:~/workspace/hadoop-1.1.2# bin/hadoop jar hadoop-examples-*.jar grep input output 'dfs[a-z.]+'
13/06/19 11:14:49 INFO util.NativeCodeLoader: Loaded the native-hadoop library
13/06/19 11:14:49 WARN snappy.LoadSnappy: Snappy native library not loaded
13/06/19 11:14:49 INFO mapred.FileInputFormat: Total input paths to process : 16
13/06/19 11:14:50 INFO mapred.JobClient: Running job: job_201306191057_0001
13/06/19 11:14:51 INFO mapred.JobClient:  map 0% reduce 0%
13/06/19 11:15:01 INFO mapred.JobClient:  map 12% reduce 0%
13/06/19 11:15:02 INFO mapred.JobClient:  map 25% reduce 0%
13/06/19 11:15:08 INFO mapred.JobClient:  map 37% reduce 0%
13/06/19 11:15:09 INFO mapred.JobClient:  map 50% reduce 0%
13/06/19 11:15:14 INFO mapred.JobClient:  map 62% reduce 0%
13/06/19 11:15:17 INFO mapred.JobClient:  map 75% reduce 0%
13/06/19 11:15:19 INFO mapred.JobClient:  map 87% reduce 0%
13/06/19 11:15:20 INFO mapred.JobClient:  map 87% reduce 20%
13/06/19 11:15:26 INFO mapred.JobClient:  map 100% reduce 29%
13/06/19 11:15:33 INFO mapred.JobClient:  map 100% reduce 100%
13/06/19 11:15:36 INFO mapred.JobClient: Job complete: job_201306191057_0001
13/06/19 11:15:36 INFO mapred.JobClient: Counters: 30
13/06/19 11:15:36 INFO mapred.JobClient:   Job Counters 
13/06/19 11:15:36 INFO mapred.JobClient:     Launched reduce tasks=1
13/06/19 11:15:36 INFO mapred.JobClient:     SLOTS_MILLIS_MAPS=110582
13/06/19 11:15:36 INFO mapred.JobClient:     Total time spent by all reduces waiting after reserving slots (ms)=0
13/06/19 11:15:36 INFO mapred.JobClient:     Total time spent by all maps waiting after reserving slots (ms)=0
13/06/19 11:15:36 INFO mapred.JobClient:     Launched map tasks=16
13/06/19 11:15:36 INFO mapred.JobClient:     Data-local map tasks=16
13/06/19 11:15:36 INFO mapred.JobClient:     SLOTS_MILLIS_REDUCES=31899
13/06/19 11:15:36 INFO mapred.JobClient:   File Input Format Counters 
13/06/19 11:15:36 INFO mapred.JobClient:     Bytes Read=26873
13/06/19 11:15:36 INFO mapred.JobClient:   File Output Format Counters 
13/06/19 11:15:36 INFO mapred.JobClient:     Bytes Written=180
13/06/19 11:15:36 INFO mapred.JobClient:   FileSystemCounters
13/06/19 11:15:36 INFO mapred.JobClient:     FILE_BYTES_READ=82
13/06/19 11:15:36 INFO mapred.JobClient:     HDFS_BYTES_READ=28595
13/06/19 11:15:36 INFO mapred.JobClient:     FILE_BYTES_WRITTEN=866947
13/06/19 11:15:36 INFO mapred.JobClient:     HDFS_BYTES_WRITTEN=180
13/06/19 11:15:36 INFO mapred.JobClient:   Map-Reduce Framework
13/06/19 11:15:36 INFO mapred.JobClient:     Map output materialized bytes=172
13/06/19 11:15:36 INFO mapred.JobClient:     Map input records=759
13/06/19 11:15:36 INFO mapred.JobClient:     Reduce shuffle bytes=172
13/06/19 11:15:36 INFO mapred.JobClient:     Spilled Records=6
13/06/19 11:15:36 INFO mapred.JobClient:     Map output bytes=70
13/06/19 11:15:36 INFO mapred.JobClient:     Total committed heap usage (bytes)=2703933440
13/06/19 11:15:36 INFO mapred.JobClient:     CPU time spent (ms)=5950
13/06/19 11:15:36 INFO mapred.JobClient:     Map input bytes=26873
13/06/19 11:15:36 INFO mapred.JobClient:     SPLIT_RAW_BYTES=1722
13/06/19 11:15:36 INFO mapred.JobClient:     Combine input records=3
13/06/19 11:15:36 INFO mapred.JobClient:     Reduce input records=3
13/06/19 11:15:36 INFO mapred.JobClient:     Reduce input groups=3
13/06/19 11:15:36 INFO mapred.JobClient:     Combine output records=3
13/06/19 11:15:36 INFO mapred.JobClient:     Physical memory (bytes) snapshot=2599989248
13/06/19 11:15:36 INFO mapred.JobClient:     Reduce output records=3
13/06/19 11:15:36 INFO mapred.JobClient:     Virtual memory (bytes) snapshot=7510061056
13/06/19 11:15:36 INFO mapred.JobClient:     Map output records=3
13/06/19 11:15:36 INFO mapred.FileInputFormat: Total input paths to process : 1
13/06/19 11:15:36 INFO mapred.JobClient: Running job: job_201306191057_0002
13/06/19 11:15:37 INFO mapred.JobClient:  map 0% reduce 0%
13/06/19 11:15:44 INFO mapred.JobClient:  map 100% reduce 0%
13/06/19 11:15:52 INFO mapred.JobClient:  map 100% reduce 33%
13/06/19 11:15:54 INFO mapred.JobClient:  map 100% reduce 100%
13/06/19 11:15:55 INFO mapred.JobClient: Job complete: job_201306191057_0002
13/06/19 11:15:55 INFO mapred.JobClient: Counters: 30
13/06/19 11:15:55 INFO mapred.JobClient:   Job Counters 
13/06/19 11:15:55 INFO mapred.JobClient:     Launched reduce tasks=1
13/06/19 11:15:55 INFO mapred.JobClient:     SLOTS_MILLIS_MAPS=7370
13/06/19 11:15:55 INFO mapred.JobClient:     Total time spent by all reduces waiting after reserving slots (ms)=0
13/06/19 11:15:55 INFO mapred.JobClient:     Total time spent by all maps waiting after reserving slots (ms)=0
13/06/19 11:15:55 INFO mapred.JobClient:     Launched map tasks=1
13/06/19 11:15:55 INFO mapred.JobClient:     Data-local map tasks=1
13/06/19 11:15:55 INFO mapred.JobClient:     SLOTS_MILLIS_REDUCES=9828
13/06/19 11:15:55 INFO mapred.JobClient:   File Input Format Counters 
13/06/19 11:15:55 INFO mapred.JobClient:     Bytes Read=180
13/06/19 11:15:55 INFO mapred.JobClient:   File Output Format Counters 
13/06/19 11:15:55 INFO mapred.JobClient:     Bytes Written=52
13/06/19 11:15:55 INFO mapred.JobClient:   FileSystemCounters
13/06/19 11:15:55 INFO mapred.JobClient:     FILE_BYTES_READ=82
13/06/19 11:15:55 INFO mapred.JobClient:     HDFS_BYTES_READ=296
13/06/19 11:15:55 INFO mapred.JobClient:     FILE_BYTES_WRITTEN=100437
13/06/19 11:15:55 INFO mapred.JobClient:     HDFS_BYTES_WRITTEN=52
13/06/19 11:15:55 INFO mapred.JobClient:   Map-Reduce Framework
13/06/19 11:15:55 INFO mapred.JobClient:     Map output materialized bytes=82
13/06/19 11:15:55 INFO mapred.JobClient:     Map input records=3
13/06/19 11:15:55 INFO mapred.JobClient:     Reduce shuffle bytes=82
13/06/19 11:15:55 INFO mapred.JobClient:     Spilled Records=6
13/06/19 11:15:55 INFO mapred.JobClient:     Map output bytes=70
13/06/19 11:15:55 INFO mapred.JobClient:     Total committed heap usage (bytes)=210501632
13/06/19 11:15:55 INFO mapred.JobClient:     CPU time spent (ms)=1090
13/06/19 11:15:55 INFO mapred.JobClient:     Map input bytes=94
13/06/19 11:15:55 INFO mapred.JobClient:     SPLIT_RAW_BYTES=116
13/06/19 11:15:55 INFO mapred.JobClient:     Combine input records=0
13/06/19 11:15:55 INFO mapred.JobClient:     Reduce input records=3
13/06/19 11:15:55 INFO mapred.JobClient:     Reduce input groups=1
13/06/19 11:15:55 INFO mapred.JobClient:     Combine output records=0
13/06/19 11:15:55 INFO mapred.JobClient:     Physical memory (bytes) snapshot=227721216
13/06/19 11:15:55 INFO mapred.JobClient:     Reduce output records=3
13/06/19 11:15:55 INFO mapred.JobClient:     Virtual memory (bytes) snapshot=898035712
13/06/19 11:15:55 INFO mapred.JobClient:     Map output records=3

查看结果

root@cluster-1:~/workspace/hadoop-1.1.2# bin/hadoop fs -cat output/*
1	dfs.replication
1	dfs.server.namenode.
1	dfsadmin

至此，Hadoop 的集群版本环境搭建完成

附上一个配置文件说明的文档，http://www.cnblogs.com/serendipity/archive/2011/08/23/2151031.html

-—————————————-

2013-06-19 16:35:52 update 修改了配置文件 conf/hdfs-site.xml

<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<!-- Put site-specific property overrides in this file. -->

<configuration>
     <property>
         <name>dfs.name.dir</name>
         <value>/var/local/hadoop/name</value>
     </property>
     <property>
         <name>dfs.data.dir</name>
         <value>/var/local/hadoop/data</value>
     </property>
     <property>
         <name>dfs.replication</name>
         <value>2</value>
     </property>
</configuration>

然后重新格式化

root@cluster-1:~/workspace/hadoop-1.1.2# bin/hadoop namenode -format
13/06/19 16:33:38 INFO namenode.NameNode: STARTUP_MSG: 
/************************************************************
STARTUP_MSG: Starting NameNode
STARTUP_MSG:   host = cluster-1/10.241.32.32
STARTUP_MSG:   args = [-format]
STARTUP_MSG:   version = 1.1.2
STARTUP_MSG:   build = https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.1 -r 1440782; compiled by 'hortonfo' on Thu Jan 31 02:03:24 UTC 2013
************************************************************/
13/06/19 16:33:38 INFO util.GSet: VM type       = 64-bit
13/06/19 16:33:38 INFO util.GSet: 2% max memory = 19.33375 MB
13/06/19 16:33:38 INFO util.GSet: capacity      = 2^21 = 2097152 entries
13/06/19 16:33:38 INFO util.GSet: recommended=2097152, actual=2097152
13/06/19 16:33:39 INFO namenode.FSNamesystem: fsOwner=root
13/06/19 16:33:39 INFO namenode.FSNamesystem: supergroup=supergroup
13/06/19 16:33:39 INFO namenode.FSNamesystem: isPermissionEnabled=true
13/06/19 16:33:39 INFO namenode.FSNamesystem: dfs.block.invalidate.limit=100
13/06/19 16:33:39 INFO namenode.FSNamesystem: isAccessTokenEnabled=false accessKeyUpdateInterval=0 min(s), accessTokenLifetime=0 min(s)
13/06/19 16:33:39 INFO namenode.NameNode: Caching file names occuring more than 10 times 
13/06/19 16:33:39 INFO common.Storage: Image file of size 110 saved in 0 seconds.
13/06/19 16:33:40 INFO namenode.FSEditLog: closing edit log: position=4, editlog=/var/local/hadoop/name/current/edits
13/06/19 16:33:40 INFO namenode.FSEditLog: close success: truncate to 4, editlog=/var/local/hadoop/name/current/edits
13/06/19 16:33:40 INFO common.Storage: Storage directory /var/local/hadoop/name has been successfully formatted.
13/06/19 16:33:40 INFO namenode.NameNode: SHUTDOWN_MSG: 
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at cluster-1/10.241.32.32
************************************************************/

格式化之后，会在 master 的 /var/local 下面自动建立 hadoop/name

然后启动集群，启动之后，会自动在各个 node 的 /var/local 下面建立 hadoop/data

于是例程的输出也有了变化

root@cluster-1:~/workspace/hadoop-1.1.2# bin/hadoop fs -cat output/*
1	dfs.data.dir
1	dfs.name.dir
1	dfs.replication
1	dfs.server.namenode.
1	dfsadmin

历史评论

ZRJ (2013-08-21 17:12:10):

在namenode启动脚本%HADOOP_HOME%/bin/start-dfs.sh的时候发现datanode报错： Error: JAVA_HOME is not set 原因是在%HADOOP_HOME%/conf/hadoop-env.sh内缺少JAVA_HOME的定义，只需要在hadoop-env.sh中增加： JAVA_HOME=/your/jdk/root/path

ZRJ (2013-08-21 20:41:24):

周一发现hadoop集群down掉了发现由于磁盘已满100% 删除无用文件后重启集群，发现还是起不来，错误如下： /************************************************************ STARTUP_MSG: Starting NameNode STARTUP_MSG: host = SFserver141.localdomain/192.168.15.141 STARTUP_MSG: args = [] STARTUP_MSG: version = 0.20.3-SNAPSHOT STARTUP_MSG: build = -r ; compiled by ‘root’ on Wed Jun 8 12:43:33 CST 2011 ************************************************************/ 2012-10-22 08:50:42,096 INFO org.apache.hadoop.ipc.metrics.RpcMetrics: Initializing RPC Metrics with hostName=NameNode, port=9000 2012-10-22 08:50:42,104 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: Namenode up at: SFserver141.localdomain/192.168.15.141:9000 2012-10-22 08:50:42,112 INFO org.apache.hadoop.metrics.jvm.JvmMetrics: Initializing JVM Metrics with processName=NameNode, sessionId=null 2012-10-22 08:50:42,113 INFO org.apache.hadoop.hdfs.server.namenode.metrics.NameNodeMetrics: Initializing NameNodeMeterics using context object:org.apache.hadoop.metrics.spi.NullContext 2012-10-22 08:50:42,169 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: fsOwner=root,root,bin,daemon,sys,adm,disk,wheel 2012-10-22 08:50:42,169 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: supergroupsupergroup=supergroup 2012-10-22 08:50:42,169 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: isPermissionEnabled=false 2012-10-22 08:50:42,187 INFO org.apache.hadoop.hdfs.server.namenode.metrics.FSNamesystemMetrics: Initializing FSNamesystemMetrics using context object:org.apache.hadoop.metrics.spi.NullContext 2012-10-22 08:50:42,188 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Registered FSNamesystemStatusMBean 2012-10-22 08:50:42,248 INFO org.apache.hadoop.hdfs.server.common.Storage: Number of files = 799968 2012-10-22 08:50:47,535 INFO org.apache.hadoop.hdfs.server.common.Storage: Number of files under construction = 13 2012-10-22 08:50:47,540 INFO org.apache.hadoop.hdfs.server.common.Storage: Image file of size 102734547 loaded in 5 seconds. 2012-10-22 08:50:48,131 INFO org.apache.hadoop.hdfs.server.common.Storage: Edits file /data/java/hadoop020/data/dfs.name.dir/current/edits of size 2749136 edits # 17772 loaded in 0 seconds. 2012-10-22 08:50:48,801 ERROR org.apache.hadoop.hdfs.server.namenode.NameNode: java.lang.NumberFormatException: For input string: "" at java.lang.NumberFormatException.forInputString(NumberFormatException.java:48) at java.lang.Integer.parseInt(Integer.java:470) at java.lang.Short.parseShort(Short.java:120) at java.lang.Short.parseShort(Short.java:78) at org.apache.hadoop.hdfs.server.namenode.FSEditLog.readShort(FSEditLog.java:1311) at org.apache.hadoop.hdfs.server.namenode.FSEditLog.loadFSEdits(FSEditLog.java:541) at org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSEdits(FSImage.java:1011) at org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:826) at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:364) at org.apache.hadoop.hdfs.server.namenode.FSDirectory.loadFSImage(FSDirectory.java:87) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.initialize(FSNamesystem.java:315) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.(FSNamesystem.java:296) at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:205) at org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:283) at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:986) at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:995)

2012-10-22 08:50:48,802 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: SHUTDOWN_MSG:
/************************************************************ SHUTDOWN_MSG: Shutting down NameNode at SFserver141.localdomain/192.168.15.141 ************************************************************/ 大致是因为edits这个文件出现问题；上网查了不少文档，但由于没有设置secondarynamenode；所以没有edits的镜像文件之后发现一篇文章写： printf “\xff\xff\xff\xee\xff” > edits 把上面一段字符串写到edits文件中重启正常注：dfs.name.dir/current文件夹下还出现了edits.new的文件，我是删除的不知道有没有影响本文出自 “工作笔记” 博客，请务必保留此出处http://693340562.blog.51cto.com/1125757/1033582

hadoop 2.5.2 集群安装 | ZRJ (2016-03-18 16:40:56):

[…] 之前写过一个，http://zrj.me/archives/888，不过现在是 2.5.2，东西不同了，安装方式有所变化。 […]

Hadoop 1.1.2 安装（单机版）

Tue, 18 Jun 2013 16:56:27 +0800

操作的环境是阿里云的服务器，http://www.aliyun.com/，还不错，按时间收费，一个小时 0.28 块钱，操作系统是 Ubuntu 10.10 64 位，单核 CPU，内存 512

首先上来先修改主机名

hostname cluster-1

然后开始按照这里，http://hadoop.apache.org/docs/stable/single_node_setup.html，的教程安装 Hadoop 1.1.2

首先看到 Prerequisites，需要 java 的环境，试一下 java 的命令

root@cluster-1:~# java
The program 'java' can be found in the following packages:
 * gcj-4.4-jre-headless
 * gcj-4.5-jre-headless
 * openjdk-6-jre-headless
Try: apt-get install <selected package>

那么首先安装 java，看到这里的教程，http://www.itkee.com/developer/detail-edd.html，首先补上源，然后安装

我的虚拟机装的是Ubuntu 10.04，安装的是Sun JDK6，因为Ubuntu 10.04后官方剔除了Sun JDK的源，提倡用OpenJDK代替（我估计是因为Sun被Oracle收购了，Oracle做了一些伤害开源组织的事，出于报复，故而提倡OpenJDK！），所以首先得加入SunJDK的源，

sudo gedit /etc/apt/sources.list编辑源列表，在文件末尾加上一句：

deb http://archive.canonical.com/ lucid partner

然后sudo apt-get update运行更新，

再输入sudo apt-get install sun-java6-jdk，提示您要不要下载软件包，输入y继续。

安装好JDK后，然后就是设置环境变量

sudo gedit /etc/profile编辑文件，在文件末尾加上

export JAVA_HOME=/usr/lib/jvm/java-6-sun-1.6.0.24

export PATH=$JAVA_HOME/bin:$JAVA_HOME/jre/bin:$PATH

export CLASSPATH=$CLASSPATH:.:$JAVA_HOME/lib:$JAVA_HOME/jre/lib

java路径是在命令行下安装时默认的路径，如果你的路径不同，只需修改第一行即可。

使配置生效，输入“source /etc/profile”

使用“java -version”查看是否配置成功，我的是：

bwk@ubuntu:~$ java -version

java version “1.6.0_24”

Java(TM) SE Runtime Environment (build 1.6.0_24-b07)

Java HotSpot(TM) Client VM (build 19.1-b02, mixed mode, sharing)

证明配置成功，好了，大功告成！

但是在 apt-get update 的时候出现问题，没有办法 update 成功，于是想到自己手工去 java 官网下载安装程序来安装，看到这篇文章，http://os.51cto.com/art/201003/189114.htm，去到这个网址，http://www.oracle.com/technetwork/java/javase/downloads/jdk6downloads-1902814.html，下载了 jdk 6 的 bin 文件，上传到 vps

root@cluster-1:~# java -version
java version "1.6.0_45"
Java(TM) SE Runtime Environment (build 1.6.0_45-b06)
Java HotSpot(TM) 64-Bit Server VM (build 20.45-b01, mixed mode)

然后从 http://www.apache.org/dyn/closer.cgi/hadoop/common/ 下载 1.1.2 的 Hadoop

hadoop-1.1.2.tar.gz                                31-Jan-2013 22:42            61927560

解压缩完了，按照要求，edit the file conf/hadoop-env.sh to define at least JAVA_HOME to be the root of your Java installation.

修改完了，按照教程上的示例，跑了一下这个代码

$ mkdir input 
$ cp conf/*.xml input 
$ bin/hadoop jar hadoop-examples-*.jar grep input output 'dfs[a-z.]+' 
$ cat output/*

得到的结果是

root@cluster-1:~/hadoop-1.1.2# cat output/*
1	dfsadmin

这个是在 standalone 模式下的，问题教程也没有给出标准答案，所以也不知道得到这样的结果是不是对的

不过使用以下这个命令，可以大致的判断出来，那个 jar 是模拟 grep 的功能的，而我们的结果也应该是对的

root@cluster-1:~/hadoop-1.1.2# cat input/* | egrep dfs[a-z.]
    dfsadmin and mradmin commands to refresh the security policy in-effect.

接下来根据教程，尝试一下那个 Pseudo-Distributed 模式，按照教程修改了配置文件之后，需要让 ssh 能够无密码登录

$ ssh-keygen -t dsa -P '' -f ~/.ssh/id_dsa 
$ cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys

按照教程跑起 start-all 的脚本之后，访问 namenode 的 web 控制页面，可以看到这样的界面

以及这个是 jobtracker 的控制界面

但是在按照教程跑这个命令的时候报错了

bin/hadoop jar hadoop-examples-*.jar grep input output 'dfs[a-z.]+'

错误是

13/06/18 14:57:52 INFO util.NativeCodeLoader: Loaded the native-hadoop library
13/06/18 14:57:52 WARN snappy.LoadSnappy: Snappy native library not loaded
13/06/18 14:57:52 INFO mapred.FileInputFormat: Total input paths to process : 16
13/06/18 14:57:52 INFO mapred.JobClient: Running job: job_201306181446_0001
13/06/18 14:57:54 INFO mapred.JobClient:  map 0% reduce 0%
13/06/18 14:58:09 INFO mapred.JobClient: Task Id : attempt_201306181446_0001_m_000001_0, Status : FAILED
Error: Java heap space
13/06/18 14:58:09 INFO mapred.JobClient: Task Id : attempt_201306181446_0001_m_000000_0, Status : FAILED
13/06/18 14:58:19 INFO mapred.JobClient: Task Id : attempt_201306181446_0001_m_000000_1, Status : FAILED
Error: Java heap space
13/06/18 14:58:19 INFO mapred.JobClient: Task Id : attempt_201306181446_0001_m_000001_1, Status : FAILED
Error: Java heap space
13/06/18 14:58:29 INFO mapred.JobClient: Task Id : attempt_201306181446_0001_m_000000_2, Status : FAILED
Error: Java heap space
13/06/18 14:58:29 INFO mapred.JobClient: Task Id : attempt_201306181446_0001_m_000001_2, Status : FAILED
Error: Java heap space
13/06/18 14:58:42 INFO mapred.JobClient: Job complete: job_201306181446_0001
13/06/18 14:58:42 INFO mapred.JobClient: Counters: 7
13/06/18 14:58:42 INFO mapred.JobClient:   Job Counters 
13/06/18 14:58:42 INFO mapred.JobClient:     SLOTS_MILLIS_MAPS=42937
13/06/18 14:58:42 INFO mapred.JobClient:     Total time spent by all reduces waiting after reserving slots (ms)=0
13/06/18 14:58:42 INFO mapred.JobClient:     Total time spent by all maps waiting after reserving slots (ms)=0
13/06/18 14:58:42 INFO mapred.JobClient:     Launched map tasks=8
13/06/18 14:58:42 INFO mapred.JobClient:     Data-local map tasks=8
13/06/18 14:58:42 INFO mapred.JobClient:     SLOTS_MILLIS_REDUCES=0
13/06/18 14:58:42 INFO mapred.JobClient:     Failed map tasks=1
13/06/18 14:58:42 INFO mapred.JobClient: Job Failed: # of failed Map Tasks exceeded allowed limit. FailedCount: 1. LastFailedTask: task_201306181446_0001_m_000000
java.io.IOException: Job failed!
	at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1327)
	at org.apache.hadoop.examples.Grep.run(Grep.java:69)
	at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
	at org.apache.hadoop.examples.Grep.main(Grep.java:93)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:68)
	at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:139)
	at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:64)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at org.apache.hadoop.util.RunJar.main(RunJar.java:156)

google 一下，看到这里，http://stackoverflow.com/questions/15609909/error-java-heap-space，需要修改 java 的堆大小

Clearly you have run out of the heap size allotted to Java. So you shall try to increase that.

For that you may execute the following before executing hadoop command:

export HADOOP_OPTS="-Xmx4096m" Alternatively, you can achieve the same thing by adding the following permanent setting in your mapred-site.xml file, this file lies in HADOOP_HOME/conf/ :

mapred.child.java.opts -Xmx4096m This would set your java heap space to 4096 MB (4GB), you may even try it with a lower value first if that works. If that too doesn’t work out then increase it more if your machine supports it, if not then move to a machine having more memory and try there. As heap space simply means you don’t have enough RAM available for Java.

然后重启整个 Hadoop

root@cluster-1:~/workspace/hadoop-1.1.2# bin/stop-all.sh 
stopping jobtracker
localhost: stopping tasktracker
stopping namenode
localhost: stopping datanode
localhost: stopping secondarynamenode
root@cluster-1:~/workspace/hadoop-1.1.2# bin/start-all.sh 
starting namenode, logging to /root/workspace/hadoop-1.1.2/libexec/../logs/hadoop-root-namenode-cluster-1.out
localhost: starting datanode, logging to /root/workspace/hadoop-1.1.2/libexec/../logs/hadoop-root-datanode-cluster-1.out
localhost: starting secondarynamenode, logging to /root/workspace/hadoop-1.1.2/libexec/../logs/hadoop-root-secondarynamenode-cluster-1.out
starting jobtracker, logging to /root/workspace/hadoop-1.1.2/libexec/../logs/hadoop-root-jobtracker-cluster-1.out
localhost: starting tasktracker, logging to /root/workspace/hadoop-1.1.2/libexec/../logs/hadoop-root-tasktracker-cluster-1.out

再次尝试之前失败的任务，依然报同样的错误，回想起之前的 conf/hadoop-env.sh 文件，把其中的

export HADOOP_HEAPSIZE=2000

这一行的注释去掉，再重启 Hadoop，运行任务，跑起来直接崩溃了

13/06/18 15:16:58 ERROR security.UserGroupInformation: PriviledgedActionException as:root cause:org.apache.hadoop.ipc.RemoteException: org.apache.hadoop.mapred.SafeModeException: JobTracker is in safe mode
	at org.apache.hadoop.mapred.JobTracker.checkSafeMode(JobTracker.java:5270)
	at org.apache.hadoop.mapred.JobTracker.getStagingAreaDir(JobTracker.java:3797)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:578)
	at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1393)
	at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1389)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:396)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149)
	at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1387)

org.apache.hadoop.ipc.RemoteException: org.apache.hadoop.hdfs.server.namenode.SafeModeException: Cannot delete /user/root/grep-temp-1080766159. Name node is in safe mode.
The reported blocks 17 has reached the threshold 0.9990 of total blocks 17. Safe mode will be turned off automatically in 19 seconds.
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.deleteInternal(FSNamesystem.java:2111)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.delete(FSNamesystem.java:2088)
	at org.apache.hadoop.hdfs.server.namenode.NameNode.delete(NameNode.java:832)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:578)
	at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1393)
	at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1389)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:396)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149)
	at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1387)

	at org.apache.hadoop.ipc.Client.call(Client.java:1107)
	at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:229)
	at com.sun.proxy.$Proxy1.delete(Unknown Source)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:85)
	at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:62)
	at com.sun.proxy.$Proxy1.delete(Unknown Source)
	at org.apache.hadoop.hdfs.DFSClient.delete(DFSClient.java:981)
	at org.apache.hadoop.hdfs.DistributedFileSystem.delete(DistributedFileSystem.java:245)
	at org.apache.hadoop.examples.Grep.run(Grep.java:87)
	at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
	at org.apache.hadoop.examples.Grep.main(Grep.java:93)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:68)
	at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:139)
	at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:64)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at org.apache.hadoop.util.RunJar.main(RunJar.java:156)

把之前的 mapred-site.xml 里面的改动恢复，再尝试，执行任务依然会直接崩溃

最后经过多个搜索，定位到问题应该是在 conf/hadoop-env.sh 这个文件中的 HADOOP_HEAPSIZE 这个变量中，对于这个变量的修改，我的步骤是这样的，首先停掉 Hadoop

# bin/stop-all.sh

然后删除 logs 和 tmp

#rm -rf logs/*
# rm -rf /tmp/hadoop-root*

然后修改配置文件

#vi conf/hadoop-env.sh

并重新格式化 namenode

#bin/hadoop namenode -format

然后重启 Hadoop

#bin/start-all.sh

并重新拷入文件

# bin/hadoop fs -mkdir input
# bin/hadoop fs -put conf/*.xml input

然后执行任务

# bin/hadoop jar hadoop-examples-*.jar grep input output 'dfs[a-z.]+'

可惜的是，当我把变量的值设为 5000 或者更高的时候，就会报内存不足，无法初始化 java 虚拟机

13/06/18 16:28:14 INFO mapred.JobClient: Task Id : attempt_201306181619_0001_m_000007_1, Status : FAILED
java.lang.Throwable: Child Error
	at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:271)
Caused by: java.io.IOException: Task process exit with nonzero status of 1.
	at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:258)

attempt_201306181619_0001_m_000007_1: Error occurred during initialization of VM
attempt_201306181619_0001_m_000007_1: Could not reserve enough space for object heap

而在 4000 或者以下的时候，就会报堆内存不足

13/06/18 16:46:13 INFO mapred.JobClient: Task Id : attempt_201306181644_0001_m_000001_1, Status : FAILED
Error: Java heap space

由于这个 vps 的内存只有 512，所以我觉得应该没有办法在单机上模拟 Pseudo-Distributed，决定跳过，尝试 Fully-Distributed