hadoop分布式集群安装
hadoop分布式集群安装
?
1、配置ssh无密码登录
安装ssh后
将slaves的ip加入/etc/hosts中
执行cd ~ ，输入ssh-keygen后连续3次回车
cat .ssh/id_rsa.pub >> .ssh/authorized_keys
分发主机公钥
scp .ssh/id_rsa.pub root@slave:/root/master.pub
cat master.pub >> .ssh/authorized_keys
测试ssh slave
#ssh master
ssh-keygen -t rsa -p ''
cat ~/.ssh/id_rsa.pub >>? ~/.ssh/authorized_keys
for slave in $HADOOP_HOME/conf/slaves) do
?scp ~/.ssh/id_rsa.pub root@slave:/root/master.pub
done
#for slaves
domain=""
master="192.168.0.100?$1?$domain"
slave="192.168.0.$3?$2?$domain"
ssh-keygen -t rsa -p ''
cat ~/.ssh/id_rsa.pub >>? ~/.ssh/authorized_keys
cat /root/master.pub >> ~/.ssh/authorized_keys
echo $master >> /etc/hosts
echo $slave >> /etc/hosts
mkdir /home/hadoop
#for master
for slave in $HADOOP_HOME/conf/slaves) do
?scp $HADOOP_HOME root@$slave:$HADOOP_HOME
done
#sync conf
for slave in $HADOOP_HOME/conf/slaves) do
?scp $HADOOP_HOME/conf/*-site.xml root@$slave:$HADOOP_HOME/conf/
?scp $HADOOP_HOME/conf/masters root@$slave:$HADOOP_HOME/conf/
?scp $HADOOP_HOME/conf/slaves root@$slave:$HADOOP_HOME/conf/
done
2、安装java1.6
到java官方网站下载解压版jdk1.6或以上
tar -xzf jdk-*-tar.gz
3、安装hadoop
下载hadoop-*-tar.gz 并解压
?
3、配置hadoop
vi /etc/profile
追加以下hadoop环境变量及hadoop快捷命令
export JAVA_HOME=/usr/jdk1.7.0_03
export HADOOP_HOME=/home/hadoop/hadoop-0.20.203
export PATH=$JAVA_HOME/bin:$HADOOP_HOME/bin:$PATH
alias hls='hadoop fs -ls'
alias hcat='hadoop fs -cat'
alias hput='hadoop fs -put'
alias hget='hadoop fs -get'
alias hrmr='hadoop fs -rmr'
alias hmkdir='hadoop fs -mkdir'
alias hjar='hadoop jar'
修改bin/hadoop加入export JAVA_HOME=/usr/jdk1.7.0_03后分发
修改site.xml后分发
配置masters、slaves文件。
?
4、启动集群
start-all.sh
jps查看启动情况
查看master机器上的http服务是否与预期一致。
http://192.168.0.100:50030/查看jobtracker的情况，http://192.168.0.100:50070/查看dfs情况
?
5、测试集群
在master上执行以下命令测试测试dfs及mr。
hls /
hmkdir input
hput conf/*.xml input
通过hjar来执行wordcount来测试map reduce
hjar hadoop-example*.jar wordcount?
?
?
