Avatar配备（转载）

Avatar配置（转载）
<!--fail message="'forrest.home' is not defined. Please pass-Dforrest.home=&lt;base of Apache Forrest installation&gt; to Ant onthe command-line." /-->
</target>
<target name="java5.check" unless="java5.home">
<!--fail message="'java5.home' is not defined.? Forrest requires Java 5.? Please pass -Djava5.home=&lt;base of Java5 distribution&gt; to Ant on the command-line." /-->
</target>

2.?????在根目录下输入ant jar（对于编译package可以参考build.xml的代码）编译hadoop，编译后的jar包会在build目录下（hadoop-0.20.3-dev-core.jar）， 拷贝该jar包到hadoop根目录下替换到原有的jar （啰嗦一句，hadoop启动时会先加载build目录下的class，所以当通过替换class修改jar包时请先把build目录暂时移除掉） 。

3.??进入src/contrib/highavailability目录下编译Avatar，编译后的jar包会在build/contrib/highavailability目录下(hadoop-${version}-highavailability.jar)，拷贝该jar包到lib目录下。

4.?????把2,3步中编译好的jar包分发到集群中所有机器的相应目录。

1.?????配置hdfs-site.xml

<?xmlversion="1.0"?>

<?xml-stylesheettype="text/xsl" href="configuration.xsl"?>

?

<!-- Put site-specificproperty overrides in this file. -->

?

<configuration>

???????<property>

???????????????<name>dfs.name.dir</name>

???????????????<value>/home/hadoop/name</value>

???????????????<description>Determineswhere on the local filesystem the?????????????????????? DFS name node shouldstore the name table. If this is a???????????????? comma-delimited list ofdirectories then the name table is????????????????????? replicated in all of thedirectories, for redundancy.

???????????????</description>

???????</property>

???????<property>

???????????????<name>dfs.data.dir</name>

???????????????<value>/home/hadoop/data</value>

???????????????<description>

???????????????</description>

???????</property>

???????<property>

???????????????<name>dfs.http.address0</name>

??????? ????????<value>hadoop1-virtual-machine:50070</value>

???????</property>

???????<property>

???????????????<name>dfs.http.address1</name>

???????????????<value>hadoop5-virtual-machine:50070</value>

???????</property>

???????<property>

???????????????<name>dfs.name.dir.shared0</name>

???????????????<value>/home/hadoop/share/shared0</value>

???????</property>

???????<property>

???????????????<name>dfs.name.dir.shared1</name>

???????????????<value>/home/hadoop/share/shared1</value>

???????</property>

???????<property>

???????????????<name>dfs.name.edits.dir.shared0</name>

???????????????<value>/home/hadoop/share/shared0</value>

???????</property>

???????<property>

???????????????<name>dfs.name.edits.dir.shared1</name>

???????????????<value>/home/hadoop/share/shared1</value>

???????</property>

???????<property>

???????????<name>dfs.replication</name>

???????????????<value>3</value>

???????????????<description>Defaultblock replication. The actual number of??????????????????? replications can bespecified when the file is created. The???????????????????? default isused ifreplication is not specified in create??????????????????????? time.

???????????????</description>

???????</property>

</configuration>

参数说明：

1)??dfs.name.dir.shared0

AvatarNode(Primary)元数据存储目录,注意不能和dfs.name.dir目录相同

2)??dfs.name.dir.shared1

AvatarNode(Standby)元数据存储目录,注意不能和dfs.name.dir目录相同

3)??dfs.name.edits.dir.shared0

AvatarNode(Primary) edits文件存储目录，默认与?????? ??? ????dfs.name.dir.shared0一致

4)??dfs.name.edits.dir.shared1

AvatarNode(Standby) edits文件存储目录，默认与???????? ?dfs.name.dir.shared1一致

5)??dfs.http.address0

AvatarNode(Primary) HTTP的监控地址

6)??dfs.http.address1

AvatarNode(Standby) HTTP的监控地址

7)??dfs.namenode.dn-address0/dfs.namenode.dn-address1

虽然在Avatar源码中有所涉及，但暂时并未用到

2.??配置core-site.xml

<?xmlversion="1.0"?>

<?xml-stylesheettype="text/xsl" href="configuration.xsl"?>

?

<!-- Put site-specificproperty overrides in this file. -->

?

<configuration>

???????<property>

???????????????<name>hadoop.tmp.dir</name>

???????????????<value>/home/hadoop/tmp</value>

???????????????<description>A base forother temporary directories.

???????????????</description>

???????</property>

???????<property>

???????????????<name>fs.default.name</name>

???????????????<value>hdfs://hadoop1-virtual-machine:9000</value>

???????????????<description>The name ofthe default file system. Either the??????????????????? literal string"local" or a host:port for DFS.

???????????????</description>

???????</property>

???????<property>

???????????????<name>fs.default.name0</name>

???????????????<value>hdfs://hadoop1-virtual-machine:9000</value>

???????????????<description>The name ofthe default file system. Either the??????????????????? literal string"local" or a host:port for DFS.

???????????????</description>

???????</property>

???????<property>

???????????????<name>fs.default.name1</name>

???????????????<value>hdfs://hadoop5-virtual-machine:9000</value>

???????????????<description>The name ofthe default file system. Either the??????????????????? literal string"local" or a host:port for DFS.

???????????????</description>

???????</property>

</configuration>

参数说明：

1)??fs.default.name

当前AvatarNode?IP地址和端口号，即Primary和Standby的配置为各自的IP地址和端口号。

2)??fs.default.name0

AvatarNode(Primary)?IP地址和端口号

3)??fs.default.name1

AvatarNode(Standby)? IP地址和端口号

3.???? 因为不涉及到mapred，故mapred-site.xml不用作修改，为原有集群配置即可。

4.???? 分发修改后的配置文件到集群节点并在Primary和Standby节点上建立好配置文件中相应目录。

5.???? 建立NFS，实现Primary与Standby?shared0目录的数据共享。有关NFS的配置请参考http://blog.csdn.net/rzhzhz/article/details/7056732

6.???? 格式化Primary与Standby，这里可以采用hadoop本身的格式化命令，也可以采用AvatarNode的格式化命令（bin/hadoop org.apache.hadoop.hdfs.AvatarShell -format），但此时shared1目录不能为空，此处有点多余。建议采用hadoop本身的格式化命令在Primary上格式化后，并且把name目录下的文件复制到shared0目录下。然后再在Standby上复制shared0目录下的文件到shared1目录下。

1.???? 由于不涉及jobtracker的单点，在这里我们只启动hdfs相关线程。Primary，Standby两个namenode（此处Standby包括SecondaryNamenode的职责）和3个AvatarDataNode数据节点。

2.???? 在Primary节点hadoop根目录下启动AvatarNode（Primary）

????? bin/hadooporg.apache.hadoop.hdfs.server.namenode.AvatarNode –zero

3.???? 在Standby节点hadoop根目录下启动AvatarNode（Standby）

????? bin/hadooporg.apache.hadoop.hdfs.server.namenode.AvatarNode -one –standby

4.???? 依次在数据节点hadoop根目录下启动AvatarDataNode

????? bin/hadooporg.apache.hadoop.hdfs.server.datanode.AvatarDataNode

5.???? 其他相关命令

bin/hadoop ??org.apache.hadoop.hdfs.server.namenode.AvatarNode，后面可??? 选参数有

[-standby] | [-sync] | [-zero] | [-one] | [-format] |? [-upgrade]?? | [-rollback] | [-finalize] |?[-importCheckpoint]

??

1)??bin/hadoop org.apache.hadoop.hdfs.AvatarShell –showAvatar?查看当前AvatarNode的状态

?

2)???? bin/hadoop org.apache.hadoop.hdfs.AvatarShell? -setAvatar primary 把当前Standby节点升级Primary节点

?

3)??bin/hadoop org.apache.hadoop.hdfs.AvatarShell?-setAvatar standby

?

1.???? 访问集群的web页

(Primary)http://hadoop1-virtual-machine:50070

(Standby)http://hadoop5-virtual-machine:50070

可见所有的AvatarDataNode都已注册到两个namenode，Primary处于正常状态，而Standby处于Safemode状态，只可读不可写。可通过AvatarShell命令查看当前AvatarNode的状态（Primary或Standby）。

2.???? 存储相关数据到集群，集群正常工作。

3.???? Kill掉Primary节点的AvatartNode线程，在Standby把当前升级为Prirmary,数据并未丢失，集群正常工作（此时web端不能正常访问文件系统，通过shell命令可查看集群数据）。但由于Avatar有转换限制，只能由Standby转换成Primary，故一次故障后,由Standby上升为Primary的节点并不能重新降级为Standby，所以不能实现像Master/Slave那种自由切换。