装配HBase Snappy

安装HBase Snappy
[talkyun@tbe192168147014 snappy-1.1.0]$ sudo yum install gccLoaded plugins: fastestmirror, securityDetermining fastest mirrorsCould not retrieve mirrorlist http://mirrorlist.centos.org/?release=6&arch=x86_64&repo=os error was14: PYCURL ERROR 6 - "Couldn't resolve host 'mirrorlist.centos.org'"Error: Cannot find a valid baseurl for repo: base[talkyun@tbe192168147014 snappy-1.1.0]$ nslookup mirrorlist.centos.org;; connection timed out; trying next origin;; connection timed out; no servers could be reached[talkyun@tbe192168147014 snappy-1.1.0]$ cat /etc/resolv.conf

? ? ? ? ? ? 解决方案：

[talkyun@tbe192168147014 snappy-1.1.0]$ sudo vim /etc/resolv.conf[talkyun@tbe192168147014 snappy-1.1.0]$ cat /etc/resolv.conf # Generated by NetworkManagernameserver 192.168.200.19

?

? ? ? ? ? ? [talkyun@tbe192168147058 snappy-1.1.0]$ sudo yum install autoconf

? ? ? ? ? ? [talkyun@tbe192168147058 snappy-1.1.0]$ sudo yum install automake

? ? ? ? ? ? [talkyun@tbe192168147058 snappy-1.1.0]$ sudo yum install libtool

? ? ? ? ? ? [talkyun@tbe192168147058 snappy-1.1.0]$ ./configure

? ? ? ? ? ? [talkyun@tbe192168147058 snappy-1.1.0]$ make

? ? ? ? ? ??错误二：./libtool: line 1125: g++: command not found

? ? ? ? ? ? 解决方案：sudo yum -y install gcc-c++

? ? ? ? ? ? 如果出现错误，解决之后，要删除原来的目录，重新创建和使用新的。

? ? ? ? ? ??[talkyun@tbe192168147058 snappy-1.1.0]$ cd ..

? ? ? ? ? ? [talkyun@tbe192168147058 snappy]$ rm -rf snappy-1.1.0

? ? ? ? ? ? [talkyun@tbe192168147058 snappy]$ tar xvfz snappy-1.1.0.tar.gz

? ? ? ? ? ? [talkyun@tbe192168147058 snappy]$ cd snappy-1.1.0

? ? ? ? ? ? [talkyun@tbe192168147058 snappy-1.1.0]$ ./configure

? ? ? ? ? ? [talkyun@tbe192168147058 snappy-1.1.0]$ make

? ? ? ? ? ? [talkyun@tbe192168147058 snappy-1.1.0]$ sudo make install

3.检查安装是否成功

? ? ??ls /usr/local/lib/libsn*

?

二、安装Hadoop Snappy

1.下载hadoop-snappy源码

? ? ??svn checkout http://hadoop-snappy.googlecode.com/svn/trunk/ hadoop-snappy

2.安装hadoop-snappy

? ? ??cd hadoop-snappy

? ? ? mvn package

? ? ??[bigdata@tbe192168147015 hadoop-snappy]$ /home/bigdata/maven/apache-maven-3.0.4/bin/mvn package

? ? ? 错误一：

     [exec] /bin/sh ./libtool --tag=CC   --mode=link gcc -g -Wall -fPIC -O2 -m64 -g -O2 -version-info 0:1:0 -L/usr/local/lib -o libhadoopsnappy.la -rpath /usr/local/lib src/org/apache/hadoop/io/compress/snappy/SnappyCompressor.lo src/org/apache/hadoop/io/compress/snappy/SnappyDecompressor.lo  -ljvm -ldl      [exec] /usr/bin/ld: cannot find -ljvm     [exec] collect2: ld returned 1 exit status     [exec] make: *** [libhadoopsnappy.la] Error 1

? ? ? ?解决方案：sudo ln -s /home/bigdata/apps/jdk/jre/lib/amd64/server/libjvm.so /usr/local/lib/

3.复制构建好的库文件到相应目录

? ? ? 注意：libsnappy.so需要依赖libhadoop.so，libsnappy和libhadoop的装载是紧耦合，因此需要把libhadoop.so一起复制到相应目录。

? ? ? [bigdata@tbe192168147015 hadoop-snappy]$ cd target/hadoop-snappy-0.0.1-SNAPSHOT-tar/hadoop-snappy-0.0.1-SNAPSHOT/

? ? ??[bigdata@tbe192168147015 hadoop-snappy-0.0.1-SNAPSHOT]$ cp /tmp/libhadoop.so lib/native/Linux-amd64-64/libhadoop.so

? ? ? [bigdata@tbe192168147015 hadoop-snappy-0.0.1-SNAPSHOT]$ cp -r lib/ /home/bigdata/apps/hadoop-mr-talkyun/

? ? ? [bigdata@tbe192168147015 hadoop-snappy-0.0.1-SNAPSHOT]$ cp -r lib/ /home/bigdata/apps/hbase-talkyun/?

?

? ? ? [bigdata@tbe192168147015 hadoop-snappy-0.0.1-SNAPSHOT]$ scp -r lib/ tbe192168147010:/home/bigdata/apps/hadoop-mr-talkyun/

? ? ? [bigdata@tbe192168147015 hadoop-snappy-0.0.1-SNAPSHOT]$ scp -r lib/ tbe192168147010:/home/bigdata/apps/hbase-talkyun/?

?

三、重启HDFS、MapReduce、HBase

?

四、检查安装是否成功

1.Testing Compression is enabled

hbase org.apache.hadoop.hbase.util.CompressionTest /tmp/testfile snappy

hbase org.apache.hadoop.hbase.util.CompressionTest file:///tmp/testfile snappy

2.create table

create 'mytable', {NAME => 'colfam', COMPRESSION => 'snappy'}

?

五、参考资料

1.snappy - A fast compressor/decompressor

2.Spark 0.7 installation on EC2 Can't read Snappy Compressed files

3.hadoop-snappy - A Hadoop library of Snappy compression

4.cloudera中hbase使用Snappy算法安装及设置

5.Hadoop HBase 配置 安装 Snappy 终极教程

6.Unable to load native-hadoop library 和 Snappy native library not loaded 的解决

7.mac os上配置CDH4.1.1版hbase并启用snappy

?