Hadoop -【IO专题-序列化机制】
?
引自http://blog.sina.com.cn/s/blog_48a45b950100sz4x.html
?
1.?基本概念?
???????序列化可被定义为将对象的状态存储到存储媒介中的过程。在此过程中，对象的公共字段和私有字段以及类的名称（包括包含该类的程序集）都被转换为字节流，然后写入数据流。在以后反序列化该对象时，创建原始对象的精确复本。
???????当在面向对象的环境中实现序列化机制时，您需要在简化使用和保持灵活性之间进行许多权衡。只要您对该过程具有充分的控制，就可以在很大程度上自动化该过程。例如，在简单二进制序列化不充分时可能导致一些情况发生，或者可能有特定原因确定在类中哪些字段需要进行序列化。
???????Serialization is the process of turning structured objects into a byte stream for trans-mission over a network or for writing to persistent storage. Deserialization is the process of turning a byte stream back into a series of structured objects.
???????Serialization?appears in two quite distinct areas of distributed data processing: forinterprocess?communication?and for?persistent?storage.
???????In Hadoop, interprocess?communication?between nodes in the system is implemented usingremote procedure calls (RPCs).?The RPC protocol uses serialization?to render the message into a binary stream?to be sent to the remote node, which?then deserializes the binary stream into the original message. In general, it is?desirable that an RPC serialization format is:
???????·??数据设计紧凑，充分利用网络带宽
???????·??系列化和反序列化的工程能够迅速完成
???????·??协议是高可扩展的
???????·??支持互操作
???????Hadoop的系列化是通过Writable接口来实现的，只满足了前两条设计，在org.apache.hadoop.io包下包含了大量的可序列化的组件，它们都实现了Writable接口，Writable接口提供了两个方法，write和readFields，分别用来序列化和反序列化，实现该接口的典型例子如下：
public?class?MyWritable?implements?Writable {
????// some data
????private?int?counter;
????private?long?timestamp;
????// 序列化方法
????public?void?write(DataOutput out)?throws?IOException {
????????out.writeInt(counter);
????????out.writeLong(timestamp);
????}
????// 反序列化方法
????public?void?readFields(DataInput in)?throws?IOException {
????????counter = in.readInt();
????????timestamp = in.readLong();
????}
????// 反序列化的对外使用接口
????public?static?MyWritable read(DataInput in)?throws?IOException {
????????MyWritable w =?new?MyWritable();
????????w.readFields(in);
????????return?w;
????}
}
?
2.?比较器?
在Map/Reduce中有多处要进行排序，所以元素的相互比较是比较重要的，Hadoop中通过如下几个接口和类来支持比较操作：
1)??WritableComparable
public?interface?WritableComparable<T>?extends?Writable, Comparable<T> {
}
2)??RawComparator
public?interface?RawComparator<T>?extends?Comparator<T> {
????public?int?compare(byte[] b1,?int?s1,?int?l1,?byte[] b2,?int?s2,?int?l2);
}
This interface permits implementors?to compare records read from a stream without
deserializing?them into objects, thereby avoiding any overhead of object creation.
3)??WritableComparator
WritableComparator is a general-purpose implementation?of??RawComparator for WritableComparable classes.?It provides two main functions：?
First, it provides a default implementation of the raw compare() method that deserializes the objects to be compared from the stream and invokes the object compare() method.
Second, it acts as a factory for RawComparator instances (that Writable implementations haveregistered，通过一个HashMap<Class, WritableComparator> comparators来保持所有注册的comparator).For?example, to obtain a comparator?for IntWritable, we just use:
????RawComparator<IntWritable> comparator?= WritableComparator.get(IntWritable.class);
????获得这个comparator之后就可以来比较两个IntWritable的对象：
????IntWritable w1 = new IntWritable(100);
????IntWritable w2 = new IntWritable(101);
????assertThat(comparator.compare(w1,w2), greatherThan(0));
????或者对序列化的对象进行比较：
????byte[] b1 = serialize(w1);
????byte[] b2 = serialize(w2);
????assertThat(comparator.compare(b1,0,b1.length, b2, 0, b2.length), greatherThan(0));
?
3.?Writable Classes
Hadoop提供的可序列化的类型主要分成如下几种：
1)??Java基本数据类型?
There are??Writable wrappers??for all??the??Java primitive??types??except short and char (both of which can be stored in an IntWritable). All have a get() and a set() method for retrieving and storing the wrapped value.
Java 基本数据类型
Writable实现
Serialized size(bytes)
boolean
BooleanWritable
1
byte
ByteWritable
1
int
IntWritable
4
VIntWritable
1-5
float
FloatWritable
4
long
LongWritable
8
VLongWritable
1-9
double
DoubleWritable
8
2)??Text
Text is a Writable for UTF-8 sequences, the max length is 2GB, 这个类的系列化和反序列化都是比较显而易见的，实现上大部分代码在做 UTF-8编解码和String（java中的String用的是unicode）的转换等。
3)??BytesWritable
BytesWritable is a wrapper?for an?array of binary data. Its serialized format is?an integerfield (4 bytes) that specifies the number of bytes to follow, followed by the bytes them-selves.?For example, the byte array of length two with values 3 and 5 is serialized as a 4-byte integer (00000002) followed by the two bytes from the array (03 and 05):
????BytesWritable b = new BytesWritable(new byte[] { 3,?5 });
????byte[] bytes = serialize(b);
????assertThat(StringUtils.byteToHexString(bytes), is("000000020305"));
????BinaryComparable是针对array of binary data而设计的比较器！
4)??NullWritable
NullWritable is a special type of Writable, as it has a zero-length serialization.?No bytes are written to, or read??from, the stream. It??is used as a placeholder; for example, in MapReduce, a key or a value can be declared as a NullWritable when you don’t need to use that position—it effectively stores a constant empty value.
5)??ObjectWritable和GenericWritable
ObjectWritable?is a general-purpose?wrapper for the following:?Java primitives, String, enum, Writable, null, or arrays of any of these types.?It is used in Hadoop RPC?to marshal（包装） and unmarshal method arguments and return types. 其实主要的通途就是对多于1个的域组成对象进行序列化！在对端进行反序列化的时候用到了WritableFactory和?WritableFactories（用来根据类名来生成对象）
6)??Writable Collections
There are four Writable?collection?types in the org.apache.hadoop.io package:
????ArrayWritable, TwoDArrayWritable, MapWritable, and SortedMapWritable.
ArrayWritable?and??TwoDArrayWritable??are??Writable??implementations??for??arrays??and two-dimensional?arrays (array of arrays) of Writable instances.?All the elements of an ArrayWritable?or a TwoDArrayWritable?must be??instances of??the same class, which??is specified at construction, as follows:
????ArrayWritable?writable = new ArrayWritable(Text.class);
MapWritable?and SortedMapWritable?are?implementations?of java.util.Map<Writable,Writable> and java.util.SortedMap<WritableComparable, Writable>, respectively. The type of each key and value field is a part of the serialization format for that field.
?
