Hadoop开发周期(三):单元测试
MapReduce一旦打包提交到分布式环境,如果出了问题,需要要定位调试,然后再打包发布。如果在发布MapRduce之前其做单元测试,消除明显的代码bug和逻辑错误,可以提高开发效率。
MRUnit是一款由Couldera公司开发的专门针对Hadoop中编写MapReduce单元测试的框架。可以用MapDriver单独测试Map,用ReduceDriver单独测试Reduce,用MapReduceDriver测试MapReduce作业。(Apache MRUnit ? is a Java library that helps developers unit test Apache Hadoop map reduce jobs.)
下面使用MRUnit对Hadoop开发周期(二):编写mapper和reducer程序 中的单词字数统计例子进行单元测试。在进行单元测试前需要导入MRUnit的jar包。单元测试代码如下:
常见错误解决:
错误:
java.lang.IncompatibleClassChangeError: Found class org.apache.hadoop.mapreduce.TaskInputOutputContext, but interface was expected
解决办法
changed the jar from mrunit-0.9.0-incubating-hadoop2.jar(which is suggested in Tutorial[hadoop-1.0.4]) to mrunit-0.9.0-incubating-hadoop1.jar and now it works well.