首页 诗词 字典 板报 句子 名言 友答 励志 学校 网站地图
当前位置: 首页 > 教程频道 > 其他教程 > 互联网 >

oozie CDH4和hadoop CDH4整合出现的有关问题(一)

2013-10-10 
oozie CDH4和hadoop CDH4整合出现的问题(一)最近在使用oozie和hadoop进行开发hadoop版本:hadoop-2.0.0-cdh

oozie CDH4和hadoop CDH4整合出现的问题(一)
最近在使用oozie和hadoop进行开发

hadoop版本:hadoop-2.0.0-cdh4.3.0
oozie版本:oozie-3.3.2-cdh4.3.0

在使用oozie执行里面的例子时,
oozie在向hadoop提交任务时报出了以下错误:

Caused by: com.google.protobuf.ServiceException: java.net.ConnectException: Call From slave4/10.95.3.65 to 0.0.0.0:8030 failed on connection exception: java.net.ConnectException: 拒绝连接; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefusedat org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:212)at com.sun.proxy.$Proxy28.registerApplicationMaster(Unknown Source)at org.apache.hadoop.yarn.api.impl.pb.client.AMRMProtocolPBClientImpl.registerApplicationMaster(AMRMProtocolPBClientImpl.java:100)... 12 moreCaused by: java.net.ConnectException: Call From slave4/10.95.3.65 to 0.0.0.0:8030 failed on connection exception: java.net.ConnectException: 拒绝连接; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefusedat sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)at java.lang.reflect.Constructor.newInstance(Constructor.java:525)at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:782)at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:729)at org.apache.hadoop.ipc.Client.call(Client.java:1229)at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:202)... 14 moreCaused by: java.net.ConnectException: 拒绝连接at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:692)at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:207)at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:528)at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:492)at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:499)at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:593)at org.apache.hadoop.ipc.Client$Connection.access$2000(Client.java:241)at org.apache.hadoop.ipc.Client.getConnection(Client.java:1278)at org.apache.hadoop.ipc.Client.call(Client.java:1196)... 15 more2013-08-22 11:03:06,534 ERROR [main] org.apache.hadoop.yarn.service.CompositeService: Error starting services org.apache.hadoop.mapreduce.v2.app.MRAppMasterorg.apache.hadoop.yarn.YarnException: java.lang.reflect.UndeclaredThrowableExceptionat org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator.register(RMCommunicator.java:160)at org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator.start(RMCommunicator.java:111)at org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator.start(RMContainerAllocator.java:213)at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$ContainerAllocatorRouter.start(MRAppMaster.java:689)at org.apache.hadoop.yarn.service.CompositeService.start(CompositeService.java:68)at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.start(MRAppMaster.java:904)at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$1.run(MRAppMaster.java:1146)at java.security.AccessController.doPrivileged(Native Method)at javax.security.auth.Subject.doAs(Subject.java:415)at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRAppMaster.java:1142)at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1097)Caused by: java.lang.reflect.UndeclaredThrowableExceptionat org.apache.hadoop.yarn.exceptions.impl.pb.YarnRemoteExceptionPBImpl.unwrapAndThrowException(YarnRemoteExceptionPBImpl.java:135)at org.apache.hadoop.yarn.api.impl.pb.client.AMRMProtocolPBClientImpl.registerApplicationMaster(AMRMProtocolPBClientImpl.java:103)at org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator.register(RMCommunicator.java:147)... 11 moreCaused by: com.google.protobuf.ServiceException: java.net.ConnectException: Call From slave4/10.95.3.65 to 0.0.0.0:8030 failed on connection exception: java.net.ConnectException: 拒绝连接; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefusedat org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:212)at com.sun.proxy.$Proxy28.registerApplicationMaster(Unknown Source)at org.apache.hadoop.yarn.api.impl.pb.client.AMRMProtocolPBClientImpl.registerApplicationMaster(AMRMProtocolPBClientImpl.java:100)... 12 moreCaused by: java.net.ConnectException: Call From slave4/10.95.3.65 to 0.0.0.0:8030 failed on connection exception: java.net.ConnectException: 拒绝连接; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefusedat sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)at java.lang.reflect.Constructor.newInstance(Constructor.java:525)at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:782)at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:729)at org.apache.hadoop.ipc.Client.call(Client.java:1229)at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:202)... 14 moreCaused by: java.net.ConnectException: 拒绝连接at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:692)at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:207)at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:528)at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:492)at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:499)at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:593)at org.apache.hadoop.ipc.Client$Connection.access$2000(Client.java:241)at org.apache.hadoop.ipc.Client.getConnection(Client.java:1278)at org.apache.hadoop.ipc.Client.call(Client.java:1196)... 15 more


通过异常信息,可以猜测应该是在执行任务时,配置没有读到,所以使用了默认的0.0.0.0:8030地址。为了验证是否是这个原因造成的,我们可以修改oozie/conf目录下的log4j的日志级别为调试。
然后通过查看和跟踪hadoop的源代码,验证了猜想的正确性,由于oozie在提交job的时候,会读取oozie/conf/hadop-conf/core-site.xml中的配置,异常我们只要在这个配置文件中添加配置:
    <property>        <name>yarn.resourcemanager.scheduler.address</name>        <value>master:8030</value>    </property>


再重启oozie服务,上述异常消失。

热点排行