求助:nutch1.2运行crawl出现ioexception
问题描述:运行crawl后出现异常。
环境描述:windowsXP/cygwin, nutch1.2, jdk1.6。
配置描述:crawl-urlfilters.txt配置+^http://www.163.com/,
nutch-site.xml配置
<configuration>
<property>
<name>http.agent.name</name>
<value>ubuntuer</value>
<description></description>
</property>
<property>
<name>http.agent.description</name>
<value>ubuntuer</value>
<description></description>
</property>
<property>
<name>http.agent.url</name>
<value></value>
<description></description>
</property>
<property>
<name>http.agent.email</name>
<value>ztjxw123@hotmail.com</value>
<description></description>
</property>
</configuration>
异常描述:
运行/bin/nutch crawl urls -dir crawl -depth 3显示
Exception in thread "main" java.io.IOException: Job failed!
at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1252)
at org.apache.nutch.crawl.Injector.inject(Injector.java:217)
at org.apache.nutch.crawl.Crawl.main(Crawl.java:124)
hadoop.log显示:
2011-09-04 14:01:33,171 WARN regex.RegexURLNormalizer - can't find rules for scope 'inject', using default
2011-09-04 14:01:33,265 WARN mapred.LocalJobRunner - job_local_0001
java.io.IOException: Expecting a line not the end of stream
at org.apache.hadoop.fs.DF.parseExecResult(DF.java:109)
at org.apache.hadoop.util.Shell.runCommand(Shell.java:179)
at org.apache.hadoop.util.Shell.run(Shell.java:134)
at org.apache.hadoop.fs.DF.getAvailable(DF.java:73)
at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:329)
at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:124)
at org.apache.hadoop.mapred.MapOutputFile.getSpillFileForWrite(MapOutputFile.java:107)
at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:1221)
at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1129)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:359)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:307)
at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:177)
[解决办法]
恭喜楼主哈~~~一模一样的问题~~~~WIN7的~~~~痛苦的要死 我弄两天了~~~有空可以交流下454274992~~~我去试下0.9
[解决办法]
我也是报这个错误。我用的也是1.2 我QQ114377413 能加我,说下是什么原因吗?
[解决办法]
这个我也遇到了 设置下cygwin的字符集环境
export "zh_CN.GBK"
执行下这个后就可以了
[解决办法]
只要打export LANG=zn_utf8就行了