[ https://issues.apache.org/jira/browse/YARN-7433?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jason Lowe updated YARN-7433: ----------------------------- Environment: (was: From centos6.5 upgrade centos7, hadoop version(2.7.1) is compiled on centos6.5 and Support snappy, the copy runs on centos7. but yarn task error ,yarn task errorlog: (hadoop Native Libraries Whether to recompile based on centos7???) Error: java.lang.RuntimeException: native snappy library not available: this version of libhadoop was built without snappy support. at org.apache.hadoop.io.compress.SnappyCodec.checkNativeCodeLoaded(SnappyCodec.java:65) at org.apache.hadoop.io.compress.SnappyCodec.getCompressorType(SnappyCodec.java:134) at org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:150) at org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:165) at org.apache.hadoop.mapred.IFile$Writer.<init>(IFile.java:114) at org.apache.hadoop.mapred.IFile$Writer.<init>(IFile.java:97) at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.mergeParts(MapTask.java:1856) at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1511) at org.apache.hadoop.mapred.MapTask$NewOutputCollector.close(MapTask.java:723) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:793) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158) Error: java.io.IOException: Spill failed) Description: >From centos6.5 upgrade centos7, hadoop version(2.7.1) is compiled on >centos6.5 and Support snappy, the copy runs on centos7. but yarn task error ,yarn task errorlog: (hadoop Native Libraries Whether to recompile based on centos7???) Error: java.lang.RuntimeException: native snappy library not available: this version of libhadoop was built without snappy support. at org.apache.hadoop.io.compress.SnappyCodec.checkNativeCodeLoaded(SnappyCodec.java:65) at org.apache.hadoop.io.compress.SnappyCodec.getCompressorType(SnappyCodec.java:134) at org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:150) at org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:165) at org.apache.hadoop.mapred.IFile$Writer.<init>(IFile.java:114) at org.apache.hadoop.mapred.IFile$Writer.<init>(IFile.java:97) at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.mergeParts(MapTask.java:1856) at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1511) at org.apache.hadoop.mapred.MapTask$NewOutputCollector.close(MapTask.java:723) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:793) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158) Error: java.io.IOException: Spill failed JIRA is for reporting bugs or tracking features rather than general user support. The [mailing lists|http://hadoop.apache.org/mailing_lists.html] would be more appropriate for this. There are two possible reasons why this error is occurring. Either the task cannot find libhadoop.so or libhadoop.so was compiled without Snappy support. For the former, see if the message "Unable to load native-hadoop library for your platform" appears in the task logs. If it does then the task is not loading libhadoop.so and Snappy cannot work. Check the LD_LIBRARY_PATH environment variable setting to make sure the task can locate libhadoop.so. If it is loading libhadoop.so properly then the error means the native library was compiled without Snappy support and needs to be rebuilt with -Drequire.snappy=true > java.lang.RuntimeException: native snappy library not available: this version > of libhadoop was built without snappy support. > ---------------------------------------------------------------------------------------------------------------------------- > > Key: YARN-7433 > URL: https://issues.apache.org/jira/browse/YARN-7433 > Project: Hadoop YARN > Issue Type: Bug > Reporter: gehaijiang > > From centos6.5 upgrade centos7, hadoop version(2.7.1) is compiled on > centos6.5 and Support snappy, the copy runs on centos7. > but yarn task error ,yarn task errorlog: > (hadoop Native Libraries Whether to recompile based on centos7???) > Error: java.lang.RuntimeException: native snappy library not available: this > version of libhadoop was built without snappy support. > at > org.apache.hadoop.io.compress.SnappyCodec.checkNativeCodeLoaded(SnappyCodec.java:65) > at > org.apache.hadoop.io.compress.SnappyCodec.getCompressorType(SnappyCodec.java:134) > at > org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:150) > at > org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:165) > at org.apache.hadoop.mapred.IFile$Writer.<init>(IFile.java:114) > at org.apache.hadoop.mapred.IFile$Writer.<init>(IFile.java:97) > at > org.apache.hadoop.mapred.MapTask$MapOutputBuffer.mergeParts(MapTask.java:1856) > at > org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1511) > at > org.apache.hadoop.mapred.MapTask$NewOutputCollector.close(MapTask.java:723) > at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:793) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341) > at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657) > at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158) > Error: java.io.IOException: Spill failed -- This message was sent by Atlassian JIRA (v6.4.14#64029) --------------------------------------------------------------------- To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org