[
https://issues.apache.org/jira/browse/MAPREDUCE-4072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13244693#comment-13244693
]
Anupam Seth commented on MAPREDUCE-4072:
----------------------------------------
One reason we might want to go for a doc change instead of stripping out and
copying the java.library.path set by the user to LD_LIBRARY_PATH is because it
is cleaner to set it right in the first place and we likely don't want the
overhead of supporting such in the long run.
Quoting Kihwal to help clarify:
"If only LD_LIBRARY_PATH is specified, the JVM (at least's Oracle's) resorts to
the dynamic linker to search for the library. If java.library.path is set, the
JVM will only look at the paths in java.library.path. Since the search
capability of the linker is superior, we prefer using the former method
whenever possible. The JVM resource loading stops searching when it locates
one match, whether the match is for the correct architecture or not. This can
be troublesome when supporting multiple architectures, which is a goal of yarn.
LD_LIBRARY_PATH is also needed if there is a chain of dependencies on native
libraries from a JNI library and those are placed in a non-standard
directory(i.e. non-system dirs and not defined in /etc/ld.so.conf.d/*)."
> User set java.library.path seems to overwrite default creating problems
> native lib loading
> ------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-4072
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4072
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Reporter: Anupam Seth
> Assignee: Anupam Seth
> Attachments: MAPREDUCE-4072-branch-23.patch,
> MAPREDUCE-4072-branch-23_documentation.patch
>
>
> This was found by Peeyush Bishnoi.
> While running a distributed cache example with Hadoop-0.23,
> tasks are failing as follows:
> ------------------------------------------------------------------------------------------------------------
> Exception from container-launch:
> org.apache.hadoop.util.Shell$ExitCodeException: at
> org.apache.hadoop.util.Shell.runCommand(Shell.java:261) at
> org.apache.hadoop.util.Shell.run(Shell.java:188) at
> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:381) at
> org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.launchContainer(LinuxContainerExecutor.java:207)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:241)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:68)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at
> java.util.concurrent.FutureTask.run(FutureTask.java:138) at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619) main : command provided 1 main : user
> is <user>
> ------------------------------------------------------------------------------------------------------------
> Same Pig script and command work successfully on 0.20
> See this in the stderr:
> Exception in thread "main" java.lang.ExceptionInInitializerError
> at java.lang.Class.forName0(Native Method)
> at java.lang.Class.forName(Class.java:247)
> at
> org.apache.hadoop.conf.Configuration.getClassByNameOrNull(Configuration.java:1179)
> at
> org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1149)
> at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1238)
> at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1264)
> at org.apache.hadoop.security.Groups.(Groups.java:54)
> at
> org.apache.hadoop.security.Groups.getUserToGroupsMappingService(Groups.java:178)
> at
> org.apache.hadoop.security.UserGroupInformation.initUGI(UserGroupInformation.java:252)
> at
> org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:223)
> at
> org.apache.hadoop.security.UserGroupInformation.setConfiguration(UserGroupInformation.java:265)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:75)
> Caused by: java.lang.RuntimeException: Bailing out since native library
> couldn't be loaded
> at
> org.apache.hadoop.security.JniBasedUnixGroupsMapping.(JniBasedUnixGroupsMapping.java:48)
> ... 12 more
> Pig command:
> $ pig -Dmapred.job.queue.name=<queue> -Dmapred.cache.archives=<archives>
> -Dmapred.child.java.opts="-Djava.library.path=./ygeo/lib
> -Dip2geo.preLoadLibraries=<some other libs>" -Djava.io.tmpdir=/grid/0/tmp
> -Dmapred.create.symlink=yes -Dmapred.job.map.memory.mb=3072 piggeoscript.pig
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
