[jira] [Commented] (HADOOP-8899) Classpath exceeds maximum OS limit
[ https://issues.apache.org/jira/browse/HADOOP-8899?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13478665#comment-13478665 ] Alejandro Abdelnur commented on HADOOP-8899: Suresh, my bad, I was missing the MANIFEST.MF *Class-path:* trick. Now it makes sense, no objection. I have not look at the patch but conceptually seems OK. Just make sure all paths to JARs listed in the *Class-path:* are relative to the location of the manifest only JAR and there are not ../ in the paths. In other words, all the JARs mentioned in the manifest only JAR must be in the same directory or in a subdirectory from where the manifest only JAR is. Thx > Classpath exceeds maximum OS limit > -- > > Key: HADOOP-8899 > URL: https://issues.apache.org/jira/browse/HADOOP-8899 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 1-win >Reporter: Ahmed El Baz > Attachments: HADOOP-8899.branch-1-win.classpath.patch > > > We hit errors in TT due to long classpath value. One example is Oozie trying > to start a hive job, and it fails with the following error: > java.io.IOException: Command exceeds the OS command length limit: 8192, > command: "set CLASSPATH="…. > The classpath includes Hadoop Core + Hive Jars which are in the distributed > cache. This is causing the classpath to be too long giving the error above. A > viable long term fix is to generate a temporary JAR file in the task cache > directory which includes all elements in the long classpath, and then provide > this as the "-classpath" argument for the JVM to be spawned. This is gated > for Windows only. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8899) Classpath exceeds maximum OS limit
[ https://issues.apache.org/jira/browse/HADOOP-8899?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13478656#comment-13478656 ] Suresh Srinivas commented on HADOOP-8899: - Alejandro, this is manifest only jar. See similar things done in maven surefire - http://maven.apache.org/plugins/maven-surefire-plugin/examples/class-loading.html. bq. then things will not work as expected as you'll lose some of those files in the consolidation I do not understand given above comment. All you have done is create a jar with manifest file that has classpath. What do mean by things will not work as expected? > Classpath exceeds maximum OS limit > -- > > Key: HADOOP-8899 > URL: https://issues.apache.org/jira/browse/HADOOP-8899 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 1-win >Reporter: Ahmed El Baz > Attachments: HADOOP-8899.branch-1-win.classpath.patch > > > We hit errors in TT due to long classpath value. One example is Oozie trying > to start a hive job, and it fails with the following error: > java.io.IOException: Command exceeds the OS command length limit: 8192, > command: "set CLASSPATH="…. > The classpath includes Hadoop Core + Hive Jars which are in the distributed > cache. This is causing the classpath to be too long giving the error above. A > viable long term fix is to generate a temporary JAR file in the task cache > directory which includes all elements in the long classpath, and then provide > this as the "-classpath" argument for the JVM to be spawned. This is gated > for Windows only. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8899) Classpath exceeds maximum OS limit
[ https://issues.apache.org/jira/browse/HADOOP-8899?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13478197#comment-13478197 ] Alejandro Abdelnur commented on HADOOP-8899: I think the approach is not correct, reiterating on my prev comment. if more than one of the JARs being consolidated contains a META-INF service definition for the same service (ie o.a.h.fs.FileSystem, o.a.h.io.compress.CompressionCodec, o.a.h.security.SecurityInfo), then things will not work as expected as you'll lose some of those files in the consolidation. > Classpath exceeds maximum OS limit > -- > > Key: HADOOP-8899 > URL: https://issues.apache.org/jira/browse/HADOOP-8899 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 1-win >Reporter: Ahmed El Baz > Attachments: HADOOP-8899.branch-1-win.classpath.patch > > > We hit errors in TT due to long classpath value. One example is Oozie trying > to start a hive job, and it fails with the following error: > java.io.IOException: Command exceeds the OS command length limit: 8192, > command: "set CLASSPATH="…. > The classpath includes Hadoop Core + Hive Jars which are in the distributed > cache. This is causing the classpath to be too long giving the error above. A > viable long term fix is to generate a temporary JAR file in the task cache > directory which includes all elements in the long classpath, and then provide > this as the "-classpath" argument for the JVM to be spawned. This is gated > for Windows only. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8899) Classpath exceeds maximum OS limit
[ https://issues.apache.org/jira/browse/HADOOP-8899?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13478069#comment-13478069 ] Suresh Srinivas commented on HADOOP-8899: - Alejandro and Joep, please look at Ahmed's comments. I am +1 for this change given maven surefire. I will commit this in a day or so, if there are no further comments. Comments post that can be addressed in a separate jira. > Classpath exceeds maximum OS limit > -- > > Key: HADOOP-8899 > URL: https://issues.apache.org/jira/browse/HADOOP-8899 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 1-win >Reporter: Ahmed El Baz > Attachments: HADOOP-8899.branch-1-win.classpath.patch > > > We hit errors in TT due to long classpath value. One example is Oozie trying > to start a hive job, and it fails with the following error: > java.io.IOException: Command exceeds the OS command length limit: 8192, > command: "set CLASSPATH="…. > The classpath includes Hadoop Core + Hive Jars which are in the distributed > cache. This is causing the classpath to be too long giving the error above. A > viable long term fix is to generate a temporary JAR file in the task cache > directory which includes all elements in the long classpath, and then provide > this as the "-classpath" argument for the JVM to be spawned. This is gated > for Windows only. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8899) Classpath exceeds maximum OS limit
[
https://issues.apache.org/jira/browse/HADOOP-8899?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13472718#comment-13472718
]
Hadoop QA commented on HADOOP-8899:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12548456/HADOOP-8899.branch-1-win.classpath.patch
against trunk revision .
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output:
https://builds.apache.org/job/PreCommit-HADOOP-Build/1591//console
This message is automatically generated.
> Classpath exceeds maximum OS limit
> --
>
> Key: HADOOP-8899
> URL: https://issues.apache.org/jira/browse/HADOOP-8899
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 1-win
>Reporter: Ahmed El Baz
> Attachments: HADOOP-8899.branch-1-win.classpath.patch
>
>
> We hit errors in TT due to long classpath value. One example is Oozie trying
> to start a hive job, and it fails with the following error:
> java.io.IOException: Command exceeds the OS command length limit: 8192,
> command: "set CLASSPATH="….
> The classpath includes Hadoop Core + Hive Jars which are in the distributed
> cache. This is causing the classpath to be too long giving the error above. A
> viable long term fix is to generate a temporary JAR file in the task cache
> directory which includes all elements in the long classpath, and then provide
> this as the "-classpath" argument for the JVM to be spawned. This is gated
> for Windows only.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8899) Classpath exceeds maximum OS limit
[ https://issues.apache.org/jira/browse/HADOOP-8899?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13472571#comment-13472571 ] Alejandro Abdelnur commented on HADOOP-8899: Argh, forgot about that. Maybe in win distributed cache symlinks should be simulated with copies. Then the proposed approach would work Alejandro > Classpath exceeds maximum OS limit > -- > > Key: HADOOP-8899 > URL: https://issues.apache.org/jira/browse/HADOOP-8899 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 1-win >Reporter: Ahmed El Baz > > We hit errors in TT due to long classpath value. One example is Oozie trying > to start a hive job, and it fails with the following error: > java.io.IOException: Command exceeds the OS command length limit: 8192, > command: "set CLASSPATH="…. > The classpath includes Hadoop Core + Hive Jars which are in the distributed > cache. This is causing the classpath to be too long giving the error above. A > viable long term fix is to generate a temporary JAR file in the task cache > directory which includes all elements in the long classpath, and then provide > this as the "-classpath" argument for the JVM to be spawned. This is gated > for Windows only. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8899) Classpath exceeds maximum OS limit
[ https://issues.apache.org/jira/browse/HADOOP-8899?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13472529#comment-13472529 ] Joep Rottinghuis commented on HADOOP-8899: -- Admittedly I have not kept up with this lately, but aren't symlinks (MKLINK) limited to NTFS? Does that interopt well with Cygwin? Not sure if that is truly a restriction anymore these days... > Classpath exceeds maximum OS limit > -- > > Key: HADOOP-8899 > URL: https://issues.apache.org/jira/browse/HADOOP-8899 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 1-win >Reporter: Ahmed El Baz > > We hit errors in TT due to long classpath value. One example is Oozie trying > to start a hive job, and it fails with the following error: > java.io.IOException: Command exceeds the OS command length limit: 8192, > command: "set CLASSPATH="…. > The classpath includes Hadoop Core + Hive Jars which are in the distributed > cache. This is causing the classpath to be too long giving the error above. A > viable long term fix is to generate a temporary JAR file in the task cache > directory which includes all elements in the long classpath, and then provide > this as the "-classpath" argument for the JVM to be spawned. This is gated > for Windows only. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8899) Classpath exceeds maximum OS limit
[ https://issues.apache.org/jira/browse/HADOOP-8899?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13472507#comment-13472507 ] Alejandro Abdelnur commented on HADOOP-8899: The potential downside of my suggested approach is that you lose the capability to order JARs in the classpath as '*' will be resolved in system specific way (I've seen this being done differently in different unix versions); which in my opinion should not be an issue as you should not have duplicate JARs in your classpath to start. The other downside is that you won't, from the logs, know exactly what is in the classpath, you'll only see a 'distributedcache-lib/*' entry in the classpath. > Classpath exceeds maximum OS limit > -- > > Key: HADOOP-8899 > URL: https://issues.apache.org/jira/browse/HADOOP-8899 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 1-win >Reporter: Ahmed El Baz > > We hit errors in TT due to long classpath value. One example is Oozie trying > to start a hive job, and it fails with the following error: > java.io.IOException: Command exceeds the OS command length limit: 8192, > command: "set CLASSPATH="…. > The classpath includes Hadoop Core + Hive Jars which are in the distributed > cache. This is causing the classpath to be too long giving the error above. A > viable long term fix is to generate a temporary JAR file in the task cache > directory which includes all elements in the long classpath, and then provide > this as the "-classpath" argument for the JVM to be spawned. This is gated > for Windows only. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8899) Classpath exceeds maximum OS limit
[ https://issues.apache.org/jira/browse/HADOOP-8899?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13472503#comment-13472503 ] Alejandro Abdelnur commented on HADOOP-8899: You don't want to unjar and jar, as you'll lose in the process any JDK services definitions in the JARs. I'd suggest the following alternate approach: all JARs in the DistributedCache classpath should be symlink-ed to a distributedcache-lib/ dir in the current directory and then 'distributedcached-lib/*' should be added to the classpath. > Classpath exceeds maximum OS limit > -- > > Key: HADOOP-8899 > URL: https://issues.apache.org/jira/browse/HADOOP-8899 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 1-win >Reporter: Ahmed El Baz > > We hit errors in TT due to long classpath value. One example is Oozie trying > to start a hive job, and it fails with the following error: > java.io.IOException: Command exceeds the OS command length limit: 8192, > command: "set CLASSPATH="…. > The classpath includes Hadoop Core + Hive Jars which are in the distributed > cache. This is causing the classpath to be too long giving the error above. A > viable long term fix is to generate a temporary JAR file in the task cache > directory which includes all elements in the long classpath, and then provide > this as the "-classpath" argument for the JVM to be spawned. This is gated > for Windows only. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8899) Classpath exceeds maximum OS limit
[ https://issues.apache.org/jira/browse/HADOOP-8899?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13471855#comment-13471855 ] Joep Rottinghuis commented on HADOOP-8899: -- IIRC this is not an OS limit, but this is a limit of cmd.exe Wouldn't jarring and unjarring end up being slow? (suppose better slow than not working). You'd also have to make sure that the classpath order is preserved. Classloaders will generally load the first hit, so if you create your own jar you need to either drop duplicates, or make sure they are in order. With signed jars duplicate entries can be a challenge. I haven't looked at the code, but can this be avoided through the user of an environment variable instead of expanding the classpath on the command-line? Along similar lines I recall switching from System.exec to the user of Procbuilder for similar problems. Not sure if relevant here. > Classpath exceeds maximum OS limit > -- > > Key: HADOOP-8899 > URL: https://issues.apache.org/jira/browse/HADOOP-8899 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 1-win >Reporter: Ahmed El Baz > > We hit errors in TT due to long classpath value. One example is Oozie trying > to start a hive job, and it fails with the following error: > java.io.IOException: Command exceeds the OS command length limit: 8192, > command: "set CLASSPATH="…. > The classpath includes Hadoop Core + Hive Jars which are in the distributed > cache. This is causing the classpath to be too long giving the error above. A > viable long term fix is to generate a temporary JAR file in the task cache > directory which includes all elements in the long classpath, and then provide > this as the "-classpath" argument for the JVM to be spawned. This is gated > for Windows only. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
