[jira] [Commented] (PIG-2957) TetsScriptUDF fail due to volume prefix in jar
[
https://issues.apache.org/jira/browse/PIG-2957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13535249#comment-13535249
]
Julien Le Dem commented on PIG-2957:
could you call the method something more explicit than "cleanupPath". Something
like "getPathForJar" maybe?
Also add comments to explain what exactly this is doing:
{noformat}
if (path.charAt(1)==':') {
newPath = path.charAt(0) + path.substring(2);
}
{noformat}
It would be useful to describe what it is changing in the path and why.
In particular the drive letter becomes a root dir in the jar (C:/foo becomes
C/foo). If that's what we want then it should be clearer.
> TetsScriptUDF fail due to volume prefix in jar
> --
>
> Key: PIG-2957
> URL: https://issues.apache.org/jira/browse/PIG-2957
> Project: Pig
> Issue Type: Sub-task
>Reporter: Daniel Dai
>Assignee: Daniel Dai
> Fix For: 0.11
>
> Attachments: PIG-2957-1.patch, PIG-2957-2_0.10.patch, PIG-2957-2.patch
>
>
> testPythonAbsolutePath fail. Stack is:
> java.io.IOException: Mkdirs failed to create
> C:\tmp\hadoop-Administrator\mapred\local\1_0\taskTracker\Administrator\jobcache\job_20120725074728013_0011\jars\C:\Users\Administrator\pig-monarch
> at org.apache.hadoop.util.RunJar.unJar(RunJar.java:47)
> at
> org.apache.hadoop.mapred.JobLocalizer.localizeJobJarFile(JobLocalizer.java:277)
> at
> org.apache.hadoop.mapred.JobLocalizer.localizeJobFiles(JobLocalizer.java:377)
> at
> org.apache.hadoop.mapred.JobLocalizer.localizeJobFiles(JobLocalizer.java:367)
> at
> org.apache.hadoop.mapred.DefaultTaskController.initializeJob(DefaultTaskController.java:214)
> at org.apache.hadoop.mapred.TaskTracker$4.run(TaskTracker.java:1237)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1107)
> at org.apache.hadoop.mapred.TaskTracker.initializeJob(TaskTracker.java:1212)
> at org.apache.hadoop.mapred.TaskTracker.localizeJob(TaskTracker.java:1127)
> at org.apache.hadoop.mapred.TaskTracker$5.run(TaskTracker.java:2417)
> at java.lang.Thread.run(Thread.java:662)
> The reason is we pack the volume prefix into the job.jar.
> jar tvf C:\Users\ADMINI~1\AppData\Local\Temp\Job6350
> 669482684441868.jar|grep testPythonAbsolutePath
> 98 Wed Jul 25 11:12:58 PDT 2012 C:\Users\Administrator\pig-monarch\testPytho
> nAbsolutePath.py
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-2957) TetsScriptUDF fail due to volume prefix in jar
[ https://issues.apache.org/jira/browse/PIG-2957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13481027#comment-13481027 ] Daniel Dai commented on PIG-2957: - Sure, all I need is to encode the path to remove those not acceptable by Windows. I didn't find an existing function to achieve this, but I can definitely put this logic into a Util function to make the main code clear. > TetsScriptUDF fail due to volume prefix in jar > -- > > Key: PIG-2957 > URL: https://issues.apache.org/jira/browse/PIG-2957 > Project: Pig > Issue Type: Sub-task >Reporter: Daniel Dai >Assignee: Daniel Dai > Fix For: 0.11 > > Attachments: PIG-2957-1.patch > > > testPythonAbsolutePath fail. Stack is: > java.io.IOException: Mkdirs failed to create > C:\tmp\hadoop-Administrator\mapred\local\1_0\taskTracker\Administrator\jobcache\job_20120725074728013_0011\jars\C:\Users\Administrator\pig-monarch > at org.apache.hadoop.util.RunJar.unJar(RunJar.java:47) > at > org.apache.hadoop.mapred.JobLocalizer.localizeJobJarFile(JobLocalizer.java:277) > at > org.apache.hadoop.mapred.JobLocalizer.localizeJobFiles(JobLocalizer.java:377) > at > org.apache.hadoop.mapred.JobLocalizer.localizeJobFiles(JobLocalizer.java:367) > at > org.apache.hadoop.mapred.DefaultTaskController.initializeJob(DefaultTaskController.java:214) > at org.apache.hadoop.mapred.TaskTracker$4.run(TaskTracker.java:1237) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:396) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1107) > at org.apache.hadoop.mapred.TaskTracker.initializeJob(TaskTracker.java:1212) > at org.apache.hadoop.mapred.TaskTracker.localizeJob(TaskTracker.java:1127) > at org.apache.hadoop.mapred.TaskTracker$5.run(TaskTracker.java:2417) > at java.lang.Thread.run(Thread.java:662) > The reason is we pack the volume prefix into the job.jar. > jar tvf C:\Users\ADMINI~1\AppData\Local\Temp\Job6350 > 669482684441868.jar|grep testPythonAbsolutePath > 98 Wed Jul 25 11:12:58 PDT 2012 C:\Users\Administrator\pig-monarch\testPytho > nAbsolutePath.py -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-2957) TetsScriptUDF fail due to volume prefix in jar
[ https://issues.apache.org/jira/browse/PIG-2957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13478549#comment-13478549 ] Jonathan Coveney commented on PIG-2957: --- Is there no cross platform way to achieve this that doesn't involve directly checking the string? > TetsScriptUDF fail due to volume prefix in jar > -- > > Key: PIG-2957 > URL: https://issues.apache.org/jira/browse/PIG-2957 > Project: Pig > Issue Type: Sub-task >Reporter: Daniel Dai >Assignee: Daniel Dai > Fix For: 0.11 > > Attachments: PIG-2957-1.patch > > > testPythonAbsolutePath fail. Stack is: > java.io.IOException: Mkdirs failed to create > C:\tmp\hadoop-Administrator\mapred\local\1_0\taskTracker\Administrator\jobcache\job_20120725074728013_0011\jars\C:\Users\Administrator\pig-monarch > at org.apache.hadoop.util.RunJar.unJar(RunJar.java:47) > at > org.apache.hadoop.mapred.JobLocalizer.localizeJobJarFile(JobLocalizer.java:277) > at > org.apache.hadoop.mapred.JobLocalizer.localizeJobFiles(JobLocalizer.java:377) > at > org.apache.hadoop.mapred.JobLocalizer.localizeJobFiles(JobLocalizer.java:367) > at > org.apache.hadoop.mapred.DefaultTaskController.initializeJob(DefaultTaskController.java:214) > at org.apache.hadoop.mapred.TaskTracker$4.run(TaskTracker.java:1237) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:396) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1107) > at org.apache.hadoop.mapred.TaskTracker.initializeJob(TaskTracker.java:1212) > at org.apache.hadoop.mapred.TaskTracker.localizeJob(TaskTracker.java:1127) > at org.apache.hadoop.mapred.TaskTracker$5.run(TaskTracker.java:2417) > at java.lang.Thread.run(Thread.java:662) > The reason is we pack the volume prefix into the job.jar. > jar tvf C:\Users\ADMINI~1\AppData\Local\Temp\Job6350 > 669482684441868.jar|grep testPythonAbsolutePath > 98 Wed Jul 25 11:12:58 PDT 2012 C:\Users\Administrator\pig-monarch\testPytho > nAbsolutePath.py -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-2957) TetsScriptUDF fail due to volume prefix in jar
[ https://issues.apache.org/jira/browse/PIG-2957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13472838#comment-13472838 ] Dmitriy V. Ryaboy commented on PIG-2957: is it safe to assume we are guaranteed to be on windows, and that lopping the volume off is safe, just by checking "scriptPath.charAt(1)==':'" ? Would it be better to isolate logic for figuring out if we are in windows behind some function? > TetsScriptUDF fail due to volume prefix in jar > -- > > Key: PIG-2957 > URL: https://issues.apache.org/jira/browse/PIG-2957 > Project: Pig > Issue Type: Sub-task >Reporter: Daniel Dai >Assignee: Daniel Dai > Fix For: 0.11 > > Attachments: PIG-2957-1.patch > > > testPythonAbsolutePath fail. Stack is: > java.io.IOException: Mkdirs failed to create > C:\tmp\hadoop-Administrator\mapred\local\1_0\taskTracker\Administrator\jobcache\job_20120725074728013_0011\jars\C:\Users\Administrator\pig-monarch > at org.apache.hadoop.util.RunJar.unJar(RunJar.java:47) > at > org.apache.hadoop.mapred.JobLocalizer.localizeJobJarFile(JobLocalizer.java:277) > at > org.apache.hadoop.mapred.JobLocalizer.localizeJobFiles(JobLocalizer.java:377) > at > org.apache.hadoop.mapred.JobLocalizer.localizeJobFiles(JobLocalizer.java:367) > at > org.apache.hadoop.mapred.DefaultTaskController.initializeJob(DefaultTaskController.java:214) > at org.apache.hadoop.mapred.TaskTracker$4.run(TaskTracker.java:1237) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:396) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1107) > at org.apache.hadoop.mapred.TaskTracker.initializeJob(TaskTracker.java:1212) > at org.apache.hadoop.mapred.TaskTracker.localizeJob(TaskTracker.java:1127) > at org.apache.hadoop.mapred.TaskTracker$5.run(TaskTracker.java:2417) > at java.lang.Thread.run(Thread.java:662) > The reason is we pack the volume prefix into the job.jar. > jar tvf C:\Users\ADMINI~1\AppData\Local\Temp\Job6350 > 669482684441868.jar|grep testPythonAbsolutePath > 98 Wed Jul 25 11:12:58 PDT 2012 C:\Users\Administrator\pig-monarch\testPytho > nAbsolutePath.py -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
