[
https://issues.apache.org/jira/browse/PIG-3886?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Cheolsoo Park updated PIG-3886:
-------------------------------
Attachment: PIG-3886-1.patch
Attaching a patch that fixes UdfDistributedCache_1.
But this patch isn't complete because the following fails in Tez while it works
in MR-
{code}
set pig.streaming.cache.files 's3n://foo/bar/zoo';
{code}
Currently, {{pig.streaming.cache.files}} only works with hdfs paths in Tez, but
it should also work with other file systems such as s3.
Here is the stack trace that I get for a s3 path-
{code}
Caused by: java.io.FileNotFoundException: File does not exist: /bar/zoo
at
org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1128)
at
org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1120)
at
org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at
org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1120)
at
org.apache.pig.backend.hadoop.executionengine.tez.TezResourceManager.getTezResources(TezResourceManager.java:138)
{code}
> UdfDistributedCache_1 fails in tez branch
> -----------------------------------------
>
> Key: PIG-3886
> URL: https://issues.apache.org/jira/browse/PIG-3886
> Project: Pig
> Issue Type: Sub-task
> Components: tez
> Affects Versions: tez-branch
> Reporter: Cheolsoo Park
> Assignee: Cheolsoo Park
> Fix For: 0.14.0
>
> Attachments: PIG-3886-1.patch
>
>
> The e2e test "UdfDistributedCache_1" fails with the following error-
> {code}
> : Caused by: java.io.FileNotFoundException: ./foodle (No
> such file or directory)
> : at java.io.FileInputStream.open(Native Method)
> : at
> java.io.FileInputStream.<init>(FileInputStream.java:146)
> : at
> java.io.FileInputStream.<init>(FileInputStream.java:101)
> : at java.io.FileReader.<init>(FileReader.java:58)
> : at
> org.apache.pig.test.udf.evalfunc.Udfcachetest.exec(Udfcachetest.java:60)
> : at
> org.apache.pig.test.udf.evalfunc.Udfcachetest.exec(Udfcachetest.java:40)
> : at
> org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOperators.POUserFunc.getNext(POUserFunc.java:353)
> : ... 14 more
> {code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
