[jira] [Commented] (PIG-1865) BinStorage/PigStorageSchema cannot load data from a different namenode
[ https://issues.apache.org/jira/browse/PIG-1865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13024927#comment-13024927 ] Richard Ding commented on PIG-1865: --- +1 > BinStorage/PigStorageSchema cannot load data from a different namenode > -- > > Key: PIG-1865 > URL: https://issues.apache.org/jira/browse/PIG-1865 > Project: Pig > Issue Type: Bug >Affects Versions: 0.7.0, 0.8.0, 0.9.0 >Reporter: Vivek Padmanabhan >Assignee: Daniel Dai > Fix For: 0.9.0 > > Attachments: PIG-1865-1.patch > > > BinStorage/PigStorageSchema cannot load data from a different namenode. The > main reason for this is that, in the getSchema method , they use > org.apache.pig.impl.io.FileLocalizer to check whether the exists, but the > filesystem in HDataStorage refers to the natively configured dfs. > The test case is simple : > a = load 'hdfs:///input' using BinStorage(); > dump a; > Here if I specify -Dmapreduce.job.hdfs-servers, it should have worked , by > pig still takes the fs from fs.default.name so to make it work i had to > override fs.default.name in pig command line. > Raising this as a bug since the same scenario works with PigStorage. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-1865) BinStorage/PigStorageSchema cannot load data from a different namenode
[ https://issues.apache.org/jira/browse/PIG-1865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13024849#comment-13024849 ] jirapos...@reviews.apache.org commented on PIG-1865: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/658/ --- Review request for pig. Summary --- See PIG-1865 This addresses bug PIG-1865. https://issues.apache.org/jira/browse/PIG-1865 Diffs - http://svn.apache.org/repos/asf/pig/trunk/src/org/apache/pig/builtin/BinStorage.java 1095143 Diff: https://reviews.apache.org/r/658/diff Testing --- Test-patch: [exec] -1 overall. [exec] [exec] +1 @author. The patch does not contain any @author tags. [exec] [exec] -1 tests included. The patch doesn't appear to include any new or modified tests. [exec] Please justify why no tests are needed for this patch. [exec] [exec] +1 javadoc. The javadoc tool did not generate any warning messages. [exec] [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings. [exec] [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings. [exec] [exec] +1 release audit. The applied patch does not increase the total number of release audit warnings. There is no test case added. It requires two clusters and is hard to do in unit test. Unit-test: all pass Manual-test: Tested using two clusters and BinStorage can access remote hdfs Thanks, Daniel > BinStorage/PigStorageSchema cannot load data from a different namenode > -- > > Key: PIG-1865 > URL: https://issues.apache.org/jira/browse/PIG-1865 > Project: Pig > Issue Type: Bug >Affects Versions: 0.7.0, 0.8.0, 0.9.0 >Reporter: Vivek Padmanabhan >Assignee: Daniel Dai > Fix For: 0.9.0 > > Attachments: PIG-1865-1.patch > > > BinStorage/PigStorageSchema cannot load data from a different namenode. The > main reason for this is that, in the getSchema method , they use > org.apache.pig.impl.io.FileLocalizer to check whether the exists, but the > filesystem in HDataStorage refers to the natively configured dfs. > The test case is simple : > a = load 'hdfs:///input' using BinStorage(); > dump a; > Here if I specify -Dmapreduce.job.hdfs-servers, it should have worked , by > pig still takes the fs from fs.default.name so to make it work i had to > override fs.default.name in pig command line. > Raising this as a bug since the same scenario works with PigStorage. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-1865) BinStorage/PigStorageSchema cannot load data from a different namenode
[ https://issues.apache.org/jira/browse/PIG-1865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13022550#comment-13022550 ] Daniel Dai commented on PIG-1865: - There is no test case added. It requires two clusters and is hard to do in unit test. Manually tested and it works. > BinStorage/PigStorageSchema cannot load data from a different namenode > -- > > Key: PIG-1865 > URL: https://issues.apache.org/jira/browse/PIG-1865 > Project: Pig > Issue Type: Bug >Affects Versions: 0.7.0, 0.8.0, 0.9.0 >Reporter: Vivek Padmanabhan >Assignee: Daniel Dai > Fix For: 0.9.0 > > Attachments: PIG-1865-1.patch > > > BinStorage/PigStorageSchema cannot load data from a different namenode. The > main reason for this is that, in the getSchema method , they use > org.apache.pig.impl.io.FileLocalizer to check whether the exists, but the > filesystem in HDataStorage refers to the natively configured dfs. > The test case is simple : > a = load 'hdfs:///input' using BinStorage(); > dump a; > Here if I specify -Dmapreduce.job.hdfs-servers, it should have worked , by > pig still takes the fs from fs.default.name so to make it work i had to > override fs.default.name in pig command line. > Raising this as a bug since the same scenario works with PigStorage. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
