[
https://issues.apache.org/jira/browse/PIG-4939?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siddhi Mehta updated PIG-4939:
------------------------------
Description:
QueryParserUtils.setHdfsServers(QueryParserUtils.java:104) should not be called
for non-dfs methods
{code}
String buildLoadOp(SourceLocation loc, String alias, String filename, FuncSpec
funcSpec, LogicalSchema schema)
throws ParserValidationException {
String absolutePath;
LoadFunc loFunc;
try {
// Load LoadFunc class from default properties if funcSpec is null.
Fallback on PigStorage if LoadFunc is not specified in properties.
funcSpec = funcSpec == null ? new
FuncSpec(pigContext.getProperties().getProperty(PigConfiguration.PIG_DEFAULT_LOAD_FUNC,
PigStorage.class.getName())) : funcSpec;
loFunc = (LoadFunc)PigContext.instantiateFuncFromSpec(funcSpec);
......
.......
if (absolutePath == null) {
absolutePath = loFunc.relativeToAbsolutePath( filename,
QueryParserUtils.getCurrentDir( pigContext ) );
if (absolutePath!=null) {
QueryParserUtils.setHdfsServers( absolutePath, pigContext );
}
.....
}
{code}
Email Conversation
On Mon, Jun 27, 2016 at 1:04 PM, Prashant Kommireddi <[email protected]>
wrote:
Agreed. This method call isn't needed for phoenix loader (or any such
non-direct-fs loaders). You should allow a config to handle it.
On Mon, Jun 27, 2016 at 12:14 PM, Siddhi Mehta <[email protected]> wrote:
> Hello All,
>
> I am getting a URISyntaxException when I try to execute my pig script using
> PHoenixHBaseLoader. Traced attached below.
> Looking through the code Pig splits multiple paths provided to it based on
> comma(',') and during the query parsing step
> QueryParserUtils.setHdfsServers(absolutePath, pigContext) tried to split
> paths based on comma(',') and create URI's/PATHS for the same.
>
> Certain loaders like 'PhoenixHBaseLoader' donot pass hdfs locations and
> instead work with passing PhoenixQueryStatement in the location.
> e.g.
> *A = load 'hbase://query/SELECT ID,NAME,DATE FROM HIRES WHERE DATE >
> TO_DATE('1990-12-21 05:55:00.000')*
>
> This locations needs not be parsed to get hdfsservers path from them.
> Does it make sense to introduce a config/loader property to annotate if the
> loader/store is dealing with hdfs locations and based on the property make
> a function call to QueryParserUtils.setHdfsServers(absolutePath,
> pigContext).
>
> *Thoughts?*
>
> ***** Stack trace *****
>
> Caused by: Failed to parse: Pig script failed to parse:
> <line 1, column 23> pig script failed to validate:
> java.lang.IllegalArgumentException: java.net.URISyntaxException: Relative
> path in absolute URI: CREATED_DATE FROM HIRES WHERE
> CREATED_DATE>=TO_DATE('1990-12-21
> 05:55:00.000')%20AND%20CREATED_DATE%3CTO_DATE('2016-03-08%2008:00:00.000')
> at
> org.apache.pig.parser.QueryParserDriver.parse(QueryParserDriver.java:199)
> at org.apache.pig.PigServer$Graph.parseQuery(PigServer.java:1712)
> ... 30 more
> Caused by:
> <line 1, column 23> pig script failed to validate:
> java.lang.IllegalArgumentException: java.net.URISyntaxException: Relative
> path in absolute URI: CREATED_DATE FROM HIRES WHERE
> CREATED_DATE>=TO_DATE('1990-12-21
> 05:55:00.000')%20AND%20CREATED_DATE%3CTO_DATE('2016-03-08%2008:00:00.000')
> at
>
> org.apache.pig.parser.LogicalPlanBuilder.buildLoadOp(LogicalPlanBuilder.java:897)
> at
>
> org.apache.pig.parser.LogicalPlanGenerator.load_clause(LogicalPlanGenerator.java:3568)
> at
>
> org.apache.pig.parser.LogicalPlanGenerator.op_clause(LogicalPlanGenerator.java:1625)
> at
>
> org.apache.pig.parser.LogicalPlanGenerator.general_statement(LogicalPlanGenerator.java:1102)
> at
>
> org.apache.pig.parser.LogicalPlanGenerator.statement(LogicalPlanGenerator.java:560)
> at
>
> org.apache.pig.parser.LogicalPlanGenerator.query(LogicalPlanGenerator.java:421)
> at
> org.apache.pig.parser.QueryParserDriver.parse(QueryParserDriver.java:191)
> ... 31 more
> Caused by: java.lang.IllegalArgumentException: java.net.URISyntaxException:
> Relative path in absolute URI: CREATED_DATE FROM HIRES WHERE
> CREATED_DATE>=TO_DATE('1990-12-21
> 05:55:00.000')%20AND%20CREATED_DATE%3CTO_DATE('2016-03-08%2008:00:00.000')
> at org.apache.hadoop.fs.Path.initialize(Path.java:206)
> at org.apache.hadoop.fs.Path.<init>(Path.java:172)
> at
>
> org.apache.pig.parser.QueryParserUtils.getRemoteHosts(QueryParserUtils.java:138)
> at
>
> org.apache.pig.parser.QueryParserUtils.setHdfsServers(QueryParserUtils.java:104)
> at
>
> org.apache.pig.parser.LogicalPlanBuilder.buildLoadOp(LogicalPlanBuilder.java:892)
> ... 37 more
> Caused by: java.net.URISyntaxException: Relative path in absolute URI:
> CREATED_DATE FROM HIRES WHERE CREATED_DATE>=TO_DATE('1990-12-21
> 05:55:00.000')%20AND%20CREATED_DATE%3CTO_DATE('2016-03-08%2008:00:00.000')
> at java.net.URI.checkPath(URI.java:1823)
> at java.net.URI.<init>(URI.java:745)
> at org.apache.hadoop.fs.Path.initialize(Path.java:203)
> ... 41 more
>
> Thanks,
> Siddhi
>
was:
On Mon, Jun 27, 2016 at 1:04 PM, Prashant Kommireddi <[email protected]>
wrote:
Agreed. This method call isn't needed for phoenix loader (or any such
non-direct-fs loaders). You should allow a config to handle it.
On Mon, Jun 27, 2016 at 12:14 PM, Siddhi Mehta <[email protected]> wrote:
> Hello All,
>
> I am getting a URISyntaxException when I try to execute my pig script using
> PHoenixHBaseLoader. Traced attached below.
> Looking through the code Pig splits multiple paths provided to it based on
> comma(',') and during the query parsing step
> QueryParserUtils.setHdfsServers(absolutePath, pigContext) tried to split
> paths based on comma(',') and create URI's/PATHS for the same.
>
> Certain loaders like 'PhoenixHBaseLoader' donot pass hdfs locations and
> instead work with passing PhoenixQueryStatement in the location.
> e.g.
> *A = load 'hbase://query/SELECT ID,NAME,DATE FROM HIRES WHERE DATE >
> TO_DATE('1990-12-21 05:55:00.000')*
>
> This locations needs not be parsed to get hdfsservers path from them.
> Does it make sense to introduce a config/loader property to annotate if the
> loader/store is dealing with hdfs locations and based on the property make
> a function call to QueryParserUtils.setHdfsServers(absolutePath,
> pigContext).
>
> *Thoughts?*
>
> ***** Stack trace *****
>
> Caused by: Failed to parse: Pig script failed to parse:
> <line 1, column 23> pig script failed to validate:
> java.lang.IllegalArgumentException: java.net.URISyntaxException: Relative
> path in absolute URI: CREATED_DATE FROM HIRES WHERE
> CREATED_DATE>=TO_DATE('1990-12-21
> 05:55:00.000')%20AND%20CREATED_DATE%3CTO_DATE('2016-03-08%2008:00:00.000')
> at
> org.apache.pig.parser.QueryParserDriver.parse(QueryParserDriver.java:199)
> at org.apache.pig.PigServer$Graph.parseQuery(PigServer.java:1712)
> ... 30 more
> Caused by:
> <line 1, column 23> pig script failed to validate:
> java.lang.IllegalArgumentException: java.net.URISyntaxException: Relative
> path in absolute URI: CREATED_DATE FROM HIRES WHERE
> CREATED_DATE>=TO_DATE('1990-12-21
> 05:55:00.000')%20AND%20CREATED_DATE%3CTO_DATE('2016-03-08%2008:00:00.000')
> at
>
> org.apache.pig.parser.LogicalPlanBuilder.buildLoadOp(LogicalPlanBuilder.java:897)
> at
>
> org.apache.pig.parser.LogicalPlanGenerator.load_clause(LogicalPlanGenerator.java:3568)
> at
>
> org.apache.pig.parser.LogicalPlanGenerator.op_clause(LogicalPlanGenerator.java:1625)
> at
>
> org.apache.pig.parser.LogicalPlanGenerator.general_statement(LogicalPlanGenerator.java:1102)
> at
>
> org.apache.pig.parser.LogicalPlanGenerator.statement(LogicalPlanGenerator.java:560)
> at
>
> org.apache.pig.parser.LogicalPlanGenerator.query(LogicalPlanGenerator.java:421)
> at
> org.apache.pig.parser.QueryParserDriver.parse(QueryParserDriver.java:191)
> ... 31 more
> Caused by: java.lang.IllegalArgumentException: java.net.URISyntaxException:
> Relative path in absolute URI: CREATED_DATE FROM HIRES WHERE
> CREATED_DATE>=TO_DATE('1990-12-21
> 05:55:00.000')%20AND%20CREATED_DATE%3CTO_DATE('2016-03-08%2008:00:00.000')
> at org.apache.hadoop.fs.Path.initialize(Path.java:206)
> at org.apache.hadoop.fs.Path.<init>(Path.java:172)
> at
>
> org.apache.pig.parser.QueryParserUtils.getRemoteHosts(QueryParserUtils.java:138)
> at
>
> org.apache.pig.parser.QueryParserUtils.setHdfsServers(QueryParserUtils.java:104)
> at
>
> org.apache.pig.parser.LogicalPlanBuilder.buildLoadOp(LogicalPlanBuilder.java:892)
> ... 37 more
> Caused by: java.net.URISyntaxException: Relative path in absolute URI:
> CREATED_DATE FROM HIRES WHERE CREATED_DATE>=TO_DATE('1990-12-21
> 05:55:00.000')%20AND%20CREATED_DATE%3CTO_DATE('2016-03-08%2008:00:00.000')
> at java.net.URI.checkPath(URI.java:1823)
> at java.net.URI.<init>(URI.java:745)
> at org.apache.hadoop.fs.Path.initialize(Path.java:203)
> ... 41 more
>
> Thanks,
> Siddhi
>
> QueryParserUtils.setHdfsServers(QueryParserUtils.java:104) should not be
> called for non-dfs methods
> ---------------------------------------------------------------------------------------------------
>
> Key: PIG-4939
> URL: https://issues.apache.org/jira/browse/PIG-4939
> Project: Pig
> Issue Type: Improvement
> Components: impl
> Reporter: Siddhi Mehta
> Priority: Minor
>
> QueryParserUtils.setHdfsServers(QueryParserUtils.java:104) should not be
> called for non-dfs methods
> {code}
> String buildLoadOp(SourceLocation loc, String alias, String filename,
> FuncSpec funcSpec, LogicalSchema schema)
> throws ParserValidationException {
> String absolutePath;
> LoadFunc loFunc;
> try {
> // Load LoadFunc class from default properties if funcSpec is
> null. Fallback on PigStorage if LoadFunc is not specified in properties.
> funcSpec = funcSpec == null ? new
> FuncSpec(pigContext.getProperties().getProperty(PigConfiguration.PIG_DEFAULT_LOAD_FUNC,
> PigStorage.class.getName())) : funcSpec;
> loFunc = (LoadFunc)PigContext.instantiateFuncFromSpec(funcSpec);
> ......
> .......
> if (absolutePath == null) {
> absolutePath = loFunc.relativeToAbsolutePath( filename,
> QueryParserUtils.getCurrentDir( pigContext ) );
> if (absolutePath!=null) {
> QueryParserUtils.setHdfsServers( absolutePath, pigContext
> );
> }
> .....
> }
> {code}
> Email Conversation
> On Mon, Jun 27, 2016 at 1:04 PM, Prashant Kommireddi <[email protected]>
> wrote:
> Agreed. This method call isn't needed for phoenix loader (or any such
> non-direct-fs loaders). You should allow a config to handle it.
> On Mon, Jun 27, 2016 at 12:14 PM, Siddhi Mehta <[email protected]> wrote:
> > Hello All,
> >
> > I am getting a URISyntaxException when I try to execute my pig script using
> > PHoenixHBaseLoader. Traced attached below.
> > Looking through the code Pig splits multiple paths provided to it based on
> > comma(',') and during the query parsing step
> > QueryParserUtils.setHdfsServers(absolutePath, pigContext) tried to split
> > paths based on comma(',') and create URI's/PATHS for the same.
> >
> > Certain loaders like 'PhoenixHBaseLoader' donot pass hdfs locations and
> > instead work with passing PhoenixQueryStatement in the location.
> > e.g.
> > *A = load 'hbase://query/SELECT ID,NAME,DATE FROM HIRES WHERE DATE >
> > TO_DATE('1990-12-21 05:55:00.000')*
> >
> > This locations needs not be parsed to get hdfsservers path from them.
> > Does it make sense to introduce a config/loader property to annotate if the
> > loader/store is dealing with hdfs locations and based on the property make
> > a function call to QueryParserUtils.setHdfsServers(absolutePath,
> > pigContext).
> >
> > *Thoughts?*
> >
> > ***** Stack trace *****
> >
> > Caused by: Failed to parse: Pig script failed to parse:
> > <line 1, column 23> pig script failed to validate:
> > java.lang.IllegalArgumentException: java.net.URISyntaxException: Relative
> > path in absolute URI: CREATED_DATE FROM HIRES WHERE
> > CREATED_DATE>=TO_DATE('1990-12-21
> > 05:55:00.000')%20AND%20CREATED_DATE%3CTO_DATE('2016-03-08%2008:00:00.000')
> > at
> > org.apache.pig.parser.QueryParserDriver.parse(QueryParserDriver.java:199)
> > at org.apache.pig.PigServer$Graph.parseQuery(PigServer.java:1712)
> > ... 30 more
> > Caused by:
> > <line 1, column 23> pig script failed to validate:
> > java.lang.IllegalArgumentException: java.net.URISyntaxException: Relative
> > path in absolute URI: CREATED_DATE FROM HIRES WHERE
> > CREATED_DATE>=TO_DATE('1990-12-21
> > 05:55:00.000')%20AND%20CREATED_DATE%3CTO_DATE('2016-03-08%2008:00:00.000')
> > at
> >
> > org.apache.pig.parser.LogicalPlanBuilder.buildLoadOp(LogicalPlanBuilder.java:897)
> > at
> >
> > org.apache.pig.parser.LogicalPlanGenerator.load_clause(LogicalPlanGenerator.java:3568)
> > at
> >
> > org.apache.pig.parser.LogicalPlanGenerator.op_clause(LogicalPlanGenerator.java:1625)
> > at
> >
> > org.apache.pig.parser.LogicalPlanGenerator.general_statement(LogicalPlanGenerator.java:1102)
> > at
> >
> > org.apache.pig.parser.LogicalPlanGenerator.statement(LogicalPlanGenerator.java:560)
> > at
> >
> > org.apache.pig.parser.LogicalPlanGenerator.query(LogicalPlanGenerator.java:421)
> > at
> > org.apache.pig.parser.QueryParserDriver.parse(QueryParserDriver.java:191)
> > ... 31 more
> > Caused by: java.lang.IllegalArgumentException: java.net.URISyntaxException:
> > Relative path in absolute URI: CREATED_DATE FROM HIRES WHERE
> > CREATED_DATE>=TO_DATE('1990-12-21
> > 05:55:00.000')%20AND%20CREATED_DATE%3CTO_DATE('2016-03-08%2008:00:00.000')
> > at org.apache.hadoop.fs.Path.initialize(Path.java:206)
> > at org.apache.hadoop.fs.Path.<init>(Path.java:172)
> > at
> >
> > org.apache.pig.parser.QueryParserUtils.getRemoteHosts(QueryParserUtils.java:138)
> > at
> >
> > org.apache.pig.parser.QueryParserUtils.setHdfsServers(QueryParserUtils.java:104)
> > at
> >
> > org.apache.pig.parser.LogicalPlanBuilder.buildLoadOp(LogicalPlanBuilder.java:892)
> > ... 37 more
> > Caused by: java.net.URISyntaxException: Relative path in absolute URI:
> > CREATED_DATE FROM HIRES WHERE CREATED_DATE>=TO_DATE('1990-12-21
> > 05:55:00.000')%20AND%20CREATED_DATE%3CTO_DATE('2016-03-08%2008:00:00.000')
> > at java.net.URI.checkPath(URI.java:1823)
> > at java.net.URI.<init>(URI.java:745)
> > at org.apache.hadoop.fs.Path.initialize(Path.java:203)
> > ... 41 more
> >
> > Thanks,
> > Siddhi
> >
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
