[jira] [Commented] (HIVE-2269) Hive --auxpath option can't handle multiple colon separated values
[ https://issues.apache.org/jira/browse/HIVE-2269?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13693139#comment-13693139 ] Josh Spiegel commented on HIVE-2269: No matter what the intent was, the case where HIVE_AUX_JARS_PATH is a list is broken. In the list case, the value of this variable is used in two contradictory ways: (1) It is colon appended to AUX_CLASSPATH, then CLASSPATH, then HADOOP_CLASSPATH (2) It is split on "," and appended to Hadoop's -libjars and hive.aux.jars.path If you use commas, (1) doesn't work. If you use colons, (2) doesn't work. I am not sure why user JARs need to go in HADOOP_CLASSPATH but I have noticed failures in local mode if one of the user JARs contains an InputFormat and it is not in HADOOP_CLASSPATH. I think this is probably due to a class caching/loading bug in Hive. But for whatever reason, HADOOP_CLASSPATH needs to be set if the jars contain a custom InputFormat. But otherwise, setting -libjars and hive.aux.jars.path seems to be sufficient. This is probably why more people haven't been impacted by this bug - for simple UDFs, using ',' works because the comma list in HADOOP_CLASSPATH is silently ignored. Consider a user with the following constraints: - Can not modify $HIVE_HOME (i.e. can not add jars to $HIVE_HOME/auxlib) - Has multiple JARs - The JARs contain at least one custom InputFormat In current releases, this user must pick one of the following options: (1) Add the list of JARs to HADOOP_CLASSPATH manually (with colons) and then set HIVE_AUX_JARS_PATH (or --auxpath) with commas. The comma list still gets appended to the end of HADOOP_CLASSPATH but it should be ignored by the JVM. (2) Copy all JARs to a single directory. Set HIVE_AUX_JARS_PATH (or --auxpath) to this single directory. Am I missing anything? Note, "add jars" is not an option for JARs with custom InputFormats as it does not change HADOOP_CLASSPATH. I should also mention that Carl's patch allows colons to be used successfully in --auxpath. His patch was picked up by or before CDH3u3 but dropped sometime later. So, there is a brief period where CDH users can successfully specify a list of JARs with just --auxpath. Finally, when defining the semantics of "--auxpath", it should support a list of local files (not URIs) that can be either comma or colon separated. The script should reformat the list depending on where it is used. Supporting just commas or just colons would not be backwards compatible. Supporting URIs would not be backwards compatible. Adding support for directories in lists may be OK. > Hive --auxpath option can't handle multiple colon separated values > -- > > Key: HIVE-2269 > URL: https://issues.apache.org/jira/browse/HIVE-2269 > Project: Hive > Issue Type: Bug > Components: CLI >Affects Versions: 0.7.0, 0.7.1 >Reporter: Carl Steinbach >Assignee: Carl Steinbach > Attachments: HIVE-2269-auxpath.1.patch.txt > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2269) Hive --auxpath option can't handle multiple colon separated values
[ https://issues.apache.org/jira/browse/HIVE-2269?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13061636#comment-13061636 ] Patrick Hunt commented on HIVE-2269: Hi. It looks like HIVE_AUX_JARS_PATH (hive --auxpath ...) can be either a directory (if so it considers all *.jar in the directory) or a list. In the case it's a list, can the entries be only jars or mixed jars & directories? I can't find any documentation on this, perhaps a JIRA should be opened to document the semantics of this parameter? > Hive --auxpath option can't handle multiple colon separated values > -- > > Key: HIVE-2269 > URL: https://issues.apache.org/jira/browse/HIVE-2269 > Project: Hive > Issue Type: Bug > Components: CLI >Affects Versions: 0.7.0, 0.7.1 >Reporter: Carl Steinbach >Assignee: Carl Steinbach > Attachments: HIVE-2269-auxpath.1.patch.txt > > -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2269) Hive --auxpath option can't handle multiple colon separated values
[
https://issues.apache.org/jira/browse/HIVE-2269?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13061520#comment-13061520
]
Edward Capriolo commented on HIVE-2269:
---
Also it seems that the AUX_PATH is being specified as URI's.
{noformat}
https://github.com/riptano/hive/wiki/Cassandra-Handler-usage-in-Hive-0.7-with-Cassandra-0.7
{noformat}
{noformat}
hive.aux.jars.path
file:///path/to/guava-r05.jar,file:///path/to/commons-lang-2.4.jar,file:///path/to/apache-cassandra-0.7.6.jar,file:///path/to/hive-cassandra-handler-0.7.0-beta1.jar
{noformat}
So splitting on ':' could break URI's?
> Hive --auxpath option can't handle multiple colon separated values
> --
>
> Key: HIVE-2269
> URL: https://issues.apache.org/jira/browse/HIVE-2269
> Project: Hive
> Issue Type: Bug
> Components: CLI
>Affects Versions: 0.7.0, 0.7.1
>Reporter: Carl Steinbach
>Assignee: Carl Steinbach
> Attachments: HIVE-2269-auxpath.1.patch.txt
>
>
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2269) Hive --auxpath option can't handle multiple colon separated values
[ https://issues.apache.org/jira/browse/HIVE-2269?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13061513#comment-13061513 ] Edward Capriolo commented on HIVE-2269: --- I am not sure this is documented anywhere. As a hadoop historian, I am guessing ',' was chosen to avoid the separator/path separator differences on different OS's. I do not see a problem supporting columns other then how it might effect the dedicated cygwin/hive users. > Hive --auxpath option can't handle multiple colon separated values > -- > > Key: HIVE-2269 > URL: https://issues.apache.org/jira/browse/HIVE-2269 > Project: Hive > Issue Type: Bug > Components: CLI >Affects Versions: 0.7.0, 0.7.1 >Reporter: Carl Steinbach >Assignee: Carl Steinbach > Attachments: HIVE-2269-auxpath.1.patch.txt > > -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2269) Hive --auxpath option can't handle multiple colon separated values
[ https://issues.apache.org/jira/browse/HIVE-2269?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13061455#comment-13061455 ] Carl Steinbach commented on HIVE-2269: -- bq. auxpath is 'comma' separate but the title says 'colon' @Ed: Where is this documented? Regardless, with this patch the value supplied to --auxpath can be either comma or colon separated. > Hive --auxpath option can't handle multiple colon separated values > -- > > Key: HIVE-2269 > URL: https://issues.apache.org/jira/browse/HIVE-2269 > Project: Hive > Issue Type: Bug > Components: CLI >Affects Versions: 0.7.0, 0.7.1 >Reporter: Carl Steinbach >Assignee: Carl Steinbach > Attachments: HIVE-2269-auxpath.1.patch.txt > > -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2269) Hive --auxpath option can't handle multiple colon separated values
[ https://issues.apache.org/jira/browse/HIVE-2269?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13061295#comment-13061295 ] Edward Capriolo commented on HIVE-2269: --- I am someone confused by the title, auxpath is 'comma' separate but the title says 'colon'. > Hive --auxpath option can't handle multiple colon separated values > -- > > Key: HIVE-2269 > URL: https://issues.apache.org/jira/browse/HIVE-2269 > Project: Hive > Issue Type: Bug > Components: CLI >Affects Versions: 0.7.0, 0.7.1 >Reporter: Carl Steinbach >Assignee: Carl Steinbach > Attachments: HIVE-2269-auxpath.1.patch.txt > > -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2269) Hive --auxpath option can't handle multiple colon separated values
[
https://issues.apache.org/jira/browse/HIVE-2269?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13061018#comment-13061018
]
Carl Steinbach commented on HIVE-2269:
--
{code}
% hive --auxpath
$HIVE_HOME/lib/hive-hbase-handler-0.8.0-SNAPSHOT.jar:$HIVE_HOME/lib/hive-contrib-0.8.0-SNAPSHOT.jar
Logging initialized using configuration in
jar:file:/Users/carl/Work/repos/hive3/build/dist/lib/hive-common-0.8.0-SNAPSHOT.jar!/hive-log4j.properties
Hive history file=/tmp/carl/hive_job_log_carl_201107062017_907171289.txt
hive> SELECT * FROM tmp WHERE x > 1;
SELECT * FROM tmp WHERE x > 1;
Total MapReduce jobs = 1
Launching Job 1 out of 1
Number of reduce tasks is set to 0 since there's no reduce operator
Execution log at:
/tmp/carl/carl_20110706201717_5294c2b7-1cdc-40ab-9aeb-d1811eb8c486.log
java.io.FileNotFoundException: File
/Users/carl/Work/repos/hive3/build/dist/lib/hive-hbase-handler-0.8.0-SNAPSHOT.jar:/Users/carl/Work/repos/hive3/build/dist/lib/hive-contrib-0.8.0-SNAPSHOT.jar
does not exist.
at
org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:361)
at
org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:245)
at
org.apache.hadoop.filecache.DistributedCache.getTimestamp(DistributedCache.java:509)
at
org.apache.hadoop.mapred.JobClient.configureCommandLineOptions(JobClient.java:629)
at
org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:761)
at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:730)
at
org.apache.hadoop.hive.ql.exec.ExecDriver.execute(ExecDriver.java:409)
at org.apache.hadoop.hive.ql.exec.ExecDriver.main(ExecDriver.java:663)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
Job Submission failed with exception 'java.io.FileNotFoundException(File
/Users/carl/Work/repos/hive3/build/dist/lib/hive-hbase-handler-0.8.0-SNAPSHOT.jar:/Users/carl/Work/repos/hive3/build/dist/lib/hive-contrib-0.8.0-SNAPSHOT.jar
does not exist.)'
FAILED: Execution Error, return code 2 from
org.apache.hadoop.hive.ql.exec.MapRedTask
hive>
{code}
> Hive --auxpath option can't handle multiple colon separated values
> --
>
> Key: HIVE-2269
> URL: https://issues.apache.org/jira/browse/HIVE-2269
> Project: Hive
> Issue Type: Bug
> Components: CLI
>Affects Versions: 0.7.0, 0.7.1
>Reporter: Carl Steinbach
>Assignee: Carl Steinbach
>
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
