[jira] [Created] (HIVE-3595) Hive should adapt new FsShell commands since Hadoop 2 has changed FsShell argument structures
Harsh J created HIVE-3595: - Summary: Hive should adapt new FsShell commands since Hadoop 2 has changed FsShell argument structures Key: HIVE-3595 URL: https://issues.apache.org/jira/browse/HIVE-3595 Project: Hive Issue Type: Improvement Components: Shims Affects Versions: 0.9.0 Reporter: Harsh J Priority: Minor A simple example is that hive calls -rmr in the FsShell class, which in Hadoop 2 is rm -r. This helps avoid printing an unnecessary Deprecated warning in Hive when the Hadoop23 (or hadoop-2) shim is in use. We should wrap the logic and call the right commands of hadoop-2 to avoid this. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3595) Hive should adapt new FsShell commands since Hadoop 2 has changed FsShell argument structures
[
https://issues.apache.org/jira/browse/HIVE-3595?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13478842#comment-13478842
]
Harsh J commented on HIVE-3595:
---
A quick scan suggests we just need to fix the one instance of rmr in use at
Hive#replaceFiles:
{code}
// point of no return -- delete oldPath
if (oldPath != null) {
try {
FileSystem fs2 = oldPath.getFileSystem(conf);
if (fs2.exists(oldPath)) {
// use FsShell to move data to .Trash first rather than delete
permanently
FsShell fshell = new FsShell();
fshell.setConf(conf);
fshell.run(new String[]{-rmr, oldPath.toString()});
}
} catch (Exception e) {
//swallow the exception
LOG.warn(Directory + oldPath.toString() + canot be removed.);
}
}
{code}
If we can wrap that -rmr to use -rmr only for 0.23 hadoop versions, this
can be closed. For higher versions the logic ought to use -rm -r.
Hive should adapt new FsShell commands since Hadoop 2 has changed FsShell
argument structures
-
Key: HIVE-3595
URL: https://issues.apache.org/jira/browse/HIVE-3595
Project: Hive
Issue Type: Improvement
Components: Shims
Affects Versions: 0.9.0
Reporter: Harsh J
Priority: Minor
A simple example is that hive calls -rmr in the FsShell class, which in
Hadoop 2 is rm -r. This helps avoid printing an unnecessary Deprecated
warning in Hive when the Hadoop23 (or hadoop-2) shim is in use.
We should wrap the logic and call the right commands of hadoop-2 to avoid
this.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HIVE-3596) Regression - HiveConf static variable causes issues in long running JVM instances with /tmp/ data
Chris McConnell created HIVE-3596: - Summary: Regression - HiveConf static variable causes issues in long running JVM instances with /tmp/ data Key: HIVE-3596 URL: https://issues.apache.org/jira/browse/HIVE-3596 Project: Hive Issue Type: Bug Components: Configuration Affects Versions: 0.9.0, 0.8.1, 0.8.0 Reporter: Chris McConnell With Hive 0.8.x, HiveConf was changed to utilize the private, static member confVarURL which points to /tmp/hive-user-tmp_number.xml for job configuration settings. During long running JVMs, such as a Beeswax server, which creates multiple HiveConf objects over time this variable does not properly get updated between jobs and can cause job failure if the OS cleans /tmp/ during a cron job. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-3596) Regression - HiveConf static variable causes issues in long running JVM instances with /tmp/ data
[ https://issues.apache.org/jira/browse/HIVE-3596?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris McConnell updated HIVE-3596: -- Attachment: HIVE-3596.patch Patch to remove static attached. Regression - HiveConf static variable causes issues in long running JVM instances with /tmp/ data - Key: HIVE-3596 URL: https://issues.apache.org/jira/browse/HIVE-3596 Project: Hive Issue Type: Bug Components: Configuration Affects Versions: 0.8.0, 0.8.1, 0.9.0 Reporter: Chris McConnell Attachments: HIVE-3596.patch With Hive 0.8.x, HiveConf was changed to utilize the private, static member confVarURL which points to /tmp/hive-user-tmp_number.xml for job configuration settings. During long running JVMs, such as a Beeswax server, which creates multiple HiveConf objects over time this variable does not properly get updated between jobs and can cause job failure if the OS cleans /tmp/ during a cron job. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-3596) Regression - HiveConf static variable causes issues in long running JVM instances with /tmp/ data
[ https://issues.apache.org/jira/browse/HIVE-3596?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris McConnell updated HIVE-3596: -- Attachment: HIVE-3596.patch New patch, added comments for change as well. Regression - HiveConf static variable causes issues in long running JVM instances with /tmp/ data - Key: HIVE-3596 URL: https://issues.apache.org/jira/browse/HIVE-3596 Project: Hive Issue Type: Bug Components: Configuration Affects Versions: 0.8.0, 0.8.1, 0.9.0 Reporter: Chris McConnell Attachments: HIVE-3596.patch With Hive 0.8.x, HiveConf was changed to utilize the private, static member confVarURL which points to /tmp/hive-user-tmp_number.xml for job configuration settings. During long running JVMs, such as a Beeswax server, which creates multiple HiveConf objects over time this variable does not properly get updated between jobs and can cause job failure if the OS cleans /tmp/ during a cron job. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
Build failed in Jenkins: Hive-0.9.1-SNAPSHOT-h0.21 #171
See https://builds.apache.org/job/Hive-0.9.1-SNAPSHOT-h0.21/171/ -- [...truncated 5823 lines...] [ivy:resolve] local: tried [ivy:resolve] /home/jenkins/.ivy2/local/log4j/log4j/1.2.15/ivys/ivy.xml [ivy:resolve] -- artifact log4j#log4j;1.2.15!log4j.jar: [ivy:resolve] /home/jenkins/.ivy2/local/log4j/log4j/1.2.15/jars/log4j.jar [ivy:resolve] apache-snapshot: tried [ivy:resolve] https://repository.apache.org/content/repositories/snapshots/log4j/log4j/1.2.15/log4j-1.2.15.pom [ivy:resolve] -- artifact log4j#log4j;1.2.15!log4j.jar: [ivy:resolve] https://repository.apache.org/content/repositories/snapshots/log4j/log4j/1.2.15/log4j-1.2.15.jar [ivy:resolve] maven2: tried [ivy:resolve] http://repo1.maven.org/maven2/log4j/log4j/1.2.15/log4j-1.2.15.pom [ivy:resolve] datanucleus-repo: tried [ivy:resolve] -- artifact log4j#log4j;1.2.15!log4j.jar: [ivy:resolve] http://www.datanucleus.org/downloads/maven2/log4j/log4j/1.2.15/log4j-1.2.15.jar [ivy:resolve] hadoop-source: tried [ivy:resolve] -- artifact log4j#log4j;1.2.15!log4j.jar: [ivy:resolve] http://mirror.facebook.net/facebook/hive-deps/hadoop/core/log4j-1.2.15/log4j-1.2.15.jar [ivy:resolve] hadoop-source2: tried [ivy:resolve] -- artifact log4j#log4j;1.2.15!log4j.jar: [ivy:resolve] http://archive.cloudera.com/hive-deps/hadoop/core/log4j-1.2.15/log4j-1.2.15.jar [ivy:resolve] problem while downloading module descriptor: http://repo1.maven.org/maven2/commons-lang/commons-lang/2.5/commons-lang-2.5.pom: /home/jenkins/.ivy2/cache/commons-lang/commons-lang/ivy-2.5.xml.original.part (No such file or directory) (37ms) [ivy:resolve] module not found: commons-lang#commons-lang;2.5 [ivy:resolve] local: tried [ivy:resolve] /home/jenkins/.ivy2/local/commons-lang/commons-lang/2.5/ivys/ivy.xml [ivy:resolve] -- artifact commons-lang#commons-lang;2.5!commons-lang.jar: [ivy:resolve] /home/jenkins/.ivy2/local/commons-lang/commons-lang/2.5/jars/commons-lang.jar [ivy:resolve] apache-snapshot: tried [ivy:resolve] https://repository.apache.org/content/repositories/snapshots/commons-lang/commons-lang/2.5/commons-lang-2.5.pom [ivy:resolve] -- artifact commons-lang#commons-lang;2.5!commons-lang.jar: [ivy:resolve] https://repository.apache.org/content/repositories/snapshots/commons-lang/commons-lang/2.5/commons-lang-2.5.jar [ivy:resolve] maven2: tried [ivy:resolve] http://repo1.maven.org/maven2/commons-lang/commons-lang/2.5/commons-lang-2.5.pom [ivy:resolve] datanucleus-repo: tried [ivy:resolve] -- artifact commons-lang#commons-lang;2.5!commons-lang.jar: [ivy:resolve] http://www.datanucleus.org/downloads/maven2/commons-lang/commons-lang/2.5/commons-lang-2.5.jar [ivy:resolve] hadoop-source: tried [ivy:resolve] -- artifact commons-lang#commons-lang;2.5!commons-lang.jar: [ivy:resolve] http://mirror.facebook.net/facebook/hive-deps/hadoop/core/commons-lang-2.5/commons-lang-2.5.jar [ivy:resolve] hadoop-source2: tried [ivy:resolve] -- artifact commons-lang#commons-lang;2.5!commons-lang.jar: [ivy:resolve] http://archive.cloudera.com/hive-deps/hadoop/core/commons-lang-2.5/commons-lang-2.5.jar [ivy:resolve] problem while downloading module descriptor: http://repo1.maven.org/maven2/org/slf4j/slf4j-api/1.6.1/slf4j-api-1.6.1.pom: /home/jenkins/.ivy2/cache/org.slf4j/slf4j-api/ivy-1.6.1.xml.original.part (No such file or directory) (62ms) [ivy:resolve] module not found: org.slf4j#slf4j-api;1.6.1 [ivy:resolve] local: tried [ivy:resolve] /home/jenkins/.ivy2/local/org.slf4j/slf4j-api/1.6.1/ivys/ivy.xml [ivy:resolve] -- artifact org.slf4j#slf4j-api;1.6.1!slf4j-api.jar: [ivy:resolve] /home/jenkins/.ivy2/local/org.slf4j/slf4j-api/1.6.1/jars/slf4j-api.jar [ivy:resolve] apache-snapshot: tried [ivy:resolve] https://repository.apache.org/content/repositories/snapshots/org/slf4j/slf4j-api/1.6.1/slf4j-api-1.6.1.pom [ivy:resolve] -- artifact org.slf4j#slf4j-api;1.6.1!slf4j-api.jar: [ivy:resolve] https://repository.apache.org/content/repositories/snapshots/org/slf4j/slf4j-api/1.6.1/slf4j-api-1.6.1.jar [ivy:resolve] maven2: tried [ivy:resolve] http://repo1.maven.org/maven2/org/slf4j/slf4j-api/1.6.1/slf4j-api-1.6.1.pom [ivy:resolve] datanucleus-repo: tried [ivy:resolve] -- artifact org.slf4j#slf4j-api;1.6.1!slf4j-api.jar: [ivy:resolve] http://www.datanucleus.org/downloads/maven2/org/slf4j/slf4j-api/1.6.1/slf4j-api-1.6.1.jar [ivy:resolve] hadoop-source: tried [ivy:resolve] -- artifact org.slf4j#slf4j-api;1.6.1!slf4j-api.jar: [ivy:resolve] http://mirror.facebook.net/facebook/hive-deps/hadoop/core/slf4j-api-1.6.1/slf4j-api-1.6.1.jar [ivy:resolve] hadoop-source2: tried [ivy:resolve] -- artifact org.slf4j#slf4j-api;1.6.1!slf4j-api.jar: [ivy:resolve]
Build failed in Jenkins: Hive-0.9.1-SNAPSHOT-h0.21-keepgoing=false #171
See https://builds.apache.org/job/Hive-0.9.1-SNAPSHOT-h0.21-keepgoing=false/171/ -- [...truncated 5746 lines...] [ivy:resolve] (184kB) [ivy:resolve] .. (0kB) [ivy:resolve] [SUCCESSFUL ] commons-beanutils#commons-beanutils;1.7.0!commons-beanutils.jar (84ms) [ivy:resolve] downloading http://repo1.maven.org/maven2/org/mortbay/jetty/servlet-api/2.5-20081211/servlet-api-2.5-20081211.jar ... [ivy:resolve] (130kB) [ivy:resolve] .. (0kB) [ivy:resolve] [SUCCESSFUL ] org.mortbay.jetty#servlet-api;2.5-20081211!servlet-api.jar (199ms) [ivy:resolve] downloading http://repo1.maven.org/maven2/org/codehaus/jackson/jackson-core-asl/1.0.1/jackson-core-asl-1.0.1.jar ... [ivy:resolve] . (132kB) [ivy:resolve] .. (0kB) [ivy:resolve] [SUCCESSFUL ] org.codehaus.jackson#jackson-core-asl;1.0.1!jackson-core-asl.jar (110ms) ivy-retrieve-hadoop-shim: [echo] Project: shims [javac] Compiling 13 source files to https://builds.apache.org/job/Hive-0.9.1-SNAPSHOT-h0.21-keepgoing=false/171/artifact/hive/build/shims/classes [javac] Note: https://builds.apache.org/job/Hive-0.9.1-SNAPSHOT-h0.21-keepgoing=false/ws/hive/shims/src/common-secure/java/org/apache/hadoop/hive/shims/HadoopShimsSecure.java uses or overrides a deprecated API. [javac] Note: Recompile with -Xlint:deprecation for details. [javac] Note: https://builds.apache.org/job/Hive-0.9.1-SNAPSHOT-h0.21-keepgoing=false/ws/hive/shims/src/common-secure/java/org/apache/hadoop/hive/shims/HadoopShimsSecure.java uses unchecked or unsafe operations. [javac] Note: Recompile with -Xlint:unchecked for details. [echo] Building shims 0.23 build_shims: [echo] Project: shims [echo] Compiling https://builds.apache.org/job/Hive-0.9.1-SNAPSHOT-h0.21-keepgoing=false/ws/hive/shims/src/common/java;/home/jenkins/jenkins-slave/workspace/Hive-0.9.1-SNAPSHOT-h0.21-keepgoing=false/hive/shims/src/common-secure/java;/home/jenkins/jenkins-slave/workspace/Hive-0.9.1-SNAPSHOT-h0.21-keepgoing=false/hive/shims/src/0.23/java against hadoop 0.23.1 (https://builds.apache.org/job/Hive-0.9.1-SNAPSHOT-h0.21-keepgoing=false/171/artifact/hive/build/hadoopcore/hadoop-0.23.1) ivy-init-settings: [echo] Project: shims ivy-resolve-hadoop-shim: [echo] Project: shims [ivy:resolve] :: loading settings :: file = https://builds.apache.org/job/Hive-0.9.1-SNAPSHOT-h0.21-keepgoing=false/ws/hive/ivy/ivysettings.xml [ivy:resolve] downloading http://repo1.maven.org/maven2/org/apache/hadoop/hadoop-common/0.23.1/hadoop-common-0.23.1.jar ... [ivy:resolve] . (1725kB) [ivy:resolve] .. (0kB) [ivy:resolve] [SUCCESSFUL ] org.apache.hadoop#hadoop-common;0.23.1!hadoop-common.jar (3266ms) [ivy:resolve] downloading http://repo1.maven.org/maven2/org/apache/hadoop/hadoop-mapreduce-client-core/0.23.1/hadoop-mapreduce-client-core-0.23.1.jar ... [ivy:resolve] . (1314kB) [ivy:resolve] .. (0kB) [ivy:resolve] [SUCCESSFUL ] org.apache.hadoop#hadoop-mapreduce-client-core;0.23.1!hadoop-mapreduce-client-core.jar (207ms) [ivy:resolve] downloading http://repo1.maven.org/maven2/org/apache/hadoop/hadoop-archives/0.23.1/hadoop-archives-0.23.1.jar ... [ivy:resolve] . (20kB) [ivy:resolve] .. (0kB) [ivy:resolve] [SUCCESSFUL ] org.apache.hadoop#hadoop-archives;0.23.1!hadoop-archives.jar (195ms) [ivy:resolve] downloading http://repo1.maven.org/maven2/org/apache/hadoop/hadoop-hdfs/0.23.1/hadoop-hdfs-0.23.1.jar ... [ivy:resolve] . (1725kB) [ivy:resolve] .. (0kB) [ivy:resolve] [SUCCESSFUL ] org.apache.hadoop#hadoop-hdfs;0.23.1!hadoop-hdfs.jar (156ms) [ivy:resolve] downloading http://repo1.maven.org/maven2/org/apache/hadoop/hadoop-hdfs/0.23.1/hadoop-hdfs-0.23.1-tests.jar ... [ivy:resolve]
[jira] [Commented] (HIVE-3581) get_json_object and json_tuple return null in the presence of new line characters
[ https://issues.apache.org/jira/browse/HIVE-3581?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13479137#comment-13479137 ] Kevin Wilfong commented on HIVE-3581: - Sorry about that. It looks like we'll either need to set mapreduce.task.classpath.user.precedence to true in Hive, or provide two versions of JSON parsing, one using JSONObject and the other using Jackson, controlled by a config. I think the former sounds preferable provided it works. get_json_object and json_tuple return null in the presence of new line characters - Key: HIVE-3581 URL: https://issues.apache.org/jira/browse/HIVE-3581 Project: Hive Issue Type: Bug Components: Query Processor Affects Versions: 0.10.0 Reporter: Kevin Wilfong Assignee: Kevin Wilfong Attachments: HIVE-3581.1.patch.txt This was introduced when these functions were updated to use Jackson. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
Review Request: HIVE-3231 msck repair should find partitions already containing data files
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/7649/ --- Review request for hive. Description --- msck repair currently will only discover partition directories if they are empty. It seems a more apt use case to copy data files into a table, creating the partition directories as you go, rather than creating a bunch of empty partition directories, then running msck repair to dynamically add them, then inserting your actual data files. This addresses bug HIVE-3231. https://issues.apache.org/jira/browse/HIVE-3231 Diffs - /trunk/ql/src/java/org/apache/hadoop/hive/ql/metadata/HiveMetaStoreChecker.java 1396831 /trunk/ql/src/test/queries/clientpositive/repair.q 1396831 /trunk/ql/src/test/queries/clientpositive/repair_hadoop23.q 1396831 Diff: https://reviews.apache.org/r/7649/diff/ Testing --- Thanks, Keegan Mosley
[jira] [Work started] (HIVE-3231) msck repair should find partitions already containing data files
[ https://issues.apache.org/jira/browse/HIVE-3231?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HIVE-3231 started by Keegan Mosley. msck repair should find partitions already containing data files Key: HIVE-3231 URL: https://issues.apache.org/jira/browse/HIVE-3231 Project: Hive Issue Type: Improvement Components: Metastore Affects Versions: 0.10.0, 0.9.1 Reporter: Keegan Mosley Assignee: Keegan Mosley Labels: msck Fix For: 0.10.0 Attachments: HIVE-3231.1.patch.txt, HIVE-3231.2.patch.txt msck repair currently will only discover partition directories if they are empty. It seems a more apt use case to copy data files into a table, creating the partition directories as you go, rather than creating a bunch of empty partition directories, then running msck repair to dynamically add them, then inserting your actual data files. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-3231) msck repair should find partitions already containing data files
[ https://issues.apache.org/jira/browse/HIVE-3231?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Keegan Mosley updated HIVE-3231: Attachment: HIVE-3231.2.patch.txt msck repair should find partitions already containing data files Key: HIVE-3231 URL: https://issues.apache.org/jira/browse/HIVE-3231 Project: Hive Issue Type: Improvement Components: Metastore Affects Versions: 0.10.0, 0.9.1 Reporter: Keegan Mosley Assignee: Keegan Mosley Labels: msck Fix For: 0.10.0 Attachments: HIVE-3231.1.patch.txt, HIVE-3231.2.patch.txt msck repair currently will only discover partition directories if they are empty. It seems a more apt use case to copy data files into a table, creating the partition directories as you go, rather than creating a bunch of empty partition directories, then running msck repair to dynamically add them, then inserting your actual data files. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-3231) msck repair should find partitions already containing data files
[ https://issues.apache.org/jira/browse/HIVE-3231?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Keegan Mosley updated HIVE-3231: Assignee: Carl Steinbach (was: Keegan Mosley) Status: Patch Available (was: In Progress) https://reviews.apache.org/r/7649/ msck repair should find partitions already containing data files Key: HIVE-3231 URL: https://issues.apache.org/jira/browse/HIVE-3231 Project: Hive Issue Type: Improvement Components: Metastore Affects Versions: 0.10.0, 0.9.1 Reporter: Keegan Mosley Assignee: Carl Steinbach Labels: msck Fix For: 0.10.0 Attachments: HIVE-3231.1.patch.txt, HIVE-3231.2.patch.txt msck repair currently will only discover partition directories if they are empty. It seems a more apt use case to copy data files into a table, creating the partition directories as you go, rather than creating a bunch of empty partition directories, then running msck repair to dynamically add them, then inserting your actual data files. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HIVE-3597) Add a flag to turn off Truncated path - alias for end-user
Gang Tim Liu created HIVE-3597: -- Summary: Add a flag to turn off Truncated path - alias for end-user Key: HIVE-3597 URL: https://issues.apache.org/jira/browse/HIVE-3597 Project: Hive Issue Type: Improvement Components: Testing Infrastructure Affects Versions: 0.10.0 Reporter: Gang Tim Liu HIVE-3556 introduced Truncated path - alias in order to test out of Path-alias. It might have less interest for end user. we can add a flag to control if we show it. By default, we don't show it but turn it on in test via QTestUtil.java -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
hive pull request: HIVE-3231.1.patch.txt
Github user kmosley closed the pull request at: https://github.com/apache/hive/pull/6
[jira] [Commented] (HIVE-3529) Incorrect partition bucket/sort metadata when overwriting partition with different metadata from table
[ https://issues.apache.org/jira/browse/HIVE-3529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13479239#comment-13479239 ] Kevin Wilfong commented on HIVE-3529: - Also, the serde parameters are wrong as well. Incorrect partition bucket/sort metadata when overwriting partition with different metadata from table -- Key: HIVE-3529 URL: https://issues.apache.org/jira/browse/HIVE-3529 Project: Hive Issue Type: Bug Components: Query Processor Affects Versions: 0.10.0 Reporter: Kevin Wilfong Assignee: Kevin Wilfong If you have a partition with bucket/sort metadata set, then you alter the table to have different bucket/sort metadata, and insert overwrite the partition with hive.enforce.bucketing=true and/or hive.enforce.sorting=true, the partition data will be bucketed/sorted by the table's metadata, but the partition will have the same metadata. This could result in wrong results. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-3544) union involving double column with a map join subquery will fail or give wrong results
[ https://issues.apache.org/jira/browse/HIVE-3544?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kevin Wilfong updated HIVE-3544: Attachment: HIVE-3544.3.patch.txt union involving double column with a map join subquery will fail or give wrong results -- Key: HIVE-3544 URL: https://issues.apache.org/jira/browse/HIVE-3544 Project: Hive Issue Type: Bug Components: Query Processor Affects Versions: 0.10.0 Reporter: Kevin Wilfong Assignee: Kevin Wilfong Attachments: HIVE-3544.1.patch.txt, HIVE-3544.2.patch.txt, HIVE-3544.3.patch.txt The following query fails: select * from (select cast(a.key as bigint) as key from src a join src b on a.key = b.key union all select cast(key as double) as key from src)a The following query gives wrong results: select * from (select cast(a.key as bigint) as key, cast(b.key as double) as value from src a join src b on a.key = b.key union all select cast(key as double) as key, cast(key as string) as value from src)a But the following query runs fine: select * from (select cast(a.key as bigint) as key from src a union all select cast(key as double) as key from src)a -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-3529) Incorrect partition bucket/sort metadata when overwriting partition with different metadata from table
[ https://issues.apache.org/jira/browse/HIVE-3529?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kevin Wilfong updated HIVE-3529: Attachment: HIVE-3529.1.patch.txt Incorrect partition bucket/sort metadata when overwriting partition with different metadata from table -- Key: HIVE-3529 URL: https://issues.apache.org/jira/browse/HIVE-3529 Project: Hive Issue Type: Bug Components: Query Processor Affects Versions: 0.10.0 Reporter: Kevin Wilfong Assignee: Kevin Wilfong Attachments: HIVE-3529.1.patch.txt If you have a partition with bucket/sort metadata set, then you alter the table to have different bucket/sort metadata, and insert overwrite the partition with hive.enforce.bucketing=true and/or hive.enforce.sorting=true, the partition data will be bucketed/sorted by the table's metadata, but the partition will have the same metadata. This could result in wrong results. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-3529) Incorrect partition bucket/sort metadata when overwriting partition with different metadata from table
[ https://issues.apache.org/jira/browse/HIVE-3529?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kevin Wilfong updated HIVE-3529: Status: Patch Available (was: Open) Incorrect partition bucket/sort metadata when overwriting partition with different metadata from table -- Key: HIVE-3529 URL: https://issues.apache.org/jira/browse/HIVE-3529 Project: Hive Issue Type: Bug Components: Query Processor Affects Versions: 0.10.0 Reporter: Kevin Wilfong Assignee: Kevin Wilfong Attachments: HIVE-3529.1.patch.txt If you have a partition with bucket/sort metadata set, then you alter the table to have different bucket/sort metadata, and insert overwrite the partition with hive.enforce.bucketing=true and/or hive.enforce.sorting=true, the partition data will be bucketed/sorted by the table's metadata, but the partition will have the same metadata. This could result in wrong results. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3529) Incorrect partition bucket/sort metadata when overwriting partition with different metadata from table
[ https://issues.apache.org/jira/browse/HIVE-3529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13479254#comment-13479254 ] Kevin Wilfong commented on HIVE-3529: - The following information is now copied from the table to the partition when writing to an existing partition: bucket columns bucket number sort columns/orders serde parameters Incorrect partition bucket/sort metadata when overwriting partition with different metadata from table -- Key: HIVE-3529 URL: https://issues.apache.org/jira/browse/HIVE-3529 Project: Hive Issue Type: Bug Components: Query Processor Affects Versions: 0.10.0 Reporter: Kevin Wilfong Assignee: Kevin Wilfong Attachments: HIVE-3529.1.patch.txt If you have a partition with bucket/sort metadata set, then you alter the table to have different bucket/sort metadata, and insert overwrite the partition with hive.enforce.bucketing=true and/or hive.enforce.sorting=true, the partition data will be bucketed/sorted by the table's metadata, but the partition will have the same metadata. This could result in wrong results. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3529) Incorrect partition bucket/sort metadata when overwriting partition with different metadata from table
[ https://issues.apache.org/jira/browse/HIVE-3529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13479253#comment-13479253 ] Kevin Wilfong commented on HIVE-3529: - https://reviews.facebook.net/D6087 Incorrect partition bucket/sort metadata when overwriting partition with different metadata from table -- Key: HIVE-3529 URL: https://issues.apache.org/jira/browse/HIVE-3529 Project: Hive Issue Type: Bug Components: Query Processor Affects Versions: 0.10.0 Reporter: Kevin Wilfong Assignee: Kevin Wilfong Attachments: HIVE-3529.1.patch.txt If you have a partition with bucket/sort metadata set, then you alter the table to have different bucket/sort metadata, and insert overwrite the partition with hive.enforce.bucketing=true and/or hive.enforce.sorting=true, the partition data will be bucketed/sorted by the table's metadata, but the partition will have the same metadata. This could result in wrong results. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3403) user should not specify mapjoin to perform sort-merge bucketed join
[ https://issues.apache.org/jira/browse/HIVE-3403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13479263#comment-13479263 ] Namit Jain commented on HIVE-3403: -- Had an offline review with Kevin - the following came out of that: 1. The policy to choose the big table should take the average size of the partition into account. We dont need to change the existing policy, can add a new policy. 2. The new policy should give preference to a table with no partitions/ 1 partition. 3. If one of the tables is small enough to fit in memory, there is no need to perform sort-merge join. The policy should check that. Eventually, that check should be removed, and that should be handled by the physical optimizer - will file a new jira for that. user should not specify mapjoin to perform sort-merge bucketed join --- Key: HIVE-3403 URL: https://issues.apache.org/jira/browse/HIVE-3403 Project: Hive Issue Type: Bug Reporter: Namit Jain Assignee: Namit Jain Attachments: hive.3403.1.patch Currently, in order to perform a sort merge bucketed join, the user needs to set hive.optimize.bucketmapjoin.sortedmerge to true, and also specify the mapjoin hint. The user should not specify any hints. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HIVE-3598) physical optimizer changes for auto sort-merge join
Namit Jain created HIVE-3598: Summary: physical optimizer changes for auto sort-merge join Key: HIVE-3598 URL: https://issues.apache.org/jira/browse/HIVE-3598 Project: Hive Issue Type: Bug Reporter: Namit Jain -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-3551) Add more tests where output of sort merge join is sorted
[ https://issues.apache.org/jira/browse/HIVE-3551?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kevin Wilfong updated HIVE-3551: Resolution: Fixed Status: Resolved (was: Patch Available) Committed, thanks Namit. Add more tests where output of sort merge join is sorted Key: HIVE-3551 URL: https://issues.apache.org/jira/browse/HIVE-3551 Project: Hive Issue Type: Test Components: Query Processor Reporter: Namit Jain Assignee: Namit Jain Attachments: hive.3551.1.patch, hive.3551.2.patch, hive.3551.3.patch As a follow-up to HIVE-3536, it would be good to have tests where a join is being performed between a single partition of the big table with multiple partitions of the small tables. The output should still be sorted -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3598) physical optimizer changes for auto sort-merge join
[ https://issues.apache.org/jira/browse/HIVE-3598?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13479310#comment-13479310 ] Namit Jain commented on HIVE-3598: -- After HIVE-3403, an auto sortmerge join can be performed. However, consider the following scenario: T1 - big table: 3 partitions : big partitions T2 - small table: 1 partition both T1 and T2 are sorted/bucketed on the same key. If a join is being performed between T1 and T2, with HIVE-3403, T2 would be picked up as the table on which the mapper would run. Ideally, if T2 is so small that it can fit in memory, a better approach would be map-join. The physical optimizer should change the sort-merge join task into a backup task, with all tables being candidates for map-join (similar to auto convert mapjoin). physical optimizer changes for auto sort-merge join --- Key: HIVE-3598 URL: https://issues.apache.org/jira/browse/HIVE-3598 Project: Hive Issue Type: Bug Reporter: Namit Jain -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-3403) user should not specify mapjoin to perform sort-merge bucketed join
[ https://issues.apache.org/jira/browse/HIVE-3403?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Namit Jain updated HIVE-3403: - Status: Open (was: Patch Available) user should not specify mapjoin to perform sort-merge bucketed join --- Key: HIVE-3403 URL: https://issues.apache.org/jira/browse/HIVE-3403 Project: Hive Issue Type: Bug Reporter: Namit Jain Assignee: Namit Jain Attachments: hive.3403.1.patch Currently, in order to perform a sort merge bucketed join, the user needs to set hive.optimize.bucketmapjoin.sortedmerge to true, and also specify the mapjoin hint. The user should not specify any hints. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-3544) union involving double column with a map join subquery will fail or give wrong results
[ https://issues.apache.org/jira/browse/HIVE-3544?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Namit Jain updated HIVE-3544: - Status: Open (was: Patch Available) minor comments on phabricator union involving double column with a map join subquery will fail or give wrong results -- Key: HIVE-3544 URL: https://issues.apache.org/jira/browse/HIVE-3544 Project: Hive Issue Type: Bug Components: Query Processor Affects Versions: 0.10.0 Reporter: Kevin Wilfong Assignee: Kevin Wilfong Attachments: HIVE-3544.1.patch.txt, HIVE-3544.2.patch.txt, HIVE-3544.3.patch.txt The following query fails: select * from (select cast(a.key as bigint) as key from src a join src b on a.key = b.key union all select cast(key as double) as key from src)a The following query gives wrong results: select * from (select cast(a.key as bigint) as key, cast(b.key as double) as value from src a join src b on a.key = b.key union all select cast(key as double) as key, cast(key as string) as value from src)a But the following query runs fine: select * from (select cast(a.key as bigint) as key from src a union all select cast(key as double) as key from src)a -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-3255) Add DBTokenStore to store Delegation Tokens in DB
[
https://issues.apache.org/jira/browse/HIVE-3255?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashutosh Chauhan updated HIVE-3255:
---
Attachment: hive-3255.patch
Updating with complete patch. Patch is ready for review.
Add DBTokenStore to store Delegation Tokens in DB
-
Key: HIVE-3255
URL: https://issues.apache.org/jira/browse/HIVE-3255
Project: Hive
Issue Type: New Feature
Components: Metastore, Security
Affects Versions: 0.9.0, 0.10.0
Reporter: Ashutosh Chauhan
Assignee: Ashutosh Chauhan
Attachments: hive-3255.patch
Before HIVE-1696 metastore was completely stateless and all the data is in
backend db. HIVE-1696 added delegation tokens for metastore which metastore
needs to keep in memory. HIVE-2467 added support for storing delegation
tokens in ZooKeeper via an interface {{DelegationTokenStore}}. This jira is
about using DB backend for storing tokens which metastore uses for storing
data.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HIVE-3599) missing return of compression codec to pool
Owen O'Malley created HIVE-3599: --- Summary: missing return of compression codec to pool Key: HIVE-3599 URL: https://issues.apache.org/jira/browse/HIVE-3599 Project: Hive Issue Type: Bug Components: Query Processor Reporter: Owen O'Malley The RCFile writer is currently missing a call to return of one of the compression codecs to the pool. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HIVE-3599) missing return of compression codec to pool
[ https://issues.apache.org/jira/browse/HIVE-3599?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Owen O'Malley reassigned HIVE-3599: --- Assignee: Owen O'Malley missing return of compression codec to pool --- Key: HIVE-3599 URL: https://issues.apache.org/jira/browse/HIVE-3599 Project: Hive Issue Type: Bug Components: Query Processor Reporter: Owen O'Malley Assignee: Owen O'Malley The RCFile writer is currently missing a call to return of one of the compression codecs to the pool. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-3599) missing return of compression codec to pool
[ https://issues.apache.org/jira/browse/HIVE-3599?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Owen O'Malley updated HIVE-3599: Attachment: hive-3599.patch Here's the obvious fix. There aren't any functional difference. missing return of compression codec to pool --- Key: HIVE-3599 URL: https://issues.apache.org/jira/browse/HIVE-3599 Project: Hive Issue Type: Bug Components: Query Processor Reporter: Owen O'Malley Assignee: Owen O'Malley Attachments: hive-3599.patch The RCFile writer is currently missing a call to return of one of the compression codecs to the pool. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-3599) missing return of compression codec to pool
[ https://issues.apache.org/jira/browse/HIVE-3599?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Owen O'Malley updated HIVE-3599: Status: Patch Available (was: Open) missing return of compression codec to pool --- Key: HIVE-3599 URL: https://issues.apache.org/jira/browse/HIVE-3599 Project: Hive Issue Type: Bug Components: Query Processor Reporter: Owen O'Malley Assignee: Owen O'Malley Attachments: hive-3599.patch The RCFile writer is currently missing a call to return of one of the compression codecs to the pool. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
Hive-trunk-h0.21 - Build # 1744 - Fixed
Changes for Build #1739 Changes for Build #1740 [namit] HIVE-3581 get_json_object and json_tuple return null in the presence of new line characters (Kevin Wilfong via namit) Changes for Build #1741 Changes for Build #1742 [kevinwilfong] HIVE-967. Implement show create table (Feng Lu via kevinwilfong) Changes for Build #1743 Changes for Build #1744 All tests passed The Apache Jenkins build system has built Hive-trunk-h0.21 (build #1744) Status: Fixed Check console output at https://builds.apache.org/job/Hive-trunk-h0.21/1744/ to view the results.
[jira] [Commented] (HIVE-3152) Disallow certain character patterns in partition names
[ https://issues.apache.org/jira/browse/HIVE-3152?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13479504#comment-13479504 ] Kevin Wilfong commented on HIVE-3152: - Ashutosh, that change wasn't really the point of this JIRA, if Ivan's ok with it, I'd be happy leaving it out, and filing a JIRA to make that change later. If that change to ListenerEvent was removed, would this be compatible with HCatalog? Disallow certain character patterns in partition names -- Key: HIVE-3152 URL: https://issues.apache.org/jira/browse/HIVE-3152 Project: Hive Issue Type: New Feature Components: Metastore Reporter: Andrew Poland Assignee: Ivan Gorbachev Priority: Minor Labels: api-addition, configuration-addition Attachments: jira-3152.0.patch New event listener to allow metastore to reject a partition name if it contains undesired character patterns such as unicode and commas. Match pattern is implemented as a regular expression Modifies append_partition to call a new MetaStorePreventListener implementation, PreAppendPartitionEvent. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3152) Disallow certain character patterns in partition names
[ https://issues.apache.org/jira/browse/HIVE-3152?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13479524#comment-13479524 ] Ivan Gorbachev commented on HIVE-3152: -- [~kevinwilfong] I agree. I think it's good idea to remove everything from this patch that doesn't belong to disallowing character patterns. Disallow certain character patterns in partition names -- Key: HIVE-3152 URL: https://issues.apache.org/jira/browse/HIVE-3152 Project: Hive Issue Type: New Feature Components: Metastore Reporter: Andrew Poland Assignee: Ivan Gorbachev Priority: Minor Labels: api-addition, configuration-addition Attachments: jira-3152.0.patch New event listener to allow metastore to reject a partition name if it contains undesired character patterns such as unicode and commas. Match pattern is implemented as a regular expression Modifies append_partition to call a new MetaStorePreventListener implementation, PreAppendPartitionEvent. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-3525) Avro Maps with Nullable Values fail with NPE
[ https://issues.apache.org/jira/browse/HIVE-3525?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Carl Steinbach updated HIVE-3525: - Assignee: Sean Busbey Avro Maps with Nullable Values fail with NPE Key: HIVE-3525 URL: https://issues.apache.org/jira/browse/HIVE-3525 Project: Hive Issue Type: Bug Components: Serializers/Deserializers Reporter: Sean Busbey Assignee: Sean Busbey Fix For: 0.10.0 Attachments: HIVE-3525.1.patch.txt, HIVE-3525.2.patch.txt When working against current trunk@1393794, using a backing Avro schema that has a Map field with nullable values causes a NPE on deserialization when the map contains a null value. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-3525) Avro Maps with Nullable Values fail with NPE
[ https://issues.apache.org/jira/browse/HIVE-3525?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Carl Steinbach updated HIVE-3525: - Resolution: Fixed Fix Version/s: 0.10.0 Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Committed to trunk. Thanks Sean! Avro Maps with Nullable Values fail with NPE Key: HIVE-3525 URL: https://issues.apache.org/jira/browse/HIVE-3525 Project: Hive Issue Type: Bug Components: Serializers/Deserializers Reporter: Sean Busbey Assignee: Sean Busbey Fix For: 0.10.0 Attachments: HIVE-3525.1.patch.txt, HIVE-3525.2.patch.txt When working against current trunk@1393794, using a backing Avro schema that has a Map field with nullable values causes a NPE on deserialization when the map contains a null value. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-3535) derby metastore upgrade script throw errors when updating from 0.7 to 0.8
[ https://issues.apache.org/jira/browse/HIVE-3535?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Carl Steinbach updated HIVE-3535: - Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Committed to trunk. Thanks Zhenxiao! derby metastore upgrade script throw errors when updating from 0.7 to 0.8 - Key: HIVE-3535 URL: https://issues.apache.org/jira/browse/HIVE-3535 Project: Hive Issue Type: Bug Components: Metastore Affects Versions: 0.9.0 Reporter: Zhenxiao Luo Assignee: Zhenxiao Luo Fix For: 0.10.0 Attachments: HIVE-3535.1.patch.txt $DERBY_HOME/bin/ij upgrade.sql ij version 10.4 ij CONNECT 'jdbc:derby:/var/lib/hive/metastore/metastore_db'; ij RUN '/usr/lib/hive/scripts/metastore/upgrade/derby/upgrade-0.7.0-to-0.8.0.derby.sql'; ij -- Upgrade MetaStore schema from 0.7.0 to 0.8.0 RUN '008-HIVE-2246.derby.sql'; ij /* * Creates the following tables: * - CDS * - COLUMNS_V2 * The new columns table is called COLUMNS_V2 * because many columns are removed, and the schema is changed. * It'd take too long to migrate and keep the same table. */ CREATE TABLE CDS ( CD_ID bigint NOT NULL, PRIMARY KEY (CD_ID) ); 0 rows inserted/updated/deleted ij CREATE TABLE COLUMNS_V2 ( CD_ID bigint NOT NULL, COMMENT varchar(4000), COLUMN_NAME varchar(128) NOT NULL, TYPE_NAME varchar(4000), INTEGER_IDX INTEGER NOT NULL, PRIMARY KEY (CD_ID, COLUMN_NAME) ); 0 rows inserted/updated/deleted ij ALTER TABLE COLUMNS_V2 ADD CONSTRAINT COLUMNS_V2_FK1 FOREIGN KEY (CD_ID) REFERENCES CDS (CD_ID) ON DELETE NO ACTION ON UPDATE NO ACTION ; 0 rows inserted/updated/deleted ij /* Alter the SDS table to: * - add the column CD_ID * - add a foreign key on CD_ID * - create an index on CD_ID */ ALTER TABLE SDS ADD COLUMN CD_ID bigint ; 0 rows inserted/updated/deleted ij ALTER TABLE SDS ADD CONSTRAINT SDS_FK2 FOREIGN KEY (CD_ID) REFERENCES CDS (CD_ID) ; 0 rows inserted/updated/deleted ij /* * Migrate the TBLS table * Add entries into CDS. * Populate the CD_ID field in SDS for tables * Add entires to COLUMNS_V2 based on this table's sd's columns */ /* In the migration, there is a 1:1 mapping between CD_ID and SD_ID * for tables. For speed, just let CD_ID = SD_ID for tables */ INSERT INTO CDS (CD_ID) SELECT t.SD_ID FROM TBLS t WHERE t.SD_ID IS NOT NULL ORDER BY t.SD_ID; ERROR 42X01: Syntax error: Encountered ORDER at line 13, column 54. ij UPDATE SDS SET CD_ID = SD_ID WHERE SD_ID in (SELECT t.SD_ID FROM TBLS t WHERE t.SD_ID IS NOT NULL ORDER BY t.SD_ID); ERROR 42X01: Syntax error: Encountered ORDER at line 4, column 55. ij INSERT INTO COLUMNS_V2 (CD_ID, COMMENT, COLUMN_NAME, TYPE_NAME, INTEGER_IDX) SELECT c.SD_ID, c.COMMENT, c.COLUMN_NAME, c.TYPE_NAME, c.INTEGER_IDX FROM COLUMNS c JOIN TBLS t ON t.SD_ID = c.SD_ID ; ERROR 23503: INSERT on table 'COLUMNS_V2' caused a violation of foreign key constraint 'COLUMNS_V2_FK1' for key (1). The statement has been rolled back. ij /* * Migrate the partitions. * Update the partitions' SDS to use the parent tables' CD_ID BEGIN * Derby does not allow joins in update statements, * so we have to make a temporary tableh */ DECLARE GLOBAL TEMPORARY TABLE TMP_TBL ( SD_ID bigint not null, CD_ID bigint not null ) ON COMMIT PRESERVE ROWS NOT LOGGED; 0 rows inserted/updated/deleted ij INSERT INTO SESSION.TMP_TBL SELECT p.SD_ID, sds.CD_ID FROM PARTITIONS p JOIN TBLS t ON t.TBL_ID = p.TBL_ID JOIN SDS sds on t.SD_ID = sds.SD_ID WHERE p.SD_ID IS NOT NULL; ERROR 23502: Column 'CD_ID' cannot accept a NULL value. ij UPDATE SDS sd SET sd.CD_ID = (SELECT tt.CD_ID FROM SESSION.TMP_TBL tt WHERE tt.SD_ID = sd.SD_ID) WHERE sd.SD_ID IN (SELECT SD_ID FROM SESSION.TMP_TBL); 0 rows inserted/updated/deleted WARNING 02000: No row was found for FETCH, UPDATE or DELETE; or the result of a query is an empty table. ij /* * Migrate IDXS */ INSERT INTO CDS (CD_ID) SELECT i.SD_ID FROM IDXS i WHERE i.SD_ID IS NOT NULL ORDER BY i.SD_ID; ERROR 42X01: Syntax error: Encountered ORDER at line 5, column 54. ij UPDATE SDS SET CD_ID = SD_ID WHERE SD_ID in (SELECT i.SD_ID FROM IDXS i WHERE i.SD_ID IS NOT NULL ORDER BY i.SD_ID); ERROR 42X01: Syntax error: Encountered ORDER at line 4, column 55. ij INSERT INTO COLUMNS_V2 (CD_ID, COMMENT, COLUMN_NAME, TYPE_NAME, INTEGER_IDX) SELECT c.SD_ID, c.COMMENT, c.COLUMN_NAME, c.TYPE_NAME, c.INTEGER_IDX FROM COLUMNS c JOIN IDXS i ON i.SD_ID = c.SD_ID ; ERROR 42X05: Table/View 'IDXS' does not exist. ij /* * rename the old COLUMNS table */ RENAME TABLE COLUMNS TO
[jira] [Updated] (HIVE-3544) union involving double column with a map join subquery will fail or give wrong results
[ https://issues.apache.org/jira/browse/HIVE-3544?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kevin Wilfong updated HIVE-3544: Attachment: HIVE-3544.4.patch.txt union involving double column with a map join subquery will fail or give wrong results -- Key: HIVE-3544 URL: https://issues.apache.org/jira/browse/HIVE-3544 Project: Hive Issue Type: Bug Components: Query Processor Affects Versions: 0.10.0 Reporter: Kevin Wilfong Assignee: Kevin Wilfong Attachments: HIVE-3544.1.patch.txt, HIVE-3544.2.patch.txt, HIVE-3544.3.patch.txt, HIVE-3544.4.patch.txt The following query fails: select * from (select cast(a.key as bigint) as key from src a join src b on a.key = b.key union all select cast(key as double) as key from src)a The following query gives wrong results: select * from (select cast(a.key as bigint) as key, cast(b.key as double) as value from src a join src b on a.key = b.key union all select cast(key as double) as key, cast(key as string) as value from src)a But the following query runs fine: select * from (select cast(a.key as bigint) as key from src a union all select cast(key as double) as key from src)a -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-3544) union involving double column with a map join subquery will fail or give wrong results
[ https://issues.apache.org/jira/browse/HIVE-3544?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kevin Wilfong updated HIVE-3544: Status: Patch Available (was: Open) union involving double column with a map join subquery will fail or give wrong results -- Key: HIVE-3544 URL: https://issues.apache.org/jira/browse/HIVE-3544 Project: Hive Issue Type: Bug Components: Query Processor Affects Versions: 0.10.0 Reporter: Kevin Wilfong Assignee: Kevin Wilfong Attachments: HIVE-3544.1.patch.txt, HIVE-3544.2.patch.txt, HIVE-3544.3.patch.txt, HIVE-3544.4.patch.txt The following query fails: select * from (select cast(a.key as bigint) as key from src a join src b on a.key = b.key union all select cast(key as double) as key from src)a The following query gives wrong results: select * from (select cast(a.key as bigint) as key, cast(b.key as double) as value from src a join src b on a.key = b.key union all select cast(key as double) as key, cast(key as string) as value from src)a But the following query runs fine: select * from (select cast(a.key as bigint) as key from src a union all select cast(key as double) as key from src)a -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HIVE-3600) Show tables within database
Krish created HIVE-3600: --- Summary: Show tables within database Key: HIVE-3600 URL: https://issues.apache.org/jira/browse/HIVE-3600 Project: Hive Issue Type: Bug Components: Metastore, Query Processor Affects Versions: 0.9.0 Reporter: Krish Priority: Minor SHOW TABLES doesn't list tables under user created databases. Only tables under [default] database are shown. 1) SHOW TABLES must provide list of all tables under each database in the form db_name.table_name 2) Enhance SHOW TABLES to support SHOW TABLES db_name to list only the tables from db_name -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3152) Disallow certain character patterns in partition names
[ https://issues.apache.org/jira/browse/HIVE-3152?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13479637#comment-13479637 ] Ashutosh Chauhan commented on HIVE-3152: Cool. Totally agree. All we need is getHandler() as long as thats there, HCatalog will work fine. And sorry for jumping on the jira so late. I will try to be more proactive on reading patches going forward. Disallow certain character patterns in partition names -- Key: HIVE-3152 URL: https://issues.apache.org/jira/browse/HIVE-3152 Project: Hive Issue Type: New Feature Components: Metastore Reporter: Andrew Poland Assignee: Ivan Gorbachev Priority: Minor Labels: api-addition, configuration-addition Attachments: jira-3152.0.patch New event listener to allow metastore to reject a partition name if it contains undesired character patterns such as unicode and commas. Match pattern is implemented as a regular expression Modifies append_partition to call a new MetaStorePreventListener implementation, PreAppendPartitionEvent. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
