[jira] [Commented] (HIVE-17529) Bucket Map Join : Sets incorrect edge type causing execution failure
[
https://issues.apache.org/jira/browse/HIVE-17529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16172776#comment-16172776
]
Hive QA commented on HIVE-17529:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12887964/HIVE-17529.4.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 12 failed/errored test(s), 11044 tests
executed
*Failed tests:*
{noformat}
TestAccumuloCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=230)
TestDummy - did not produce a TEST-*.xml file (likely timed out) (batchId=230)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[create_view] (batchId=39)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[udf_mask_hash]
(batchId=28)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[union_fast_stats]
(batchId=156)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_explainuser_1]
(batchId=170)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_2]
(batchId=100)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[drop_table_failure2]
(batchId=89)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query14]
(batchId=234)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[bucket_map_join_tez1]
(batchId=137)
org.apache.hadoop.hive.ql.TestAcidOnTez.testCtasTezUnion (batchId=215)
org.apache.hadoop.hive.ql.TestAcidOnTez.testNonStandardConversion01
(batchId=215)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/6896/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/6896/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-6896/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 12 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12887964 - PreCommit-HIVE-Build
> Bucket Map Join : Sets incorrect edge type causing execution failure
>
>
> Key: HIVE-17529
> URL: https://issues.apache.org/jira/browse/HIVE-17529
> Project: Hive
> Issue Type: Bug
>Reporter: Deepak Jaiswal
>Assignee: Deepak Jaiswal
> Attachments: HIVE-17529.1.patch, HIVE-17529.2.patch,
> HIVE-17529.3.patch, HIVE-17529.4.patch
>
>
> If while traversing the tree to generate tasks, a bucket mapjoin may set its
> edge as CUSTOM_SIMPLE_EDGE against CUSTOM_EDGE if the bigtable is already not
> traversed causing Tez to assert and fail the vertex.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-16898) Validation of source file after distcp in repl load
[ https://issues.apache.org/jira/browse/HIVE-16898?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Dai updated HIVE-16898: -- Attachment: HIVE-16898.5.patch Rebase with master. > Validation of source file after distcp in repl load > > > Key: HIVE-16898 > URL: https://issues.apache.org/jira/browse/HIVE-16898 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Affects Versions: 3.0.0 >Reporter: anishek >Assignee: Daniel Dai > Fix For: 3.0.0 > > Attachments: HIVE-16898.1.patch, HIVE-16898.2.patch, > HIVE-16898.3.patch, HIVE-16898.4.patch, HIVE-16898.5.patch > > > time between deciding the source and destination path for distcp to invoking > of distcp can have a change of the source file, hence distcp might copy the > wrong file to destination, hence we should an additional check on the > checksum of the source file path after distcp finishes to make sure the path > didnot change during the copy process. if it has take additional steps to > delete the previous file on destination and copy the new source and repeat > the same process as above till we copy the correct file. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-15653) Some ALTER TABLE commands drop table stats
[
https://issues.apache.org/jira/browse/HIVE-15653?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16172752#comment-16172752
]
Alexander Behm commented on HIVE-15653:
---
[~ctang.ma], am I understanding correctly that there is no interest in fixing
this issue on the Metastore side? I understand that all clients can pass
DO_NOT_UPDATE_STATS.
> Some ALTER TABLE commands drop table stats
> --
>
> Key: HIVE-15653
> URL: https://issues.apache.org/jira/browse/HIVE-15653
> Project: Hive
> Issue Type: Bug
> Components: Metastore, Statistics
>Affects Versions: 1.1.0
>Reporter: Alexander Behm
>Assignee: Chaoyu Tang
>Priority: Critical
> Fix For: 2.3.0
>
> Attachments: HIVE-15653.1.patch, HIVE-15653.2.patch,
> HIVE-15653.3.patch, HIVE-15653.4.patch, HIVE-15653.5.patch,
> HIVE-15653.6.patch, HIVE-15653.patch
>
>
> Some ALTER TABLE commands drop the table stats. That may make sense for some
> ALTER TABLE operations, but certainly not for others. Personally, I I think
> ALTER TABLE should only change what was requested by the user without any
> side effects that may be unclear to users. In particular, collecting stats
> can be an expensive operation so it's rather inconvenient for users if they
> get wiped accidentally.
> Repro:
> {code}
> create table t (i int);
> insert into t values(1);
> analyze table t compute statistics;
> alter table t set tblproperties('test'='test');
> hive> describe formatted t;
> OK
> # col_namedata_type comment
>
> i int
>
> # Detailed Table Information
> Database: default
> Owner:abehm
> CreateTime: Tue Jan 17 18:13:34 PST 2017
> LastAccessTime: UNKNOWN
> Protect Mode: None
> Retention:0
> Location: hdfs://localhost:20500/test-warehouse/t
> Table Type: MANAGED_TABLE
> Table Parameters:
> COLUMN_STATS_ACCURATE false
> last_modified_byabehm
> last_modified_time 1484705748
> numFiles1
> numRows -1
> rawDataSize -1
> testtest
> totalSize 2
> transient_lastDdlTime 1484705748
>
> # Storage Information
> SerDe Library:org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
>
> InputFormat: org.apache.hadoop.mapred.TextInputFormat
> OutputFormat:
> org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
> Compressed: No
> Num Buckets: -1
> Bucket Columns: []
> Sort Columns: []
> Storage Desc Params:
> serialization.format1
> Time taken: 0.169 seconds, Fetched: 34 row(s)
> {code}
> The same behavior can be observed with several other ALTER TABLE commands.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-17402) Provide object location in the HMS notification messages
[
https://issues.apache.org/jira/browse/HIVE-17402?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16172731#comment-16172731
]
Alexander Kolbasov commented on HIVE-17402:
---
The goal is to add information about object location for the following messages:
- {{JSONAddPartitionMessage}}
- {{JSONAlterPartitionMessage}} (old and new location)
- {{JSONAlterTableMessage}} (old and new location)
- {{JSONCreateDatabaseMessage}}
- {{JSONCreateTableMessage}}
- {{JSONDropDatabaseMessage}}
- {{JSONDropPartitionMessage}}
- {{JSONDropTableMessage}}
> Provide object location in the HMS notification messages
>
>
> Key: HIVE-17402
> URL: https://issues.apache.org/jira/browse/HIVE-17402
> Project: Hive
> Issue Type: Improvement
> Components: Hive
>Affects Versions: 2.2.0
>Reporter: Alexander Kolbasov
>Assignee: Alexander Kolbasov
>
> While working on the Apache Sentry project that uses HMS notifications we
> noticed that these notifications are using some useful data - e.g. location
> information for the objects. To get around these, ApacheSentry implemented
> its own version of events
> (https://github.com/apache/sentry/tree/master/sentry-binding/sentry-binding-hive-follower/src/main/java/org/apache/sentry/binding/metastore/messaging/json).
> It seems to be a useful information for Hive as well, so why not add it
> directly into the standard message factory?
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-17402) Provide object location in the HMS notification messages
[ https://issues.apache.org/jira/browse/HIVE-17402?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexander Kolbasov updated HIVE-17402: -- Status: Patch Available (was: Open) > Provide object location in the HMS notification messages > > > Key: HIVE-17402 > URL: https://issues.apache.org/jira/browse/HIVE-17402 > Project: Hive > Issue Type: Improvement > Components: Hive >Affects Versions: 2.2.0 >Reporter: Alexander Kolbasov >Assignee: Alexander Kolbasov > Attachments: HIVE-17402.01.patch > > > While working on the Apache Sentry project that uses HMS notifications we > noticed that these notifications are using some useful data - e.g. location > information for the objects. To get around these, ApacheSentry implemented > its own version of events > (https://github.com/apache/sentry/tree/master/sentry-binding/sentry-binding-hive-follower/src/main/java/org/apache/sentry/binding/metastore/messaging/json). > It seems to be a useful information for Hive as well, so why not add it > directly into the standard message factory? -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17402) Provide object location in the HMS notification messages
[ https://issues.apache.org/jira/browse/HIVE-17402?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexander Kolbasov updated HIVE-17402: -- Attachment: HIVE-17402.01.patch > Provide object location in the HMS notification messages > > > Key: HIVE-17402 > URL: https://issues.apache.org/jira/browse/HIVE-17402 > Project: Hive > Issue Type: Improvement > Components: Hive >Affects Versions: 2.2.0 >Reporter: Alexander Kolbasov >Assignee: Alexander Kolbasov > Attachments: HIVE-17402.01.patch > > > While working on the Apache Sentry project that uses HMS notifications we > noticed that these notifications are using some useful data - e.g. location > information for the objects. To get around these, ApacheSentry implemented > its own version of events > (https://github.com/apache/sentry/tree/master/sentry-binding/sentry-binding-hive-follower/src/main/java/org/apache/sentry/binding/metastore/messaging/json). > It seems to be a useful information for Hive as well, so why not add it > directly into the standard message factory? -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-16898) Validation of source file after distcp in repl load
[
https://issues.apache.org/jira/browse/HIVE-16898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16172727#comment-16172727
]
Hive QA commented on HIVE-16898:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12887961/HIVE-16898.4.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/6895/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/6895/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-6895/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2017-09-20 04:29:05.312
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-6895/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2017-09-20 04:29:05.315
+ cd apache-github-source-source
+ git fetch origin
+ git reset --hard HEAD
HEAD is now at 5608300 HIVE-17537 Move Warehouse class to standalone metastore.
This closes #252. (Alan Gates, reviewed by Zoltan Haindrich)
+ git clean -f -d
Removing standalone-metastore/src/gen/org/
+ git checkout master
Already on 'master'
Your branch is up-to-date with 'origin/master'.
+ git reset --hard origin/master
HEAD is now at 5608300 HIVE-17537 Move Warehouse class to standalone metastore.
This closes #252. (Alan Gates, reviewed by Zoltan Haindrich)
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2017-09-20 04:29:05.930
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh
/data/hiveptest/working/scratch/build.patch
error:
a/metastore/src/java/org/apache/hadoop/hive/metastore/ReplChangeManager.java:
No such file or directory
error: a/ql/src/java/org/apache/hadoop/hive/ql/exec/ReplCopyTask.java: No such
file or directory
error: a/ql/src/java/org/apache/hadoop/hive/ql/parse/repl/CopyUtils.java: No
such file or directory
The patch does not appear to apply with p0, p1, or p2
+ exit 1
'
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12887961 - PreCommit-HIVE-Build
> Validation of source file after distcp in repl load
>
>
> Key: HIVE-16898
> URL: https://issues.apache.org/jira/browse/HIVE-16898
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 3.0.0
>Reporter: anishek
>Assignee: Daniel Dai
> Fix For: 3.0.0
>
> Attachments: HIVE-16898.1.patch, HIVE-16898.2.patch,
> HIVE-16898.3.patch, HIVE-16898.4.patch
>
>
> time between deciding the source and destination path for distcp to invoking
> of distcp can have a change of the source file, hence distcp might copy the
> wrong file to destination, hence we should an additional check on the
> checksum of the source file path after distcp finishes to make sure the path
> didnot change during the copy process. if it has take additional steps to
> delete the previous file on destination and copy the new source and repeat
> the same process as above till we copy the correct file.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-16360) Improve "No delta files or original files found to compact in" message
[
https://issues.apache.org/jira/browse/HIVE-16360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16172725#comment-16172725
]
Hive QA commented on HIVE-16360:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12887941/HIVE-16360.01.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/6894/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/6894/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-6894/
Messages:
{noformat}
This message was trimmed, see log for full details
[loading
ZipFileIndexFileObject[/data/hiveptest/working/maven/org/apache/hadoop/hadoop-hdfs/2.8.0/hadoop-hdfs-2.8.0.jar(org/apache/hadoop/hdfs/web/AuthFilter.class)]]
[loading
ZipFileIndexFileObject[/data/hiveptest/working/apache-github-source-source/shims/common/target/hive-shims-common-3.0.0-SNAPSHOT.jar(org/apache/hadoop/hive/shims/Utils.class)]]
[loading
ZipFileIndexFileObject[/data/hiveptest/working/maven/org/apache/hadoop/hadoop-common/2.8.0/hadoop-common-2.8.0.jar(org/apache/hadoop/security/UserGroupInformation.class)]]
[loading
ZipFileIndexFileObject[/data/hiveptest/working/maven/org/apache/hadoop/hadoop-auth/2.8.0/hadoop-auth-2.8.0.jar(org/apache/hadoop/security/authentication/client/PseudoAuthenticator.class)]]
[loading
ZipFileIndexFileObject[/data/hiveptest/working/maven/org/apache/hadoop/hadoop-auth/2.8.0/hadoop-auth-2.8.0.jar(org/apache/hadoop/security/authentication/server/PseudoAuthenticationHandler.class)]]
[loading
ZipFileIndexFileObject[/data/hiveptest/working/maven/org/apache/hadoop/hadoop-common/2.8.0/hadoop-common-2.8.0.jar(org/apache/hadoop/util/GenericOptionsParser.class)]]
[loading
ZipFileIndexFileObject[/data/hiveptest/working/maven/org/eclipse/jetty/jetty-rewrite/9.3.8.v20160314/jetty-rewrite-9.3.8.v20160314.jar(org/eclipse/jetty/rewrite/handler/RedirectPatternRule.class)]]
[loading
ZipFileIndexFileObject[/data/hiveptest/working/maven/org/eclipse/jetty/jetty-rewrite/9.3.8.v20160314/jetty-rewrite-9.3.8.v20160314.jar(org/eclipse/jetty/rewrite/handler/RewriteHandler.class)]]
[loading
ZipFileIndexFileObject[/data/hiveptest/working/maven/org/eclipse/jetty/jetty-server/9.3.8.v20160314/jetty-server-9.3.8.v20160314.jar(org/eclipse/jetty/server/Handler.class)]]
[loading
ZipFileIndexFileObject[/data/hiveptest/working/maven/org/eclipse/jetty/jetty-server/9.3.8.v20160314/jetty-server-9.3.8.v20160314.jar(org/eclipse/jetty/server/Server.class)]]
[loading
ZipFileIndexFileObject[/data/hiveptest/working/maven/org/eclipse/jetty/jetty-server/9.3.8.v20160314/jetty-server-9.3.8.v20160314.jar(org/eclipse/jetty/server/ServerConnector.class)]]
[loading
ZipFileIndexFileObject[/data/hiveptest/working/maven/org/eclipse/jetty/jetty-server/9.3.8.v20160314/jetty-server-9.3.8.v20160314.jar(org/eclipse/jetty/server/handler/HandlerList.class)]]
[loading
ZipFileIndexFileObject[/data/hiveptest/working/maven/org/eclipse/jetty/jetty-servlet/9.3.8.v20160314/jetty-servlet-9.3.8.v20160314.jar(org/eclipse/jetty/servlet/FilterHolder.class)]]
[loading
ZipFileIndexFileObject[/data/hiveptest/working/maven/org/eclipse/jetty/jetty-servlet/9.3.8.v20160314/jetty-servlet-9.3.8.v20160314.jar(org/eclipse/jetty/servlet/ServletContextHandler.class)]]
[loading
ZipFileIndexFileObject[/data/hiveptest/working/maven/org/eclipse/jetty/jetty-servlet/9.3.8.v20160314/jetty-servlet-9.3.8.v20160314.jar(org/eclipse/jetty/servlet/ServletHolder.class)]]
[loading
ZipFileIndexFileObject[/data/hiveptest/working/maven/org/eclipse/jetty/jetty-xml/9.3.8.v20160314/jetty-xml-9.3.8.v20160314.jar(org/eclipse/jetty/xml/XmlConfiguration.class)]]
[loading
ZipFileIndexFileObject[/data/hiveptest/working/maven/org/slf4j/jul-to-slf4j/1.7.10/jul-to-slf4j-1.7.10.jar(org/slf4j/bridge/SLF4JBridgeHandler.class)]]
[loading
ZipFileIndexFileObject[/data/hiveptest/working/maven/javax/servlet/javax.servlet-api/3.1.0/javax.servlet-api-3.1.0.jar(javax/servlet/DispatcherType.class)]]
[loading
ZipFileIndexFileObject[/data/hiveptest/working/maven/javax/servlet/servlet-api/2.5/servlet-api-2.5.jar(javax/servlet/http/HttpServletRequest.class)]]
[loading
ZipFileIndexFileObject[/data/hiveptest/working/apache-github-source-source/common/target/hive-common-3.0.0-SNAPSHOT.jar(org/apache/hadoop/hive/common/classification/InterfaceAudience$LimitedPrivate.class)]]
[loading
ZipFileIndexFileObject[/data/hiveptest/working/apache-github-source-source/common/target/hive-common-3.0.0-SNAPSHOT.jar(org/apache/hadoop/hive/common/classification/InterfaceStability$Unstable.class)]]
[loading
ZipFileIndexFileObject[/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/rt.jar(java/io/ByteArrayOutputStream.class)]]
[loading
ZipFileIndexFileObject[/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/rt.jar(java/io/OutputStream.class)]]
[loading
[jira] [Updated] (HIVE-17402) Provide object location in the HMS notification messages
[ https://issues.apache.org/jira/browse/HIVE-17402?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexander Kolbasov updated HIVE-17402: -- Summary: Provide object location in the HMS notification messages (was: Provide more useful information in the HMS notification messages) > Provide object location in the HMS notification messages > > > Key: HIVE-17402 > URL: https://issues.apache.org/jira/browse/HIVE-17402 > Project: Hive > Issue Type: Improvement > Components: Hive >Affects Versions: 2.2.0 >Reporter: Alexander Kolbasov >Assignee: Alexander Kolbasov > > While working on the Apache Sentry project that uses HMS notifications we > noticed that these notifications are using some useful data - e.g. location > information for the objects. To get around these, ApacheSentry implemented > its own version of events > (https://github.com/apache/sentry/tree/master/sentry-binding/sentry-binding-hive-follower/src/main/java/org/apache/sentry/binding/metastore/messaging/json). > It seems to be a useful information for Hive as well, so why not add it > directly into the standard message factory? -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17542) Make HoS CombineEquivalentWorkResolver Configurable
[
https://issues.apache.org/jira/browse/HIVE-17542?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16172721#comment-16172721
]
Hive QA commented on HIVE-17542:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12887927/HIVE-17542.3.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 12 failed/errored test(s), 11045 tests
executed
*Failed tests:*
{noformat}
TestAccumuloCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=230)
TestDummy - did not produce a TEST-*.xml file (likely timed out) (batchId=230)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[create_view] (batchId=39)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[udf_mask_hash]
(batchId=28)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[union_fast_stats]
(batchId=156)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_explainuser_1]
(batchId=170)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_2]
(batchId=100)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[drop_table_failure2]
(batchId=89)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query14]
(batchId=234)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query23]
(batchId=234)
org.apache.hadoop.hive.ql.TestAcidOnTez.testCtasTezUnion (batchId=215)
org.apache.hadoop.hive.ql.TestAcidOnTez.testNonStandardConversion01
(batchId=215)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/6893/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/6893/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-6893/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 12 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12887927 - PreCommit-HIVE-Build
> Make HoS CombineEquivalentWorkResolver Configurable
> ---
>
> Key: HIVE-17542
> URL: https://issues.apache.org/jira/browse/HIVE-17542
> Project: Hive
> Issue Type: Improvement
> Components: Physical Optimizer, Spark
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
> Attachments: HIVE-17542.1.patch, HIVE-17542.2.patch,
> HIVE-17542.3.patch
>
>
> The {{CombineEquivalentWorkResolver}} is run by default. We should make it
> configurable so that users can disable it in case there are any issues. We
> can enable it by default to preserve backwards compatibility.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-17554) Occurr java.lang.ArithmeticException: / by zero at hplsql component
[
https://issues.apache.org/jira/browse/HIVE-17554?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ZhangBing Lin updated HIVE-17554:
-
Attachment: HIVE-17554.2.patch
> Occurr java.lang.ArithmeticException: / by zero at hplsql component
> ---
>
> Key: HIVE-17554
> URL: https://issues.apache.org/jira/browse/HIVE-17554
> Project: Hive
> Issue Type: Bug
> Components: hpl/sql
>Affects Versions: 3.0.0
>Reporter: ZhangBing Lin
>Assignee: ZhangBing Lin
> Attachments: HIVE-17554.1.patch, HIVE-17554.2.patch
>
>
> When I execute the copyToTable method in hplsql, the following exception
> occurs:
> java.lang.ArithmeticException: / by zero
> at org.apache.hive.hplsql.Copy.copyToTable(Copy.java:170)
> at org.apache.hive.hplsql.Copy.run(Copy.java:93)
> at org.apache.hive.hplsql.Exec.visitCopy_stmt(Exec.java:1284)
> at org.apache.hive.hplsql.Exec.visitCopy_stmt(Exec.java:52)
> at
> org.apache.hive.hplsql.HplsqlParser$Copy_stmtContext.accept(HplsqlParser.java:12148)
> at
> org.antlr.v4.runtime.tree.AbstractParseTreeVisitor.visitChildren(AbstractParseTreeVisitor.java:70)
> at org.apache.hive.hplsql.Exec.visitStmt(Exec.java:994)
> at org.apache.hive.hplsql.Exec.visitStmt(Exec.java:52)
> at
> org.apache.hive.hplsql.HplsqlParser$StmtContext.accept(HplsqlParser.java:1012)
> at
> org.antlr.v4.runtime.tree.AbstractParseTreeVisitor.visitChildren(AbstractParseTreeVisitor.java:70)
> at
> org.apache.hive.hplsql.HplsqlBaseVisitor.visitBlock(HplsqlBaseVisitor.java:28)
> at
> org.apache.hive.hplsql.HplsqlParser$BlockContext.accept(HplsqlParser.java:446)
> at
> org.antlr.v4.runtime.tree.AbstractParseTreeVisitor.visitChildren(AbstractParseTreeVisitor.java:70)
> at org.apache.hive.hplsql.Exec.visitProgram(Exec.java:901)
> at org.apache.hive.hplsql.Exec.visitProgram(Exec.java:52)
> at
> org.apache.hive.hplsql.HplsqlParser$ProgramContext.accept(HplsqlParser.java:389)
> at
> org.antlr.v4.runtime.tree.AbstractParseTreeVisitor.visit(AbstractParseTreeVisitor.java:42)
> at org.apache.hive.hplsql.Exec.run(Exec.java:760)
> at org.apache.hive.hplsql.Exec.run(Exec.java:736)
> at org.apache.hive.hplsql.Hplsql.main(Hplsql.java:23)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
> View source code:
> if (info) {
> info(ctx, "COPY completed: " + rows + " row(s), " + timer.format() + ",
> " + rows/(elapsed/1000) + " rows/sec");
> }
> When elapsed less than one thousand will cause the divisor to be zero
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-17554) Occurr java.lang.ArithmeticException: / by zero at hplsql component
[
https://issues.apache.org/jira/browse/HIVE-17554?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ZhangBing Lin updated HIVE-17554:
-
Status: Patch Available (was: Open)
> Occurr java.lang.ArithmeticException: / by zero at hplsql component
> ---
>
> Key: HIVE-17554
> URL: https://issues.apache.org/jira/browse/HIVE-17554
> Project: Hive
> Issue Type: Bug
> Components: hpl/sql
>Affects Versions: 3.0.0
>Reporter: ZhangBing Lin
>Assignee: ZhangBing Lin
> Attachments: HIVE-17554.1.patch, HIVE-17554.2.patch
>
>
> When I execute the copyToTable method in hplsql, the following exception
> occurs:
> java.lang.ArithmeticException: / by zero
> at org.apache.hive.hplsql.Copy.copyToTable(Copy.java:170)
> at org.apache.hive.hplsql.Copy.run(Copy.java:93)
> at org.apache.hive.hplsql.Exec.visitCopy_stmt(Exec.java:1284)
> at org.apache.hive.hplsql.Exec.visitCopy_stmt(Exec.java:52)
> at
> org.apache.hive.hplsql.HplsqlParser$Copy_stmtContext.accept(HplsqlParser.java:12148)
> at
> org.antlr.v4.runtime.tree.AbstractParseTreeVisitor.visitChildren(AbstractParseTreeVisitor.java:70)
> at org.apache.hive.hplsql.Exec.visitStmt(Exec.java:994)
> at org.apache.hive.hplsql.Exec.visitStmt(Exec.java:52)
> at
> org.apache.hive.hplsql.HplsqlParser$StmtContext.accept(HplsqlParser.java:1012)
> at
> org.antlr.v4.runtime.tree.AbstractParseTreeVisitor.visitChildren(AbstractParseTreeVisitor.java:70)
> at
> org.apache.hive.hplsql.HplsqlBaseVisitor.visitBlock(HplsqlBaseVisitor.java:28)
> at
> org.apache.hive.hplsql.HplsqlParser$BlockContext.accept(HplsqlParser.java:446)
> at
> org.antlr.v4.runtime.tree.AbstractParseTreeVisitor.visitChildren(AbstractParseTreeVisitor.java:70)
> at org.apache.hive.hplsql.Exec.visitProgram(Exec.java:901)
> at org.apache.hive.hplsql.Exec.visitProgram(Exec.java:52)
> at
> org.apache.hive.hplsql.HplsqlParser$ProgramContext.accept(HplsqlParser.java:389)
> at
> org.antlr.v4.runtime.tree.AbstractParseTreeVisitor.visit(AbstractParseTreeVisitor.java:42)
> at org.apache.hive.hplsql.Exec.run(Exec.java:760)
> at org.apache.hive.hplsql.Exec.run(Exec.java:736)
> at org.apache.hive.hplsql.Hplsql.main(Hplsql.java:23)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
> View source code:
> if (info) {
> info(ctx, "COPY completed: " + rows + " row(s), " + timer.format() + ",
> " + rows/(elapsed/1000) + " rows/sec");
> }
> When elapsed less than one thousand will cause the divisor to be zero
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-17554) Occurr java.lang.ArithmeticException: / by zero at hplsql component
[
https://issues.apache.org/jira/browse/HIVE-17554?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ZhangBing Lin updated HIVE-17554:
-
Status: Open (was: Patch Available)
> Occurr java.lang.ArithmeticException: / by zero at hplsql component
> ---
>
> Key: HIVE-17554
> URL: https://issues.apache.org/jira/browse/HIVE-17554
> Project: Hive
> Issue Type: Bug
> Components: hpl/sql
>Affects Versions: 3.0.0
>Reporter: ZhangBing Lin
>Assignee: ZhangBing Lin
> Attachments: HIVE-17554.1.patch
>
>
> When I execute the copyToTable method in hplsql, the following exception
> occurs:
> java.lang.ArithmeticException: / by zero
> at org.apache.hive.hplsql.Copy.copyToTable(Copy.java:170)
> at org.apache.hive.hplsql.Copy.run(Copy.java:93)
> at org.apache.hive.hplsql.Exec.visitCopy_stmt(Exec.java:1284)
> at org.apache.hive.hplsql.Exec.visitCopy_stmt(Exec.java:52)
> at
> org.apache.hive.hplsql.HplsqlParser$Copy_stmtContext.accept(HplsqlParser.java:12148)
> at
> org.antlr.v4.runtime.tree.AbstractParseTreeVisitor.visitChildren(AbstractParseTreeVisitor.java:70)
> at org.apache.hive.hplsql.Exec.visitStmt(Exec.java:994)
> at org.apache.hive.hplsql.Exec.visitStmt(Exec.java:52)
> at
> org.apache.hive.hplsql.HplsqlParser$StmtContext.accept(HplsqlParser.java:1012)
> at
> org.antlr.v4.runtime.tree.AbstractParseTreeVisitor.visitChildren(AbstractParseTreeVisitor.java:70)
> at
> org.apache.hive.hplsql.HplsqlBaseVisitor.visitBlock(HplsqlBaseVisitor.java:28)
> at
> org.apache.hive.hplsql.HplsqlParser$BlockContext.accept(HplsqlParser.java:446)
> at
> org.antlr.v4.runtime.tree.AbstractParseTreeVisitor.visitChildren(AbstractParseTreeVisitor.java:70)
> at org.apache.hive.hplsql.Exec.visitProgram(Exec.java:901)
> at org.apache.hive.hplsql.Exec.visitProgram(Exec.java:52)
> at
> org.apache.hive.hplsql.HplsqlParser$ProgramContext.accept(HplsqlParser.java:389)
> at
> org.antlr.v4.runtime.tree.AbstractParseTreeVisitor.visit(AbstractParseTreeVisitor.java:42)
> at org.apache.hive.hplsql.Exec.run(Exec.java:760)
> at org.apache.hive.hplsql.Exec.run(Exec.java:736)
> at org.apache.hive.hplsql.Hplsql.main(Hplsql.java:23)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
> View source code:
> if (info) {
> info(ctx, "COPY completed: " + rows + " row(s), " + timer.format() + ",
> " + rows/(elapsed/1000) + " rows/sec");
> }
> When elapsed less than one thousand will cause the divisor to be zero
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-17554) Occurr java.lang.ArithmeticException: / by zero at hplsql component
[
https://issues.apache.org/jira/browse/HIVE-17554?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16172676#comment-16172676
]
ZhangBing Lin commented on HIVE-17554:
--
Hi,[~lirui],I think you are right,I update the patch.
> Occurr java.lang.ArithmeticException: / by zero at hplsql component
> ---
>
> Key: HIVE-17554
> URL: https://issues.apache.org/jira/browse/HIVE-17554
> Project: Hive
> Issue Type: Bug
> Components: hpl/sql
>Affects Versions: 3.0.0
>Reporter: ZhangBing Lin
>Assignee: ZhangBing Lin
> Attachments: HIVE-17554.1.patch
>
>
> When I execute the copyToTable method in hplsql, the following exception
> occurs:
> java.lang.ArithmeticException: / by zero
> at org.apache.hive.hplsql.Copy.copyToTable(Copy.java:170)
> at org.apache.hive.hplsql.Copy.run(Copy.java:93)
> at org.apache.hive.hplsql.Exec.visitCopy_stmt(Exec.java:1284)
> at org.apache.hive.hplsql.Exec.visitCopy_stmt(Exec.java:52)
> at
> org.apache.hive.hplsql.HplsqlParser$Copy_stmtContext.accept(HplsqlParser.java:12148)
> at
> org.antlr.v4.runtime.tree.AbstractParseTreeVisitor.visitChildren(AbstractParseTreeVisitor.java:70)
> at org.apache.hive.hplsql.Exec.visitStmt(Exec.java:994)
> at org.apache.hive.hplsql.Exec.visitStmt(Exec.java:52)
> at
> org.apache.hive.hplsql.HplsqlParser$StmtContext.accept(HplsqlParser.java:1012)
> at
> org.antlr.v4.runtime.tree.AbstractParseTreeVisitor.visitChildren(AbstractParseTreeVisitor.java:70)
> at
> org.apache.hive.hplsql.HplsqlBaseVisitor.visitBlock(HplsqlBaseVisitor.java:28)
> at
> org.apache.hive.hplsql.HplsqlParser$BlockContext.accept(HplsqlParser.java:446)
> at
> org.antlr.v4.runtime.tree.AbstractParseTreeVisitor.visitChildren(AbstractParseTreeVisitor.java:70)
> at org.apache.hive.hplsql.Exec.visitProgram(Exec.java:901)
> at org.apache.hive.hplsql.Exec.visitProgram(Exec.java:52)
> at
> org.apache.hive.hplsql.HplsqlParser$ProgramContext.accept(HplsqlParser.java:389)
> at
> org.antlr.v4.runtime.tree.AbstractParseTreeVisitor.visit(AbstractParseTreeVisitor.java:42)
> at org.apache.hive.hplsql.Exec.run(Exec.java:760)
> at org.apache.hive.hplsql.Exec.run(Exec.java:736)
> at org.apache.hive.hplsql.Hplsql.main(Hplsql.java:23)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
> View source code:
> if (info) {
> info(ctx, "COPY completed: " + rows + " row(s), " + timer.format() + ",
> " + rows/(elapsed/1000) + " rows/sec");
> }
> When elapsed less than one thousand will cause the divisor to be zero
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-17510) Make comparison of filter predicates in q files deterministic
[
https://issues.apache.org/jira/browse/HIVE-17510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16172656#comment-16172656
]
Jesus Camacho Rodriguez commented on HIVE-17510:
I have regenerated all q files. [~ashutoshc], could you take a look since this
is blocking HIVE-17432? Thanks
https://reviews.apache.org/r/62428/
> Make comparison of filter predicates in q files deterministic
> -
>
> Key: HIVE-17510
> URL: https://issues.apache.org/jira/browse/HIVE-17510
> Project: Hive
> Issue Type: Bug
> Components: Testing Infrastructure
>Affects Versions: 3.0.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
> Attachments: HIVE-17510.01.patch, HIVE-17510.patch
>
>
> I have been hitting this issue while submitting patches to test HIVE-17432.
> Basically, the order on which the rewriting might create the children of AND
> operations is not deterministic. Thus, tests might fail because the created
> golden file is not the same, thought the test should pass because they just
> simply do not follow same order:
> {code}
> predicate: ((d_year >= 1992) and (d_year <= 1997) and ((c_city = 'UNITED
> KI1') or (c_city = 'UNITED KI5')) and ((s_city = 'UNITED KI1') or (s_city =
> 'UNITED KI5'))) (type: boolean)
> {code}
> {code}
> predicate: ((d_year <= 1997) and (d_year >= 1992) and ((c_city = 'UNITED
> KI1') or (c_city = 'UNITED KI5')) and ((s_city = 'UNITED KI1') or (s_city =
> 'UNITED KI5'))) (type: boolean)
> {code}
> This patches fixes the issue by sorting the children of some expressions
> (currently AND and OR children) when we run explain plan and we are running
> in test mode.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-17510) Make comparison of filter predicates in q files deterministic
[
https://issues.apache.org/jira/browse/HIVE-17510?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jesus Camacho Rodriguez updated HIVE-17510:
---
Attachment: HIVE-17510.01.patch
> Make comparison of filter predicates in q files deterministic
> -
>
> Key: HIVE-17510
> URL: https://issues.apache.org/jira/browse/HIVE-17510
> Project: Hive
> Issue Type: Bug
> Components: Testing Infrastructure
>Affects Versions: 3.0.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
> Attachments: HIVE-17510.01.patch, HIVE-17510.patch
>
>
> I have been hitting this issue while submitting patches to test HIVE-17432.
> Basically, the order on which the rewriting might create the children of AND
> operations is not deterministic. Thus, tests might fail because the created
> golden file is not the same, thought the test should pass because they just
> simply do not follow same order:
> {code}
> predicate: ((d_year >= 1992) and (d_year <= 1997) and ((c_city = 'UNITED
> KI1') or (c_city = 'UNITED KI5')) and ((s_city = 'UNITED KI1') or (s_city =
> 'UNITED KI5'))) (type: boolean)
> {code}
> {code}
> predicate: ((d_year <= 1997) and (d_year >= 1992) and ((c_city = 'UNITED
> KI1') or (c_city = 'UNITED KI5')) and ((s_city = 'UNITED KI1') or (s_city =
> 'UNITED KI5'))) (type: boolean)
> {code}
> This patches fixes the issue by sorting the children of some expressions
> (currently AND and OR children) when we run explain plan and we are running
> in test mode.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-15899) Make CTAS with acid target table and insert into acid_tbl select ... union all ... work
[
https://issues.apache.org/jira/browse/HIVE-15899?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16172654#comment-16172654
]
Hive QA commented on HIVE-15899:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12887985/HIVE-15899.13.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 8 failed/errored test(s), 11049 tests
executed
*Failed tests:*
{noformat}
TestAccumuloCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=230)
TestDummy - did not produce a TEST-*.xml file (likely timed out) (batchId=230)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[create_view] (batchId=39)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[udf_mask_hash]
(batchId=28)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[union_fast_stats]
(batchId=156)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_explainuser_1]
(batchId=170)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[drop_table_failure2]
(batchId=89)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query14]
(batchId=234)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/6892/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/6892/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-6892/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 8 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12887985 - PreCommit-HIVE-Build
> Make CTAS with acid target table and insert into acid_tbl select ... union
> all ... work
> ---
>
> Key: HIVE-15899
> URL: https://issues.apache.org/jira/browse/HIVE-15899
> Project: Hive
> Issue Type: Sub-task
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
> Attachments: HIVE-15899.01.patch, HIVE-15899.02.patch,
> HIVE-15899.03.patch, HIVE-15899.04.patch, HIVE-15899.05.patch,
> HIVE-15899.07.patch, HIVE-15899.08.patch, HIVE-15899.09.patch,
> HIVE-15899.10.patch, HIVE-15899.11.patch, HIVE-15899.12.patch,
> HIVE-15899.13.patch
>
>
> Consider:
> {noformat}
> create table T stored as ORC TBLPROPERTIES('transactional'='true') as
> select a, b from A where a <= 5 union all select a, b from B where a >=
> 5
> {noformat}
> and
> {noformat}
> create table T (a int, b int) stored as ORC TBLPROPERTIES
> ('transactional'='false';
> insert into T(a,b) select a, b from T where a between 1 and 3 group by a, b
> union all select a, b from A where a between 5 and 7 union all select a, b
> from B where a >= 9
> {noformat}
> On Tez, there is an optimization that removes Union All operator writes the
> data into
> subdirectories of T (in this case T is unpartitioned).
> This also happens on MR but requires
> {noformat}
> hiveConf.setBoolVar(HiveConf.ConfVars.HIVE_OPTIMIZE_UNION_REMOVE, true);
> hiveConf.setVar(HiveConf.ConfVars.HIVEFETCHTASKCONVERSION, "none");
> {noformat}
> Need to ensure that when target table is Acid, we generate unique ROW__IDs
> When target is not acid, that we can convert it to Acid via Alter Table even
> when data layout includes subdirectories.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-17554) Occurr java.lang.ArithmeticException: / by zero at hplsql component
[
https://issues.apache.org/jira/browse/HIVE-17554?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16172653#comment-16172653
]
Rui Li commented on HIVE-17554:
---
Hi [~linzhangbing], I think it'd be better to use a double (together with a
DecimalFormat) to represent {{rows/sec}}. What do you think?
> Occurr java.lang.ArithmeticException: / by zero at hplsql component
> ---
>
> Key: HIVE-17554
> URL: https://issues.apache.org/jira/browse/HIVE-17554
> Project: Hive
> Issue Type: Bug
> Components: hpl/sql
>Affects Versions: 3.0.0
>Reporter: ZhangBing Lin
>Assignee: ZhangBing Lin
> Attachments: HIVE-17554.1.patch
>
>
> When I execute the copyToTable method in hplsql, the following exception
> occurs:
> java.lang.ArithmeticException: / by zero
> at org.apache.hive.hplsql.Copy.copyToTable(Copy.java:170)
> at org.apache.hive.hplsql.Copy.run(Copy.java:93)
> at org.apache.hive.hplsql.Exec.visitCopy_stmt(Exec.java:1284)
> at org.apache.hive.hplsql.Exec.visitCopy_stmt(Exec.java:52)
> at
> org.apache.hive.hplsql.HplsqlParser$Copy_stmtContext.accept(HplsqlParser.java:12148)
> at
> org.antlr.v4.runtime.tree.AbstractParseTreeVisitor.visitChildren(AbstractParseTreeVisitor.java:70)
> at org.apache.hive.hplsql.Exec.visitStmt(Exec.java:994)
> at org.apache.hive.hplsql.Exec.visitStmt(Exec.java:52)
> at
> org.apache.hive.hplsql.HplsqlParser$StmtContext.accept(HplsqlParser.java:1012)
> at
> org.antlr.v4.runtime.tree.AbstractParseTreeVisitor.visitChildren(AbstractParseTreeVisitor.java:70)
> at
> org.apache.hive.hplsql.HplsqlBaseVisitor.visitBlock(HplsqlBaseVisitor.java:28)
> at
> org.apache.hive.hplsql.HplsqlParser$BlockContext.accept(HplsqlParser.java:446)
> at
> org.antlr.v4.runtime.tree.AbstractParseTreeVisitor.visitChildren(AbstractParseTreeVisitor.java:70)
> at org.apache.hive.hplsql.Exec.visitProgram(Exec.java:901)
> at org.apache.hive.hplsql.Exec.visitProgram(Exec.java:52)
> at
> org.apache.hive.hplsql.HplsqlParser$ProgramContext.accept(HplsqlParser.java:389)
> at
> org.antlr.v4.runtime.tree.AbstractParseTreeVisitor.visit(AbstractParseTreeVisitor.java:42)
> at org.apache.hive.hplsql.Exec.run(Exec.java:760)
> at org.apache.hive.hplsql.Exec.run(Exec.java:736)
> at org.apache.hive.hplsql.Hplsql.main(Hplsql.java:23)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
> View source code:
> if (info) {
> info(ctx, "COPY completed: " + rows + " row(s), " + timer.format() + ",
> " + rows/(elapsed/1000) + " rows/sec");
> }
> When elapsed less than one thousand will cause the divisor to be zero
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-17549) Use SHA-256 for RowContainer to improve security
[ https://issues.apache.org/jira/browse/HIVE-17549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16172650#comment-16172650 ] Rui Li commented on HIVE-17549: --- Thanks for the explanations [~txhsj]. It seems row container doesn't make efforts to secure the rows written to disk. That's why I'm not sure whether we need a stronger digest for the file name. [~taoli-hwx] what's your opinions about this? > Use SHA-256 for RowContainer to improve security > > > Key: HIVE-17549 > URL: https://issues.apache.org/jira/browse/HIVE-17549 > Project: Hive > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: Saijin Huang >Assignee: Saijin Huang > Attachments: HIVE-17549.1.patch > > > Use SHA-256 to replace md5 for RowContainer to improve security -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17541) Move testing related methods from MetaStoreUtils to some testing related utility
[
https://issues.apache.org/jira/browse/HIVE-17541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16172637#comment-16172637
]
Alan Gates commented on HIVE-17541:
---
Yes.
> Move testing related methods from MetaStoreUtils to some testing related
> utility
>
>
> Key: HIVE-17541
> URL: https://issues.apache.org/jira/browse/HIVE-17541
> Project: Hive
> Issue Type: Sub-task
> Components: Metastore
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
> Attachments: HIVE-17541.01.patch, HIVE-17541.02.patch
>
>
> MetaStoreUtils has a very wide range of methods...when the last time tried to
> do some modularization related with it - it always came back problematic :)
> The most usefull observation I made that it doesn't neccessarily needs the
> {{HMSHandler}} import.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-16084) SHOW COMPACTIONS should display CompactionID
[ https://issues.apache.org/jira/browse/HIVE-16084?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman updated HIVE-16084: -- Attachment: HIVE-16084.01.patch > SHOW COMPACTIONS should display CompactionID > > > Key: HIVE-16084 > URL: https://issues.apache.org/jira/browse/HIVE-16084 > Project: Hive > Issue Type: Improvement > Components: Transactions >Reporter: Eugene Koifman >Assignee: Eugene Koifman > Attachments: HIVE-16084.01.patch > > > together with HIVE-13353 it will let users search for specific job -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-16084) SHOW COMPACTIONS should display CompactionID
[ https://issues.apache.org/jira/browse/HIVE-16084?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman updated HIVE-16084: -- Status: Patch Available (was: Open) > SHOW COMPACTIONS should display CompactionID > > > Key: HIVE-16084 > URL: https://issues.apache.org/jira/browse/HIVE-16084 > Project: Hive > Issue Type: Improvement > Components: Transactions >Reporter: Eugene Koifman >Assignee: Eugene Koifman > Attachments: HIVE-16084.01.patch > > > together with HIVE-13353 it will let users search for specific job -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-15212) merge branch into master
[ https://issues.apache.org/jira/browse/HIVE-15212?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-15212: Attachment: HIVE-15212.15.patch > merge branch into master > > > Key: HIVE-15212 > URL: https://issues.apache.org/jira/browse/HIVE-15212 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-15212.01.patch, HIVE-15212.02.patch, > HIVE-15212.03.patch, HIVE-15212.04.patch, HIVE-15212.05.patch, > HIVE-15212.06.patch, HIVE-15212.07.patch, HIVE-15212.08.patch, > HIVE-15212.09.patch, HIVE-15212.10.patch, HIVE-15212.11.patch, > HIVE-15212.12.patch, HIVE-15212.12.patch, HIVE-15212.13.patch, > HIVE-15212.13.patch, HIVE-15212.14.patch, HIVE-15212.15.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17537) Move Warehouse class to standalone metastore
[ https://issues.apache.org/jira/browse/HIVE-17537?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16172593#comment-16172593 ] Sergey Shelukhin commented on HIVE-17537: - Is it possible to make these commits in the branch? Or maybe not commit them so often. It's impossible to get HiveQA runs for any moderately complicated patch because of the move-related conflicts > Move Warehouse class to standalone metastore > > > Key: HIVE-17537 > URL: https://issues.apache.org/jira/browse/HIVE-17537 > Project: Hive > Issue Type: Sub-task > Components: Metastore >Reporter: Alan Gates >Assignee: Alan Gates > Labels: pull-request-available > Fix For: 3.0.0 > > Attachments: HIVE-17537.patch > > > Move the Warehouse class. This is done in its own JIRA as it was somewhat > more involved than some of the other classes. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17519) Transpose column stats display
[
https://issues.apache.org/jira/browse/HIVE-17519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16172589#comment-16172589
]
Hive QA commented on HIVE-17519:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12887899/HIVE-17519.02.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 16 failed/errored test(s), 11044 tests
executed
*Failed tests:*
{noformat}
TestAccumuloCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=230)
TestDummy - did not produce a TEST-*.xml file (likely timed out) (batchId=230)
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[escape_comments]
(batchId=239)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[create_view] (batchId=39)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[udf_mask_hash]
(batchId=28)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_outer_join4]
(batchId=82)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[dynpart_sort_opt_vectorization]
(batchId=156)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[dynpart_sort_optimization]
(batchId=156)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_explainuser_1]
(batchId=170)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[drop_table_failure2]
(batchId=89)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query14]
(batchId=234)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query23]
(batchId=234)
org.apache.hadoop.hive.ql.TestAcidOnTez.testCtasTezUnion (batchId=215)
org.apache.hadoop.hive.ql.TestAcidOnTez.testNonStandardConversion01
(batchId=215)
org.apache.hive.jdbc.TestJdbcDriver2.testDescribeTable (batchId=225)
org.apache.hive.jdbc.TestJdbcDriver2.testShowColumns (batchId=225)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/6891/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/6891/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-6891/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 16 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12887899 - PreCommit-HIVE-Build
> Transpose column stats display
> --
>
> Key: HIVE-17519
> URL: https://issues.apache.org/jira/browse/HIVE-17519
> Project: Hive
> Issue Type: Improvement
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
> Attachments: HIVE-17519.01.patch, HIVE-17519.02.patch
>
>
> currently {{describe formatted table1 insert_num}} shows the column
> informations in a table like format...which is very hard to read - because
> there are to many columns
> {code}
> # col_namedata_type min
> max num_nulls distinct_count
> avg_col_len max_col_len num_trues
> num_falses comment bitVector
>
>
> insert_numint
>
>
> from deserializer
> {code}
> I think it would be better to show the same information like this:
> {code}
> col_name insert_num
> data_type int
> min
> max
> num_nulls
> distinct_count
> avg_col_len
> max_col_len
> num_trues
> num_falses
> comment from deserializer
> bitVector
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-17466) Metastore API to list unique partition-key-value combinations
[
https://issues.apache.org/jira/browse/HIVE-17466?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16172572#comment-16172572
]
Lefty Leverenz commented on HIVE-17466:
---
Does this need to be documented in the wiki?
* [Metastore Administration |
https://cwiki.apache.org/confluence/display/Hive/AdminManual+MetastoreAdmin]
> Metastore API to list unique partition-key-value combinations
> -
>
> Key: HIVE-17466
> URL: https://issues.apache.org/jira/browse/HIVE-17466
> Project: Hive
> Issue Type: New Feature
> Components: Metastore
>Affects Versions: 2.2.0, 3.0.0
>Reporter: Mithun Radhakrishnan
>Assignee: Thiruvel Thirumoolan
> Fix For: 3.0.0, 2.4.0
>
> Attachments: HIVE-17466.1.patch, HIVE-17466.2-branch-2.patch,
> HIVE-17466.2.patch, HIVE-17466.3.patch
>
>
> Raising this on behalf of [~thiruvel], who wrote this initially as part of a
> tangential "data-discovery" system.
> Programs like Apache Oozie, Apache Falcon (or Yahoo GDM), etc. launch
> workflows based on the availability of table/partitions. Partitions are
> currently discovered by listing partitions using (what boils down to)
> {{HiveMetaStoreClient.listPartitions()}}. This can be slow and cumbersome,
> given that {{Partition}} objects are heavyweight and carry redundant
> information. The alternative is to use partition-names, which will need
> client-side parsing to extract part-key values.
> When checking which hourly partitions for a particular day have been
> published already, it would be preferable to have an API that pushed down
> part-key extraction into the {{RawStore}} layer, and returned key-values as
> the result. This would be similar to how {{SELECT DISTINCT part_key FROM
> my_table;}} would run, but at the {{HiveMetaStoreClient}} level.
> Here's what we've been using at Yahoo.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-17502) Reuse of default session should not throw an exception in LLAP w/ Tez
[
https://issues.apache.org/jira/browse/HIVE-17502?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16172568#comment-16172568
]
Sergey Shelukhin commented on HIVE-17502:
-
[~thai.bui] there's a little bit of confusion because there are 2 separate
sessions, and 3 objects, involved.
There's Hive (HS2) session, related to the JDBC connection and represented by
HiveSession(Impl) and SessionState (separate for historical reasons that are
probably not relevant anymore), and then there's TezClient, that represents the
Tez session and AM.
The patch addresses the reuse of the latter correctly; it skips the one in use
and returns a new one instead. However, it still uses the same Hive session
(incl. SessionState) in parallel, which is also not supported.
Is it possible to change Hue to only issue one query at a time per JDBC/Hive
session? It might be possible to add behavior to Hive to catch such behavior
and to use/open a separate Hive session (starting with HiveSessionImpl...,
etc.), but I'm not sure how practical that is and it's doesn't seen to me like
a good solution.
> Reuse of default session should not throw an exception in LLAP w/ Tez
> -
>
> Key: HIVE-17502
> URL: https://issues.apache.org/jira/browse/HIVE-17502
> Project: Hive
> Issue Type: Bug
> Components: llap, Tez
>Affects Versions: 2.1.1, 2.2.0
> Environment: HDP 2.6.1.0-129, Hue 4
>Reporter: Thai Bui
>Assignee: Thai Bui
>
> Hive2 w/ LLAP on Tez doesn't allow a currently used, default session to be
> skipped mostly because of this line
> https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionPoolManager.java#L365.
> However, some clients such as Hue 4, allow multiple sessions to be used per
> user. Under this configuration, a Thrift client will send a request to either
> reuse or open a new session. The reuse request could include the session id
> of a currently used snippet being executed in Hue, this causes HS2 to throw
> an exception:
> {noformat}
> 2017-09-10T17:51:36,548 INFO [Thread-89]: tez.TezSessionPoolManager
> (TezSessionPoolManager.java:canWorkWithSameSession(512)) - The current user:
> hive, session user: hive
> 2017-09-10T17:51:36,549 ERROR [Thread-89]: exec.Task
> (TezTask.java:execute(232)) - Failed to execute tez graph.
> org.apache.hadoop.hive.ql.metadata.HiveException: The pool session
> sessionId=5b61a578-6336-41c5-860d-9838166f97fe, queueName=llap, user=hive,
> doAs=false, isOpen=true, isDefault=true, expires in 591015330ms should have
> been returned to the pool
> at
> org.apache.hadoop.hive.ql.exec.tez.TezSessionPoolManager.canWorkWithSameSession(TezSessionPoolManager.java:534)
> ~[hive-exec-2.1.0.2.6.1.0-129.jar:2.1.0.2.6.1.0-129]
> at
> org.apache.hadoop.hive.ql.exec.tez.TezSessionPoolManager.getSession(TezSessionPoolManager.java:544)
> ~[hive-exec-2.1.0.2.6.1.0-129.jar:2.1.0.2.6.1.0-129]
> at org.apache.hadoop.hive.ql.exec.tez.TezTask.execute(TezTask.java:147)
> [hive-exec-2.1.0.2.6.1.0-129.jar:2.1.0.2.6.1.0-129]
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:197)
> [hive-exec-2.1.0.2.6.1.0-129.jar:2.1.0.2.6.1.0-129]
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:100)
> [hive-exec-2.1.0.2.6.1.0-129.jar:2.1.0.2.6.1.0-129]
> at org.apache.hadoop.hive.ql.exec.TaskRunner.run(TaskRunner.java:79)
> [hive-exec-2.1.0.2.6.1.0-129.jar:2.1.0.2.6.1.0-129]
> {noformat}
> Note that every query is issued as a single 'hive' user to share the LLAP
> daemon pool, a set of pre-determined number of AMs is initialized at setup
> time. Thus, HS2 should allow new sessions from a Thrift client to be used out
> of the pool, or an existing session to be skipped and an unused session from
> the pool to be returned. The logic to throw an exception in the
> `canWorkWithSameSession` doesn't make sense to me.
> I have a solution to fix this issue in my local branch at
> https://github.com/thaibui/hive/commit/078a521b9d0906fe6c0323b63e567f6eee2f3a70.
> When applied, the log will become like so
> {noformat}
> 2017-09-10T09:15:33,578 INFO [Thread-239]: tez.TezSessionPoolManager
> (TezSessionPoolManager.java:canWorkWithSameSession(533)) - Skipping default
> session sessionId=6638b1da-0f8a-405e-85f0-9586f484e6de, queueName=llap,
> user=hive, doAs=false, isOpen=true, isDefault=true, expires in 591868732ms
> since it is being used.
> {noformat}
> A test case is provided in my branch to demonstrate how it works. If possible
> I would like this patch to be applied to version 2.1, 2.2 and master. Since
> we are using 2.1 LLAP in production with Hue 4, this patch is critical to our
> success.
> Alternatively, if this patch is too broad in scope, I
[jira] [Updated] (HIVE-17558) Skip non-native/temporary tables for constraint related scenarios
[ https://issues.apache.org/jira/browse/HIVE-17558?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Li updated HIVE-17558: -- Attachment: HIVE-17558.1.patch > Skip non-native/temporary tables for constraint related scenarios > - > > Key: HIVE-17558 > URL: https://issues.apache.org/jira/browse/HIVE-17558 > Project: Hive > Issue Type: Bug >Reporter: Tao Li >Assignee: Tao Li > Attachments: HIVE-17558.1.patch > > > The change would be similar to HIVE-17422. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17558) Skip non-native/temporary tables for constraint related scenarios
[ https://issues.apache.org/jira/browse/HIVE-17558?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Li updated HIVE-17558: -- Status: Patch Available (was: Open) > Skip non-native/temporary tables for constraint related scenarios > - > > Key: HIVE-17558 > URL: https://issues.apache.org/jira/browse/HIVE-17558 > Project: Hive > Issue Type: Bug >Reporter: Tao Li >Assignee: Tao Li > Attachments: HIVE-17558.1.patch > > > The change would be similar to HIVE-17422. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17558) Skip non-native/temporary tables for constraint related scenarios
[ https://issues.apache.org/jira/browse/HIVE-17558?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Li updated HIVE-17558: -- Attachment: (was: HIVE-17558.1.patch) > Skip non-native/temporary tables for constraint related scenarios > - > > Key: HIVE-17558 > URL: https://issues.apache.org/jira/browse/HIVE-17558 > Project: Hive > Issue Type: Bug >Reporter: Tao Li >Assignee: Tao Li > > The change would be similar to HIVE-17422. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17558) Skip non-native/temporary tables for constraint related scenarios
[ https://issues.apache.org/jira/browse/HIVE-17558?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Li updated HIVE-17558: -- Status: Open (was: Patch Available) > Skip non-native/temporary tables for constraint related scenarios > - > > Key: HIVE-17558 > URL: https://issues.apache.org/jira/browse/HIVE-17558 > Project: Hive > Issue Type: Bug >Reporter: Tao Li >Assignee: Tao Li > Attachments: HIVE-17558.1.patch > > > The change would be similar to HIVE-17422. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17558) Skip non-native/temporary tables for constraint related scenarios
[ https://issues.apache.org/jira/browse/HIVE-17558?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Li updated HIVE-17558: -- Attachment: HIVE-17558.1.patch > Skip non-native/temporary tables for constraint related scenarios > - > > Key: HIVE-17558 > URL: https://issues.apache.org/jira/browse/HIVE-17558 > Project: Hive > Issue Type: Bug >Reporter: Tao Li >Assignee: Tao Li > Attachments: HIVE-17558.1.patch > > > The change would be similar to HIVE-17422. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (HIVE-17293) ETL split strategy not accounting for empty base and non-empty delta buckets
[ https://issues.apache.org/jira/browse/HIVE-17293?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran resolved HIVE-17293. -- Resolution: Duplicate This is observed only in Acid 1.0. Tracking this issue in HIVE-17562. Closing this as duplicate. > ETL split strategy not accounting for empty base and non-empty delta buckets > > > Key: HIVE-17293 > URL: https://issues.apache.org/jira/browse/HIVE-17293 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 1.3.0, 3.0.0, 2.4.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Critical > > Observed an issue with customer case where there are 2 buckets (bucket_0 > and bucket_1). > Based bucket 0 had some rows whereas bucket 1 was empty. > Delta bucket 0 and 1 had some rows. > ETL split strategy did not generate OrcSplit for bucket 1 even though it had > some rows in delta directories. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17558) Skip non-native/temporary tables for constraint related scenarios
[ https://issues.apache.org/jira/browse/HIVE-17558?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Li updated HIVE-17558: -- Status: Patch Available (was: Open) > Skip non-native/temporary tables for constraint related scenarios > - > > Key: HIVE-17558 > URL: https://issues.apache.org/jira/browse/HIVE-17558 > Project: Hive > Issue Type: Bug >Reporter: Tao Li >Assignee: Tao Li > Attachments: HIVE-17558.1.patch > > > The change would be similar to HIVE-17422. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17562) ACID 1.0 + ETL strategy should treat empty compacted files as uncovered deltas
[ https://issues.apache.org/jira/browse/HIVE-17562?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-17562: - Status: Patch Available (was: Open) > ACID 1.0 + ETL strategy should treat empty compacted files as uncovered deltas > -- > > Key: HIVE-17562 > URL: https://issues.apache.org/jira/browse/HIVE-17562 > Project: Hive > Issue Type: Bug >Affects Versions: 2.4.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran > Attachments: HIVE-17562.1.branch-2.patch > > > In branch-2, with ACID 1.0, following sequence will result in incorrect > results > 1) Set split strategy to ETL > 2) Insert some rows > 3) Delete all rows > 4) Alter table compact MAJOR > 5) Insert some rows > 6) Select * query will not return any rows that is written at last (step 5) > The reason for that, compaction essentially voids the first insert in step 2. > Now when ETL split strategy is chosen, there will not be any stripes in the > base files. So no split gets generated and any subsequent deltas gets ignored. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17562) ACID 1.0 + ETL strategy should treat empty compacted files as uncovered deltas
[ https://issues.apache.org/jira/browse/HIVE-17562?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-17562: - Attachment: HIVE-17562.1.branch-2.patch [~ekoifman] Can you please review this patch? > ACID 1.0 + ETL strategy should treat empty compacted files as uncovered deltas > -- > > Key: HIVE-17562 > URL: https://issues.apache.org/jira/browse/HIVE-17562 > Project: Hive > Issue Type: Bug >Affects Versions: 2.4.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran > Attachments: HIVE-17562.1.branch-2.patch > > > In branch-2, with ACID 1.0, following sequence will result in incorrect > results > 1) Set split strategy to ETL > 2) Insert some rows > 3) Delete all rows > 4) Alter table compact MAJOR > 5) Insert some rows > 6) Select * query will not return any rows that is written at last (step 5) > The reason for that, compaction essentially voids the first insert in step 2. > Now when ETL split strategy is chosen, there will not be any stripes in the > base files. So no split gets generated and any subsequent deltas gets ignored. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Assigned] (HIVE-17562) ACID 1.0 + ETL strategy should treat empty compacted files as uncovered deltas
[ https://issues.apache.org/jira/browse/HIVE-17562?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran reassigned HIVE-17562: > ACID 1.0 + ETL strategy should treat empty compacted files as uncovered deltas > -- > > Key: HIVE-17562 > URL: https://issues.apache.org/jira/browse/HIVE-17562 > Project: Hive > Issue Type: Bug >Affects Versions: 2.4.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran > > In branch-2, with ACID 1.0, following sequence will result in incorrect > results > 1) Set split strategy to ETL > 2) Insert some rows > 3) Delete all rows > 4) Alter table compact MAJOR > 5) Insert some rows > 6) Select * query will not return any rows that is written at last (step 5) > The reason for that, compaction essentially voids the first insert in step 2. > Now when ETL split strategy is chosen, there will not be any stripes in the > base files. So no split gets generated and any subsequent deltas gets ignored. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17554) Occurr java.lang.ArithmeticException: / by zero at hplsql component
[
https://issues.apache.org/jira/browse/HIVE-17554?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ZhangBing Lin updated HIVE-17554:
-
Affects Version/s: 3.0.0
> Occurr java.lang.ArithmeticException: / by zero at hplsql component
> ---
>
> Key: HIVE-17554
> URL: https://issues.apache.org/jira/browse/HIVE-17554
> Project: Hive
> Issue Type: Bug
> Components: hpl/sql
>Affects Versions: 3.0.0
>Reporter: ZhangBing Lin
>Assignee: ZhangBing Lin
> Attachments: HIVE-17554.1.patch
>
>
> When I execute the copyToTable method in hplsql, the following exception
> occurs:
> java.lang.ArithmeticException: / by zero
> at org.apache.hive.hplsql.Copy.copyToTable(Copy.java:170)
> at org.apache.hive.hplsql.Copy.run(Copy.java:93)
> at org.apache.hive.hplsql.Exec.visitCopy_stmt(Exec.java:1284)
> at org.apache.hive.hplsql.Exec.visitCopy_stmt(Exec.java:52)
> at
> org.apache.hive.hplsql.HplsqlParser$Copy_stmtContext.accept(HplsqlParser.java:12148)
> at
> org.antlr.v4.runtime.tree.AbstractParseTreeVisitor.visitChildren(AbstractParseTreeVisitor.java:70)
> at org.apache.hive.hplsql.Exec.visitStmt(Exec.java:994)
> at org.apache.hive.hplsql.Exec.visitStmt(Exec.java:52)
> at
> org.apache.hive.hplsql.HplsqlParser$StmtContext.accept(HplsqlParser.java:1012)
> at
> org.antlr.v4.runtime.tree.AbstractParseTreeVisitor.visitChildren(AbstractParseTreeVisitor.java:70)
> at
> org.apache.hive.hplsql.HplsqlBaseVisitor.visitBlock(HplsqlBaseVisitor.java:28)
> at
> org.apache.hive.hplsql.HplsqlParser$BlockContext.accept(HplsqlParser.java:446)
> at
> org.antlr.v4.runtime.tree.AbstractParseTreeVisitor.visitChildren(AbstractParseTreeVisitor.java:70)
> at org.apache.hive.hplsql.Exec.visitProgram(Exec.java:901)
> at org.apache.hive.hplsql.Exec.visitProgram(Exec.java:52)
> at
> org.apache.hive.hplsql.HplsqlParser$ProgramContext.accept(HplsqlParser.java:389)
> at
> org.antlr.v4.runtime.tree.AbstractParseTreeVisitor.visit(AbstractParseTreeVisitor.java:42)
> at org.apache.hive.hplsql.Exec.run(Exec.java:760)
> at org.apache.hive.hplsql.Exec.run(Exec.java:736)
> at org.apache.hive.hplsql.Hplsql.main(Hplsql.java:23)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
> View source code:
> if (info) {
> info(ctx, "COPY completed: " + rows + " row(s), " + timer.format() + ",
> " + rows/(elapsed/1000) + " rows/sec");
> }
> When elapsed less than one thousand will cause the divisor to be zero
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-17554) Occurr java.lang.ArithmeticException: / by zero at hplsql component
[
https://issues.apache.org/jira/browse/HIVE-17554?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16172540#comment-16172540
]
ZhangBing Lin commented on HIVE-17554:
--
[~lirui],can you plz take a quick review?
> Occurr java.lang.ArithmeticException: / by zero at hplsql component
> ---
>
> Key: HIVE-17554
> URL: https://issues.apache.org/jira/browse/HIVE-17554
> Project: Hive
> Issue Type: Bug
> Components: hpl/sql
>Reporter: ZhangBing Lin
>Assignee: ZhangBing Lin
> Attachments: HIVE-17554.1.patch
>
>
> When I execute the copyToTable method in hplsql, the following exception
> occurs:
> java.lang.ArithmeticException: / by zero
> at org.apache.hive.hplsql.Copy.copyToTable(Copy.java:170)
> at org.apache.hive.hplsql.Copy.run(Copy.java:93)
> at org.apache.hive.hplsql.Exec.visitCopy_stmt(Exec.java:1284)
> at org.apache.hive.hplsql.Exec.visitCopy_stmt(Exec.java:52)
> at
> org.apache.hive.hplsql.HplsqlParser$Copy_stmtContext.accept(HplsqlParser.java:12148)
> at
> org.antlr.v4.runtime.tree.AbstractParseTreeVisitor.visitChildren(AbstractParseTreeVisitor.java:70)
> at org.apache.hive.hplsql.Exec.visitStmt(Exec.java:994)
> at org.apache.hive.hplsql.Exec.visitStmt(Exec.java:52)
> at
> org.apache.hive.hplsql.HplsqlParser$StmtContext.accept(HplsqlParser.java:1012)
> at
> org.antlr.v4.runtime.tree.AbstractParseTreeVisitor.visitChildren(AbstractParseTreeVisitor.java:70)
> at
> org.apache.hive.hplsql.HplsqlBaseVisitor.visitBlock(HplsqlBaseVisitor.java:28)
> at
> org.apache.hive.hplsql.HplsqlParser$BlockContext.accept(HplsqlParser.java:446)
> at
> org.antlr.v4.runtime.tree.AbstractParseTreeVisitor.visitChildren(AbstractParseTreeVisitor.java:70)
> at org.apache.hive.hplsql.Exec.visitProgram(Exec.java:901)
> at org.apache.hive.hplsql.Exec.visitProgram(Exec.java:52)
> at
> org.apache.hive.hplsql.HplsqlParser$ProgramContext.accept(HplsqlParser.java:389)
> at
> org.antlr.v4.runtime.tree.AbstractParseTreeVisitor.visit(AbstractParseTreeVisitor.java:42)
> at org.apache.hive.hplsql.Exec.run(Exec.java:760)
> at org.apache.hive.hplsql.Exec.run(Exec.java:736)
> at org.apache.hive.hplsql.Hplsql.main(Hplsql.java:23)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
> View source code:
> if (info) {
> info(ctx, "COPY completed: " + rows + " row(s), " + timer.format() + ",
> " + rows/(elapsed/1000) + " rows/sec");
> }
> When elapsed less than one thousand will cause the divisor to be zero
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-17554) Occurr java.lang.ArithmeticException: / by zero at hplsql component
[
https://issues.apache.org/jira/browse/HIVE-17554?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16172539#comment-16172539
]
ZhangBing Lin commented on HIVE-17554:
--
Unit tests failed not related to the patch
> Occurr java.lang.ArithmeticException: / by zero at hplsql component
> ---
>
> Key: HIVE-17554
> URL: https://issues.apache.org/jira/browse/HIVE-17554
> Project: Hive
> Issue Type: Bug
> Components: hpl/sql
>Reporter: ZhangBing Lin
>Assignee: ZhangBing Lin
> Attachments: HIVE-17554.1.patch
>
>
> When I execute the copyToTable method in hplsql, the following exception
> occurs:
> java.lang.ArithmeticException: / by zero
> at org.apache.hive.hplsql.Copy.copyToTable(Copy.java:170)
> at org.apache.hive.hplsql.Copy.run(Copy.java:93)
> at org.apache.hive.hplsql.Exec.visitCopy_stmt(Exec.java:1284)
> at org.apache.hive.hplsql.Exec.visitCopy_stmt(Exec.java:52)
> at
> org.apache.hive.hplsql.HplsqlParser$Copy_stmtContext.accept(HplsqlParser.java:12148)
> at
> org.antlr.v4.runtime.tree.AbstractParseTreeVisitor.visitChildren(AbstractParseTreeVisitor.java:70)
> at org.apache.hive.hplsql.Exec.visitStmt(Exec.java:994)
> at org.apache.hive.hplsql.Exec.visitStmt(Exec.java:52)
> at
> org.apache.hive.hplsql.HplsqlParser$StmtContext.accept(HplsqlParser.java:1012)
> at
> org.antlr.v4.runtime.tree.AbstractParseTreeVisitor.visitChildren(AbstractParseTreeVisitor.java:70)
> at
> org.apache.hive.hplsql.HplsqlBaseVisitor.visitBlock(HplsqlBaseVisitor.java:28)
> at
> org.apache.hive.hplsql.HplsqlParser$BlockContext.accept(HplsqlParser.java:446)
> at
> org.antlr.v4.runtime.tree.AbstractParseTreeVisitor.visitChildren(AbstractParseTreeVisitor.java:70)
> at org.apache.hive.hplsql.Exec.visitProgram(Exec.java:901)
> at org.apache.hive.hplsql.Exec.visitProgram(Exec.java:52)
> at
> org.apache.hive.hplsql.HplsqlParser$ProgramContext.accept(HplsqlParser.java:389)
> at
> org.antlr.v4.runtime.tree.AbstractParseTreeVisitor.visit(AbstractParseTreeVisitor.java:42)
> at org.apache.hive.hplsql.Exec.run(Exec.java:760)
> at org.apache.hive.hplsql.Exec.run(Exec.java:736)
> at org.apache.hive.hplsql.Hplsql.main(Hplsql.java:23)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
> View source code:
> if (info) {
> info(ctx, "COPY completed: " + rows + " row(s), " + timer.format() + ",
> " + rows/(elapsed/1000) + " rows/sec");
> }
> When elapsed less than one thousand will cause the divisor to be zero
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-17538) Enhance estimation of stats to estimate even if only one column is missing stats
[ https://issues.apache.org/jira/browse/HIVE-17538?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16172535#comment-16172535 ] Vineet Garg commented on HIVE-17538: Unfortunately the test report is not available anymore so will have to re-run tests to see if they are related or unrelated. > Enhance estimation of stats to estimate even if only one column is missing > stats > > > Key: HIVE-17538 > URL: https://issues.apache.org/jira/browse/HIVE-17538 > Project: Hive > Issue Type: Improvement >Reporter: Vineet Garg >Assignee: Vineet Garg > Attachments: HIVE-17538.1.patch > > > HIVE-16811 provided support for estimating statistics in absence of stats. > But that estimation is done if and only if statistics are missing for all > columns. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-16084) SHOW COMPACTIONS should display CompactionID
[ https://issues.apache.org/jira/browse/HIVE-16084?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman updated HIVE-16084: -- Issue Type: Improvement (was: Test) > SHOW COMPACTIONS should display CompactionID > > > Key: HIVE-16084 > URL: https://issues.apache.org/jira/browse/HIVE-16084 > Project: Hive > Issue Type: Improvement > Components: Transactions >Reporter: Eugene Koifman >Assignee: Eugene Koifman > > together with HIVE-13353 it will let users search for specific job -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17378) CBO: HiveReduceExpressionsWithStatsRule can operate on IS_NULL and IS_NOT_NULL
[
https://issues.apache.org/jira/browse/HIVE-17378?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16172528#comment-16172528
]
Hive QA commented on HIVE-17378:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12887893/HIVE-17378.01.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 12 failed/errored test(s), 11049 tests
executed
*Failed tests:*
{noformat}
TestAccumuloCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=230)
TestDummy - did not produce a TEST-*.xml file (likely timed out) (batchId=230)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_sortmerge_join_2]
(batchId=47)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[create_view] (batchId=39)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[udf_mask_hash]
(batchId=28)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[union_fast_stats]
(batchId=156)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_explainuser_1]
(batchId=170)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_2]
(batchId=100)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[drop_table_failure2]
(batchId=89)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query14]
(batchId=234)
org.apache.hadoop.hive.ql.TestAcidOnTez.testCtasTezUnion (batchId=215)
org.apache.hadoop.hive.ql.TestAcidOnTez.testNonStandardConversion01
(batchId=215)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/6890/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/6890/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-6890/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 12 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12887893 - PreCommit-HIVE-Build
> CBO: HiveReduceExpressionsWithStatsRule can operate on IS_NULL and IS_NOT_NULL
> --
>
> Key: HIVE-17378
> URL: https://issues.apache.org/jira/browse/HIVE-17378
> Project: Hive
> Issue Type: Bug
> Components: CBO
>Affects Versions: 3.0.0
>Reporter: Gopal V
>Assignee: Zoltan Haindrich
> Attachments: HIVE-17378.01.patch
>
>
> {code}
> * Currently we support the simplification of =, >=, <=, >, <, and
> * IN operations.
> */
> {code}
> IS_NULL and IS_NOT_NULL are closely related and can be processed by this rule.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-17271) log base/delta for each split
[ https://issues.apache.org/jira/browse/HIVE-17271?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman updated HIVE-17271: -- Description: check to make sure we properly log all files included in the split - not sure if we log the deltas easiest to log base file name, min/max key if any and ValidTxnList need to be careful TxnList - if compactor is not keeping up this could very large was:check to make sure we properly log all files included in the split - not sure if we log the deltas > log base/delta for each split > - > > Key: HIVE-17271 > URL: https://issues.apache.org/jira/browse/HIVE-17271 > Project: Hive > Issue Type: Improvement > Components: Transactions >Reporter: Eugene Koifman >Assignee: Eugene Koifman > > check to make sure we properly log all files included in the split - not sure > if we log the deltas > easiest to log base file name, min/max key if any and ValidTxnList > need to be careful TxnList - if compactor is not keeping up this could very > large -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17537) Move Warehouse class to standalone metastore
[ https://issues.apache.org/jira/browse/HIVE-17537?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alan Gates updated HIVE-17537: -- Resolution: Fixed Fix Version/s: 3.0.0 Status: Resolved (was: Patch Available) Patch committed. Thanks Zoltan for the review. > Move Warehouse class to standalone metastore > > > Key: HIVE-17537 > URL: https://issues.apache.org/jira/browse/HIVE-17537 > Project: Hive > Issue Type: Sub-task > Components: Metastore >Reporter: Alan Gates >Assignee: Alan Gates > Labels: pull-request-available > Fix For: 3.0.0 > > Attachments: HIVE-17537.patch > > > Move the Warehouse class. This is done in its own JIRA as it was somewhat > more involved than some of the other classes. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17537) Move Warehouse class to standalone metastore
[ https://issues.apache.org/jira/browse/HIVE-17537?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16172510#comment-16172510 ] ASF GitHub Bot commented on HIVE-17537: --- Github user asfgit closed the pull request at: https://github.com/apache/hive/pull/252 > Move Warehouse class to standalone metastore > > > Key: HIVE-17537 > URL: https://issues.apache.org/jira/browse/HIVE-17537 > Project: Hive > Issue Type: Sub-task > Components: Metastore >Reporter: Alan Gates >Assignee: Alan Gates > Labels: pull-request-available > Attachments: HIVE-17537.patch > > > Move the Warehouse class. This is done in its own JIRA as it was somewhat > more involved than some of the other classes. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Assigned] (HIVE-17561) Move TxnStore and implementations to standalone metastore
[ https://issues.apache.org/jira/browse/HIVE-17561?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alan Gates reassigned HIVE-17561: - > Move TxnStore and implementations to standalone metastore > - > > Key: HIVE-17561 > URL: https://issues.apache.org/jira/browse/HIVE-17561 > Project: Hive > Issue Type: Sub-task > Components: Metastore, Transactions >Reporter: Alan Gates >Assignee: Alan Gates > > We need to move the metastore handling of transactions into the standalone > metastore. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17536) StatsUtil::getBasicStatForTable doesn't distinguish b/w absence of statistics or zero stats
[ https://issues.apache.org/jira/browse/HIVE-17536?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-17536: --- Status: Patch Available (was: Open) > StatsUtil::getBasicStatForTable doesn't distinguish b/w absence of statistics > or zero stats > --- > > Key: HIVE-17536 > URL: https://issues.apache.org/jira/browse/HIVE-17536 > Project: Hive > Issue Type: Improvement > Components: Statistics >Reporter: Vineet Garg >Assignee: Vineet Garg > Attachments: HIVE-17536.1.patch, HIVE-17536.2.patch, > HIVE-17536.3.patch, HIVE-17536.4.patch > > > This method returns zero for both of the following cases: > * Statistics are missing in metastore > * Actual stats e.g. number of rows are zero > It'll be good for this method to return e.g. -1 in absence of statistics > instead of assuming it to be zero. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17536) StatsUtil::getBasicStatForTable doesn't distinguish b/w absence of statistics or zero stats
[ https://issues.apache.org/jira/browse/HIVE-17536?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-17536: --- Attachment: HIVE-17536.4.patch > StatsUtil::getBasicStatForTable doesn't distinguish b/w absence of statistics > or zero stats > --- > > Key: HIVE-17536 > URL: https://issues.apache.org/jira/browse/HIVE-17536 > Project: Hive > Issue Type: Improvement > Components: Statistics >Reporter: Vineet Garg >Assignee: Vineet Garg > Attachments: HIVE-17536.1.patch, HIVE-17536.2.patch, > HIVE-17536.3.patch, HIVE-17536.4.patch > > > This method returns zero for both of the following cases: > * Statistics are missing in metastore > * Actual stats e.g. number of rows are zero > It'll be good for this method to return e.g. -1 in absence of statistics > instead of assuming it to be zero. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17536) StatsUtil::getBasicStatForTable doesn't distinguish b/w absence of statistics or zero stats
[ https://issues.apache.org/jira/browse/HIVE-17536?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-17536: --- Status: Open (was: Patch Available) > StatsUtil::getBasicStatForTable doesn't distinguish b/w absence of statistics > or zero stats > --- > > Key: HIVE-17536 > URL: https://issues.apache.org/jira/browse/HIVE-17536 > Project: Hive > Issue Type: Improvement > Components: Statistics >Reporter: Vineet Garg >Assignee: Vineet Garg > Attachments: HIVE-17536.1.patch, HIVE-17536.2.patch, > HIVE-17536.3.patch > > > This method returns zero for both of the following cases: > * Statistics are missing in metastore > * Actual stats e.g. number of rows are zero > It'll be good for this method to return e.g. -1 in absence of statistics > instead of assuming it to be zero. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-15899) Make CTAS with acid target table and insert into acid_tbl select ... union all ... work
[
https://issues.apache.org/jira/browse/HIVE-15899?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eugene Koifman updated HIVE-15899:
--
Attachment: HIVE-15899.13.patch
patch13 updates the comments per RB
> Make CTAS with acid target table and insert into acid_tbl select ... union
> all ... work
> ---
>
> Key: HIVE-15899
> URL: https://issues.apache.org/jira/browse/HIVE-15899
> Project: Hive
> Issue Type: Sub-task
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
> Attachments: HIVE-15899.01.patch, HIVE-15899.02.patch,
> HIVE-15899.03.patch, HIVE-15899.04.patch, HIVE-15899.05.patch,
> HIVE-15899.07.patch, HIVE-15899.08.patch, HIVE-15899.09.patch,
> HIVE-15899.10.patch, HIVE-15899.11.patch, HIVE-15899.12.patch,
> HIVE-15899.13.patch
>
>
> Consider:
> {noformat}
> create table T stored as ORC TBLPROPERTIES('transactional'='true') as
> select a, b from A where a <= 5 union all select a, b from B where a >=
> 5
> {noformat}
> and
> {noformat}
> create table T (a int, b int) stored as ORC TBLPROPERTIES
> ('transactional'='false';
> insert into T(a,b) select a, b from T where a between 1 and 3 group by a, b
> union all select a, b from A where a between 5 and 7 union all select a, b
> from B where a >= 9
> {noformat}
> On Tez, there is an optimization that removes Union All operator writes the
> data into
> subdirectories of T (in this case T is unpartitioned).
> This also happens on MR but requires
> {noformat}
> hiveConf.setBoolVar(HiveConf.ConfVars.HIVE_OPTIMIZE_UNION_REMOVE, true);
> hiveConf.setVar(HiveConf.ConfVars.HIVEFETCHTASKCONVERSION, "none");
> {noformat}
> Need to ensure that when target table is Acid, we generate unique ROW__IDs
> When target is not acid, that we can convert it to Acid via Alter Table even
> when data layout includes subdirectories.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-17560) HiveMetastore doesn't start in secure cluster if repl change manager is enabled
[ https://issues.apache.org/jira/browse/HIVE-17560?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16172473#comment-16172473 ] Daniel Dai commented on HIVE-17560: --- +1 > HiveMetastore doesn't start in secure cluster if repl change manager is > enabled > --- > > Key: HIVE-17560 > URL: https://issues.apache.org/jira/browse/HIVE-17560 > Project: Hive > Issue Type: Bug >Reporter: Thejas M Nair >Assignee: Thejas M Nair > Fix For: 3.0.0 > > Attachments: HIVE-17560.1.patch > > > When hive.repl.cm.enabled=true, ReplChangeManager tries to access HDFS before > metastore does kerberos login using keytab. > Metastore startup code doesn't do an explicit login using keytab, but instead > relies on kinit by saslserver for use by thrift to do it. > It would be cleaner to do an explicit UGI.loginFromKeytab instead to avoid > such issues in future as well. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17483) HS2 kill command to kill queries using query id
[
https://issues.apache.org/jira/browse/HIVE-17483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16172468#comment-16172468
]
Hive QA commented on HIVE-17483:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12887881/HIVE-17483.5.patch
{color:green}SUCCESS:{color} +1 due to 6 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 20 failed/errored test(s), 11050 tests
executed
*Failed tests:*
{noformat}
TestAccumuloCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=230)
TestDummy - did not produce a TEST-*.xml file (likely timed out) (batchId=230)
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[smb_mapjoin_13]
(batchId=239)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[create_view] (batchId=39)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[udf_mask_hash]
(batchId=28)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[union_fast_stats]
(batchId=156)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_explainuser_1]
(batchId=170)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[drop_table_failure2]
(batchId=89)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query14]
(batchId=234)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query23]
(batchId=234)
org.apache.hadoop.hive.ql.TestAcidOnTez.testCtasTezUnion (batchId=215)
org.apache.hadoop.hive.ql.TestAcidOnTez.testNonStandardConversion01
(batchId=215)
org.apache.hive.service.cli.session.TestHiveSessionImpl.testLeakOperationHandle
(batchId=223)
org.apache.hive.service.cli.session.TestQueryDisplay.testQueryDisplay
(batchId=223)
org.apache.hive.service.cli.session.TestQueryDisplay.testWebUI (batchId=223)
org.apache.hive.service.cli.session.TestSessionManagerMetrics.testAbandonedSessionMetrics
(batchId=197)
org.apache.hive.service.cli.session.TestSessionManagerMetrics.testActiveSessionMetrics
(batchId=197)
org.apache.hive.service.cli.session.TestSessionManagerMetrics.testActiveSessionTimeMetrics
(batchId=197)
org.apache.hive.service.cli.session.TestSessionManagerMetrics.testOpenSessionMetrics
(batchId=197)
org.apache.hive.service.cli.session.TestSessionManagerMetrics.testOpenSessionTimeMetrics
(batchId=197)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/6889/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/6889/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-6889/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 20 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12887881 - PreCommit-HIVE-Build
> HS2 kill command to kill queries using query id
> ---
>
> Key: HIVE-17483
> URL: https://issues.apache.org/jira/browse/HIVE-17483
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Reporter: Thejas M Nair
>Assignee: Teddy Choi
> Attachments: HIVE-17483.1.patch, HIVE-17483.2.patch,
> HIVE-17483.2.patch, HIVE-17483.3.patch, HIVE-17483.4.patch, HIVE-17483.5.patch
>
>
> For administrators, it is important to be able to kill queries if required.
> Currently, there is no clean way to do it.
> It would help to have a "kill query " command that can be run using
> odbc/jdbc against a HiveServer2 instance, to kill a query with that queryid
> running in that instance.
> Authorization will have to be done to ensure that the user that is invoking
> the API is allowed to perform this action.
> In case of SQL std authorization, this would require admin role.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-17112) Reduce logging in HiveSparkClientFactory and RemoteHiveSparkClient
[
https://issues.apache.org/jira/browse/HIVE-17112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16172442#comment-16172442
]
Sahil Takiar commented on HIVE-17112:
-
CC: [~pvary], [~zsombor.klara] could you review this? What do you think about
making this change?
> Reduce logging in HiveSparkClientFactory and RemoteHiveSparkClient
> --
>
> Key: HIVE-17112
> URL: https://issues.apache.org/jira/browse/HIVE-17112
> Project: Hive
> Issue Type: Improvement
> Components: Spark
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
> Attachments: HIVE-17112.1.patch
>
>
> HiveSparkClientFactory has the following line that introduces excess logging:
> {code}
> LOG.info(String.format(
> "load spark property from %s (%s -> %s).",
> SPARK_DEFAULT_CONF_FILE, propertyName,
> LogUtils.maskIfPassword(propertyName,value)));
> {code}
> It basically dumps the entire configuration object to the logs, we can
> probably change this from INFO to DEBUG.
> Same thing happens in {{RemoteHiveSparkClient#logConfigurations}}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-17542) Make HoS CombineEquivalentWorkResolver Configurable
[
https://issues.apache.org/jira/browse/HIVE-17542?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16172440#comment-16172440
]
Sahil Takiar commented on HIVE-17542:
-
[~pvary] added calculating query results. Attached updated patch.
> Make HoS CombineEquivalentWorkResolver Configurable
> ---
>
> Key: HIVE-17542
> URL: https://issues.apache.org/jira/browse/HIVE-17542
> Project: Hive
> Issue Type: Improvement
> Components: Physical Optimizer, Spark
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
> Attachments: HIVE-17542.1.patch, HIVE-17542.2.patch,
> HIVE-17542.3.patch
>
>
> The {{CombineEquivalentWorkResolver}} is run by default. We should make it
> configurable so that users can disable it in case there are any issues. We
> can enable it by default to preserve backwards compatibility.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-17472) Drop-partition for multi-level partition fails, if data does not exist.
[
https://issues.apache.org/jira/browse/HIVE-17472?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mithun Radhakrishnan updated HIVE-17472:
Status: Open (was: Patch Available)
> Drop-partition for multi-level partition fails, if data does not exist.
> ---
>
> Key: HIVE-17472
> URL: https://issues.apache.org/jira/browse/HIVE-17472
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 2.2.0, 3.0.0
>Reporter: Mithun Radhakrishnan
>Assignee: Chris Drome
> Attachments: HIVE-17472.1.patch, HIVE-17472.2-branch-2.patch,
> HIVE-17472.2.patch, HIVE-17472.3-branch-2.2.patch,
> HIVE-17472.3-branch-2.patch, HIVE-17472.3.patch,
> HIVE-17472.4-branch-2.2.patch, HIVE-17472.4-branch-2.patch, HIVE-17472.4.patch
>
>
> Raising this on behalf of [~cdrome] and [~selinazh].
> Here's how to reproduce the problem:
> {code:sql}
> CREATE TABLE foobar ( foo STRING, bar STRING ) PARTITIONED BY ( dt STRING,
> region STRING ) STORED AS RCFILE LOCATION '/tmp/foobar';
> ALTER TABLE foobar ADD PARTITION ( dt='1', region='A' ) ;
> dfs -rm -R -skipTrash /tmp/foobar/dt=1;
> ALTER TABLE foobar DROP PARTITION ( dt='1' );
> {code}
> This causes a client-side error as follows:
> {code}
> 15/02/26 23:08:32 ERROR exec.DDLTask:
> org.apache.hadoop.hive.ql.metadata.HiveException: Unknown error. Please check
> logs.
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-17472) Drop-partition for multi-level partition fails, if data does not exist.
[
https://issues.apache.org/jira/browse/HIVE-17472?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mithun Radhakrishnan updated HIVE-17472:
Status: Patch Available (was: Open)
> Drop-partition for multi-level partition fails, if data does not exist.
> ---
>
> Key: HIVE-17472
> URL: https://issues.apache.org/jira/browse/HIVE-17472
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 2.2.0, 3.0.0
>Reporter: Mithun Radhakrishnan
>Assignee: Chris Drome
> Attachments: HIVE-17472.1.patch, HIVE-17472.2-branch-2.patch,
> HIVE-17472.2.patch, HIVE-17472.3-branch-2.2.patch,
> HIVE-17472.3-branch-2.patch, HIVE-17472.3.patch,
> HIVE-17472.4-branch-2.2.patch, HIVE-17472.4-branch-2.patch, HIVE-17472.4.patch
>
>
> Raising this on behalf of [~cdrome] and [~selinazh].
> Here's how to reproduce the problem:
> {code:sql}
> CREATE TABLE foobar ( foo STRING, bar STRING ) PARTITIONED BY ( dt STRING,
> region STRING ) STORED AS RCFILE LOCATION '/tmp/foobar';
> ALTER TABLE foobar ADD PARTITION ( dt='1', region='A' ) ;
> dfs -rm -R -skipTrash /tmp/foobar/dt=1;
> ALTER TABLE foobar DROP PARTITION ( dt='1' );
> {code}
> This causes a client-side error as follows:
> {code}
> 15/02/26 23:08:32 ERROR exec.DDLTask:
> org.apache.hadoop.hive.ql.metadata.HiveException: Unknown error. Please check
> logs.
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-17466) Metastore API to list unique partition-key-value combinations
[
https://issues.apache.org/jira/browse/HIVE-17466?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mithun Radhakrishnan updated HIVE-17466:
Resolution: Fixed
Fix Version/s: 2.4.0
3.0.0
Status: Resolved (was: Patch Available)
I've checked this into {{master}}, and {{branch-2}}. [~owen.omalley],
[~thejas], would it be incorrect to check this into {{branch-2.2}}?
> Metastore API to list unique partition-key-value combinations
> -
>
> Key: HIVE-17466
> URL: https://issues.apache.org/jira/browse/HIVE-17466
> Project: Hive
> Issue Type: New Feature
> Components: Metastore
>Affects Versions: 2.2.0, 3.0.0
>Reporter: Mithun Radhakrishnan
>Assignee: Thiruvel Thirumoolan
> Fix For: 3.0.0, 2.4.0
>
> Attachments: HIVE-17466.1.patch, HIVE-17466.2-branch-2.patch,
> HIVE-17466.2.patch, HIVE-17466.3.patch
>
>
> Raising this on behalf of [~thiruvel], who wrote this initially as part of a
> tangential "data-discovery" system.
> Programs like Apache Oozie, Apache Falcon (or Yahoo GDM), etc. launch
> workflows based on the availability of table/partitions. Partitions are
> currently discovered by listing partitions using (what boils down to)
> {{HiveMetaStoreClient.listPartitions()}}. This can be slow and cumbersome,
> given that {{Partition}} objects are heavyweight and carry redundant
> information. The alternative is to use partition-names, which will need
> client-side parsing to extract part-key values.
> When checking which hourly partitions for a particular day have been
> published already, it would be preferable to have an API that pushed down
> part-key extraction into the {{RawStore}} layer, and returned key-values as
> the result. This would be similar to how {{SELECT DISTINCT part_key FROM
> my_table;}} would run, but at the {{HiveMetaStoreClient}} level.
> Here's what we've been using at Yahoo.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-17560) HiveMetastore doesn't start in secure cluster if repl change manager is enabled
[ https://issues.apache.org/jira/browse/HIVE-17560?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thejas M Nair updated HIVE-17560: - Attachment: HIVE-17560.1.patch > HiveMetastore doesn't start in secure cluster if repl change manager is > enabled > --- > > Key: HIVE-17560 > URL: https://issues.apache.org/jira/browse/HIVE-17560 > Project: Hive > Issue Type: Bug >Reporter: Thejas M Nair >Assignee: Thejas M Nair > Fix For: 3.0.0 > > Attachments: HIVE-17560.1.patch > > > When hive.repl.cm.enabled=true, ReplChangeManager tries to access HDFS before > metastore does kerberos login using keytab. > Metastore startup code doesn't do an explicit login using keytab, but instead > relies on kinit by saslserver for use by thrift to do it. > It would be cleaner to do an explicit UGI.loginFromKeytab instead to avoid > such issues in future as well. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17560) HiveMetastore doesn't start in secure cluster if repl change manager is enabled
[ https://issues.apache.org/jira/browse/HIVE-17560?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thejas M Nair updated HIVE-17560: - Status: Patch Available (was: Open) > HiveMetastore doesn't start in secure cluster if repl change manager is > enabled > --- > > Key: HIVE-17560 > URL: https://issues.apache.org/jira/browse/HIVE-17560 > Project: Hive > Issue Type: Bug >Reporter: Thejas M Nair >Assignee: Thejas M Nair > Fix For: 3.0.0 > > Attachments: HIVE-17560.1.patch > > > When hive.repl.cm.enabled=true, ReplChangeManager tries to access HDFS before > metastore does kerberos login using keytab. > Metastore startup code doesn't do an explicit login using keytab, but instead > relies on kinit by saslserver for use by thrift to do it. > It would be cleaner to do an explicit UGI.loginFromKeytab instead to avoid > such issues in future as well. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17530) ClassCastException when converting uniontype
[
https://issues.apache.org/jira/browse/HIVE-17530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16172368#comment-16172368
]
Anthony Hsu commented on HIVE-17530:
Thanks, [~cwsteinbach] and [~rdsr]!
> ClassCastException when converting uniontype
>
>
> Key: HIVE-17530
> URL: https://issues.apache.org/jira/browse/HIVE-17530
> Project: Hive
> Issue Type: Bug
>Affects Versions: 1.1.0, 3.0.0
>Reporter: Anthony Hsu
>Assignee: Anthony Hsu
> Fix For: 3.0.0
>
> Attachments: HIVE-17530.1.patch, HIVE-17530.2.patch
>
>
> To repro:
> {noformat}
> SET hive.exec.schema.evolution = false;
> CREATE TABLE avro_orc_partitioned_uniontype (a uniontype)
> PARTITIONED BY (b int) STORED AS ORC;
> INSERT INTO avro_orc_partitioned_uniontype PARTITION (b=1) SELECT
> create_union(1, true, value) FROM src LIMIT 5;
> ALTER TABLE avro_orc_partitioned_uniontype SET FILEFORMAT AVRO;
> SELECT * FROM avro_orc_partitioned_uniontype;
> {noformat}
> The exception you get is:
> {code}
> java.io.IOException:org.apache.hadoop.hive.ql.metadata.HiveException:
> java.lang.ClassCastException: java.util.ArrayList cannot be cast to
> org.apache.hadoop.hive.serde2.objectinspector.UnionObject
> {code}
> The issue is that StandardUnionObjectInspector was creating and returning an
> ArrayList rather than a UnionObject.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Assigned] (HIVE-17560) HiveMetastore doesn't start in secure cluster if repl change manager is enabled
[ https://issues.apache.org/jira/browse/HIVE-17560?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thejas M Nair reassigned HIVE-17560: > HiveMetastore doesn't start in secure cluster if repl change manager is > enabled > --- > > Key: HIVE-17560 > URL: https://issues.apache.org/jira/browse/HIVE-17560 > Project: Hive > Issue Type: Bug >Reporter: Thejas M Nair >Assignee: Thejas M Nair > Fix For: 3.0.0 > > > When hive.repl.cm.enabled=true, ReplChangeManager tries to access HDFS before > metastore does kerberos login using keytab. > Metastore startup code doesn't do an explicit login using keytab, but instead > relies on kinit by saslserver for use by thrift to do it. > It would be cleaner to do an explicit UGI.loginFromKeytab instead to avoid > such issues in future as well. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17556) The test udf_mask_hash.q is failing
[
https://issues.apache.org/jira/browse/HIVE-17556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16172356#comment-16172356
]
Hive QA commented on HIVE-17556:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12887879/HIVE-17556.1.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 10 failed/errored test(s), 11037 tests
executed
*Failed tests:*
{noformat}
TestAccumuloCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=230)
TestDummy - did not produce a TEST-*.xml file (likely timed out) (batchId=230)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[create_view] (batchId=39)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[union_fast_stats]
(batchId=156)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_explainuser_1]
(batchId=170)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[drop_table_failure2]
(batchId=89)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query14]
(batchId=234)
org.apache.hadoop.hive.cli.TestSparkNegativeCliDriver.org.apache.hadoop.hive.cli.TestSparkNegativeCliDriver
(batchId=241)
org.apache.hadoop.hive.ql.TestAcidOnTez.testCtasTezUnion (batchId=215)
org.apache.hadoop.hive.ql.TestAcidOnTez.testNonStandardConversion01
(batchId=215)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/6888/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/6888/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-6888/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 10 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12887879 - PreCommit-HIVE-Build
> The test udf_mask_hash.q is failing
> ---
>
> Key: HIVE-17556
> URL: https://issues.apache.org/jira/browse/HIVE-17556
> Project: Hive
> Issue Type: Bug
> Components: Test
>Affects Versions: 3.0.0
>Reporter: Marta Kuczora
>Assignee: Marta Kuczora
> Attachments: HIVE-17556.1.patch
>
>
> After committing
> [HIVE-17515|https://issues.apache.org/jira/browse/HIVE-17515], the
> udf_mask_hash.q test is failing with the following error message:
> {noformat}
> Error Message
> Client Execution succeeded but contained differences (error code = 1) after
> executing udf_mask_hash.q
> 34c34
> < expressions:
> '8b44d559dc5d60e4453c9b4edf2a455fbce054bb8504cd3eb9b5f391bd239c90' (type:
> string)
> ---
> > expressions: 'dd78d68ad1b23bde126812482dd70ac6' (type: string)
> 36c36
> < Statistics: Num rows: 1 Data size: 148 Basic stats: COMPLETE
> Column stats: COMPLETE
> ---
> > Statistics: Num rows: 1 Data size: 116 Basic stats: COMPLETE
> > Column stats: COMPLETE
> 61c61
> < 8b44d559dc5d60e4453c9b4edf2a455fbce054bb8504cd3eb9b5f391bd239c90
> 8b44d559dc5d60e4453c9b4edf2a455fbce054bb8504cd3eb9b5f391bd239c90
> 30a88603135d3a6f7a66b4f9193da1ab4423aed45fb8fe736c2f2a08977f2bdd
>
>
> NULL NULL NULL NULL NULL
> ---
> > dd78d68ad1b23bde126812482dd70ac6 dd78d68ad1b23bde126812482dd70ac6
> > 835735ba20f1297683efca69fabd0fba
> >
> >
> > NULL NULL NULL NULL NULL
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Resolved] (HIVE-17485) Hive-Druid table on indexing for few segments- DruidRecordWriter.pushSegments throws ArrayIndexOutOfBoundsException
[
https://issues.apache.org/jira/browse/HIVE-17485?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashutosh Chauhan resolved HIVE-17485.
-
Resolution: Duplicate
Fix Version/s: 3.0.0
> Hive-Druid table on indexing for few segments- DruidRecordWriter.pushSegments
> throws ArrayIndexOutOfBoundsException
> ---
>
> Key: HIVE-17485
> URL: https://issues.apache.org/jira/browse/HIVE-17485
> Project: Hive
> Issue Type: Bug
> Components: Druid integration
>Affects Versions: 3.0.0
>Reporter: Dileep Kumar Chiguruvada
>Assignee: slim bouguerra
> Fix For: 3.0.0
>
>
> Hive-Druid table on indexing for few segments DruidRecordWriter.pushSegments
> throws ArrayIndexOutOfBoundsException.
> Error says
> {code}
> ERROR : Vertex failed, vertexName=Reducer 2,
> vertexId=vertex_1502725432788_0017_2_01, diagnostics=[Task failed,
> taskId=task_1502725432788_0017_2_01_02, diagnostics=[TaskAttempt 0
> failed, info=[Error: Error while running task ( failure ) :
> attempt_1502725432788_0017_2_01_02_0:java.lang.RuntimeException:
> java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException:
> Hive Runtime Error while processing vector batch (tag=0) (vectorizedVertexNum
> 1) Column vector types: 1:TIMESTAMP, 2:LONG, 3:BYTES, 4:LONG, 5:LONG, 6:LONG,
> 7:LONG, 8:LONG, 9:LONG, 10:LONG, 11:LONG, 12:LONG, 13:LONG, 14:LONG,
> 15:BYTES, 16:BYTES, 17:BYTES, 18:BYTES, 19:BYTES, 20:LONG, 21:LONG, 22:LONG,
> 23:LONG, 24:BYTES, 25:BYTES, 26:BYTES, 27:BYTES, 28:BYTES, 0:TIMESTAMP
> [1900-01-18 00:00:00.0, 2415038, "OLJNECAA", 0, 3, 1, 1900, 3, 1, 18,
> 1, 1900, 1, 3, "Wednesday", "1900Q1", "N", "N", "N", 2415021, 2415020,
> 2414673, 2414946, "N", "N", "N", "N", "N", 1900-01-18 00:00:00.0]
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:218)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:172)
> at
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:370)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:73)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:61)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1866)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:61)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:37)
> at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
> at
> org.apache.hadoop.hive.llap.daemon.impl.StatsRecordingThreadPool$WrappedCallable.call(StatsRecordingThreadPool.java:110)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.lang.RuntimeException:
> org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while
> processing vector batch (tag=0) (vectorizedVertexNum 1) Column vector types:
> 1:TIMESTAMP, 2:LONG, 3:BYTES, 4:LONG, 5:LONG, 6:LONG, 7:LONG, 8:LONG, 9:LONG,
> 10:LONG, 11:LONG, 12:LONG, 13:LONG, 14:LONG, 15:BYTES, 16:BYTES, 17:BYTES,
> 18:BYTES, 19:BYTES, 20:LONG, 21:LONG, 22:LONG, 23:LONG, 24:BYTES, 25:BYTES,
> 26:BYTES, 27:BYTES, 28:BYTES, 0:TIMESTAMP
> [1900-01-18 00:00:00.0, 2415038, "OLJNECAA", 0, 3, 1, 1900, 3, 1, 18,
> 1, 1900, 1, 3, "Wednesday", "1900Q1", "N", "N", "N", 2415021, 2415020,
> 2414673, 2414946, "N", "N", "N", "N", "N", 1900-01-18 00:00:00.0]
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.pushRecordVector(ReduceRecordSource.java:406)
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.pushRecord(ReduceRecordSource.java:248)
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.run(ReduceRecordProcessor.java:319)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:189)
> ... 15 more
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime
> Error while processing vector batch (tag=0) (vectorizedVertexNum 1) Column
> vector types: 1:TIMESTAMP, 2:LONG, 3:BYTES, 4:LONG, 5:LONG, 6:LONG, 7:LONG,
> 8:LONG, 9:LONG, 10:LONG, 11:LONG, 12:LONG, 13:LONG, 14:LONG, 15:BYTES,
> 16:BYTES, 17:BYTES, 18:BYTES, 19:BYTES, 20:LONG,
[jira] [Updated] (HIVE-17196) CM: ReplCopyTask should retain the original file names even if copied from CM path.
[ https://issues.apache.org/jira/browse/HIVE-17196?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Dai updated HIVE-17196: -- Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Patch pushed to master. > CM: ReplCopyTask should retain the original file names even if copied from CM > path. > --- > > Key: HIVE-17196 > URL: https://issues.apache.org/jira/browse/HIVE-17196 > Project: Hive > Issue Type: Sub-task > Components: repl >Affects Versions: 2.1.0 >Reporter: Sankar Hariappan >Assignee: Daniel Dai > Fix For: 3.0.0 > > Attachments: HIVE-17196.1.patch, HIVE-17196.2.patch > > > Consider the below scenario, > 1. Insert into table T1 with value(X). > 2. Insert into table T1 with value(X). > 3. Truncate the table T1. > – This step backs up 2 files with same content to cmroot which ends up with > one file in cmroot as checksum matches. > 4. Incremental repl with above 3 operations. > – In this step, both the insert event files will be read from cmroot where > copy of one leads to overwrite the other one as the file name is same in cm > path (checksum as file name). > So, this leads to data loss and hence it is necessary to retain the original > file names even if we copy from cm path. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17196) CM: ReplCopyTask should retain the original file names even if copied from CM path.
[ https://issues.apache.org/jira/browse/HIVE-17196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16172324#comment-16172324 ] Thejas M Nair commented on HIVE-17196: -- +1 > CM: ReplCopyTask should retain the original file names even if copied from CM > path. > --- > > Key: HIVE-17196 > URL: https://issues.apache.org/jira/browse/HIVE-17196 > Project: Hive > Issue Type: Sub-task > Components: repl >Affects Versions: 2.1.0 >Reporter: Sankar Hariappan >Assignee: Daniel Dai > Fix For: 3.0.0 > > Attachments: HIVE-17196.1.patch, HIVE-17196.2.patch > > > Consider the below scenario, > 1. Insert into table T1 with value(X). > 2. Insert into table T1 with value(X). > 3. Truncate the table T1. > – This step backs up 2 files with same content to cmroot which ends up with > one file in cmroot as checksum matches. > 4. Incremental repl with above 3 operations. > – In this step, both the insert event files will be read from cmroot where > copy of one leads to overwrite the other one as the file name is same in cm > path (checksum as file name). > So, this leads to data loss and hence it is necessary to retain the original > file names even if we copy from cm path. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17535) Select 1 EXCEPT Select 1 fails with NPE
[
https://issues.apache.org/jira/browse/HIVE-17535?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vineet Garg updated HIVE-17535:
---
Status: Patch Available (was: Open)
Latest patch(5) should fix test failures
> Select 1 EXCEPT Select 1 fails with NPE
> ---
>
> Key: HIVE-17535
> URL: https://issues.apache.org/jira/browse/HIVE-17535
> Project: Hive
> Issue Type: Bug
> Components: Query Planning
>Reporter: Vineet Garg
>Assignee: Vineet Garg
> Attachments: HIVE-17535.1.patch, HIVE-17535.2.patch,
> HIVE-17535.3.patch, HIVE-17535.4.patch, HIVE-17535.5.patch
>
>
> Since Hive CBO isn't able to handle queries with no table e.g. {{select 1}}
> queries with SET operators fail (intersect requires CBO).
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-17535) Select 1 EXCEPT Select 1 fails with NPE
[
https://issues.apache.org/jira/browse/HIVE-17535?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vineet Garg updated HIVE-17535:
---
Attachment: HIVE-17535.5.patch
> Select 1 EXCEPT Select 1 fails with NPE
> ---
>
> Key: HIVE-17535
> URL: https://issues.apache.org/jira/browse/HIVE-17535
> Project: Hive
> Issue Type: Bug
> Components: Query Planning
>Reporter: Vineet Garg
>Assignee: Vineet Garg
> Attachments: HIVE-17535.1.patch, HIVE-17535.2.patch,
> HIVE-17535.3.patch, HIVE-17535.4.patch, HIVE-17535.5.patch
>
>
> Since Hive CBO isn't able to handle queries with no table e.g. {{select 1}}
> queries with SET operators fail (intersect requires CBO).
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-17535) Select 1 EXCEPT Select 1 fails with NPE
[
https://issues.apache.org/jira/browse/HIVE-17535?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vineet Garg updated HIVE-17535:
---
Status: Open (was: Patch Available)
> Select 1 EXCEPT Select 1 fails with NPE
> ---
>
> Key: HIVE-17535
> URL: https://issues.apache.org/jira/browse/HIVE-17535
> Project: Hive
> Issue Type: Bug
> Components: Query Planning
>Reporter: Vineet Garg
>Assignee: Vineet Garg
> Attachments: HIVE-17535.1.patch, HIVE-17535.2.patch,
> HIVE-17535.3.patch, HIVE-17535.4.patch, HIVE-17535.5.patch
>
>
> Since Hive CBO isn't able to handle queries with no table e.g. {{select 1}}
> queries with SET operators fail (intersect requires CBO).
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-17538) Enhance estimation of stats to estimate even if only one column is missing stats
[ https://issues.apache.org/jira/browse/HIVE-17538?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16172317#comment-16172317 ] Ashutosh Chauhan commented on HIVE-17538: - Are test failures related? Also, instead of linearly searching in list, you may use sets to efficiently figure out membership. > Enhance estimation of stats to estimate even if only one column is missing > stats > > > Key: HIVE-17538 > URL: https://issues.apache.org/jira/browse/HIVE-17538 > Project: Hive > Issue Type: Improvement >Reporter: Vineet Garg >Assignee: Vineet Garg > Attachments: HIVE-17538.1.patch > > > HIVE-16811 provided support for estimating statistics in absence of stats. > But that estimation is done if and only if statistics are missing for all > columns. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17275) Auto-merge fails on writes of UNION ALL output to ORC file with dynamic partitioning
[
https://issues.apache.org/jira/browse/HIVE-17275?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16172314#comment-16172314
]
Mithun Radhakrishnan commented on HIVE-17275:
-
Hey, [~leftylev]. Pardon my negligence. I've updated the JIRA with the fix
versions from my previous comment. [~owen.omalley], does this need to be
committed to the {{branch-2.3}} release branch?
> Auto-merge fails on writes of UNION ALL output to ORC file with dynamic
> partitioning
>
>
> Key: HIVE-17275
> URL: https://issues.apache.org/jira/browse/HIVE-17275
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
>Affects Versions: 2.2.0
>Reporter: Chris Drome
>Assignee: Chris Drome
> Fix For: 3.0.0, 2.4.0, 2.2.1
>
> Attachments: HIVE-17275.2-branch-2.2.patch,
> HIVE-17275.2-branch-2.patch, HIVE-17275.2.patch, HIVE-17275-branch-2.2.patch,
> HIVE-17275-branch-2.patch, HIVE-17275.patch
>
>
> If dynamic partitioning is used to write the output of UNION or UNION ALL
> queries into ORC files with hive.merge.tezfiles=true, the merge step fails as
> follows:
> {noformat}
> 2017-08-08T11:27:19,958 ERROR [e7b1f06d-d632-408a-9dff-f7ae042cd25a main]
> SessionState: Vertex failed, vertexName=File Merge,
> vertexId=vertex_1502216690354_0001_33_00, diagnostics=[Task failed,
> taskId=task_1502216690354_0001_33_00_00, diagnostics=[TaskAttempt 0
> failed, info=[Error: Error while running task ( failure ) :
> attempt_1502216690354_0001_33_00_00_0:java.lang.RuntimeException:
> java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException:
> java.io.IOException: Multiple partitions for one merge mapper:
> hdfs://localhost:39943/build/ql/test/data/warehouse/partunion1/.hive-staging_hive_2017-08-08_11-27-09_105_286405133968521828-1/-ext-10002/part1=2014/1
> NOT EQUAL TO
> hdfs://localhost:39943/build/ql/test/data/warehouse/partunion1/.hive-staging_hive_2017-08-08_11-27-09_105_286405133968521828-1/-ext-10002/part1=2014/2
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:211)
> at
> org.apache.hadoop.hive.ql.exec.tez.MergeFileTezProcessor.run(MergeFileTezProcessor.java:42)
> at
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:370)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:73)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:61)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1807)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:61)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:37)
> at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.RuntimeException:
> org.apache.hadoop.hive.ql.metadata.HiveException: java.io.IOException:
> Multiple partitions for one merge mapper:
> hdfs://localhost:39943/build/ql/test/data/warehouse/partunion1/.hive-staging_hive_2017-08-08_11-27-09_105_286405133968521828-1/-ext-10002/part1=2014/1
> NOT EQUAL TO
> hdfs://localhost:39943/build/ql/test/data/warehouse/partunion1/.hive-staging_hive_2017-08-08_11-27-09_105_286405133968521828-1/-ext-10002/part1=2014/2
> at
> org.apache.hadoop.hive.ql.exec.tez.MergeFileRecordProcessor.processRow(MergeFileRecordProcessor.java:225)
> at
> org.apache.hadoop.hive.ql.exec.tez.MergeFileRecordProcessor.run(MergeFileRecordProcessor.java:154)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:185)
> ... 14 more
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
> java.io.IOException: Multiple partitions for one merge mapper:
> hdfs://localhost:39943/build/ql/test/data/warehouse/partunion1/.hive-staging_hive_2017-08-08_11-27-09_105_286405133968521828-1/-ext-10002/part1=2014/1
> NOT EQUAL TO
> hdfs://localhost:39943/build/ql/test/data/warehouse/partunion1/.hive-staging_hive_2017-08-08_11-27-09_105_286405133968521828-1/-ext-10002/part1=2014/2
> at
> org.apache.hadoop.hive.ql.exec.OrcFileMergeOperator.processKeyValuePairs(OrcFileMergeOperator.java:169)
> at
>
[jira] [Updated] (HIVE-17275) Auto-merge fails on writes of UNION ALL output to ORC file with dynamic partitioning
[
https://issues.apache.org/jira/browse/HIVE-17275?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mithun Radhakrishnan updated HIVE-17275:
Fix Version/s: 2.2.1
2.4.0
3.0.0
> Auto-merge fails on writes of UNION ALL output to ORC file with dynamic
> partitioning
>
>
> Key: HIVE-17275
> URL: https://issues.apache.org/jira/browse/HIVE-17275
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
>Affects Versions: 2.2.0
>Reporter: Chris Drome
>Assignee: Chris Drome
> Fix For: 3.0.0, 2.4.0, 2.2.1
>
> Attachments: HIVE-17275.2-branch-2.2.patch,
> HIVE-17275.2-branch-2.patch, HIVE-17275.2.patch, HIVE-17275-branch-2.2.patch,
> HIVE-17275-branch-2.patch, HIVE-17275.patch
>
>
> If dynamic partitioning is used to write the output of UNION or UNION ALL
> queries into ORC files with hive.merge.tezfiles=true, the merge step fails as
> follows:
> {noformat}
> 2017-08-08T11:27:19,958 ERROR [e7b1f06d-d632-408a-9dff-f7ae042cd25a main]
> SessionState: Vertex failed, vertexName=File Merge,
> vertexId=vertex_1502216690354_0001_33_00, diagnostics=[Task failed,
> taskId=task_1502216690354_0001_33_00_00, diagnostics=[TaskAttempt 0
> failed, info=[Error: Error while running task ( failure ) :
> attempt_1502216690354_0001_33_00_00_0:java.lang.RuntimeException:
> java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException:
> java.io.IOException: Multiple partitions for one merge mapper:
> hdfs://localhost:39943/build/ql/test/data/warehouse/partunion1/.hive-staging_hive_2017-08-08_11-27-09_105_286405133968521828-1/-ext-10002/part1=2014/1
> NOT EQUAL TO
> hdfs://localhost:39943/build/ql/test/data/warehouse/partunion1/.hive-staging_hive_2017-08-08_11-27-09_105_286405133968521828-1/-ext-10002/part1=2014/2
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:211)
> at
> org.apache.hadoop.hive.ql.exec.tez.MergeFileTezProcessor.run(MergeFileTezProcessor.java:42)
> at
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:370)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:73)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:61)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1807)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:61)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:37)
> at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.RuntimeException:
> org.apache.hadoop.hive.ql.metadata.HiveException: java.io.IOException:
> Multiple partitions for one merge mapper:
> hdfs://localhost:39943/build/ql/test/data/warehouse/partunion1/.hive-staging_hive_2017-08-08_11-27-09_105_286405133968521828-1/-ext-10002/part1=2014/1
> NOT EQUAL TO
> hdfs://localhost:39943/build/ql/test/data/warehouse/partunion1/.hive-staging_hive_2017-08-08_11-27-09_105_286405133968521828-1/-ext-10002/part1=2014/2
> at
> org.apache.hadoop.hive.ql.exec.tez.MergeFileRecordProcessor.processRow(MergeFileRecordProcessor.java:225)
> at
> org.apache.hadoop.hive.ql.exec.tez.MergeFileRecordProcessor.run(MergeFileRecordProcessor.java:154)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:185)
> ... 14 more
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
> java.io.IOException: Multiple partitions for one merge mapper:
> hdfs://localhost:39943/build/ql/test/data/warehouse/partunion1/.hive-staging_hive_2017-08-08_11-27-09_105_286405133968521828-1/-ext-10002/part1=2014/1
> NOT EQUAL TO
> hdfs://localhost:39943/build/ql/test/data/warehouse/partunion1/.hive-staging_hive_2017-08-08_11-27-09_105_286405133968521828-1/-ext-10002/part1=2014/2
> at
> org.apache.hadoop.hive.ql.exec.OrcFileMergeOperator.processKeyValuePairs(OrcFileMergeOperator.java:169)
> at
> org.apache.hadoop.hive.ql.exec.OrcFileMergeOperator.process(OrcFileMergeOperator.java:72)
> at
> org.apache.hadoop.hive.ql.exec.tez.MergeFileRecordProcessor.processRow(MergeFileRecordProcessor.java:216)
> ... 16 more
> Caused by:
[jira] [Commented] (HIVE-17535) Select 1 EXCEPT Select 1 fails with NPE
[
https://issues.apache.org/jira/browse/HIVE-17535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16172302#comment-16172302
]
Ashutosh Chauhan commented on HIVE-17535:
-
+1 Lets do those 2 exposed issues in a follow-up
> Select 1 EXCEPT Select 1 fails with NPE
> ---
>
> Key: HIVE-17535
> URL: https://issues.apache.org/jira/browse/HIVE-17535
> Project: Hive
> Issue Type: Bug
> Components: Query Planning
>Reporter: Vineet Garg
>Assignee: Vineet Garg
> Attachments: HIVE-17535.1.patch, HIVE-17535.2.patch,
> HIVE-17535.3.patch, HIVE-17535.4.patch
>
>
> Since Hive CBO isn't able to handle queries with no table e.g. {{select 1}}
> queries with SET operators fail (intersect requires CBO).
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-17529) Bucket Map Join : Sets incorrect edge type causing execution failure
[ https://issues.apache.org/jira/browse/HIVE-17529?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Deepak Jaiswal updated HIVE-17529: -- Attachment: HIVE-17529.4.patch Retrying again after refreshing the code. > Bucket Map Join : Sets incorrect edge type causing execution failure > > > Key: HIVE-17529 > URL: https://issues.apache.org/jira/browse/HIVE-17529 > Project: Hive > Issue Type: Bug >Reporter: Deepak Jaiswal >Assignee: Deepak Jaiswal > Attachments: HIVE-17529.1.patch, HIVE-17529.2.patch, > HIVE-17529.3.patch, HIVE-17529.4.patch > > > If while traversing the tree to generate tasks, a bucket mapjoin may set its > edge as CUSTOM_SIMPLE_EDGE against CUSTOM_EDGE if the bigtable is already not > traversed causing Tez to assert and fail the vertex. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-16898) Validation of source file after distcp in repl load
[ https://issues.apache.org/jira/browse/HIVE-16898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16172266#comment-16172266 ] Daniel Dai commented on HIVE-16898: --- Discussed with Anishek, update the patch again. > Validation of source file after distcp in repl load > > > Key: HIVE-16898 > URL: https://issues.apache.org/jira/browse/HIVE-16898 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Affects Versions: 3.0.0 >Reporter: anishek >Assignee: Daniel Dai > Fix For: 3.0.0 > > Attachments: HIVE-16898.1.patch, HIVE-16898.2.patch, > HIVE-16898.3.patch, HIVE-16898.4.patch > > > time between deciding the source and destination path for distcp to invoking > of distcp can have a change of the source file, hence distcp might copy the > wrong file to destination, hence we should an additional check on the > checksum of the source file path after distcp finishes to make sure the path > didnot change during the copy process. if it has take additional steps to > delete the previous file on destination and copy the new source and repeat > the same process as above till we copy the correct file. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-16898) Validation of source file after distcp in repl load
[ https://issues.apache.org/jira/browse/HIVE-16898?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Dai updated HIVE-16898: -- Attachment: HIVE-16898.4.patch > Validation of source file after distcp in repl load > > > Key: HIVE-16898 > URL: https://issues.apache.org/jira/browse/HIVE-16898 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Affects Versions: 3.0.0 >Reporter: anishek >Assignee: Daniel Dai > Fix For: 3.0.0 > > Attachments: HIVE-16898.1.patch, HIVE-16898.2.patch, > HIVE-16898.3.patch, HIVE-16898.4.patch > > > time between deciding the source and destination path for distcp to invoking > of distcp can have a change of the source file, hence distcp might copy the > wrong file to destination, hence we should an additional check on the > checksum of the source file path after distcp finishes to make sure the path > didnot change during the copy process. if it has take additional steps to > delete the previous file on destination and copy the new source and repeat > the same process as above till we copy the correct file. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-16602) Implement shared scans with Tez
[ https://issues.apache.org/jira/browse/HIVE-16602?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16172260#comment-16172260 ] Jesus Camacho Rodriguez commented on HIVE-16602: [~kellyzly], this has been tested and it makes a huge difference, specially for IO intensive queries. bq. ...it appears multiple times in the query. What do you mean? When you use "explain plan", you should see that TS is reused for the same table across different tasks. Otherwise the optimization might not have been trigger. You can see multiple examples in the commit for this issue. > Implement shared scans with Tez > --- > > Key: HIVE-16602 > URL: https://issues.apache.org/jira/browse/HIVE-16602 > Project: Hive > Issue Type: New Feature > Components: Physical Optimizer >Affects Versions: 3.0.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez > Labels: TODOC3.0 > Fix For: 3.0.0 > > Attachments: HIVE-16602.01.patch, HIVE-16602.02.patch, > HIVE-16602.03.patch, HIVE-16602.04.patch, HIVE-16602.patch > > > Given a query plan, the goal is to identify scans on input tables that can be > merged so the data is read only once. Optimization will be carried out at the > physical level. > In the longer term, identification of equivalent expressions and > reutilization of intermediary results should be done at the logical layer via > Spool operator. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17554) Occurr java.lang.ArithmeticException: / by zero at hplsql component
[
https://issues.apache.org/jira/browse/HIVE-17554?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16172259#comment-16172259
]
Hive QA commented on HIVE-17554:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12887835/HIVE-17554.1.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 13 failed/errored test(s), 11029 tests
executed
*Failed tests:*
{noformat}
TestAccumuloCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=230)
TestDummy - did not produce a TEST-*.xml file (likely timed out) (batchId=230)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[create_view] (batchId=39)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[udf_mask_hash]
(batchId=28)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[union_fast_stats]
(batchId=156)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_explainuser_1]
(batchId=170)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_2]
(batchId=100)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[drop_table_failure2]
(batchId=89)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query14]
(batchId=234)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query23]
(batchId=234)
org.apache.hadoop.hive.cli.TestSparkCliDriver.org.apache.hadoop.hive.cli.TestSparkCliDriver
(batchId=115)
org.apache.hadoop.hive.ql.TestAcidOnTez.testCtasTezUnion (batchId=215)
org.apache.hadoop.hive.ql.TestAcidOnTez.testNonStandardConversion01
(batchId=215)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/6887/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/6887/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-6887/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 13 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12887835 - PreCommit-HIVE-Build
> Occurr java.lang.ArithmeticException: / by zero at hplsql component
> ---
>
> Key: HIVE-17554
> URL: https://issues.apache.org/jira/browse/HIVE-17554
> Project: Hive
> Issue Type: Bug
> Components: hpl/sql
>Reporter: ZhangBing Lin
>Assignee: ZhangBing Lin
> Attachments: HIVE-17554.1.patch
>
>
> When I execute the copyToTable method in hplsql, the following exception
> occurs:
> java.lang.ArithmeticException: / by zero
> at org.apache.hive.hplsql.Copy.copyToTable(Copy.java:170)
> at org.apache.hive.hplsql.Copy.run(Copy.java:93)
> at org.apache.hive.hplsql.Exec.visitCopy_stmt(Exec.java:1284)
> at org.apache.hive.hplsql.Exec.visitCopy_stmt(Exec.java:52)
> at
> org.apache.hive.hplsql.HplsqlParser$Copy_stmtContext.accept(HplsqlParser.java:12148)
> at
> org.antlr.v4.runtime.tree.AbstractParseTreeVisitor.visitChildren(AbstractParseTreeVisitor.java:70)
> at org.apache.hive.hplsql.Exec.visitStmt(Exec.java:994)
> at org.apache.hive.hplsql.Exec.visitStmt(Exec.java:52)
> at
> org.apache.hive.hplsql.HplsqlParser$StmtContext.accept(HplsqlParser.java:1012)
> at
> org.antlr.v4.runtime.tree.AbstractParseTreeVisitor.visitChildren(AbstractParseTreeVisitor.java:70)
> at
> org.apache.hive.hplsql.HplsqlBaseVisitor.visitBlock(HplsqlBaseVisitor.java:28)
> at
> org.apache.hive.hplsql.HplsqlParser$BlockContext.accept(HplsqlParser.java:446)
> at
> org.antlr.v4.runtime.tree.AbstractParseTreeVisitor.visitChildren(AbstractParseTreeVisitor.java:70)
> at org.apache.hive.hplsql.Exec.visitProgram(Exec.java:901)
> at org.apache.hive.hplsql.Exec.visitProgram(Exec.java:52)
> at
> org.apache.hive.hplsql.HplsqlParser$ProgramContext.accept(HplsqlParser.java:389)
> at
> org.antlr.v4.runtime.tree.AbstractParseTreeVisitor.visit(AbstractParseTreeVisitor.java:42)
> at org.apache.hive.hplsql.Exec.run(Exec.java:760)
> at org.apache.hive.hplsql.Exec.run(Exec.java:736)
> at org.apache.hive.hplsql.Hplsql.main(Hplsql.java:23)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
>
[jira] [Updated] (HIVE-15899) Make CTAS with acid target table and insert into acid_tbl select ... union all ... work
[
https://issues.apache.org/jira/browse/HIVE-15899?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eugene Koifman updated HIVE-15899:
--
Description:
Consider:
{noformat}
create table T stored as ORC TBLPROPERTIES('transactional'='true') as
select a, b from A where a <= 5 union all select a, b from B where a >= 5
{noformat}
and
{noformat}
create table T (a int, b int) stored as ORC TBLPROPERTIES
('transactional'='false';
insert into T(a,b) select a, b from T where a between 1 and 3 group by a, b
union all select a, b from A where a between 5 and 7 union all select a, b from
B where a >= 9
{noformat}
On Tez, there is an optimization that removes Union All operator writes the
data into
subdirectories of T (in this case T is unpartitioned).
This also happens on MR but requires
{noformat}
hiveConf.setBoolVar(HiveConf.ConfVars.HIVE_OPTIMIZE_UNION_REMOVE, true);
hiveConf.setVar(HiveConf.ConfVars.HIVEFETCHTASKCONVERSION, "none");
{noformat}
Need to ensure that when target table is Acid, we generate unique ROW__IDs
When target is not acid, that we can convert it to Acid via Alter Table even
when data layout includes subdirectories.
was:need to add a test to check if create table as works correctly with acid
tables
> Make CTAS with acid target table and insert into acid_tbl select ... union
> all ... work
> ---
>
> Key: HIVE-15899
> URL: https://issues.apache.org/jira/browse/HIVE-15899
> Project: Hive
> Issue Type: Sub-task
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
> Attachments: HIVE-15899.01.patch, HIVE-15899.02.patch,
> HIVE-15899.03.patch, HIVE-15899.04.patch, HIVE-15899.05.patch,
> HIVE-15899.07.patch, HIVE-15899.08.patch, HIVE-15899.09.patch,
> HIVE-15899.10.patch, HIVE-15899.11.patch, HIVE-15899.12.patch
>
>
> Consider:
> {noformat}
> create table T stored as ORC TBLPROPERTIES('transactional'='true') as
> select a, b from A where a <= 5 union all select a, b from B where a >=
> 5
> {noformat}
> and
> {noformat}
> create table T (a int, b int) stored as ORC TBLPROPERTIES
> ('transactional'='false';
> insert into T(a,b) select a, b from T where a between 1 and 3 group by a, b
> union all select a, b from A where a between 5 and 7 union all select a, b
> from B where a >= 9
> {noformat}
> On Tez, there is an optimization that removes Union All operator writes the
> data into
> subdirectories of T (in this case T is unpartitioned).
> This also happens on MR but requires
> {noformat}
> hiveConf.setBoolVar(HiveConf.ConfVars.HIVE_OPTIMIZE_UNION_REMOVE, true);
> hiveConf.setVar(HiveConf.ConfVars.HIVEFETCHTASKCONVERSION, "none");
> {noformat}
> Need to ensure that when target table is Acid, we generate unique ROW__IDs
> When target is not acid, that we can convert it to Acid via Alter Table even
> when data layout includes subdirectories.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-17139) Conditional expressions optimization: skip the expression evaluation if the condition is not satisfied for vectorization engine.
[
https://issues.apache.org/jira/browse/HIVE-17139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16172144#comment-16172144
]
Hive QA commented on HIVE-17139:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12887830/HIVE-17139.11.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 12 failed/errored test(s), 11043 tests
executed
*Failed tests:*
{noformat}
TestAccumuloCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=230)
TestDummy - did not produce a TEST-*.xml file (likely timed out) (batchId=230)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[create_view] (batchId=39)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[udf_mask_hash]
(batchId=28)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[union_fast_stats]
(batchId=156)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_explainuser_1]
(batchId=170)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_2]
(batchId=100)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[drop_table_failure2]
(batchId=89)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query14]
(batchId=234)
org.apache.hadoop.hive.ql.TestAcidOnTez.testCtasTezUnion (batchId=215)
org.apache.hadoop.hive.ql.TestAcidOnTez.testNonStandardConversion01
(batchId=215)
org.apache.hive.jdbc.TestJdbcWithMiniHS2.testHttpRetryOnServerIdleTimeout
(batchId=227)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/6886/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/6886/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-6886/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 12 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12887830 - PreCommit-HIVE-Build
> Conditional expressions optimization: skip the expression evaluation if the
> condition is not satisfied for vectorization engine.
>
>
> Key: HIVE-17139
> URL: https://issues.apache.org/jira/browse/HIVE-17139
> Project: Hive
> Issue Type: Improvement
>Reporter: Ke Jia
>Assignee: Ke Jia
> Attachments: HIVE-17139.10.patch, HIVE-17139.11.patch,
> HIVE-17139.1.patch, HIVE-17139.2.patch, HIVE-17139.3.patch,
> HIVE-17139.4.patch, HIVE-17139.5.patch, HIVE-17139.6.patch,
> HIVE-17139.7.patch, HIVE-17139.8.patch, HIVE-17139.9.patch
>
>
> The case when and if statement execution for Hive vectorization is not
> optimal, which all the conditional and else expressions are evaluated for
> current implementation. The optimized approach is to update the selected
> array of batch parameter after the conditional expression is executed. Then
> the else expression will only do the selected rows instead of all.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-13390) HiveServer2: Add more test to ZK service discovery using MiniHS2
[ https://issues.apache.org/jira/browse/HIVE-13390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16172127#comment-16172127 ] Vaibhav Gumashta commented on HIVE-13390: - Useful reference to keytool for testing purpose: https://docs.oracle.com/javase/8/docs/technotes/tools/windows/keytool.html > HiveServer2: Add more test to ZK service discovery using MiniHS2 > > > Key: HIVE-13390 > URL: https://issues.apache.org/jira/browse/HIVE-13390 > Project: Hive > Issue Type: Bug > Components: HiveServer2, JDBC >Affects Versions: 1.2.1, 2.0.0 >Reporter: Vaibhav Gumashta >Assignee: Vaibhav Gumashta > Fix For: 1.3.0, 1.2.2, 2.0.1, 2.1.0 > > Attachments: HIVE-13390.1.patch, HIVE-13390.1.patch, > HIVE-13390.2.patch, HIVE-13390.3.patch, HIVE-13390.branch-1.2.patch, > HIVE-13390.branch-1.patch, keystore_exampledotcom.jks, keystore.jks, > truststore.jks > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-13390) HiveServer2: Add more test to ZK service discovery using MiniHS2
[ https://issues.apache.org/jira/browse/HIVE-13390?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vaibhav Gumashta updated HIVE-13390: Description: (was: https://docs.oracle.com/javase/8/docs/technotes/tools/windows/keytool.html) > HiveServer2: Add more test to ZK service discovery using MiniHS2 > > > Key: HIVE-13390 > URL: https://issues.apache.org/jira/browse/HIVE-13390 > Project: Hive > Issue Type: Bug > Components: HiveServer2, JDBC >Affects Versions: 1.2.1, 2.0.0 >Reporter: Vaibhav Gumashta >Assignee: Vaibhav Gumashta > Fix For: 1.3.0, 1.2.2, 2.0.1, 2.1.0 > > Attachments: HIVE-13390.1.patch, HIVE-13390.1.patch, > HIVE-13390.2.patch, HIVE-13390.3.patch, HIVE-13390.branch-1.2.patch, > HIVE-13390.branch-1.patch, keystore_exampledotcom.jks, keystore.jks, > truststore.jks > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-16360) Improve "No delta files or original files found to compact in" message
[
https://issues.apache.org/jira/browse/HIVE-16360?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eugene Koifman updated HIVE-16360:
--
Status: Patch Available (was: Open)
> Improve "No delta files or original files found to compact in" message
> --
>
> Key: HIVE-16360
> URL: https://issues.apache.org/jira/browse/HIVE-16360
> Project: Hive
> Issue Type: Improvement
> Components: Transactions
>Affects Versions: 1.2.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
> Attachments: HIVE-16360.01.patch
>
>
> CompactorMR.run() produces
> {noformat}
> LOG.error("No delta files or original files found to compact in " +
> sd.getLocation() + " for compactionId=" + ci.id);
> {noformat}
> in some cases even where there are plenty of delta directories in the
> partition.
> The reason is that compactor won't compact anything above the level of an
> open transaction. So if there is a very long running txn, it may filter out
> all deltas in a partition. This looks very confusing when looking at the
> logs.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-16360) Improve "No delta files or original files found to compact in" message
[
https://issues.apache.org/jira/browse/HIVE-16360?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eugene Koifman updated HIVE-16360:
--
Attachment: HIVE-16360.01.patch
> Improve "No delta files or original files found to compact in" message
> --
>
> Key: HIVE-16360
> URL: https://issues.apache.org/jira/browse/HIVE-16360
> Project: Hive
> Issue Type: Improvement
> Components: Transactions
>Affects Versions: 1.2.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
> Attachments: HIVE-16360.01.patch
>
>
> CompactorMR.run() produces
> {noformat}
> LOG.error("No delta files or original files found to compact in " +
> sd.getLocation() + " for compactionId=" + ci.id);
> {noformat}
> in some cases even where there are plenty of delta directories in the
> partition.
> The reason is that compactor won't compact anything above the level of an
> open transaction. So if there is a very long running txn, it may filter out
> all deltas in a partition. This looks very confusing when looking at the
> logs.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-13390) HiveServer2: Add more test to ZK service discovery using MiniHS2
[ https://issues.apache.org/jira/browse/HIVE-13390?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vaibhav Gumashta updated HIVE-13390: Description: https://docs.oracle.com/javase/8/docs/technotes/tools/windows/keytool.html > HiveServer2: Add more test to ZK service discovery using MiniHS2 > > > Key: HIVE-13390 > URL: https://issues.apache.org/jira/browse/HIVE-13390 > Project: Hive > Issue Type: Bug > Components: HiveServer2, JDBC >Affects Versions: 1.2.1, 2.0.0 >Reporter: Vaibhav Gumashta >Assignee: Vaibhav Gumashta > Fix For: 1.3.0, 1.2.2, 2.0.1, 2.1.0 > > Attachments: HIVE-13390.1.patch, HIVE-13390.1.patch, > HIVE-13390.2.patch, HIVE-13390.3.patch, HIVE-13390.branch-1.2.patch, > HIVE-13390.branch-1.patch, keystore_exampledotcom.jks, keystore.jks, > truststore.jks > > > https://docs.oracle.com/javase/8/docs/technotes/tools/windows/keytool.html -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17550) Remove unreferenced q.out-s
[ https://issues.apache.org/jira/browse/HIVE-17550?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16172049#comment-16172049 ] Ashutosh Chauhan commented on HIVE-17550: - +1 > Remove unreferenced q.out-s > --- > > Key: HIVE-17550 > URL: https://issues.apache.org/jira/browse/HIVE-17550 > Project: Hive > Issue Type: Improvement > Components: Tests >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich > Attachments: HIVE-17550.01.patch > > > it's sometimes a bit misleading to see q.out-s which are never even used.. > I'll also add a small utility which is able to remove them - and add a test > which will help to avoid them in the future -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17483) HS2 kill command to kill queries using query id
[
https://issues.apache.org/jira/browse/HIVE-17483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16172034#comment-16172034
]
Hive QA commented on HIVE-17483:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12887881/HIVE-17483.5.patch
{color:green}SUCCESS:{color} +1 due to 6 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 18 failed/errored test(s), 11049 tests
executed
*Failed tests:*
{noformat}
TestAccumuloCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=230)
TestDummy - did not produce a TEST-*.xml file (likely timed out) (batchId=230)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[create_view] (batchId=39)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[udf_mask_hash]
(batchId=28)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[union_fast_stats]
(batchId=156)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_explainuser_1]
(batchId=170)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[drop_table_failure2]
(batchId=89)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query14]
(batchId=234)
org.apache.hadoop.hive.ql.TestAcidOnTez.testCtasTezUnion (batchId=215)
org.apache.hadoop.hive.ql.TestAcidOnTez.testNonStandardConversion01
(batchId=215)
org.apache.hive.service.cli.session.TestHiveSessionImpl.testLeakOperationHandle
(batchId=223)
org.apache.hive.service.cli.session.TestQueryDisplay.testQueryDisplay
(batchId=223)
org.apache.hive.service.cli.session.TestQueryDisplay.testWebUI (batchId=223)
org.apache.hive.service.cli.session.TestSessionManagerMetrics.testAbandonedSessionMetrics
(batchId=197)
org.apache.hive.service.cli.session.TestSessionManagerMetrics.testActiveSessionMetrics
(batchId=197)
org.apache.hive.service.cli.session.TestSessionManagerMetrics.testActiveSessionTimeMetrics
(batchId=197)
org.apache.hive.service.cli.session.TestSessionManagerMetrics.testOpenSessionMetrics
(batchId=197)
org.apache.hive.service.cli.session.TestSessionManagerMetrics.testOpenSessionTimeMetrics
(batchId=197)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/6885/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/6885/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-6885/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 18 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12887881 - PreCommit-HIVE-Build
> HS2 kill command to kill queries using query id
> ---
>
> Key: HIVE-17483
> URL: https://issues.apache.org/jira/browse/HIVE-17483
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Reporter: Thejas M Nair
>Assignee: Teddy Choi
> Attachments: HIVE-17483.1.patch, HIVE-17483.2.patch,
> HIVE-17483.2.patch, HIVE-17483.3.patch, HIVE-17483.4.patch, HIVE-17483.5.patch
>
>
> For administrators, it is important to be able to kill queries if required.
> Currently, there is no clean way to do it.
> It would help to have a "kill query " command that can be run using
> odbc/jdbc against a HiveServer2 instance, to kill a query with that queryid
> running in that instance.
> Authorization will have to be done to ensure that the user that is invoking
> the API is allowed to perform this action.
> In case of SQL std authorization, this would require admin role.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-17542) Make HoS CombineEquivalentWorkResolver Configurable
[
https://issues.apache.org/jira/browse/HIVE-17542?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sahil Takiar updated HIVE-17542:
Attachment: HIVE-17542.3.patch
> Make HoS CombineEquivalentWorkResolver Configurable
> ---
>
> Key: HIVE-17542
> URL: https://issues.apache.org/jira/browse/HIVE-17542
> Project: Hive
> Issue Type: Improvement
> Components: Physical Optimizer, Spark
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
> Attachments: HIVE-17542.1.patch, HIVE-17542.2.patch,
> HIVE-17542.3.patch
>
>
> The {{CombineEquivalentWorkResolver}} is run by default. We should make it
> configurable so that users can disable it in case there are any issues. We
> can enable it by default to preserve backwards compatibility.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-17558) Skip non-native/temporary tables for constraint related scenarios
[ https://issues.apache.org/jira/browse/HIVE-17558?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16171970#comment-16171970 ] Tao Li commented on HIVE-17558: --- cc [~daijy] > Skip non-native/temporary tables for constraint related scenarios > - > > Key: HIVE-17558 > URL: https://issues.apache.org/jira/browse/HIVE-17558 > Project: Hive > Issue Type: Bug >Reporter: Tao Li >Assignee: Tao Li > > The change would be similar to HIVE-17422. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Assigned] (HIVE-17558) Skip non-native/temporary tables for constraint related scenarios
[ https://issues.apache.org/jira/browse/HIVE-17558?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Li reassigned HIVE-17558: - > Skip non-native/temporary tables for constraint related scenarios > - > > Key: HIVE-17558 > URL: https://issues.apache.org/jira/browse/HIVE-17558 > Project: Hive > Issue Type: Bug >Reporter: Tao Li >Assignee: Tao Li > > The change would be similar to HIVE-17422. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17519) Transpose column stats display
[
https://issues.apache.org/jira/browse/HIVE-17519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16171954#comment-16171954
]
Ashutosh Chauhan commented on HIVE-17519:
-
+1 few minor comments on RB.
Also mark it as incompatible change while resolving jira.
> Transpose column stats display
> --
>
> Key: HIVE-17519
> URL: https://issues.apache.org/jira/browse/HIVE-17519
> Project: Hive
> Issue Type: Improvement
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
> Attachments: HIVE-17519.01.patch, HIVE-17519.02.patch
>
>
> currently {{describe formatted table1 insert_num}} shows the column
> informations in a table like format...which is very hard to read - because
> there are to many columns
> {code}
> # col_namedata_type min
> max num_nulls distinct_count
> avg_col_len max_col_len num_trues
> num_falses comment bitVector
>
>
> insert_numint
>
>
> from deserializer
> {code}
> I think it would be better to show the same information like this:
> {code}
> col_name insert_num
> data_type int
> min
> max
> num_nulls
> distinct_count
> avg_col_len
> max_col_len
> num_trues
> num_falses
> comment from deserializer
> bitVector
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-15899) Make CTAS with acid target table and insert into acid_tbl select ... union all ... work
[ https://issues.apache.org/jira/browse/HIVE-15899?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman updated HIVE-15899: -- Attachment: HIVE-15899.12.patch [~ashutoshc] could you review please (patch 12) > Make CTAS with acid target table and insert into acid_tbl select ... union > all ... work > --- > > Key: HIVE-15899 > URL: https://issues.apache.org/jira/browse/HIVE-15899 > Project: Hive > Issue Type: Sub-task >Reporter: Eugene Koifman >Assignee: Eugene Koifman > Attachments: HIVE-15899.01.patch, HIVE-15899.02.patch, > HIVE-15899.03.patch, HIVE-15899.04.patch, HIVE-15899.05.patch, > HIVE-15899.07.patch, HIVE-15899.08.patch, HIVE-15899.09.patch, > HIVE-15899.10.patch, HIVE-15899.11.patch, HIVE-15899.12.patch > > > need to add a test to check if create table as works correctly with acid > tables -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17508) Implement pool rules and triggers based on counters
[
https://issues.apache.org/jira/browse/HIVE-17508?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16171930#comment-16171930
]
Hive QA commented on HIVE-17508:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12887807/HIVE-17508.2.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 12 failed/errored test(s), 11054 tests
executed
*Failed tests:*
{noformat}
TestAccumuloCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=230)
TestDummy - did not produce a TEST-*.xml file (likely timed out) (batchId=230)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[create_view] (batchId=39)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[udf_mask_hash]
(batchId=28)
org.apache.hadoop.hive.cli.TestHBaseCliDriver.org.apache.hadoop.hive.cli.TestHBaseCliDriver
(batchId=96)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[union_fast_stats]
(batchId=156)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_explainuser_1]
(batchId=170)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[drop_table_failure2]
(batchId=89)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query14]
(batchId=234)
org.apache.hadoop.hive.ql.TestAcidOnTez.testCtasTezUnion (batchId=215)
org.apache.hadoop.hive.ql.TestAcidOnTez.testNonStandardConversion01
(batchId=215)
org.apache.hadoop.hive.ql.lockmgr.TestDbTxnManager2.testMerge3Way02
(batchId=282)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/6884/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/6884/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-6884/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 12 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12887807 - PreCommit-HIVE-Build
> Implement pool rules and triggers based on counters
> ---
>
> Key: HIVE-17508
> URL: https://issues.apache.org/jira/browse/HIVE-17508
> Project: Hive
> Issue Type: Sub-task
>Affects Versions: 3.0.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
> Attachments: HIVE-17508.1.patch, HIVE-17508.2.patch,
> HIVE-17508.WIP.2.patch, HIVE-17508.WIP.patch
>
>
> Workload management can defined Rules that are bound to a resource plan. Each
> rule can have a trigger expression and an action associated with it. Trigger
> expressions are evaluated at runtime after configurable check interval, based
> on which actions like killing a query, moving a query to different pool etc.
> will get invoked. Simple rule could be something like
> {code}
> CREATE RULE slow_query IN resource_plan_name
> WHEN execution_time_ms > 1
> MOVE TO slow_queue
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-17515) Use SHA-256 for GenericUDFMaskHash to improve security
[ https://issues.apache.org/jira/browse/HIVE-17515?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16171896#comment-16171896 ] Tao Li commented on HIVE-17515: --- [~kgyrtkirk] Thanks for that fix! > Use SHA-256 for GenericUDFMaskHash to improve security > -- > > Key: HIVE-17515 > URL: https://issues.apache.org/jira/browse/HIVE-17515 > Project: Hive > Issue Type: Sub-task > Components: UDF >Reporter: Tao Li >Assignee: Tao Li > Fix For: 3.0.0 > > Attachments: HIVE-17515.1.patch > > > See HIVE-17226 for detailed description. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17555) StatsUtils considers all ranges to be 'long'; and loose precision / introduce bugs in some cases
[
https://issues.apache.org/jira/browse/HIVE-17555?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16171893#comment-16171893
]
Zoltan Haindrich commented on HIVE-17555:
-
seems like the affected methods are only plays some roles in case of row number
estimations...that's not that critical..
> StatsUtils considers all ranges to be 'long'; and loose precision / introduce
> bugs in some cases
>
>
> Key: HIVE-17555
> URL: https://issues.apache.org/jira/browse/HIVE-17555
> Project: Hive
> Issue Type: Bug
> Components: Statistics
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>
> The following test fails because the combined range is: {{\[0:0\]}}
> This problem is present in other methods of StatsUtil as well
> {code}
> package org.apache.hadoop.hive.ql.stats;
> import static org.junit.Assert.assertTrue;
> import org.apache.hadoop.hive.ql.plan.ColStatistics.Range;
> import org.junit.Test;
> public class TestStatsUtils {
> @Test
> public void test1() {
> Range r1 = new Range(0.1f, 0.4f);
> Range r2 = new Range(0.3f, 0.9f);
> assertTrue(rangeContains(r1, 0.2f));
> Range r3 = StatsUtils.combineRange(r1, r2);
> System.out.println(r3);
> assertTrue(rangeContains(r3, 0.2f));
> }
> private boolean rangeContains(Range range, Number f) {
> double m = range.minValue.doubleValue();
> double M = range.maxValue.doubleValue();
> double v = f.doubleValue();
> return m <= v && v <= M;
> }
> }
> {code}
> https://github.com/apache/hive/blob/32e854ef1c25f21d53f7932723cfc76bf75a71cd/ql/src/java/org/apache/hadoop/hive/ql/stats/StatsUtils.java#L1955
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Assigned] (HIVE-17555) StatsUtils considers all ranges to be 'long'; and loose precision / introduce bugs in some cases
[
https://issues.apache.org/jira/browse/HIVE-17555?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zoltan Haindrich reassigned HIVE-17555:
---
Assignee: Zoltan Haindrich
> StatsUtils considers all ranges to be 'long'; and loose precision / introduce
> bugs in some cases
>
>
> Key: HIVE-17555
> URL: https://issues.apache.org/jira/browse/HIVE-17555
> Project: Hive
> Issue Type: Bug
> Components: Statistics
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>
> The following test fails because the combined range is: {{\[0:0\]}}
> This problem is present in other methods of StatsUtil as well
> {code}
> package org.apache.hadoop.hive.ql.stats;
> import static org.junit.Assert.assertTrue;
> import org.apache.hadoop.hive.ql.plan.ColStatistics.Range;
> import org.junit.Test;
> public class TestStatsUtils {
> @Test
> public void test1() {
> Range r1 = new Range(0.1f, 0.4f);
> Range r2 = new Range(0.3f, 0.9f);
> assertTrue(rangeContains(r1, 0.2f));
> Range r3 = StatsUtils.combineRange(r1, r2);
> System.out.println(r3);
> assertTrue(rangeContains(r3, 0.2f));
> }
> private boolean rangeContains(Range range, Number f) {
> double m = range.minValue.doubleValue();
> double M = range.maxValue.doubleValue();
> double v = f.doubleValue();
> return m <= v && v <= M;
> }
> }
> {code}
> https://github.com/apache/hive/blob/32e854ef1c25f21d53f7932723cfc76bf75a71cd/ql/src/java/org/apache/hadoop/hive/ql/stats/StatsUtils.java#L1955
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-15899) Make CTAS with acid target table and insert into acid_tbl select ... union all ... work
[
https://issues.apache.org/jira/browse/HIVE-15899?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16171785#comment-16171785
]
Hive QA commented on HIVE-15899:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12887792/HIVE-15899.11.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 14 failed/errored test(s), 11048 tests
executed
*Failed tests:*
{noformat}
TestAccumuloCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=230)
TestDummy - did not produce a TEST-*.xml file (likely timed out) (batchId=230)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[autoColumnStats_4]
(batchId=12)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[create_view] (batchId=39)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[udf_mask_hash]
(batchId=28)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[acid_no_buckets]
(batchId=159)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[dynamic_semijoin_reduction_3]
(batchId=160)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[dynpart_sort_optimization_acid]
(batchId=155)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sqlmerge]
(batchId=161)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[union_fast_stats]
(batchId=156)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_explainuser_1]
(batchId=170)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[drop_table_failure2]
(batchId=89)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query14]
(batchId=234)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query23]
(batchId=234)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/6883/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/6883/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-6883/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 14 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12887792 - PreCommit-HIVE-Build
> Make CTAS with acid target table and insert into acid_tbl select ... union
> all ... work
> ---
>
> Key: HIVE-15899
> URL: https://issues.apache.org/jira/browse/HIVE-15899
> Project: Hive
> Issue Type: Sub-task
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
> Attachments: HIVE-15899.01.patch, HIVE-15899.02.patch,
> HIVE-15899.03.patch, HIVE-15899.04.patch, HIVE-15899.05.patch,
> HIVE-15899.07.patch, HIVE-15899.08.patch, HIVE-15899.09.patch,
> HIVE-15899.10.patch, HIVE-15899.11.patch
>
>
> need to add a test to check if create table as works correctly with acid
> tables
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-17519) Transpose column stats display
[
https://issues.apache.org/jira/browse/HIVE-17519?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zoltan Haindrich updated HIVE-17519:
Attachment: HIVE-17519.02.patch
#2) add q.out changes...
* most of them because of the removal of the newline
* further differences are because of the usage of the formatted output; which
is now transposed
> Transpose column stats display
> --
>
> Key: HIVE-17519
> URL: https://issues.apache.org/jira/browse/HIVE-17519
> Project: Hive
> Issue Type: Improvement
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
> Attachments: HIVE-17519.01.patch, HIVE-17519.02.patch
>
>
> currently {{describe formatted table1 insert_num}} shows the column
> informations in a table like format...which is very hard to read - because
> there are to many columns
> {code}
> # col_namedata_type min
> max num_nulls distinct_count
> avg_col_len max_col_len num_trues
> num_falses comment bitVector
>
>
> insert_numint
>
>
> from deserializer
> {code}
> I think it would be better to show the same information like this:
> {code}
> col_name insert_num
> data_type int
> min
> max
> num_nulls
> distinct_count
> avg_col_len
> max_col_len
> num_trues
> num_falses
> comment from deserializer
> bitVector
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-15079) Hive cannot read Parquet string timetamps as TIMESTAMP data type
[
https://issues.apache.org/jira/browse/HIVE-15079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16171748#comment-16171748
]
Gabor Szadovszky commented on HIVE-15079:
-
Hi, [~kam338uk]. Unfortunately, I don't have time to work on Hive anymore. I
don't think it's fixed. Feel free to contribute ;)
> Hive cannot read Parquet string timetamps as TIMESTAMP data type
>
>
> Key: HIVE-15079
> URL: https://issues.apache.org/jira/browse/HIVE-15079
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Reporter: Sergio Peña
>
> The Hive Wiki for timestamps specifies that strings timestamps can be read by
> Hive.
> https://cwiki.apache.org/confluence/display/Hive/LanguageManual+Types#LanguageManualTypes-Timestamps
> {noformat}
> Supported conversions:
> Integer numeric types: Interpreted as UNIX timestamp in seconds
> Floating point numeric types: Interpreted as UNIX timestamp in seconds with
> decimal precision
> Strings: JDBC compliant java.sql.Timestamp format "-MM-DD
> HH:MM:SS.f" (9 decimal place precision)
> {noformat}
> This works fine with Text table formats, but when Parquet is used, then it
> throws the following exception:
> {noformat}
> java.io.IOException: org.apache.hadoop.hive.ql.metadata.HiveException:
> java.lang.ClassCastException: org.apache.hadoop.io.Text cannot be cast to
> org.apache.hadoop.hive.serde2.io.TimestampWritable
> {noformat}
> How to reproduce
> {noformat}
> > create table t1 (id int, time string) stored as parquet;
> > insert into table t1 values (1,'2016-07-17 14:42:18');
> > alter table t1 replace columns (id int, time timestamp);
> > select * from t1
> {noformat}
> The above example will run fine if you use a TEXT format instead of PARQUET.
> This issue was raised on PARQUET-723
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Assigned] (HIVE-15079) Hive cannot read Parquet string timetamps as TIMESTAMP data type
[
https://issues.apache.org/jira/browse/HIVE-15079?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gabor Szadovszky reassigned HIVE-15079:
---
Assignee: (was: Gabor Szadovszky)
> Hive cannot read Parquet string timetamps as TIMESTAMP data type
>
>
> Key: HIVE-15079
> URL: https://issues.apache.org/jira/browse/HIVE-15079
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Reporter: Sergio Peña
>
> The Hive Wiki for timestamps specifies that strings timestamps can be read by
> Hive.
> https://cwiki.apache.org/confluence/display/Hive/LanguageManual+Types#LanguageManualTypes-Timestamps
> {noformat}
> Supported conversions:
> Integer numeric types: Interpreted as UNIX timestamp in seconds
> Floating point numeric types: Interpreted as UNIX timestamp in seconds with
> decimal precision
> Strings: JDBC compliant java.sql.Timestamp format "-MM-DD
> HH:MM:SS.f" (9 decimal place precision)
> {noformat}
> This works fine with Text table formats, but when Parquet is used, then it
> throws the following exception:
> {noformat}
> java.io.IOException: org.apache.hadoop.hive.ql.metadata.HiveException:
> java.lang.ClassCastException: org.apache.hadoop.io.Text cannot be cast to
> org.apache.hadoop.hive.serde2.io.TimestampWritable
> {noformat}
> How to reproduce
> {noformat}
> > create table t1 (id int, time string) stored as parquet;
> > insert into table t1 values (1,'2016-07-17 14:42:18');
> > alter table t1 replace columns (id int, time timestamp);
> > select * from t1
> {noformat}
> The above example will run fine if you use a TEXT format instead of PARQUET.
> This issue was raised on PARQUET-723
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
