[jira] [Commented] (HIVE-19228) Remove commons-httpclient 3.x usage
[
https://issues.apache.org/jira/browse/HIVE-19228?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16463467#comment-16463467
]
Hive QA commented on HIVE-19228:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12921741/HIVE-19228.5.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 35 failed/errored test(s), 14316 tests
executed
*Failed tests:*
{noformat}
TestDbNotificationListener - did not produce a TEST-*.xml file (likely timed
out) (batchId=247)
TestHCatHiveCompatibility - did not produce a TEST-*.xml file (likely timed
out) (batchId=247)
TestNonCatCallsWithCatalog - did not produce a TEST-*.xml file (likely timed
out) (batchId=217)
TestSequenceFileReadWrite - did not produce a TEST-*.xml file (likely timed
out) (batchId=247)
TestTxnExIm - did not produce a TEST-*.xml file (likely timed out) (batchId=286)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucket_map_join_tez1]
(batchId=175)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucket_map_join_tez2]
(batchId=156)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[materialized_view_create_rewrite_5]
(batchId=154)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[materialized_view_create_rewrite_rebuild_dummy]
(batchId=161)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[materialized_view_create_rewrite_time_window]
(batchId=156)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb]
(batchId=163)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[tez_dynpart_hashjoin_1]
(batchId=174)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[union_fast_stats]
(batchId=167)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[union_stats]
(batchId=159)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_5]
(batchId=105)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[udf_reflect_neg]
(batchId=96)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[udf_test_error]
(batchId=96)
org.apache.hadoop.hive.ql.TestAcidOnTez.testCtasTezUnion (batchId=228)
org.apache.hadoop.hive.ql.TestAcidOnTez.testNonStandardConversion01
(batchId=228)
org.apache.hadoop.hive.ql.TestMTQueries.testMTQueries1 (batchId=232)
org.apache.hadoop.hive.ql.parse.TestCopyUtils.testPrivilegedDistCpWithSameUserAsCurrentDoesNotTryToImpersonate
(batchId=231)
org.apache.hadoop.hive.ql.parse.TestReplicationOnHDFSEncryptedZones.targetAndSourceHaveDifferentEncryptionZoneKeys
(batchId=231)
org.apache.hive.beeline.TestBeeLineWithArgs.testQueryProgress (batchId=235)
org.apache.hive.jdbc.TestSSL.testSSLFetchHttp (batchId=239)
org.apache.hive.jdbc.TestTriggersMoveWorkloadManager.testTriggerMoveAndKill
(batchId=241)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testMultipleTriggers2
(batchId=241)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerCustomCreatedFiles
(batchId=241)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerCustomNonExistent
(batchId=241)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerCustomReadOps
(batchId=241)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerHighBytesRead
(batchId=241)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerHighBytesWrite
(batchId=241)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerHighShuffleBytes
(batchId=241)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerSlowQueryElapsedTime
(batchId=241)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerSlowQueryExecutionTime
(batchId=241)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerVertexRawInputSplitsNoKill
(batchId=241)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/10667/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/10667/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-10667/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 35 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12921741 - PreCommit-HIVE-Build
> Remove commons-httpclient 3.x usage
> ---
>
> Key: HIVE-19228
> URL: https://issues.apache.org/jira/browse/HIVE-19228
> Project: Hive
> Issue Type: Improvement
>Reporter: Janaki Lahorani
>Assignee: Janaki Lahorani
>Priority: Major
> Attac
[jira] [Commented] (HIVE-18977) Listing partitions returns different results with JDO and direct SQL
[
https://issues.apache.org/jira/browse/HIVE-18977?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16463497#comment-16463497
]
Hive QA commented on HIVE-18977:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
1s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
12s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
50s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
26s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
0s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
30s{color} | {color:green} standalone-metastore: The patch generated 0 new +
789 unchanged - 2 fixed = 789 total (was 791) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
0s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 15m 19s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-10668/dev-support/hive-personality.sh
|
| git revision | master / 39917ef |
| Default Java | 1.8.0_111 |
| modules | C: standalone-metastore U: standalone-metastore |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-10668/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Listing partitions returns different results with JDO and direct SQL
>
>
> Key: HIVE-18977
> URL: https://issues.apache.org/jira/browse/HIVE-18977
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 3.0.0
>Reporter: Alan Gates
>Assignee: Marta Kuczora
>Priority: Major
> Attachments: HIVE-18977.1.patch
>
>
> Some of the tests in TestListPartitions fail when using JDO instead of direct
> SQL.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (HIVE-19420) Support LOAD from SeqFile to ORC table
[ https://issues.apache.org/jira/browse/HIVE-19420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Deepak Jaiswal reassigned HIVE-19420: - > Support LOAD from SeqFile to ORC table > -- > > Key: HIVE-19420 > URL: https://issues.apache.org/jira/browse/HIVE-19420 > Project: Hive > Issue Type: Bug >Reporter: Deepak Jaiswal >Assignee: Deepak Jaiswal >Priority: Major > > Support LOAD from SeqFile to ORC table. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19420) Support LOAD from SeqFile to ORC table
[ https://issues.apache.org/jira/browse/HIVE-19420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Deepak Jaiswal updated HIVE-19420: -- Status: Patch Available (was: Open) > Support LOAD from SeqFile to ORC table > -- > > Key: HIVE-19420 > URL: https://issues.apache.org/jira/browse/HIVE-19420 > Project: Hive > Issue Type: Bug >Reporter: Deepak Jaiswal >Assignee: Deepak Jaiswal >Priority: Major > > Support LOAD from SeqFile to ORC table. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19420) Support LOAD from SeqFile to ORC table
[ https://issues.apache.org/jira/browse/HIVE-19420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Deepak Jaiswal updated HIVE-19420: -- Attachment: HIVE-19420.1.patch > Support LOAD from SeqFile to ORC table > -- > > Key: HIVE-19420 > URL: https://issues.apache.org/jira/browse/HIVE-19420 > Project: Hive > Issue Type: Bug >Reporter: Deepak Jaiswal >Assignee: Deepak Jaiswal >Priority: Major > Attachments: HIVE-19420.1.patch > > > Support LOAD from SeqFile to ORC table. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19420) Support LOAD from SeqFile to ORC table
[ https://issues.apache.org/jira/browse/HIVE-19420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16463523#comment-16463523 ] Deepak Jaiswal commented on HIVE-19420: --- [~prasanth_j] and [~ewohlstadter] can you please review? > Support LOAD from SeqFile to ORC table > -- > > Key: HIVE-19420 > URL: https://issues.apache.org/jira/browse/HIVE-19420 > Project: Hive > Issue Type: Bug >Reporter: Deepak Jaiswal >Assignee: Deepak Jaiswal >Priority: Major > Attachments: HIVE-19420.1.patch > > > Support LOAD from SeqFile to ORC table. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work started] (HIVE-19248) REPL LOAD doesn't throw error if file copy fails.
[
https://issues.apache.org/jira/browse/HIVE-19248?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on HIVE-19248 started by Sankar Hariappan.
---
> REPL LOAD doesn't throw error if file copy fails.
> -
>
> Key: HIVE-19248
> URL: https://issues.apache.org/jira/browse/HIVE-19248
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2, repl
>Affects Versions: 3.0.0
>Reporter: Sankar Hariappan
>Assignee: Sankar Hariappan
>Priority: Major
> Labels: DR, pull-request-available, replication
> Fix For: 3.1.0
>
>
> Hive replication uses Hadoop distcp to copy files from primary to replica
> warehouse. If the HDFS block size is different across clusters, it cause file
> copy failures.
> {code:java}
> 2018-04-09 14:32:06,690 ERROR [main]
> org.apache.hadoop.tools.mapred.CopyMapper: Failure in copying
> hdfs://chelsea/apps/hive/warehouse/tpch_flat_orc_1000.db/customer/000259_0 to
> hdfs://marilyn/apps/hive/warehouse/tpch_flat_orc_1000.db/customer/.hive-staging_hive_2018-04-09_14-30-45_723_7153496419225102220-2/-ext-10001/000259_0
> java.io.IOException: File copy failed:
> hdfs://chelsea/apps/hive/warehouse/tpch_flat_orc_1000.db/customer/000259_0
> -->
> hdfs://marilyn/apps/hive/warehouse/tpch_flat_orc_1000.db/customer/.hive-staging_hive_2018-04-09_14-30-45_723_7153496419225102220-2/-ext-10001/000259_0
> at
> org.apache.hadoop.tools.mapred.CopyMapper.copyFileWithRetry(CopyMapper.java:299)
> at org.apache.hadoop.tools.mapred.CopyMapper.map(CopyMapper.java:266)
> at org.apache.hadoop.tools.mapred.CopyMapper.map(CopyMapper.java:52)
> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:146)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:787)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:170)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1869)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:164)
> Caused by: java.io.IOException: Couldn't run retriable-command: Copying
> hdfs://chelsea/apps/hive/warehouse/tpch_flat_orc_1000.db/customer/000259_0 to
> hdfs://marilyn/apps/hive/warehouse/tpch_flat_orc_1000.db/customer/.hive-staging_hive_2018-04-09_14-30-45_723_7153496419225102220-2/-ext-10001/000259_0
> at
> org.apache.hadoop.tools.util.RetriableCommand.execute(RetriableCommand.java:101)
> at
> org.apache.hadoop.tools.mapred.CopyMapper.copyFileWithRetry(CopyMapper.java:296)
> ... 10 more
> Caused by: java.io.IOException: Check-sum mismatch between
> hdfs://chelsea/apps/hive/warehouse/tpch_flat_orc_1000.db/customer/000259_0
> and
> hdfs://marilyn/apps/hive/warehouse/tpch_flat_orc_1000.db/customer/.hive-staging_hive_2018-04-09_14-30-45_723_7153496419225102220-2/-ext-10001/.distcp.tmp.attempt_1522833620762_4416_m_00_0.
> Source and target differ in block-size. Use -pb to preserve block-sizes
> during copy. Alternatively, skip checksum-checks altogether, using -skipCrc.
> (NOTE: By skipping checksums, one runs the risk of masking data-corruption
> during file-transfer.)
> at
> org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.compareCheckSums(RetriableFileCopyCommand.java:212)
> at
> org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doCopy(RetriableFileCopyCommand.java:130)
> at
> org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doExecute(RetriableFileCopyCommand.java:99)
> at
> org.apache.hadoop.tools.util.RetriableCommand.execute(RetriableCommand.java:87)
> ... 11 more
> {code}
> But REPL LOAD returns success even if distcp jobs failed.
> CopyUtils.doCopyRetry doesn't throw error if copy failed even after maximum
> attempts.
> So, need to perform 2 things.
> # If copy of multiple files fail for some reason, need to check if any files
> completely copied by verifying the checksum and file size and skip those from
> retry.
> # If source path is moved to CM path, then delete the incorrectly copied
> files.
> # If copy fails for maximum attempt, then throw error.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19248) REPL LOAD fails when try to access source CM path and also doesn't throw error if file copy fails.
[
https://issues.apache.org/jira/browse/HIVE-19248?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sankar Hariappan updated HIVE-19248:
Summary: REPL LOAD fails when try to access source CM path and also doesn't
throw error if file copy fails. (was: REPL LOAD doesn't throw error if file
copy fails.)
> REPL LOAD fails when try to access source CM path and also doesn't throw
> error if file copy fails.
> --
>
> Key: HIVE-19248
> URL: https://issues.apache.org/jira/browse/HIVE-19248
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2, repl
>Affects Versions: 3.0.0
>Reporter: Sankar Hariappan
>Assignee: Sankar Hariappan
>Priority: Major
> Labels: DR, pull-request-available, replication
> Fix For: 3.1.0
>
>
> Hive replication uses Hadoop distcp to copy files from primary to replica
> warehouse. If the HDFS block size is different across clusters, it cause file
> copy failures.
> {code:java}
> 2018-04-09 14:32:06,690 ERROR [main]
> org.apache.hadoop.tools.mapred.CopyMapper: Failure in copying
> hdfs://chelsea/apps/hive/warehouse/tpch_flat_orc_1000.db/customer/000259_0 to
> hdfs://marilyn/apps/hive/warehouse/tpch_flat_orc_1000.db/customer/.hive-staging_hive_2018-04-09_14-30-45_723_7153496419225102220-2/-ext-10001/000259_0
> java.io.IOException: File copy failed:
> hdfs://chelsea/apps/hive/warehouse/tpch_flat_orc_1000.db/customer/000259_0
> -->
> hdfs://marilyn/apps/hive/warehouse/tpch_flat_orc_1000.db/customer/.hive-staging_hive_2018-04-09_14-30-45_723_7153496419225102220-2/-ext-10001/000259_0
> at
> org.apache.hadoop.tools.mapred.CopyMapper.copyFileWithRetry(CopyMapper.java:299)
> at org.apache.hadoop.tools.mapred.CopyMapper.map(CopyMapper.java:266)
> at org.apache.hadoop.tools.mapred.CopyMapper.map(CopyMapper.java:52)
> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:146)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:787)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:170)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1869)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:164)
> Caused by: java.io.IOException: Couldn't run retriable-command: Copying
> hdfs://chelsea/apps/hive/warehouse/tpch_flat_orc_1000.db/customer/000259_0 to
> hdfs://marilyn/apps/hive/warehouse/tpch_flat_orc_1000.db/customer/.hive-staging_hive_2018-04-09_14-30-45_723_7153496419225102220-2/-ext-10001/000259_0
> at
> org.apache.hadoop.tools.util.RetriableCommand.execute(RetriableCommand.java:101)
> at
> org.apache.hadoop.tools.mapred.CopyMapper.copyFileWithRetry(CopyMapper.java:296)
> ... 10 more
> Caused by: java.io.IOException: Check-sum mismatch between
> hdfs://chelsea/apps/hive/warehouse/tpch_flat_orc_1000.db/customer/000259_0
> and
> hdfs://marilyn/apps/hive/warehouse/tpch_flat_orc_1000.db/customer/.hive-staging_hive_2018-04-09_14-30-45_723_7153496419225102220-2/-ext-10001/.distcp.tmp.attempt_1522833620762_4416_m_00_0.
> Source and target differ in block-size. Use -pb to preserve block-sizes
> during copy. Alternatively, skip checksum-checks altogether, using -skipCrc.
> (NOTE: By skipping checksums, one runs the risk of masking data-corruption
> during file-transfer.)
> at
> org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.compareCheckSums(RetriableFileCopyCommand.java:212)
> at
> org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doCopy(RetriableFileCopyCommand.java:130)
> at
> org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doExecute(RetriableFileCopyCommand.java:99)
> at
> org.apache.hadoop.tools.util.RetriableCommand.execute(RetriableCommand.java:87)
> ... 11 more
> {code}
> But REPL LOAD returns success even if distcp jobs failed.
> CopyUtils.doCopyRetry doesn't throw error if copy failed even after maximum
> attempts.
> So, need to perform 2 things.
> # If copy of multiple files fail for some reason, need to check if any files
> completely copied by verifying the checksum and file size and skip those from
> retry.
> # If source path is moved to CM path, then delete the incorrectly copied
> files.
> # If copy fails for maximum attempt, then throw error.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19248) REPL LOAD fails to copy file from source CM path and also doesn't throw error if file copy fails.
[
https://issues.apache.org/jira/browse/HIVE-19248?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sankar Hariappan updated HIVE-19248:
Summary: REPL LOAD fails to copy file from source CM path and also doesn't
throw error if file copy fails. (was: REPL LOAD fails when try to access
source CM path and also doesn't throw error if file copy fails.)
> REPL LOAD fails to copy file from source CM path and also doesn't throw error
> if file copy fails.
> -
>
> Key: HIVE-19248
> URL: https://issues.apache.org/jira/browse/HIVE-19248
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2, repl
>Affects Versions: 3.0.0
>Reporter: Sankar Hariappan
>Assignee: Sankar Hariappan
>Priority: Major
> Labels: DR, pull-request-available, replication
> Fix For: 3.1.0
>
>
> Hive replication uses Hadoop distcp to copy files from primary to replica
> warehouse. If the HDFS block size is different across clusters, it cause file
> copy failures.
> {code:java}
> 2018-04-09 14:32:06,690 ERROR [main]
> org.apache.hadoop.tools.mapred.CopyMapper: Failure in copying
> hdfs://chelsea/apps/hive/warehouse/tpch_flat_orc_1000.db/customer/000259_0 to
> hdfs://marilyn/apps/hive/warehouse/tpch_flat_orc_1000.db/customer/.hive-staging_hive_2018-04-09_14-30-45_723_7153496419225102220-2/-ext-10001/000259_0
> java.io.IOException: File copy failed:
> hdfs://chelsea/apps/hive/warehouse/tpch_flat_orc_1000.db/customer/000259_0
> -->
> hdfs://marilyn/apps/hive/warehouse/tpch_flat_orc_1000.db/customer/.hive-staging_hive_2018-04-09_14-30-45_723_7153496419225102220-2/-ext-10001/000259_0
> at
> org.apache.hadoop.tools.mapred.CopyMapper.copyFileWithRetry(CopyMapper.java:299)
> at org.apache.hadoop.tools.mapred.CopyMapper.map(CopyMapper.java:266)
> at org.apache.hadoop.tools.mapred.CopyMapper.map(CopyMapper.java:52)
> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:146)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:787)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:170)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1869)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:164)
> Caused by: java.io.IOException: Couldn't run retriable-command: Copying
> hdfs://chelsea/apps/hive/warehouse/tpch_flat_orc_1000.db/customer/000259_0 to
> hdfs://marilyn/apps/hive/warehouse/tpch_flat_orc_1000.db/customer/.hive-staging_hive_2018-04-09_14-30-45_723_7153496419225102220-2/-ext-10001/000259_0
> at
> org.apache.hadoop.tools.util.RetriableCommand.execute(RetriableCommand.java:101)
> at
> org.apache.hadoop.tools.mapred.CopyMapper.copyFileWithRetry(CopyMapper.java:296)
> ... 10 more
> Caused by: java.io.IOException: Check-sum mismatch between
> hdfs://chelsea/apps/hive/warehouse/tpch_flat_orc_1000.db/customer/000259_0
> and
> hdfs://marilyn/apps/hive/warehouse/tpch_flat_orc_1000.db/customer/.hive-staging_hive_2018-04-09_14-30-45_723_7153496419225102220-2/-ext-10001/.distcp.tmp.attempt_1522833620762_4416_m_00_0.
> Source and target differ in block-size. Use -pb to preserve block-sizes
> during copy. Alternatively, skip checksum-checks altogether, using -skipCrc.
> (NOTE: By skipping checksums, one runs the risk of masking data-corruption
> during file-transfer.)
> at
> org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.compareCheckSums(RetriableFileCopyCommand.java:212)
> at
> org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doCopy(RetriableFileCopyCommand.java:130)
> at
> org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doExecute(RetriableFileCopyCommand.java:99)
> at
> org.apache.hadoop.tools.util.RetriableCommand.execute(RetriableCommand.java:87)
> ... 11 more
> {code}
> But REPL LOAD returns success even if distcp jobs failed.
> CopyUtils.doCopyRetry doesn't throw error if copy failed even after maximum
> attempts.
> So, need to perform 2 things.
> # If copy of multiple files fail for some reason, need to check if any files
> completely copied by verifying the checksum and file size and skip those from
> retry.
> # If source path is moved to CM path, then delete the incorrectly copied
> files.
> # If copy fails for maximum attempt, then throw error.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18977) Listing partitions returns different results with JDO and direct SQL
[
https://issues.apache.org/jira/browse/HIVE-18977?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16463570#comment-16463570
]
Hive QA commented on HIVE-18977:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12921771/HIVE-18977.1.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 35 failed/errored test(s), 14320 tests
executed
*Failed tests:*
{noformat}
TestDbNotificationListener - did not produce a TEST-*.xml file (likely timed
out) (batchId=247)
TestHCatHiveCompatibility - did not produce a TEST-*.xml file (likely timed
out) (batchId=247)
TestNonCatCallsWithCatalog - did not produce a TEST-*.xml file (likely timed
out) (batchId=217)
TestSequenceFileReadWrite - did not produce a TEST-*.xml file (likely timed
out) (batchId=247)
TestTxnExIm - did not produce a TEST-*.xml file (likely timed out) (batchId=286)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucket_map_join_tez1]
(batchId=175)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucket_map_join_tez2]
(batchId=156)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[explainuser_4]
(batchId=164)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[materialized_view_create_rewrite_5]
(batchId=154)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[materialized_view_create_rewrite_rebuild_dummy]
(batchId=161)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[materialized_view_create_rewrite_time_window]
(batchId=156)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb]
(batchId=163)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[tez_dynpart_hashjoin_1]
(batchId=174)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[union_fast_stats]
(batchId=167)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_5]
(batchId=105)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[udf_reflect_neg]
(batchId=96)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[udf_test_error]
(batchId=96)
org.apache.hadoop.hive.ql.TestAcidOnTez.testCtasTezUnion (batchId=228)
org.apache.hadoop.hive.ql.TestAcidOnTez.testNonStandardConversion01
(batchId=228)
org.apache.hadoop.hive.ql.TestMTQueries.testMTQueries1 (batchId=232)
org.apache.hadoop.hive.ql.parse.TestCopyUtils.testPrivilegedDistCpWithSameUserAsCurrentDoesNotTryToImpersonate
(batchId=231)
org.apache.hadoop.hive.ql.parse.TestReplicationOnHDFSEncryptedZones.targetAndSourceHaveDifferentEncryptionZoneKeys
(batchId=231)
org.apache.hive.beeline.TestBeeLineWithArgs.testQueryProgress (batchId=235)
org.apache.hive.beeline.TestBeeLineWithArgs.testQueryProgressParallel
(batchId=235)
org.apache.hive.jdbc.TestSSL.testSSLFetchHttp (batchId=239)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testMultipleTriggers2
(batchId=241)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerCustomCreatedFiles
(batchId=241)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerCustomNonExistent
(batchId=241)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerCustomReadOps
(batchId=241)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerHighBytesRead
(batchId=241)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerHighBytesWrite
(batchId=241)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerHighShuffleBytes
(batchId=241)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerSlowQueryElapsedTime
(batchId=241)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerSlowQueryExecutionTime
(batchId=241)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerVertexRawInputSplitsNoKill
(batchId=241)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/10668/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/10668/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-10668/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 35 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12921771 - PreCommit-HIVE-Build
> Listing partitions returns different results with JDO and direct SQL
>
>
> Key: HIVE-18977
> URL: https://issues.apache.org/jira/browse/HIVE-18977
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 3.0.0
>
[jira] [Updated] (HIVE-19248) REPL LOAD doesn't copy file from source CM path and also doesn't throw error if file copy fails.
[
https://issues.apache.org/jira/browse/HIVE-19248?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sankar Hariappan updated HIVE-19248:
Summary: REPL LOAD doesn't copy file from source CM path and also doesn't
throw error if file copy fails. (was: REPL LOAD fails to copy file from source
CM path and also doesn't throw error if file copy fails.)
> REPL LOAD doesn't copy file from source CM path and also doesn't throw error
> if file copy fails.
>
>
> Key: HIVE-19248
> URL: https://issues.apache.org/jira/browse/HIVE-19248
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2, repl
>Affects Versions: 3.0.0
>Reporter: Sankar Hariappan
>Assignee: Sankar Hariappan
>Priority: Major
> Labels: DR, pull-request-available, replication
> Fix For: 3.1.0
>
>
> Hive replication uses Hadoop distcp to copy files from primary to replica
> warehouse. If the HDFS block size is different across clusters, it cause file
> copy failures.
> {code:java}
> 2018-04-09 14:32:06,690 ERROR [main]
> org.apache.hadoop.tools.mapred.CopyMapper: Failure in copying
> hdfs://chelsea/apps/hive/warehouse/tpch_flat_orc_1000.db/customer/000259_0 to
> hdfs://marilyn/apps/hive/warehouse/tpch_flat_orc_1000.db/customer/.hive-staging_hive_2018-04-09_14-30-45_723_7153496419225102220-2/-ext-10001/000259_0
> java.io.IOException: File copy failed:
> hdfs://chelsea/apps/hive/warehouse/tpch_flat_orc_1000.db/customer/000259_0
> -->
> hdfs://marilyn/apps/hive/warehouse/tpch_flat_orc_1000.db/customer/.hive-staging_hive_2018-04-09_14-30-45_723_7153496419225102220-2/-ext-10001/000259_0
> at
> org.apache.hadoop.tools.mapred.CopyMapper.copyFileWithRetry(CopyMapper.java:299)
> at org.apache.hadoop.tools.mapred.CopyMapper.map(CopyMapper.java:266)
> at org.apache.hadoop.tools.mapred.CopyMapper.map(CopyMapper.java:52)
> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:146)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:787)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:170)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1869)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:164)
> Caused by: java.io.IOException: Couldn't run retriable-command: Copying
> hdfs://chelsea/apps/hive/warehouse/tpch_flat_orc_1000.db/customer/000259_0 to
> hdfs://marilyn/apps/hive/warehouse/tpch_flat_orc_1000.db/customer/.hive-staging_hive_2018-04-09_14-30-45_723_7153496419225102220-2/-ext-10001/000259_0
> at
> org.apache.hadoop.tools.util.RetriableCommand.execute(RetriableCommand.java:101)
> at
> org.apache.hadoop.tools.mapred.CopyMapper.copyFileWithRetry(CopyMapper.java:296)
> ... 10 more
> Caused by: java.io.IOException: Check-sum mismatch between
> hdfs://chelsea/apps/hive/warehouse/tpch_flat_orc_1000.db/customer/000259_0
> and
> hdfs://marilyn/apps/hive/warehouse/tpch_flat_orc_1000.db/customer/.hive-staging_hive_2018-04-09_14-30-45_723_7153496419225102220-2/-ext-10001/.distcp.tmp.attempt_1522833620762_4416_m_00_0.
> Source and target differ in block-size. Use -pb to preserve block-sizes
> during copy. Alternatively, skip checksum-checks altogether, using -skipCrc.
> (NOTE: By skipping checksums, one runs the risk of masking data-corruption
> during file-transfer.)
> at
> org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.compareCheckSums(RetriableFileCopyCommand.java:212)
> at
> org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doCopy(RetriableFileCopyCommand.java:130)
> at
> org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doExecute(RetriableFileCopyCommand.java:99)
> at
> org.apache.hadoop.tools.util.RetriableCommand.execute(RetriableCommand.java:87)
> ... 11 more
> {code}
> But REPL LOAD returns success even if distcp jobs failed.
> CopyUtils.doCopyRetry doesn't throw error if copy failed even after maximum
> attempts.
> So, need to perform 2 things.
> # If copy of multiple files fail for some reason, need to check if any files
> completely copied by verifying the checksum and file size and skip those from
> retry.
> # If source path is moved to CM path, then delete the incorrectly copied
> files.
> # If copy fails for maximum attempt, then throw error.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19248) REPL LOAD doesn't copy file from source CM path and also doesn't throw error if file copy fails.
[
https://issues.apache.org/jira/browse/HIVE-19248?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sankar Hariappan updated HIVE-19248:
Description:
Hive replication uses Hadoop distcp to copy files from primary to replica
warehouse. If the HDFS block size is different across clusters, it cause file
copy failures.
{code:java}
2018-04-09 14:32:06,690 ERROR [main] org.apache.hadoop.tools.mapred.CopyMapper:
Failure in copying
hdfs://chelsea/apps/hive/warehouse/tpch_flat_orc_1000.db/customer/000259_0 to
hdfs://marilyn/apps/hive/warehouse/tpch_flat_orc_1000.db/customer/.hive-staging_hive_2018-04-09_14-30-45_723_7153496419225102220-2/-ext-10001/000259_0
java.io.IOException: File copy failed:
hdfs://chelsea/apps/hive/warehouse/tpch_flat_orc_1000.db/customer/000259_0 -->
hdfs://marilyn/apps/hive/warehouse/tpch_flat_orc_1000.db/customer/.hive-staging_hive_2018-04-09_14-30-45_723_7153496419225102220-2/-ext-10001/000259_0
at
org.apache.hadoop.tools.mapred.CopyMapper.copyFileWithRetry(CopyMapper.java:299)
at org.apache.hadoop.tools.mapred.CopyMapper.map(CopyMapper.java:266)
at org.apache.hadoop.tools.mapred.CopyMapper.map(CopyMapper.java:52)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:146)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:787)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:170)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1869)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:164)
Caused by: java.io.IOException: Couldn't run retriable-command: Copying
hdfs://chelsea/apps/hive/warehouse/tpch_flat_orc_1000.db/customer/000259_0 to
hdfs://marilyn/apps/hive/warehouse/tpch_flat_orc_1000.db/customer/.hive-staging_hive_2018-04-09_14-30-45_723_7153496419225102220-2/-ext-10001/000259_0
at
org.apache.hadoop.tools.util.RetriableCommand.execute(RetriableCommand.java:101)
at
org.apache.hadoop.tools.mapred.CopyMapper.copyFileWithRetry(CopyMapper.java:296)
... 10 more
Caused by: java.io.IOException: Check-sum mismatch between
hdfs://chelsea/apps/hive/warehouse/tpch_flat_orc_1000.db/customer/000259_0 and
hdfs://marilyn/apps/hive/warehouse/tpch_flat_orc_1000.db/customer/.hive-staging_hive_2018-04-09_14-30-45_723_7153496419225102220-2/-ext-10001/.distcp.tmp.attempt_1522833620762_4416_m_00_0.
Source and target differ in block-size. Use -pb to preserve block-sizes during
copy. Alternatively, skip checksum-checks altogether, using -skipCrc. (NOTE: By
skipping checksums, one runs the risk of masking data-corruption during
file-transfer.)
at
org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.compareCheckSums(RetriableFileCopyCommand.java:212)
at
org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doCopy(RetriableFileCopyCommand.java:130)
at
org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doExecute(RetriableFileCopyCommand.java:99)
at
org.apache.hadoop.tools.util.RetriableCommand.execute(RetriableCommand.java:87)
... 11 more
{code}
Distcp failed as the CM path for the file doesn't point to source file system.
So, it is needed to get the qualified cm root URI as part of files listed in
dump.
Also, REPL LOAD returns success even if distcp jobs failed.
CopyUtils.doCopyRetry doesn't throw error if copy failed even after maximum
attempts.
So, need to perform 2 things.
# If copy of multiple files fail for some reason, then retry with same set of
files again but need to set CM path if original source file is missing or
modified based on checksum. Let distcp to skip the properly copied files.
FileUtil.copy will always overwrite the files.
# If source path is moved to CM path, then delete the incorrectly copied files.
# If copy fails for maximum attempt, then throw error.
was:
Hive replication uses Hadoop distcp to copy files from primary to replica
warehouse. If the HDFS block size is different across clusters, it cause file
copy failures.
{code:java}
2018-04-09 14:32:06,690 ERROR [main] org.apache.hadoop.tools.mapred.CopyMapper:
Failure in copying
hdfs://chelsea/apps/hive/warehouse/tpch_flat_orc_1000.db/customer/000259_0 to
hdfs://marilyn/apps/hive/warehouse/tpch_flat_orc_1000.db/customer/.hive-staging_hive_2018-04-09_14-30-45_723_7153496419225102220-2/-ext-10001/000259_0
java.io.IOException: File copy failed:
hdfs://chelsea/apps/hive/warehouse/tpch_flat_orc_1000.db/customer/000259_0 -->
hdfs://marilyn/apps/hive/warehouse/tpch_flat_orc_1000.db/customer/.hive-staging_hive_2018-04-09_14-30-45_723_7153496419225102220-2/-ext-10001/000259_0
at
org.apache.hadoop.tools.mapred.CopyMapper.copyFileWithRetry(CopyMapper.java:299)
at org.apache.hadoop.tools.mapred.CopyMapper.map(CopyMapper.java:266)
at org.apache.hadoop.tools.mapred.Co
[jira] [Updated] (HIVE-18906) Lower Logging for "Using direct SQL"
[
https://issues.apache.org/jira/browse/HIVE-18906?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Antal Sinkovits updated HIVE-18906:
---
Attachment: HIVE-18906.1.patch
> Lower Logging for "Using direct SQL"
>
>
> Key: HIVE-18906
> URL: https://issues.apache.org/jira/browse/HIVE-18906
> Project: Hive
> Issue Type: Improvement
>Reporter: BELUGA BEHR
>Assignee: Antal Sinkovits
>Priority: Trivial
> Labels: noob
> Attachments: HIVE-18906.1.patch
>
>
> https://github.com/apache/hive/blob/4047befe48c8f762c58d8854e058385c1df151c6/standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java#L181
> {code}
> 2018-02-26 14:32:50,620 INFO
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql: [pool-4-thread-199]:
> Using direct SQL, underlying DB is MYSQL
> 2018-02-26 14:33:09,566 INFO
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql: [pool-4-thread-172]:
> Using direct SQL, underlying DB is MYSQL
> 2018-02-26 14:33:43,170 INFO
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql: [pool-4-thread-172]:
> Using direct SQL, underlying DB is MYSQL
> 2018-02-26 14:33:46,301 INFO
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql: [pool-4-thread-199]:
> Using direct SQL, underlying DB is MYSQL
> 2018-02-26 14:34:16,966 INFO
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql: [pool-4-thread-172]:
> Using direct SQL, underlying DB is MYSQL
> 2018-02-26 14:34:18,072 INFO
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql: [pool-4-thread-196]:
> Using direct SQL, underlying DB is MYSQL
> 2018-02-26 14:34:25,064 INFO
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql: [pool-4-thread-199]:
> Using direct SQL, underlying DB is MYSQL
> 2018-02-26 14:34:51,781 INFO
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql: [pool-4-thread-172]:
> Using direct SQL, underlying DB is MYSQL
> 2018-02-26 14:35:04,685 INFO
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql: [pool-4-thread-199]:
> Using direct SQL, underlying DB is MYSQL
> 2018-02-26 14:35:26,689 INFO
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql: [pool-4-thread-172]:
> Using direct SQL, underlying DB is MYSQL
> {code}
> I wouldn't mind knowing this information when the service starts up and
> detects that there is a Direct SQL compatible database, but I don't need to
> know about this every time.
> Please lower to _debug_ level logging and parameterize with SLF4J. Direct
> SQL is a performance thing and not something I care about during the normal
> course of operation (unless there is an error which would presumably be
> covered by an _error_ logging message).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18906) Lower Logging for "Using direct SQL"
[
https://issues.apache.org/jira/browse/HIVE-18906?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Antal Sinkovits updated HIVE-18906:
---
Status: Patch Available (was: Open)
> Lower Logging for "Using direct SQL"
>
>
> Key: HIVE-18906
> URL: https://issues.apache.org/jira/browse/HIVE-18906
> Project: Hive
> Issue Type: Improvement
>Reporter: BELUGA BEHR
>Assignee: Antal Sinkovits
>Priority: Trivial
> Labels: noob
> Attachments: HIVE-18906.1.patch
>
>
> https://github.com/apache/hive/blob/4047befe48c8f762c58d8854e058385c1df151c6/standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java#L181
> {code}
> 2018-02-26 14:32:50,620 INFO
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql: [pool-4-thread-199]:
> Using direct SQL, underlying DB is MYSQL
> 2018-02-26 14:33:09,566 INFO
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql: [pool-4-thread-172]:
> Using direct SQL, underlying DB is MYSQL
> 2018-02-26 14:33:43,170 INFO
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql: [pool-4-thread-172]:
> Using direct SQL, underlying DB is MYSQL
> 2018-02-26 14:33:46,301 INFO
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql: [pool-4-thread-199]:
> Using direct SQL, underlying DB is MYSQL
> 2018-02-26 14:34:16,966 INFO
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql: [pool-4-thread-172]:
> Using direct SQL, underlying DB is MYSQL
> 2018-02-26 14:34:18,072 INFO
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql: [pool-4-thread-196]:
> Using direct SQL, underlying DB is MYSQL
> 2018-02-26 14:34:25,064 INFO
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql: [pool-4-thread-199]:
> Using direct SQL, underlying DB is MYSQL
> 2018-02-26 14:34:51,781 INFO
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql: [pool-4-thread-172]:
> Using direct SQL, underlying DB is MYSQL
> 2018-02-26 14:35:04,685 INFO
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql: [pool-4-thread-199]:
> Using direct SQL, underlying DB is MYSQL
> 2018-02-26 14:35:26,689 INFO
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql: [pool-4-thread-172]:
> Using direct SQL, underlying DB is MYSQL
> {code}
> I wouldn't mind knowing this information when the service starts up and
> detects that there is a Direct SQL compatible database, but I don't need to
> know about this every time.
> Please lower to _debug_ level logging and parameterize with SLF4J. Direct
> SQL is a performance thing and not something I care about during the normal
> course of operation (unless there is an error which would presumably be
> covered by an _error_ logging message).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18193) Migrate existing ACID tables to use write id per table rather than global transaction id
[
https://issues.apache.org/jira/browse/HIVE-18193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16463594#comment-16463594
]
Hive QA commented on HIVE-18193:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
50s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
16s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
5s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
30s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
15s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
9s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
29s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch 2 line(s) with tabs. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
17s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
16s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 16m 57s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-10669/dev-support/hive-personality.sh
|
| git revision | master / 39917ef |
| Default Java | 1.8.0_111 |
| whitespace |
http://104.198.109.242/logs//PreCommit-HIVE-Build-10669/yetus/whitespace-tabs.txt
|
| modules | C: metastore standalone-metastore U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-10669/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Migrate existing ACID tables to use write id per table rather than global
> transaction id
>
>
> Key: HIVE-18193
> URL: https://issues.apache.org/jira/browse/HIVE-18193
> Project: Hive
> Issue Type: Sub-task
> Components: HiveServer2, Transactions
>Affects Versions: 3.0.0
>Reporter: anishek
>Assignee: Sankar Hariappan
>Priority: Blocker
> Labels: ACID, Upgrade
> Attachments: HIVE-18193.01.patch
>
>
> dependent upon HIVE-18192
> For existing ACID Tables we need to update the table level write id
> metatables/sequences so any new operations on these tables works seamlessly
> without any conflicting data in existing base/delta files.
> 1. Need to create metadata tables such as NEXT_WRITE_ID and TXN_TO_WRITE_ID.
> 2. Add entries for each ACID/MM tables into NEXT_WRITE_ID where NWI_NEXT is
> set to current value of NEXT_TXN_ID.NTXN_NEXT.
> 3. All current open/abort transactions to have an entry in TXN_TO_WRITE_ID
> such that T2W_TXNID=T2W_WRITEID=Open/AbortedTxnId.
> 4. Added new column TC_WRITEID in TXN_COMPONENTS and CTC_WRITEID in
> COMPLETED_TXN_COMPONENTS to store the write id which should be set as
> respective values of TC_TXNID and CTC_TXNID from the same row.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19248) REPL LOAD couldn't copy file from source CM path and also doesn't throw error if file copy fails.
[
https://issues.apache.org/jira/browse/HIVE-19248?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sankar Hariappan updated HIVE-19248:
Summary: REPL LOAD couldn't copy file from source CM path and also doesn't
throw error if file copy fails. (was: REPL LOAD doesn't copy file from source
CM path and also doesn't throw error if file copy fails.)
> REPL LOAD couldn't copy file from source CM path and also doesn't throw error
> if file copy fails.
> -
>
> Key: HIVE-19248
> URL: https://issues.apache.org/jira/browse/HIVE-19248
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2, repl
>Affects Versions: 3.0.0
>Reporter: Sankar Hariappan
>Assignee: Sankar Hariappan
>Priority: Major
> Labels: DR, pull-request-available, replication
> Fix For: 3.1.0
>
>
> Hive replication uses Hadoop distcp to copy files from primary to replica
> warehouse. If the HDFS block size is different across clusters, it cause file
> copy failures.
> {code:java}
> 2018-04-09 14:32:06,690 ERROR [main]
> org.apache.hadoop.tools.mapred.CopyMapper: Failure in copying
> hdfs://chelsea/apps/hive/warehouse/tpch_flat_orc_1000.db/customer/000259_0 to
> hdfs://marilyn/apps/hive/warehouse/tpch_flat_orc_1000.db/customer/.hive-staging_hive_2018-04-09_14-30-45_723_7153496419225102220-2/-ext-10001/000259_0
> java.io.IOException: File copy failed:
> hdfs://chelsea/apps/hive/warehouse/tpch_flat_orc_1000.db/customer/000259_0
> -->
> hdfs://marilyn/apps/hive/warehouse/tpch_flat_orc_1000.db/customer/.hive-staging_hive_2018-04-09_14-30-45_723_7153496419225102220-2/-ext-10001/000259_0
> at
> org.apache.hadoop.tools.mapred.CopyMapper.copyFileWithRetry(CopyMapper.java:299)
> at org.apache.hadoop.tools.mapred.CopyMapper.map(CopyMapper.java:266)
> at org.apache.hadoop.tools.mapred.CopyMapper.map(CopyMapper.java:52)
> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:146)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:787)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:170)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1869)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:164)
> Caused by: java.io.IOException: Couldn't run retriable-command: Copying
> hdfs://chelsea/apps/hive/warehouse/tpch_flat_orc_1000.db/customer/000259_0 to
> hdfs://marilyn/apps/hive/warehouse/tpch_flat_orc_1000.db/customer/.hive-staging_hive_2018-04-09_14-30-45_723_7153496419225102220-2/-ext-10001/000259_0
> at
> org.apache.hadoop.tools.util.RetriableCommand.execute(RetriableCommand.java:101)
> at
> org.apache.hadoop.tools.mapred.CopyMapper.copyFileWithRetry(CopyMapper.java:296)
> ... 10 more
> Caused by: java.io.IOException: Check-sum mismatch between
> hdfs://chelsea/apps/hive/warehouse/tpch_flat_orc_1000.db/customer/000259_0
> and
> hdfs://marilyn/apps/hive/warehouse/tpch_flat_orc_1000.db/customer/.hive-staging_hive_2018-04-09_14-30-45_723_7153496419225102220-2/-ext-10001/.distcp.tmp.attempt_1522833620762_4416_m_00_0.
> Source and target differ in block-size. Use -pb to preserve block-sizes
> during copy. Alternatively, skip checksum-checks altogether, using -skipCrc.
> (NOTE: By skipping checksums, one runs the risk of masking data-corruption
> during file-transfer.)
> at
> org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.compareCheckSums(RetriableFileCopyCommand.java:212)
> at
> org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doCopy(RetriableFileCopyCommand.java:130)
> at
> org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doExecute(RetriableFileCopyCommand.java:99)
> at
> org.apache.hadoop.tools.util.RetriableCommand.execute(RetriableCommand.java:87)
> ... 11 more
> {code}
> Distcp failed as the CM path for the file doesn't point to source file
> system. So, it is needed to get the qualified cm root URI as part of files
> listed in dump.
> Also, REPL LOAD returns success even if distcp jobs failed.
> CopyUtils.doCopyRetry doesn't throw error if copy failed even after maximum
> attempts.
> So, need to perform 2 things.
> # If copy of multiple files fail for some reason, then retry with same set
> of files again but need to set CM path if original source file is missing or
> modified based on checksum. Let distcp to skip the properly copied files.
> FileUtil.copy will always overwrite the files.
> # If source path is moved to CM path, then delete the incorrectly copied
> files.
> # If copy fails for maximum attempt, then throw error.
>
--
This m
[jira] [Updated] (HIVE-18193) Migrate existing ACID tables to use write id per table rather than global transaction id
[ https://issues.apache.org/jira/browse/HIVE-18193?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sankar Hariappan updated HIVE-18193: Fix Version/s: 3.1.0 3.0.0 > Migrate existing ACID tables to use write id per table rather than global > transaction id > > > Key: HIVE-18193 > URL: https://issues.apache.org/jira/browse/HIVE-18193 > Project: Hive > Issue Type: Sub-task > Components: HiveServer2, Transactions >Affects Versions: 3.0.0 >Reporter: anishek >Assignee: Sankar Hariappan >Priority: Blocker > Labels: ACID, Upgrade > Fix For: 3.0.0, 3.1.0 > > Attachments: HIVE-18193.01.patch > > > dependent upon HIVE-18192 > For existing ACID Tables we need to update the table level write id > metatables/sequences so any new operations on these tables works seamlessly > without any conflicting data in existing base/delta files. > 1. Need to create metadata tables such as NEXT_WRITE_ID and TXN_TO_WRITE_ID. > 2. Add entries for each ACID/MM tables into NEXT_WRITE_ID where NWI_NEXT is > set to current value of NEXT_TXN_ID.NTXN_NEXT. > 3. All current open/abort transactions to have an entry in TXN_TO_WRITE_ID > such that T2W_TXNID=T2W_WRITEID=Open/AbortedTxnId. > 4. Added new column TC_WRITEID in TXN_COMPONENTS and CTC_WRITEID in > COMPLETED_TXN_COMPONENTS to store the write id which should be set as > respective values of TC_TXNID and CTC_TXNID from the same row. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18193) Migrate existing ACID tables to use write id per table rather than global transaction id
[
https://issues.apache.org/jira/browse/HIVE-18193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16463661#comment-16463661
]
Hive QA commented on HIVE-18193:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12921772/HIVE-18193.01.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 83 failed/errored test(s), 14316 tests
executed
*Failed tests:*
{noformat}
TestDbNotificationListener - did not produce a TEST-*.xml file (likely timed
out) (batchId=247)
TestHCatHiveCompatibility - did not produce a TEST-*.xml file (likely timed
out) (batchId=247)
TestNonCatCallsWithCatalog - did not produce a TEST-*.xml file (likely timed
out) (batchId=217)
TestSequenceFileReadWrite - did not produce a TEST-*.xml file (likely timed
out) (batchId=247)
TestTxnExIm - did not produce a TEST-*.xml file (likely timed out) (batchId=286)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucket_map_join_tez1]
(batchId=175)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucket_map_join_tez2]
(batchId=156)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[explainuser_4]
(batchId=164)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[materialized_view_create_rewrite_5]
(batchId=154)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[materialized_view_create_rewrite_rebuild_dummy]
(batchId=161)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[materialized_view_create_rewrite_time_window]
(batchId=156)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb]
(batchId=163)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[union_fast_stats]
(batchId=167)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[bucketizedhiveinputformat]
(batchId=183)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_5]
(batchId=105)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[udf_reflect_neg]
(batchId=96)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[udf_test_error]
(batchId=96)
org.apache.hadoop.hive.metastore.TestHiveMetaStoreTxns.testLocksWithTxn
(batchId=219)
org.apache.hadoop.hive.metastore.tools.TestSchemaToolForMetastore.testMetastoreDbPropertiesAfterUpgrade
(batchId=211)
org.apache.hadoop.hive.metastore.tools.TestSchemaToolForMetastore.testSchemaUpgrade
(batchId=211)
org.apache.hadoop.hive.metastore.tools.TestSchemaToolForMetastore.testValidateSchemaTables
(batchId=211)
org.apache.hadoop.hive.metastore.txn.TestCompactionTxnHandler.testFindPotentialCompactions
(batchId=264)
org.apache.hadoop.hive.metastore.txn.TestCompactionTxnHandler.testMarkCleanedCleansTxnsAndTxnComponents
(batchId=264)
org.apache.hadoop.hive.metastore.txn.TestTxnHandler.testCheckLockAcquireAfterWaiting
(batchId=264)
org.apache.hadoop.hive.metastore.txn.TestTxnHandler.testCheckLockTxnAborted
(batchId=264)
org.apache.hadoop.hive.metastore.txn.TestTxnHandler.testLockEESW (batchId=264)
org.apache.hadoop.hive.metastore.txn.TestTxnHandler.testLockESRSW (batchId=264)
org.apache.hadoop.hive.metastore.txn.TestTxnHandler.testLockSRSW (batchId=264)
org.apache.hadoop.hive.metastore.txn.TestTxnHandler.testLockSWSR (batchId=264)
org.apache.hadoop.hive.metastore.txn.TestTxnHandler.testLockSWSWSR (batchId=264)
org.apache.hadoop.hive.metastore.txn.TestTxnHandler.testLockSWSWSW (batchId=264)
org.apache.hadoop.hive.metastore.txn.TestTxnHandler.testUnlockOnAbort
(batchId=264)
org.apache.hadoop.hive.metastore.txn.TestTxnHandler.testUnlockOnCommit
(batchId=264)
org.apache.hadoop.hive.metastore.txn.TestTxnHandler.testUnlockWithTxn
(batchId=264)
org.apache.hadoop.hive.ql.TestAcidOnTez.testCtasTezUnion (batchId=228)
org.apache.hadoop.hive.ql.TestAcidOnTez.testNonStandardConversion01
(batchId=228)
org.apache.hadoop.hive.ql.TestMTQueries.testMTQueries1 (batchId=232)
org.apache.hadoop.hive.ql.lockmgr.TestDbTxnManager.testDelete (batchId=301)
org.apache.hadoop.hive.ql.lockmgr.TestDbTxnManager.testReadWrite (batchId=301)
org.apache.hadoop.hive.ql.lockmgr.TestDbTxnManager.testRollback (batchId=301)
org.apache.hadoop.hive.ql.lockmgr.TestDbTxnManager.testSingleWritePartition
(batchId=301)
org.apache.hadoop.hive.ql.lockmgr.TestDbTxnManager.testSingleWriteTable
(batchId=301)
org.apache.hadoop.hive.ql.lockmgr.TestDbTxnManager.testUpdate (batchId=301)
org.apache.hadoop.hive.ql.lockmgr.TestDbTxnManager.testWriteDynamicPartition
(batchId=301)
org.apache.hadoop.hive.ql.parse.TestCopyUtils.testPrivilegedDistCpWithSameUserAsCurrentDoesNotTryToImpersonate
(batchId=231)
org.apache.hadoop.hive.ql.parse.TestReplicationOnHDFSEncryptedZones.targetAndSourceHaveDifferentEncryptionZoneKeys
(batchId=231)
org.apache.hadoop.hive.ql.txn.compactor.TestCleaner.blockedByLockPartition
(ba
[jira] [Commented] (HIVE-19388) ClassCastException during VectorMapJoinCommonOperator initialization
[
https://issues.apache.org/jira/browse/HIVE-19388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16463686#comment-16463686
]
Hive QA commented on HIVE-19388:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
52s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
16s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
44s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
10s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
5s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
16s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 17m 24s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-10670/dev-support/hive-personality.sh
|
| git revision | master / 39917ef |
| Default Java | 1.8.0_111 |
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-10670/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> ClassCastException during VectorMapJoinCommonOperator initialization
>
>
> Key: HIVE-19388
> URL: https://issues.apache.org/jira/browse/HIVE-19388
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.1.1, 2.2.0, 3.0.0, 2.3.2, 3.1.0
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
>Priority: Major
> Attachments: HIVE-19388.01.patch, HIVE-19388.02.patch
>
>
> I see the following exceptions when I a mapjoin operator is being initialized
> on Hive-on-Spark and when vectorization is turned on.
> This happens when the hashTable is empty. The code in
> {{MapJoinTableContainerSerDe#getDefaultEmptyContainer}} method returns a
> HashMapWrapper while the VectorMapJoinOperator expects a
> {{MapJoinBytesTableContainer}} when {{hive.mapjoin.optimized.hashtable}} is
> set to true.
> {noformat}
> Caused by: java.lang.ClassCastException:
> org.apache.hadoop.hive.ql.exec.persistence.HashMapWrapper cannot be cast to
> org.apache.hadoop.hive.ql.exec.persistence.MapJoinTableContainerDirectAccess
> at
> org.apache.hadoop.hive.ql.exec.vector.mapjoin.optimized.VectorMapJoinOptimizedHashTable.(VectorMapJoinOptimizedHashTable.java:92)
> ~[hive-exec-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.vector.mapjoin.optimized.VectorMapJoinOptimizedHashMap.(VectorMapJoinOptimizedHashMap.java:127)
> ~[hive-exec-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.vector.mapjoin.optimized.VectorMapJoinOptimizedStringHashMap.(VectorMapJoinOptimizedStringHashMap.java:60)
> ~[hive-exec-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.vector.mapjoin.optimized.VectorMapJoinOptimizedCreateHashTable.createHashTabl
[jira] [Work stopped] (HIVE-19248) REPL LOAD couldn't copy file from source CM path and also doesn't throw error if file copy fails.
[
https://issues.apache.org/jira/browse/HIVE-19248?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on HIVE-19248 stopped by Sankar Hariappan.
---
> REPL LOAD couldn't copy file from source CM path and also doesn't throw error
> if file copy fails.
> -
>
> Key: HIVE-19248
> URL: https://issues.apache.org/jira/browse/HIVE-19248
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2, repl
>Affects Versions: 3.0.0
>Reporter: Sankar Hariappan
>Assignee: Sankar Hariappan
>Priority: Major
> Labels: DR, pull-request-available, replication
> Fix For: 3.1.0
>
>
> Hive replication uses Hadoop distcp to copy files from primary to replica
> warehouse. If the HDFS block size is different across clusters, it cause file
> copy failures.
> {code:java}
> 2018-04-09 14:32:06,690 ERROR [main]
> org.apache.hadoop.tools.mapred.CopyMapper: Failure in copying
> hdfs://chelsea/apps/hive/warehouse/tpch_flat_orc_1000.db/customer/000259_0 to
> hdfs://marilyn/apps/hive/warehouse/tpch_flat_orc_1000.db/customer/.hive-staging_hive_2018-04-09_14-30-45_723_7153496419225102220-2/-ext-10001/000259_0
> java.io.IOException: File copy failed:
> hdfs://chelsea/apps/hive/warehouse/tpch_flat_orc_1000.db/customer/000259_0
> -->
> hdfs://marilyn/apps/hive/warehouse/tpch_flat_orc_1000.db/customer/.hive-staging_hive_2018-04-09_14-30-45_723_7153496419225102220-2/-ext-10001/000259_0
> at
> org.apache.hadoop.tools.mapred.CopyMapper.copyFileWithRetry(CopyMapper.java:299)
> at org.apache.hadoop.tools.mapred.CopyMapper.map(CopyMapper.java:266)
> at org.apache.hadoop.tools.mapred.CopyMapper.map(CopyMapper.java:52)
> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:146)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:787)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:170)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1869)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:164)
> Caused by: java.io.IOException: Couldn't run retriable-command: Copying
> hdfs://chelsea/apps/hive/warehouse/tpch_flat_orc_1000.db/customer/000259_0 to
> hdfs://marilyn/apps/hive/warehouse/tpch_flat_orc_1000.db/customer/.hive-staging_hive_2018-04-09_14-30-45_723_7153496419225102220-2/-ext-10001/000259_0
> at
> org.apache.hadoop.tools.util.RetriableCommand.execute(RetriableCommand.java:101)
> at
> org.apache.hadoop.tools.mapred.CopyMapper.copyFileWithRetry(CopyMapper.java:296)
> ... 10 more
> Caused by: java.io.IOException: Check-sum mismatch between
> hdfs://chelsea/apps/hive/warehouse/tpch_flat_orc_1000.db/customer/000259_0
> and
> hdfs://marilyn/apps/hive/warehouse/tpch_flat_orc_1000.db/customer/.hive-staging_hive_2018-04-09_14-30-45_723_7153496419225102220-2/-ext-10001/.distcp.tmp.attempt_1522833620762_4416_m_00_0.
> Source and target differ in block-size. Use -pb to preserve block-sizes
> during copy. Alternatively, skip checksum-checks altogether, using -skipCrc.
> (NOTE: By skipping checksums, one runs the risk of masking data-corruption
> during file-transfer.)
> at
> org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.compareCheckSums(RetriableFileCopyCommand.java:212)
> at
> org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doCopy(RetriableFileCopyCommand.java:130)
> at
> org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doExecute(RetriableFileCopyCommand.java:99)
> at
> org.apache.hadoop.tools.util.RetriableCommand.execute(RetriableCommand.java:87)
> ... 11 more
> {code}
> Distcp failed as the CM path for the file doesn't point to source file
> system. So, it is needed to get the qualified cm root URI as part of files
> listed in dump.
> Also, REPL LOAD returns success even if distcp jobs failed.
> CopyUtils.doCopyRetry doesn't throw error if copy failed even after maximum
> attempts.
> So, need to perform 2 things.
> # If copy of multiple files fail for some reason, then retry with same set
> of files again but need to set CM path if original source file is missing or
> modified based on checksum. Let distcp to skip the properly copied files.
> FileUtil.copy will always overwrite the files.
> # If source path is moved to CM path, then delete the incorrectly copied
> files.
> # If copy fails for maximum attempt, then throw error.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19248) REPL LOAD couldn't copy file from source CM path and also doesn't throw error if file copy fails.
[
https://issues.apache.org/jira/browse/HIVE-19248?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sankar Hariappan updated HIVE-19248:
Attachment: HIVE-19248.01.patch
> REPL LOAD couldn't copy file from source CM path and also doesn't throw error
> if file copy fails.
> -
>
> Key: HIVE-19248
> URL: https://issues.apache.org/jira/browse/HIVE-19248
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2, repl
>Affects Versions: 3.0.0
>Reporter: Sankar Hariappan
>Assignee: Sankar Hariappan
>Priority: Major
> Labels: DR, pull-request-available, replication
> Fix For: 3.1.0
>
> Attachments: HIVE-19248.01.patch
>
>
> Hive replication uses Hadoop distcp to copy files from primary to replica
> warehouse. If the HDFS block size is different across clusters, it cause file
> copy failures.
> {code:java}
> 2018-04-09 14:32:06,690 ERROR [main]
> org.apache.hadoop.tools.mapred.CopyMapper: Failure in copying
> hdfs://chelsea/apps/hive/warehouse/tpch_flat_orc_1000.db/customer/000259_0 to
> hdfs://marilyn/apps/hive/warehouse/tpch_flat_orc_1000.db/customer/.hive-staging_hive_2018-04-09_14-30-45_723_7153496419225102220-2/-ext-10001/000259_0
> java.io.IOException: File copy failed:
> hdfs://chelsea/apps/hive/warehouse/tpch_flat_orc_1000.db/customer/000259_0
> -->
> hdfs://marilyn/apps/hive/warehouse/tpch_flat_orc_1000.db/customer/.hive-staging_hive_2018-04-09_14-30-45_723_7153496419225102220-2/-ext-10001/000259_0
> at
> org.apache.hadoop.tools.mapred.CopyMapper.copyFileWithRetry(CopyMapper.java:299)
> at org.apache.hadoop.tools.mapred.CopyMapper.map(CopyMapper.java:266)
> at org.apache.hadoop.tools.mapred.CopyMapper.map(CopyMapper.java:52)
> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:146)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:787)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:170)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1869)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:164)
> Caused by: java.io.IOException: Couldn't run retriable-command: Copying
> hdfs://chelsea/apps/hive/warehouse/tpch_flat_orc_1000.db/customer/000259_0 to
> hdfs://marilyn/apps/hive/warehouse/tpch_flat_orc_1000.db/customer/.hive-staging_hive_2018-04-09_14-30-45_723_7153496419225102220-2/-ext-10001/000259_0
> at
> org.apache.hadoop.tools.util.RetriableCommand.execute(RetriableCommand.java:101)
> at
> org.apache.hadoop.tools.mapred.CopyMapper.copyFileWithRetry(CopyMapper.java:296)
> ... 10 more
> Caused by: java.io.IOException: Check-sum mismatch between
> hdfs://chelsea/apps/hive/warehouse/tpch_flat_orc_1000.db/customer/000259_0
> and
> hdfs://marilyn/apps/hive/warehouse/tpch_flat_orc_1000.db/customer/.hive-staging_hive_2018-04-09_14-30-45_723_7153496419225102220-2/-ext-10001/.distcp.tmp.attempt_1522833620762_4416_m_00_0.
> Source and target differ in block-size. Use -pb to preserve block-sizes
> during copy. Alternatively, skip checksum-checks altogether, using -skipCrc.
> (NOTE: By skipping checksums, one runs the risk of masking data-corruption
> during file-transfer.)
> at
> org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.compareCheckSums(RetriableFileCopyCommand.java:212)
> at
> org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doCopy(RetriableFileCopyCommand.java:130)
> at
> org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doExecute(RetriableFileCopyCommand.java:99)
> at
> org.apache.hadoop.tools.util.RetriableCommand.execute(RetriableCommand.java:87)
> ... 11 more
> {code}
> Distcp failed as the CM path for the file doesn't point to source file
> system. So, it is needed to get the qualified cm root URI as part of files
> listed in dump.
> Also, REPL LOAD returns success even if distcp jobs failed.
> CopyUtils.doCopyRetry doesn't throw error if copy failed even after maximum
> attempts.
> So, need to perform 2 things.
> # If copy of multiple files fail for some reason, then retry with same set
> of files again but need to set CM path if original source file is missing or
> modified based on checksum. Let distcp to skip the properly copied files.
> FileUtil.copy will always overwrite the files.
> # If source path is moved to CM path, then delete the incorrectly copied
> files.
> # If copy fails for maximum attempt, then throw error.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19248) REPL LOAD couldn't copy file from source CM path and also doesn't throw error if file copy fails.
[
https://issues.apache.org/jira/browse/HIVE-19248?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sankar Hariappan updated HIVE-19248:
Status: Patch Available (was: Open)
Added 01.patch with
* Retry logic for file copy when distcp or Fileutil.copy fails.
* Throw error/exception when both source and cm files are missing.
* Pass the source CM root URI as part of file URI encoded part of dumped event
and target to use it when copy from CM path.
Existing test cases would test the changes. Also, manually tested the retry
logic.
[~maheshk114], [~thejas],
Can you please review the patch?
> REPL LOAD couldn't copy file from source CM path and also doesn't throw error
> if file copy fails.
> -
>
> Key: HIVE-19248
> URL: https://issues.apache.org/jira/browse/HIVE-19248
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2, repl
>Affects Versions: 3.0.0
>Reporter: Sankar Hariappan
>Assignee: Sankar Hariappan
>Priority: Major
> Labels: DR, pull-request-available, replication
> Fix For: 3.1.0
>
> Attachments: HIVE-19248.01.patch
>
>
> Hive replication uses Hadoop distcp to copy files from primary to replica
> warehouse. If the HDFS block size is different across clusters, it cause file
> copy failures.
> {code:java}
> 2018-04-09 14:32:06,690 ERROR [main]
> org.apache.hadoop.tools.mapred.CopyMapper: Failure in copying
> hdfs://chelsea/apps/hive/warehouse/tpch_flat_orc_1000.db/customer/000259_0 to
> hdfs://marilyn/apps/hive/warehouse/tpch_flat_orc_1000.db/customer/.hive-staging_hive_2018-04-09_14-30-45_723_7153496419225102220-2/-ext-10001/000259_0
> java.io.IOException: File copy failed:
> hdfs://chelsea/apps/hive/warehouse/tpch_flat_orc_1000.db/customer/000259_0
> -->
> hdfs://marilyn/apps/hive/warehouse/tpch_flat_orc_1000.db/customer/.hive-staging_hive_2018-04-09_14-30-45_723_7153496419225102220-2/-ext-10001/000259_0
> at
> org.apache.hadoop.tools.mapred.CopyMapper.copyFileWithRetry(CopyMapper.java:299)
> at org.apache.hadoop.tools.mapred.CopyMapper.map(CopyMapper.java:266)
> at org.apache.hadoop.tools.mapred.CopyMapper.map(CopyMapper.java:52)
> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:146)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:787)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:170)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1869)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:164)
> Caused by: java.io.IOException: Couldn't run retriable-command: Copying
> hdfs://chelsea/apps/hive/warehouse/tpch_flat_orc_1000.db/customer/000259_0 to
> hdfs://marilyn/apps/hive/warehouse/tpch_flat_orc_1000.db/customer/.hive-staging_hive_2018-04-09_14-30-45_723_7153496419225102220-2/-ext-10001/000259_0
> at
> org.apache.hadoop.tools.util.RetriableCommand.execute(RetriableCommand.java:101)
> at
> org.apache.hadoop.tools.mapred.CopyMapper.copyFileWithRetry(CopyMapper.java:296)
> ... 10 more
> Caused by: java.io.IOException: Check-sum mismatch between
> hdfs://chelsea/apps/hive/warehouse/tpch_flat_orc_1000.db/customer/000259_0
> and
> hdfs://marilyn/apps/hive/warehouse/tpch_flat_orc_1000.db/customer/.hive-staging_hive_2018-04-09_14-30-45_723_7153496419225102220-2/-ext-10001/.distcp.tmp.attempt_1522833620762_4416_m_00_0.
> Source and target differ in block-size. Use -pb to preserve block-sizes
> during copy. Alternatively, skip checksum-checks altogether, using -skipCrc.
> (NOTE: By skipping checksums, one runs the risk of masking data-corruption
> during file-transfer.)
> at
> org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.compareCheckSums(RetriableFileCopyCommand.java:212)
> at
> org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doCopy(RetriableFileCopyCommand.java:130)
> at
> org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doExecute(RetriableFileCopyCommand.java:99)
> at
> org.apache.hadoop.tools.util.RetriableCommand.execute(RetriableCommand.java:87)
> ... 11 more
> {code}
> Distcp failed as the CM path for the file doesn't point to source file
> system. So, it is needed to get the qualified cm root URI as part of files
> listed in dump.
> Also, REPL LOAD returns success even if distcp jobs failed.
> CopyUtils.doCopyRetry doesn't throw error if copy failed even after maximum
> attempts.
> So, need to perform 2 things.
> # If copy of multiple files fail for some reason, then retry with same set
> of files again but need to set CM path if original source file is miss
[jira] [Updated] (HIVE-18394) Materialized view: "Create Materialized View" should default to rewritable ones
[
https://issues.apache.org/jira/browse/HIVE-18394?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Damien Carol updated HIVE-18394:
Description:
This is a usability ticket, since it is possible to end up creating
materialized views and realize that they need an additional flag to be picked
up by the optimizer to do rewrites to.
{code:sql}
create materialized view ca as select * from customer, customer_address where
c_current_addr_sk = ca_address_sk;
set hive.materializedview.rewriting=true;
select count(1) from customer, customer_address where c_current_addr_sk =
ca_address_sk; -- does not use materialized view
{code}
Needs another step
{code:sql}
alter materialized view ca enable rewrite;
{code}
And then, it kicks in
{code:sql}
select count(1) from customer, customer_address where c_current_addr_sk =
ca_address_sk;
OK
1200
Time taken: 0.494 seconds, Fetched: 1 row(s)
{code}
was:
This is a usability ticket, since it is possible to end up creating
materialized views and realize that they need an additional flag to be picked
up by the optimizer to do rewrites to.
{code}
create materialized view ca as select * from customer, customer_address where
c_current_addr_sk = ca_address_sk;
set hive.materializedview.rewriting=true;
select count(1) from customer, customer_address where c_current_addr_sk =
ca_address_sk; -- does not use materialized view
{code}
Needs another step
{code}
alter materialized view ca enable rewrite;
{code}
And then, it kicks in
{code}
select count(1) from customer, customer_address where c_current_addr_sk =
ca_address_sk;
OK
1200
Time taken: 0.494 seconds, Fetched: 1 row(s)
{code}
> Materialized view: "Create Materialized View" should default to rewritable
> ones
> ---
>
> Key: HIVE-18394
> URL: https://issues.apache.org/jira/browse/HIVE-18394
> Project: Hive
> Issue Type: Improvement
> Components: Materialized views
>Reporter: Gopal V
>Assignee: Jesus Camacho Rodriguez
>Priority: Blocker
>
> This is a usability ticket, since it is possible to end up creating
> materialized views and realize that they need an additional flag to be picked
> up by the optimizer to do rewrites to.
> {code:sql}
> create materialized view ca as select * from customer, customer_address where
> c_current_addr_sk = ca_address_sk;
> set hive.materializedview.rewriting=true;
> select count(1) from customer, customer_address where c_current_addr_sk =
> ca_address_sk; -- does not use materialized view
> {code}
> Needs another step
> {code:sql}
> alter materialized view ca enable rewrite;
> {code}
> And then, it kicks in
> {code:sql}
> select count(1) from customer, customer_address where c_current_addr_sk =
> ca_address_sk;
> OK
> 1200
> Time taken: 0.494 seconds, Fetched: 1 row(s)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18193) Migrate existing ACID tables to use write id per table rather than global transaction id
[ https://issues.apache.org/jira/browse/HIVE-18193?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sankar Hariappan updated HIVE-18193: Status: Open (was: Patch Available) > Migrate existing ACID tables to use write id per table rather than global > transaction id > > > Key: HIVE-18193 > URL: https://issues.apache.org/jira/browse/HIVE-18193 > Project: Hive > Issue Type: Sub-task > Components: HiveServer2, Transactions >Affects Versions: 3.0.0 >Reporter: anishek >Assignee: Sankar Hariappan >Priority: Blocker > Labels: ACID, Upgrade > Fix For: 3.0.0, 3.1.0 > > Attachments: HIVE-18193.01.patch > > > dependent upon HIVE-18192 > For existing ACID Tables we need to update the table level write id > metatables/sequences so any new operations on these tables works seamlessly > without any conflicting data in existing base/delta files. > 1. Need to create metadata tables such as NEXT_WRITE_ID and TXN_TO_WRITE_ID. > 2. Add entries for each ACID/MM tables into NEXT_WRITE_ID where NWI_NEXT is > set to current value of NEXT_TXN_ID.NTXN_NEXT. > 3. All current open/abort transactions to have an entry in TXN_TO_WRITE_ID > such that T2W_TXNID=T2W_WRITEID=Open/AbortedTxnId. > 4. Added new column TC_WRITEID in TXN_COMPONENTS and CTC_WRITEID in > COMPLETED_TXN_COMPONENTS to store the write id which should be set as > respective values of TC_TXNID and CTC_TXNID from the same row. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18193) Migrate existing ACID tables to use write id per table rather than global transaction id
[ https://issues.apache.org/jira/browse/HIVE-18193?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sankar Hariappan updated HIVE-18193: Attachment: HIVE-18193.02.patch > Migrate existing ACID tables to use write id per table rather than global > transaction id > > > Key: HIVE-18193 > URL: https://issues.apache.org/jira/browse/HIVE-18193 > Project: Hive > Issue Type: Sub-task > Components: HiveServer2, Transactions >Affects Versions: 3.0.0 >Reporter: anishek >Assignee: Sankar Hariappan >Priority: Blocker > Labels: ACID, Upgrade > Fix For: 3.0.0, 3.1.0 > > Attachments: HIVE-18193.01.patch, HIVE-18193.02.patch > > > dependent upon HIVE-18192 > For existing ACID Tables we need to update the table level write id > metatables/sequences so any new operations on these tables works seamlessly > without any conflicting data in existing base/delta files. > 1. Need to create metadata tables such as NEXT_WRITE_ID and TXN_TO_WRITE_ID. > 2. Add entries for each ACID/MM tables into NEXT_WRITE_ID where NWI_NEXT is > set to current value of NEXT_TXN_ID.NTXN_NEXT. > 3. All current open/abort transactions to have an entry in TXN_TO_WRITE_ID > such that T2W_TXNID=T2W_WRITEID=Open/AbortedTxnId. > 4. Added new column TC_WRITEID in TXN_COMPONENTS and CTC_WRITEID in > COMPLETED_TXN_COMPONENTS to store the write id which should be set as > respective values of TC_TXNID and CTC_TXNID from the same row. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18193) Migrate existing ACID tables to use write id per table rather than global transaction id
[ https://issues.apache.org/jira/browse/HIVE-18193?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sankar Hariappan updated HIVE-18193: Status: Patch Available (was: Open) Attached 02.patch with NOT NULL constraint removed on TXN_COMPONENTS.TC_WRITEID and COMPLETED_TXN_COMPONENTS.CTC_WRITEID columns as they can be NULL if it is DB lock. However Txn ID in both these tables are marked as NOT NULL. > Migrate existing ACID tables to use write id per table rather than global > transaction id > > > Key: HIVE-18193 > URL: https://issues.apache.org/jira/browse/HIVE-18193 > Project: Hive > Issue Type: Sub-task > Components: HiveServer2, Transactions >Affects Versions: 3.0.0 >Reporter: anishek >Assignee: Sankar Hariappan >Priority: Blocker > Labels: ACID, Upgrade > Fix For: 3.0.0, 3.1.0 > > Attachments: HIVE-18193.01.patch, HIVE-18193.02.patch > > > dependent upon HIVE-18192 > For existing ACID Tables we need to update the table level write id > metatables/sequences so any new operations on these tables works seamlessly > without any conflicting data in existing base/delta files. > 1. Need to create metadata tables such as NEXT_WRITE_ID and TXN_TO_WRITE_ID. > 2. Add entries for each ACID/MM tables into NEXT_WRITE_ID where NWI_NEXT is > set to current value of NEXT_TXN_ID.NTXN_NEXT. > 3. All current open/abort transactions to have an entry in TXN_TO_WRITE_ID > such that T2W_TXNID=T2W_WRITEID=Open/AbortedTxnId. > 4. Added new column TC_WRITEID in TXN_COMPONENTS and CTC_WRITEID in > COMPLETED_TXN_COMPONENTS to store the write id which should be set as > respective values of TC_TXNID and CTC_TXNID from the same row. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19388) ClassCastException during VectorMapJoinCommonOperator initialization
[
https://issues.apache.org/jira/browse/HIVE-19388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16463774#comment-16463774
]
Hive QA commented on HIVE-19388:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12921778/HIVE-19388.02.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 35 failed/errored test(s), 14316 tests
executed
*Failed tests:*
{noformat}
TestDbNotificationListener - did not produce a TEST-*.xml file (likely timed
out) (batchId=247)
TestHCatHiveCompatibility - did not produce a TEST-*.xml file (likely timed
out) (batchId=247)
TestNonCatCallsWithCatalog - did not produce a TEST-*.xml file (likely timed
out) (batchId=217)
TestSequenceFileReadWrite - did not produce a TEST-*.xml file (likely timed
out) (batchId=247)
TestTxnExIm - did not produce a TEST-*.xml file (likely timed out) (batchId=286)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[authorization_view_disable_cbo_4]
(batchId=13)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucket_map_join_tez1]
(batchId=175)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucket_map_join_tez2]
(batchId=156)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[materialized_view_create_rewrite_5]
(batchId=154)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[materialized_view_create_rewrite_rebuild_dummy]
(batchId=161)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[materialized_view_create_rewrite_time_window]
(batchId=156)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb]
(batchId=163)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[tez_dynpart_hashjoin_1]
(batchId=174)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[union_fast_stats]
(batchId=167)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_5]
(batchId=105)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[udf_reflect_neg]
(batchId=96)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[udf_test_error]
(batchId=96)
org.apache.hadoop.hive.metastore.TestStats.partitionedTableOtherCatalog
(batchId=211)
org.apache.hadoop.hive.ql.TestAcidOnTez.testCtasTezUnion (batchId=228)
org.apache.hadoop.hive.ql.TestAcidOnTez.testNonStandardConversion01
(batchId=228)
org.apache.hadoop.hive.ql.TestMTQueries.testMTQueries1 (batchId=232)
org.apache.hadoop.hive.ql.parse.TestCopyUtils.testPrivilegedDistCpWithSameUserAsCurrentDoesNotTryToImpersonate
(batchId=231)
org.apache.hadoop.hive.ql.parse.TestReplicationOnHDFSEncryptedZones.targetAndSourceHaveDifferentEncryptionZoneKeys
(batchId=231)
org.apache.hive.beeline.TestBeeLineWithArgs.testQueryProgressParallel
(batchId=235)
org.apache.hive.jdbc.TestSSL.testSSLFetchHttp (batchId=239)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testMultipleTriggers2
(batchId=241)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerCustomCreatedFiles
(batchId=241)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerCustomNonExistent
(batchId=241)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerCustomReadOps
(batchId=241)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerHighBytesRead
(batchId=241)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerHighBytesWrite
(batchId=241)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerHighShuffleBytes
(batchId=241)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerSlowQueryElapsedTime
(batchId=241)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerSlowQueryExecutionTime
(batchId=241)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerVertexRawInputSplitsNoKill
(batchId=241)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/10670/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/10670/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-10670/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 35 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12921778 - PreCommit-HIVE-Build
> ClassCastException during VectorMapJoinCommonOperator initialization
>
>
> Key: HIVE-19388
> URL: https://issues.apache.org/jira/browse/HIVE-19388
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.1.1, 2.2.0, 3.0.0, 2.3.2, 3.1
[jira] [Updated] (HIVE-19159) TestMTQueries.testMTQueries1 failure
[ https://issues.apache.org/jira/browse/HIVE-19159?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Bodor updated HIVE-19159: Attachment: HIVE-19159.02.patch > TestMTQueries.testMTQueries1 failure > > > Key: HIVE-19159 > URL: https://issues.apache.org/jira/browse/HIVE-19159 > Project: Hive > Issue Type: Sub-task > Components: Test >Reporter: Vineet Garg >Assignee: Laszlo Bodor >Priority: Critical > Attachments: HIVE-19159.01.patch, HIVE-19159.02.patch > > > I have confirmed that HIVE-18051 caused this failure -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19159) TestMTQueries.testMTQueries1 failure
[ https://issues.apache.org/jira/browse/HIVE-19159?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Bodor updated HIVE-19159: Attachment: HIVE-19159.02-branch-3.patch > TestMTQueries.testMTQueries1 failure > > > Key: HIVE-19159 > URL: https://issues.apache.org/jira/browse/HIVE-19159 > Project: Hive > Issue Type: Sub-task > Components: Test >Reporter: Vineet Garg >Assignee: Laszlo Bodor >Priority: Critical > Attachments: HIVE-19159.01.patch, HIVE-19159.02-branch-3.patch, > HIVE-19159.02.patch > > > I have confirmed that HIVE-18051 caused this failure -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-12342) Set default value of hive.optimize.index.filter to true
[
https://issues.apache.org/jira/browse/HIVE-12342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16463796#comment-16463796
]
Hive QA commented on HIVE-12342:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
1s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
50s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
48s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
36s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
3s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
18s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
10s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
1s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
18s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
16s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 20m 11s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-10671/dev-support/hive-personality.sh
|
| git revision | master / 39917ef |
| Default Java | 1.8.0_111 |
| modules | C: common ql U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-10671/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Set default value of hive.optimize.index.filter to true
> ---
>
> Key: HIVE-12342
> URL: https://issues.apache.org/jira/browse/HIVE-12342
> Project: Hive
> Issue Type: Task
> Components: Configuration
>Reporter: Ashutosh Chauhan
>Assignee: Igor Kryvenko
>Priority: Major
> Attachments: HIVE-12342.05.patch, HIVE-12342.1.patch,
> HIVE-12342.2.patch, HIVE-12342.3.patch, HIVE-12342.4.patch, HIVE-12342.patch
>
>
> This configuration governs ppd for storage layer. When applicable, it will
> always help. It should be on by default.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18862) qfiles: prepare .q files for using datasets
[ https://issues.apache.org/jira/browse/HIVE-18862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16463809#comment-16463809 ] Laszlo Bodor commented on HIVE-18862: - I'm working on the branch-3 patch. > qfiles: prepare .q files for using datasets > --- > > Key: HIVE-18862 > URL: https://issues.apache.org/jira/browse/HIVE-18862 > Project: Hive > Issue Type: Sub-task >Reporter: Laszlo Bodor >Assignee: Laszlo Bodor >Priority: Major > Fix For: 3.1.0 > > Attachments: HIVE-18862.01.patch, HIVE-18862.02.patch, > HIVE-18862.03.patch, HIVE-18862.04.patch, HIVE-18862.05.patch, > HIVE-18862.06.patch, HIVE-18862.07.patch, HIVE-18862.08.patch, > HIVE-18862.09-branch-3.patch, HIVE-18862.09.patch > > > # Parse .q files for source table usage > # Add needed dataset annotations > # Remove create table statements from "q_test_init.sql" like files > # Handle oncoming issues related to dataset introduction -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-12342) Set default value of hive.optimize.index.filter to true
[
https://issues.apache.org/jira/browse/HIVE-12342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16463854#comment-16463854
]
Hive QA commented on HIVE-12342:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12921788/HIVE-12342.05.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1507 failed/errored test(s), 14306 tests
executed
*Failed tests:*
{noformat}
TestDbNotificationListener - did not produce a TEST-*.xml file (likely timed
out) (batchId=247)
TestHCatHiveCompatibility - did not produce a TEST-*.xml file (likely timed
out) (batchId=247)
TestMinimrCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=94)
[infer_bucket_sort_num_buckets.q,infer_bucket_sort_reducers_power_two.q,parallel_orderby.q,bucket_num_reducers_acid.q,scriptfile1.q,infer_bucket_sort_map_operators.q,infer_bucket_sort_merge.q,root_dir_external_table.q,infer_bucket_sort_dyn_part.q,udf_using.q]
TestNonCatCallsWithCatalog - did not produce a TEST-*.xml file (likely timed
out) (batchId=217)
TestSequenceFileReadWrite - did not produce a TEST-*.xml file (likely timed
out) (batchId=247)
TestTxnExIm - did not produce a TEST-*.xml file (likely timed out) (batchId=286)
org.apache.hadoop.hive.cli.TestAccumuloCliDriver.testCliDriver[accumulo_predicate_pushdown]
(batchId=245)
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[explain_outputs]
(batchId=253)
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[materialized_view_create_rewrite]
(batchId=253)
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[smb_mapjoin_10]
(batchId=253)
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[smb_mapjoin_11]
(batchId=253)
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[smb_mapjoin_12]
(batchId=253)
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[smb_mapjoin_13]
(batchId=253)
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[smb_mapjoin_16]
(batchId=253)
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[smb_mapjoin_1]
(batchId=253)
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[smb_mapjoin_2]
(batchId=253)
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[smb_mapjoin_3]
(batchId=253)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_mapjoin]
(batchId=11)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_table_stats]
(batchId=55)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[allcolref_in_udf]
(batchId=54)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[alterColumnStatsPart]
(batchId=88)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[alter_partition_coltype]
(batchId=27)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ambiguous_col]
(batchId=50)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[analyze_tbl_date]
(batchId=36)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[annotate_stats_deep_filters]
(batchId=92)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[annotate_stats_filter]
(batchId=9)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[annotate_stats_join]
(batchId=55)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[annotate_stats_join_pkfk]
(batchId=15)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[annotate_stats_part]
(batchId=16)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[annotate_stats_select]
(batchId=64)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[autoColumnStats_1]
(batchId=22)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[autoColumnStats_2]
(batchId=86)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[autoColumnStats_4]
(batchId=13)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[autoColumnStats_9]
(batchId=37)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join0] (batchId=90)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join10] (batchId=36)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join11] (batchId=9)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join12] (batchId=25)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join13] (batchId=83)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join14] (batchId=15)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join15] (batchId=16)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join16] (batchId=41)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join17] (batchId=84)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join19] (batchId=67)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join19_inclause]
(batchId=18)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join1] (batchId=80)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join20] (batchId=91)
org.apa
[jira] [Updated] (HIVE-18193) Migrate existing ACID tables to use write id per table rather than global transaction id
[ https://issues.apache.org/jira/browse/HIVE-18193?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sankar Hariappan updated HIVE-18193: Attachment: (was: HIVE-18193.02.patch) > Migrate existing ACID tables to use write id per table rather than global > transaction id > > > Key: HIVE-18193 > URL: https://issues.apache.org/jira/browse/HIVE-18193 > Project: Hive > Issue Type: Sub-task > Components: HiveServer2, Transactions >Affects Versions: 3.0.0 >Reporter: anishek >Assignee: Sankar Hariappan >Priority: Blocker > Labels: ACID, Upgrade > Fix For: 3.0.0, 3.1.0 > > Attachments: HIVE-18193.01.patch > > > dependent upon HIVE-18192 > For existing ACID Tables we need to update the table level write id > metatables/sequences so any new operations on these tables works seamlessly > without any conflicting data in existing base/delta files. > 1. Need to create metadata tables such as NEXT_WRITE_ID and TXN_TO_WRITE_ID. > 2. Add entries for each ACID/MM tables into NEXT_WRITE_ID where NWI_NEXT is > set to current value of NEXT_TXN_ID.NTXN_NEXT. > 3. All current open/abort transactions to have an entry in TXN_TO_WRITE_ID > such that T2W_TXNID=T2W_WRITEID=Open/AbortedTxnId. > 4. Added new column TC_WRITEID in TXN_COMPONENTS and CTC_WRITEID in > COMPLETED_TXN_COMPONENTS to store the write id which should be set as > respective values of TC_TXNID and CTC_TXNID from the same row. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18193) Migrate existing ACID tables to use write id per table rather than global transaction id
[ https://issues.apache.org/jira/browse/HIVE-18193?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sankar Hariappan updated HIVE-18193: Status: Open (was: Patch Available) > Migrate existing ACID tables to use write id per table rather than global > transaction id > > > Key: HIVE-18193 > URL: https://issues.apache.org/jira/browse/HIVE-18193 > Project: Hive > Issue Type: Sub-task > Components: HiveServer2, Transactions >Affects Versions: 3.0.0 >Reporter: anishek >Assignee: Sankar Hariappan >Priority: Blocker > Labels: ACID, Upgrade > Fix For: 3.0.0, 3.1.0 > > Attachments: HIVE-18193.01.patch > > > dependent upon HIVE-18192 > For existing ACID Tables we need to update the table level write id > metatables/sequences so any new operations on these tables works seamlessly > without any conflicting data in existing base/delta files. > 1. Need to create metadata tables such as NEXT_WRITE_ID and TXN_TO_WRITE_ID. > 2. Add entries for each ACID/MM tables into NEXT_WRITE_ID where NWI_NEXT is > set to current value of NEXT_TXN_ID.NTXN_NEXT. > 3. All current open/abort transactions to have an entry in TXN_TO_WRITE_ID > such that T2W_TXNID=T2W_WRITEID=Open/AbortedTxnId. > 4. Added new column TC_WRITEID in TXN_COMPONENTS and CTC_WRITEID in > COMPLETED_TXN_COMPONENTS to store the write id which should be set as > respective values of TC_TXNID and CTC_TXNID from the same row. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18193) Migrate existing ACID tables to use write id per table rather than global transaction id
[ https://issues.apache.org/jira/browse/HIVE-18193?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sankar Hariappan updated HIVE-18193: Attachment: HIVE-18193.02.patch > Migrate existing ACID tables to use write id per table rather than global > transaction id > > > Key: HIVE-18193 > URL: https://issues.apache.org/jira/browse/HIVE-18193 > Project: Hive > Issue Type: Sub-task > Components: HiveServer2, Transactions >Affects Versions: 3.0.0 >Reporter: anishek >Assignee: Sankar Hariappan >Priority: Blocker > Labels: ACID, Upgrade > Fix For: 3.0.0, 3.1.0 > > Attachments: HIVE-18193.01.patch, HIVE-18193.02.patch > > > dependent upon HIVE-18192 > For existing ACID Tables we need to update the table level write id > metatables/sequences so any new operations on these tables works seamlessly > without any conflicting data in existing base/delta files. > 1. Need to create metadata tables such as NEXT_WRITE_ID and TXN_TO_WRITE_ID. > 2. Add entries for each ACID/MM tables into NEXT_WRITE_ID where NWI_NEXT is > set to current value of NEXT_TXN_ID.NTXN_NEXT. > 3. All current open/abort transactions to have an entry in TXN_TO_WRITE_ID > such that T2W_TXNID=T2W_WRITEID=Open/AbortedTxnId. > 4. Added new column TC_WRITEID in TXN_COMPONENTS and CTC_WRITEID in > COMPLETED_TXN_COMPONENTS to store the write id which should be set as > respective values of TC_TXNID and CTC_TXNID from the same row. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18193) Migrate existing ACID tables to use write id per table rather than global transaction id
[ https://issues.apache.org/jira/browse/HIVE-18193?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sankar Hariappan updated HIVE-18193: Attachment: (was: HIVE-18193.02.patch) > Migrate existing ACID tables to use write id per table rather than global > transaction id > > > Key: HIVE-18193 > URL: https://issues.apache.org/jira/browse/HIVE-18193 > Project: Hive > Issue Type: Sub-task > Components: HiveServer2, Transactions >Affects Versions: 3.0.0 >Reporter: anishek >Assignee: Sankar Hariappan >Priority: Blocker > Labels: ACID, Upgrade > Fix For: 3.0.0, 3.1.0 > > Attachments: HIVE-18193.01.patch > > > dependent upon HIVE-18192 > For existing ACID Tables we need to update the table level write id > metatables/sequences so any new operations on these tables works seamlessly > without any conflicting data in existing base/delta files. > 1. Need to create metadata tables such as NEXT_WRITE_ID and TXN_TO_WRITE_ID. > 2. Add entries for each ACID/MM tables into NEXT_WRITE_ID where NWI_NEXT is > set to current value of NEXT_TXN_ID.NTXN_NEXT. > 3. All current open/abort transactions to have an entry in TXN_TO_WRITE_ID > such that T2W_TXNID=T2W_WRITEID=Open/AbortedTxnId. > 4. Added new column TC_WRITEID in TXN_COMPONENTS and CTC_WRITEID in > COMPLETED_TXN_COMPONENTS to store the write id which should be set as > respective values of TC_TXNID and CTC_TXNID from the same row. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19161) Add authorizations to information schema
[
https://issues.apache.org/jira/browse/HIVE-19161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16463861#comment-16463861
]
Hive QA commented on HIVE-19161:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12921494/HIVE-19161.12.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/10673/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/10673/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-10673/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2018-05-04 13:22:02.365
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-10673/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2018-05-04 13:22:02.369
+ cd apache-github-source-source
+ git fetch origin
+ git reset --hard HEAD
HEAD is now at 39917ef Addendum to HIVE-19396
+ git clean -f -d
+ git checkout master
Already on 'master'
Your branch is up-to-date with 'origin/master'.
+ git reset --hard origin/master
HEAD is now at 39917ef Addendum to HIVE-19396
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2018-05-04 13:22:11.828
+ rm -rf ../yetus_PreCommit-HIVE-Build-10673
+ mkdir ../yetus_PreCommit-HIVE-Build-10673
+ git gc
+ cp -R . ../yetus_PreCommit-HIVE-Build-10673
+ mkdir /data/hiveptest/logs/PreCommit-HIVE-Build-10673/yetus
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh
/data/hiveptest/working/scratch/build.patch
error:
a/accumulo-handler/src/java/org/apache/hadoop/hive/accumulo/HiveAccumuloHelper.java:
does not exist in index
error: a/accumulo-handler/src/java/org/apache/hadoop/hive/accumulo/Utils.java:
does not exist in index
error:
a/accumulo-handler/src/java/org/apache/hadoop/hive/accumulo/serde/CompositeAccumuloRowIdFactory.java:
does not exist in index
error:
a/accumulo-handler/src/java/org/apache/hadoop/hive/accumulo/serde/DefaultAccumuloRowIdFactory.java:
does not exist in index
error: a/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java: does not
exist in index
error:
a/itests/hcatalog-unit/src/test/java/org/apache/hive/hcatalog/listener/DummyRawStoreFailEvent.java:
does not exist in index
error: a/itests/hive-unit/pom.xml: does not exist in index
error:
a/itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestRestrictedList.java:
does not exist in index
error:
a/jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/JdbcStorageHandler.java:
does not exist in index
error:
a/jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/dao/DatabaseAccessorFactory.java:
does not exist in index
error:
a/jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/dao/GenericJdbcDatabaseAccessor.java:
does not exist in index
error:
a/jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/dao/JdbcRecordIterator.java:
does not exist in index
error: a/metastore/scripts/upgrade/hive/hive-schema-3.0.0.hive.sql: does not
exist in index
error: a/ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java: does
not exist in index
error:
a/ql/src/java/org/apache/hadoop/hive/ql/security/authorization/HiveAuthorizationProvider.java:
does not exist in index
error: a/ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java: does
not exist in index
error: a/ql/src/test/results/clientpositive/llap/resourceplan.q.out: does not
exist in index
error: a/ql/src/test/results/clientpositive/show_functions.q.out: does not
exist in index
error: a/service/src/java/org/apache/hive/service/server/HiveServer2.java: does
not exist in index
error: a/standalone-metastore/src/gen/thrift/gen-cpp/ThriftH
[jira] [Updated] (HIVE-18193) Migrate existing ACID tables to use write id per table rather than global transaction id
[ https://issues.apache.org/jira/browse/HIVE-18193?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sankar Hariappan updated HIVE-18193: Attachment: HIVE-18193.02.patch > Migrate existing ACID tables to use write id per table rather than global > transaction id > > > Key: HIVE-18193 > URL: https://issues.apache.org/jira/browse/HIVE-18193 > Project: Hive > Issue Type: Sub-task > Components: HiveServer2, Transactions >Affects Versions: 3.0.0 >Reporter: anishek >Assignee: Sankar Hariappan >Priority: Blocker > Labels: ACID, Upgrade > Fix For: 3.0.0, 3.1.0 > > Attachments: HIVE-18193.01.patch, HIVE-18193.02.patch > > > dependent upon HIVE-18192 > For existing ACID Tables we need to update the table level write id > metatables/sequences so any new operations on these tables works seamlessly > without any conflicting data in existing base/delta files. > 1. Need to create metadata tables such as NEXT_WRITE_ID and TXN_TO_WRITE_ID. > 2. Add entries for each ACID/MM tables into NEXT_WRITE_ID where NWI_NEXT is > set to current value of NEXT_TXN_ID.NTXN_NEXT. > 3. All current open/abort transactions to have an entry in TXN_TO_WRITE_ID > such that T2W_TXNID=T2W_WRITEID=Open/AbortedTxnId. > 4. Added new column TC_WRITEID in TXN_COMPONENTS and CTC_WRITEID in > COMPLETED_TXN_COMPONENTS to store the write id which should be set as > respective values of TC_TXNID and CTC_TXNID from the same row. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18193) Migrate existing ACID tables to use write id per table rather than global transaction id
[ https://issues.apache.org/jira/browse/HIVE-18193?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sankar Hariappan updated HIVE-18193: Status: Patch Available (was: Open) > Migrate existing ACID tables to use write id per table rather than global > transaction id > > > Key: HIVE-18193 > URL: https://issues.apache.org/jira/browse/HIVE-18193 > Project: Hive > Issue Type: Sub-task > Components: HiveServer2, Transactions >Affects Versions: 3.0.0 >Reporter: anishek >Assignee: Sankar Hariappan >Priority: Blocker > Labels: ACID, Upgrade > Fix For: 3.0.0, 3.1.0 > > Attachments: HIVE-18193.01.patch, HIVE-18193.02.patch > > > dependent upon HIVE-18192 > For existing ACID Tables we need to update the table level write id > metatables/sequences so any new operations on these tables works seamlessly > without any conflicting data in existing base/delta files. > 1. Need to create metadata tables such as NEXT_WRITE_ID and TXN_TO_WRITE_ID. > 2. Add entries for each ACID/MM tables into NEXT_WRITE_ID where NWI_NEXT is > set to current value of NEXT_TXN_ID.NTXN_NEXT. > 3. All current open/abort transactions to have an entry in TXN_TO_WRITE_ID > such that T2W_TXNID=T2W_WRITEID=Open/AbortedTxnId. > 4. Added new column TC_WRITEID in TXN_COMPONENTS and CTC_WRITEID in > COMPLETED_TXN_COMPONENTS to store the write id which should be set as > respective values of TC_TXNID and CTC_TXNID from the same row. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19326) union_fast_stats MiniLlapLocal golden file has incorrect "accurate" stats (incorrect query results possible)
[
https://issues.apache.org/jira/browse/HIVE-19326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16463877#comment-16463877
]
Zoltan Haindrich commented on HIVE-19326:
-
actually this seems to be 2 issues:
* there was an issue which arised from the fact that all the rs-es have
overwritten eachothers output - I've fixed this..and now ctas for unions work
as expected
* if {{hive.merge.tezfiles}} is enabled ; somehow the counters also get merged
or something...I'm not sure if this is enabled in production environments or
not...I'll take another look
I think {{hive.optimize.metadataonly}} should be enabled for all tests...or at
least check that all tests would pass with it...
> union_fast_stats MiniLlapLocal golden file has incorrect "accurate" stats
> (incorrect query results possible)
>
>
> Key: HIVE-19326
> URL: https://issues.apache.org/jira/browse/HIVE-19326
> Project: Hive
> Issue Type: Bug
> Components: Statistics
>Reporter: Sergey Shelukhin
>Assignee: Zoltan Haindrich
>Priority: Blocker
> Fix For: 3.0.0
>
>
> Found when investigating the results change after converting tables to MM,
> turns out the MM result is correct but the current one is not.
> The test ends like so:
> {noformat}
> desc formatted small_alltypesorc_a;
> ANALYZE TABLE small_alltypesorc_a COMPUTE STATISTICS;
> desc formatted small_alltypesorc_a;
> insert into table small_alltypesorc_a select * from small_alltypesorc1a;
> desc formatted small_alltypesorc_a;
> {noformat}
> The results from the descs in the golden file are:
> {noformat}
> COLUMN_STATS_ACCURATE {\"BASIC_STATS\":\"true\"}
> numFiles1
> numRows 5
> ...
> COLUMN_STATS_ACCURATE {\"BASIC_STATS\":\"true\"}
> numFiles1
> numRows 15
> ...
> COLUMN_STATS_ACCURATE {\"BASIC_STATS\":\"true\"}
> numFiles2
> numRows 20
> {noformat}
> Note the result change after analyze - the original nomRows is inaccurate,
> but BASIC_STATS is set to true.
> I am assuming with metadata only optimization this can produce incorrect
> results.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19248) REPL LOAD couldn't copy file from source CM path and also doesn't throw error if file copy fails.
[
https://issues.apache.org/jira/browse/HIVE-19248?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sankar Hariappan updated HIVE-19248:
Fix Version/s: 3.0.0
> REPL LOAD couldn't copy file from source CM path and also doesn't throw error
> if file copy fails.
> -
>
> Key: HIVE-19248
> URL: https://issues.apache.org/jira/browse/HIVE-19248
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2, repl
>Affects Versions: 3.0.0
>Reporter: Sankar Hariappan
>Assignee: Sankar Hariappan
>Priority: Major
> Labels: DR, pull-request-available, replication
> Fix For: 3.0.0, 3.1.0
>
> Attachments: HIVE-19248.01.patch
>
>
> Hive replication uses Hadoop distcp to copy files from primary to replica
> warehouse. If the HDFS block size is different across clusters, it cause file
> copy failures.
> {code:java}
> 2018-04-09 14:32:06,690 ERROR [main]
> org.apache.hadoop.tools.mapred.CopyMapper: Failure in copying
> hdfs://chelsea/apps/hive/warehouse/tpch_flat_orc_1000.db/customer/000259_0 to
> hdfs://marilyn/apps/hive/warehouse/tpch_flat_orc_1000.db/customer/.hive-staging_hive_2018-04-09_14-30-45_723_7153496419225102220-2/-ext-10001/000259_0
> java.io.IOException: File copy failed:
> hdfs://chelsea/apps/hive/warehouse/tpch_flat_orc_1000.db/customer/000259_0
> -->
> hdfs://marilyn/apps/hive/warehouse/tpch_flat_orc_1000.db/customer/.hive-staging_hive_2018-04-09_14-30-45_723_7153496419225102220-2/-ext-10001/000259_0
> at
> org.apache.hadoop.tools.mapred.CopyMapper.copyFileWithRetry(CopyMapper.java:299)
> at org.apache.hadoop.tools.mapred.CopyMapper.map(CopyMapper.java:266)
> at org.apache.hadoop.tools.mapred.CopyMapper.map(CopyMapper.java:52)
> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:146)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:787)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:170)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1869)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:164)
> Caused by: java.io.IOException: Couldn't run retriable-command: Copying
> hdfs://chelsea/apps/hive/warehouse/tpch_flat_orc_1000.db/customer/000259_0 to
> hdfs://marilyn/apps/hive/warehouse/tpch_flat_orc_1000.db/customer/.hive-staging_hive_2018-04-09_14-30-45_723_7153496419225102220-2/-ext-10001/000259_0
> at
> org.apache.hadoop.tools.util.RetriableCommand.execute(RetriableCommand.java:101)
> at
> org.apache.hadoop.tools.mapred.CopyMapper.copyFileWithRetry(CopyMapper.java:296)
> ... 10 more
> Caused by: java.io.IOException: Check-sum mismatch between
> hdfs://chelsea/apps/hive/warehouse/tpch_flat_orc_1000.db/customer/000259_0
> and
> hdfs://marilyn/apps/hive/warehouse/tpch_flat_orc_1000.db/customer/.hive-staging_hive_2018-04-09_14-30-45_723_7153496419225102220-2/-ext-10001/.distcp.tmp.attempt_1522833620762_4416_m_00_0.
> Source and target differ in block-size. Use -pb to preserve block-sizes
> during copy. Alternatively, skip checksum-checks altogether, using -skipCrc.
> (NOTE: By skipping checksums, one runs the risk of masking data-corruption
> during file-transfer.)
> at
> org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.compareCheckSums(RetriableFileCopyCommand.java:212)
> at
> org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doCopy(RetriableFileCopyCommand.java:130)
> at
> org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doExecute(RetriableFileCopyCommand.java:99)
> at
> org.apache.hadoop.tools.util.RetriableCommand.execute(RetriableCommand.java:87)
> ... 11 more
> {code}
> Distcp failed as the CM path for the file doesn't point to source file
> system. So, it is needed to get the qualified cm root URI as part of files
> listed in dump.
> Also, REPL LOAD returns success even if distcp jobs failed.
> CopyUtils.doCopyRetry doesn't throw error if copy failed even after maximum
> attempts.
> So, need to perform 2 things.
> # If copy of multiple files fail for some reason, then retry with same set
> of files again but need to set CM path if original source file is missing or
> modified based on checksum. Let distcp to skip the properly copied files.
> FileUtil.copy will always overwrite the files.
> # If source path is moved to CM path, then delete the incorrectly copied
> files.
> # If copy fails for maximum attempt, then throw error.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19344) Change default value of msck.repair.batch.size
[
https://issues.apache.org/jira/browse/HIVE-19344?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16463882#comment-16463882
]
Hive QA commented on HIVE-19344:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
9s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
20s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
18s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
16s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
14s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 11m 44s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-10674/dev-support/hive-personality.sh
|
| git revision | master / 39917ef |
| Default Java | 1.8.0_111 |
| modules | C: common U: common |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-10674/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Change default value of msck.repair.batch.size
> ---
>
> Key: HIVE-19344
> URL: https://issues.apache.org/jira/browse/HIVE-19344
> Project: Hive
> Issue Type: Improvement
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
>Priority: Minor
> Attachments: HIVE-19344.01.patch
>
>
> {{msck.repair.batch.size}} default to 0 which means msck will try to add all
> the partitions in one API call to HMS. This can potentially add huge memory
> pressure on HMS. The default value should be changed to a reasonable number
> so that in case of large number of partitions we can batch the addition of
> partitions. Same goes for {{msck.repair.batch.max.retries}}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18862) qfiles: prepare .q files for using datasets
[ https://issues.apache.org/jira/browse/HIVE-18862?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Bodor updated HIVE-18862: Attachment: HIVE-18862.10-branch-3.patch > qfiles: prepare .q files for using datasets > --- > > Key: HIVE-18862 > URL: https://issues.apache.org/jira/browse/HIVE-18862 > Project: Hive > Issue Type: Sub-task >Reporter: Laszlo Bodor >Assignee: Laszlo Bodor >Priority: Major > Fix For: 3.1.0 > > Attachments: HIVE-18862.01.patch, HIVE-18862.02.patch, > HIVE-18862.03.patch, HIVE-18862.04.patch, HIVE-18862.05.patch, > HIVE-18862.06.patch, HIVE-18862.07.patch, HIVE-18862.08.patch, > HIVE-18862.09-branch-3.patch, HIVE-18862.09.patch, > HIVE-18862.10-branch-3.patch > > > # Parse .q files for source table usage > # Add needed dataset annotations > # Remove create table statements from "q_test_init.sql" like files > # Handle oncoming issues related to dataset introduction -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (HIVE-18862) qfiles: prepare .q files for using datasets
[ https://issues.apache.org/jira/browse/HIVE-18862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16463809#comment-16463809 ] Laszlo Bodor edited comment on HIVE-18862 at 5/4/18 2:03 PM: - Uploaded HIVE-18862.10-branch-3.patch which handles the conflicts above. was (Author: abstractdog): I'm working on the branch-3 patch. > qfiles: prepare .q files for using datasets > --- > > Key: HIVE-18862 > URL: https://issues.apache.org/jira/browse/HIVE-18862 > Project: Hive > Issue Type: Sub-task >Reporter: Laszlo Bodor >Assignee: Laszlo Bodor >Priority: Major > Fix For: 3.1.0 > > Attachments: HIVE-18862.01.patch, HIVE-18862.02.patch, > HIVE-18862.03.patch, HIVE-18862.04.patch, HIVE-18862.05.patch, > HIVE-18862.06.patch, HIVE-18862.07.patch, HIVE-18862.08.patch, > HIVE-18862.09-branch-3.patch, HIVE-18862.09.patch, > HIVE-18862.10-branch-3.patch > > > # Parse .q files for source table usage > # Add needed dataset annotations > # Remove create table statements from "q_test_init.sql" like files > # Handle oncoming issues related to dataset introduction -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19415) Support CORS for all HS2 web endpoints
[
https://issues.apache.org/jira/browse/HIVE-19415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16464021#comment-16464021
]
Hive QA commented on HIVE-19415:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
49s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
16s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
27s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
49s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
58s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
9s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
25s{color} | {color:red} service in the patch failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m
23s{color} | {color:red} service in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 23s{color}
| {color:red} service in the patch failed. {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
18s{color} | {color:red} itests/hive-unit: The patch generated 2 new + 19
unchanged - 1 fixed = 21 total (was 20) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
14s{color} | {color:red} service: The patch generated 1 new + 42 unchanged - 0
fixed = 43 total (was 42) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
16s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 17m 46s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-10690/dev-support/hive-personality.sh
|
| git revision | master / 39917ef |
| Default Java | 1.8.0_111 |
| mvninstall |
http://104.198.109.242/logs//PreCommit-HIVE-Build-10690/yetus/patch-mvninstall-service.txt
|
| compile |
http://104.198.109.242/logs//PreCommit-HIVE-Build-10690/yetus/patch-compile-service.txt
|
| javac |
http://104.198.109.242/logs//PreCommit-HIVE-Build-10690/yetus/patch-compile-service.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-10690/yetus/diff-checkstyle-itests_hive-unit.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-10690/yetus/diff-checkstyle-service.txt
|
| modules | C: common itests/hive-unit service U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-10690/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Support CORS for all HS2 web endpoints
> --
>
> Key: HIVE-19415
> URL: https://issues.apache.org/jira/browse/HIVE-19415
> Project: Hive
> Issue Type: Improvement
> Components: Web UI
>Affects Versions: 3.0.0, 3.1.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
>Priority: Minor
> Attachments: HIVE-19415.1.patch
>
>
> HIVE-19277 changes alone are not sufficient to support CORS.
> CrossOriginFilter has to be added to jetty which will serve appropriate
> response for OPTIONS pre-flight request.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (HIVE-5207) Support data encryption for Hive tables
[ https://issues.apache.org/jira/browse/HIVE-5207?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergio Peña resolved HIVE-5207. --- Resolution: Duplicate Closing this issue as HDFS encryption is supported in Hive by HIVE-8065 > Support data encryption for Hive tables > --- > > Key: HIVE-5207 > URL: https://issues.apache.org/jira/browse/HIVE-5207 > Project: Hive > Issue Type: New Feature >Affects Versions: 0.12.0 >Reporter: Haifeng Chen >Priority: Major > Labels: Rhino > Attachments: HIVE-5207.patch, HIVE-5207.patch > > Original Estimate: 504h > Remaining Estimate: 504h > > For sensitive and legally protected data such as personal information, it is > a common practice that the data is stored encrypted in the file system. To > enable Hive with the ability to store and query the encrypted data is very > crucial for Hive data analysis in enterprise. > > When creating table, user can specify whether a table is an encrypted table > or not by specify a property in TBLPROPERTIES. Once an encrypted table is > created, query on the encrypted table is transparent as long as the > corresponding key management facilities are set in the running environment of > query. We can use hadoop crypto provided by HADOOP-9331 for underlying data > encryption and decryption. > > As to key management, we would support several common key management use > cases. First, the table key (data key) can be stored in the Hive metastore > associated with the table in properties. The table key can be explicit > specified or auto generated and will be encrypted with a master key. There > are cases that the data being processed is generated by other applications, > we need to support externally managed or imported table keys. Also, the data > generated by Hive may be consumed by other applications in the system. We > need to a tool or command for exporting the table key to a java keystore for > using externally. > > To handle versions of Hadoop that do not have crypto support, we can avoid > compilation problems by segregating crypto API usage into separate files > (shims) to be included only if a flag is defined on the Ant command line > (something like –Dcrypto=true). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19212) Fix findbugs yetus pre-commit checks
[
https://issues.apache.org/jira/browse/HIVE-19212?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16464040#comment-16464040
]
Sahil Takiar commented on HIVE-19212:
-
Pushed to master, but it looks like I have to run
{{/usr/local/hiveptest/bin/update-hiveptest.sh}} to deploy this. I'll do that
over the weekend when there are less pending tests in the queue.
> Fix findbugs yetus pre-commit checks
>
>
> Key: HIVE-19212
> URL: https://issues.apache.org/jira/browse/HIVE-19212
> Project: Hive
> Issue Type: Sub-task
> Components: Testing Infrastructure
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
>Priority: Major
> Attachments: HIVE-19212.1.patch, HIVE-19212.2.patch,
> HIVE-19212.3.patch
>
>
> Follow up from HIVE-18883, the committed patch isn't working and Findbugs is
> still not working.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19317) Handle schema evolution from int like types to decimal
[ https://issues.apache.org/jira/browse/HIVE-19317?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Janaki Lahorani updated HIVE-19317: --- Attachment: (was: HIVE-19317.4.patch) > Handle schema evolution from int like types to decimal > -- > > Key: HIVE-19317 > URL: https://issues.apache.org/jira/browse/HIVE-19317 > Project: Hive > Issue Type: Bug >Reporter: Janaki Lahorani >Assignee: Janaki Lahorani >Priority: Major > Attachments: HIVE-19317.1.patch, HIVE-19317.2.patch, > HIVE-19317.3.patch, HIVE-19317.4.patch > > > If int like type is changed to decimal on parquet data, select results in > errors. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19317) Handle schema evolution from int like types to decimal
[ https://issues.apache.org/jira/browse/HIVE-19317?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Janaki Lahorani updated HIVE-19317: --- Attachment: (was: HIVE-19317.4.patch) > Handle schema evolution from int like types to decimal > -- > > Key: HIVE-19317 > URL: https://issues.apache.org/jira/browse/HIVE-19317 > Project: Hive > Issue Type: Bug >Reporter: Janaki Lahorani >Assignee: Janaki Lahorani >Priority: Major > Attachments: HIVE-19317.1.patch, HIVE-19317.2.patch, > HIVE-19317.3.patch, HIVE-19317.4.patch > > > If int like type is changed to decimal on parquet data, select results in > errors. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19421) Upgrade versions of Jetty and Jackson
[ https://issues.apache.org/jira/browse/HIVE-19421?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Janaki Lahorani updated HIVE-19421: --- Status: Patch Available (was: Open) > Upgrade versions of Jetty and Jackson > - > > Key: HIVE-19421 > URL: https://issues.apache.org/jira/browse/HIVE-19421 > Project: Hive > Issue Type: Improvement >Reporter: Janaki Lahorani >Assignee: Janaki Lahorani >Priority: Major > > Move Jackson up to 2.9.5 > Move Jetty up to 9.3.20.v20170531 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-19421) Upgrade versions of Jetty and Jackson
[ https://issues.apache.org/jira/browse/HIVE-19421?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Janaki Lahorani reassigned HIVE-19421: -- Assignee: Janaki Lahorani > Upgrade versions of Jetty and Jackson > - > > Key: HIVE-19421 > URL: https://issues.apache.org/jira/browse/HIVE-19421 > Project: Hive > Issue Type: Improvement >Reporter: Janaki Lahorani >Assignee: Janaki Lahorani >Priority: Major > > Move Jackson up to 2.9.5 > Move Jetty up to 9.3.20.v20170531 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19421) Upgrade versions of Jetty and Jackson
[ https://issues.apache.org/jira/browse/HIVE-19421?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Janaki Lahorani updated HIVE-19421: --- Attachment: HIVE-19421.1.patch > Upgrade versions of Jetty and Jackson > - > > Key: HIVE-19421 > URL: https://issues.apache.org/jira/browse/HIVE-19421 > Project: Hive > Issue Type: Improvement >Reporter: Janaki Lahorani >Assignee: Janaki Lahorani >Priority: Major > Attachments: HIVE-19421.1.patch > > > Move Jackson up to 2.9.5 > Move Jetty up to 9.3.20.v20170531 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18131) Truncate table for Acid tables
[ https://issues.apache.org/jira/browse/HIVE-18131?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman updated HIVE-18131: -- Attachment: HIVE-18131.03.patch > Truncate table for Acid tables > -- > > Key: HIVE-18131 > URL: https://issues.apache.org/jira/browse/HIVE-18131 > Project: Hive > Issue Type: Task > Components: Transactions >Reporter: Eugene Koifman >Assignee: Eugene Koifman >Priority: Major > Attachments: HIVE-18131.01.patch, HIVE-18131.02.patch, > HIVE-18131.03.patch > > > How should this work? Should it work like Insert Overwrite T select * from T > where 1=2? > This should create a new empty base_x/ and thus operate w/o violating > Snapshot Isolation semantics. > This makes sense for specific partition or unpartitioned table. What about > "Truncate T" where T is partitioned? Is the expectation to wipe out all > partition info or to make each partition empty? -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18131) Truncate table for Acid tables
[ https://issues.apache.org/jira/browse/HIVE-18131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16464068#comment-16464068 ] Eugene Koifman commented on HIVE-18131: --- same patch again since the last one was again lost in the Bermuda triangle of the build queue > Truncate table for Acid tables > -- > > Key: HIVE-18131 > URL: https://issues.apache.org/jira/browse/HIVE-18131 > Project: Hive > Issue Type: Task > Components: Transactions >Reporter: Eugene Koifman >Assignee: Eugene Koifman >Priority: Major > Attachments: HIVE-18131.01.patch, HIVE-18131.02.patch, > HIVE-18131.03.patch > > > How should this work? Should it work like Insert Overwrite T select * from T > where 1=2? > This should create a new empty base_x/ and thus operate w/o violating > Snapshot Isolation semantics. > This makes sense for specific partition or unpartitioned table. What about > "Truncate T" where T is partitioned? Is the expectation to wipe out all > partition info or to make each partition empty? -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18288) merge/concat not supported on Acid table
[
https://issues.apache.org/jira/browse/HIVE-18288?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eugene Koifman updated HIVE-18288:
--
Attachment: HIVE-18288.02.patch
> merge/concat not supported on Acid table
>
>
> Key: HIVE-18288
> URL: https://issues.apache.org/jira/browse/HIVE-18288
> Project: Hive
> Issue Type: Sub-task
> Components: Transactions
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
>Priority: Major
> Attachments: HIVE-18288.01.patch, HIVE-18288.02.patch,
> HIVE-18288.02.patch
>
>
> For example, mvn test -Dtest=TestCliDriver -Dqfile=orc_merge10.q
> now ends up with
> {noformat}
> 2017-12-15T15:12:30,753 ERROR [7c3ff5b2-285c-44f2-8b13-5c3ccbd41b13 main]
> ql.Driver: FAILED: SemanticException
> org.apache.hadoop.hive.ql.parse.SemanticException: Concatenate/M\
> erge can not be performed on transactional tables
> org.apache.hadoop.hive.ql.parse.SemanticException:
> org.apache.hadoop.hive.ql.parse.SemanticException: Concatenate/Merge can not
> be performed on transactional tables
> at
> org.apache.hadoop.hive.ql.parse.DDLSemanticAnalyzer.analyzeAlterTablePartMergeFiles(DDLSemanticAnalyzer.java:2172)
> at
> org.apache.hadoop.hive.ql.parse.DDLSemanticAnalyzer.analyzeInternal(DDLSemanticAnalyzer.java:343)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18288) merge/concat not supported on Acid table
[
https://issues.apache.org/jira/browse/HIVE-18288?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eugene Koifman updated HIVE-18288:
--
Attachment: (was: HIVE-18288.02.patch)
> merge/concat not supported on Acid table
>
>
> Key: HIVE-18288
> URL: https://issues.apache.org/jira/browse/HIVE-18288
> Project: Hive
> Issue Type: Sub-task
> Components: Transactions
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
>Priority: Major
> Attachments: HIVE-18288.01.patch, HIVE-18288.02.patch
>
>
> For example, mvn test -Dtest=TestCliDriver -Dqfile=orc_merge10.q
> now ends up with
> {noformat}
> 2017-12-15T15:12:30,753 ERROR [7c3ff5b2-285c-44f2-8b13-5c3ccbd41b13 main]
> ql.Driver: FAILED: SemanticException
> org.apache.hadoop.hive.ql.parse.SemanticException: Concatenate/M\
> erge can not be performed on transactional tables
> org.apache.hadoop.hive.ql.parse.SemanticException:
> org.apache.hadoop.hive.ql.parse.SemanticException: Concatenate/Merge can not
> be performed on transactional tables
> at
> org.apache.hadoop.hive.ql.parse.DDLSemanticAnalyzer.analyzeAlterTablePartMergeFiles(DDLSemanticAnalyzer.java:2172)
> at
> org.apache.hadoop.hive.ql.parse.DDLSemanticAnalyzer.analyzeInternal(DDLSemanticAnalyzer.java:343)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18288) merge/concat not supported on Acid table
[
https://issues.apache.org/jira/browse/HIVE-18288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16464073#comment-16464073
]
Eugene Koifman commented on HIVE-18288:
---
same patch again
> merge/concat not supported on Acid table
>
>
> Key: HIVE-18288
> URL: https://issues.apache.org/jira/browse/HIVE-18288
> Project: Hive
> Issue Type: Sub-task
> Components: Transactions
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
>Priority: Major
> Attachments: HIVE-18288.01.patch, HIVE-18288.02.patch
>
>
> For example, mvn test -Dtest=TestCliDriver -Dqfile=orc_merge10.q
> now ends up with
> {noformat}
> 2017-12-15T15:12:30,753 ERROR [7c3ff5b2-285c-44f2-8b13-5c3ccbd41b13 main]
> ql.Driver: FAILED: SemanticException
> org.apache.hadoop.hive.ql.parse.SemanticException: Concatenate/M\
> erge can not be performed on transactional tables
> org.apache.hadoop.hive.ql.parse.SemanticException:
> org.apache.hadoop.hive.ql.parse.SemanticException: Concatenate/Merge can not
> be performed on transactional tables
> at
> org.apache.hadoop.hive.ql.parse.DDLSemanticAnalyzer.analyzeAlterTablePartMergeFiles(DDLSemanticAnalyzer.java:2172)
> at
> org.apache.hadoop.hive.ql.parse.DDLSemanticAnalyzer.analyzeInternal(DDLSemanticAnalyzer.java:343)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18288) merge/concat not supported on Acid table
[
https://issues.apache.org/jira/browse/HIVE-18288?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eugene Koifman updated HIVE-18288:
--
Attachment: (was: HIVE-18288.02.patch)
> merge/concat not supported on Acid table
>
>
> Key: HIVE-18288
> URL: https://issues.apache.org/jira/browse/HIVE-18288
> Project: Hive
> Issue Type: Sub-task
> Components: Transactions
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
>Priority: Major
> Attachments: HIVE-18288.01.patch, HIVE-18288.02.patch
>
>
> For example, mvn test -Dtest=TestCliDriver -Dqfile=orc_merge10.q
> now ends up with
> {noformat}
> 2017-12-15T15:12:30,753 ERROR [7c3ff5b2-285c-44f2-8b13-5c3ccbd41b13 main]
> ql.Driver: FAILED: SemanticException
> org.apache.hadoop.hive.ql.parse.SemanticException: Concatenate/M\
> erge can not be performed on transactional tables
> org.apache.hadoop.hive.ql.parse.SemanticException:
> org.apache.hadoop.hive.ql.parse.SemanticException: Concatenate/Merge can not
> be performed on transactional tables
> at
> org.apache.hadoop.hive.ql.parse.DDLSemanticAnalyzer.analyzeAlterTablePartMergeFiles(DDLSemanticAnalyzer.java:2172)
> at
> org.apache.hadoop.hive.ql.parse.DDLSemanticAnalyzer.analyzeInternal(DDLSemanticAnalyzer.java:343)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18288) merge/concat not supported on Acid table
[
https://issues.apache.org/jira/browse/HIVE-18288?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eugene Koifman updated HIVE-18288:
--
Attachment: HIVE-18288.02.patch
> merge/concat not supported on Acid table
>
>
> Key: HIVE-18288
> URL: https://issues.apache.org/jira/browse/HIVE-18288
> Project: Hive
> Issue Type: Sub-task
> Components: Transactions
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
>Priority: Major
> Attachments: HIVE-18288.01.patch, HIVE-18288.02.patch
>
>
> For example, mvn test -Dtest=TestCliDriver -Dqfile=orc_merge10.q
> now ends up with
> {noformat}
> 2017-12-15T15:12:30,753 ERROR [7c3ff5b2-285c-44f2-8b13-5c3ccbd41b13 main]
> ql.Driver: FAILED: SemanticException
> org.apache.hadoop.hive.ql.parse.SemanticException: Concatenate/M\
> erge can not be performed on transactional tables
> org.apache.hadoop.hive.ql.parse.SemanticException:
> org.apache.hadoop.hive.ql.parse.SemanticException: Concatenate/Merge can not
> be performed on transactional tables
> at
> org.apache.hadoop.hive.ql.parse.DDLSemanticAnalyzer.analyzeAlterTablePartMergeFiles(DDLSemanticAnalyzer.java:2172)
> at
> org.apache.hadoop.hive.ql.parse.DDLSemanticAnalyzer.analyzeInternal(DDLSemanticAnalyzer.java:343)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19298) Fix operator tree of CTAS for Druid Storage Handler
[
https://issues.apache.org/jira/browse/HIVE-19298?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
slim bouguerra updated HIVE-19298:
--
Attachment: HIVE-19298.3.patch
> Fix operator tree of CTAS for Druid Storage Handler
> ---
>
> Key: HIVE-19298
> URL: https://issues.apache.org/jira/browse/HIVE-19298
> Project: Hive
> Issue Type: Bug
> Components: Druid integration
>Reporter: slim bouguerra
>Assignee: slim bouguerra
>Priority: Major
> Fix For: 3.1.0
>
> Attachments: HIVE-19298.2.patch, HIVE-19298.3.patch,
> HIVE-19298.patch, HIVE-19298.patch
>
>
> Current operator plan of CTAS for Druid storage handler is broken when used
> enables the property \{code} hive.exec.parallel\{code} as \{code} true\{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19322) broken test: TestNegativeMinimrCliDriver#testCliDriver[minimr_broken_pipe]
[
https://issues.apache.org/jira/browse/HIVE-19322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16464077#comment-16464077
]
Eugene Koifman commented on HIVE-19322:
---
rebase patch
> broken test: TestNegativeMinimrCliDriver#testCliDriver[minimr_broken_pipe]
> --
>
> Key: HIVE-19322
> URL: https://issues.apache.org/jira/browse/HIVE-19322
> Project: Hive
> Issue Type: Sub-task
> Components: Test, Transactions
>Affects Versions: 3.0.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
>Priority: Major
> Attachments: HIVE-19322.01-branch-3.patch,
> HIVE-19322.02-branch-3.patch, HIVE-19322.02.patch,
> HIVE-19322.03-branch-3.patch, HIVE-19322.03.patch,
> HIVE-19322.04-branch-3.patch, HIVE-19322.04.patch, HIVE-19322.05.patch,
> HIVE-19322.06.patch, HIVE-19322.07.patch
>
>
> this is apparently caused by HIVE-18739, specifically changing
> {{private static ThreadLocal tss}} in {{SessionState}} to
> {{private static InheritableThreadLocal tss}}
> need to figure out why this is.
> Looks like
> {{TestNegativeMinimrCliDriver
> -Dqfile=mapreduce_stack_trace_turnoff.q,mapreduce_stack_trace.q,cluster_tasklog_retrieval.q}}
> are also broken by this
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19322) broken test: TestNegativeMinimrCliDriver#testCliDriver[minimr_broken_pipe]
[
https://issues.apache.org/jira/browse/HIVE-19322?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eugene Koifman updated HIVE-19322:
--
Attachment: HIVE-19322.04-branch-3.patch
> broken test: TestNegativeMinimrCliDriver#testCliDriver[minimr_broken_pipe]
> --
>
> Key: HIVE-19322
> URL: https://issues.apache.org/jira/browse/HIVE-19322
> Project: Hive
> Issue Type: Sub-task
> Components: Test, Transactions
>Affects Versions: 3.0.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
>Priority: Major
> Attachments: HIVE-19322.01-branch-3.patch,
> HIVE-19322.02-branch-3.patch, HIVE-19322.02.patch,
> HIVE-19322.03-branch-3.patch, HIVE-19322.03.patch,
> HIVE-19322.04-branch-3.patch, HIVE-19322.04.patch, HIVE-19322.05.patch,
> HIVE-19322.06.patch, HIVE-19322.07.patch
>
>
> this is apparently caused by HIVE-18739, specifically changing
> {{private static ThreadLocal tss}} in {{SessionState}} to
> {{private static InheritableThreadLocal tss}}
> need to figure out why this is.
> Looks like
> {{TestNegativeMinimrCliDriver
> -Dqfile=mapreduce_stack_trace_turnoff.q,mapreduce_stack_trace.q,cluster_tasklog_retrieval.q}}
> are also broken by this
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19298) Fix operator tree of CTAS for Druid Storage Handler
[
https://issues.apache.org/jira/browse/HIVE-19298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16464090#comment-16464090
]
Ashutosh Chauhan commented on HIVE-19298:
-
+1
> Fix operator tree of CTAS for Druid Storage Handler
> ---
>
> Key: HIVE-19298
> URL: https://issues.apache.org/jira/browse/HIVE-19298
> Project: Hive
> Issue Type: Bug
> Components: Druid integration
>Reporter: slim bouguerra
>Assignee: slim bouguerra
>Priority: Major
> Fix For: 3.1.0
>
> Attachments: HIVE-19298.2.patch, HIVE-19298.3.patch,
> HIVE-19298.patch, HIVE-19298.patch
>
>
> Current operator plan of CTAS for Druid storage handler is broken when used
> enables the property \{code} hive.exec.parallel\{code} as \{code} true\{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-17978) TPCDS queries 58 and 83 generate exceptions in the vectorization.
[
https://issues.apache.org/jira/browse/HIVE-17978?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16464095#comment-16464095
]
Ashutosh Chauhan commented on HIVE-17978:
-
+1
> TPCDS queries 58 and 83 generate exceptions in the vectorization.
> -
>
> Key: HIVE-17978
> URL: https://issues.apache.org/jira/browse/HIVE-17978
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Reporter: Nita Dembla
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-17978.patch
>
>
> Failed with the following Exception
> {code}
> ERROR [6c707c4e-2849-4ff2-809d-946581e6b83a HiveServer2-Handler-Pool:
> Thread-78] ql.Driver: FAILED: SemanticException
> org.apache.hadoop.hive.ql.metadata.HiveException: The column KEY._col0 is not
> in the vectorization context column map {_col0=0}.
> org.apache.hadoop.hive.ql.parse.SemanticException:
> org.apache.hadoop.hive.ql.metadata.HiveException: The column KEY._col0 is not
> in the vectorization context column map {_col0=0}.
> at
> org.apache.hadoop.hive.ql.optimizer.physical.Vectorizer$VectorizationNodeProcessor.doVectorize(Vectorizer.java:1665)
> at
> org.apache.hadoop.hive.ql.optimizer.physical.Vectorizer$MapWorkVectorizationNodeProcessor.process(Vectorizer.java:1725)
> at
> org.apache.hadoop.hive.ql.lib.DefaultRuleDispatcher.dispatch(DefaultRuleDispatcher.java:90)
> at
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatchAndReturn(DefaultGraphWalker.java:105)
> at
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatch(DefaultGraphWalker.java:89)
> at
> org.apache.hadoop.hive.ql.lib.PreOrderOnceWalker.walk(PreOrderOnceWalker.java:43)
> at
> org.apache.hadoop.hive.ql.lib.PreOrderOnceWalker.walk(PreOrderOnceWalker.java:54)
> at
> org.apache.hadoop.hive.ql.lib.PreOrderOnceWalker.walk(PreOrderOnceWalker.java:54)
> at
> org.apache.hadoop.hive.ql.lib.PreOrderOnceWalker.walk(PreOrderOnceWalker.java:54)
> at
> org.apache.hadoop.hive.ql.lib.PreOrderOnceWalker.walk(PreOrderOnceWalker.java:54)
> at
> org.apache.hadoop.hive.ql.lib.PreOrderOnceWalker.walk(PreOrderOnceWalker.java:54)
> at
> org.apache.hadoop.hive.ql.lib.PreOrderOnceWalker.walk(PreOrderOnceWalker.java:54)
> at
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.startWalking(DefaultGraphWalker.java:120)
> at
> org.apache.hadoop.hive.ql.optimizer.physical.Vectorizer$VectorizationDispatcher.vectorizeMapWork(Vectorizer.java:1245)
> at
> org.apache.hadoop.hive.ql.optimizer.physical.Vectorizer$VectorizationDispatcher.convertMapWork(Vectorizer.java:671)
> at
> org.apache.hadoop.hive.ql.optimizer.physical.Vectorizer$VectorizationDispatcher.dispatch(Vectorizer.java:616)
> at
> org.apache.hadoop.hive.ql.lib.TaskGraphWalker.dispatch(TaskGraphWalker.java:111)
> at
> org.apache.hadoop.hive.ql.lib.TaskGraphWalker.walk(TaskGraphWalker.java:180)
> at
> org.apache.hadoop.hive.ql.lib.TaskGraphWalker.startWalking(TaskGraphWalker.java:125)
> at
> org.apache.hadoop.hive.ql.optimizer.physical.Vectorizer.resolve(Vectorizer.java:1902)
> at
> org.apache.hadoop.hive.ql.parse.TezCompiler.optimizeTaskPlan(TezCompiler.java:674)
> at
> org.apache.hadoop.hive.ql.parse.TaskCompiler.compile(TaskCompiler.java:271)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:11621)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:296)
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:268)
> at
> org.apache.hadoop.hive.ql.parse.ExplainSemanticAnalyzer.analyzeInternal(ExplainSemanticAnalyzer.java:169)
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:268)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:599)
> at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1463)
> at
> org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1420)
> at
> org.apache.hive.service.cli.operation.SQLOperation.prepare(SQLOperation.java:201)
> at
> org.apache.hive.service.cli.operation.SQLOperation.runInternal(SQLOperation.java:288)
> at
> org.apache.hive.service.cli.operation.Operation.run(Operation.java:249)
> at
> org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementInternal(HiveSessionImpl.java:532)
> at
> org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementAsync(HiveSessionImpl.java:518)
> at
> org.apache.hive.service.cli.CLIService.executeStatementAsync(CLIServic
[jira] [Commented] (HIVE-17978) TPCDS queries 58 and 83 generate exceptions in the vectorization.
[
https://issues.apache.org/jira/browse/HIVE-17978?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16464098#comment-16464098
]
Ashutosh Chauhan commented on HIVE-17978:
-
In a follow-up we shall allow merging of 2 TSs both of which have semijoin
branch.
Also, we should consider running sharedwork optimizer after semijoin branches
are removed.
> TPCDS queries 58 and 83 generate exceptions in the vectorization.
> -
>
> Key: HIVE-17978
> URL: https://issues.apache.org/jira/browse/HIVE-17978
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Reporter: Nita Dembla
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-17978.patch
>
>
> Failed with the following Exception
> {code}
> ERROR [6c707c4e-2849-4ff2-809d-946581e6b83a HiveServer2-Handler-Pool:
> Thread-78] ql.Driver: FAILED: SemanticException
> org.apache.hadoop.hive.ql.metadata.HiveException: The column KEY._col0 is not
> in the vectorization context column map {_col0=0}.
> org.apache.hadoop.hive.ql.parse.SemanticException:
> org.apache.hadoop.hive.ql.metadata.HiveException: The column KEY._col0 is not
> in the vectorization context column map {_col0=0}.
> at
> org.apache.hadoop.hive.ql.optimizer.physical.Vectorizer$VectorizationNodeProcessor.doVectorize(Vectorizer.java:1665)
> at
> org.apache.hadoop.hive.ql.optimizer.physical.Vectorizer$MapWorkVectorizationNodeProcessor.process(Vectorizer.java:1725)
> at
> org.apache.hadoop.hive.ql.lib.DefaultRuleDispatcher.dispatch(DefaultRuleDispatcher.java:90)
> at
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatchAndReturn(DefaultGraphWalker.java:105)
> at
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatch(DefaultGraphWalker.java:89)
> at
> org.apache.hadoop.hive.ql.lib.PreOrderOnceWalker.walk(PreOrderOnceWalker.java:43)
> at
> org.apache.hadoop.hive.ql.lib.PreOrderOnceWalker.walk(PreOrderOnceWalker.java:54)
> at
> org.apache.hadoop.hive.ql.lib.PreOrderOnceWalker.walk(PreOrderOnceWalker.java:54)
> at
> org.apache.hadoop.hive.ql.lib.PreOrderOnceWalker.walk(PreOrderOnceWalker.java:54)
> at
> org.apache.hadoop.hive.ql.lib.PreOrderOnceWalker.walk(PreOrderOnceWalker.java:54)
> at
> org.apache.hadoop.hive.ql.lib.PreOrderOnceWalker.walk(PreOrderOnceWalker.java:54)
> at
> org.apache.hadoop.hive.ql.lib.PreOrderOnceWalker.walk(PreOrderOnceWalker.java:54)
> at
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.startWalking(DefaultGraphWalker.java:120)
> at
> org.apache.hadoop.hive.ql.optimizer.physical.Vectorizer$VectorizationDispatcher.vectorizeMapWork(Vectorizer.java:1245)
> at
> org.apache.hadoop.hive.ql.optimizer.physical.Vectorizer$VectorizationDispatcher.convertMapWork(Vectorizer.java:671)
> at
> org.apache.hadoop.hive.ql.optimizer.physical.Vectorizer$VectorizationDispatcher.dispatch(Vectorizer.java:616)
> at
> org.apache.hadoop.hive.ql.lib.TaskGraphWalker.dispatch(TaskGraphWalker.java:111)
> at
> org.apache.hadoop.hive.ql.lib.TaskGraphWalker.walk(TaskGraphWalker.java:180)
> at
> org.apache.hadoop.hive.ql.lib.TaskGraphWalker.startWalking(TaskGraphWalker.java:125)
> at
> org.apache.hadoop.hive.ql.optimizer.physical.Vectorizer.resolve(Vectorizer.java:1902)
> at
> org.apache.hadoop.hive.ql.parse.TezCompiler.optimizeTaskPlan(TezCompiler.java:674)
> at
> org.apache.hadoop.hive.ql.parse.TaskCompiler.compile(TaskCompiler.java:271)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:11621)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:296)
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:268)
> at
> org.apache.hadoop.hive.ql.parse.ExplainSemanticAnalyzer.analyzeInternal(ExplainSemanticAnalyzer.java:169)
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:268)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:599)
> at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1463)
> at
> org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1420)
> at
> org.apache.hive.service.cli.operation.SQLOperation.prepare(SQLOperation.java:201)
> at
> org.apache.hive.service.cli.operation.SQLOperation.runInternal(SQLOperation.java:288)
> at
> org.apache.hive.service.cli.operation.Operation.run(Operation.java:249)
> at
> org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementInternal(HiveSessionImpl.java:532)
> at
> org.apache.h
[jira] [Commented] (HIVE-19415) Support CORS for all HS2 web endpoints
[
https://issues.apache.org/jira/browse/HIVE-19415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16464103#comment-16464103
]
Hive QA commented on HIVE-19415:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12921853/HIVE-19415.1.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 35 failed/errored test(s), 14316 tests
executed
*Failed tests:*
{noformat}
TestDbNotificationListener - did not produce a TEST-*.xml file (likely timed
out) (batchId=247)
TestHCatHiveCompatibility - did not produce a TEST-*.xml file (likely timed
out) (batchId=247)
TestNonCatCallsWithCatalog - did not produce a TEST-*.xml file (likely timed
out) (batchId=217)
TestSequenceFileReadWrite - did not produce a TEST-*.xml file (likely timed
out) (batchId=247)
TestTxnExIm - did not produce a TEST-*.xml file (likely timed out) (batchId=286)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucket_map_join_tez1]
(batchId=175)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucket_map_join_tez2]
(batchId=156)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[materialized_view_create_rewrite_5]
(batchId=154)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[materialized_view_create_rewrite_rebuild_dummy]
(batchId=161)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[materialized_view_create_rewrite_time_window]
(batchId=156)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb]
(batchId=163)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[tez_dynpart_hashjoin_1]
(batchId=174)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[tez_vector_dynpart_hashjoin_1]
(batchId=172)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[union_fast_stats]
(batchId=167)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_5]
(batchId=105)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[udf_reflect_neg]
(batchId=96)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[udf_test_error]
(batchId=96)
org.apache.hadoop.hive.ql.TestAcidOnTez.testCtasTezUnion (batchId=228)
org.apache.hadoop.hive.ql.TestAcidOnTez.testNonStandardConversion01
(batchId=228)
org.apache.hadoop.hive.ql.TestMTQueries.testMTQueries1 (batchId=232)
org.apache.hadoop.hive.ql.parse.TestCopyUtils.testPrivilegedDistCpWithSameUserAsCurrentDoesNotTryToImpersonate
(batchId=231)
org.apache.hadoop.hive.ql.parse.TestReplicationOnHDFSEncryptedZones.targetAndSourceHaveDifferentEncryptionZoneKeys
(batchId=231)
org.apache.hive.beeline.TestBeeLineWithArgs.testQueryProgress (batchId=235)
org.apache.hive.beeline.TestBeeLineWithArgs.testQueryProgressParallel
(batchId=235)
org.apache.hive.jdbc.TestSSL.testSSLFetchHttp (batchId=239)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testMultipleTriggers2
(batchId=241)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerCustomCreatedFiles
(batchId=241)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerCustomNonExistent
(batchId=241)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerCustomReadOps
(batchId=241)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerHighBytesRead
(batchId=241)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerHighBytesWrite
(batchId=241)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerHighShuffleBytes
(batchId=241)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerSlowQueryElapsedTime
(batchId=241)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerSlowQueryExecutionTime
(batchId=241)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerVertexRawInputSplitsNoKill
(batchId=241)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/10690/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/10690/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-10690/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 35 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12921853 - PreCommit-HIVE-Build
> Support CORS for all HS2 web endpoints
> --
>
> Key: HIVE-19415
> URL: https://issues.apache.org/jira/browse/HIVE-19415
> Project: Hive
> Issue Type: Improvement
> Components: Web UI
>Affects Versions: 3.0.0, 3.1.0
>Reporter: Prasanth Jayach
[jira] [Commented] (HIVE-19400) Adjust Hive 1.0 to 2.0 conversion utility to the upgrade
[ https://issues.apache.org/jira/browse/HIVE-19400?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16464104#comment-16464104 ] Ashutosh Chauhan commented on HIVE-19400: - +1 > Adjust Hive 1.0 to 2.0 conversion utility to the upgrade > > > Key: HIVE-19400 > URL: https://issues.apache.org/jira/browse/HIVE-19400 > Project: Hive > Issue Type: Sub-task > Components: Hive >Affects Versions: 3.0.0 >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Fix For: 3.0.0 > > Attachments: HIVE-19400.01.patch > > > Conversion utility should allow specification of the output dir, and create > files only if there is actually something to do. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19184) Hive 3.0.0 release branch preparation
[
https://issues.apache.org/jira/browse/HIVE-19184?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16464136#comment-16464136
]
Hive QA commented on HIVE-19184:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
39s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 1m 5s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-10692/dev-support/hive-personality.sh
|
| git revision | master / 39917ef |
| modules | C: . U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-10692/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Hive 3.0.0 release branch preparation
> -
>
> Key: HIVE-19184
> URL: https://issues.apache.org/jira/browse/HIVE-19184
> Project: Hive
> Issue Type: Task
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HIVE-19184.01-branch-3.patch
>
>
> Need to do bunch of things to prepare branch-3 for release e.g.
> * Update pom.xml to delete SNAPSHOT
> * Update .reviewboardrc
> * Remove storage-api module to build
> * Change storage-api depdency etc
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19354) from_utc_timestamp returns incorrect results for datetime values with timezone
[
https://issues.apache.org/jira/browse/HIVE-19354?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bharathkrishna Guruvayoor Murali updated HIVE-19354:
Attachment: (was: HIVE-19354.01.patch)
> from_utc_timestamp returns incorrect results for datetime values with timezone
> --
>
> Key: HIVE-19354
> URL: https://issues.apache.org/jira/browse/HIVE-19354
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 3.1.0
>Reporter: Bruce Robbins
>Assignee: Bharathkrishna Guruvayoor Murali
>Priority: Major
>
> On the master branch, from_utc_timestamp returns incorrect results for
> datetime strings that contain a timezone:
> {noformat}
> hive> select from_utc_timestamp('2000-10-10 00:00:00+00:00',
> 'America/Los_Angeles');
> OK
> 2000-10-09 10:00:00
> Time taken: 0.294 seconds, Fetched: 1 row(s)
> hive> select from_utc_timestamp('2000-10-10 00:00:00', 'America/Los_Angeles');
> OK
> 2000-10-09 17:00:00
> Time taken: 0.121 seconds, Fetched: 1 row(s)
> hive>
> {noformat}
> Both inputs are 2000-10-10 00:00:00 in UTC time, but I got two different
> results.
> In version 2.3.3, from_utc_timestamp doesn't accept timezones in its input
> strings, so it does not have this bug:
> {noformat}
> hive> select from_utc_timestamp('2000-10-10 00:00:00+00:00',
> 'America/Los_Angeles');
> OK
> NULL
> Time taken: 5.152 seconds, Fetched: 1 row(s)
> hive> select from_utc_timestamp('2000-10-10 00:00:00', 'America/Los_Angeles');
> OK
> 2000-10-09 17:00:00
> Time taken: 0.069 seconds, Fetched: 1 row(s)
> hive>
> {noformat}
> Since the function is expecting a UTC datetime value, it probably should
> continue to reject input that contains a timezone component.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19354) from_utc_timestamp returns incorrect results for datetime values with timezone
[
https://issues.apache.org/jira/browse/HIVE-19354?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bharathkrishna Guruvayoor Murali updated HIVE-19354:
Attachment: HIVE-19354.01.patch
> from_utc_timestamp returns incorrect results for datetime values with timezone
> --
>
> Key: HIVE-19354
> URL: https://issues.apache.org/jira/browse/HIVE-19354
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 3.1.0
>Reporter: Bruce Robbins
>Assignee: Bharathkrishna Guruvayoor Murali
>Priority: Major
> Attachments: HIVE-19354.01.patch
>
>
> On the master branch, from_utc_timestamp returns incorrect results for
> datetime strings that contain a timezone:
> {noformat}
> hive> select from_utc_timestamp('2000-10-10 00:00:00+00:00',
> 'America/Los_Angeles');
> OK
> 2000-10-09 10:00:00
> Time taken: 0.294 seconds, Fetched: 1 row(s)
> hive> select from_utc_timestamp('2000-10-10 00:00:00', 'America/Los_Angeles');
> OK
> 2000-10-09 17:00:00
> Time taken: 0.121 seconds, Fetched: 1 row(s)
> hive>
> {noformat}
> Both inputs are 2000-10-10 00:00:00 in UTC time, but I got two different
> results.
> In version 2.3.3, from_utc_timestamp doesn't accept timezones in its input
> strings, so it does not have this bug:
> {noformat}
> hive> select from_utc_timestamp('2000-10-10 00:00:00+00:00',
> 'America/Los_Angeles');
> OK
> NULL
> Time taken: 5.152 seconds, Fetched: 1 row(s)
> hive> select from_utc_timestamp('2000-10-10 00:00:00', 'America/Los_Angeles');
> OK
> 2000-10-09 17:00:00
> Time taken: 0.069 seconds, Fetched: 1 row(s)
> hive>
> {noformat}
> Since the function is expecting a UTC datetime value, it probably should
> continue to reject input that contains a timezone component.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18533) Add option to use InProcessLauncher to submit spark jobs
[ https://issues.apache.org/jira/browse/HIVE-18533?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sahil Takiar updated HIVE-18533: Attachment: HIVE-18533.91.patch > Add option to use InProcessLauncher to submit spark jobs > > > Key: HIVE-18533 > URL: https://issues.apache.org/jira/browse/HIVE-18533 > Project: Hive > Issue Type: Improvement > Components: Spark >Reporter: Sahil Takiar >Assignee: Sahil Takiar >Priority: Major > Attachments: HIVE-18533.1.patch, HIVE-18533.2.patch, > HIVE-18533.3.patch, HIVE-18533.4.patch, HIVE-18533.5.patch, > HIVE-18533.6.patch, HIVE-18533.7.patch, HIVE-18533.8.patch, > HIVE-18533.9.patch, HIVE-18533.91.patch > > > See discussion in HIVE-16484 for details. > I think this will help with reducing the amount of time it takes to open a > HoS session + debuggability (no need launch a separate process to run a Spark > app). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19161) Add authorizations to information schema
[ https://issues.apache.org/jira/browse/HIVE-19161?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Dai updated HIVE-19161: -- Attachment: HIVE-19161.13.patch > Add authorizations to information schema > > > Key: HIVE-19161 > URL: https://issues.apache.org/jira/browse/HIVE-19161 > Project: Hive > Issue Type: Improvement >Reporter: Daniel Dai >Assignee: Daniel Dai >Priority: Major > Attachments: HIVE-19161.1.patch, HIVE-19161.10.patch, > HIVE-19161.11.patch, HIVE-19161.12.patch, HIVE-19161.13.patch, > HIVE-19161.2.patch, HIVE-19161.3.patch, HIVE-19161.4.patch, > HIVE-19161.5.patch, HIVE-19161.6.patch, HIVE-19161.7.patch, > HIVE-19161.8.patch, HIVE-19161.9.patch > > > We need to control the access of information schema so user can only query > the information authorized to. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19161) Add authorizations to information schema
[ https://issues.apache.org/jira/browse/HIVE-19161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16464196#comment-16464196 ] Daniel Dai commented on HIVE-19161: --- HIVE-19161.13.patch resync with master. > Add authorizations to information schema > > > Key: HIVE-19161 > URL: https://issues.apache.org/jira/browse/HIVE-19161 > Project: Hive > Issue Type: Improvement >Reporter: Daniel Dai >Assignee: Daniel Dai >Priority: Major > Attachments: HIVE-19161.1.patch, HIVE-19161.10.patch, > HIVE-19161.11.patch, HIVE-19161.12.patch, HIVE-19161.13.patch, > HIVE-19161.2.patch, HIVE-19161.3.patch, HIVE-19161.4.patch, > HIVE-19161.5.patch, HIVE-19161.6.patch, HIVE-19161.7.patch, > HIVE-19161.8.patch, HIVE-19161.9.patch > > > We need to control the access of information schema so user can only query > the information authorized to. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19389) Schematool: For Hive's Information Schema, use embedded HS2 as default
[ https://issues.apache.org/jira/browse/HIVE-19389?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vaibhav Gumashta updated HIVE-19389: Attachment: HIVE-19389.2.patch > Schematool: For Hive's Information Schema, use embedded HS2 as default > -- > > Key: HIVE-19389 > URL: https://issues.apache.org/jira/browse/HIVE-19389 > Project: Hive > Issue Type: Bug > Components: Metastore >Affects Versions: 3.0.0, 3.1.0 >Reporter: Vaibhav Gumashta >Assignee: Vaibhav Gumashta >Priority: Major > Attachments: HIVE-19389.1.patch, HIVE-19389.2.patch, > HIVE-19389.2.patch > > > Currently, for initializing/upgrading Hive's information schema, we require a > full jdbc url (for HS2). It will be good to have it connect using embedded > HS2 by default. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19415) Support CORS for all HS2 web endpoints
[ https://issues.apache.org/jira/browse/HIVE-19415?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-19415: - Resolution: Fixed Fix Version/s: 3.1.0 3.0.0 Status: Resolved (was: Patch Available) Test failures are unrelated. Committed to master and branch-3. Thanks for the review! > Support CORS for all HS2 web endpoints > -- > > Key: HIVE-19415 > URL: https://issues.apache.org/jira/browse/HIVE-19415 > Project: Hive > Issue Type: Improvement > Components: Web UI >Affects Versions: 3.0.0, 3.1.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Minor > Fix For: 3.0.0, 3.1.0 > > Attachments: HIVE-19415.1.patch > > > HIVE-19277 changes alone are not sufficient to support CORS. > CrossOriginFilter has to be added to jetty which will serve appropriate > response for OPTIONS pre-flight request. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19184) Hive 3.0.0 release branch preparation
[
https://issues.apache.org/jira/browse/HIVE-19184?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16464226#comment-16464226
]
Hive QA commented on HIVE-19184:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12919517/HIVE-19184.01-branch-3.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 72 failed/errored test(s), 14149 tests
executed
*Failed tests:*
{noformat}
TestBeeLineDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=253)
TestDummy - did not produce a TEST-*.xml file (likely timed out) (batchId=253)
TestMiniDruidCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=253)
TestMiniDruidKafkaCliDriver - did not produce a TEST-*.xml file (likely timed
out) (batchId=253)
TestNonCatCallsWithCatalog - did not produce a TEST-*.xml file (likely timed
out) (batchId=217)
TestTezPerfCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=253)
TestTxnExIm - did not produce a TEST-*.xml file (likely timed out) (batchId=286)
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
(batchId=90)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[bucket_map_join_1]
(batchId=68)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[bucket_map_join_2]
(batchId=60)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[bucketmapjoin_negative3]
(batchId=29)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[groupby_sort_1_23]
(batchId=81)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[groupby_sort_skew_1_23]
(batchId=9)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[smb_mapjoin_13]
(batchId=32)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sort_merge_join_desc_7]
(batchId=27)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb]
(batchId=163)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[tez_dynpart_hashjoin_1]
(batchId=174)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[tez_smb_1]
(batchId=172)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[tez_smb_main]
(batchId=160)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[tez_vector_dynpart_hashjoin_1]
(batchId=172)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[union_fast_stats]
(batchId=167)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_5]
(batchId=105)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[avro_non_nullable_union]
(batchId=96)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[cachingprintstream]
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[compute_stats_long]
(batchId=96)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[dyn_part3]
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[dyn_part_max_per_node]
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[dynamic_partitions_with_whitelist]
(batchId=96)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[load_data_parquet_empty]
(batchId=96)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[script_broken_pipe2]
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[script_broken_pipe3]
(batchId=96)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[script_error]
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[serde_regex2]
(batchId=96)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[stats_aggregator_error_2]
(batchId=96)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[stats_publisher_error_1]
(batchId=96)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[stats_publisher_error_2]
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[subquery_corr_in_agg]
(batchId=96)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[subquery_in_implicit_gby]
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[subquery_notin_implicit_gby]
(batchId=96)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[subquery_scalar_corr_multi_rows]
(batchId=96)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[subquery_scalar_multi_rows]
(batchId=96)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[udf_assert_true2]
(batchId=96)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[udf_assert_true]
(batchId=96)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[udf_reflect_neg]
(batchId=96)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[udf_test_error]
(batchId=96)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[udf_test_error_reduce]
(batchId=95
[jira] [Assigned] (HIVE-19423) REPL LOAD creates staging directory in source dump directory instead of table data location
[ https://issues.apache.org/jira/browse/HIVE-19423?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] mahesh kumar behera reassigned HIVE-19423: -- Assignee: mahesh kumar behera > REPL LOAD creates staging directory in source dump directory instead of table > data location > --- > > Key: HIVE-19423 > URL: https://issues.apache.org/jira/browse/HIVE-19423 > Project: Hive > Issue Type: Bug > Components: Hive, HiveServer2, repl >Reporter: mahesh kumar behera >Assignee: mahesh kumar behera >Priority: Major > Labels: Hive, Repl > Fix For: 3.0.0 > > > REPL LOAD creates staging directory in source dump directory instead of table > data location. In case of replication from on-perm to cloud it can create > problem. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19423) REPL LOAD creates staging directory in source dump directory instead of table data location
[ https://issues.apache.org/jira/browse/HIVE-19423?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] mahesh kumar behera updated HIVE-19423: --- Attachment: HIVE-19423.01.patch > REPL LOAD creates staging directory in source dump directory instead of table > data location > --- > > Key: HIVE-19423 > URL: https://issues.apache.org/jira/browse/HIVE-19423 > Project: Hive > Issue Type: Bug > Components: Hive, HiveServer2, repl >Reporter: mahesh kumar behera >Assignee: mahesh kumar behera >Priority: Major > Labels: Hive, Repl > Fix For: 3.0.0 > > Attachments: HIVE-19423.01.patch > > > REPL LOAD creates staging directory in source dump directory instead of table > data location. In case of replication from on-perm to cloud it can create > problem. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19423) REPL LOAD creates staging directory in source dump directory instead of table data location
[ https://issues.apache.org/jira/browse/HIVE-19423?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] mahesh kumar behera updated HIVE-19423: --- Status: Patch Available (was: Open) [~sankarh] [~thejas] Please review.Thanks. > REPL LOAD creates staging directory in source dump directory instead of table > data location > --- > > Key: HIVE-19423 > URL: https://issues.apache.org/jira/browse/HIVE-19423 > Project: Hive > Issue Type: Bug > Components: Hive, HiveServer2, repl >Reporter: mahesh kumar behera >Assignee: mahesh kumar behera >Priority: Major > Labels: Hive, Repl > Fix For: 3.0.0 > > Attachments: HIVE-19423.01.patch > > > REPL LOAD creates staging directory in source dump directory instead of table > data location. In case of replication from on-perm to cloud it can create > problem. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19423) REPL LOAD creates staging directory in source dump directory instead of table data location
[ https://issues.apache.org/jira/browse/HIVE-19423?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HIVE-19423: -- Labels: Hive Repl pull-request-available (was: Hive Repl) > REPL LOAD creates staging directory in source dump directory instead of table > data location > --- > > Key: HIVE-19423 > URL: https://issues.apache.org/jira/browse/HIVE-19423 > Project: Hive > Issue Type: Bug > Components: Hive, HiveServer2, repl >Reporter: mahesh kumar behera >Assignee: mahesh kumar behera >Priority: Major > Labels: Hive, Repl, pull-request-available > Fix For: 3.0.0 > > Attachments: HIVE-19423.01.patch > > > REPL LOAD creates staging directory in source dump directory instead of table > data location. In case of replication from on-perm to cloud it can create > problem. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19423) REPL LOAD creates staging directory in source dump directory instead of table data location
[ https://issues.apache.org/jira/browse/HIVE-19423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16464238#comment-16464238 ] ASF GitHub Bot commented on HIVE-19423: --- GitHub user maheshk114 opened a pull request: https://github.com/apache/hive/pull/340 HIVE-19423 : REPL LOAD creates staging directory in source dump direc… REPL LOAD creates staging directory in source dump directory instead of table data location. In case of replication from on-perm to cloud it can create problem. You can merge this pull request into a Git repository by running: $ git pull https://github.com/maheshk114/hive HIVE-19423 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/hive/pull/340.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #340 commit 44865176f1a4f3c9598c46473861482c9e64fde3 Author: Mahesh Kumar Behera Date: 2018-05-04T18:22:47Z HIVE-19423 : REPL LOAD creates staging directory in source dump directory instead of table data location > REPL LOAD creates staging directory in source dump directory instead of table > data location > --- > > Key: HIVE-19423 > URL: https://issues.apache.org/jira/browse/HIVE-19423 > Project: Hive > Issue Type: Bug > Components: Hive, HiveServer2, repl >Reporter: mahesh kumar behera >Assignee: mahesh kumar behera >Priority: Major > Labels: Hive, Repl, pull-request-available > Fix For: 3.0.0 > > Attachments: HIVE-19423.01.patch > > > REPL LOAD creates staging directory in source dump directory instead of table > data location. In case of replication from on-perm to cloud it can create > problem. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19420) Support LOAD from SeqFile to ORC table
[ https://issues.apache.org/jira/browse/HIVE-19420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16464253#comment-16464253 ] Prasanth Jayachandran commented on HIVE-19420: -- +1 > Support LOAD from SeqFile to ORC table > -- > > Key: HIVE-19420 > URL: https://issues.apache.org/jira/browse/HIVE-19420 > Project: Hive > Issue Type: Bug >Reporter: Deepak Jaiswal >Assignee: Deepak Jaiswal >Priority: Major > Attachments: HIVE-19420.1.patch > > > Support LOAD from SeqFile to ORC table. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18570) ACID IOW implemented using base may delete too much data
[
https://issues.apache.org/jira/browse/HIVE-18570?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16464260#comment-16464260
]
Hive QA commented on HIVE-18570:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
1s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
43s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
27s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
32s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
10s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
26s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
9s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
31s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
50s{color} | {color:red} ql: The patch generated 6 new + 1140 unchanged - 3
fixed = 1146 total (was 1143) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
16s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
16s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 18m 56s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-10693/dev-support/hive-personality.sh
|
| git revision | master / 7188506 |
| Default Java | 1.8.0_111 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-10693/yetus/diff-checkstyle-ql.txt
|
| modules | C: common ql U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-10693/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> ACID IOW implemented using base may delete too much data
>
>
> Key: HIVE-18570
> URL: https://issues.apache.org/jira/browse/HIVE-18570
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Reporter: Sergey Shelukhin
>Assignee: Eugene Koifman
>Priority: Blocker
> Attachments: HIVE-18570.01-branch-3.patch, HIVE-18570.01.patch,
> HIVE-18570.02-branch-3.patch, HIVE-18570.02.patch,
> HIVE-18570.03-branch-3.patch, HIVE-18570.03.patch,
> HIVE-18570.04-branch-3.patch
>
>
> Suppose we have a table with delta_0 insert data.
> Txn 1 starts an insert into delta_1.
> Txn 2 starts an IOW into base_2.
> Txn 2 commits.
> Txn 1 commits after txn 2 but its results would be invisible.
> Txn 2 deletes rows committed by txn 1 that according to standard ACID
> semantics it could have never observed and affected; this sequence of events
> is only possible under read-uncommitted isolation level (so, 2 deletes rows
> written by 1 before 1 commits them).
> This is if we look at IOW as transactional delete+insert. Otherwise we are
> just saying IOW performs "semi"-transactional delete.
> If 1 ran an update on rows instead of an insert, and 2 still ran an
> IOW/delete, row lock conflict (or equivalent) should cause one of them to
> fail.
-
[jira] [Updated] (HIVE-19400) Adjust Hive 1.0 to 2.0 conversion utility to the upgrade
[ https://issues.apache.org/jira/browse/HIVE-19400?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-19400: Resolution: Fixed Status: Resolved (was: Patch Available) Pushed to master and branch-3. Thanks, Miklos! > Adjust Hive 1.0 to 2.0 conversion utility to the upgrade > > > Key: HIVE-19400 > URL: https://issues.apache.org/jira/browse/HIVE-19400 > Project: Hive > Issue Type: Sub-task > Components: Hive >Affects Versions: 3.0.0 >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Fix For: 3.0.0 > > Attachments: HIVE-19400.01.patch > > > Conversion utility should allow specification of the output dir, and create > files only if there is actually something to do. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19418) add background stats updater similar to compactor
[ https://issues.apache.org/jira/browse/HIVE-19418?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16464272#comment-16464272 ] Sergey Shelukhin commented on HIVE-19418: - HyperLogLog is not precise; ndv is used for count distinct queries, that cannot be merged. W.r.t. background logic yeah, [~alangates] had some plans for compactor, we'd have to do the same for this one. I'm assuming we'll move them into HS2, however for load balanced case there'd need to be a primary hosting them. There's also phase 4 btw - where all the updates write increments, instead of total stats, to metastore... then, we can get stats state with one DB query, and also "compact" stats. > add background stats updater similar to compactor > - > > Key: HIVE-19418 > URL: https://issues.apache.org/jira/browse/HIVE-19418 > Project: Hive > Issue Type: Bug > Components: Transactions >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > > There's a JIRA HIVE-19416 to add snapshot version to stats for MM/ACID tables > to make them usable in a transaction without breaking ACID (for metadata-only > optimization). However, stats for ACID tables can still become unusable if > e.g. two parallel inserts run - neither sees the data written by the other, > so after both finish, the snapshots on either set of stats won't match the > current snapshot and the stats will be unusable. > Additionally, for ACID and non-ACID tables alike, a lot of the stats, with > some exceptions like numRows, cannot be aggregated (i.e. you cannot combine > ndvs from two inserts), and for ACID even less can be aggregated (you cannot > derive min/max if some rows are deleted but you don't scan the rest of the > dataset). > Therefore we will add background logic to metastore (similar to, and > partially inside, the ACID compactor) to update stats. > It will have 3 modes of operation. > 1) Off. > 2) Update only the stats that exist but are out of date (generating stats can > be expensive, so if the user is only analyzing a subset of tables it should > be able to only update that subset). We can simply look at existing stats and > only analyze for the relevant partitions and columns. > 3) On: 2 + create stats for all tables and columns missing stats. > There will also be a table parameter to skip stats update. > In phase 1, the process will operate outside of compactor, and run analyze > command on the table. The analyze command will automatically save the stats > with ACID snapshot information if needed, based on HIVE-19416, so we don't > need to do any special state management and this will work for all table > types. However it's also more expensive. > In phase 2, we can explore adding stats collection during MM compaction that > uses a temp table. If we don't have open writers during major compaction (so > we overwrite all of the data), the temp table stats can simply be copied over > to the main table with correct snapshot information, saving us a table scan. > In phase 3, we can add custom stats collection logic to full ACID compactor > that is not query based, the same way as we'd do for (2). Alternatively we > can wait for ACID compactor to become query based and just reuse (2). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19041) Thrift deserialization of Partition objects should intern fields
[
https://issues.apache.org/jira/browse/HIVE-19041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16464296#comment-16464296
]
Vihang Karajgaonkar commented on HIVE-19041:
just realized {{AddPartitionsRequest}} is not handled by default too. Will
update the patch.
> Thrift deserialization of Partition objects should intern fields
>
>
> Key: HIVE-19041
> URL: https://issues.apache.org/jira/browse/HIVE-19041
> Project: Hive
> Issue Type: Improvement
> Components: Metastore
>Affects Versions: 3.0.0, 2.3.2
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
>Priority: Major
> Attachments: HIVE-19041.01.patch, HIVE-19041.02.patch
>
>
> When a client is creating large number of partitions, the thrift objects are
> deserialized into Partition objects. The read method of these objects does
> not intern the inputformat, location, outputformat which cause large number
> of duplicate Strings in the HMS memory. We should intern these objects while
> deserialization to reduce memory pressure.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18570) ACID IOW implemented using base may delete too much data
[
https://issues.apache.org/jira/browse/HIVE-18570?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16464327#comment-16464327
]
Hive QA commented on HIVE-18570:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12921863/HIVE-18570.03.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 35 failed/errored test(s), 14319 tests
executed
*Failed tests:*
{noformat}
TestDbNotificationListener - did not produce a TEST-*.xml file (likely timed
out) (batchId=247)
TestHCatHiveCompatibility - did not produce a TEST-*.xml file (likely timed
out) (batchId=247)
TestNonCatCallsWithCatalog - did not produce a TEST-*.xml file (likely timed
out) (batchId=217)
TestSequenceFileReadWrite - did not produce a TEST-*.xml file (likely timed
out) (batchId=247)
TestTxnExIm - did not produce a TEST-*.xml file (likely timed out) (batchId=286)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucket_map_join_tez1]
(batchId=175)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucket_map_join_tez2]
(batchId=156)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[explainuser_4]
(batchId=164)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[materialized_view_create_rewrite_5]
(batchId=154)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[materialized_view_create_rewrite_rebuild_dummy]
(batchId=161)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[materialized_view_create_rewrite_time_window]
(batchId=156)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb]
(batchId=163)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[union_fast_stats]
(batchId=167)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_5]
(batchId=105)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[udf_reflect_neg]
(batchId=96)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[udf_test_error]
(batchId=96)
org.apache.hadoop.hive.ql.TestAcidOnTez.testCtasTezUnion (batchId=228)
org.apache.hadoop.hive.ql.TestAcidOnTez.testNonStandardConversion01
(batchId=228)
org.apache.hadoop.hive.ql.TestAutoPurgeTables.testAutoPurgeUnset (batchId=233)
org.apache.hadoop.hive.ql.TestMTQueries.testMTQueries1 (batchId=232)
org.apache.hadoop.hive.ql.parse.TestCopyUtils.testPrivilegedDistCpWithSameUserAsCurrentDoesNotTryToImpersonate
(batchId=231)
org.apache.hadoop.hive.ql.parse.TestReplicationOnHDFSEncryptedZones.targetAndSourceHaveDifferentEncryptionZoneKeys
(batchId=231)
org.apache.hive.beeline.TestBeeLineWithArgs.testQueryProgress (batchId=235)
org.apache.hive.beeline.TestBeeLineWithArgs.testQueryProgressParallel
(batchId=235)
org.apache.hive.jdbc.TestSSL.testSSLFetchHttp (batchId=239)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testMultipleTriggers2
(batchId=241)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerCustomCreatedFiles
(batchId=241)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerCustomNonExistent
(batchId=241)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerCustomReadOps
(batchId=241)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerHighBytesRead
(batchId=241)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerHighBytesWrite
(batchId=241)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerHighShuffleBytes
(batchId=241)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerSlowQueryElapsedTime
(batchId=241)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerSlowQueryExecutionTime
(batchId=241)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerVertexRawInputSplitsNoKill
(batchId=241)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/10693/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/10693/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-10693/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 35 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12921863 - PreCommit-HIVE-Build
> ACID IOW implemented using base may delete too much data
>
>
> Key: HIVE-18570
> URL: https://issues.apache.org/jira/browse/HIVE-18570
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Reporter: Sergey Shelukhin
>Assignee: Eugene Koifman
>
[jira] [Commented] (HIVE-14065) Provide an API for making Hive read-only for a short period
[ https://issues.apache.org/jira/browse/HIVE-14065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16464330#comment-16464330 ] Alexander Kolbasov commented on HIVE-14065: --- Apache Sentry no longer needs this API. > Provide an API for making Hive read-only for a short period > --- > > Key: HIVE-14065 > URL: https://issues.apache.org/jira/browse/HIVE-14065 > Project: Hive > Issue Type: Improvement >Reporter: Colin P. McCabe >Assignee: Colin P. McCabe >Priority: Major > > HIVE-7973 added a notification log which allows clients to do incremental > replication of the Hive metastore. However, it is a challenge to get the > initial state of the Hive database. Using existing APIs may give us an > inconsistent state. For example, if a Hive table is renamed while we're > loading all tables, we may miss that information. > The easiest way to fix this would be to provide an API for making Hive > read-only for a short period. This locking API would come with a timeout so > that if the locker failed, the system would not stay down. It would return > an ID which uniquely identified the lock instance. The read-only lock itself > could be implemented by taking all the ZooKeeper locks. The RPC for removing > the lock would return back a status indicating whether the lock had timed out > before being removed or not. If it had timed out, we could retry our > snapshot loading process with a longer timeout period. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18570) ACID IOW implemented using base may delete too much data
[ https://issues.apache.org/jira/browse/HIVE-18570?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman updated HIVE-18570: -- Attachment: HIVE-18570.05-branch-3.patch > ACID IOW implemented using base may delete too much data > > > Key: HIVE-18570 > URL: https://issues.apache.org/jira/browse/HIVE-18570 > Project: Hive > Issue Type: Bug > Components: Transactions >Reporter: Sergey Shelukhin >Assignee: Eugene Koifman >Priority: Blocker > Attachments: HIVE-18570.01-branch-3.patch, HIVE-18570.01.patch, > HIVE-18570.02-branch-3.patch, HIVE-18570.02.patch, > HIVE-18570.03-branch-3.patch, HIVE-18570.03.patch, > HIVE-18570.04-branch-3.patch, HIVE-18570.05-branch-3.patch > > > Suppose we have a table with delta_0 insert data. > Txn 1 starts an insert into delta_1. > Txn 2 starts an IOW into base_2. > Txn 2 commits. > Txn 1 commits after txn 2 but its results would be invisible. > Txn 2 deletes rows committed by txn 1 that according to standard ACID > semantics it could have never observed and affected; this sequence of events > is only possible under read-uncommitted isolation level (so, 2 deletes rows > written by 1 before 1 commits them). > This is if we look at IOW as transactional delete+insert. Otherwise we are > just saying IOW performs "semi"-transactional delete. > If 1 ran an update on rows instead of an insert, and 2 still ran an > IOW/delete, row lock conflict (or equivalent) should cause one of them to > fail. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18570) ACID IOW implemented using base may delete too much data
[ https://issues.apache.org/jira/browse/HIVE-18570?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16464335#comment-16464335 ] Eugene Koifman commented on HIVE-18570: --- no related failures for HIVE-18570.03.patch on master attaching HIVE-18570.05-branch-3.patch > ACID IOW implemented using base may delete too much data > > > Key: HIVE-18570 > URL: https://issues.apache.org/jira/browse/HIVE-18570 > Project: Hive > Issue Type: Bug > Components: Transactions >Reporter: Sergey Shelukhin >Assignee: Eugene Koifman >Priority: Blocker > Attachments: HIVE-18570.01-branch-3.patch, HIVE-18570.01.patch, > HIVE-18570.02-branch-3.patch, HIVE-18570.02.patch, > HIVE-18570.03-branch-3.patch, HIVE-18570.03.patch, > HIVE-18570.04-branch-3.patch, HIVE-18570.05-branch-3.patch > > > Suppose we have a table with delta_0 insert data. > Txn 1 starts an insert into delta_1. > Txn 2 starts an IOW into base_2. > Txn 2 commits. > Txn 1 commits after txn 2 but its results would be invisible. > Txn 2 deletes rows committed by txn 1 that according to standard ACID > semantics it could have never observed and affected; this sequence of events > is only possible under read-uncommitted isolation level (so, 2 deletes rows > written by 1 before 1 commits them). > This is if we look at IOW as transactional delete+insert. Otherwise we are > just saying IOW performs "semi"-transactional delete. > If 1 ran an update on rows instead of an insert, and 2 still ran an > IOW/delete, row lock conflict (or equivalent) should cause one of them to > fail. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18288) merge/concat not supported on Acid table
[
https://issues.apache.org/jira/browse/HIVE-18288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16464343#comment-16464343
]
Sergey Shelukhin commented on HIVE-18288:
-
Does this basically trigger a major compaction? +1 pending tests
> merge/concat not supported on Acid table
>
>
> Key: HIVE-18288
> URL: https://issues.apache.org/jira/browse/HIVE-18288
> Project: Hive
> Issue Type: Sub-task
> Components: Transactions
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
>Priority: Major
> Attachments: HIVE-18288.01.patch, HIVE-18288.02.patch
>
>
> For example, mvn test -Dtest=TestCliDriver -Dqfile=orc_merge10.q
> now ends up with
> {noformat}
> 2017-12-15T15:12:30,753 ERROR [7c3ff5b2-285c-44f2-8b13-5c3ccbd41b13 main]
> ql.Driver: FAILED: SemanticException
> org.apache.hadoop.hive.ql.parse.SemanticException: Concatenate/M\
> erge can not be performed on transactional tables
> org.apache.hadoop.hive.ql.parse.SemanticException:
> org.apache.hadoop.hive.ql.parse.SemanticException: Concatenate/Merge can not
> be performed on transactional tables
> at
> org.apache.hadoop.hive.ql.parse.DDLSemanticAnalyzer.analyzeAlterTablePartMergeFiles(DDLSemanticAnalyzer.java:2172)
> at
> org.apache.hadoop.hive.ql.parse.DDLSemanticAnalyzer.analyzeInternal(DDLSemanticAnalyzer.java:343)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18288) merge/concat not supported on Acid table
[
https://issues.apache.org/jira/browse/HIVE-18288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16464350#comment-16464350
]
Eugene Koifman commented on HIVE-18288:
---
yes
> merge/concat not supported on Acid table
>
>
> Key: HIVE-18288
> URL: https://issues.apache.org/jira/browse/HIVE-18288
> Project: Hive
> Issue Type: Sub-task
> Components: Transactions
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
>Priority: Major
> Attachments: HIVE-18288.01.patch, HIVE-18288.02.patch
>
>
> For example, mvn test -Dtest=TestCliDriver -Dqfile=orc_merge10.q
> now ends up with
> {noformat}
> 2017-12-15T15:12:30,753 ERROR [7c3ff5b2-285c-44f2-8b13-5c3ccbd41b13 main]
> ql.Driver: FAILED: SemanticException
> org.apache.hadoop.hive.ql.parse.SemanticException: Concatenate/M\
> erge can not be performed on transactional tables
> org.apache.hadoop.hive.ql.parse.SemanticException:
> org.apache.hadoop.hive.ql.parse.SemanticException: Concatenate/Merge can not
> be performed on transactional tables
> at
> org.apache.hadoop.hive.ql.parse.DDLSemanticAnalyzer.analyzeAlterTablePartMergeFiles(DDLSemanticAnalyzer.java:2172)
> at
> org.apache.hadoop.hive.ql.parse.DDLSemanticAnalyzer.analyzeInternal(DDLSemanticAnalyzer.java:343)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19041) Thrift deserialization of Partition objects should intern fields
[
https://issues.apache.org/jira/browse/HIVE-19041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16464356#comment-16464356
]
Hive QA commented on HIVE-19041:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
1s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
54s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
49s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
19s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
12s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
8s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 15m 10s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-10694/dev-support/hive-personality.sh
|
| git revision | master / 552dbaf |
| Default Java | 1.8.0_111 |
| modules | C: standalone-metastore U: standalone-metastore |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-10694/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Thrift deserialization of Partition objects should intern fields
>
>
> Key: HIVE-19041
> URL: https://issues.apache.org/jira/browse/HIVE-19041
> Project: Hive
> Issue Type: Improvement
> Components: Metastore
>Affects Versions: 3.0.0, 2.3.2
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
>Priority: Major
> Attachments: HIVE-19041.01.patch, HIVE-19041.02.patch
>
>
> When a client is creating large number of partitions, the thrift objects are
> deserialized into Partition objects. The read method of these objects does
> not intern the inputformat, location, outputformat which cause large number
> of duplicate Strings in the HMS memory. We should intern these objects while
> deserialization to reduce memory pressure.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18570) ACID IOW implemented using base may delete too much data
[ https://issues.apache.org/jira/browse/HIVE-18570?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman updated HIVE-18570: -- Resolution: Fixed Release Note: Insert Overwrite commands even on transactional tables will acquire Exclusive locks to ensure correctness. This will be improved upon to allow greater concurrency. Status: Resolved (was: Patch Available) committed to branch-3/master thanks Sergey for the review > ACID IOW implemented using base may delete too much data > > > Key: HIVE-18570 > URL: https://issues.apache.org/jira/browse/HIVE-18570 > Project: Hive > Issue Type: Bug > Components: Transactions >Reporter: Sergey Shelukhin >Assignee: Eugene Koifman >Priority: Blocker > Attachments: HIVE-18570.01-branch-3.patch, HIVE-18570.01.patch, > HIVE-18570.02-branch-3.patch, HIVE-18570.02.patch, > HIVE-18570.03-branch-3.patch, HIVE-18570.03.patch, > HIVE-18570.04-branch-3.patch, HIVE-18570.05-branch-3.patch > > > Suppose we have a table with delta_0 insert data. > Txn 1 starts an insert into delta_1. > Txn 2 starts an IOW into base_2. > Txn 2 commits. > Txn 1 commits after txn 2 but its results would be invisible. > Txn 2 deletes rows committed by txn 1 that according to standard ACID > semantics it could have never observed and affected; this sequence of events > is only possible under read-uncommitted isolation level (so, 2 deletes rows > written by 1 before 1 commits them). > This is if we look at IOW as transactional delete+insert. Otherwise we are > just saying IOW performs "semi"-transactional delete. > If 1 ran an update on rows instead of an insert, and 2 still ran an > IOW/delete, row lock conflict (or equivalent) should cause one of them to > fail. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-17824) msck repair table should drop the missing partitions from metastore
[
https://issues.apache.org/jira/browse/HIVE-17824?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Janaki Lahorani updated HIVE-17824:
---
Attachment: (was: HIVE-17824.01-branch-2.patch)
> msck repair table should drop the missing partitions from metastore
> ---
>
> Key: HIVE-17824
> URL: https://issues.apache.org/jira/browse/HIVE-17824
> Project: Hive
> Issue Type: Improvement
>Reporter: Vihang Karajgaonkar
>Assignee: Janaki Lahorani
>Priority: Major
> Fix For: 3.0.0, 2.4.0, 3.1.0
>
> Attachments: HIVE-17824-branch-2.01.patch,
> HIVE-17824.01-branch-2.patch, HIVE-17824.1.patch, HIVE-17824.2.patch,
> HIVE-17824.3.patch, HIVE-17824.4.patch
>
>
> {{msck repair table }} is often used in environments where the new
> partitions are loaded as directories on HDFS or S3 and users want to create
> the missing partitions in bulk. However, currently it only supports addition
> of missing partitions. If there are any partitions which are present in
> metastore but not on the FileSystem, it should also delete them so that it
> truly repairs the table metadata.
> We should be careful not to break backwards compatibility so we should either
> introduce a new config or keyword to add support to delete unnecessary
> partitions from the metastore. This way users who want the old behavior can
> easily turn it off.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-17824) msck repair table should drop the missing partitions from metastore
[
https://issues.apache.org/jira/browse/HIVE-17824?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Janaki Lahorani updated HIVE-17824:
---
Attachment: (was: HIVE-17824.01-branch-2.patch)
> msck repair table should drop the missing partitions from metastore
> ---
>
> Key: HIVE-17824
> URL: https://issues.apache.org/jira/browse/HIVE-17824
> Project: Hive
> Issue Type: Improvement
>Reporter: Vihang Karajgaonkar
>Assignee: Janaki Lahorani
>Priority: Major
> Fix For: 3.0.0, 2.4.0, 3.1.0
>
> Attachments: HIVE-17824-branch-2.01.patch,
> HIVE-17824.01-branch-2.patch, HIVE-17824.1.patch, HIVE-17824.2.patch,
> HIVE-17824.3.patch, HIVE-17824.4.patch
>
>
> {{msck repair table }} is often used in environments where the new
> partitions are loaded as directories on HDFS or S3 and users want to create
> the missing partitions in bulk. However, currently it only supports addition
> of missing partitions. If there are any partitions which are present in
> metastore but not on the FileSystem, it should also delete them so that it
> truly repairs the table metadata.
> We should be careful not to break backwards compatibility so we should either
> introduce a new config or keyword to add support to delete unnecessary
> partitions from the metastore. This way users who want the old behavior can
> easily turn it off.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19396) HiveOperation is incorrectly set for analyze statement
[ https://issues.apache.org/jira/browse/HIVE-19396?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-19396: Fix Version/s: (was: 3.1.0) 3.0.0 > HiveOperation is incorrectly set for analyze statement > -- > > Key: HIVE-19396 > URL: https://issues.apache.org/jira/browse/HIVE-19396 > Project: Hive > Issue Type: Bug > Components: Query Planning >Reporter: Ashutosh Chauhan >Assignee: Ashutosh Chauhan >Priority: Major > Fix For: 3.0.0 > > Attachments: HIVE-19396.patch > > > Because we rewrite analyze to select compute_stats() operation enum gets set > to Query. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19041) Thrift deserialization of Partition objects should intern fields
[
https://issues.apache.org/jira/browse/HIVE-19041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16464418#comment-16464418
]
Hive QA commented on HIVE-19041:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12921874/HIVE-19041.02.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 35 failed/errored test(s), 14316 tests
executed
*Failed tests:*
{noformat}
TestDbNotificationListener - did not produce a TEST-*.xml file (likely timed
out) (batchId=247)
TestHCatHiveCompatibility - did not produce a TEST-*.xml file (likely timed
out) (batchId=247)
TestNonCatCallsWithCatalog - did not produce a TEST-*.xml file (likely timed
out) (batchId=217)
TestSequenceFileReadWrite - did not produce a TEST-*.xml file (likely timed
out) (batchId=247)
TestTxnExIm - did not produce a TEST-*.xml file (likely timed out) (batchId=286)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucket_map_join_tez1]
(batchId=175)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucket_map_join_tez2]
(batchId=156)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[materialized_view_create_rewrite_5]
(batchId=154)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[materialized_view_create_rewrite_rebuild_dummy]
(batchId=161)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[materialized_view_create_rewrite_time_window]
(batchId=156)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb]
(batchId=163)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[tez_dynpart_hashjoin_1]
(batchId=174)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[union_fast_stats]
(batchId=167)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[union_stats]
(batchId=159)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_5]
(batchId=105)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[udf_reflect_neg]
(batchId=96)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[udf_test_error]
(batchId=96)
org.apache.hadoop.hive.ql.TestAcidOnTez.testCtasTezUnion (batchId=228)
org.apache.hadoop.hive.ql.TestAcidOnTez.testNonStandardConversion01
(batchId=228)
org.apache.hadoop.hive.ql.TestMTQueries.testMTQueries1 (batchId=232)
org.apache.hadoop.hive.ql.parse.TestCopyUtils.testPrivilegedDistCpWithSameUserAsCurrentDoesNotTryToImpersonate
(batchId=231)
org.apache.hadoop.hive.ql.parse.TestReplicationOnHDFSEncryptedZones.targetAndSourceHaveDifferentEncryptionZoneKeys
(batchId=231)
org.apache.hive.beeline.TestBeeLineWithArgs.testQueryProgress (batchId=235)
org.apache.hive.jdbc.TestSSL.testSSLFetchHttp (batchId=239)
org.apache.hive.jdbc.TestTriggersMoveWorkloadManager.testTriggerMoveBackKill
(batchId=241)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testMultipleTriggers2
(batchId=241)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerCustomCreatedFiles
(batchId=241)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerCustomNonExistent
(batchId=241)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerCustomReadOps
(batchId=241)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerHighBytesRead
(batchId=241)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerHighBytesWrite
(batchId=241)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerHighShuffleBytes
(batchId=241)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerSlowQueryElapsedTime
(batchId=241)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerSlowQueryExecutionTime
(batchId=241)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerVertexRawInputSplitsNoKill
(batchId=241)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/10694/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/10694/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-10694/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 35 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12921874 - PreCommit-HIVE-Build
> Thrift deserialization of Partition objects should intern fields
>
>
> Key: HIVE-19041
> URL: https://issues.apache.org/jira/browse/HIVE-19041
> Project: Hive
> Issue Type: Improvement
> Components: Metastore
>Affects Versions: 3.0.0, 2.
[jira] [Updated] (HIVE-17838) Make org.apache.hive.spark.client.rpc logging HoS specific and other logging cleanup
[
https://issues.apache.org/jira/browse/HIVE-17838?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sahil Takiar updated HIVE-17838:
Resolution: Fixed
Fix Version/s: 3.1.0
Status: Resolved (was: Patch Available)
Pushed to master. Thanks Aihua for the review!
> Make org.apache.hive.spark.client.rpc logging HoS specific and other logging
> cleanup
>
>
> Key: HIVE-17838
> URL: https://issues.apache.org/jira/browse/HIVE-17838
> Project: Hive
> Issue Type: Sub-task
> Components: Hive
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
>Priority: Major
> Fix For: 3.1.0
>
> Attachments: HIVE-17838.1.patch, HIVE-17838.2.patch,
> HIVE-17838.4.patch
>
>
> A lot of the logging in {{org.apache.hive.spark.client.rpc}} emits
> information that a generic RPC library would, but this package is only used
> by HoS. We should make the logging more HoS specific so users don't get
> confused as to what this RPC library is doing. Ideally, they aren't even
> aware that a RPC library is in place, it should be more transparent to the
> user.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18378) Explain plan should show if a Map/Reduce Work is being cached
[
https://issues.apache.org/jira/browse/HIVE-18378?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16464430#comment-16464430
]
Sahil Takiar commented on HIVE-18378:
-
Hey [~kellyzly] are you still working on this?
> Explain plan should show if a Map/Reduce Work is being cached
> -
>
> Key: HIVE-18378
> URL: https://issues.apache.org/jira/browse/HIVE-18378
> Project: Hive
> Issue Type: Sub-task
> Components: Spark
>Reporter: Sahil Takiar
>Assignee: liyunzhang
>Priority: Major
> Attachments: HIVE-18378.1.patch
>
>
> It would be nice if the explain plan showed what {{MapWork}} / {{ReduceWork}}
> objects are being cached by Spark.
> The {{CombineEquivalentWorkResolver}} is the only code that triggers Spark
> cache-ing, so we should be able to modify it so that it displays if a work
> object will be cached or not.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
