[jira] [Work logged] (HIVE-26716) Query based Rebalance compaction on full acid tables
[ https://issues.apache.org/jira/browse/HIVE-26716?focusedWorklogId=836840&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-836840 ] ASF GitHub Bot logged work on HIVE-26716: - Author: ASF GitHub Bot Created on: 04/Jan/23 09:04 Start Date: 04/Jan/23 09:04 Worklog Time Spent: 10m Work Description: deniskuzZ merged PR #3904: URL: https://github.com/apache/hive/pull/3904 Issue Time Tracking --- Worklog Id: (was: 836840) Time Spent: 18h 20m (was: 18h 10m) > Query based Rebalance compaction on full acid tables > > > Key: HIVE-26716 > URL: https://issues.apache.org/jira/browse/HIVE-26716 > Project: Hive > Issue Type: Sub-task > Components: Hive >Reporter: László Végh >Assignee: László Végh >Priority: Major > Labels: ACID, compaction, pull-request-available > Time Spent: 18h 20m > Remaining Estimate: 0h > > Support rebalancing compaction on fully ACID tables. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-26869) Backport of HIVE-25250 Branch-3: Fix TestHS2ImpersonationWithRemoteMS.testImpersonation
[ https://issues.apache.org/jira/browse/HIVE-26869?focusedWorklogId=836844&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-836844 ] ASF GitHub Bot logged work on HIVE-26869: - Author: ASF GitHub Bot Created on: 04/Jan/23 09:22 Start Date: 04/Jan/23 09:22 Worklog Time Spent: 10m Work Description: amanraj2520 closed pull request #3873: HIVE-26869 : Backport of HIVE-25250: Fix TestHS2ImpersonationWithRemoteMS.testImpersonation URL: https://github.com/apache/hive/pull/3873 Issue Time Tracking --- Worklog Id: (was: 836844) Time Spent: 2h 10m (was: 2h) > Backport of HIVE-25250 Branch-3: Fix > TestHS2ImpersonationWithRemoteMS.testImpersonation > --- > > Key: HIVE-26869 > URL: https://issues.apache.org/jira/browse/HIVE-26869 > Project: Hive > Issue Type: Sub-task >Reporter: Aman Raj >Assignee: Aman Raj >Priority: Critical > Labels: pull-request-available > Time Spent: 2h 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-26869) Backport of HIVE-25250 Branch-3: Fix TestHS2ImpersonationWithRemoteMS.testImpersonation
[ https://issues.apache.org/jira/browse/HIVE-26869?focusedWorklogId=836845&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-836845 ] ASF GitHub Bot logged work on HIVE-26869: - Author: ASF GitHub Bot Created on: 04/Jan/23 09:25 Start Date: 04/Jan/23 09:25 Worklog Time Spent: 10m Work Description: amanraj2520 opened a new pull request, #3873: URL: https://github.com/apache/hive/pull/3873 HIVE-26869 : Backport of HIVE-25250: Fix TestHS2ImpersonationWithRemoteMS.testImpersonation(Ashish Sharma, reviewed by Zoltan Haindrich) Signed-off-by: Sankar Hariappan (cherry picked from commit dc8891ec9459d2eff5a23154383ec3bd19481fd2) ### What changes were proposed in this pull request? ### Why are the changes needed? ### Does this PR introduce _any_ user-facing change? ### How was this patch tested? Issue Time Tracking --- Worklog Id: (was: 836845) Time Spent: 2h 20m (was: 2h 10m) > Backport of HIVE-25250 Branch-3: Fix > TestHS2ImpersonationWithRemoteMS.testImpersonation > --- > > Key: HIVE-26869 > URL: https://issues.apache.org/jira/browse/HIVE-26869 > Project: Hive > Issue Type: Sub-task >Reporter: Aman Raj >Assignee: Aman Raj >Priority: Critical > Labels: pull-request-available > Time Spent: 2h 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-26869) Backport of HIVE-25250 Branch-3: Fix TestHS2ImpersonationWithRemoteMS.testImpersonation
[ https://issues.apache.org/jira/browse/HIVE-26869?focusedWorklogId=836846&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-836846 ] ASF GitHub Bot logged work on HIVE-26869: - Author: ASF GitHub Bot Created on: 04/Jan/23 09:26 Start Date: 04/Jan/23 09:26 Worklog Time Spent: 10m Work Description: amanraj2520 commented on PR #3873: URL: https://github.com/apache/hive/pull/3873#issuecomment-1370678673 @cnauroth Cherry picked the patch and tested locally. It works as expected. Thanks for your suggestion updated this PR. Issue Time Tracking --- Worklog Id: (was: 836846) Time Spent: 2.5h (was: 2h 20m) > Backport of HIVE-25250 Branch-3: Fix > TestHS2ImpersonationWithRemoteMS.testImpersonation > --- > > Key: HIVE-26869 > URL: https://issues.apache.org/jira/browse/HIVE-26869 > Project: Hive > Issue Type: Sub-task >Reporter: Aman Raj >Assignee: Aman Raj >Priority: Critical > Labels: pull-request-available > Time Spent: 2.5h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HIVE-26869) Backport of HIVE-19104: When test MetaStore is started with retry the instances should be independent
[ https://issues.apache.org/jira/browse/HIVE-26869?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aman Raj updated HIVE-26869: Summary: Backport of HIVE-19104: When test MetaStore is started with retry the instances should be independent (was: Backport of HIVE-25250 Branch-3: Fix TestHS2ImpersonationWithRemoteMS.testImpersonation) > Backport of HIVE-19104: When test MetaStore is started with retry the > instances should be independent > - > > Key: HIVE-26869 > URL: https://issues.apache.org/jira/browse/HIVE-26869 > Project: Hive > Issue Type: Sub-task >Reporter: Aman Raj >Assignee: Aman Raj >Priority: Critical > Labels: pull-request-available > Time Spent: 2.5h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HIVE-26869) Backport of HIVE-19104: When test MetaStore is started with retry the instances should be independent
[
https://issues.apache.org/jira/browse/HIVE-26869?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aman Raj updated HIVE-26869:
Description:
This fixes TestHS2ImpersonationWithRemoteMS which was failing with the
following error :
{code:java}
[ERROR] Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 703.357
s <<< FAILURE! - in org.apache.hive.service.TestHS2ImpersonationWithRemoteMS
[ERROR]
testImpersonation(org.apache.hive.service.TestHS2ImpersonationWithRemoteMS)
Time elapsed: 668.923 s <<< FAILURE!
java.lang.AssertionError: Unexpected table directory '34015' in warehouse
at org.junit.Assert.fail(Assert.java:88)
at
org.apache.hive.service.TestHS2ImpersonationWithRemoteMS.testImpersonation(TestHS2ImpersonationWithRemoteMS.java:115)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43){code}
was:
This fixes TestHS2ImpersonationWithRemoteMS which was failing with the
following error :
{code:java}
{code}
> Backport of HIVE-19104: When test MetaStore is started with retry the
> instances should be independent
> -
>
> Key: HIVE-26869
> URL: https://issues.apache.org/jira/browse/HIVE-26869
> Project: Hive
> Issue Type: Sub-task
>Reporter: Aman Raj
>Assignee: Aman Raj
>Priority: Critical
> Labels: pull-request-available
> Time Spent: 2.5h
> Remaining Estimate: 0h
>
> This fixes TestHS2ImpersonationWithRemoteMS which was failing with the
> following error :
> {code:java}
> [ERROR] Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed:
> 703.357 s <<< FAILURE! - in
> org.apache.hive.service.TestHS2ImpersonationWithRemoteMS
> [ERROR]
> testImpersonation(org.apache.hive.service.TestHS2ImpersonationWithRemoteMS)
> Time elapsed: 668.923 s <<< FAILURE!
> java.lang.AssertionError: Unexpected table directory '34015' in warehouse
> at org.junit.Assert.fail(Assert.java:88)
> at
> org.apache.hive.service.TestHS2ImpersonationWithRemoteMS.testImpersonation(TestHS2ImpersonationWithRemoteMS.java:115)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43){code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (HIVE-26869) Backport of HIVE-19104: When test MetaStore is started with retry the instances should be independent
[
https://issues.apache.org/jira/browse/HIVE-26869?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aman Raj updated HIVE-26869:
Description:
This fixes TestHS2ImpersonationWithRemoteMS which was failing with the
following error :
{code:java}
{code}
> Backport of HIVE-19104: When test MetaStore is started with retry the
> instances should be independent
> -
>
> Key: HIVE-26869
> URL: https://issues.apache.org/jira/browse/HIVE-26869
> Project: Hive
> Issue Type: Sub-task
>Reporter: Aman Raj
>Assignee: Aman Raj
>Priority: Critical
> Labels: pull-request-available
> Time Spent: 2.5h
> Remaining Estimate: 0h
>
> This fixes TestHS2ImpersonationWithRemoteMS which was failing with the
> following error :
> {code:java}
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-26894) After using scratchdir for staging final job, CTAS and IOW on ACID tables are failing with wrongFS exception
[ https://issues.apache.org/jira/browse/HIVE-26894?focusedWorklogId=836856&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-836856 ] ASF GitHub Bot logged work on HIVE-26894: - Author: ASF GitHub Bot Created on: 04/Jan/23 10:17 Start Date: 04/Jan/23 10:17 Worklog Time Spent: 10m Work Description: warriersruthi closed pull request #3906: HIVE-26894: After using scratchdir for staging final job, CTAS and IOW on ACID tables are failing with wrongFS exception URL: https://github.com/apache/hive/pull/3906 Issue Time Tracking --- Worklog Id: (was: 836856) Time Spent: 0.5h (was: 20m) > After using scratchdir for staging final job, CTAS and IOW on ACID tables are > failing with wrongFS exception > > > Key: HIVE-26894 > URL: https://issues.apache.org/jira/browse/HIVE-26894 > Project: Hive > Issue Type: Sub-task > Components: Hive >Affects Versions: 3.1.3 >Reporter: Sruthi M >Assignee: Sruthi M >Priority: Major > Labels: pull-request-available > Time Spent: 0.5h > Remaining Estimate: 0h > > ERROR : Failed with exception Wrong FS: > abfs:///hive/warehouse/managed/tpcds_orc.db/test_sales/delta_001_001_, > expected: hdfs://mycluster -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-26894) After using scratchdir for staging final job, CTAS and IOW on ACID tables are failing with wrongFS exception
[ https://issues.apache.org/jira/browse/HIVE-26894?focusedWorklogId=836857&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-836857 ] ASF GitHub Bot logged work on HIVE-26894: - Author: ASF GitHub Bot Created on: 04/Jan/23 10:20 Start Date: 04/Jan/23 10:20 Worklog Time Spent: 10m Work Description: warriersruthi opened a new pull request, #3909: URL: https://github.com/apache/hive/pull/3909 ###What changes were proposed in this pull request? moveAcidFiles() method in Hive.java is modified to cater to the case where the src FS and dest FS are different. Instead of rename() operation, in the above case, we need to do copy(). ###Why are the changes needed? With the Jira: HIVE-26815, we can change the hive staging directory to be the same as that of scratchdir. In that case, staging files will be in HDFS while the target location would be blob or some other FS. In such scenarios, while creating and updating ACID tables, the final rename operation was not working fine (throws Wrong FS error) since the src and dest FS are different. This is fixed in this patch by changing the rename() operation to copy() if FS are in different encryption zones. ###Does this PR introduce any user-facing change? Nope. ###How was this patch tested? Tested with patch on hive 3.1.2 Issue Time Tracking --- Worklog Id: (was: 836857) Time Spent: 40m (was: 0.5h) > After using scratchdir for staging final job, CTAS and IOW on ACID tables are > failing with wrongFS exception > > > Key: HIVE-26894 > URL: https://issues.apache.org/jira/browse/HIVE-26894 > Project: Hive > Issue Type: Sub-task > Components: Hive >Affects Versions: 3.1.3 >Reporter: Sruthi M >Assignee: Sruthi M >Priority: Major > Labels: pull-request-available > Time Spent: 40m > Remaining Estimate: 0h > > ERROR : Failed with exception Wrong FS: > abfs:///hive/warehouse/managed/tpcds_orc.db/test_sales/delta_001_001_, > expected: hdfs://mycluster -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-26711) The very first REPL Load should make the Target Database read-only
[ https://issues.apache.org/jira/browse/HIVE-26711?focusedWorklogId=836862&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-836862 ] ASF GitHub Bot logged work on HIVE-26711: - Author: ASF GitHub Bot Created on: 04/Jan/23 10:44 Start Date: 04/Jan/23 10:44 Worklog Time Spent: 10m Work Description: sonarcloud[bot] commented on PR #3736: URL: https://github.com/apache/hive/pull/3736#issuecomment-1370762915 Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=3736) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3736&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3736&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3736&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3736&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3736&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3736&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=3736&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=3736&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=3736&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3736&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3736&resolved=false&types=CODE_SMELL) [87 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3736&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=3736&metric=coverage&view=list) No Coverage information [](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=3736&metric=duplicated_lines_density&view=list) No Duplication information Issue Time Tracking --- Worklog Id: (was: 836862) Time Spent: 20m (was: 10m) > The very first REPL Load should make the Target Database read-only > -- > > Key: HIVE-26711 > URL: https://issues.apache.org/jira/browse/HIVE-26711 > Project: Hive > Issue Type: Task >Reporter: Shreenidhi >Assignee: Shreenidhi >Priority: Major > Labels: pull-request-available > Time Spent: 20m > Remaining Estimate: 0h > > Use EnforceReadOnly hook to set TARGET database read only during BootStrap > load. > Also ensure backward compatibility. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-26808) Port Iceberg catalog changes
[ https://issues.apache.org/jira/browse/HIVE-26808?focusedWorklogId=836866&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-836866 ] ASF GitHub Bot logged work on HIVE-26808: - Author: ASF GitHub Bot Created on: 04/Jan/23 10:53 Start Date: 04/Jan/23 10:53 Worklog Time Spent: 10m Work Description: sonarcloud[bot] commented on PR #3907: URL: https://github.com/apache/hive/pull/3907#issuecomment-1370772531 Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=3907) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3907&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3907&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3907&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3907&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3907&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3907&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=3907&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=3907&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=3907&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3907&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3907&resolved=false&types=CODE_SMELL) [87 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3907&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=3907&metric=coverage&view=list) No Coverage information [](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=3907&metric=duplicated_lines_density&view=list) No Duplication information Issue Time Tracking --- Worklog Id: (was: 836866) Time Spent: 0.5h (was: 20m) > Port Iceberg catalog changes > > > Key: HIVE-26808 > URL: https://issues.apache.org/jira/browse/HIVE-26808 > Project: Hive > Issue Type: Improvement > Components: Iceberg integration >Reporter: Zsolt Miskolczi >Assignee: Zsolt Miskolczi >Priority: Major > Labels: pull-request-available > Time Spent: 0.5h > Remaining Estimate: 0h > > The last round of porting happened in 2022 april, there were a couple of > changes especially in HiveTableOperations worth porting into iceberg-catalog. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (HIVE-26904) QueryCompactor failed in commitCompaction if the tmp table dir is already removed
[
https://issues.apache.org/jira/browse/HIVE-26904?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Quanlong Huang reassigned HIVE-26904:
-
> QueryCompactor failed in commitCompaction if the tmp table dir is already
> removed
> --
>
> Key: HIVE-26904
> URL: https://issues.apache.org/jira/browse/HIVE-26904
> Project: Hive
> Issue Type: Bug
>Reporter: Quanlong Huang
>Assignee: Quanlong Huang
>Priority: Major
>
> commitCompaction() of query-based compactions just remove the dirs of tmp
> tables. It should not fail the compaction if the dirs are already removed.
> We've seen such a failure in Impala's test (IMPALA-11756):
> {noformat}
> 2023-01-02T02:09:26,306 INFO [HiveServer2-Background-Pool: Thread-695]
> ql.Driver: Executing
> command(queryId=jenkins_20230102020926_69112755-b783-4214-89e5-1c7111dfe15f):
> alter table partial_catalog_info_test.insert_only_partitioned partition
> (part=1) compact 'minor' and wait
> 2023-01-02T02:09:26,306 INFO [HiveServer2-Background-Pool: Thread-695]
> ql.Driver: Starting task [Stage-0:DDL] in serial mode
> 2023-01-02T02:09:26,317 INFO [HiveServer2-Background-Pool: Thread-695]
> exec.Task: Compaction enqueued with id 15
> ...
> 2023-01-02T02:12:55,849 ERROR
> [impala-ec2-centos79-m6i-4xlarge-ondemand-1428.vpc.cloudera.com-48_executor]
> compactor.Worker: Caught exception while trying to compact
> id:15,dbname:partial_catalog_info_test,tableName:insert_only_partitioned,partName:part=1,state:^@,type:MINOR,enqueueTime:0,start:0,properties:null,runAs:jenkins,tooManyAborts:false,hasOldAbort:false,highestWriteId:3,errorMessage:null,workerId:
> null,initiatorId: null,retryRetention0. Marking failed to avoid repeated
> failures
> java.io.FileNotFoundException: File
> hdfs://localhost:20500/tmp/hive/jenkins/092b533a-81c8-4b95-88e4-9472cf6f365d/_tmp_space.db/62ec04fb-e2d2-4a99-a454-ae709a3cccfe
> does not exist.
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$DirListingIterator.(DistributedFileSystem.java:1275)
> ~[hadoop-hdfs-client-3.1.1.7.2.15.4-6.jar:?]
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$DirListingIterator.(DistributedFileSystem.java:1249)
> ~[hadoop-hdfs-client-3.1.1.7.2.15.4-6.jar:?]
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$25.doCall(DistributedFileSystem.java:1194)
> ~[hadoop-hdfs-client-3.1.1.7.2.15.4-6.jar:?]
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$25.doCall(DistributedFileSystem.java:1190)
> ~[hadoop-hdfs-client-3.1.1.7.2.15.4-6.jar:?]
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> ~[hadoop-common-3.1.1.7.2.15.4-6.jar:?]
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.listLocatedStatus(DistributedFileSystem.java:1208)
> ~[hadoop-hdfs-client-3.1.1.7.2.15.4-6.jar:?]
> at
> org.apache.hadoop.fs.FileSystem.listLocatedStatus(FileSystem.java:2144)
> ~[hadoop-common-3.1.1.7.2.15.4-6.jar:?]
> at org.apache.hadoop.fs.FileSystem$5.(FileSystem.java:2302)
> ~[hadoop-common-3.1.1.7.2.15.4-6.jar:?]
> at org.apache.hadoop.fs.FileSystem.listFiles(FileSystem.java:2299)
> ~[hadoop-common-3.1.1.7.2.15.4-6.jar:?]
> at
> org.apache.hadoop.hive.ql.txn.compactor.QueryCompactor$Util.cleanupEmptyDir(QueryCompactor.java:261)
> ~[hive-exec-3.1.3000.2022.0.13.0-60.jar:3.1.3000.2022.0.13.0-60]
> at
> org.apache.hadoop.hive.ql.txn.compactor.MmMinorQueryCompactor.commitCompaction(MmMinorQueryCompactor.java:72)
> ~[hive-exec-3.1.3000.2022.0.13.0-60.jar:3.1.3000.2022.0.13.0-60]
> at
> org.apache.hadoop.hive.ql.txn.compactor.QueryCompactor.runCompactionQueries(QueryCompactor.java:146)
> ~[hive-exec-3.1.3000.2022.0.13.0-60.jar:3.1.3000.2022.0.13.0-60]
> at
> org.apache.hadoop.hive.ql.txn.compactor.MmMinorQueryCompactor.runCompaction(MmMinorQueryCompactor.java:63)
> ~[hive-exec-3.1.3000.2022.0.13.0-60.jar:3.1.3000.2022.0.13.0-60]
> at
> org.apache.hadoop.hive.ql.txn.compactor.Worker.findNextCompactionAndExecute(Worker.java:435)
> ~[hive-exec-3.1.3000.2022.0.13.0-60.jar:3.1.3000.2022.0.13.0-60]
> at
> org.apache.hadoop.hive.ql.txn.compactor.Worker.lambda$run$0(Worker.java:115)
> ~[hive-exec-3.1.3000.2022.0.13.0-60.jar:3.1.3000.2022.0.13.0-60]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> ~[?:1.8.0_261]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> ~[?:1.8.0_261]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> ~[?:1.8.0_261]
> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_261]
> 2023-01-02T02:12:55,858 INFO
> [impala-ec2-centos79-m6i-4xlarge-ondemand-1428.vpc.cloudera.com-48_execut
[jira] [Work logged] (HIVE-11495) Add aborted reason to transaction information.
[
https://issues.apache.org/jira/browse/HIVE-11495?focusedWorklogId=836869&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-836869
]
ASF GitHub Bot logged work on HIVE-11495:
-
Author: ASF GitHub Bot
Created on: 04/Jan/23 11:38
Start Date: 04/Jan/23 11:38
Worklog Time Spent: 10m
Work Description: veghlaci05 commented on code in PR #3908:
URL: https://github.com/apache/hive/pull/3908#discussion_r1061364735
##
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnErrorMsg.java:
##
@@ -0,0 +1,83 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.metastore.txn;
+
+/**
+ * The following class represents all the error messages that are handled for
aborts.
+ */
+public enum TxnErrorMsg {
+// 5 to 5: Txn Errors Codes.
+// Query runtime aborts
+NONE(5, "None"),
+ABORT_QUERY(50001, " Txn aborted by Abort Query Command"),
+ABORT_CONCURRENT(50002, " Txn aborted due to concurrent committed
transaction"),
+ABORT_WRITE_CONFLICT(50003, " Txn aborted due to write conflicts"),
+ABORT_TIMEOUT(50004, " Txn aborted due to heartbeat time-out"),
+ABORT_ROLLBACK(50005, "Txn aborted due to rollback"),
+ABORT_COMPACTION_TXN(50006, "Compaction txn is aborted"),
+ABORT_MSCK_TXN(50007, "Msck txn is aborted"),
+ABORT_MIGRATION_TXN(50008, "Managed Migration transaction is aborted"),
+
+// Replication related aborts
+ABORT_DEFAULT_REPL_TXN(50009, " Replication:" +
Review Comment:
You should reserve digits for error grouping. For example 5-50999 can be
reserved for runtime errors, and 51000-51999 can be reserved for replication
errors, and so on. As a result you can add new codes later in the same group,
and even users/code can categorize the errors by their code.
##
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnErrorMsg.java:
##
@@ -0,0 +1,83 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.metastore.txn;
+
+/**
+ * The following class represents all the error messages that are handled for
aborts.
+ */
+public enum TxnErrorMsg {
Review Comment:
Some messages are not real reasons, rather indicate only the type of the
aborted txn. Please try to add a real reason as well, whenever possible.
##
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnErrorMsg.java:
##
@@ -0,0 +1,83 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations un
[jira] [Updated] (HIVE-26904) QueryCompactor failed in commitCompaction if the tmp table dir is already removed
[
https://issues.apache.org/jira/browse/HIVE-26904?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated HIVE-26904:
--
Labels: pull-request-available (was: )
> QueryCompactor failed in commitCompaction if the tmp table dir is already
> removed
> --
>
> Key: HIVE-26904
> URL: https://issues.apache.org/jira/browse/HIVE-26904
> Project: Hive
> Issue Type: Bug
>Reporter: Quanlong Huang
>Assignee: Quanlong Huang
>Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> commitCompaction() of query-based compactions just remove the dirs of tmp
> tables. It should not fail the compaction if the dirs are already removed.
> We've seen such a failure in Impala's test (IMPALA-11756):
> {noformat}
> 2023-01-02T02:09:26,306 INFO [HiveServer2-Background-Pool: Thread-695]
> ql.Driver: Executing
> command(queryId=jenkins_20230102020926_69112755-b783-4214-89e5-1c7111dfe15f):
> alter table partial_catalog_info_test.insert_only_partitioned partition
> (part=1) compact 'minor' and wait
> 2023-01-02T02:09:26,306 INFO [HiveServer2-Background-Pool: Thread-695]
> ql.Driver: Starting task [Stage-0:DDL] in serial mode
> 2023-01-02T02:09:26,317 INFO [HiveServer2-Background-Pool: Thread-695]
> exec.Task: Compaction enqueued with id 15
> ...
> 2023-01-02T02:12:55,849 ERROR
> [impala-ec2-centos79-m6i-4xlarge-ondemand-1428.vpc.cloudera.com-48_executor]
> compactor.Worker: Caught exception while trying to compact
> id:15,dbname:partial_catalog_info_test,tableName:insert_only_partitioned,partName:part=1,state:^@,type:MINOR,enqueueTime:0,start:0,properties:null,runAs:jenkins,tooManyAborts:false,hasOldAbort:false,highestWriteId:3,errorMessage:null,workerId:
> null,initiatorId: null,retryRetention0. Marking failed to avoid repeated
> failures
> java.io.FileNotFoundException: File
> hdfs://localhost:20500/tmp/hive/jenkins/092b533a-81c8-4b95-88e4-9472cf6f365d/_tmp_space.db/62ec04fb-e2d2-4a99-a454-ae709a3cccfe
> does not exist.
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$DirListingIterator.(DistributedFileSystem.java:1275)
> ~[hadoop-hdfs-client-3.1.1.7.2.15.4-6.jar:?]
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$DirListingIterator.(DistributedFileSystem.java:1249)
> ~[hadoop-hdfs-client-3.1.1.7.2.15.4-6.jar:?]
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$25.doCall(DistributedFileSystem.java:1194)
> ~[hadoop-hdfs-client-3.1.1.7.2.15.4-6.jar:?]
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$25.doCall(DistributedFileSystem.java:1190)
> ~[hadoop-hdfs-client-3.1.1.7.2.15.4-6.jar:?]
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> ~[hadoop-common-3.1.1.7.2.15.4-6.jar:?]
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.listLocatedStatus(DistributedFileSystem.java:1208)
> ~[hadoop-hdfs-client-3.1.1.7.2.15.4-6.jar:?]
> at
> org.apache.hadoop.fs.FileSystem.listLocatedStatus(FileSystem.java:2144)
> ~[hadoop-common-3.1.1.7.2.15.4-6.jar:?]
> at org.apache.hadoop.fs.FileSystem$5.(FileSystem.java:2302)
> ~[hadoop-common-3.1.1.7.2.15.4-6.jar:?]
> at org.apache.hadoop.fs.FileSystem.listFiles(FileSystem.java:2299)
> ~[hadoop-common-3.1.1.7.2.15.4-6.jar:?]
> at
> org.apache.hadoop.hive.ql.txn.compactor.QueryCompactor$Util.cleanupEmptyDir(QueryCompactor.java:261)
> ~[hive-exec-3.1.3000.2022.0.13.0-60.jar:3.1.3000.2022.0.13.0-60]
> at
> org.apache.hadoop.hive.ql.txn.compactor.MmMinorQueryCompactor.commitCompaction(MmMinorQueryCompactor.java:72)
> ~[hive-exec-3.1.3000.2022.0.13.0-60.jar:3.1.3000.2022.0.13.0-60]
> at
> org.apache.hadoop.hive.ql.txn.compactor.QueryCompactor.runCompactionQueries(QueryCompactor.java:146)
> ~[hive-exec-3.1.3000.2022.0.13.0-60.jar:3.1.3000.2022.0.13.0-60]
> at
> org.apache.hadoop.hive.ql.txn.compactor.MmMinorQueryCompactor.runCompaction(MmMinorQueryCompactor.java:63)
> ~[hive-exec-3.1.3000.2022.0.13.0-60.jar:3.1.3000.2022.0.13.0-60]
> at
> org.apache.hadoop.hive.ql.txn.compactor.Worker.findNextCompactionAndExecute(Worker.java:435)
> ~[hive-exec-3.1.3000.2022.0.13.0-60.jar:3.1.3000.2022.0.13.0-60]
> at
> org.apache.hadoop.hive.ql.txn.compactor.Worker.lambda$run$0(Worker.java:115)
> ~[hive-exec-3.1.3000.2022.0.13.0-60.jar:3.1.3000.2022.0.13.0-60]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> ~[?:1.8.0_261]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> ~[?:1.8.0_261]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> ~[?:1.8.0_261]
> at java.lang.Thread.run(Thr
[jira] [Work logged] (HIVE-26904) QueryCompactor failed in commitCompaction if the tmp table dir is already removed
[
https://issues.apache.org/jira/browse/HIVE-26904?focusedWorklogId=836871&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-836871
]
ASF GitHub Bot logged work on HIVE-26904:
-
Author: ASF GitHub Bot
Created on: 04/Jan/23 11:53
Start Date: 04/Jan/23 11:53
Worklog Time Spent: 10m
Work Description: stiga-huang opened a new pull request, #3910:
URL: https://github.com/apache/hive/pull/3910
### What changes were proposed in this pull request?
commitCompaction of query-based compactors just do some cleanup on tmp table
dirs. It could fail if the tmp table dirs is already removed. See an example in
https://issues.apache.org/jira/browse/IMPALA-11756
It doesn't make sense to fail the compaction due to this.
This PR catches the FileNotFoundException in commitCompaction() when the tmp
table dir doesn't exist. Ignore and skip removing the dir, and don't fail the
compaction.
### Why are the changes needed?
The compaction is already finished so don't need to fail it if the tmp dirs
are already cleaned up.
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
Run existing tests.
Issue Time Tracking
---
Worklog Id: (was: 836871)
Remaining Estimate: 0h
Time Spent: 10m
> QueryCompactor failed in commitCompaction if the tmp table dir is already
> removed
> --
>
> Key: HIVE-26904
> URL: https://issues.apache.org/jira/browse/HIVE-26904
> Project: Hive
> Issue Type: Bug
>Reporter: Quanlong Huang
>Assignee: Quanlong Huang
>Priority: Major
> Time Spent: 10m
> Remaining Estimate: 0h
>
> commitCompaction() of query-based compactions just remove the dirs of tmp
> tables. It should not fail the compaction if the dirs are already removed.
> We've seen such a failure in Impala's test (IMPALA-11756):
> {noformat}
> 2023-01-02T02:09:26,306 INFO [HiveServer2-Background-Pool: Thread-695]
> ql.Driver: Executing
> command(queryId=jenkins_20230102020926_69112755-b783-4214-89e5-1c7111dfe15f):
> alter table partial_catalog_info_test.insert_only_partitioned partition
> (part=1) compact 'minor' and wait
> 2023-01-02T02:09:26,306 INFO [HiveServer2-Background-Pool: Thread-695]
> ql.Driver: Starting task [Stage-0:DDL] in serial mode
> 2023-01-02T02:09:26,317 INFO [HiveServer2-Background-Pool: Thread-695]
> exec.Task: Compaction enqueued with id 15
> ...
> 2023-01-02T02:12:55,849 ERROR
> [impala-ec2-centos79-m6i-4xlarge-ondemand-1428.vpc.cloudera.com-48_executor]
> compactor.Worker: Caught exception while trying to compact
> id:15,dbname:partial_catalog_info_test,tableName:insert_only_partitioned,partName:part=1,state:^@,type:MINOR,enqueueTime:0,start:0,properties:null,runAs:jenkins,tooManyAborts:false,hasOldAbort:false,highestWriteId:3,errorMessage:null,workerId:
> null,initiatorId: null,retryRetention0. Marking failed to avoid repeated
> failures
> java.io.FileNotFoundException: File
> hdfs://localhost:20500/tmp/hive/jenkins/092b533a-81c8-4b95-88e4-9472cf6f365d/_tmp_space.db/62ec04fb-e2d2-4a99-a454-ae709a3cccfe
> does not exist.
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$DirListingIterator.(DistributedFileSystem.java:1275)
> ~[hadoop-hdfs-client-3.1.1.7.2.15.4-6.jar:?]
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$DirListingIterator.(DistributedFileSystem.java:1249)
> ~[hadoop-hdfs-client-3.1.1.7.2.15.4-6.jar:?]
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$25.doCall(DistributedFileSystem.java:1194)
> ~[hadoop-hdfs-client-3.1.1.7.2.15.4-6.jar:?]
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$25.doCall(DistributedFileSystem.java:1190)
> ~[hadoop-hdfs-client-3.1.1.7.2.15.4-6.jar:?]
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> ~[hadoop-common-3.1.1.7.2.15.4-6.jar:?]
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.listLocatedStatus(DistributedFileSystem.java:1208)
> ~[hadoop-hdfs-client-3.1.1.7.2.15.4-6.jar:?]
> at
> org.apache.hadoop.fs.FileSystem.listLocatedStatus(FileSystem.java:2144)
> ~[hadoop-common-3.1.1.7.2.15.4-6.jar:?]
> at org.apache.hadoop.fs.FileSystem$5.(FileSystem.java:2302)
> ~[hadoop-common-3.1.1.7.2.15.4-6.jar:?]
> at org.apache.hadoop.fs.FileSystem.listFiles(FileSystem.java:2299)
> ~[hadoop-common-3.1.1.7.2.15.4-6.jar:?]
> at
> org.apache.hadoop.hive.ql.txn.compactor.QueryCompactor$Util.cleanupEmptyDir(QueryCompactor.java:261)
> ~[hive-exec-3.1.3000.2022.0.13.0-60.jar:3.1.3000.2022.0.13.0-60]
> at
> org.apache.hadoop.hive.ql.txn.compactor.MmMinorQueryCompactor.commitCo
[jira] [Work logged] (HIVE-11495) Add aborted reason to transaction information.
[ https://issues.apache.org/jira/browse/HIVE-11495?focusedWorklogId=836872&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-836872 ] ASF GitHub Bot logged work on HIVE-11495: - Author: ASF GitHub Bot Created on: 04/Jan/23 11:57 Start Date: 04/Jan/23 11:57 Worklog Time Spent: 10m Work Description: SourabhBadhya commented on PR #3908: URL: https://github.com/apache/hive/pull/3908#issuecomment-1370838223 @veghlaci05 Based on a earlier conversation for the same ticket - https://github.com/apache/hive/pull/3656#issuecomment-1274240038 It was decided that we must opt for logging instead of persisting it. I will be editing the same in the Jira as well. Issue Time Tracking --- Worklog Id: (was: 836872) Time Spent: 2h 50m (was: 2h 40m) > Add aborted reason to transaction information. > -- > > Key: HIVE-11495 > URL: https://issues.apache.org/jira/browse/HIVE-11495 > Project: Hive > Issue Type: Improvement > Components: Metastore, Transactions >Affects Versions: 1.0.0 >Reporter: Eugene Koifman >Assignee: Sourabh Badhya >Priority: Major > Labels: pull-request-available > Time Spent: 2h 50m > Remaining Estimate: 0h > > Should add TXNS.COMMENT field or something like that so that if the system > aborts a transaction (due to timeout, for example) we can add a message to > that effect to the aborted transaction. > Another reason: Commit can fail due to a conflicting write from another txn > (since HIVE-13395) -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-11495) Add aborted reason to transaction information.
[
https://issues.apache.org/jira/browse/HIVE-11495?focusedWorklogId=836873&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-836873
]
ASF GitHub Bot logged work on HIVE-11495:
-
Author: ASF GitHub Bot
Created on: 04/Jan/23 12:02
Start Date: 04/Jan/23 12:02
Worklog Time Spent: 10m
Work Description: SourabhBadhya commented on code in PR #3908:
URL: https://github.com/apache/hive/pull/3908#discussion_r1061412088
##
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnErrorMsg.java:
##
@@ -0,0 +1,83 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.metastore.txn;
+
+/**
+ * The following class represents all the error messages that are handled for
aborts.
+ */
+public enum TxnErrorMsg {
+// 5 to 5: Txn Errors Codes.
+// Query runtime aborts
+NONE(5, "None"),
+ABORT_QUERY(50001, " Txn aborted by Abort Query Command"),
+ABORT_CONCURRENT(50002, " Txn aborted due to concurrent committed
transaction"),
+ABORT_WRITE_CONFLICT(50003, " Txn aborted due to write conflicts"),
+ABORT_TIMEOUT(50004, " Txn aborted due to heartbeat time-out"),
+ABORT_ROLLBACK(50005, "Txn aborted due to rollback"),
+ABORT_COMPACTION_TXN(50006, "Compaction txn is aborted"),
+ABORT_MSCK_TXN(50007, "Msck txn is aborted"),
+ABORT_MIGRATION_TXN(50008, "Managed Migration transaction is aborted"),
+
+// Replication related aborts
+ABORT_DEFAULT_REPL_TXN(50009, " Replication:" +
+"Abort default replication transaction"),
+ABORT_REPLAYED_REPL_TXN(50010, " Replication:" +
+"Abort replayed replication transaction"),
+ABORT_REPL_WRITEID_TXN(50011, " Replication:" +
+"Abort all the allocated txns so that the mapped write ids are
referred as aborted ones."),
+ABORT_FETCH_FAILOVER_METADATA(50012, " Replication:" +
+"Abort all transactions while trying to fetch failover metadata."),
+ABORT_WRITE_TXN_AFTER_TIMEOUT(50013, " Replication:" +
+"Abort only write transactions for the db under replication"),
+ABORT_ONGOING_TXN_FOR_TARGET_DB(50014, " Replication:" +
+"Abort the ongoing transactions(opened prior to failover) for the
target database.");
+
+private final long errorCode;
+private final String txnErrorMsg;
+
+TxnErrorMsg(int errorCode, String txnErrorMsg) {
+ this.errorCode = errorCode;
+ this.txnErrorMsg = txnErrorMsg;
+}
+
+public long getErrorCode() {
+return errorCode;
+}
+
+public String getTxnErrorMsg() {
Review Comment:
I think I will remove this function. The mapping of errorCode to TxnErrorMsg
is done here -
https://github.com/apache/hive/pull/3908/files#diff-22cbee1c93bb4d0cedd41531cd2d8631483ec310a6727bbdb2d73f8c830b6d95R74-R81
Issue Time Tracking
---
Worklog Id: (was: 836873)
Time Spent: 3h (was: 2h 50m)
> Add aborted reason to transaction information.
> --
>
> Key: HIVE-11495
> URL: https://issues.apache.org/jira/browse/HIVE-11495
> Project: Hive
> Issue Type: Improvement
> Components: Metastore, Transactions
>Affects Versions: 1.0.0
>Reporter: Eugene Koifman
>Assignee: Sourabh Badhya
>Priority: Major
> Labels: pull-request-available
> Time Spent: 3h
> Remaining Estimate: 0h
>
> Should add TXNS.COMMENT field or something like that so that if the system
> aborts a transaction (due to timeout, for example) we can add a message to
> that effect to the aborted transaction.
> Another reason: Commit can fail due to a conflicting write from another txn
> (since HIVE-13395)
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-11495) Add aborted reason to transaction information.
[
https://issues.apache.org/jira/browse/HIVE-11495?focusedWorklogId=836874&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-836874
]
ASF GitHub Bot logged work on HIVE-11495:
-
Author: ASF GitHub Bot
Created on: 04/Jan/23 12:07
Start Date: 04/Jan/23 12:07
Worklog Time Spent: 10m
Work Description: SourabhBadhya commented on code in PR #3908:
URL: https://github.com/apache/hive/pull/3908#discussion_r1061415809
##
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnErrorMsg.java:
##
@@ -0,0 +1,83 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.metastore.txn;
+
+/**
+ * The following class represents all the error messages that are handled for
aborts.
+ */
+public enum TxnErrorMsg {
Review Comment:
All these messages are written based on where `abortTxn()`/`abortTxns()` is
called. This represents the totality of how transactions are aborted from
HiveServer. Anything else has not come from Hiveserver but rather from other
services which might use transactional features of Hive (I have represented
them here as NONE). These are the only reasons where abort is done.
Issue Time Tracking
---
Worklog Id: (was: 836874)
Time Spent: 3h 10m (was: 3h)
> Add aborted reason to transaction information.
> --
>
> Key: HIVE-11495
> URL: https://issues.apache.org/jira/browse/HIVE-11495
> Project: Hive
> Issue Type: Improvement
> Components: Metastore, Transactions
>Affects Versions: 1.0.0
>Reporter: Eugene Koifman
>Assignee: Sourabh Badhya
>Priority: Major
> Labels: pull-request-available
> Time Spent: 3h 10m
> Remaining Estimate: 0h
>
> Should add TXNS.COMMENT field or something like that so that if the system
> aborts a transaction (due to timeout, for example) we can add a message to
> that effect to the aborted transaction.
> Another reason: Commit can fail due to a conflicting write from another txn
> (since HIVE-13395)
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (HIVE-11495) Add aborted reason to transaction information.
[ https://issues.apache.org/jira/browse/HIVE-11495?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sourabh Badhya updated HIVE-11495: -- Description: Add logging messages mentioning the reason to abort the transaction. Another reason: Commit can fail due to a conflicting write from another txn (since HIVE-13395) was: Should add TXNS.COMMENT field or something like that so that if the system aborts a transaction (due to timeout, for example) we can add a message to that effect to the aborted transaction. Another reason: Commit can fail due to a conflicting write from another txn (since HIVE-13395) > Add aborted reason to transaction information. > -- > > Key: HIVE-11495 > URL: https://issues.apache.org/jira/browse/HIVE-11495 > Project: Hive > Issue Type: Improvement > Components: Metastore, Transactions >Affects Versions: 1.0.0 >Reporter: Eugene Koifman >Assignee: Sourabh Badhya >Priority: Major > Labels: pull-request-available > Time Spent: 3h 10m > Remaining Estimate: 0h > > Add logging messages mentioning the reason to abort the transaction. > Another reason: Commit can fail due to a conflicting write from another txn > (since HIVE-13395) -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-11495) Add aborted reason to transaction information.
[
https://issues.apache.org/jira/browse/HIVE-11495?focusedWorklogId=836876&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-836876
]
ASF GitHub Bot logged work on HIVE-11495:
-
Author: ASF GitHub Bot
Created on: 04/Jan/23 12:09
Start Date: 04/Jan/23 12:09
Worklog Time Spent: 10m
Work Description: SourabhBadhya commented on code in PR #3908:
URL: https://github.com/apache/hive/pull/3908#discussion_r1061415809
##
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnErrorMsg.java:
##
@@ -0,0 +1,83 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.metastore.txn;
+
+/**
+ * The following class represents all the error messages that are handled for
aborts.
+ */
+public enum TxnErrorMsg {
Review Comment:
All these messages are written based on where `abortTxn()`/`abortTxns()` is
called. This represents the totality of how transactions are aborted from
HiveServer. Anything else has not come from Hiveserver but rather from other
services which might use transactional features of Hive (I have represented
them here as NONE). These are the only reasons where abort is done from
Hiveserver.
Issue Time Tracking
---
Worklog Id: (was: 836876)
Time Spent: 3h 20m (was: 3h 10m)
> Add aborted reason to transaction information.
> --
>
> Key: HIVE-11495
> URL: https://issues.apache.org/jira/browse/HIVE-11495
> Project: Hive
> Issue Type: Improvement
> Components: Metastore, Transactions
>Affects Versions: 1.0.0
>Reporter: Eugene Koifman
>Assignee: Sourabh Badhya
>Priority: Major
> Labels: pull-request-available
> Time Spent: 3h 20m
> Remaining Estimate: 0h
>
> Add logging messages mentioning the reason to abort the transaction.
> Another reason: Commit can fail due to a conflicting write from another txn
> (since HIVE-13395)
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-26880) Upgrade Apache Directory Server to 1.5.7 for release 3.2.
[
https://issues.apache.org/jira/browse/HIVE-26880?focusedWorklogId=836877&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-836877
]
ASF GitHub Bot logged work on HIVE-26880:
-
Author: ASF GitHub Bot
Created on: 04/Jan/23 12:14
Start Date: 04/Jan/23 12:14
Worklog Time Spent: 10m
Work Description: zabetak commented on code in PR #3886:
URL: https://github.com/apache/hive/pull/3886#discussion_r1061421382
##
service/pom.xml:
##
@@ -291,13 +291,6 @@
test
-
- org.apache.directory.client.ldap
- ldap-client-api
- ${apache-directory-clientapi.version}
- test
-
-
org.apache.directory.server
apacheds-server-integ
Review Comment:
Oh you are right. I was confused while checking the diff and though that the
exclusion was added rather than removed. Apologies for the back and forth!
Issue Time Tracking
---
Worklog Id: (was: 836877)
Time Spent: 1h 10m (was: 1h)
> Upgrade Apache Directory Server to 1.5.7 for release 3.2.
> -
>
> Key: HIVE-26880
> URL: https://issues.apache.org/jira/browse/HIVE-26880
> Project: Hive
> Issue Type: Improvement
> Components: Test
>Reporter: Chris Nauroth
>Assignee: Chris Nauroth
>Priority: Minor
> Labels: hive-3.2.0-must, pull-request-available
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> branch-3 uses Apache Directory Server in some tests. It currently uses
> version 1.5.6. This version has a transitive dependency to a SNAPSHOT, making
> it awkward to build and release. We can upgrade to 1.5.7 to remove the
> SNAPSHOT dependency.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-26719) Ability to set number of buckets manually
[ https://issues.apache.org/jira/browse/HIVE-26719?focusedWorklogId=836878&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-836878 ] ASF GitHub Bot logged work on HIVE-26719: - Author: ASF GitHub Bot Created on: 04/Jan/23 12:15 Start Date: 04/Jan/23 12:15 Worklog Time Spent: 10m Work Description: sonarcloud[bot] commented on PR #3759: URL: https://github.com/apache/hive/pull/3759#issuecomment-1370856621 Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=3759) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3759&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3759&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3759&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3759&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3759&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3759&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=3759&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=3759&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=3759&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3759&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3759&resolved=false&types=CODE_SMELL) [90 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3759&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=3759&metric=coverage&view=list) No Coverage information [](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=3759&metric=duplicated_lines_density&view=list) No Duplication information Issue Time Tracking --- Worklog Id: (was: 836878) Time Spent: 1h 20m (was: 1h 10m) > Ability to set number of buckets manually > - > > Key: HIVE-26719 > URL: https://issues.apache.org/jira/browse/HIVE-26719 > Project: Hive > Issue Type: Sub-task >Reporter: László Végh >Assignee: László Végh >Priority: Major > Labels: pull-request-available > Time Spent: 1h 20m > Remaining Estimate: 0h > > Add the ability to set the number of implicit buckets manually through the > alter table compact command -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-11495) Add aborted reason to transaction information.
[
https://issues.apache.org/jira/browse/HIVE-11495?focusedWorklogId=836879&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-836879
]
ASF GitHub Bot logged work on HIVE-11495:
-
Author: ASF GitHub Bot
Created on: 04/Jan/23 12:20
Start Date: 04/Jan/23 12:20
Worklog Time Spent: 10m
Work Description: SourabhBadhya commented on code in PR #3908:
URL: https://github.com/apache/hive/pull/3908#discussion_r1061415809
##
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnErrorMsg.java:
##
@@ -0,0 +1,83 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.metastore.txn;
+
+/**
+ * The following class represents all the error messages that are handled for
aborts.
+ */
+public enum TxnErrorMsg {
Review Comment:
All these messages are written based on where `abortTxn()`/`abortTxns()` is
called. This represents the totality of how transactions are aborted from
HiveServer. Anything else has not come from Hiveserver but rather from other
services which might use transactional features of Hive (I have tried to
represent them here as NONE). These are the only reasons where abort is done
from Hiveserver.
Issue Time Tracking
---
Worklog Id: (was: 836879)
Time Spent: 3.5h (was: 3h 20m)
> Add aborted reason to transaction information.
> --
>
> Key: HIVE-11495
> URL: https://issues.apache.org/jira/browse/HIVE-11495
> Project: Hive
> Issue Type: Improvement
> Components: Metastore, Transactions
>Affects Versions: 1.0.0
>Reporter: Eugene Koifman
>Assignee: Sourabh Badhya
>Priority: Major
> Labels: pull-request-available
> Time Spent: 3.5h
> Remaining Estimate: 0h
>
> Add logging messages mentioning the reason to abort the transaction.
> Another reason: Commit can fail due to a conflicting write from another txn
> (since HIVE-13395)
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-26880) Upgrade Apache Directory Server to 1.5.7 for release 3.2.
[ https://issues.apache.org/jira/browse/HIVE-26880?focusedWorklogId=836880&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-836880 ] ASF GitHub Bot logged work on HIVE-26880: - Author: ASF GitHub Bot Created on: 04/Jan/23 12:21 Start Date: 04/Jan/23 12:21 Worklog Time Spent: 10m Work Description: zabetak closed pull request #3886: HIVE-26880: Upgrade Apache Directory Server to 1.5.7 for release 3.2. URL: https://github.com/apache/hive/pull/3886 Issue Time Tracking --- Worklog Id: (was: 836880) Time Spent: 1h 20m (was: 1h 10m) > Upgrade Apache Directory Server to 1.5.7 for release 3.2. > - > > Key: HIVE-26880 > URL: https://issues.apache.org/jira/browse/HIVE-26880 > Project: Hive > Issue Type: Improvement > Components: Test >Reporter: Chris Nauroth >Assignee: Chris Nauroth >Priority: Minor > Labels: hive-3.2.0-must, pull-request-available > Time Spent: 1h 20m > Remaining Estimate: 0h > > branch-3 uses Apache Directory Server in some tests. It currently uses > version 1.5.6. This version has a transitive dependency to a SNAPSHOT, making > it awkward to build and release. We can upgrade to 1.5.7 to remove the > SNAPSHOT dependency. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (HIVE-26880) Upgrade Apache Directory Server to 1.5.7 for release 3.2.
[ https://issues.apache.org/jira/browse/HIVE-26880?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stamatis Zampetakis resolved HIVE-26880. Fix Version/s: 3.2.0 Resolution: Fixed Fixed in https://github.com/apache/hive/commit/771b8055dae71e0e154ef75d2fd2295bbac7c171. Thanks for the PR [~cnauroth]! > Upgrade Apache Directory Server to 1.5.7 for release 3.2. > - > > Key: HIVE-26880 > URL: https://issues.apache.org/jira/browse/HIVE-26880 > Project: Hive > Issue Type: Improvement > Components: Test >Reporter: Chris Nauroth >Assignee: Chris Nauroth >Priority: Minor > Labels: hive-3.2.0-must, pull-request-available > Fix For: 3.2.0 > > Time Spent: 1h 20m > Remaining Estimate: 0h > > branch-3 uses Apache Directory Server in some tests. It currently uses > version 1.5.6. This version has a transitive dependency to a SNAPSHOT, making > it awkward to build and release. We can upgrade to 1.5.7 to remove the > SNAPSHOT dependency. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-11495) Add aborted reason to transaction information.
[
https://issues.apache.org/jira/browse/HIVE-11495?focusedWorklogId=836881&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-836881
]
ASF GitHub Bot logged work on HIVE-11495:
-
Author: ASF GitHub Bot
Created on: 04/Jan/23 12:26
Start Date: 04/Jan/23 12:26
Worklog Time Spent: 10m
Work Description: veghlaci05 commented on code in PR #3908:
URL: https://github.com/apache/hive/pull/3908#discussion_r1061430475
##
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnErrorMsg.java:
##
@@ -0,0 +1,83 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.metastore.txn;
+
+/**
+ * The following class represents all the error messages that are handled for
aborts.
+ */
+public enum TxnErrorMsg {
Review Comment:
Ok
Issue Time Tracking
---
Worklog Id: (was: 836881)
Time Spent: 3h 40m (was: 3.5h)
> Add aborted reason to transaction information.
> --
>
> Key: HIVE-11495
> URL: https://issues.apache.org/jira/browse/HIVE-11495
> Project: Hive
> Issue Type: Improvement
> Components: Metastore, Transactions
>Affects Versions: 1.0.0
>Reporter: Eugene Koifman
>Assignee: Sourabh Badhya
>Priority: Major
> Labels: pull-request-available
> Time Spent: 3h 40m
> Remaining Estimate: 0h
>
> Add logging messages mentioning the reason to abort the transaction.
> Another reason: Commit can fail due to a conflicting write from another txn
> (since HIVE-13395)
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-11495) Add aborted reason to transaction information.
[
https://issues.apache.org/jira/browse/HIVE-11495?focusedWorklogId=836882&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-836882
]
ASF GitHub Bot logged work on HIVE-11495:
-
Author: ASF GitHub Bot
Created on: 04/Jan/23 12:30
Start Date: 04/Jan/23 12:30
Worklog Time Spent: 10m
Work Description: veghlaci05 commented on code in PR #3908:
URL: https://github.com/apache/hive/pull/3908#discussion_r1061433523
##
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnErrorMsg.java:
##
@@ -0,0 +1,83 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.metastore.txn;
+
+/**
+ * The following class represents all the error messages that are handled for
aborts.
+ */
+public enum TxnErrorMsg {
+// 5 to 5: Txn Errors Codes.
+// Query runtime aborts
+NONE(5, "None"),
+ABORT_QUERY(50001, " Txn aborted by Abort Query Command"),
+ABORT_CONCURRENT(50002, " Txn aborted due to concurrent committed
transaction"),
+ABORT_WRITE_CONFLICT(50003, " Txn aborted due to write conflicts"),
+ABORT_TIMEOUT(50004, " Txn aborted due to heartbeat time-out"),
+ABORT_ROLLBACK(50005, "Txn aborted due to rollback"),
+ABORT_COMPACTION_TXN(50006, "Compaction txn is aborted"),
+ABORT_MSCK_TXN(50007, "Msck txn is aborted"),
+ABORT_MIGRATION_TXN(50008, "Managed Migration transaction is aborted"),
+
+// Replication related aborts
+ABORT_DEFAULT_REPL_TXN(50009, " Replication:" +
+"Abort default replication transaction"),
+ABORT_REPLAYED_REPL_TXN(50010, " Replication:" +
+"Abort replayed replication transaction"),
+ABORT_REPL_WRITEID_TXN(50011, " Replication:" +
+"Abort all the allocated txns so that the mapped write ids are
referred as aborted ones."),
+ABORT_FETCH_FAILOVER_METADATA(50012, " Replication:" +
+"Abort all transactions while trying to fetch failover metadata."),
+ABORT_WRITE_TXN_AFTER_TIMEOUT(50013, " Replication:" +
+"Abort only write transactions for the db under replication"),
+ABORT_ONGOING_TXN_FOR_TARGET_DB(50014, " Replication:" +
+"Abort the ongoing transactions(opened prior to failover) for the
target database.");
+
+private final long errorCode;
+private final String txnErrorMsg;
+
+TxnErrorMsg(int errorCode, String txnErrorMsg) {
+ this.errorCode = errorCode;
+ this.txnErrorMsg = txnErrorMsg;
+}
+
+public long getErrorCode() {
+return errorCode;
+}
+
+public String getTxnErrorMsg() {
Review Comment:
This link redirects to iteslf for some reason but I guess you wanted to
point to line 1099 in TxnHandler. My main concern about the messages what I
have also added as a general comment: The messages are not persisted and not
shown to the users in any way (except in the logs).
Issue Time Tracking
---
Worklog Id: (was: 836882)
Time Spent: 3h 50m (was: 3h 40m)
> Add aborted reason to transaction information.
> --
>
> Key: HIVE-11495
> URL: https://issues.apache.org/jira/browse/HIVE-11495
> Project: Hive
> Issue Type: Improvement
> Components: Metastore, Transactions
>Affects Versions: 1.0.0
>Reporter: Eugene Koifman
>Assignee: Sourabh Badhya
>Priority: Major
> Labels: pull-request-available
> Time Spent: 3h 50m
> Remaining Estimate: 0h
>
> Add logging messages mentioning the reason to abort the transaction.
> Another reason: Commit can fail due to a conflicting write from another txn
> (since HIVE-13395)
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-11495) Add aborted reason to transaction information.
[ https://issues.apache.org/jira/browse/HIVE-11495?focusedWorklogId=836884&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-836884 ] ASF GitHub Bot logged work on HIVE-11495: - Author: ASF GitHub Bot Created on: 04/Jan/23 12:38 Start Date: 04/Jan/23 12:38 Worklog Time Spent: 10m Work Description: veghlaci05 commented on PR #3908: URL: https://github.com/apache/hive/pull/3908#issuecomment-1370877808 > @veghlaci05 Based on a earlier conversation for the same ticket - [#3656 (comment)](https://github.com/apache/hive/pull/3656#issuecomment-1274240038) It was decided that we must opt for logging instead of persisting it. I have edited the same in the Jira as well. In this please ignore my comments regarding the presistence. While I still think persisting would be better, I have to admit it would make this task significantly more complicated. (As @deniskuzZ pointed out, aborted TXNs are cleaned up quickly). Issue Time Tracking --- Worklog Id: (was: 836884) Time Spent: 4h (was: 3h 50m) > Add aborted reason to transaction information. > -- > > Key: HIVE-11495 > URL: https://issues.apache.org/jira/browse/HIVE-11495 > Project: Hive > Issue Type: Improvement > Components: Metastore, Transactions >Affects Versions: 1.0.0 >Reporter: Eugene Koifman >Assignee: Sourabh Badhya >Priority: Major > Labels: pull-request-available > Time Spent: 4h > Remaining Estimate: 0h > > Add logging messages mentioning the reason to abort the transaction. > Another reason: Commit can fail due to a conflicting write from another txn > (since HIVE-13395) -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-11495) Add aborted reason to transaction information.
[
https://issues.apache.org/jira/browse/HIVE-11495?focusedWorklogId=836890&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-836890
]
ASF GitHub Bot logged work on HIVE-11495:
-
Author: ASF GitHub Bot
Created on: 04/Jan/23 13:00
Start Date: 04/Jan/23 13:00
Worklog Time Spent: 10m
Work Description: SourabhBadhya commented on code in PR #3908:
URL: https://github.com/apache/hive/pull/3908#discussion_r1061458528
##
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnErrorMsg.java:
##
@@ -0,0 +1,83 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.metastore.txn;
+
+/**
+ * The following class represents all the error messages that are handled for
aborts.
+ */
+public enum TxnErrorMsg {
+// 5 to 5: Txn Errors Codes.
+// Query runtime aborts
+NONE(5, "None"),
+ABORT_QUERY(50001, " Txn aborted by Abort Query Command"),
+ABORT_CONCURRENT(50002, " Txn aborted due to concurrent committed
transaction"),
+ABORT_WRITE_CONFLICT(50003, " Txn aborted due to write conflicts"),
+ABORT_TIMEOUT(50004, " Txn aborted due to heartbeat time-out"),
+ABORT_ROLLBACK(50005, "Txn aborted due to rollback"),
+ABORT_COMPACTION_TXN(50006, "Compaction txn is aborted"),
+ABORT_MSCK_TXN(50007, "Msck txn is aborted"),
+ABORT_MIGRATION_TXN(50008, "Managed Migration transaction is aborted"),
+
+// Replication related aborts
+ABORT_DEFAULT_REPL_TXN(50009, " Replication:" +
Review Comment:
Nice point. I have grouped them accordingly. Done.
Issue Time Tracking
---
Worklog Id: (was: 836890)
Time Spent: 4h 10m (was: 4h)
> Add aborted reason to transaction information.
> --
>
> Key: HIVE-11495
> URL: https://issues.apache.org/jira/browse/HIVE-11495
> Project: Hive
> Issue Type: Improvement
> Components: Metastore, Transactions
>Affects Versions: 1.0.0
>Reporter: Eugene Koifman
>Assignee: Sourabh Badhya
>Priority: Major
> Labels: pull-request-available
> Time Spent: 4h 10m
> Remaining Estimate: 0h
>
> Add logging messages mentioning the reason to abort the transaction.
> Another reason: Commit can fail due to a conflicting write from another txn
> (since HIVE-13395)
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-26904) QueryCompactor failed in commitCompaction if the tmp table dir is already removed
[
https://issues.apache.org/jira/browse/HIVE-26904?focusedWorklogId=836904&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-836904
]
ASF GitHub Bot logged work on HIVE-26904:
-
Author: ASF GitHub Bot
Created on: 04/Jan/23 14:03
Start Date: 04/Jan/23 14:03
Worklog Time Spent: 10m
Work Description: sonarcloud[bot] commented on PR #3910:
URL: https://github.com/apache/hive/pull/3910#issuecomment-1370967210
Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=3910)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3910&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3910&resolved=false&types=BUG)
[0
Bugs](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3910&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3910&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3910&resolved=false&types=VULNERABILITY)
[0
Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3910&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=3910&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=3910&resolved=false&types=SECURITY_HOTSPOT)
[0 Security
Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=3910&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3910&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3910&resolved=false&types=CODE_SMELL)
[87 Code
Smells](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3910&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=3910&metric=coverage&view=list)
No Coverage information
[](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=3910&metric=duplicated_lines_density&view=list)
No Duplication information
Issue Time Tracking
---
Worklog Id: (was: 836904)
Time Spent: 20m (was: 10m)
> QueryCompactor failed in commitCompaction if the tmp table dir is already
> removed
> --
>
> Key: HIVE-26904
> URL: https://issues.apache.org/jira/browse/HIVE-26904
> Project: Hive
> Issue Type: Bug
>Reporter: Quanlong Huang
>Assignee: Quanlong Huang
>Priority: Major
> Labels: pull-request-available
> Time Spent: 20m
> Remaining Estimate: 0h
>
> commitCompaction() of query-based compactions just remove the dirs of tmp
> tables. It should not fail the compaction if the dirs are already removed.
> We've seen such a failure in Impala's test (IMPALA-11756):
> {noformat}
> 2023-01-02T02:09:26,306 INFO [HiveServer2-Background-Pool: Thread-695]
> ql.Driver: Executing
> command(queryId=jenkins_20230102020926_69112755-b783-4214-89e5-1c7111dfe15f):
> alter table partial_catalog_info_test.insert_only_partitioned partition
> (part=1) compact 'minor' and wait
> 2023-01-02T02:09:26,306 INFO [HiveSe
[jira] [Work logged] (HIVE-26035) Explore moving to directsql for ObjectStore::addPartitions
[
https://issues.apache.org/jira/browse/HIVE-26035?focusedWorklogId=836909&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-836909
]
ASF GitHub Bot logged work on HIVE-26035:
-
Author: ASF GitHub Bot
Created on: 04/Jan/23 14:19
Start Date: 04/Jan/23 14:19
Worklog Time Spent: 10m
Work Description: sonarcloud[bot] commented on PR #3905:
URL: https://github.com/apache/hive/pull/3905#issuecomment-1370987766
Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=3905)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3905&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3905&resolved=false&types=BUG)
[0
Bugs](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3905&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3905&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3905&resolved=false&types=VULNERABILITY)
[0
Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3905&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=3905&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=3905&resolved=false&types=SECURITY_HOTSPOT)
[0 Security
Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=3905&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3905&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3905&resolved=false&types=CODE_SMELL)
[93 Code
Smells](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3905&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=3905&metric=coverage&view=list)
No Coverage information
[](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=3905&metric=duplicated_lines_density&view=list)
No Duplication information
Issue Time Tracking
---
Worklog Id: (was: 836909)
Time Spent: 0.5h (was: 20m)
> Explore moving to directsql for ObjectStore::addPartitions
> --
>
> Key: HIVE-26035
> URL: https://issues.apache.org/jira/browse/HIVE-26035
> Project: Hive
> Issue Type: Bug
>Reporter: Rajesh Balamohan
>Priority: Major
> Labels: pull-request-available
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> Currently {{addPartitions}} uses datanuclues and is super slow for large
> number of partitions. It will be good to move to direct sql. Lots of repeated
> SQLs can be avoided as well (e.g SDS, SERDE, TABLE_PARAMS)
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-26802) Create qtest running QB compaction queries
[
https://issues.apache.org/jira/browse/HIVE-26802?focusedWorklogId=836934&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-836934
]
ASF GitHub Bot logged work on HIVE-26802:
-
Author: ASF GitHub Bot
Created on: 04/Jan/23 15:28
Start Date: 04/Jan/23 15:28
Worklog Time Spent: 10m
Work Description: zratkai commented on code in PR #3882:
URL: https://github.com/apache/hive/pull/3882#discussion_r1061602461
##
itests/qtest/src/test/java/org/apache/hadoop/hive/cli/TestMiniLlapLocalCompactorCliDriver.java:
##
@@ -0,0 +1,84 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.cli;
+
+import org.apache.hadoop.hive.cli.control.CliAdapter;
+import org.apache.hadoop.hive.cli.control.CliConfigs;
+import org.apache.hadoop.hive.cli.control.SplitSupport;
+import org.apache.hadoop.hive.ql.session.SessionState;
+import org.apache.hadoop.hive.ql.txn.compactor.Worker;
+import org.junit.AfterClass;
+import org.junit.BeforeClass;
+import org.junit.ClassRule;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.rules.TestRule;
+import org.junit.runner.RunWith;
+import org.junit.runners.Parameterized;
+import org.junit.runners.Parameterized.Parameters;
+
+import java.io.File;
+import java.util.List;
+import java.util.concurrent.atomic.AtomicBoolean;
+
+@RunWith(Parameterized.class)
+public class TestMiniLlapLocalCompactorCliDriver {
+
+ static CliAdapter adapter = new
CliConfigs.MiniLlapLocalCompactorCliConfig().getCliAdapter();
+
+ private static int N_SPLITS = 32;
+
+ private static final AtomicBoolean stop = new AtomicBoolean();
+ private static Worker worker;
+ @Parameters(name = "{0}")
+ public static List getParameters() throws Exception {
+return SplitSupport.process(adapter.getParameters(),
TestMiniLlapLocalCompactorCliDriver.class, N_SPLITS);
+ }
+
+ @ClassRule
+ public static TestRule cliClassRule = adapter.buildClassRule();
+
+ @Rule
+ public TestRule cliTestRule = adapter.buildTestRule();
+
+ @BeforeClass
+ public static void setup() throws Exception {
+worker = new Worker();
Review Comment:
Would be good idea, but the implementation has a bug in it.It won't start
the Worker, because the stop flag is set to true at the beginning:
AtomicBoolean stop = new AtomicBoolean(true);
Issue Time Tracking
---
Worklog Id: (was: 836934)
Time Spent: 2h (was: 1h 50m)
> Create qtest running QB compaction queries
> --
>
> Key: HIVE-26802
> URL: https://issues.apache.org/jira/browse/HIVE-26802
> Project: Hive
> Issue Type: Improvement
>Reporter: Zoltán Rátkai
>Assignee: Zoltán Rátkai
>Priority: Minor
> Labels: pull-request-available
> Time Spent: 2h
> Remaining Estimate: 0h
>
> Create a qtest that runs the queries that query-based compaction runs.
> Not so much to check for correct data but more to check the query plans, to
> simplify tracing changes in compilation that might affect QB compaction.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-26802) Create qtest running QB compaction queries
[
https://issues.apache.org/jira/browse/HIVE-26802?focusedWorklogId=836942&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-836942
]
ASF GitHub Bot logged work on HIVE-26802:
-
Author: ASF GitHub Bot
Created on: 04/Jan/23 16:04
Start Date: 04/Jan/23 16:04
Worklog Time Spent: 10m
Work Description: zratkai commented on code in PR #3882:
URL: https://github.com/apache/hive/pull/3882#discussion_r1061641312
##
itests/qtest/src/test/java/org/apache/hadoop/hive/cli/TestMiniLlapLocalCompactorCliDriver.java:
##
@@ -0,0 +1,84 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.cli;
+
+import org.apache.hadoop.hive.cli.control.CliAdapter;
+import org.apache.hadoop.hive.cli.control.CliConfigs;
+import org.apache.hadoop.hive.cli.control.SplitSupport;
+import org.apache.hadoop.hive.ql.session.SessionState;
+import org.apache.hadoop.hive.ql.txn.compactor.Worker;
+import org.junit.AfterClass;
+import org.junit.BeforeClass;
+import org.junit.ClassRule;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.rules.TestRule;
+import org.junit.runner.RunWith;
+import org.junit.runners.Parameterized;
+import org.junit.runners.Parameterized.Parameters;
+
+import java.io.File;
+import java.util.List;
+import java.util.concurrent.atomic.AtomicBoolean;
+
+@RunWith(Parameterized.class)
+public class TestMiniLlapLocalCompactorCliDriver {
Review Comment:
Good idea. Moved the Worker init to TestMiniLlapLocalCliDriver.
Issue Time Tracking
---
Worklog Id: (was: 836942)
Time Spent: 2h 10m (was: 2h)
> Create qtest running QB compaction queries
> --
>
> Key: HIVE-26802
> URL: https://issues.apache.org/jira/browse/HIVE-26802
> Project: Hive
> Issue Type: Improvement
>Reporter: Zoltán Rátkai
>Assignee: Zoltán Rátkai
>Priority: Minor
> Labels: pull-request-available
> Time Spent: 2h 10m
> Remaining Estimate: 0h
>
> Create a qtest that runs the queries that query-based compaction runs.
> Not so much to check for correct data but more to check the query plans, to
> simplify tracing changes in compilation that might affect QB compaction.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-26711) The very first REPL Load should make the Target Database read-only
[ https://issues.apache.org/jira/browse/HIVE-26711?focusedWorklogId=836949&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-836949 ] ASF GitHub Bot logged work on HIVE-26711: - Author: ASF GitHub Bot Created on: 04/Jan/23 16:35 Start Date: 04/Jan/23 16:35 Worklog Time Spent: 10m Work Description: shreenidhiSaigaonkar commented on PR #3736: URL: https://github.com/apache/hive/pull/3736#issuecomment-1371155745 @pudidic could you please review the change Issue Time Tracking --- Worklog Id: (was: 836949) Time Spent: 0.5h (was: 20m) > The very first REPL Load should make the Target Database read-only > -- > > Key: HIVE-26711 > URL: https://issues.apache.org/jira/browse/HIVE-26711 > Project: Hive > Issue Type: Task >Reporter: Shreenidhi >Assignee: Shreenidhi >Priority: Major > Labels: pull-request-available > Time Spent: 0.5h > Remaining Estimate: 0h > > Use EnforceReadOnly hook to set TARGET database read only during BootStrap > load. > Also ensure backward compatibility. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-26875) Transaction conflict retry loop only executes once
[
https://issues.apache.org/jira/browse/HIVE-26875?focusedWorklogId=836960&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-836960
]
ASF GitHub Bot logged work on HIVE-26875:
-
Author: ASF GitHub Bot
Created on: 04/Jan/23 17:24
Start Date: 04/Jan/23 17:24
Worklog Time Spent: 10m
Work Description: deniskuzZ merged PR #3887:
URL: https://github.com/apache/hive/pull/3887
Issue Time Tracking
---
Worklog Id: (was: 836960)
Time Spent: 2h (was: 1h 50m)
> Transaction conflict retry loop only executes once
> --
>
> Key: HIVE-26875
> URL: https://issues.apache.org/jira/browse/HIVE-26875
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Reporter: John Sherman
>Assignee: John Sherman
>Priority: Critical
> Labels: pull-request-available
> Time Spent: 2h
> Remaining Estimate: 0h
>
> Currently the "conflict retry loop" only executes once.
> [https://github.com/apache/hive/blob/ab4c53de82d4aaa33706510441167f2df55df15e/ql/src/java/org/apache/hadoop/hive/ql/Driver.java#L264]
> The intent of this loop is to detect if a conflicting transaction has
> committed while we were waiting to acquire locks. If there is a conflicting
> transaction, it invalidates the snapshot, rolls-back the transaction, opens a
> new transaction and tries to re-acquire locks (and then recompile). It then
> checks again if a conflicting transaction has committed and if so, redoes the
> above steps again, up to HIVE_TXN_MAX_RETRYSNAPSHOT_COUNT times.
> However - isValidTxnState relies on getNonSharedLockedTable():
> [https://github.com/apache/hive/blob/ab4c53de82d4aaa33706510441167f2df55df15e/ql/src/java/org/apache/hadoop/hive/ql/DriverTxnHandler.java#L422]
> which does:
> {code:java}
> private Set getNonSharedLockedTables() {
> if (CollectionUtils.isEmpty(driver.getContext().getHiveLocks())) {
> return Collections.emptySet(); // Nothing to check
> }{code}
> getHiveLocks gets populated by lockAndRespond... HOWEVER -
> compileInternal ends up calling compile which ends up calling
> preparForCompile which ends up calling prepareContext which ends up
> destroying the context with the information lockAndRespond populated. So when
> the loop executes after all of this, it will never detect a 2nd conflict
> because isValidTxnState will always return true (because it thinks there are
> no locked objects).
> This manifests as duplicate records being created during concurrent UPDATEs
> if a transaction get conflicted twice.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-11495) Add aborted reason to transaction information.
[ https://issues.apache.org/jira/browse/HIVE-11495?focusedWorklogId=836966&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-836966 ] ASF GitHub Bot logged work on HIVE-11495: - Author: ASF GitHub Bot Created on: 04/Jan/23 17:40 Start Date: 04/Jan/23 17:40 Worklog Time Spent: 10m Work Description: sonarcloud[bot] commented on PR #3908: URL: https://github.com/apache/hive/pull/3908#issuecomment-1371229162 Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=3908) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3908&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3908&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3908&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3908&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3908&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3908&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=3908&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=3908&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=3908&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3908&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3908&resolved=false&types=CODE_SMELL) [10 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3908&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=3908&metric=coverage&view=list) No Coverage information [](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=3908&metric=duplicated_lines_density&view=list) No Duplication information Issue Time Tracking --- Worklog Id: (was: 836966) Time Spent: 4h 20m (was: 4h 10m) > Add aborted reason to transaction information. > -- > > Key: HIVE-11495 > URL: https://issues.apache.org/jira/browse/HIVE-11495 > Project: Hive > Issue Type: Improvement > Components: Metastore, Transactions >Affects Versions: 1.0.0 >Reporter: Eugene Koifman >Assignee: Sourabh Badhya >Priority: Major > Labels: pull-request-available > Time Spent: 4h 20m > Remaining Estimate: 0h > > Add logging messages mentioning the reason to abort the transaction. > Another reason: Commit can fail due to a conflicting write from another txn > (since HIVE-13395) -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-26808) Port Iceberg catalog changes
[ https://issues.apache.org/jira/browse/HIVE-26808?focusedWorklogId=836987&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-836987 ] ASF GitHub Bot logged work on HIVE-26808: - Author: ASF GitHub Bot Created on: 04/Jan/23 18:15 Start Date: 04/Jan/23 18:15 Worklog Time Spent: 10m Work Description: sonarcloud[bot] commented on PR #3907: URL: https://github.com/apache/hive/pull/3907#issuecomment-1371263394 Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=3907) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3907&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3907&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3907&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3907&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3907&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3907&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=3907&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=3907&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=3907&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3907&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3907&resolved=false&types=CODE_SMELL) [0 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3907&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=3907&metric=coverage&view=list) No Coverage information [](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=3907&metric=duplicated_lines_density&view=list) No Duplication information Issue Time Tracking --- Worklog Id: (was: 836987) Time Spent: 40m (was: 0.5h) > Port Iceberg catalog changes > > > Key: HIVE-26808 > URL: https://issues.apache.org/jira/browse/HIVE-26808 > Project: Hive > Issue Type: Improvement > Components: Iceberg integration >Reporter: Zsolt Miskolczi >Assignee: Zsolt Miskolczi >Priority: Major > Labels: pull-request-available > Time Spent: 40m > Remaining Estimate: 0h > > The last round of porting happened in 2022 april, there were a couple of > changes especially in HiveTableOperations worth porting into iceberg-catalog. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-26882) Allow transactional check of Table parameter before altering the Table
[
https://issues.apache.org/jira/browse/HIVE-26882?focusedWorklogId=837001&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-837001
]
ASF GitHub Bot logged work on HIVE-26882:
-
Author: ASF GitHub Bot
Created on: 04/Jan/23 18:45
Start Date: 04/Jan/23 18:45
Worklog Time Spent: 10m
Work Description: prasanthj commented on code in PR #3888:
URL: https://github.com/apache/hive/pull/3888#discussion_r1061784678
##
standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/client/TestTablesCreateDropAlterTruncate.java:
##
@@ -1183,6 +1193,90 @@ public void testAlterTableAlreadyExists() throws
Exception {
}
}
+ @Test
+ public void testAlterTableExpectedPropertyMatch() throws Exception {
+Table originalTable = testTables[0];
+
+EnvironmentContext context = new EnvironmentContext();
+context.putToProperties(hive_metastoreConstants.EXPECTED_PARAMETER_KEY,
"transient_lastDdlTime");
+context.putToProperties(hive_metastoreConstants.EXPECTED_PARAMETER_VALUE,
+originalTable.getParameters().get("transient_lastDdlTime"));
+
+client.alter_table(originalTable.getCatName(), originalTable.getDbName(),
originalTable.getTableName(),
+originalTable, context);
+ }
+
+ @Test(expected = MetaException.class)
+ public void testAlterTableExpectedPropertyDifferent() throws Exception {
+Table originalTable = testTables[0];
+
+EnvironmentContext context = new EnvironmentContext();
+context.putToProperties(hive_metastoreConstants.EXPECTED_PARAMETER_KEY,
"transient_lastDdlTime");
+context.putToProperties(hive_metastoreConstants.EXPECTED_PARAMETER_VALUE,
"alma");
+
+client.alter_table(originalTable.getCatName(), originalTable.getDbName(),
originalTable.getTableName(),
+originalTable, context);
+ }
+
+ /**
+ * This tests ensures that concurrent Iceberg commits will fail. Acceptable
as a first sanity check.
+ *
+ * I have not found a good way to check that HMS side database commits are
parallel in the
+ * {@link
org.apache.hadoop.hive.metastore.HiveAlterHandler#alterTable(RawStore,
Warehouse, String, String, String, Table, EnvironmentContext, IHMSHandler,
String)}
+ * call, but this test could be used to manually ensure that using
breakpoints.
+ */
+ @Test
+ public void testAlterTableExpectedPropertyConcurrent() throws Exception {
+Table originalTable = testTables[0];
+
+originalTable.getParameters().put("snapshot", "1");
+client.alter_table(originalTable.getCatName(), originalTable.getDbName(),
originalTable.getTableName(),
+originalTable, null);
+
+EnvironmentContext context = new EnvironmentContext();
Review Comment:
nit: small change to concurrency test. if you can run a loop for few
iterations with snapshot=i and expected value=i-1 that will be useful.. just
some repetition to avoid flakiness..
Issue Time Tracking
---
Worklog Id: (was: 837001)
Time Spent: 1h 40m (was: 1.5h)
> Allow transactional check of Table parameter before altering the Table
> --
>
> Key: HIVE-26882
> URL: https://issues.apache.org/jira/browse/HIVE-26882
> Project: Hive
> Issue Type: Improvement
> Components: Standalone Metastore
>Reporter: Peter Vary
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h 40m
> Remaining Estimate: 0h
>
> We should add the possibility to transactionally check if a Table parameter
> is changed before altering the table in the HMS.
> This would provide an alternative, less error-prone and faster way to commit
> an Iceberg table, as the Iceberg table currently needs to:
> - Create an exclusive lock
> - Get the table metadata to check if the current snapshot is not changed
> - Update the table metadata
> - Release the lock
> After the change these 4 HMS calls could be substituted with a single alter
> table call.
> Also we could avoid cases where the locks are left hanging by failed processes
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-25707) SchemaTool may leave the metastore in-between upgrade steps
[ https://issues.apache.org/jira/browse/HIVE-25707?focusedWorklogId=837004&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-837004 ] ASF GitHub Bot logged work on HIVE-25707: - Author: ASF GitHub Bot Created on: 04/Jan/23 18:58 Start Date: 04/Jan/23 18:58 Worklog Time Spent: 10m Work Description: nrg4878 merged PR #3863: URL: https://github.com/apache/hive/pull/3863 Issue Time Tracking --- Worklog Id: (was: 837004) Time Spent: 2h (was: 1h 50m) > SchemaTool may leave the metastore in-between upgrade steps > --- > > Key: HIVE-25707 > URL: https://issues.apache.org/jira/browse/HIVE-25707 > Project: Hive > Issue Type: Bug >Reporter: Zoltan Haindrich >Priority: Major > Labels: pull-request-available > Time Spent: 2h > Remaining Estimate: 0h > > it seems like: > * schematool runs the sql files via beeline > * autocommit is turned on > * pressing ctrl+c or killing the process will result in an invalid schema > https://github.com/apache/hive/blob/6e02f6164385a370ee8014c795bee1fa423d7937/beeline/src/java/org/apache/hive/beeline/schematool/HiveSchemaTool.java#L79 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-26802) Create qtest running QB compaction queries
[ https://issues.apache.org/jira/browse/HIVE-26802?focusedWorklogId=837005&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-837005 ] ASF GitHub Bot logged work on HIVE-26802: - Author: ASF GitHub Bot Created on: 04/Jan/23 19:04 Start Date: 04/Jan/23 19:04 Worklog Time Spent: 10m Work Description: sonarcloud[bot] commented on PR #3882: URL: https://github.com/apache/hive/pull/3882#issuecomment-1371308381 Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=3882) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3882&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3882&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3882&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3882&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3882&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3882&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=3882&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=3882&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=3882&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3882&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3882&resolved=false&types=CODE_SMELL) [0 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3882&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=3882&metric=coverage&view=list) No Coverage information [](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=3882&metric=duplicated_lines_density&view=list) No Duplication information Issue Time Tracking --- Worklog Id: (was: 837005) Time Spent: 2h 20m (was: 2h 10m) > Create qtest running QB compaction queries > -- > > Key: HIVE-26802 > URL: https://issues.apache.org/jira/browse/HIVE-26802 > Project: Hive > Issue Type: Improvement >Reporter: Zoltán Rátkai >Assignee: Zoltán Rátkai >Priority: Minor > Labels: pull-request-available > Time Spent: 2h 20m > Remaining Estimate: 0h > > Create a qtest that runs the queries that query-based compaction runs. > Not so much to check for correct data but more to check the query plans, to > simplify tracing changes in compilation that might affect QB compaction. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-26882) Allow transactional check of Table parameter before altering the Table
[
https://issues.apache.org/jira/browse/HIVE-26882?focusedWorklogId=837016&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-837016
]
ASF GitHub Bot logged work on HIVE-26882:
-
Author: ASF GitHub Bot
Created on: 04/Jan/23 19:47
Start Date: 04/Jan/23 19:47
Worklog Time Spent: 10m
Work Description: pvary commented on code in PR #3888:
URL: https://github.com/apache/hive/pull/3888#discussion_r1061829069
##
standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/client/TestTablesCreateDropAlterTruncate.java:
##
@@ -1183,6 +1193,90 @@ public void testAlterTableAlreadyExists() throws
Exception {
}
}
+ @Test
+ public void testAlterTableExpectedPropertyMatch() throws Exception {
+Table originalTable = testTables[0];
+
+EnvironmentContext context = new EnvironmentContext();
+context.putToProperties(hive_metastoreConstants.EXPECTED_PARAMETER_KEY,
"transient_lastDdlTime");
+context.putToProperties(hive_metastoreConstants.EXPECTED_PARAMETER_VALUE,
+originalTable.getParameters().get("transient_lastDdlTime"));
+
+client.alter_table(originalTable.getCatName(), originalTable.getDbName(),
originalTable.getTableName(),
+originalTable, context);
+ }
+
+ @Test(expected = MetaException.class)
+ public void testAlterTableExpectedPropertyDifferent() throws Exception {
+Table originalTable = testTables[0];
+
+EnvironmentContext context = new EnvironmentContext();
+context.putToProperties(hive_metastoreConstants.EXPECTED_PARAMETER_KEY,
"transient_lastDdlTime");
+context.putToProperties(hive_metastoreConstants.EXPECTED_PARAMETER_VALUE,
"alma");
+
+client.alter_table(originalTable.getCatName(), originalTable.getDbName(),
originalTable.getTableName(),
+originalTable, context);
+ }
+
+ /**
+ * This tests ensures that concurrent Iceberg commits will fail. Acceptable
as a first sanity check.
+ *
+ * I have not found a good way to check that HMS side database commits are
parallel in the
+ * {@link
org.apache.hadoop.hive.metastore.HiveAlterHandler#alterTable(RawStore,
Warehouse, String, String, String, Table, EnvironmentContext, IHMSHandler,
String)}
+ * call, but this test could be used to manually ensure that using
breakpoints.
+ */
+ @Test
+ public void testAlterTableExpectedPropertyConcurrent() throws Exception {
+Table originalTable = testTables[0];
+
+originalTable.getParameters().put("snapshot", "1");
+client.alter_table(originalTable.getCatName(), originalTable.getDbName(),
originalTable.getTableName(),
+originalTable, null);
+
+EnvironmentContext context = new EnvironmentContext();
Review Comment:
Added a short loop (3), since in Derby the concurrent change causes a
deadlock and the deadlock timeout causes the test to run slowly.
Issue Time Tracking
---
Worklog Id: (was: 837016)
Time Spent: 1h 50m (was: 1h 40m)
> Allow transactional check of Table parameter before altering the Table
> --
>
> Key: HIVE-26882
> URL: https://issues.apache.org/jira/browse/HIVE-26882
> Project: Hive
> Issue Type: Improvement
> Components: Standalone Metastore
>Reporter: Peter Vary
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h 50m
> Remaining Estimate: 0h
>
> We should add the possibility to transactionally check if a Table parameter
> is changed before altering the table in the HMS.
> This would provide an alternative, less error-prone and faster way to commit
> an Iceberg table, as the Iceberg table currently needs to:
> - Create an exclusive lock
> - Get the table metadata to check if the current snapshot is not changed
> - Update the table metadata
> - Release the lock
> After the change these 4 HMS calls could be substituted with a single alter
> table call.
> Also we could avoid cases where the locks are left hanging by failed processes
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-26894) After using scratchdir for staging final job, CTAS and IOW on ACID tables are failing with wrongFS exception
[ https://issues.apache.org/jira/browse/HIVE-26894?focusedWorklogId=837020&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-837020 ] ASF GitHub Bot logged work on HIVE-26894: - Author: ASF GitHub Bot Created on: 04/Jan/23 20:02 Start Date: 04/Jan/23 20:02 Worklog Time Spent: 10m Work Description: sonarcloud[bot] commented on PR #3909: URL: https://github.com/apache/hive/pull/3909#issuecomment-1371361113 Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=3909) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3909&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3909&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3909&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3909&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3909&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3909&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=3909&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=3909&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=3909&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3909&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3909&resolved=false&types=CODE_SMELL) [8 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3909&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=3909&metric=coverage&view=list) No Coverage information [](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=3909&metric=duplicated_lines_density&view=list) No Duplication information Issue Time Tracking --- Worklog Id: (was: 837020) Time Spent: 50m (was: 40m) > After using scratchdir for staging final job, CTAS and IOW on ACID tables are > failing with wrongFS exception > > > Key: HIVE-26894 > URL: https://issues.apache.org/jira/browse/HIVE-26894 > Project: Hive > Issue Type: Sub-task > Components: Hive >Affects Versions: 3.1.3 >Reporter: Sruthi M >Assignee: Sruthi M >Priority: Major > Labels: pull-request-available > Time Spent: 50m > Remaining Estimate: 0h > > ERROR : Failed with exception Wrong FS: > abfs:///hive/warehouse/managed/tpcds_orc.db/test_sales/delta_001_001_, > expected: hdfs://mycluster -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (HIVE-26872) INSERT UNION with LATERAL VIEW does not produce data
[
https://issues.apache.org/jira/browse/HIVE-26872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17654622#comment-17654622
]
Jan Filipiak commented on HIVE-26872:
-
https://issues.apache.org/jira/browse/HIVE-21100 this should fix it for you
> INSERT UNION with LATERAL VIEW does not produce data
> -
>
> Key: HIVE-26872
> URL: https://issues.apache.org/jira/browse/HIVE-26872
> Project: Hive
> Issue Type: Bug
> Components: Tez
>Affects Versions: 2.3.9, 3.1.2
>Reporter: FangBO
>Priority: Major
>
> {code:java}
> // code placeholder
> DROP TABLE union_test;
> CREATE TABLE union_test(id INT) PARTITIONED BY (`dt` STRING);
> DROP TABLE json_src;
> CREATE TABLE json_src(message STRING) PARTITIONED BY (`dt` STRING);
> INSERT OVERWRITE TABLE json_src PARTITION(dt='1219') VALUES('{"id":1}');
> INSERT OVERWRITE TABLE json_src PARTITION(dt='1220') VALUES('{"id":2}');
> INSERT OVERWRITE TABLE union_test PARTITION (dt='1221')
> SELECT id FROM json_src LATERAL VIEW json_tuple(message, 'id') b AS id WHERE
> dt='1219'
> UNION ALL
> SELECT id FROM json_src LATERAL VIEW json_tuple(message, 'id') b AS id WHERE
> dt='1220'
> ; {code}
> The script above does not produce data in partition dt='1221'
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-26869) Backport of HIVE-19104: When test MetaStore is started with retry the instances should be independent
[
https://issues.apache.org/jira/browse/HIVE-26869?focusedWorklogId=837033&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-837033
]
ASF GitHub Bot logged work on HIVE-26869:
-
Author: ASF GitHub Bot
Created on: 04/Jan/23 20:50
Start Date: 04/Jan/23 20:50
Worklog Time Spent: 10m
Work Description: amanraj2520 commented on PR #3873:
URL: https://github.com/apache/hive/pull/3873#issuecomment-1371404437
@cnauroth You suggestion worked. The concerned tests have passed
But now new tests are failing :
1. org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs
ERROR :
`
Error
Could not open client transport with JDBC Uri:
jdbc:hive2://localhost:42959/default;auth=delegationToken: Peer indicated
failure: DIGEST-MD5: IO error acquiring password
Stacktrace
java.sql.SQLException: Could not open client transport with JDBC Uri:
jdbc:hive2://localhost:42959/default;auth=delegationToken: Peer indicated
failure: DIGEST-MD5: IO error acquiring password
at org.apache.hive.jdbc.HiveConnection.(HiveConnection.java:269)
at org.apache.hive.jdbc.HiveDriver.connect(HiveDriver.java:107)
at java.sql.DriverManager.getConnection(DriverManager.java:664)
at java.sql.DriverManager.getConnection(DriverManager.java:270)
at
org.apache.hive.minikdc.TestJdbcWithMiniKdc.testTokenAuth(TestJdbcWithMiniKdc.java:172)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
`
3. udaf_context_ngrams.q (org.apache.hadoop.hive.cli.TestCliDriver)
ERROR :
`
Client Execution succeeded but contained differences (error code = 1) after
executing udaf_context_ngrams.q
25c25
<
[9.0,8.0,7.0,6.0,5.0,5.0,4.0,4.0,4.0,4.0,4.0,3.0,3.0,3.0,3.0,2.0,2.0,2.0,2.0,2.0,2.0,2.0,2.0,2.0,2.0,2.0,2.0,2.0,2.0,2.0,2.0,2.0,2.0,2.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0]
---
>
[267.0,171.0,164.0,119.0,108.0,106.0,106.0,82.0,79.0,67.0,67.0,46.0,45.0,42.0,42.0,40.0,39.0,37.0,37.0,36.0,34.0,32.0,32.0,30.0,30.0,29.0,28.0,28.0,28.0,28.0,26.0,25.0,24.0,23.0,23.0,22.0,22.0,21.0,20.0,19.0,18.0,18.0,18.0,17.0,17.0,17.0,16.0,16.0,16.0,16.0,15.0,15.0,14.0,14.0,14.0,13.0,13.0,13.0,13.0,13.0,13.0,13.0,12.0,12.0,12.0,12.0,12.0,12.0,12.0,11.0,11.0,11.0,11.0,11.0,11.0,11.0,10.0,10.0,10.0,10.0,10.0,10.0,10.0,10.0,10.0,10.0,10.0,9.0,9.0,9.0,9.0,9.0,9.0,9.0,9.0,9.0,9.0,9.0,9.0,8.0]
52c52
`
Issue Time Tracking
---
Worklog Id: (was: 837033)
Time Spent: 2h 40m (was: 2.5h)
> Backport of HIVE-19104: When test MetaStore is started with retry the
> instances should be independent
> -
>
> Key: HIVE-26869
> URL: https://issues.apache.org/jira/browse/HIVE-26869
> Project: Hive
> Issue Type: Sub-task
>Reporter: Aman Raj
>Assignee: Aman Raj
>Priority: Critical
> Labels: pull-request-available
> Time Spent: 2h 40m
> Remaining Estimate: 0h
>
> This fixes TestHS2ImpersonationWithRemoteMS which was failing with the
> following error :
> {code:java}
> [ERROR] Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed:
> 703.357 s <<< FAILURE! - in
> org.apache.hive.service.TestHS2ImpersonationWithRemoteMS
> [ERROR]
> testImpersonation(org.apache.hive.service.TestHS2ImpersonationWithRemoteMS)
> Time elapsed: 668.923 s <<< FAILURE!
> java.lang.AssertionError: Unexpected t
[jira] [Work logged] (HIVE-26869) Backport of HIVE-19104: When test MetaStore is started with retry the instances should be independent
[
https://issues.apache.org/jira/browse/HIVE-26869?focusedWorklogId=837034&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-837034
]
ASF GitHub Bot logged work on HIVE-26869:
-
Author: ASF GitHub Bot
Created on: 04/Jan/23 20:52
Start Date: 04/Jan/23 20:52
Worklog Time Spent: 10m
Work Description: amanraj2520 commented on PR #3873:
URL: https://github.com/apache/hive/pull/3873#issuecomment-1371405635
I found a fix for the first test in HIVE-19313
(https://issues.apache.org/jira/browse/HIVE-19313). Will cherry pick the same
Issue Time Tracking
---
Worklog Id: (was: 837034)
Time Spent: 2h 50m (was: 2h 40m)
> Backport of HIVE-19104: When test MetaStore is started with retry the
> instances should be independent
> -
>
> Key: HIVE-26869
> URL: https://issues.apache.org/jira/browse/HIVE-26869
> Project: Hive
> Issue Type: Sub-task
>Reporter: Aman Raj
>Assignee: Aman Raj
>Priority: Critical
> Labels: pull-request-available
> Time Spent: 2h 50m
> Remaining Estimate: 0h
>
> This fixes TestHS2ImpersonationWithRemoteMS which was failing with the
> following error :
> {code:java}
> [ERROR] Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed:
> 703.357 s <<< FAILURE! - in
> org.apache.hive.service.TestHS2ImpersonationWithRemoteMS
> [ERROR]
> testImpersonation(org.apache.hive.service.TestHS2ImpersonationWithRemoteMS)
> Time elapsed: 668.923 s <<< FAILURE!
> java.lang.AssertionError: Unexpected table directory '34015' in warehouse
> at org.junit.Assert.fail(Assert.java:88)
> at
> org.apache.hive.service.TestHS2ImpersonationWithRemoteMS.testImpersonation(TestHS2ImpersonationWithRemoteMS.java:115)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43){code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-26869) Backport of HIVE-19104: When test MetaStore is started with retry the instances should be independent
[
https://issues.apache.org/jira/browse/HIVE-26869?focusedWorklogId=837035&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-837035
]
ASF GitHub Bot logged work on HIVE-26869:
-
Author: ASF GitHub Bot
Created on: 04/Jan/23 21:00
Start Date: 04/Jan/23 21:00
Worklog Time Spent: 10m
Work Description: amanraj2520 commented on PR #3873:
URL: https://github.com/apache/hive/pull/3873#issuecomment-1371412554
Remaining tests which failed were disabled in
https://issues.apache.org/jira/browse/HIVE-20715 and
https://issues.apache.org/jira/browse/HIVE-20741 so cherry picked the same
Issue Time Tracking
---
Worklog Id: (was: 837035)
Time Spent: 3h (was: 2h 50m)
> Backport of HIVE-19104: When test MetaStore is started with retry the
> instances should be independent
> -
>
> Key: HIVE-26869
> URL: https://issues.apache.org/jira/browse/HIVE-26869
> Project: Hive
> Issue Type: Sub-task
>Reporter: Aman Raj
>Assignee: Aman Raj
>Priority: Critical
> Labels: pull-request-available
> Time Spent: 3h
> Remaining Estimate: 0h
>
> This fixes TestHS2ImpersonationWithRemoteMS which was failing with the
> following error :
> {code:java}
> [ERROR] Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed:
> 703.357 s <<< FAILURE! - in
> org.apache.hive.service.TestHS2ImpersonationWithRemoteMS
> [ERROR]
> testImpersonation(org.apache.hive.service.TestHS2ImpersonationWithRemoteMS)
> Time elapsed: 668.923 s <<< FAILURE!
> java.lang.AssertionError: Unexpected table directory '34015' in warehouse
> at org.junit.Assert.fail(Assert.java:88)
> at
> org.apache.hive.service.TestHS2ImpersonationWithRemoteMS.testImpersonation(TestHS2ImpersonationWithRemoteMS.java:115)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43){code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (HIVE-26880) Upgrade Apache Directory Server to 1.5.7 for release 3.2.
[ https://issues.apache.org/jira/browse/HIVE-26880?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17654647#comment-17654647 ] Chris Nauroth commented on HIVE-26880: -- Thanks very much, [~zabetak]! > Upgrade Apache Directory Server to 1.5.7 for release 3.2. > - > > Key: HIVE-26880 > URL: https://issues.apache.org/jira/browse/HIVE-26880 > Project: Hive > Issue Type: Improvement > Components: Test >Reporter: Chris Nauroth >Assignee: Chris Nauroth >Priority: Minor > Labels: hive-3.2.0-must, pull-request-available > Fix For: 3.2.0 > > Time Spent: 1h 20m > Remaining Estimate: 0h > > branch-3 uses Apache Directory Server in some tests. It currently uses > version 1.5.6. This version has a transitive dependency to a SNAPSHOT, making > it awkward to build and release. We can upgrade to 1.5.7 to remove the > SNAPSHOT dependency. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-26035) Explore moving to directsql for ObjectStore::addPartitions
[
https://issues.apache.org/jira/browse/HIVE-26035?focusedWorklogId=837059&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-837059
]
ASF GitHub Bot logged work on HIVE-26035:
-
Author: ASF GitHub Bot
Created on: 04/Jan/23 22:02
Start Date: 04/Jan/23 22:02
Worklog Time Spent: 10m
Work Description: sonarcloud[bot] commented on PR #3905:
URL: https://github.com/apache/hive/pull/3905#issuecomment-1371480128
Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=3905)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3905&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3905&resolved=false&types=BUG)
[0
Bugs](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3905&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3905&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3905&resolved=false&types=VULNERABILITY)
[0
Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3905&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=3905&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=3905&resolved=false&types=SECURITY_HOTSPOT)
[0 Security
Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=3905&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3905&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3905&resolved=false&types=CODE_SMELL)
[6 Code
Smells](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3905&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=3905&metric=coverage&view=list)
No Coverage information
[](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=3905&metric=duplicated_lines_density&view=list)
No Duplication information
Issue Time Tracking
---
Worklog Id: (was: 837059)
Time Spent: 40m (was: 0.5h)
> Explore moving to directsql for ObjectStore::addPartitions
> --
>
> Key: HIVE-26035
> URL: https://issues.apache.org/jira/browse/HIVE-26035
> Project: Hive
> Issue Type: Bug
>Reporter: Rajesh Balamohan
>Priority: Major
> Labels: pull-request-available
> Time Spent: 40m
> Remaining Estimate: 0h
>
> Currently {{addPartitions}} uses datanuclues and is super slow for large
> number of partitions. It will be good to move to direct sql. Lots of repeated
> SQLs can be avoided as well (e.g SDS, SERDE, TABLE_PARAMS)
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-26869) Backport of HIVE-19104: When test MetaStore is started with retry the instances should be independent
[
https://issues.apache.org/jira/browse/HIVE-26869?focusedWorklogId=837063&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-837063
]
ASF GitHub Bot logged work on HIVE-26869:
-
Author: ASF GitHub Bot
Created on: 04/Jan/23 22:18
Start Date: 04/Jan/23 22:18
Worklog Time Spent: 10m
Work Description: cnauroth commented on PR #3873:
URL: https://github.com/apache/hive/pull/3873#issuecomment-1371495446
Thanks, @amanraj2520 . It looks like you've picked up the cherry-pick I had
in mind as well as a few others. I don't know if committers would prefer to see
them split out individually for cleaner revision history. If so, I'll post my
PR for just the [HIVE-19104](https://issues.apache.org/jira/browse/HIVE-19104)
backport.
Issue Time Tracking
---
Worklog Id: (was: 837063)
Time Spent: 3h 10m (was: 3h)
> Backport of HIVE-19104: When test MetaStore is started with retry the
> instances should be independent
> -
>
> Key: HIVE-26869
> URL: https://issues.apache.org/jira/browse/HIVE-26869
> Project: Hive
> Issue Type: Sub-task
>Reporter: Aman Raj
>Assignee: Aman Raj
>Priority: Critical
> Labels: pull-request-available
> Time Spent: 3h 10m
> Remaining Estimate: 0h
>
> This fixes TestHS2ImpersonationWithRemoteMS which was failing with the
> following error :
> {code:java}
> [ERROR] Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed:
> 703.357 s <<< FAILURE! - in
> org.apache.hive.service.TestHS2ImpersonationWithRemoteMS
> [ERROR]
> testImpersonation(org.apache.hive.service.TestHS2ImpersonationWithRemoteMS)
> Time elapsed: 668.923 s <<< FAILURE!
> java.lang.AssertionError: Unexpected table directory '34015' in warehouse
> at org.junit.Assert.fail(Assert.java:88)
> at
> org.apache.hive.service.TestHS2ImpersonationWithRemoteMS.testImpersonation(TestHS2ImpersonationWithRemoteMS.java:115)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43){code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-26882) Allow transactional check of Table parameter before altering the Table
[ https://issues.apache.org/jira/browse/HIVE-26882?focusedWorklogId=837072&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-837072 ] ASF GitHub Bot logged work on HIVE-26882: - Author: ASF GitHub Bot Created on: 04/Jan/23 23:07 Start Date: 04/Jan/23 23:07 Worklog Time Spent: 10m Work Description: sonarcloud[bot] commented on PR #3888: URL: https://github.com/apache/hive/pull/3888#issuecomment-1371529261 Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=3888) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3888&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3888&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3888&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3888&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3888&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3888&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=3888&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=3888&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=3888&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3888&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3888&resolved=false&types=CODE_SMELL) [7 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3888&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=3888&metric=coverage&view=list) No Coverage information [](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=3888&metric=duplicated_lines_density&view=list) No Duplication information Issue Time Tracking --- Worklog Id: (was: 837072) Time Spent: 2h (was: 1h 50m) > Allow transactional check of Table parameter before altering the Table > -- > > Key: HIVE-26882 > URL: https://issues.apache.org/jira/browse/HIVE-26882 > Project: Hive > Issue Type: Improvement > Components: Standalone Metastore >Reporter: Peter Vary >Priority: Major > Labels: pull-request-available > Time Spent: 2h > Remaining Estimate: 0h > > We should add the possibility to transactionally check if a Table parameter > is changed before altering the table in the HMS. > This would provide an alternative, less error-prone and faster way to commit > an Iceberg table, as the Iceberg table currently needs to: > - Create an exclusive lock > - Get the table metadata to check if the current snapshot is not changed > - Update the table metadata > - Release the lock > After the change these 4 HMS calls could be substituted with a single alter > table call. > Also we could avoid cases where the locks are left han
[jira] [Work logged] (HIVE-26837) CTLT with hive.create.as.external.legacy as true creates managed table instead of external table
[
https://issues.apache.org/jira/browse/HIVE-26837?focusedWorklogId=837085&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-837085
]
ASF GitHub Bot logged work on HIVE-26837:
-
Author: ASF GitHub Bot
Created on: 04/Jan/23 23:43
Start Date: 04/Jan/23 23:43
Worklog Time Spent: 10m
Work Description: ramesh0201 commented on code in PR #3854:
URL: https://github.com/apache/hive/pull/3854#discussion_r1061974132
##
ql/src/test/results/clientpositive/llap/ctlt_translate_external.q.out:
##
@@ -0,0 +1,108 @@
+PREHOOK: query: create table test_mm(empno int, name string) partitioned
by(dept string) stored as orc tblproperties('transactional'='true',
'transactional_properties'='default')
+PREHOOK: type: CREATETABLE
+PREHOOK: Output: database:default
+PREHOOK: Output: default@test_mm
+POSTHOOK: query: create table test_mm(empno int, name string) partitioned
by(dept string) stored as orc tblproperties('transactional'='true',
'transactional_properties'='default')
+POSTHOOK: type: CREATETABLE
+POSTHOOK: Output: database:default
+POSTHOOK: Output: default@test_mm
+PREHOOK: query: create table test_external like test_mm
+PREHOOK: type: CREATETABLE
+PREHOOK: Output: database:default
+PREHOOK: Output: default@test_external
+POSTHOOK: query: create table test_external like test_mm
+POSTHOOK: type: CREATETABLE
+POSTHOOK: Output: database:default
+POSTHOOK: Output: default@test_external
+PREHOOK: query: desc formatted test_external
+PREHOOK: type: DESCTABLE
+PREHOOK: Input: default@test_external
+POSTHOOK: query: desc formatted test_external
+POSTHOOK: type: DESCTABLE
+POSTHOOK: Input: default@test_external
+# col_name data_type comment
+empno int
+name string
+
+# Partition Information
+# col_name data_type comment
+dept string
+
+# Detailed Table Information
+Database: default
+ A masked pattern was here
+Retention: 0
+ A masked pattern was here
+Table Type:MANAGED_TABLE
+Table Parameters:
+ COLUMN_STATS_ACCURATE {\"BASIC_STATS\":\"true\"}
+ bucketing_version 2
+ numFiles0
+ numPartitions 0
+ numRows 0
+ rawDataSize 0
+ totalSize 0
+ transactional true
+ transactional_propertiesdefault
+ A masked pattern was here
Review Comment:
@saihemanth-cloudera Can you review the latest?
Issue Time Tracking
---
Worklog Id: (was: 837085)
Time Spent: 1h 40m (was: 1.5h)
> CTLT with hive.create.as.external.legacy as true creates managed table
> instead of external table
>
>
> Key: HIVE-26837
> URL: https://issues.apache.org/jira/browse/HIVE-26837
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Reporter: Ramesh Kumar Thangarajan
>Assignee: Ramesh Kumar Thangarajan
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h 40m
> Remaining Estimate: 0h
>
> When CTLT is used with the config hive.create.as.external.legacy=true, it
> still creates managed table by default. Use below to reproduce.
> create external table test_ext(empno int, name string) partitioned by(dept
> string) stored as orc;
> desc formatted test_ext;
> set hive.create.as.external.legacy=true;
> create table test_external like test_ext;
> desc formatted test_external;
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-26837) CTLT with hive.create.as.external.legacy as true creates managed table instead of external table
[ https://issues.apache.org/jira/browse/HIVE-26837?focusedWorklogId=837093&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-837093 ] ASF GitHub Bot logged work on HIVE-26837: - Author: ASF GitHub Bot Created on: 05/Jan/23 00:17 Start Date: 05/Jan/23 00:17 Worklog Time Spent: 10m Work Description: saihemanth-cloudera commented on PR #3854: URL: https://github.com/apache/hive/pull/3854#issuecomment-1371572142 Pending green jenkins tests. Issue Time Tracking --- Worklog Id: (was: 837093) Time Spent: 1h 50m (was: 1h 40m) > CTLT with hive.create.as.external.legacy as true creates managed table > instead of external table > > > Key: HIVE-26837 > URL: https://issues.apache.org/jira/browse/HIVE-26837 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Reporter: Ramesh Kumar Thangarajan >Assignee: Ramesh Kumar Thangarajan >Priority: Major > Labels: pull-request-available > Time Spent: 1h 50m > Remaining Estimate: 0h > > When CTLT is used with the config hive.create.as.external.legacy=true, it > still creates managed table by default. Use below to reproduce. > create external table test_ext(empno int, name string) partitioned by(dept > string) stored as orc; > desc formatted test_ext; > set hive.create.as.external.legacy=true; > create table test_external like test_ext; > desc formatted test_external; -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-26837) CTLT with hive.create.as.external.legacy as true creates managed table instead of external table
[ https://issues.apache.org/jira/browse/HIVE-26837?focusedWorklogId=837096&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-837096 ] ASF GitHub Bot logged work on HIVE-26837: - Author: ASF GitHub Bot Created on: 05/Jan/23 00:49 Start Date: 05/Jan/23 00:49 Worklog Time Spent: 10m Work Description: sonarcloud[bot] commented on PR #3854: URL: https://github.com/apache/hive/pull/3854#issuecomment-1371601883 Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=3854) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3854&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3854&resolved=false&types=BUG) [1 Bug](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3854&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3854&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3854&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3854&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=3854&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=3854&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=3854&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3854&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3854&resolved=false&types=CODE_SMELL) [0 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3854&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=3854&metric=coverage&view=list) No Coverage information [](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=3854&metric=duplicated_lines_density&view=list) No Duplication information Issue Time Tracking --- Worklog Id: (was: 837096) Time Spent: 2h (was: 1h 50m) > CTLT with hive.create.as.external.legacy as true creates managed table > instead of external table > > > Key: HIVE-26837 > URL: https://issues.apache.org/jira/browse/HIVE-26837 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Reporter: Ramesh Kumar Thangarajan >Assignee: Ramesh Kumar Thangarajan >Priority: Major > Labels: pull-request-available > Time Spent: 2h > Remaining Estimate: 0h > > When CTLT is used with the config hive.create.as.external.legacy=true, it > still creates managed table by default. Use below to reproduce. > create external table test_ext(empno int, name string) partitioned by(dept > string) stored as orc; > desc formatted test_ext; > set hive.create.as.external.legacy=true; > create table test_external like test_ext; > desc formatted test_external; -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-26837) CTLT with hive.create.as.external.legacy as true creates managed table instead of external table
[ https://issues.apache.org/jira/browse/HIVE-26837?focusedWorklogId=837123&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-837123 ] ASF GitHub Bot logged work on HIVE-26837: - Author: ASF GitHub Bot Created on: 05/Jan/23 04:38 Start Date: 05/Jan/23 04:38 Worklog Time Spent: 10m Work Description: sonarcloud[bot] commented on PR #3854: URL: https://github.com/apache/hive/pull/3854#issuecomment-1371773551 Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=3854) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3854&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3854&resolved=false&types=BUG) [1 Bug](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3854&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3854&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3854&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3854&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=3854&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=3854&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=3854&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3854&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3854&resolved=false&types=CODE_SMELL) [0 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3854&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=3854&metric=coverage&view=list) No Coverage information [](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=3854&metric=duplicated_lines_density&view=list) No Duplication information Issue Time Tracking --- Worklog Id: (was: 837123) Time Spent: 2h 10m (was: 2h) > CTLT with hive.create.as.external.legacy as true creates managed table > instead of external table > > > Key: HIVE-26837 > URL: https://issues.apache.org/jira/browse/HIVE-26837 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Reporter: Ramesh Kumar Thangarajan >Assignee: Ramesh Kumar Thangarajan >Priority: Major > Labels: pull-request-available > Time Spent: 2h 10m > Remaining Estimate: 0h > > When CTLT is used with the config hive.create.as.external.legacy=true, it > still creates managed table by default. Use below to reproduce. > create external table test_ext(empno int, name string) partitioned by(dept > string) stored as orc; > desc formatted test_ext; > set hive.create.as.external.legacy=true; > create table test_external like test_ext; > desc formatted test_external; -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (HIVE-26872) INSERT UNION with LATERAL VIEW does not produce data
[
https://issues.apache.org/jira/browse/HIVE-26872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17654738#comment-17654738
]
FangBO commented on HIVE-26872:
---
resolved by https://issues.apache.org/jira/browse/HIVE-21660
> INSERT UNION with LATERAL VIEW does not produce data
> -
>
> Key: HIVE-26872
> URL: https://issues.apache.org/jira/browse/HIVE-26872
> Project: Hive
> Issue Type: Bug
> Components: Tez
>Affects Versions: 2.3.9, 3.1.2
>Reporter: FangBO
>Priority: Major
>
> {code:java}
> // code placeholder
> DROP TABLE union_test;
> CREATE TABLE union_test(id INT) PARTITIONED BY (`dt` STRING);
> DROP TABLE json_src;
> CREATE TABLE json_src(message STRING) PARTITIONED BY (`dt` STRING);
> INSERT OVERWRITE TABLE json_src PARTITION(dt='1219') VALUES('{"id":1}');
> INSERT OVERWRITE TABLE json_src PARTITION(dt='1220') VALUES('{"id":2}');
> INSERT OVERWRITE TABLE union_test PARTITION (dt='1221')
> SELECT id FROM json_src LATERAL VIEW json_tuple(message, 'id') b AS id WHERE
> dt='1219'
> UNION ALL
> SELECT id FROM json_src LATERAL VIEW json_tuple(message, 'id') b AS id WHERE
> dt='1220'
> ; {code}
> The script above does not produce data in partition dt='1221'
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (HIVE-26872) INSERT UNION with LATERAL VIEW does not produce data
[
https://issues.apache.org/jira/browse/HIVE-26872?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
FangBO resolved HIVE-26872.
---
Resolution: Duplicate
> INSERT UNION with LATERAL VIEW does not produce data
> -
>
> Key: HIVE-26872
> URL: https://issues.apache.org/jira/browse/HIVE-26872
> Project: Hive
> Issue Type: Bug
> Components: Tez
>Affects Versions: 2.3.9, 3.1.2
>Reporter: FangBO
>Priority: Major
>
> {code:java}
> // code placeholder
> DROP TABLE union_test;
> CREATE TABLE union_test(id INT) PARTITIONED BY (`dt` STRING);
> DROP TABLE json_src;
> CREATE TABLE json_src(message STRING) PARTITIONED BY (`dt` STRING);
> INSERT OVERWRITE TABLE json_src PARTITION(dt='1219') VALUES('{"id":1}');
> INSERT OVERWRITE TABLE json_src PARTITION(dt='1220') VALUES('{"id":2}');
> INSERT OVERWRITE TABLE union_test PARTITION (dt='1221')
> SELECT id FROM json_src LATERAL VIEW json_tuple(message, 'id') b AS id WHERE
> dt='1219'
> UNION ALL
> SELECT id FROM json_src LATERAL VIEW json_tuple(message, 'id') b AS id WHERE
> dt='1220'
> ; {code}
> The script above does not produce data in partition dt='1221'
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Assigned] (HIVE-26905) Backport HIVE-25173 to 3.2.0: Exclude pentaho-aggdesigner-algorithm from upgrade-acid build.
[ https://issues.apache.org/jira/browse/HIVE-26905?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Nauroth reassigned HIVE-26905: > Backport HIVE-25173 to 3.2.0: Exclude pentaho-aggdesigner-algorithm from > upgrade-acid build. > > > Key: HIVE-26905 > URL: https://issues.apache.org/jira/browse/HIVE-26905 > Project: Hive > Issue Type: Bug > Components: Build Infrastructure >Reporter: Chris Nauroth >Assignee: Chris Nauroth >Priority: Major > Labels: hive-3.2.0-must > > In the current branch-3, upgrade-acid has a dependency on an old hive-exec > version that has a transitive dependency to > org.pentaho:pentaho-aggdesigner-algorithm. This artifact is no longer > available in commonly supported Maven repositories, which causes a build > failure. We can safely exclude the dependency, as was originally done in > HIVE-25173. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-26905) Backport HIVE-25173 to 3.2.0: Exclude pentaho-aggdesigner-algorithm from upgrade-acid build.
[ https://issues.apache.org/jira/browse/HIVE-26905?focusedWorklogId=837128&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-837128 ] ASF GitHub Bot logged work on HIVE-26905: - Author: ASF GitHub Bot Created on: 05/Jan/23 05:27 Start Date: 05/Jan/23 05:27 Worklog Time Spent: 10m Work Description: cnauroth commented on PR #3911: URL: https://github.com/apache/hive/pull/3911#issuecomment-1371797763 @amanraj2520 , FYI on another branch-3 patch. Issue Time Tracking --- Worklog Id: (was: 837128) Time Spent: 20m (was: 10m) > Backport HIVE-25173 to 3.2.0: Exclude pentaho-aggdesigner-algorithm from > upgrade-acid build. > > > Key: HIVE-26905 > URL: https://issues.apache.org/jira/browse/HIVE-26905 > Project: Hive > Issue Type: Bug > Components: Build Infrastructure >Reporter: Chris Nauroth >Assignee: Chris Nauroth >Priority: Major > Labels: hive-3.2.0-must, pull-request-available > Time Spent: 20m > Remaining Estimate: 0h > > In the current branch-3, upgrade-acid has a dependency on an old hive-exec > version that has a transitive dependency to > org.pentaho:pentaho-aggdesigner-algorithm. This artifact is no longer > available in commonly supported Maven repositories, which causes a build > failure. We can safely exclude the dependency, as was originally done in > HIVE-25173. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HIVE-26905) Backport HIVE-25173 to 3.2.0: Exclude pentaho-aggdesigner-algorithm from upgrade-acid build.
[ https://issues.apache.org/jira/browse/HIVE-26905?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HIVE-26905: -- Labels: hive-3.2.0-must pull-request-available (was: hive-3.2.0-must) > Backport HIVE-25173 to 3.2.0: Exclude pentaho-aggdesigner-algorithm from > upgrade-acid build. > > > Key: HIVE-26905 > URL: https://issues.apache.org/jira/browse/HIVE-26905 > Project: Hive > Issue Type: Bug > Components: Build Infrastructure >Reporter: Chris Nauroth >Assignee: Chris Nauroth >Priority: Major > Labels: hive-3.2.0-must, pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > In the current branch-3, upgrade-acid has a dependency on an old hive-exec > version that has a transitive dependency to > org.pentaho:pentaho-aggdesigner-algorithm. This artifact is no longer > available in commonly supported Maven repositories, which causes a build > failure. We can safely exclude the dependency, as was originally done in > HIVE-25173. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-26905) Backport HIVE-25173 to 3.2.0: Exclude pentaho-aggdesigner-algorithm from upgrade-acid build.
[ https://issues.apache.org/jira/browse/HIVE-26905?focusedWorklogId=837127&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-837127 ] ASF GitHub Bot logged work on HIVE-26905: - Author: ASF GitHub Bot Created on: 05/Jan/23 05:27 Start Date: 05/Jan/23 05:27 Worklog Time Spent: 10m Work Description: cnauroth opened a new pull request, #3911: URL: https://github.com/apache/hive/pull/3911 ### What changes were proposed in this pull request? Exclude pentaho-aggdesigner-algorithm from upgrade-acid build. ### Why are the changes needed? In the current branch-3, upgrade-acid has a dependency on an old hive-exec version that has a transitive dependency to org.pentaho:pentaho-aggdesigner-algorithm. This artifact is no longer available in commonly supported Maven repositories, which causes a build failure. We can safely exclude the dependency, as was originally done in [HIVE-25173](https://issues.apache.org/jira/browse/HIVE-25173). Differences from the master patch branch are: 1. On master, this applied to the pre-upgrade sub-module. This sub-module doesn't exist in branch-3, so the patch was rebased to the parent upgrade-acid module. 2. Additionally, the pom.xml code had changed quite a bit on master. This is just applying the equivalent exclusion from the HIVE-25173 diff: a1d4c8a6b3cf8465ac1ae074748a8f5a04bb473f. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? I can run a full local build from branch-3 after applying this patch. ``` mvn -B -T 8 clean install -Pitests -DskipTests ``` Prior to this patch, my build failed while trying to download the org.pentaho:pentaho-aggdesigner-algorithm artifact. Issue Time Tracking --- Worklog Id: (was: 837127) Remaining Estimate: 0h Time Spent: 10m > Backport HIVE-25173 to 3.2.0: Exclude pentaho-aggdesigner-algorithm from > upgrade-acid build. > > > Key: HIVE-26905 > URL: https://issues.apache.org/jira/browse/HIVE-26905 > Project: Hive > Issue Type: Bug > Components: Build Infrastructure >Reporter: Chris Nauroth >Assignee: Chris Nauroth >Priority: Major > Labels: hive-3.2.0-must > Time Spent: 10m > Remaining Estimate: 0h > > In the current branch-3, upgrade-acid has a dependency on an old hive-exec > version that has a transitive dependency to > org.pentaho:pentaho-aggdesigner-algorithm. This artifact is no longer > available in commonly supported Maven repositories, which causes a build > failure. We can safely exclude the dependency, as was originally done in > HIVE-25173. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-26711) The very first REPL Load should make the Target Database read-only
[ https://issues.apache.org/jira/browse/HIVE-26711?focusedWorklogId=837139&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-837139 ] ASF GitHub Bot logged work on HIVE-26711: - Author: ASF GitHub Bot Created on: 05/Jan/23 06:25 Start Date: 05/Jan/23 06:25 Worklog Time Spent: 10m Work Description: sonarcloud[bot] commented on PR #3736: URL: https://github.com/apache/hive/pull/3736#issuecomment-1371830690 Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=3736) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3736&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3736&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3736&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3736&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3736&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3736&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=3736&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=3736&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=3736&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3736&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3736&resolved=false&types=CODE_SMELL) [0 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3736&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=3736&metric=coverage&view=list) No Coverage information [](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=3736&metric=duplicated_lines_density&view=list) No Duplication information Issue Time Tracking --- Worklog Id: (was: 837139) Time Spent: 40m (was: 0.5h) > The very first REPL Load should make the Target Database read-only > -- > > Key: HIVE-26711 > URL: https://issues.apache.org/jira/browse/HIVE-26711 > Project: Hive > Issue Type: Task >Reporter: Shreenidhi >Assignee: Shreenidhi >Priority: Major > Labels: pull-request-available > Time Spent: 40m > Remaining Estimate: 0h > > Use EnforceReadOnly hook to set TARGET database read only during BootStrap > load. > Also ensure backward compatibility. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-26869) Backport of HIVE-19104: When test MetaStore is started with retry the instances should be independent
[
https://issues.apache.org/jira/browse/HIVE-26869?focusedWorklogId=837142&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-837142
]
ASF GitHub Bot logged work on HIVE-26869:
-
Author: ASF GitHub Bot
Created on: 05/Jan/23 06:30
Start Date: 05/Jan/23 06:30
Worklog Time Spent: 10m
Work Description: amanraj2520 commented on PR #3873:
URL: https://github.com/apache/hive/pull/3873#issuecomment-1371833492
@cnauroth I agree with you. I was going to revert the additional commits
just was making sure that these test work fine when combined together. This PR
tests that. This PR will now going to be for HIVE-19104. Will raise new tickets.
Issue Time Tracking
---
Worklog Id: (was: 837142)
Time Spent: 3h 20m (was: 3h 10m)
> Backport of HIVE-19104: When test MetaStore is started with retry the
> instances should be independent
> -
>
> Key: HIVE-26869
> URL: https://issues.apache.org/jira/browse/HIVE-26869
> Project: Hive
> Issue Type: Sub-task
>Reporter: Aman Raj
>Assignee: Aman Raj
>Priority: Critical
> Labels: pull-request-available
> Time Spent: 3h 20m
> Remaining Estimate: 0h
>
> This fixes TestHS2ImpersonationWithRemoteMS which was failing with the
> following error :
> {code:java}
> [ERROR] Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed:
> 703.357 s <<< FAILURE! - in
> org.apache.hive.service.TestHS2ImpersonationWithRemoteMS
> [ERROR]
> testImpersonation(org.apache.hive.service.TestHS2ImpersonationWithRemoteMS)
> Time elapsed: 668.923 s <<< FAILURE!
> java.lang.AssertionError: Unexpected table directory '34015' in warehouse
> at org.junit.Assert.fail(Assert.java:88)
> at
> org.apache.hive.service.TestHS2ImpersonationWithRemoteMS.testImpersonation(TestHS2ImpersonationWithRemoteMS.java:115)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43){code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-26869) Backport of HIVE-19104: When test MetaStore is started with retry the instances should be independent
[
https://issues.apache.org/jira/browse/HIVE-26869?focusedWorklogId=837143&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-837143
]
ASF GitHub Bot logged work on HIVE-26869:
-
Author: ASF GitHub Bot
Created on: 05/Jan/23 06:33
Start Date: 05/Jan/23 06:33
Worklog Time Spent: 10m
Work Description: amanraj2520 commented on PR #3873:
URL: https://github.com/apache/hive/pull/3873#issuecomment-1371835089
@cnauroth @zabetak @abstractdog I have done the changes from my side and
tested it in my local as well. Can you please approve and merge this PR.
Issue Time Tracking
---
Worklog Id: (was: 837143)
Time Spent: 3.5h (was: 3h 20m)
> Backport of HIVE-19104: When test MetaStore is started with retry the
> instances should be independent
> -
>
> Key: HIVE-26869
> URL: https://issues.apache.org/jira/browse/HIVE-26869
> Project: Hive
> Issue Type: Sub-task
>Reporter: Aman Raj
>Assignee: Aman Raj
>Priority: Critical
> Labels: pull-request-available
> Time Spent: 3.5h
> Remaining Estimate: 0h
>
> This fixes TestHS2ImpersonationWithRemoteMS which was failing with the
> following error :
> {code:java}
> [ERROR] Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed:
> 703.357 s <<< FAILURE! - in
> org.apache.hive.service.TestHS2ImpersonationWithRemoteMS
> [ERROR]
> testImpersonation(org.apache.hive.service.TestHS2ImpersonationWithRemoteMS)
> Time elapsed: 668.923 s <<< FAILURE!
> java.lang.AssertionError: Unexpected table directory '34015' in warehouse
> at org.junit.Assert.fail(Assert.java:88)
> at
> org.apache.hive.service.TestHS2ImpersonationWithRemoteMS.testImpersonation(TestHS2ImpersonationWithRemoteMS.java:115)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43){code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Assigned] (HIVE-26906) Backport of HIVE-19313 to branch-3 : TestJdbcWithDBTokenStoreNoDoAs tests are failing
[
https://issues.apache.org/jira/browse/HIVE-26906?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aman Raj reassigned HIVE-26906:
---
> Backport of HIVE-19313 to branch-3 : TestJdbcWithDBTokenStoreNoDoAs tests are
> failing
> -
>
> Key: HIVE-26906
> URL: https://issues.apache.org/jira/browse/HIVE-26906
> Project: Hive
> Issue Type: Sub-task
>Reporter: Aman Raj
>Assignee: Aman Raj
>Priority: Critical
>
> # org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs
> ERROR :
> {{Error Could not open client transport with JDBC Uri:
> jdbc:hive2://localhost:42959/default;auth=delegationToken: Peer indicated
> failure: DIGEST-MD5: IO error acquiring password Stacktrace
> java.sql.SQLException: Could not open client transport with JDBC Uri:
> jdbc:hive2://localhost:42959/default;auth=delegationToken: Peer indicated
> failure: DIGEST-MD5: IO error acquiring password at
> org.apache.hive.jdbc.HiveConnection.(HiveConnection.java:269) at
> org.apache.hive.jdbc.HiveDriver.connect(HiveDriver.java:107) at
> java.sql.DriverManager.getConnection(DriverManager.java:664) at
> java.sql.DriverManager.getConnection(DriverManager.java:270) at
> org.apache.hive.minikdc.TestJdbcWithMiniKdc.testTokenAuth(TestJdbcWithMiniKdc.java:172)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498) at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271) at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)}}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-26906) Backport of HIVE-19313 to branch-3 : TestJdbcWithDBTokenStoreNoDoAs tests are failing
[
https://issues.apache.org/jira/browse/HIVE-26906?focusedWorklogId=837146&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-837146
]
ASF GitHub Bot logged work on HIVE-26906:
-
Author: ASF GitHub Bot
Created on: 05/Jan/23 06:42
Start Date: 05/Jan/23 06:42
Worklog Time Spent: 10m
Work Description: amanraj2520 opened a new pull request, #3912:
URL: https://github.com/apache/hive/pull/3912
JIRA link : https://issues.apache.org/jira/browse/HIVE-26906
### What changes were proposed in this pull request?
### Why are the changes needed?
### Does this PR introduce _any_ user-facing change?
### How was this patch tested?
Issue Time Tracking
---
Worklog Id: (was: 837146)
Remaining Estimate: 0h
Time Spent: 10m
> Backport of HIVE-19313 to branch-3 : TestJdbcWithDBTokenStoreNoDoAs tests are
> failing
> -
>
> Key: HIVE-26906
> URL: https://issues.apache.org/jira/browse/HIVE-26906
> Project: Hive
> Issue Type: Sub-task
>Reporter: Aman Raj
>Assignee: Aman Raj
>Priority: Critical
> Time Spent: 10m
> Remaining Estimate: 0h
>
> # org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs
> ERROR :
> {{Error Could not open client transport with JDBC Uri:
> jdbc:hive2://localhost:42959/default;auth=delegationToken: Peer indicated
> failure: DIGEST-MD5: IO error acquiring password Stacktrace
> java.sql.SQLException: Could not open client transport with JDBC Uri:
> jdbc:hive2://localhost:42959/default;auth=delegationToken: Peer indicated
> failure: DIGEST-MD5: IO error acquiring password at
> org.apache.hive.jdbc.HiveConnection.(HiveConnection.java:269) at
> org.apache.hive.jdbc.HiveDriver.connect(HiveDriver.java:107) at
> java.sql.DriverManager.getConnection(DriverManager.java:664) at
> java.sql.DriverManager.getConnection(DriverManager.java:270) at
> org.apache.hive.minikdc.TestJdbcWithMiniKdc.testTokenAuth(TestJdbcWithMiniKdc.java:172)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498) at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271) at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)}}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (HIVE-26906) Backport of HIVE-19313 to branch-3 : TestJdbcWithDBTokenStoreNoDoAs tests are failing
[
https://issues.apache.org/jira/browse/HIVE-26906?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated HIVE-26906:
--
Labels: pull-request-available (was: )
> Backport of HIVE-19313 to branch-3 : TestJdbcWithDBTokenStoreNoDoAs tests are
> failing
> -
>
> Key: HIVE-26906
> URL: https://issues.apache.org/jira/browse/HIVE-26906
> Project: Hive
> Issue Type: Sub-task
>Reporter: Aman Raj
>Assignee: Aman Raj
>Priority: Critical
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> # org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs
> ERROR :
> {{Error Could not open client transport with JDBC Uri:
> jdbc:hive2://localhost:42959/default;auth=delegationToken: Peer indicated
> failure: DIGEST-MD5: IO error acquiring password Stacktrace
> java.sql.SQLException: Could not open client transport with JDBC Uri:
> jdbc:hive2://localhost:42959/default;auth=delegationToken: Peer indicated
> failure: DIGEST-MD5: IO error acquiring password at
> org.apache.hive.jdbc.HiveConnection.(HiveConnection.java:269) at
> org.apache.hive.jdbc.HiveDriver.connect(HiveDriver.java:107) at
> java.sql.DriverManager.getConnection(DriverManager.java:664) at
> java.sql.DriverManager.getConnection(DriverManager.java:270) at
> org.apache.hive.minikdc.TestJdbcWithMiniKdc.testTokenAuth(TestJdbcWithMiniKdc.java:172)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498) at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271) at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)}}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-26906) Backport of HIVE-19313 to branch-3 : TestJdbcWithDBTokenStoreNoDoAs tests are failing
[
https://issues.apache.org/jira/browse/HIVE-26906?focusedWorklogId=837147&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-837147
]
ASF GitHub Bot logged work on HIVE-26906:
-
Author: ASF GitHub Bot
Created on: 05/Jan/23 06:42
Start Date: 05/Jan/23 06:42
Worklog Time Spent: 10m
Work Description: amanraj2520 commented on PR #3912:
URL: https://github.com/apache/hive/pull/3912#issuecomment-1371840163
@cnauroth FYI raised this a separate PR.
Issue Time Tracking
---
Worklog Id: (was: 837147)
Time Spent: 20m (was: 10m)
> Backport of HIVE-19313 to branch-3 : TestJdbcWithDBTokenStoreNoDoAs tests are
> failing
> -
>
> Key: HIVE-26906
> URL: https://issues.apache.org/jira/browse/HIVE-26906
> Project: Hive
> Issue Type: Sub-task
>Reporter: Aman Raj
>Assignee: Aman Raj
>Priority: Critical
> Labels: pull-request-available
> Time Spent: 20m
> Remaining Estimate: 0h
>
> # org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs
> ERROR :
> {{Error Could not open client transport with JDBC Uri:
> jdbc:hive2://localhost:42959/default;auth=delegationToken: Peer indicated
> failure: DIGEST-MD5: IO error acquiring password Stacktrace
> java.sql.SQLException: Could not open client transport with JDBC Uri:
> jdbc:hive2://localhost:42959/default;auth=delegationToken: Peer indicated
> failure: DIGEST-MD5: IO error acquiring password at
> org.apache.hive.jdbc.HiveConnection.(HiveConnection.java:269) at
> org.apache.hive.jdbc.HiveDriver.connect(HiveDriver.java:107) at
> java.sql.DriverManager.getConnection(DriverManager.java:664) at
> java.sql.DriverManager.getConnection(DriverManager.java:270) at
> org.apache.hive.minikdc.TestJdbcWithMiniKdc.testTokenAuth(TestJdbcWithMiniKdc.java:172)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498) at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271) at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)}}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Assigned] (HIVE-26907) Backport of HIVE-20741: Disable udaf_context_ngrams.q and udaf_corr.q tests
[ https://issues.apache.org/jira/browse/HIVE-26907?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aman Raj reassigned HIVE-26907: --- > Backport of HIVE-20741: Disable udaf_context_ngrams.q and udaf_corr.q tests > --- > > Key: HIVE-26907 > URL: https://issues.apache.org/jira/browse/HIVE-26907 > Project: Hive > Issue Type: Sub-task >Reporter: Aman Raj >Assignee: Aman Raj >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-26907) Backport of HIVE-20741: Disable udaf_context_ngrams.q and udaf_corr.q tests
[ https://issues.apache.org/jira/browse/HIVE-26907?focusedWorklogId=837150&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-837150 ] ASF GitHub Bot logged work on HIVE-26907: - Author: ASF GitHub Bot Created on: 05/Jan/23 06:46 Start Date: 05/Jan/23 06:46 Worklog Time Spent: 10m Work Description: amanraj2520 commented on PR #3913: URL: https://github.com/apache/hive/pull/3913#issuecomment-1371842246 @cnauroth FYI Issue Time Tracking --- Worklog Id: (was: 837150) Time Spent: 20m (was: 10m) > Backport of HIVE-20741: Disable udaf_context_ngrams.q and udaf_corr.q tests > --- > > Key: HIVE-26907 > URL: https://issues.apache.org/jira/browse/HIVE-26907 > Project: Hive > Issue Type: Sub-task >Reporter: Aman Raj >Assignee: Aman Raj >Priority: Major > Labels: pull-request-available > Time Spent: 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HIVE-26907) Backport of HIVE-20741: Disable udaf_context_ngrams.q and udaf_corr.q tests
[ https://issues.apache.org/jira/browse/HIVE-26907?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HIVE-26907: -- Labels: pull-request-available (was: ) > Backport of HIVE-20741: Disable udaf_context_ngrams.q and udaf_corr.q tests > --- > > Key: HIVE-26907 > URL: https://issues.apache.org/jira/browse/HIVE-26907 > Project: Hive > Issue Type: Sub-task >Reporter: Aman Raj >Assignee: Aman Raj >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-26907) Backport of HIVE-20741: Disable udaf_context_ngrams.q and udaf_corr.q tests
[ https://issues.apache.org/jira/browse/HIVE-26907?focusedWorklogId=837149&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-837149 ] ASF GitHub Bot logged work on HIVE-26907: - Author: ASF GitHub Bot Created on: 05/Jan/23 06:46 Start Date: 05/Jan/23 06:46 Worklog Time Spent: 10m Work Description: amanraj2520 opened a new pull request, #3913: URL: https://github.com/apache/hive/pull/3913 JIRA link : https://issues.apache.org/jira/browse/HIVE-26907 ### What changes were proposed in this pull request? ### Why are the changes needed? ### Does this PR introduce _any_ user-facing change? ### How was this patch tested? Issue Time Tracking --- Worklog Id: (was: 837149) Remaining Estimate: 0h Time Spent: 10m > Backport of HIVE-20741: Disable udaf_context_ngrams.q and udaf_corr.q tests > --- > > Key: HIVE-26907 > URL: https://issues.apache.org/jira/browse/HIVE-26907 > Project: Hive > Issue Type: Sub-task >Reporter: Aman Raj >Assignee: Aman Raj >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (HIVE-26908) Disable Initiator on HMS instance at the same time enable Cleaner thread
[
https://issues.apache.org/jira/browse/HIVE-26908?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Taraka Rama Rao Lethavadla reassigned HIVE-26908:
-
> Disable Initiator on HMS instance at the same time enable Cleaner thread
>
>
> Key: HIVE-26908
> URL: https://issues.apache.org/jira/browse/HIVE-26908
> Project: Hive
> Issue Type: New Feature
> Components: Standalone Metastore
>Reporter: Taraka Rama Rao Lethavadla
>Assignee: Taraka Rama Rao Lethavadla
>Priority: Major
>
> In the current implementation, both Initiator and Cleaner are either enabled
> or disabled using the same config
> {noformat}
> hive.compactor.initiator.on{noformat}
> So there is no way to selectively disable initiator and enable cleaner or
> vice versa.
> Introducing another config to handle Cleaner thread alone like
> hive.compactor.cleaner.on
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Assigned] (HIVE-26909) Backport of HIVE-20715: Disable test: udaf_histogram_numeric
[ https://issues.apache.org/jira/browse/HIVE-26909?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aman Raj reassigned HIVE-26909: --- > Backport of HIVE-20715: Disable test: udaf_histogram_numeric > - > > Key: HIVE-26909 > URL: https://issues.apache.org/jira/browse/HIVE-26909 > Project: Hive > Issue Type: Sub-task >Reporter: Aman Raj >Assignee: Aman Raj >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HIVE-26909) Backport of HIVE-20715: Disable test: udaf_histogram_numeric
[ https://issues.apache.org/jira/browse/HIVE-26909?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HIVE-26909: -- Labels: pull-request-available (was: ) > Backport of HIVE-20715: Disable test: udaf_histogram_numeric > - > > Key: HIVE-26909 > URL: https://issues.apache.org/jira/browse/HIVE-26909 > Project: Hive > Issue Type: Sub-task >Reporter: Aman Raj >Assignee: Aman Raj >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-26909) Backport of HIVE-20715: Disable test: udaf_histogram_numeric
[ https://issues.apache.org/jira/browse/HIVE-26909?focusedWorklogId=837151&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-837151 ] ASF GitHub Bot logged work on HIVE-26909: - Author: ASF GitHub Bot Created on: 05/Jan/23 07:00 Start Date: 05/Jan/23 07:00 Worklog Time Spent: 10m Work Description: amanraj2520 opened a new pull request, #3914: URL: https://github.com/apache/hive/pull/3914 JIRA link : https://issues.apache.org/jira/browse/HIVE-26909 ### What changes were proposed in this pull request? ### Why are the changes needed? ### Does this PR introduce _any_ user-facing change? ### How was this patch tested? Issue Time Tracking --- Worklog Id: (was: 837151) Remaining Estimate: 0h Time Spent: 10m > Backport of HIVE-20715: Disable test: udaf_histogram_numeric > - > > Key: HIVE-26909 > URL: https://issues.apache.org/jira/browse/HIVE-26909 > Project: Hive > Issue Type: Sub-task >Reporter: Aman Raj >Assignee: Aman Raj >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-26909) Backport of HIVE-20715: Disable test: udaf_histogram_numeric
[ https://issues.apache.org/jira/browse/HIVE-26909?focusedWorklogId=837152&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-837152 ] ASF GitHub Bot logged work on HIVE-26909: - Author: ASF GitHub Bot Created on: 05/Jan/23 07:01 Start Date: 05/Jan/23 07:01 Worklog Time Spent: 10m Work Description: amanraj2520 commented on PR #3914: URL: https://github.com/apache/hive/pull/3914#issuecomment-1371850140 @cnauroth FYI Issue Time Tracking --- Worklog Id: (was: 837152) Time Spent: 20m (was: 10m) > Backport of HIVE-20715: Disable test: udaf_histogram_numeric > - > > Key: HIVE-26909 > URL: https://issues.apache.org/jira/browse/HIVE-26909 > Project: Hive > Issue Type: Sub-task >Reporter: Aman Raj >Assignee: Aman Raj >Priority: Major > Labels: pull-request-available > Time Spent: 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-26909) Backport of HIVE-20715: Disable test: udaf_histogram_numeric
[ https://issues.apache.org/jira/browse/HIVE-26909?focusedWorklogId=837153&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-837153 ] ASF GitHub Bot logged work on HIVE-26909: - Author: ASF GitHub Bot Created on: 05/Jan/23 07:01 Start Date: 05/Jan/23 07:01 Worklog Time Spent: 10m Work Description: amanraj2520 commented on PR #3914: URL: https://github.com/apache/hive/pull/3914#issuecomment-1371850699 @zabetak @abstractdog Can you please check this and approve. Just a simple exclusion of test. Issue Time Tracking --- Worklog Id: (was: 837153) Time Spent: 0.5h (was: 20m) > Backport of HIVE-20715: Disable test: udaf_histogram_numeric > - > > Key: HIVE-26909 > URL: https://issues.apache.org/jira/browse/HIVE-26909 > Project: Hive > Issue Type: Sub-task >Reporter: Aman Raj >Assignee: Aman Raj >Priority: Major > Labels: pull-request-available > Time Spent: 0.5h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-26907) Backport of HIVE-20741: Disable udaf_context_ngrams.q and udaf_corr.q tests
[ https://issues.apache.org/jira/browse/HIVE-26907?focusedWorklogId=837155&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-837155 ] ASF GitHub Bot logged work on HIVE-26907: - Author: ASF GitHub Bot Created on: 05/Jan/23 07:02 Start Date: 05/Jan/23 07:02 Worklog Time Spent: 10m Work Description: amanraj2520 commented on PR #3913: URL: https://github.com/apache/hive/pull/3913#issuecomment-1371851247 @zabetak @abstractdog Can you please check this and approve. Just a simple exclusion of a test. Issue Time Tracking --- Worklog Id: (was: 837155) Time Spent: 0.5h (was: 20m) > Backport of HIVE-20741: Disable udaf_context_ngrams.q and udaf_corr.q tests > --- > > Key: HIVE-26907 > URL: https://issues.apache.org/jira/browse/HIVE-26907 > Project: Hive > Issue Type: Sub-task >Reporter: Aman Raj >Assignee: Aman Raj >Priority: Major > Labels: pull-request-available > Time Spent: 0.5h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HIVE-26898) Split Notification logging so that busy clusters can have better performance
[ https://issues.apache.org/jira/browse/HIVE-26898?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Taraka Rama Rao Lethavadla updated HIVE-26898: -- Summary: Split Notification logging so that busy clusters can have better performance (was: Split Notification logging so that we can busy clusters can have better performance) > Split Notification logging so that busy clusters can have better performance > > > Key: HIVE-26898 > URL: https://issues.apache.org/jira/browse/HIVE-26898 > Project: Hive > Issue Type: New Feature >Reporter: Taraka Rama Rao Lethavadla >Priority: Major > > For DDL & DML events are logged into notifications log table and those get > cleaned as soon as ttl got expired. > In most of the busy clusters, the notification log is growing even though > cleaner is running and kept on cleaning the events. It means the rate of Hive > db operations are very high compared to rate at which cleaning is happening. > So any query on this table is becoming bottle neck at backend DB causing slow > response > The proposal is to split the notification log table in to multiple tables > like > notification_log_dml - for all DML queries > notification_log_insert - for all insert queries > .. > etc. > > So that load on that single table gets reduced improving the performance of > the backend db as well as Hive -- This message was sent by Atlassian Jira (v8.20.10#820010)
