[jira] [Updated] (HIVE-20906) Are Hive locks ephemeral ?
[ https://issues.apache.org/jira/browse/HIVE-20906?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajesh KSV updated HIVE-20906: -- Summary: Are Hive locks ephemeral ? (was: Hive locks are not really ephemeral) > Are Hive locks ephemeral ? > -- > > Key: HIVE-20906 > URL: https://issues.apache.org/jira/browse/HIVE-20906 > Project: Hive > Issue Type: Bug >Affects Versions: 2.3.0, 2.3.3, 3.1.1 >Reporter: Rajesh KSV >Priority: Major > Fix For: 3.1.1 > > Attachments: ephemeral.patch > > > I am using Hive 2.3.3. > > According to docs > ([https://cwiki.apache.org/confluence/display/Hive/Locking]), they are > ephemeral. But in the Zookeeper and Hive source Code, they seemed to be > non-ephemeral. > > I ran the following query in Hive. > > *lock table discovery.defaultns* > > When I see the lock in zookeeper, it doesn't look like ephemeral node. > > [zk: localhost:2181(CONNECTED) 55] get > /hive_zookeeper_namespace/discovery/defaultns/LOCK-EXCLUSIVE-00 > > hive_20181112191555_d6640955-2806-4fa7-9e52-30d04c377a04:1542030355565:EXPLICIT:lock > table discovery.defaultns exclusive:10.33.222.139 > cZxid = 0x2400296808 > ctime = Mon Nov 12 19:15:55 IST 2018 > mZxid = 0x2400296808 > mtime = Mon Nov 12 19:15:55 IST 2018 > pZxid = 0x2400296808 > cversion = 0 > dataVersion = 0 > aclVersion = 0 > *ephemeralOwner = 0x0* > dataLength = 134 > numChildren = 0 > > Also in the code - > [https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/lockmgr/HiveTxnManagerImpl.java], > the lock is KeepAlive/PERSISTENT and not EPHIMERAL. Current code is > > *HiveLock lck = lockMgr.lock(new HiveLockObject(tbl, lockData), mode, > true);* > Either the docs are misleading or code is incorrect. > If docs are correct, then it should be > *HiveLock lck = lockMgr.lock(new HiveLockObject(tbl, lockData), mode, > false);* > The patch is the same. > Please correct me if I am wrong. This is my first ever Jira/Patch in Hive -- This message was sent by Atlassian JIRA (v7.6.3#76005)
***UNCHECKED*** [jira] [Updated] (HIVE-20906) Hive locks are not really ephemeral
[ https://issues.apache.org/jira/browse/HIVE-20906?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajesh KSV updated HIVE-20906: -- Description: I am using Hive 2.3.3. According to docs ([https://cwiki.apache.org/confluence/display/Hive/Locking]), they are ephemeral. But in the Zookeeper and Hive source Code, they seemed to be non-ephemeral. I ran the following query in Hive. *lock table discovery.defaultns* When I see the lock in zookeeper, it doesn't look like ephemeral node. [zk: localhost:2181(CONNECTED) 55] get /hive_zookeeper_namespace/discovery/defaultns/LOCK-EXCLUSIVE-00 hive_20181112191555_d6640955-2806-4fa7-9e52-30d04c377a04:1542030355565:EXPLICIT:lock table discovery.defaultns exclusive:10.33.222.139 cZxid = 0x2400296808 ctime = Mon Nov 12 19:15:55 IST 2018 mZxid = 0x2400296808 mtime = Mon Nov 12 19:15:55 IST 2018 pZxid = 0x2400296808 cversion = 0 dataVersion = 0 aclVersion = 0 *ephemeralOwner = 0x0* dataLength = 134 numChildren = 0 Also in the code - [https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/lockmgr/HiveTxnManagerImpl.java], the lock is KeepAlive/PERSISTENT and not EPHIMERAL. Current code is *HiveLock lck = lockMgr.lock(new HiveLockObject(tbl, lockData), mode, true);* Either the docs are misleading or code is incorrect. If docs are correct, then it should be *HiveLock lck = lockMgr.lock(new HiveLockObject(tbl, lockData), mode, false);* The patch is the same. Please correct me if I am wrong. This is my first ever Jira/Patch in Hive was: I am using Hive 2.3.3. According to docs ([https://cwiki.apache.org/confluence/display/Hive/Locking]), they are ephemeral. But in the Zookeeper and Hive source Code, they seemed to be non-ephemeral. I ran the following query in Hive. *lock table discovery.defaultns* When I see the lock in zookeeper, it doesn't look like ephemeral node. [zk: localhost:2181(CONNECTED) 55] get /hive_zookeeper_namespace/discovery/defaultns/LOCK-EXCLUSIVE-00 hive_20181112191555_d6640955-2806-4fa7-9e52-30d04c377a04:1542030355565:EXPLICIT:lock table discovery.defaultns exclusive:10.33.222.139 cZxid = 0x2400296808 ctime = Mon Nov 12 19:15:55 IST 2018 mZxid = 0x2400296808 mtime = Mon Nov 12 19:15:55 IST 2018 pZxid = 0x2400296808 cversion = 0 dataVersion = 0 aclVersion = 0 *ephemeralOwner = 0x0* dataLength = 134 numChildren = 0 Also in the code - [https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/lockmgr/HiveTxnManagerImpl.java], the lock is KeepAlive/PERSISTENT and not EPHIMERAL. Current code is *HiveLock lck = lockMgr.lock(new HiveLockObject(tbl, lockData), mode, true);* Instead it should be *HiveLock lck = lockMgr.lock(new HiveLockObject(tbl, lockData), mode, false);* The patch is the same. Please correct me if I am wrong. This is my first ever Jira/Patch in Hive > Hive locks are not really ephemeral > --- > > Key: HIVE-20906 > URL: https://issues.apache.org/jira/browse/HIVE-20906 > Project: Hive > Issue Type: Bug >Affects Versions: 2.3.0, 2.3.3, 3.1.1 >Reporter: Rajesh KSV >Priority: Major > Fix For: 3.1.1 > > Attachments: ephemeral.patch > > > I am using Hive 2.3.3. > > According to docs > ([https://cwiki.apache.org/confluence/display/Hive/Locking]), they are > ephemeral. But in the Zookeeper and Hive source Code, they seemed to be > non-ephemeral. > > I ran the following query in Hive. > > *lock table discovery.defaultns* > > When I see the lock in zookeeper, it doesn't look like ephemeral node. > > [zk: localhost:2181(CONNECTED) 55] get > /hive_zookeeper_namespace/discovery/defaultns/LOCK-EXCLUSIVE-00 > > hive_20181112191555_d6640955-2806-4fa7-9e52-30d04c377a04:1542030355565:EXPLICIT:lock > table discovery.defaultns exclusive:10.33.222.139 > cZxid = 0x2400296808 > ctime = Mon Nov 12 19:15:55 IST 2018 > mZxid = 0x2400296808 > mtime = Mon Nov 12 19:15:55 IST 2018 > pZxid = 0x2400296808 > cversion = 0 > dataVersion = 0 > aclVersion = 0 > *ephemeralOwner = 0x0* > dataLength = 134 > numChildren = 0 > > Also in the code - > [https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/lockmgr/HiveTxnManagerImpl.java], > the lock is KeepAlive/PERSISTENT and not EPHIMERAL. Current code is > > *HiveLock lck = lockMgr.lock(new HiveLockObject(tbl, lockData), mode, > true);* > Either the docs are misleading or code is incorrect. > If docs are correct, then it should be > *HiveLock lck = lockMgr.lock(new HiveLockObject(tbl, lockData), mode, > false);* > The patch is the same. > Please correct me if I am wrong. This is my first ever Jira/Patch in Hive -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20908) Avoid multiple getTableMeta calls during GetTablesOperation.
[
https://issues.apache.org/jira/browse/HIVE-20908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16684799#comment-16684799
]
Hive QA commented on HIVE-20908:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
24s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
22s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
11s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
34s{color} | {color:blue} service in master has 48 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
15s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
21s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
11s{color} | {color:red} service: The patch generated 13 new + 13 unchanged - 0
fixed = 26 total (was 13) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
15s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 11m 21s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-14902/dev-support/hive-personality.sh
|
| git revision | master / af40170 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-14902/yetus/diff-checkstyle-service.txt
|
| modules | C: service U: service |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-14902/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Avoid multiple getTableMeta calls during GetTablesOperation.
>
>
> Key: HIVE-20908
> URL: https://issues.apache.org/jira/browse/HIVE-20908
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 3.1.1
>Reporter: Rajkumar Singh
>Assignee: Rajkumar Singh
>Priority: Minor
> Labels: performance
> Attachments: HIVE-20908.patch
>
>
> following HIVE-19432, we are doing getTableMeta for each authorized db
> instead of that we can pass pattern for metastore.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20842) Fix logic introduced in HIVE-20660 to estimate statistics for group by
[
https://issues.apache.org/jira/browse/HIVE-20842?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16684790#comment-16684790
]
Hive QA commented on HIVE-20842:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12947925/HIVE-20842.10.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 4 failed/errored test(s), 15539 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vectorization_short_regress]
(batchId=171)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query77]
(batchId=272)
org.apache.hive.jdbc.TestJdbcWithMiniLlapArrow.testComplexQuery (batchId=258)
org.apache.hive.jdbc.TestJdbcWithMiniLlapArrow.testKillQuery (batchId=258)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/14901/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/14901/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-14901/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 4 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12947925 - PreCommit-HIVE-Build
> Fix logic introduced in HIVE-20660 to estimate statistics for group by
> --
>
> Key: HIVE-20842
> URL: https://issues.apache.org/jira/browse/HIVE-20842
> Project: Hive
> Issue Type: Bug
> Components: Statistics
>Affects Versions: 4.0.0
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
> Attachments: HIVE-20842.1.patch, HIVE-20842.10.patch,
> HIVE-20842.2.patch, HIVE-20842.3.patch, HIVE-20842.4.patch,
> HIVE-20842.5.patch, HIVE-20842.6.patch, HIVE-20842.7.patch,
> HIVE-20842.8.patch, HIVE-20842.9.patch
>
>
> HIVE-20660 introduced better estimation for group by operator. But the logic
> did not account for Partial and Full group by separately.
> For partial group by parallelism (i.e. number of tasks) should be taken into
> account.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20905) querying streaming table fails with out of memory exception
[ https://issues.apache.org/jira/browse/HIVE-20905?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] mahesh kumar behera updated HIVE-20905: --- Status: Patch Available (was: Open) > querying streaming table fails with out of memory exception > --- > > Key: HIVE-20905 > URL: https://issues.apache.org/jira/browse/HIVE-20905 > Project: Hive > Issue Type: Bug > Components: Metastore, Transactions >Affects Versions: 4.0.0 >Reporter: mahesh kumar behera >Assignee: mahesh kumar behera >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-20905.01.patch, HIVE-20905.02.patch, > HIVE-20905.03.patch > > > Streaming app was ran for 24hrs post which it went down due authentication > issue . The table was accessible for 12hrs into the run, however currently > querying the table fails with OOM exception. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20905) querying streaming table fails with out of memory exception
[ https://issues.apache.org/jira/browse/HIVE-20905?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] mahesh kumar behera updated HIVE-20905: --- Attachment: HIVE-20905.03.patch > querying streaming table fails with out of memory exception > --- > > Key: HIVE-20905 > URL: https://issues.apache.org/jira/browse/HIVE-20905 > Project: Hive > Issue Type: Bug > Components: Metastore, Transactions >Affects Versions: 4.0.0 >Reporter: mahesh kumar behera >Assignee: mahesh kumar behera >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-20905.01.patch, HIVE-20905.02.patch, > HIVE-20905.03.patch > > > Streaming app was ran for 24hrs post which it went down due authentication > issue . The table was accessible for 12hrs into the run, however currently > querying the table fails with OOM exception. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
***UNCHECKED*** [jira] [Updated] (HIVE-20905) querying streaming table fails with out of memory exception
[ https://issues.apache.org/jira/browse/HIVE-20905?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] mahesh kumar behera updated HIVE-20905: --- Attachment: (was: HIVE-20905.02.patch) > querying streaming table fails with out of memory exception > --- > > Key: HIVE-20905 > URL: https://issues.apache.org/jira/browse/HIVE-20905 > Project: Hive > Issue Type: Bug > Components: Metastore, Transactions >Affects Versions: 4.0.0 >Reporter: mahesh kumar behera >Assignee: mahesh kumar behera >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-20905.01.patch, HIVE-20905.02.patch > > > Streaming app was ran for 24hrs post which it went down due authentication > issue . The table was accessible for 12hrs into the run, however currently > querying the table fails with OOM exception. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
***UNCHECKED*** [jira] [Updated] (HIVE-20905) querying streaming table fails with out of memory exception
[ https://issues.apache.org/jira/browse/HIVE-20905?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] mahesh kumar behera updated HIVE-20905: --- Status: Open (was: Patch Available) > querying streaming table fails with out of memory exception > --- > > Key: HIVE-20905 > URL: https://issues.apache.org/jira/browse/HIVE-20905 > Project: Hive > Issue Type: Bug > Components: Metastore, Transactions >Affects Versions: 4.0.0 >Reporter: mahesh kumar behera >Assignee: mahesh kumar behera >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-20905.01.patch, HIVE-20905.02.patch > > > Streaming app was ran for 24hrs post which it went down due authentication > issue . The table was accessible for 12hrs into the run, however currently > querying the table fails with OOM exception. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20905) querying streaming table fails with out of memory exception
[ https://issues.apache.org/jira/browse/HIVE-20905?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] mahesh kumar behera updated HIVE-20905: --- Attachment: HIVE-20905.02.patch > querying streaming table fails with out of memory exception > --- > > Key: HIVE-20905 > URL: https://issues.apache.org/jira/browse/HIVE-20905 > Project: Hive > Issue Type: Bug > Components: Metastore, Transactions >Affects Versions: 4.0.0 >Reporter: mahesh kumar behera >Assignee: mahesh kumar behera >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-20905.01.patch, HIVE-20905.02.patch > > > Streaming app was ran for 24hrs post which it went down due authentication > issue . The table was accessible for 12hrs into the run, however currently > querying the table fails with OOM exception. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20842) Fix logic introduced in HIVE-20660 to estimate statistics for group by
[
https://issues.apache.org/jira/browse/HIVE-20842?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16684760#comment-16684760
]
Hive QA commented on HIVE-20842:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
32s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
59s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
37s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
38s{color} | {color:blue} ql in master has 2316 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
54s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
40s{color} | {color:red} ql: The patch generated 5 new + 38 unchanged - 0 fixed
= 43 total (was 38) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
12s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 22m 18s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-14901/dev-support/hive-personality.sh
|
| git revision | master / af40170 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-14901/yetus/diff-checkstyle-ql.txt
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-14901/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Fix logic introduced in HIVE-20660 to estimate statistics for group by
> --
>
> Key: HIVE-20842
> URL: https://issues.apache.org/jira/browse/HIVE-20842
> Project: Hive
> Issue Type: Bug
> Components: Statistics
>Affects Versions: 4.0.0
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
> Attachments: HIVE-20842.1.patch, HIVE-20842.10.patch,
> HIVE-20842.2.patch, HIVE-20842.3.patch, HIVE-20842.4.patch,
> HIVE-20842.5.patch, HIVE-20842.6.patch, HIVE-20842.7.patch,
> HIVE-20842.8.patch, HIVE-20842.9.patch
>
>
> HIVE-20660 introduced better estimation for group by operator. But the logic
> did not account for Partial and Full group by separately.
> For partial group by parallelism (i.e. number of tasks) should be taken into
> account.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20823) Make Compactor run in a transaction
[
https://issues.apache.org/jira/browse/HIVE-20823?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16684745#comment-16684745
]
Hive QA commented on HIVE-20823:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12947931/HIVE-20823.11.patch
{color:green}SUCCESS:{color} +1 due to 20 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 28 failed/errored test(s), 15531 tests
executed
*Failed tests:*
{noformat}
TestMiniDruidCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=196)
[druidmini_masking.q,druidmini_test1.q,druidkafkamini_basic.q,druidmini_joins.q,druid_timestamptz.q]
org.apache.hadoop.hive.ql.TestTxnCommands.testNonAcidToAcidConversion01
(batchId=323)
org.apache.hadoop.hive.ql.TestTxnNoBuckets.testCompactStatsGather (batchId=307)
org.apache.hadoop.hive.ql.TestTxnNoBucketsVectorized.testCompactStatsGather
(batchId=307)
org.apache.hadoop.hive.ql.txn.compactor.TestCompactor.majorCompactAfterAbort[0]
(batchId=243)
org.apache.hadoop.hive.ql.txn.compactor.TestCompactor.majorCompactAfterAbort[1]
(batchId=243)
org.apache.hadoop.hive.ql.txn.compactor.TestCompactor.majorCompactWhileStreamingForSplitUpdate[0]
(batchId=243)
org.apache.hadoop.hive.ql.txn.compactor.TestCompactor.majorCompactWhileStreamingForSplitUpdate[1]
(batchId=243)
org.apache.hadoop.hive.ql.txn.compactor.TestCompactor.majorCompactWhileStreaming[0]
(batchId=243)
org.apache.hadoop.hive.ql.txn.compactor.TestCompactor.majorCompactWhileStreaming[1]
(batchId=243)
org.apache.hadoop.hive.ql.txn.compactor.TestCompactor.minorCompactAfterAbort[0]
(batchId=243)
org.apache.hadoop.hive.ql.txn.compactor.TestCompactor.minorCompactAfterAbort[1]
(batchId=243)
org.apache.hadoop.hive.ql.txn.compactor.TestCompactor.minorCompactWhileStreamingWithSplitUpdate[0]
(batchId=243)
org.apache.hadoop.hive.ql.txn.compactor.TestCompactor.minorCompactWhileStreamingWithSplitUpdate[1]
(batchId=243)
org.apache.hadoop.hive.ql.txn.compactor.TestCompactor.minorCompactWhileStreaming[0]
(batchId=243)
org.apache.hadoop.hive.ql.txn.compactor.TestCompactor.minorCompactWhileStreaming[1]
(batchId=243)
org.apache.hadoop.hive.ql.txn.compactor.TestCompactor.testMinorCompactionForSplitUpdateWithInsertsAndDeletes[0]
(batchId=243)
org.apache.hadoop.hive.ql.txn.compactor.TestCompactor.testMinorCompactionForSplitUpdateWithInsertsAndDeletes[1]
(batchId=243)
org.apache.hadoop.hive.ql.txn.compactor.TestCompactor.testMinorCompactionForSplitUpdateWithOnlyInserts[0]
(batchId=243)
org.apache.hadoop.hive.ql.txn.compactor.TestCompactor.testMinorCompactionForSplitUpdateWithOnlyInserts[1]
(batchId=243)
org.apache.hadoop.hive.ql.txn.compactor.TestWorker.majorTableNoBase
(batchId=293)
org.apache.hadoop.hive.ql.txn.compactor.TestWorker.majorWithOpenInMiddle
(batchId=293)
org.apache.hadoop.hive.ql.txn.compactor.TestWorker.minorPartitionWithBase
(batchId=293)
org.apache.hadoop.hive.ql.txn.compactor.TestWorker.minorWithOpenInMiddle
(batchId=293)
org.apache.hadoop.hive.ql.txn.compactor.TestWorker2.majorTableNoBase
(batchId=293)
org.apache.hadoop.hive.ql.txn.compactor.TestWorker2.majorWithOpenInMiddle
(batchId=293)
org.apache.hadoop.hive.ql.txn.compactor.TestWorker2.minorPartitionWithBase
(batchId=293)
org.apache.hadoop.hive.ql.txn.compactor.TestWorker2.minorWithOpenInMiddle
(batchId=293)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/14900/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/14900/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-14900/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 28 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12947931 - PreCommit-HIVE-Build
> Make Compactor run in a transaction
> ---
>
> Key: HIVE-20823
> URL: https://issues.apache.org/jira/browse/HIVE-20823
> Project: Hive
> Issue Type: Improvement
> Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
>Priority: Major
> Attachments: HIVE-20823.01.patch, HIVE-20823.03.patch,
> HIVE-20823.04.patch, HIVE-20823.05.patch, HIVE-20823.07.patch,
> HIVE-20823.08.patch, HIVE-20823.09.patch, HIVE-20823.10.patch,
> HIVE-20823.11.patch, HIVE-20823.11.patch
>
>
> Have compactor open a transaction and run the job in that transaction.
> # make compactor produced base/delta include this txn id in the folder name,
> e.g. base_7_c17 where 17 is the txnid.
> #
[jira] [Commented] (HIVE-20823) Make Compactor run in a transaction
[
https://issues.apache.org/jira/browse/HIVE-20823?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16684733#comment-16684733
]
Hive QA commented on HIVE-20823:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
28s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
10s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m
15s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
6s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
23s{color} | {color:blue} storage-api in master has 48 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 2m
11s{color} | {color:blue} standalone-metastore/metastore-common in master has
29 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
27s{color} | {color:blue} hcatalog/streaming in master has 11 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
35s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
46s{color} | {color:blue} ql in master has 2316 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 1m
4s{color} | {color:blue} standalone-metastore/metastore-server in master has
185 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
24s{color} | {color:blue} streaming in master has 2 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
54s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
8s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
54s{color} | {color:red} ql in the patch failed. {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 3m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
10s{color} | {color:green} storage-api: The patch generated 0 new + 7 unchanged
- 1 fixed = 7 total (was 8) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
6s{color} | {color:green} The patch metastore-common passed checkstyle {color}
|
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
13s{color} | {color:green} The patch streaming passed checkstyle {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
17s{color} | {color:red} itests/hive-unit: The patch generated 10 new + 179
unchanged - 10 fixed = 189 total (was 189) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
56s{color} | {color:red} ql: The patch generated 88 new + 2410 unchanged - 51
fixed = 2498 total (was 2461) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
7s{color} | {color:green} The patch metastore-server passed checkstyle {color}
|
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
13s{color} | {color:red} streaming: The patch generated 2 new + 455 unchanged -
1 fixed = 457 total (was 456) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 3m
51s{color} | {color:red} ql generated 1 new + 2315 unchanged - 1 fixed = 2316
total (was 2316) {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
10s{color} | {color:red} standalone-metastore/metastore-server generated 1 new
+ 184 unchanged - 1 fixed = 185 total (was 185) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
9s{color} | {color:green}
[jira] [Updated] (HIVE-18661) CachedStore: Use metastore notification log events to update cache
[ https://issues.apache.org/jira/browse/HIVE-18661?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] mahesh kumar behera updated HIVE-18661: --- Status: Patch Available (was: Open) > CachedStore: Use metastore notification log events to update cache > -- > > Key: HIVE-18661 > URL: https://issues.apache.org/jira/browse/HIVE-18661 > Project: Hive > Issue Type: Sub-task > Components: Metastore >Reporter: Vaibhav Gumashta >Assignee: mahesh kumar behera >Priority: Major > Labels: pull-request-available > Attachments: HIVE-18661.02.patch, HIVE-18661.03.patch, > HIVE-18661.04.patch, HIVE-18661.05.patch, HIVE-18661.06.patch, > HIVE-18661.07.patch > > > Currently, a background thread updates the entire cache which is pretty > inefficient. We capture the updates to metadata in NOTIFICATION_LOG table > which is getting used in the Replication work. We should have the background > thread apply these notifications to incrementally update the cache. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18661) CachedStore: Use metastore notification log events to update cache
[ https://issues.apache.org/jira/browse/HIVE-18661?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] mahesh kumar behera updated HIVE-18661: --- Attachment: HIVE-18661.07.patch > CachedStore: Use metastore notification log events to update cache > -- > > Key: HIVE-18661 > URL: https://issues.apache.org/jira/browse/HIVE-18661 > Project: Hive > Issue Type: Sub-task > Components: Metastore >Reporter: Vaibhav Gumashta >Assignee: mahesh kumar behera >Priority: Major > Labels: pull-request-available > Attachments: HIVE-18661.02.patch, HIVE-18661.03.patch, > HIVE-18661.04.patch, HIVE-18661.05.patch, HIVE-18661.06.patch, > HIVE-18661.07.patch > > > Currently, a background thread updates the entire cache which is pretty > inefficient. We capture the updates to metadata in NOTIFICATION_LOG table > which is getting used in the Replication work. We should have the background > thread apply these notifications to incrementally update the cache. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18661) CachedStore: Use metastore notification log events to update cache
[ https://issues.apache.org/jira/browse/HIVE-18661?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] mahesh kumar behera updated HIVE-18661: --- Status: Open (was: Patch Available) > CachedStore: Use metastore notification log events to update cache > -- > > Key: HIVE-18661 > URL: https://issues.apache.org/jira/browse/HIVE-18661 > Project: Hive > Issue Type: Sub-task > Components: Metastore >Reporter: Vaibhav Gumashta >Assignee: mahesh kumar behera >Priority: Major > Labels: pull-request-available > Attachments: HIVE-18661.02.patch, HIVE-18661.03.patch, > HIVE-18661.04.patch, HIVE-18661.05.patch, HIVE-18661.06.patch > > > Currently, a background thread updates the entire cache which is pretty > inefficient. We capture the updates to metadata in NOTIFICATION_LOG table > which is getting used in the Replication work. We should have the background > thread apply these notifications to incrementally update the cache. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20787) MapJoinBytesTableContainer dummyRow case doesn't handle reuse
[
https://issues.apache.org/jira/browse/HIVE-20787?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16684709#comment-16684709
]
Hive QA commented on HIVE-20787:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12947901/HIVE-20787.02.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 15539 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/14899/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/14899/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-14899/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12947901 - PreCommit-HIVE-Build
> MapJoinBytesTableContainer dummyRow case doesn't handle reuse
> -
>
> Key: HIVE-20787
> URL: https://issues.apache.org/jira/browse/HIVE-20787
> Project: Hive
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Major
> Attachments: HIVE-20787.01.patch, HIVE-20787.02.patch,
> HIVE-20787.patch
>
>
> Discovered while investigating some (probably) unrelated issue.
> MapJoinBytesTableContainer was not intended to be reused, but it looks like
> some code might reuse it. If that happens, the dummyRow case will not work
> correctly (dummyRow is cleared on first(), so another call to first() will
> behave differently).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20905) querying streaming table fails with out of memory exception
[ https://issues.apache.org/jira/browse/HIVE-20905?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16684700#comment-16684700 ] Thejas M Nair commented on HIVE-20905: -- Setting it to empty for whole metastore server sounds like a good solution. > querying streaming table fails with out of memory exception > --- > > Key: HIVE-20905 > URL: https://issues.apache.org/jira/browse/HIVE-20905 > Project: Hive > Issue Type: Bug > Components: Metastore, Transactions >Affects Versions: 4.0.0 >Reporter: mahesh kumar behera >Assignee: mahesh kumar behera >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-20905.01.patch, HIVE-20905.02.patch > > > Streaming app was ran for 24hrs post which it went down due authentication > issue . The table was accessible for 12hrs into the run, however currently > querying the table fails with OOM exception. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18661) CachedStore: Use metastore notification log events to update cache
[ https://issues.apache.org/jira/browse/HIVE-18661?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] mahesh kumar behera updated HIVE-18661: --- Status: Patch Available (was: Open) fixed issue related to stale read from an active transaction > CachedStore: Use metastore notification log events to update cache > -- > > Key: HIVE-18661 > URL: https://issues.apache.org/jira/browse/HIVE-18661 > Project: Hive > Issue Type: Sub-task > Components: Metastore >Reporter: Vaibhav Gumashta >Assignee: mahesh kumar behera >Priority: Major > Labels: pull-request-available > Attachments: HIVE-18661.02.patch, HIVE-18661.03.patch, > HIVE-18661.04.patch, HIVE-18661.05.patch, HIVE-18661.06.patch > > > Currently, a background thread updates the entire cache which is pretty > inefficient. We capture the updates to metadata in NOTIFICATION_LOG table > which is getting used in the Replication work. We should have the background > thread apply these notifications to incrementally update the cache. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18661) CachedStore: Use metastore notification log events to update cache
[ https://issues.apache.org/jira/browse/HIVE-18661?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] mahesh kumar behera updated HIVE-18661: --- Status: Open (was: Patch Available) > CachedStore: Use metastore notification log events to update cache > -- > > Key: HIVE-18661 > URL: https://issues.apache.org/jira/browse/HIVE-18661 > Project: Hive > Issue Type: Sub-task > Components: Metastore >Reporter: Vaibhav Gumashta >Assignee: mahesh kumar behera >Priority: Major > Labels: pull-request-available > Attachments: HIVE-18661.02.patch, HIVE-18661.03.patch, > HIVE-18661.04.patch, HIVE-18661.05.patch, HIVE-18661.06.patch > > > Currently, a background thread updates the entire cache which is pretty > inefficient. We capture the updates to metadata in NOTIFICATION_LOG table > which is getting used in the Replication work. We should have the background > thread apply these notifications to incrementally update the cache. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18661) CachedStore: Use metastore notification log events to update cache
[ https://issues.apache.org/jira/browse/HIVE-18661?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] mahesh kumar behera updated HIVE-18661: --- Attachment: HIVE-18661.06.patch > CachedStore: Use metastore notification log events to update cache > -- > > Key: HIVE-18661 > URL: https://issues.apache.org/jira/browse/HIVE-18661 > Project: Hive > Issue Type: Sub-task > Components: Metastore >Reporter: Vaibhav Gumashta >Assignee: mahesh kumar behera >Priority: Major > Labels: pull-request-available > Attachments: HIVE-18661.02.patch, HIVE-18661.03.patch, > HIVE-18661.04.patch, HIVE-18661.05.patch, HIVE-18661.06.patch > > > Currently, a background thread updates the entire cache which is pretty > inefficient. We capture the updates to metadata in NOTIFICATION_LOG table > which is getting used in the Replication work. We should have the background > thread apply these notifications to incrementally update the cache. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20787) MapJoinBytesTableContainer dummyRow case doesn't handle reuse
[
https://issues.apache.org/jira/browse/HIVE-20787?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16684684#comment-16684684
]
Hive QA commented on HIVE-20787:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
40s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
1s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
36s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
42s{color} | {color:blue} ql in master has 2316 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
54s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
37s{color} | {color:red} ql: The patch generated 2 new + 13 unchanged - 0 fixed
= 15 total (was 13) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
55s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 22m 25s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-14899/dev-support/hive-personality.sh
|
| git revision | master / af40170 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-14899/yetus/diff-checkstyle-ql.txt
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-14899/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> MapJoinBytesTableContainer dummyRow case doesn't handle reuse
> -
>
> Key: HIVE-20787
> URL: https://issues.apache.org/jira/browse/HIVE-20787
> Project: Hive
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Major
> Attachments: HIVE-20787.01.patch, HIVE-20787.02.patch,
> HIVE-20787.patch
>
>
> Discovered while investigating some (probably) unrelated issue.
> MapJoinBytesTableContainer was not intended to be reused, but it looks like
> some code might reuse it. If that happens, the dummyRow case will not work
> correctly (dummyRow is cleared on first(), so another call to first() will
> behave differently).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20512) Improve record and memory usage logging in SparkRecordHandler
[
https://issues.apache.org/jira/browse/HIVE-20512?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bharathkrishna Guruvayoor Murali updated HIVE-20512:
Attachment: HIVE-20512.92.patch
> Improve record and memory usage logging in SparkRecordHandler
> -
>
> Key: HIVE-20512
> URL: https://issues.apache.org/jira/browse/HIVE-20512
> Project: Hive
> Issue Type: Sub-task
> Components: Spark
>Reporter: Sahil Takiar
>Assignee: Bharathkrishna Guruvayoor Murali
>Priority: Major
> Attachments: HIVE-20512.1.patch, HIVE-20512.2.patch,
> HIVE-20512.3.patch, HIVE-20512.4.patch, HIVE-20512.5.patch,
> HIVE-20512.6.patch, HIVE-20512.7.patch, HIVE-20512.8.patch,
> HIVE-20512.9.patch, HIVE-20512.91.patch, HIVE-20512.92.patch
>
>
> We currently log memory usage and # of records processed in Spark tasks, but
> we should improve the methodology for how frequently we log this info.
> Currently we use the following code:
> {code:java}
> private long getNextLogThreshold(long currentThreshold) {
> // A very simple counter to keep track of number of rows processed by the
> // reducer. It dumps
> // every 1 million times, and quickly before that
> if (currentThreshold >= 100) {
> return currentThreshold + 100;
> }
> return 10 * currentThreshold;
> }
> {code}
> The issue is that after a while, the increase by 10x factor means that you
> have to process a huge # of records before this gets triggered.
> A better approach would be to log this info at a given interval. This would
> help in debugging tasks that are seemingly hung.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20900) serde2.JsonSerDe no longer supports timestamp.formats
[
https://issues.apache.org/jira/browse/HIVE-20900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16684660#comment-16684660
]
Hive QA commented on HIVE-20900:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12947896/HIVE-20900.1.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 3 failed/errored test(s), 15330 tests
executed
*Failed tests:*
{noformat}
TestCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=32)
[kafka_storage_handler.q,input11_limit.q,nonreserved_keywords_input37.q,acid_globallimit.q,confirm_initial_tbl_stats.q,orc_llap_counters1.q,alter_char1.q,inputddl1.q,udf_add.q,join16.q,spark_explain_groupbyshuffle.q,parquet_vectorization_decimal_date.q,udf_chr.q,vectorization_part.q,auto_join30.q,columnStatsUpdateForStatsOptimizer_2.q,dynamic_partition_pruning.q,input44.q,udf_size.q,parquet_vectorization_9.q,timestamp.q,bucketmapjoin7.q,smb_mapjoin_18.q,vector_varchar_4.q,partition_wise_fileformat15.q,udf_std.q,vector_decimal_4.q,cbo_subq_in.q,show_create_table_alter.q,vectorized_timestamp_funcs.q]
org.apache.hadoop.hive.cli.TestMiniDruidCliDriver.testCliDriver[kafka_storage_handler]
(batchId=194)
org.apache.hadoop.hive.cli.TestTezPerfConstraintsCliDriver.org.apache.hadoop.hive.cli.TestTezPerfConstraintsCliDriver
(batchId=272)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/14898/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/14898/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-14898/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 3 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12947896 - PreCommit-HIVE-Build
> serde2.JsonSerDe no longer supports timestamp.formats
> -
>
> Key: HIVE-20900
> URL: https://issues.apache.org/jira/browse/HIVE-20900
> Project: Hive
> Issue Type: Bug
> Components: Serializers/Deserializers
>Reporter: Jason Dere
>Assignee: Jason Dere
>Priority: Major
> Attachments: HIVE-20900.1.patch
>
>
> Looks like HIVE-18545 broke this.
> Also json_serde_tsformat.q only tested the hcat version of JsonSerde, and the
> format in that test used the ISO timestamp format which apparently is now
> parsed by the default timestamp parsing, so the test was too simple.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20905) querying streaming table fails with out of memory exception
[ https://issues.apache.org/jira/browse/HIVE-20905?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16684647#comment-16684647 ] mahesh kumar behera commented on HIVE-20905: [~ekoifman] [~thejas] [~anishek] yes, i also felt that the metastore uri should be set to empty string during metastore init (main method in hive metastore). I don't see any side effect of that. If thats fine i can change the patch accordingly. > querying streaming table fails with out of memory exception > --- > > Key: HIVE-20905 > URL: https://issues.apache.org/jira/browse/HIVE-20905 > Project: Hive > Issue Type: Bug > Components: Metastore, Transactions >Affects Versions: 4.0.0 >Reporter: mahesh kumar behera >Assignee: mahesh kumar behera >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-20905.01.patch, HIVE-20905.02.patch > > > Streaming app was ran for 24hrs post which it went down due authentication > issue . The table was accessible for 12hrs into the run, however currently > querying the table fails with OOM exception. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20900) serde2.JsonSerDe no longer supports timestamp.formats
[
https://issues.apache.org/jira/browse/HIVE-20900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16684650#comment-16684650
]
Hive QA commented on HIVE-20900:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
22s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
18s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
5s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
9s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
39s{color} | {color:blue} serde in master has 198 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
49s{color} | {color:blue} ql in master has 2316 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 7m
27s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
8s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 7m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
13s{color} | {color:green} serde: The patch generated 0 new + 5 unchanged - 1
fixed = 5 total (was 6) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
18s{color} | {color:green} root: The patch generated 0 new + 5 unchanged - 1
fixed = 5 total (was 6) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
37s{color} | {color:green} The patch ql passed checkstyle {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
48s{color} | {color:red} serde generated 1 new + 197 unchanged - 1 fixed = 198
total (was 198) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 7m
19s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
12s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 57m 0s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:serde |
| | Found reliance on default encoding in
org.apache.hadoop.hive.serde2.json.HiveJsonStructReader.parsePrimitiveValue(String,
PrimitiveObjectInspector):in
org.apache.hadoop.hive.serde2.json.HiveJsonStructReader.parsePrimitiveValue(String,
PrimitiveObjectInspector): String.getBytes() At
HiveJsonStructReader.java:[line 353] |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-14898/dev-support/hive-personality.sh
|
| git revision | master / af40170 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-14898/yetus/new-findbugs-serde.html
|
| modules | C: serde . ql U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-14898/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> serde2.JsonSerDe no longer supports timestamp.formats
> -
>
> Key: HIVE-20900
> URL: https://issues.apache.org/jira/browse/HIVE-20900
> Project: Hive
> Issue Type: Bug
> Components: Serializers/Deserializers
>
[jira] [Updated] (HIVE-20823) Make Compactor run in a transaction
[
https://issues.apache.org/jira/browse/HIVE-20823?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eugene Koifman updated HIVE-20823:
--
Attachment: HIVE-20823.11.patch
> Make Compactor run in a transaction
> ---
>
> Key: HIVE-20823
> URL: https://issues.apache.org/jira/browse/HIVE-20823
> Project: Hive
> Issue Type: Improvement
> Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
>Priority: Major
> Attachments: HIVE-20823.01.patch, HIVE-20823.03.patch,
> HIVE-20823.04.patch, HIVE-20823.05.patch, HIVE-20823.07.patch,
> HIVE-20823.08.patch, HIVE-20823.09.patch, HIVE-20823.10.patch,
> HIVE-20823.11.patch, HIVE-20823.11.patch
>
>
> Have compactor open a transaction and run the job in that transaction.
> # make compactor produced base/delta include this txn id in the folder name,
> e.g. base_7_c17 where 17 is the txnid.
> # add {{CQ_TXN_ID bigint}} to COMPACTION_QUEUE and COMPLETED_COMPACTIONS to
> record this txn id
> # make sure {{AcidUtils.getAcidState()}} pays attention to this transaction
> on read and ignores this dir if this txn id is not committed in the current
> snapshot
> ## this means not only validWriteIdList but ValidTxnIdList should be passed
> along in config (if it isn't yet)
> # once this is done, {{CompactorMR.createCompactorMarker()}} can be
> eliminated and {{AcidUtils.isValidBase}} modified accordingly
> # modify Cleaner so that it doesn't clean old files until new file is visible
> to all readers
> #
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20823) Make Compactor run in a transaction
[
https://issues.apache.org/jira/browse/HIVE-20823?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eugene Koifman updated HIVE-20823:
--
Attachment: HIVE-20823.11.patch
> Make Compactor run in a transaction
> ---
>
> Key: HIVE-20823
> URL: https://issues.apache.org/jira/browse/HIVE-20823
> Project: Hive
> Issue Type: Improvement
> Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
>Priority: Major
> Attachments: HIVE-20823.01.patch, HIVE-20823.03.patch,
> HIVE-20823.04.patch, HIVE-20823.05.patch, HIVE-20823.07.patch,
> HIVE-20823.08.patch, HIVE-20823.09.patch, HIVE-20823.10.patch,
> HIVE-20823.11.patch
>
>
> Have compactor open a transaction and run the job in that transaction.
> # make compactor produced base/delta include this txn id in the folder name,
> e.g. base_7_c17 where 17 is the txnid.
> # add {{CQ_TXN_ID bigint}} to COMPACTION_QUEUE and COMPLETED_COMPACTIONS to
> record this txn id
> # make sure {{AcidUtils.getAcidState()}} pays attention to this transaction
> on read and ignores this dir if this txn id is not committed in the current
> snapshot
> ## this means not only validWriteIdList but ValidTxnIdList should be passed
> along in config (if it isn't yet)
> # once this is done, {{CompactorMR.createCompactorMarker()}} can be
> eliminated and {{AcidUtils.isValidBase}} modified accordingly
> # modify Cleaner so that it doesn't clean old files until new file is visible
> to all readers
> #
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20512) Improve record and memory usage logging in SparkRecordHandler
[
https://issues.apache.org/jira/browse/HIVE-20512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16684609#comment-16684609
]
Hive QA commented on HIVE-20512:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12947894/HIVE-20512.91.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 15504 tests
executed
*Failed tests:*
{noformat}
TestMiniSparkOnYarnCliDriver - did not produce a TEST-*.xml file (likely timed
out) (batchId=189)
[infer_bucket_sort_reducers_power_two.q,list_bucket_dml_10.q,orc_merge9.q,leftsemijoin_mr.q,bucket6.q,bucketmapjoin7.q,uber_reduce.q,empty_dir_in_table.q,vector_outer_join2.q,spark_explain_groupbyshuffle.q,spark_dynamic_partition_pruning.q,spark_combine_equivalent_work.q,orc_merge1.q,spark_use_op_stats.q,orc_merge_diff_fs.q,quotedid_smb.q,truncate_column_buckets.q,spark_vectorized_dynamic_partition_pruning.q,spark_in_process_launcher.q,orc_merge3.q]
TestSparkCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=128)
[load_dyn_part15.q,explaindenpendencydiffengs.q,transform2.q,groupby5.q,cbo_semijoin.q,bucketmapjoin13.q,alter_merge_stats_orc.q,subquery_scalar.q,union_remove_2.q,groupby_position.q,join12.q,smb_mapjoin_8.q,subquery_select.q,join21.q,auto_join16.q]
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/14897/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/14897/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-14897/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12947894 - PreCommit-HIVE-Build
> Improve record and memory usage logging in SparkRecordHandler
> -
>
> Key: HIVE-20512
> URL: https://issues.apache.org/jira/browse/HIVE-20512
> Project: Hive
> Issue Type: Sub-task
> Components: Spark
>Reporter: Sahil Takiar
>Assignee: Bharathkrishna Guruvayoor Murali
>Priority: Major
> Attachments: HIVE-20512.1.patch, HIVE-20512.2.patch,
> HIVE-20512.3.patch, HIVE-20512.4.patch, HIVE-20512.5.patch,
> HIVE-20512.6.patch, HIVE-20512.7.patch, HIVE-20512.8.patch,
> HIVE-20512.9.patch, HIVE-20512.91.patch
>
>
> We currently log memory usage and # of records processed in Spark tasks, but
> we should improve the methodology for how frequently we log this info.
> Currently we use the following code:
> {code:java}
> private long getNextLogThreshold(long currentThreshold) {
> // A very simple counter to keep track of number of rows processed by the
> // reducer. It dumps
> // every 1 million times, and quickly before that
> if (currentThreshold >= 100) {
> return currentThreshold + 100;
> }
> return 10 * currentThreshold;
> }
> {code}
> The issue is that after a while, the increase by 10x factor means that you
> have to process a huge # of records before this gets triggered.
> A better approach would be to log this info at a given interval. This would
> help in debugging tasks that are seemingly hung.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20838) Timestamps with timezone are set to null when using the streaming API
[
https://issues.apache.org/jira/browse/HIVE-20838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16684575#comment-16684575
]
Jaume M commented on HIVE-20838:
Can you take another look [~gopalv]?
> Timestamps with timezone are set to null when using the streaming API
> -
>
> Key: HIVE-20838
> URL: https://issues.apache.org/jira/browse/HIVE-20838
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 3.1.0
>Reporter: Jaume M
>Assignee: Jaume M
>Priority: Major
> Attachments: HIVE-20838.1.patch, HIVE-20838.10.patch,
> HIVE-20838.2.patch, HIVE-20838.3.patch, HIVE-20838.3.patch,
> HIVE-20838.4.patch, HIVE-20838.5.patch, HIVE-20838.6.patch,
> HIVE-20838.7.patch, HIVE-20838.8.patch, HIVE-20838.9.patch
>
>
> For example:
> {code}
> beeline> create table default.timest (a TIMESTAMP) stored as orc " +
> "TBLPROPERTIES('transactional'='true')
> # And then:
> connection.write("2018-10-19 10:35:00 America/Los_Angeles".getBytes());
> {code}
> inserts NULL.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20842) Fix logic introduced in HIVE-20660 to estimate statistics for group by
[ https://issues.apache.org/jira/browse/HIVE-20842?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-20842: --- Status: Patch Available (was: Open) Latest patch (10) fixes an issue in the logic . > Fix logic introduced in HIVE-20660 to estimate statistics for group by > -- > > Key: HIVE-20842 > URL: https://issues.apache.org/jira/browse/HIVE-20842 > Project: Hive > Issue Type: Bug > Components: Statistics >Affects Versions: 4.0.0 >Reporter: Vineet Garg >Assignee: Vineet Garg >Priority: Major > Attachments: HIVE-20842.1.patch, HIVE-20842.10.patch, > HIVE-20842.2.patch, HIVE-20842.3.patch, HIVE-20842.4.patch, > HIVE-20842.5.patch, HIVE-20842.6.patch, HIVE-20842.7.patch, > HIVE-20842.8.patch, HIVE-20842.9.patch > > > HIVE-20660 introduced better estimation for group by operator. But the logic > did not account for Partial and Full group by separately. > For partial group by parallelism (i.e. number of tasks) should be taken into > account. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20842) Fix logic introduced in HIVE-20660 to estimate statistics for group by
[ https://issues.apache.org/jira/browse/HIVE-20842?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-20842: --- Attachment: HIVE-20842.10.patch > Fix logic introduced in HIVE-20660 to estimate statistics for group by > -- > > Key: HIVE-20842 > URL: https://issues.apache.org/jira/browse/HIVE-20842 > Project: Hive > Issue Type: Bug > Components: Statistics >Affects Versions: 4.0.0 >Reporter: Vineet Garg >Assignee: Vineet Garg >Priority: Major > Attachments: HIVE-20842.1.patch, HIVE-20842.10.patch, > HIVE-20842.2.patch, HIVE-20842.3.patch, HIVE-20842.4.patch, > HIVE-20842.5.patch, HIVE-20842.6.patch, HIVE-20842.7.patch, > HIVE-20842.8.patch, HIVE-20842.9.patch > > > HIVE-20660 introduced better estimation for group by operator. But the logic > did not account for Partial and Full group by separately. > For partial group by parallelism (i.e. number of tasks) should be taken into > account. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20842) Fix logic introduced in HIVE-20660 to estimate statistics for group by
[ https://issues.apache.org/jira/browse/HIVE-20842?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-20842: --- Status: Open (was: Patch Available) > Fix logic introduced in HIVE-20660 to estimate statistics for group by > -- > > Key: HIVE-20842 > URL: https://issues.apache.org/jira/browse/HIVE-20842 > Project: Hive > Issue Type: Bug > Components: Statistics >Affects Versions: 4.0.0 >Reporter: Vineet Garg >Assignee: Vineet Garg >Priority: Major > Attachments: HIVE-20842.1.patch, HIVE-20842.10.patch, > HIVE-20842.2.patch, HIVE-20842.3.patch, HIVE-20842.4.patch, > HIVE-20842.5.patch, HIVE-20842.6.patch, HIVE-20842.7.patch, > HIVE-20842.8.patch, HIVE-20842.9.patch > > > HIVE-20660 introduced better estimation for group by operator. But the logic > did not account for Partial and Full group by separately. > For partial group by parallelism (i.e. number of tasks) should be taken into > account. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20512) Improve record and memory usage logging in SparkRecordHandler
[
https://issues.apache.org/jira/browse/HIVE-20512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16684565#comment-16684565
]
Hive QA commented on HIVE-20512:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
21s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
4s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
36s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
48s{color} | {color:blue} ql in master has 2316 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
53s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
35s{color} | {color:green} ql: The patch generated 0 new + 4 unchanged - 5
fixed = 4 total (was 9) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 3m
51s{color} | {color:red} ql generated 1 new + 2316 unchanged - 0 fixed = 2317
total (was 2316) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 23m 16s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:ql |
| | Increment of volatile field
org.apache.hadoop.hive.ql.exec.spark.SparkRecordHandler.rowNumber in
org.apache.hadoop.hive.ql.exec.spark.SparkRecordHandler.incrementRowNumber()
At SparkRecordHandler.java:in
org.apache.hadoop.hive.ql.exec.spark.SparkRecordHandler.incrementRowNumber()
At SparkRecordHandler.java:[line 113] |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-14897/dev-support/hive-personality.sh
|
| git revision | master / af40170 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-14897/yetus/new-findbugs-ql.html
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-14897/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Improve record and memory usage logging in SparkRecordHandler
> -
>
> Key: HIVE-20512
> URL: https://issues.apache.org/jira/browse/HIVE-20512
> Project: Hive
> Issue Type: Sub-task
> Components: Spark
>Reporter: Sahil Takiar
>Assignee: Bharathkrishna Guruvayoor Murali
>Priority: Major
> Attachments: HIVE-20512.1.patch, HIVE-20512.2.patch,
> HIVE-20512.3.patch, HIVE-20512.4.patch, HIVE-20512.5.patch,
> HIVE-20512.6.patch, HIVE-20512.7.patch, HIVE-20512.8.patch,
> HIVE-20512.9.patch, HIVE-20512.91.patch
>
>
> We currently log memory usage and # of records processed in Spark tasks, but
> we should improve the methodology for how frequently we log this info.
> Currently we use the following code:
> {code:java}
> private long getNextLogThreshold(long currentThreshold) {
> // A very simple counter to keep track of number of rows processed by the
> // reducer. It dumps
> // every 1 million times, and quickly before that
> if
[jira] [Updated] (HIVE-20908) Avoid multiple getTableMeta calls during GetTablesOperation.
[ https://issues.apache.org/jira/browse/HIVE-20908?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajkumar Singh updated HIVE-20908: -- Attachment: HIVE-20908.patch Status: Patch Available (was: In Progress) > Avoid multiple getTableMeta calls during GetTablesOperation. > > > Key: HIVE-20908 > URL: https://issues.apache.org/jira/browse/HIVE-20908 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Affects Versions: 3.1.1 >Reporter: Rajkumar Singh >Assignee: Rajkumar Singh >Priority: Minor > Labels: performance > Attachments: HIVE-20908.patch > > > following HIVE-19432, we are doing getTableMeta for each authorized db > instead of that we can pass pattern for metastore. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20740) Remove global lock in ObjectStore.setConf method
[
https://issues.apache.org/jira/browse/HIVE-20740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16684538#comment-16684538
]
Hive QA commented on HIVE-20740:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12947891/HIVE-20740.12.patch
{color:green}SUCCESS:{color} +1 due to 3 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 12 failed/errored test(s), 15312 tests
executed
*Failed tests:*
{noformat}
TestDataSourceProviderFactory - did not produce a TEST-*.xml file (likely timed
out) (batchId=231)
TestFMSketchSerialization - did not produce a TEST-*.xml file (likely timed
out) (batchId=231)
TestHLLNoBias - did not produce a TEST-*.xml file (likely timed out)
(batchId=231)
TestHLLSerialization - did not produce a TEST-*.xml file (likely timed out)
(batchId=231)
TestHyperLogLog - did not produce a TEST-*.xml file (likely timed out)
(batchId=231)
TestHyperLogLogDense - did not produce a TEST-*.xml file (likely timed out)
(batchId=231)
TestHyperLogLogMerge - did not produce a TEST-*.xml file (likely timed out)
(batchId=231)
TestHyperLogLogSparse - did not produce a TEST-*.xml file (likely timed out)
(batchId=231)
TestObjectStore - did not produce a TEST-*.xml file (likely timed out)
(batchId=231)
TestPartitionProjectionEvaluator - did not produce a TEST-*.xml file (likely
timed out) (batchId=231)
TestRetriesInRetryingHMSHandler - did not produce a TEST-*.xml file (likely
timed out) (batchId=231)
TestSparseEncodeHash - did not produce a TEST-*.xml file (likely timed out)
(batchId=231)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/14896/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/14896/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-14896/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 12 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12947891 - PreCommit-HIVE-Build
> Remove global lock in ObjectStore.setConf method
>
>
> Key: HIVE-20740
> URL: https://issues.apache.org/jira/browse/HIVE-20740
> Project: Hive
> Issue Type: Improvement
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
>Priority: Major
> Attachments: HIVE-20740.01.patch, HIVE-20740.02.patch,
> HIVE-20740.04.patch, HIVE-20740.05.patch, HIVE-20740.06.patch,
> HIVE-20740.08.patch, HIVE-20740.09.patch, HIVE-20740.10.patch,
> HIVE-20740.11.patch, HIVE-20740.12.patch
>
>
> The ObjectStore#setConf method has a global lock which can block other
> clients in concurrent workloads.
> {code}
> @Override
> @SuppressWarnings("nls")
> public void setConf(Configuration conf) {
> // Although an instance of ObjectStore is accessed by one thread, there
> may
> // be many threads with ObjectStore instances. So the static variables
> // pmf and prop need to be protected with locks.
> pmfPropLock.lock();
> try {
> isInitialized = false;
> this.conf = conf;
> this.areTxnStatsSupported = MetastoreConf.getBoolVar(conf,
> ConfVars.HIVE_TXN_STATS_ENABLED);
> configureSSL(conf);
> Properties propsFromConf = getDataSourceProps(conf);
> boolean propsChanged = !propsFromConf.equals(prop);
> if (propsChanged) {
> if (pmf != null){
> clearOutPmfClassLoaderCache(pmf);
> if (!forTwoMetastoreTesting) {
> // close the underlying connection pool to avoid leaks
> pmf.close();
> }
> }
> pmf = null;
> prop = null;
> }
> assert(!isActiveTransaction());
> shutdown();
> // Always want to re-create pm as we don't know if it were created by
> the
> // most recent instance of the pmf
> pm = null;
> directSql = null;
> expressionProxy = null;
> openTrasactionCalls = 0;
> currentTransaction = null;
> transactionStatus = TXN_STATUS.NO_STATE;
> initialize(propsFromConf);
> String partitionValidationRegex =
> MetastoreConf.getVar(this.conf,
> ConfVars.PARTITION_NAME_WHITELIST_PATTERN);
> if (partitionValidationRegex != null &&
> !partitionValidationRegex.isEmpty()) {
> partitionValidationPattern =
> Pattern.compile(partitionValidationRegex);
> } else {
> partitionValidationPattern = null;
> }
> // Note, if metrics have not been initialized this will return null,
> which means we aren't
> //
[jira] [Updated] (HIVE-20807) Refactor LlapStatusServiceDriver
[ https://issues.apache.org/jira/browse/HIVE-20807?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-20807: Resolution: Fixed Status: Resolved (was: Patch Available) Pushed to master. Thanks, Miklos! > Refactor LlapStatusServiceDriver > > > Key: HIVE-20807 > URL: https://issues.apache.org/jira/browse/HIVE-20807 > Project: Hive > Issue Type: Improvement > Components: Hive >Affects Versions: 4.0.0 >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-20807.01.patch, HIVE-20807.02.patch, > HIVE-20807.03.patch, HIVE-20807.04.patch, HIVE-20807.05.patch, > HIVE-20807.06.patch > > > LlapStatusServiceDriver is the class used to determine if LLAP has started. > The following problems should be solved by refactoring: > 1. The main class is more than 800 lines long,should be cut into multiple > smaller classes. > 2. The current design makes it extremely hard to write unit tests. > 3. There are some overcomplicated, over-engineered parts of the code. > 4. Most of the code is under org.apache.hadoop.hive.llap.cli, but some parts > are under org.apache.hadoop.hive.llap.cli.status. The whole program could be > moved to the latter. > 5. LlapStatusHelpers serves as a class for holding classes, which doesn't > make much sense. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20740) Remove global lock in ObjectStore.setConf method
[
https://issues.apache.org/jira/browse/HIVE-20740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16684487#comment-16684487
]
Hive QA commented on HIVE-20740:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
27s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
17s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
0s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
1s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
37s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
42s{color} | {color:blue} ql in master has 2316 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 1m
4s{color} | {color:blue} standalone-metastore/metastore-server in master has
185 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
36s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
9s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
33s{color} | {color:red} hive-unit in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
31s{color} | {color:red} ql in the patch failed. {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
1s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
20s{color} | {color:red} itests/hive-unit: The patch generated 1 new + 608
unchanged - 0 fixed = 609 total (was 608) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
10s{color} | {color:red} standalone-metastore/metastore-server generated 1 new
+ 183 unchanged - 2 fixed = 184 total (was 185) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
37s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 30m 54s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:standalone-metastore/metastore-server |
| |
org.apache.hadoop.hive.metastore.PersistenceManagerProvider.updatePmfProperties(Configuration)
does not release lock on all paths At PersistenceManagerProvider.java:on all
paths At PersistenceManagerProvider.java:[line 152] |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-14896/dev-support/hive-personality.sh
|
| git revision | master / bc39c49 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| mvninstall |
http://104.198.109.242/logs//PreCommit-HIVE-Build-14896/yetus/patch-mvninstall-itests_hive-unit.txt
|
| mvninstall |
http://104.198.109.242/logs//PreCommit-HIVE-Build-14896/yetus/patch-mvninstall-ql.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-14896/yetus/diff-checkstyle-itests_hive-unit.txt
|
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-14896/yetus/new-findbugs-standalone-metastore_metastore-server.html
|
| modules | C: itests/hive-unit ql standalone-metastore/metastore-server U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-14896/yetus.txt |
| Powered by | Apache Yetus
[jira] [Commented] (HIVE-20807) Refactor LlapStatusServiceDriver
[ https://issues.apache.org/jira/browse/HIVE-20807?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16684486#comment-16684486 ] Miklos Gergely commented on HIVE-20807: --- Thank you [~sershe]! About the line lengths: there is kind of a confusion here, the following are contradicting each other: https://cwiki.apache.org/confluence/display/Hive/HowToContribute#HowToContribute-CodingConventions https://github.com/apache/hive/blob/bc39c49988c8a5d881a23ed7dd5d4adba0509ee9/checkstyle/checkstyle.xml#L163 The latter was introduced this January by [~kgyrtkirk], reviewed by [~ashutoshc], and there is some discussion about it at HIVE-18222. I personally prefer 120, but will go with 100 and fix all lines going forward in may patches to that, but then we should modify the checkstyle.xml as well. > Refactor LlapStatusServiceDriver > > > Key: HIVE-20807 > URL: https://issues.apache.org/jira/browse/HIVE-20807 > Project: Hive > Issue Type: Improvement > Components: Hive >Affects Versions: 4.0.0 >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-20807.01.patch, HIVE-20807.02.patch, > HIVE-20807.03.patch, HIVE-20807.04.patch, HIVE-20807.05.patch, > HIVE-20807.06.patch > > > LlapStatusServiceDriver is the class used to determine if LLAP has started. > The following problems should be solved by refactoring: > 1. The main class is more than 800 lines long,should be cut into multiple > smaller classes. > 2. The current design makes it extremely hard to write unit tests. > 3. There are some overcomplicated, over-engineered parts of the code. > 4. Most of the code is under org.apache.hadoop.hive.llap.cli, but some parts > are under org.apache.hadoop.hive.llap.cli.status. The whole program could be > moved to the latter. > 5. LlapStatusHelpers serves as a class for holding classes, which doesn't > make much sense. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19026) Configurable serde for druid kafka indexing
[
https://issues.apache.org/jira/browse/HIVE-19026?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16684463#comment-16684463
]
Hive QA commented on HIVE-19026:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12947890/HIVE-19026.5.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/14895/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/14895/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-14895/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2018-11-12 22:59:22.967
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-14895/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2018-11-12 22:59:22.970
+ cd apache-github-source-source
+ git fetch origin
+ git reset --hard HEAD
HEAD is now at bc39c49 HIVE-20899: Keytab URI for LLAP YARN Service is
restrictive to support HDFS only (Gour Saha reviewd by Prasanth Jayachandran)
+ git clean -f -d
Removing standalone-metastore/metastore-server/src/gen/
+ git checkout master
Already on 'master'
Your branch is up-to-date with 'origin/master'.
+ git reset --hard origin/master
HEAD is now at bc39c49 HIVE-20899: Keytab URI for LLAP YARN Service is
restrictive to support HDFS only (Gour Saha reviewd by Prasanth Jayachandran)
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2018-11-12 22:59:23.798
+ rm -rf ../yetus_PreCommit-HIVE-Build-14895
+ mkdir ../yetus_PreCommit-HIVE-Build-14895
+ git gc
+ cp -R . ../yetus_PreCommit-HIVE-Build-14895
+ mkdir /data/hiveptest/logs/PreCommit-HIVE-Build-14895/yetus
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh
/data/hiveptest/working/scratch/build.patch
error: a/common/src/java/org/apache/hadoop/hive/conf/Constants.java: does not
exist in index
error:
a/druid-handler/src/java/org/apache/hadoop/hive/druid/DruidStorageHandler.java:
does not exist in index
error:
a/druid-handler/src/java/org/apache/hadoop/hive/druid/DruidStorageHandlerUtils.java:
does not exist in index
error:
a/druid-handler/src/java/org/apache/hadoop/hive/druid/io/DruidOutputFormat.java:
does not exist in index
error:
a/druid-handler/src/java/org/apache/hadoop/hive/druid/io/DruidQueryBasedInputFormat.java:
does not exist in index
error: a/itests/qtest-druid/pom.xml: does not exist in index
error:
a/itests/qtest-druid/src/main/java/org/apache/hive/druid/MiniDruidCluster.java:
does not exist in index
error: a/itests/src/test/resources/testconfiguration.properties: does not exist
in index
error: a/itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java:
does not exist in index
error: patch failed:
common/src/java/org/apache/hadoop/hive/conf/Constants.java:32
Falling back to three-way merge...
Applied patch to 'common/src/java/org/apache/hadoop/hive/conf/Constants.java'
with conflicts.
error: patch failed:
druid-handler/src/java/org/apache/hadoop/hive/druid/DruidStorageHandler.java:17
Falling back to three-way merge...
Applied patch to
'druid-handler/src/java/org/apache/hadoop/hive/druid/DruidStorageHandler.java'
with conflicts.
error: patch failed:
druid-handler/src/java/org/apache/hadoop/hive/druid/DruidStorageHandlerUtils.java:92
Falling back to three-way merge...
Applied patch to
'druid-handler/src/java/org/apache/hadoop/hive/druid/DruidStorageHandlerUtils.java'
with conflicts.
error: patch failed:
druid-handler/src/java/org/apache/hadoop/hive/druid/io/DruidOutputFormat.java:89
Falling back to three-way merge...
Applied patch to
'druid-handler/src/java/org/apache/hadoop/hive/druid/io/DruidOutputFormat.java'
[jira] [Commented] (HIVE-20838) Timestamps with timezone are set to null when using the streaming API
[
https://issues.apache.org/jira/browse/HIVE-20838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16684461#comment-16684461
]
Hive QA commented on HIVE-20838:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12947877/HIVE-20838.10.patch
{color:green}SUCCESS:{color} +1 due to 3 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 15540 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/14894/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/14894/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-14894/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12947877 - PreCommit-HIVE-Build
> Timestamps with timezone are set to null when using the streaming API
> -
>
> Key: HIVE-20838
> URL: https://issues.apache.org/jira/browse/HIVE-20838
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 3.1.0
>Reporter: Jaume M
>Assignee: Jaume M
>Priority: Major
> Attachments: HIVE-20838.1.patch, HIVE-20838.10.patch,
> HIVE-20838.2.patch, HIVE-20838.3.patch, HIVE-20838.3.patch,
> HIVE-20838.4.patch, HIVE-20838.5.patch, HIVE-20838.6.patch,
> HIVE-20838.7.patch, HIVE-20838.8.patch, HIVE-20838.9.patch
>
>
> For example:
> {code}
> beeline> create table default.timest (a TIMESTAMP) stored as orc " +
> "TBLPROPERTIES('transactional'='true')
> # And then:
> connection.write("2018-10-19 10:35:00 America/Los_Angeles".getBytes());
> {code}
> inserts NULL.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20842) Fix logic introduced in HIVE-20660 to estimate statistics for group by
[ https://issues.apache.org/jira/browse/HIVE-20842?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-20842: --- Status: Open (was: Patch Available) > Fix logic introduced in HIVE-20660 to estimate statistics for group by > -- > > Key: HIVE-20842 > URL: https://issues.apache.org/jira/browse/HIVE-20842 > Project: Hive > Issue Type: Bug > Components: Statistics >Affects Versions: 4.0.0 >Reporter: Vineet Garg >Assignee: Vineet Garg >Priority: Major > Attachments: HIVE-20842.1.patch, HIVE-20842.2.patch, > HIVE-20842.3.patch, HIVE-20842.4.patch, HIVE-20842.5.patch, > HIVE-20842.6.patch, HIVE-20842.7.patch, HIVE-20842.8.patch, HIVE-20842.9.patch > > > HIVE-20660 introduced better estimation for group by operator. But the logic > did not account for Partial and Full group by separately. > For partial group by parallelism (i.e. number of tasks) should be taken into > account. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20842) Fix logic introduced in HIVE-20660 to estimate statistics for group by
[ https://issues.apache.org/jira/browse/HIVE-20842?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-20842: --- Attachment: HIVE-20842.9.patch > Fix logic introduced in HIVE-20660 to estimate statistics for group by > -- > > Key: HIVE-20842 > URL: https://issues.apache.org/jira/browse/HIVE-20842 > Project: Hive > Issue Type: Bug > Components: Statistics >Affects Versions: 4.0.0 >Reporter: Vineet Garg >Assignee: Vineet Garg >Priority: Major > Attachments: HIVE-20842.1.patch, HIVE-20842.2.patch, > HIVE-20842.3.patch, HIVE-20842.4.patch, HIVE-20842.5.patch, > HIVE-20842.6.patch, HIVE-20842.7.patch, HIVE-20842.8.patch, HIVE-20842.9.patch > > > HIVE-20660 introduced better estimation for group by operator. But the logic > did not account for Partial and Full group by separately. > For partial group by parallelism (i.e. number of tasks) should be taken into > account. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20842) Fix logic introduced in HIVE-20660 to estimate statistics for group by
[ https://issues.apache.org/jira/browse/HIVE-20842?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-20842: --- Status: Patch Available (was: Open) > Fix logic introduced in HIVE-20660 to estimate statistics for group by > -- > > Key: HIVE-20842 > URL: https://issues.apache.org/jira/browse/HIVE-20842 > Project: Hive > Issue Type: Bug > Components: Statistics >Affects Versions: 4.0.0 >Reporter: Vineet Garg >Assignee: Vineet Garg >Priority: Major > Attachments: HIVE-20842.1.patch, HIVE-20842.2.patch, > HIVE-20842.3.patch, HIVE-20842.4.patch, HIVE-20842.5.patch, > HIVE-20842.6.patch, HIVE-20842.7.patch, HIVE-20842.8.patch, HIVE-20842.9.patch > > > HIVE-20660 introduced better estimation for group by operator. But the logic > did not account for Partial and Full group by separately. > For partial group by parallelism (i.e. number of tasks) should be taken into > account. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20908) Avoid multiple getTableMeta calls during GetTablesOperation.
[ https://issues.apache.org/jira/browse/HIVE-20908?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajkumar Singh updated HIVE-20908: -- Summary: Avoid multiple getTableMeta calls during GetTablesOperation. (was: Avoid multiple getTableMeta calls per database) > Avoid multiple getTableMeta calls during GetTablesOperation. > > > Key: HIVE-20908 > URL: https://issues.apache.org/jira/browse/HIVE-20908 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Affects Versions: 3.1.1 >Reporter: Rajkumar Singh >Assignee: Rajkumar Singh >Priority: Minor > Labels: performance > > following HIVE-19432, we are doing getTableMeta for each authorized db > instead of that we can pass pattern for metastore. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20823) Make Compactor run in a transaction
[
https://issues.apache.org/jira/browse/HIVE-20823?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eugene Koifman updated HIVE-20823:
--
Attachment: HIVE-20823.10.patch
> Make Compactor run in a transaction
> ---
>
> Key: HIVE-20823
> URL: https://issues.apache.org/jira/browse/HIVE-20823
> Project: Hive
> Issue Type: Improvement
> Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
>Priority: Major
> Attachments: HIVE-20823.01.patch, HIVE-20823.03.patch,
> HIVE-20823.04.patch, HIVE-20823.05.patch, HIVE-20823.07.patch,
> HIVE-20823.08.patch, HIVE-20823.09.patch, HIVE-20823.10.patch
>
>
> Have compactor open a transaction and run the job in that transaction.
> # make compactor produced base/delta include this txn id in the folder name,
> e.g. base_7_c17 where 17 is the txnid.
> # add {{CQ_TXN_ID bigint}} to COMPACTION_QUEUE and COMPLETED_COMPACTIONS to
> record this txn id
> # make sure {{AcidUtils.getAcidState()}} pays attention to this transaction
> on read and ignores this dir if this txn id is not committed in the current
> snapshot
> ## this means not only validWriteIdList but ValidTxnIdList should be passed
> along in config (if it isn't yet)
> # once this is done, {{CompactorMR.createCompactorMarker()}} can be
> eliminated and {{AcidUtils.isValidBase}} modified accordingly
> # modify Cleaner so that it doesn't clean old files until new file is visible
> to all readers
> #
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20807) Refactor LlapStatusServiceDriver
[ https://issues.apache.org/jira/browse/HIVE-20807?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16684439#comment-16684439 ] Sergey Shelukhin commented on HIVE-20807: - +1 Nit: some lines are made longer than 100 chars that used to be shorter, and there weren't any logic changes to them. I think your auto-formatting might need to be set to a 100-char limit. > Refactor LlapStatusServiceDriver > > > Key: HIVE-20807 > URL: https://issues.apache.org/jira/browse/HIVE-20807 > Project: Hive > Issue Type: Improvement > Components: Hive >Affects Versions: 4.0.0 >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-20807.01.patch, HIVE-20807.02.patch, > HIVE-20807.03.patch, HIVE-20807.04.patch, HIVE-20807.05.patch, > HIVE-20807.06.patch > > > LlapStatusServiceDriver is the class used to determine if LLAP has started. > The following problems should be solved by refactoring: > 1. The main class is more than 800 lines long,should be cut into multiple > smaller classes. > 2. The current design makes it extremely hard to write unit tests. > 3. There are some overcomplicated, over-engineered parts of the code. > 4. Most of the code is under org.apache.hadoop.hive.llap.cli, but some parts > are under org.apache.hadoop.hive.llap.cli.status. The whole program could be > moved to the latter. > 5. LlapStatusHelpers serves as a class for holding classes, which doesn't > make much sense. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20838) Timestamps with timezone are set to null when using the streaming API
[
https://issues.apache.org/jira/browse/HIVE-20838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16684438#comment-16684438
]
Hive QA commented on HIVE-20838:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
33s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
14s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
50s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
16s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
30s{color} | {color:blue} common in master has 65 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
36s{color} | {color:blue} serde in master has 198 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
48s{color} | {color:blue} ql in master has 2316 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
26s{color} | {color:blue} streaming in master has 2 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
35s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
9s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
37s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 31m 49s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-14894/dev-support/hive-personality.sh
|
| git revision | master / bc39c49 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: common serde ql streaming U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-14894/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Timestamps with timezone are set to null when using the streaming API
> -
>
> Key: HIVE-20838
> URL: https://issues.apache.org/jira/browse/HIVE-20838
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 3.1.0
>Reporter: Jaume M
>Assignee: Jaume M
>Priority: Major
> Attachments: HIVE-20838.1.patch, HIVE-20838.10.patch,
> HIVE-20838.2.patch, HIVE-20838.3.patch, HIVE-20838.3.patch,
> HIVE-20838.4.patch, HIVE-20838.5.patch, HIVE-20838.6.patch,
> HIVE-20838.7.patch, HIVE-20838.8.patch, HIVE-20838.9.patch
>
>
> For example:
> {code}
> beeline> create table default.timest (a TIMESTAMP) stored as orc " +
> "TBLPROPERTIES('transactional'='true')
> # And then:
> connection.write("2018-10-19 10:35:00
[jira] [Commented] (HIVE-20907) TestGetPartitionsUsingProjectionAndFilterSpecs is flaky
[ https://issues.apache.org/jira/browse/HIVE-20907?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16684421#comment-16684421 ] Antal Sinkovits commented on HIVE-20907: [~vihangk1] I've faced the issue in https://issues.apache.org/jira/browse/HIVE-20904 > TestGetPartitionsUsingProjectionAndFilterSpecs is flaky > --- > > Key: HIVE-20907 > URL: https://issues.apache.org/jira/browse/HIVE-20907 > Project: Hive > Issue Type: Bug >Affects Versions: 4.0.0 >Reporter: Antal Sinkovits >Assignee: Antal Sinkovits >Priority: Major > Attachments: HIVE-20907.01.patch, HIVE-20907.02.patch > > > private void verifyLocations(List origPartitions, > StorageDescriptor sharedSD, > List partitionWithoutSDS) > method expects, that the order of the two list are the same. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20787) MapJoinBytesTableContainer dummyRow case doesn't handle reuse
[ https://issues.apache.org/jira/browse/HIVE-20787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-20787: Attachment: HIVE-20787.02.patch > MapJoinBytesTableContainer dummyRow case doesn't handle reuse > - > > Key: HIVE-20787 > URL: https://issues.apache.org/jira/browse/HIVE-20787 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-20787.01.patch, HIVE-20787.02.patch, > HIVE-20787.patch > > > Discovered while investigating some (probably) unrelated issue. > MapJoinBytesTableContainer was not intended to be reused, but it looks like > some code might reuse it. If that happens, the dummyRow case will not work > correctly (dummyRow is cleared on first(), so another call to first() will > behave differently). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20380) LLAP cache should cache small buffers more efficiently
[ https://issues.apache.org/jira/browse/HIVE-20380?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-20380: Description: The main thing to do is https://issues.apache.org/jira/browse/HIVE-20380?focusedCommentId=16591009=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16591009 There are also minor heap improvements possible. 1) Intern tracking tag. 2) Replace AtomicLong object with a long and unsafe CAS method, we only ever use one method, compareAndSwap. One more idea is making tracking less object oriented, in particular passing around integer indexes instead of objects and storing state in giant arrays somewhere (potentially with some optimizations for less common things), instead of every buffers getting its own object. cc [~gopalv] [~prasanth_j] was: Lately ORC CBs are becoming ridiculously small. First there's the 4Kb minimum (instead of 256Kb), then after we moved metadata cache off-heap, the index streams that are all tiny take up a lot of CBs and waste space. Wasted space can require larger cache and lead to cache OOMs on some workloads. Reducing min.alloc solves this problem, but then there's a lot of heap (and probably compute) overhead to track all these buffers. Arguably even the 4Kb min.alloc is too small. The initial idea was to store multiple CBs per block, however this is a lot of work all over the place (cache mapping, cache lookups, everywhere in the readers, etc.). Consolidating and reducing allocation sizes after we know the "real" size after decompression is the new idea (see comments) that can be confined to the allocator and is also more flexible - no dependence on cache map, so we don't need to make sure stuff is contiguous and such (for example, R_I streams that we want to consolidate are interleaved with large bloom filters, that we don't want to read or consolidate when they are not needed - but cache key structure depends on offsets, so we'd need a new cache map for R_I and separate logic for these streams). Also streams like PRESENT with one small CB cannot be combined with anything realistically speaking, but shrinking the allocation will help them. There are also minor heap improvements possible. 1) Intern tracking tag. 2) Replace AtomicLong object with a long and unsafe CAS method, we only ever use one method, compareAndSwap. One more idea is making tracking less object oriented, in particular passing around integer indexes instead of objects and storing state in giant arrays somewhere (potentially with some optimizations for less common things), instead of every buffers getting its own object. cc [~gopalv] [~prasanth_j] > LLAP cache should cache small buffers more efficiently > -- > > Key: HIVE-20380 > URL: https://issues.apache.org/jira/browse/HIVE-20380 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > > The main thing to do is > https://issues.apache.org/jira/browse/HIVE-20380?focusedCommentId=16591009=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16591009 > There are also minor heap improvements possible. > 1) Intern tracking tag. > 2) Replace AtomicLong object with a long and unsafe CAS method, we only ever > use one method, compareAndSwap. > One more idea is making tracking less object oriented, in particular passing > around integer indexes instead of objects and storing state in giant arrays > somewhere (potentially with some optimizations for less common things), > instead of every buffers getting its own object. > cc [~gopalv] [~prasanth_j] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20906) Hive locks are not really ephemeral
[
https://issues.apache.org/jira/browse/HIVE-20906?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16684389#comment-16684389
]
Hive QA commented on HIVE-20906:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12947863/ephemeral.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 15538 tests
executed
*Failed tests:*
{noformat}
org.apache.hive.jdbc.authorization.TestJdbcMetadataApiAuth.org.apache.hive.jdbc.authorization.TestJdbcMetadataApiAuth
(batchId=262)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/14893/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/14893/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-14893/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12947863 - PreCommit-HIVE-Build
> Hive locks are not really ephemeral
> ---
>
> Key: HIVE-20906
> URL: https://issues.apache.org/jira/browse/HIVE-20906
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.3.0, 2.3.3, 3.1.1
>Reporter: Rajesh KSV
>Priority: Major
> Fix For: 3.1.1
>
> Attachments: ephemeral.patch
>
>
> I am using Hive 2.3.3.
>
> According to docs
> ([https://cwiki.apache.org/confluence/display/Hive/Locking]), they are
> ephemeral. But in the Zookeeper and Hive source Code, they seemed to be
> non-ephemeral.
>
> I ran the following query in Hive.
>
> *lock table discovery.defaultns*
>
> When I see the lock in zookeeper, it doesn't look like ephemeral node.
>
> [zk: localhost:2181(CONNECTED) 55] get
> /hive_zookeeper_namespace/discovery/defaultns/LOCK-EXCLUSIVE-00
>
> hive_20181112191555_d6640955-2806-4fa7-9e52-30d04c377a04:1542030355565:EXPLICIT:lock
> table discovery.defaultns exclusive:10.33.222.139
> cZxid = 0x2400296808
> ctime = Mon Nov 12 19:15:55 IST 2018
> mZxid = 0x2400296808
> mtime = Mon Nov 12 19:15:55 IST 2018
> pZxid = 0x2400296808
> cversion = 0
> dataVersion = 0
> aclVersion = 0
> *ephemeralOwner = 0x0*
> dataLength = 134
> numChildren = 0
>
> Also in the code -
> [https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/lockmgr/HiveTxnManagerImpl.java],
> the lock is KeepAlive/PERSISTENT and not EPHIMERAL. Current code is
>
> *HiveLock lck = lockMgr.lock(new HiveLockObject(tbl, lockData), mode,
> true);*
> Instead it should be
> *HiveLock lck = lockMgr.lock(new HiveLockObject(tbl, lockData), mode,
> false);*
> The patch is the same.
> Please correct me if I am wrong. This is my first ever Jira/Patch in Hive
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20117) schema changes for txn stats
[ https://issues.apache.org/jira/browse/HIVE-20117?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-20117: Resolution: Implemented Status: Resolved (was: Patch Available) > schema changes for txn stats > > > Key: HIVE-20117 > URL: https://issues.apache.org/jira/browse/HIVE-20117 > Project: Hive > Issue Type: Bug > Components: Statistics, Transactions >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-20117.01.patch, HIVE-20117.02.patch, > HIVE-20117.03.patch, HIVE-20117.04.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19593) CLONE - CLONE - CLONE - NOOP jira to see which tests are flaky on HiveQA
[ https://issues.apache.org/jira/browse/HIVE-19593?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-19593: Resolution: Not A Problem Status: Resolved (was: Patch Available) > CLONE - CLONE - CLONE - NOOP jira to see which tests are flaky on HiveQA > > > Key: HIVE-19593 > URL: https://issues.apache.org/jira/browse/HIVE-19593 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-19593.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (HIVE-19596) NOOP jira to see which tests are flaky on HiveQA
[ https://issues.apache.org/jira/browse/HIVE-19596?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin resolved HIVE-19596. - Resolution: Not A Problem > NOOP jira to see which tests are flaky on HiveQA > > > Key: HIVE-19596 > URL: https://issues.apache.org/jira/browse/HIVE-19596 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-19596.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work started] (HIVE-20908) Avoid multiple getTableMeta calls per database
[ https://issues.apache.org/jira/browse/HIVE-20908?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HIVE-20908 started by Rajkumar Singh. - > Avoid multiple getTableMeta calls per database > -- > > Key: HIVE-20908 > URL: https://issues.apache.org/jira/browse/HIVE-20908 > Project: Hive > Issue Type: Bug >Affects Versions: 3.1.1 >Reporter: Rajkumar Singh >Assignee: Rajkumar Singh >Priority: Minor > > following HIVE-19432, we are doing getTableMeta for each authorized db > instead of that we can pass pattern for metastore. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20900) serde2.JsonSerDe no longer supports timestamp.formats
[ https://issues.apache.org/jira/browse/HIVE-20900?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Dere updated HIVE-20900: -- Status: Patch Available (was: Open) > serde2.JsonSerDe no longer supports timestamp.formats > - > > Key: HIVE-20900 > URL: https://issues.apache.org/jira/browse/HIVE-20900 > Project: Hive > Issue Type: Bug > Components: Serializers/Deserializers >Reporter: Jason Dere >Assignee: Jason Dere >Priority: Major > Attachments: HIVE-20900.1.patch > > > Looks like HIVE-18545 broke this. > Also json_serde_tsformat.q only tested the hcat version of JsonSerde, and the > format in that test used the ISO timestamp format which apparently is now > parsed by the default timestamp parsing, so the test was too simple. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20908) Avoid multiple getTableMeta calls per database
[ https://issues.apache.org/jira/browse/HIVE-20908?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajkumar Singh updated HIVE-20908: -- Component/s: HiveServer2 > Avoid multiple getTableMeta calls per database > -- > > Key: HIVE-20908 > URL: https://issues.apache.org/jira/browse/HIVE-20908 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Affects Versions: 3.1.1 >Reporter: Rajkumar Singh >Assignee: Rajkumar Singh >Priority: Minor > Labels: performance > > following HIVE-19432, we are doing getTableMeta for each authorized db > instead of that we can pass pattern for metastore. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20908) Avoid multiple getTableMeta calls per database
[ https://issues.apache.org/jira/browse/HIVE-20908?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajkumar Singh updated HIVE-20908: -- Labels: performance (was: ) > Avoid multiple getTableMeta calls per database > -- > > Key: HIVE-20908 > URL: https://issues.apache.org/jira/browse/HIVE-20908 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Affects Versions: 3.1.1 >Reporter: Rajkumar Singh >Assignee: Rajkumar Singh >Priority: Minor > Labels: performance > > following HIVE-19432, we are doing getTableMeta for each authorized db > instead of that we can pass pattern for metastore. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-20908) Avoid multiple getTableMeta calls per database
[ https://issues.apache.org/jira/browse/HIVE-20908?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajkumar Singh reassigned HIVE-20908: - Assignee: Rajkumar Singh > Avoid multiple getTableMeta calls per database > -- > > Key: HIVE-20908 > URL: https://issues.apache.org/jira/browse/HIVE-20908 > Project: Hive > Issue Type: Bug >Affects Versions: 3.1.1 >Reporter: Rajkumar Singh >Assignee: Rajkumar Singh >Priority: Minor > > following HIVE-19432, we are doing getTableMeta for each authorized db > instead of that we can pass pattern for metastore. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20908) Avoid multiple getTableMeta calls per database
[ https://issues.apache.org/jira/browse/HIVE-20908?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajkumar Singh updated HIVE-20908: -- Affects Version/s: 3.1.1 > Avoid multiple getTableMeta calls per database > -- > > Key: HIVE-20908 > URL: https://issues.apache.org/jira/browse/HIVE-20908 > Project: Hive > Issue Type: Bug >Affects Versions: 3.1.1 >Reporter: Rajkumar Singh >Assignee: Rajkumar Singh >Priority: Minor > > following HIVE-19432, we are doing getTableMeta for each authorized db > instead of that we can pass pattern for metastore. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20899) Keytab URI for LLAP YARN Service is restrictive to support HDFS only
[ https://issues.apache.org/jira/browse/HIVE-20899?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-20899: - Resolution: Fixed Fix Version/s: 3.2.0 4.0.0 Status: Resolved (was: Patch Available) Committed to master and branch-3. Thanks [~gsaha] for the patch! > Keytab URI for LLAP YARN Service is restrictive to support HDFS only > > > Key: HIVE-20899 > URL: https://issues.apache.org/jira/browse/HIVE-20899 > Project: Hive > Issue Type: Bug > Components: llap >Affects Versions: 3.1.1 >Reporter: Gour Saha >Assignee: Gour Saha >Priority: Major > Fix For: 4.0.0, 3.2.0 > > Attachments: HIVE-20899.01.patch, HIVE-20899.02.patch, > HIVE-20899.03.patch, HIVE-20899.04.patch > > > llap-server/src/main/resources/package.py restricts the keytab URI to support > HDFS only and hence fails for other FileSystem API conforming FSs like s3a, > wasb, gs, etc. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20900) serde2.JsonSerDe no longer supports timestamp.formats
[ https://issues.apache.org/jira/browse/HIVE-20900?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Dere updated HIVE-20900: -- Attachment: HIVE-20900.1.patch > serde2.JsonSerDe no longer supports timestamp.formats > - > > Key: HIVE-20900 > URL: https://issues.apache.org/jira/browse/HIVE-20900 > Project: Hive > Issue Type: Bug > Components: Serializers/Deserializers >Reporter: Jason Dere >Assignee: Jason Dere >Priority: Major > Attachments: HIVE-20900.1.patch > > > Looks like HIVE-18545 broke this. > Also json_serde_tsformat.q only tested the hcat version of JsonSerde, and the > format in that test used the ISO timestamp format which apparently is now > parsed by the default timestamp parsing, so the test was too simple. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20906) Hive locks are not really ephemeral
[
https://issues.apache.org/jira/browse/HIVE-20906?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16684356#comment-16684356
]
Hive QA commented on HIVE-20906:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
33s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
0s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
35s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
37s{color} | {color:blue} ql in master has 2316 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
53s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
12s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 22m 8s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-14893/dev-support/hive-personality.sh
|
| git revision | master / 5588764 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-14893/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Hive locks are not really ephemeral
> ---
>
> Key: HIVE-20906
> URL: https://issues.apache.org/jira/browse/HIVE-20906
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.3.0, 2.3.3, 3.1.1
>Reporter: Rajesh KSV
>Priority: Major
> Fix For: 3.1.1
>
> Attachments: ephemeral.patch
>
>
> I am using Hive 2.3.3.
>
> According to docs
> ([https://cwiki.apache.org/confluence/display/Hive/Locking]), they are
> ephemeral. But in the Zookeeper and Hive source Code, they seemed to be
> non-ephemeral.
>
> I ran the following query in Hive.
>
> *lock table discovery.defaultns*
>
> When I see the lock in zookeeper, it doesn't look like ephemeral node.
>
> [zk: localhost:2181(CONNECTED) 55] get
> /hive_zookeeper_namespace/discovery/defaultns/LOCK-EXCLUSIVE-00
>
> hive_20181112191555_d6640955-2806-4fa7-9e52-30d04c377a04:1542030355565:EXPLICIT:lock
> table discovery.defaultns exclusive:10.33.222.139
> cZxid = 0x2400296808
> ctime = Mon Nov 12 19:15:55 IST 2018
> mZxid = 0x2400296808
> mtime = Mon Nov 12 19:15:55 IST 2018
> pZxid = 0x2400296808
> cversion = 0
> dataVersion = 0
> aclVersion = 0
> *ephemeralOwner = 0x0*
> dataLength = 134
> numChildren = 0
>
> Also in the code -
> [https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/lockmgr/HiveTxnManagerImpl.java],
> the lock is KeepAlive/PERSISTENT and not EPHIMERAL. Current code is
>
> *HiveLock lck = lockMgr.lock(new HiveLockObject(tbl, lockData), mode,
> true);*
> Instead it should be
> *HiveLock lck = lockMgr.lock(new HiveLockObject(tbl,
[jira] [Updated] (HIVE-20512) Improve record and memory usage logging in SparkRecordHandler
[
https://issues.apache.org/jira/browse/HIVE-20512?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bharathkrishna Guruvayoor Murali updated HIVE-20512:
Attachment: HIVE-20512.91.patch
> Improve record and memory usage logging in SparkRecordHandler
> -
>
> Key: HIVE-20512
> URL: https://issues.apache.org/jira/browse/HIVE-20512
> Project: Hive
> Issue Type: Sub-task
> Components: Spark
>Reporter: Sahil Takiar
>Assignee: Bharathkrishna Guruvayoor Murali
>Priority: Major
> Attachments: HIVE-20512.1.patch, HIVE-20512.2.patch,
> HIVE-20512.3.patch, HIVE-20512.4.patch, HIVE-20512.5.patch,
> HIVE-20512.6.patch, HIVE-20512.7.patch, HIVE-20512.8.patch,
> HIVE-20512.9.patch, HIVE-20512.91.patch
>
>
> We currently log memory usage and # of records processed in Spark tasks, but
> we should improve the methodology for how frequently we log this info.
> Currently we use the following code:
> {code:java}
> private long getNextLogThreshold(long currentThreshold) {
> // A very simple counter to keep track of number of rows processed by the
> // reducer. It dumps
> // every 1 million times, and quickly before that
> if (currentThreshold >= 100) {
> return currentThreshold + 100;
> }
> return 10 * currentThreshold;
> }
> {code}
> The issue is that after a while, the increase by 10x factor means that you
> have to process a huge # of records before this gets triggered.
> A better approach would be to log this info at a given interval. This would
> help in debugging tasks that are seemingly hung.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20512) Improve record and memory usage logging in SparkRecordHandler
[
https://issues.apache.org/jira/browse/HIVE-20512?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bharathkrishna Guruvayoor Murali updated HIVE-20512:
Attachment: (was: HIVE-20512.10.patch)
> Improve record and memory usage logging in SparkRecordHandler
> -
>
> Key: HIVE-20512
> URL: https://issues.apache.org/jira/browse/HIVE-20512
> Project: Hive
> Issue Type: Sub-task
> Components: Spark
>Reporter: Sahil Takiar
>Assignee: Bharathkrishna Guruvayoor Murali
>Priority: Major
> Attachments: HIVE-20512.1.patch, HIVE-20512.2.patch,
> HIVE-20512.3.patch, HIVE-20512.4.patch, HIVE-20512.5.patch,
> HIVE-20512.6.patch, HIVE-20512.7.patch, HIVE-20512.8.patch, HIVE-20512.9.patch
>
>
> We currently log memory usage and # of records processed in Spark tasks, but
> we should improve the methodology for how frequently we log this info.
> Currently we use the following code:
> {code:java}
> private long getNextLogThreshold(long currentThreshold) {
> // A very simple counter to keep track of number of rows processed by the
> // reducer. It dumps
> // every 1 million times, and quickly before that
> if (currentThreshold >= 100) {
> return currentThreshold + 100;
> }
> return 10 * currentThreshold;
> }
> {code}
> The issue is that after a while, the increase by 10x factor means that you
> have to process a huge # of records before this gets triggered.
> A better approach would be to log this info at a given interval. This would
> help in debugging tasks that are seemingly hung.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-12014) re-enable most LLAP tests after merge
[ https://issues.apache.org/jira/browse/HIVE-12014?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-12014: Resolution: Not A Problem Status: Resolved (was: Patch Available) > re-enable most LLAP tests after merge > - > > Key: HIVE-12014 > URL: https://issues.apache.org/jira/browse/HIVE-12014 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-12014.patch > > > see HIVE-12013 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20512) Improve record and memory usage logging in SparkRecordHandler
[
https://issues.apache.org/jira/browse/HIVE-20512?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bharathkrishna Guruvayoor Murali updated HIVE-20512:
Attachment: HIVE-20512.10.patch
> Improve record and memory usage logging in SparkRecordHandler
> -
>
> Key: HIVE-20512
> URL: https://issues.apache.org/jira/browse/HIVE-20512
> Project: Hive
> Issue Type: Sub-task
> Components: Spark
>Reporter: Sahil Takiar
>Assignee: Bharathkrishna Guruvayoor Murali
>Priority: Major
> Attachments: HIVE-20512.1.patch, HIVE-20512.10.patch,
> HIVE-20512.2.patch, HIVE-20512.3.patch, HIVE-20512.4.patch,
> HIVE-20512.5.patch, HIVE-20512.6.patch, HIVE-20512.7.patch,
> HIVE-20512.8.patch, HIVE-20512.9.patch
>
>
> We currently log memory usage and # of records processed in Spark tasks, but
> we should improve the methodology for how frequently we log this info.
> Currently we use the following code:
> {code:java}
> private long getNextLogThreshold(long currentThreshold) {
> // A very simple counter to keep track of number of rows processed by the
> // reducer. It dumps
> // every 1 million times, and quickly before that
> if (currentThreshold >= 100) {
> return currentThreshold + 100;
> }
> return 10 * currentThreshold;
> }
> {code}
> The issue is that after a while, the increase by 10x factor means that you
> have to process a huge # of records before this gets triggered.
> A better approach would be to log this info at a given interval. This would
> help in debugging tasks that are seemingly hung.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20899) Keytab URI for LLAP YARN Service is restrictive to support HDFS only
[ https://issues.apache.org/jira/browse/HIVE-20899?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16684340#comment-16684340 ] Gour Saha commented on HIVE-20899: -- bq. Wouldn't this break backward compat for paths specified the old way? We can check the path for being the full URI and append if it's not. +-0 [~sershe] not to worry. This will not cause any backward incompatibility. Any store conforming to hadoop FS API will work, including relative paths based on fs.defaultFS value. In fact, if you look at history including Slider - these 2 lines of code did not even exist. It was added only recently due to the constraint that YARN Service imposed to support hdfs and file URIs only, which now has been removed with YARN-9002. > Keytab URI for LLAP YARN Service is restrictive to support HDFS only > > > Key: HIVE-20899 > URL: https://issues.apache.org/jira/browse/HIVE-20899 > Project: Hive > Issue Type: Bug > Components: llap >Affects Versions: 3.1.1 >Reporter: Gour Saha >Assignee: Gour Saha >Priority: Major > Attachments: HIVE-20899.01.patch, HIVE-20899.02.patch, > HIVE-20899.03.patch, HIVE-20899.04.patch > > > llap-server/src/main/resources/package.py restricts the keytab URI to support > HDFS only and hence fails for other FileSystem API conforming FSs like s3a, > wasb, gs, etc. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19026) Configurable serde for druid kafka indexing
[ https://issues.apache.org/jira/browse/HIVE-19026?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nishant Bangarwa updated HIVE-19026: Attachment: HIVE-19026.5.patch > Configurable serde for druid kafka indexing > > > Key: HIVE-19026 > URL: https://issues.apache.org/jira/browse/HIVE-19026 > Project: Hive > Issue Type: Task >Reporter: Nishant Bangarwa >Assignee: Nishant Bangarwa >Priority: Major > Attachments: HIVE-19026.1.patch, HIVE-19026.2.patch, > HIVE-19026.3.patch, HIVE-19026.4.patch, HIVE-19026.5.patch, HIVE-19026.patch > > > https://issues.apache.org/jira/browse/HIVE-18976 introduces support for > setting up druid kafka-indexing service. > Input serialization should be configurable. for now we can say we only > support json, but there should be a mechanism to support other formats. > Perhaps, we can make use of Hive's serde library like LazySimpleSerde etc. > Also add support to ingest timestamp column when the input timestamp column > name in input is not `__time`. > e.g. > CREATE EXTERNAL TABLE druid_kafka_test_avro(__time timestamp , other > columns...) > STORED BY 'org.apache.hadoop.hive.druid.DruidStorageHandler' > TBLPROPERTIES ( > "druid.timestamp.column" = "myinputColumnTimestamp" > other ppts > ) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20899) Keytab URI for LLAP YARN Service is restrictive to support HDFS only
[
https://issues.apache.org/jira/browse/HIVE-20899?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16684334#comment-16684334
]
Hive QA commented on HIVE-20899:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12947860/HIVE-20899.04.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 15539 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/14892/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/14892/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-14892/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12947860 - PreCommit-HIVE-Build
> Keytab URI for LLAP YARN Service is restrictive to support HDFS only
>
>
> Key: HIVE-20899
> URL: https://issues.apache.org/jira/browse/HIVE-20899
> Project: Hive
> Issue Type: Bug
> Components: llap
>Affects Versions: 3.1.1
>Reporter: Gour Saha
>Assignee: Gour Saha
>Priority: Major
> Attachments: HIVE-20899.01.patch, HIVE-20899.02.patch,
> HIVE-20899.03.patch, HIVE-20899.04.patch
>
>
> llap-server/src/main/resources/package.py restricts the keytab URI to support
> HDFS only and hence fails for other FileSystem API conforming FSs like s3a,
> wasb, gs, etc.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20740) Remove global lock in ObjectStore.setConf method
[
https://issues.apache.org/jira/browse/HIVE-20740?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vihang Karajgaonkar updated HIVE-20740:
---
Attachment: HIVE-20740.12.patch
> Remove global lock in ObjectStore.setConf method
>
>
> Key: HIVE-20740
> URL: https://issues.apache.org/jira/browse/HIVE-20740
> Project: Hive
> Issue Type: Improvement
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
>Priority: Major
> Attachments: HIVE-20740.01.patch, HIVE-20740.02.patch,
> HIVE-20740.04.patch, HIVE-20740.05.patch, HIVE-20740.06.patch,
> HIVE-20740.08.patch, HIVE-20740.09.patch, HIVE-20740.10.patch,
> HIVE-20740.11.patch, HIVE-20740.12.patch
>
>
> The ObjectStore#setConf method has a global lock which can block other
> clients in concurrent workloads.
> {code}
> @Override
> @SuppressWarnings("nls")
> public void setConf(Configuration conf) {
> // Although an instance of ObjectStore is accessed by one thread, there
> may
> // be many threads with ObjectStore instances. So the static variables
> // pmf and prop need to be protected with locks.
> pmfPropLock.lock();
> try {
> isInitialized = false;
> this.conf = conf;
> this.areTxnStatsSupported = MetastoreConf.getBoolVar(conf,
> ConfVars.HIVE_TXN_STATS_ENABLED);
> configureSSL(conf);
> Properties propsFromConf = getDataSourceProps(conf);
> boolean propsChanged = !propsFromConf.equals(prop);
> if (propsChanged) {
> if (pmf != null){
> clearOutPmfClassLoaderCache(pmf);
> if (!forTwoMetastoreTesting) {
> // close the underlying connection pool to avoid leaks
> pmf.close();
> }
> }
> pmf = null;
> prop = null;
> }
> assert(!isActiveTransaction());
> shutdown();
> // Always want to re-create pm as we don't know if it were created by
> the
> // most recent instance of the pmf
> pm = null;
> directSql = null;
> expressionProxy = null;
> openTrasactionCalls = 0;
> currentTransaction = null;
> transactionStatus = TXN_STATUS.NO_STATE;
> initialize(propsFromConf);
> String partitionValidationRegex =
> MetastoreConf.getVar(this.conf,
> ConfVars.PARTITION_NAME_WHITELIST_PATTERN);
> if (partitionValidationRegex != null &&
> !partitionValidationRegex.isEmpty()) {
> partitionValidationPattern =
> Pattern.compile(partitionValidationRegex);
> } else {
> partitionValidationPattern = null;
> }
> // Note, if metrics have not been initialized this will return null,
> which means we aren't
> // using metrics. Thus we should always check whether this is non-null
> before using.
> MetricRegistry registry = Metrics.getRegistry();
> if (registry != null) {
> directSqlErrors =
> Metrics.getOrCreateCounter(MetricsConstants.DIRECTSQL_ERRORS);
> }
> this.batchSize = MetastoreConf.getIntVar(conf,
> ConfVars.RAWSTORE_PARTITION_BATCH_SIZE);
> if (!isInitialized) {
> throw new RuntimeException(
> "Unable to create persistence manager. Check dss.log for details");
> } else {
> LOG.debug("Initialized ObjectStore");
> }
> } finally {
> pmfPropLock.unlock();
> }
> }
> {code}
> The {{pmfPropLock}} is a static object and it disallows any other new
> connection to HMS which is trying to instantiate ObjectStore. We should
> either remove the lock or reduce the scope of the lock so that it is held for
> a very small amount of time.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20907) TestGetPartitionsUsingProjectionAndFilterSpecs is flaky
[
https://issues.apache.org/jira/browse/HIVE-20907?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16684328#comment-16684328
]
Vihang Karajgaonkar commented on HIVE-20907:
Hi [~asinkovits] Thanks for the patch. Can you give a example of test failure?
The partitions are returned sorted by PART_ID so I wonder why the order of
partitions is different that partitions returned by {{getPartitions}}
> TestGetPartitionsUsingProjectionAndFilterSpecs is flaky
> ---
>
> Key: HIVE-20907
> URL: https://issues.apache.org/jira/browse/HIVE-20907
> Project: Hive
> Issue Type: Bug
>Affects Versions: 4.0.0
>Reporter: Antal Sinkovits
>Assignee: Antal Sinkovits
>Priority: Major
> Attachments: HIVE-20907.01.patch, HIVE-20907.02.patch
>
>
> private void verifyLocations(List origPartitions,
> StorageDescriptor sharedSD,
> List partitionWithoutSDS)
> method expects, that the order of the two list are the same.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20845) Fix TestJdbcWithDBTokenStoreNoDoAs flakiness
[ https://issues.apache.org/jira/browse/HIVE-20845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16684323#comment-16684323 ] Sergey Shelukhin commented on HIVE-20845: - [~pvary] that is specific to some unit tests that start HS2 in passive mode (HA tests). HS2 that comes up as passive will not accept the session, so starting one is not a valid liveness check. Not sure why it won't repro... I think I hit it on the Mac before fixing. > Fix TestJdbcWithDBTokenStoreNoDoAs flakiness > > > Key: HIVE-20845 > URL: https://issues.apache.org/jira/browse/HIVE-20845 > Project: Hive > Issue Type: Test >Reporter: Peter Vary >Assignee: Peter Vary >Priority: Major > Attachments: HIVE-20845.2.patch, HIVE-20845.patch > > > Previously did a dirty fix for TestJdbcWithDBTokenStoreNoDoAs and > TestJdbcWithDBTokenStore > Found out the issue is that we do not wait enough for HS2 to come up. > Need to fix in MiniHS2.waitForStartup() -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20899) Keytab URI for LLAP YARN Service is restrictive to support HDFS only
[ https://issues.apache.org/jira/browse/HIVE-20899?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16684291#comment-16684291 ] Sergey Shelukhin commented on HIVE-20899: - Wouldn't this break backward compat for paths specified the old way? We can check the path for being the full URI and append if it's not. +-0 > Keytab URI for LLAP YARN Service is restrictive to support HDFS only > > > Key: HIVE-20899 > URL: https://issues.apache.org/jira/browse/HIVE-20899 > Project: Hive > Issue Type: Bug > Components: llap >Affects Versions: 3.1.1 >Reporter: Gour Saha >Assignee: Gour Saha >Priority: Major > Attachments: HIVE-20899.01.patch, HIVE-20899.02.patch, > HIVE-20899.03.patch, HIVE-20899.04.patch > > > llap-server/src/main/resources/package.py restricts the keytab URI to support > HDFS only and hence fails for other FileSystem API conforming FSs like s3a, > wasb, gs, etc. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20899) Keytab URI for LLAP YARN Service is restrictive to support HDFS only
[
https://issues.apache.org/jira/browse/HIVE-20899?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16684288#comment-16684288
]
Hive QA commented on HIVE-20899:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
39s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
14s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
12s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 9m 1s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-14892/dev-support/hive-personality.sh
|
| git revision | master / 5588764 |
| Default Java | 1.8.0_111 |
| modules | C: llap-server U: llap-server |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-14892/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Keytab URI for LLAP YARN Service is restrictive to support HDFS only
>
>
> Key: HIVE-20899
> URL: https://issues.apache.org/jira/browse/HIVE-20899
> Project: Hive
> Issue Type: Bug
> Components: llap
>Affects Versions: 3.1.1
>Reporter: Gour Saha
>Assignee: Gour Saha
>Priority: Major
> Attachments: HIVE-20899.01.patch, HIVE-20899.02.patch,
> HIVE-20899.03.patch, HIVE-20899.04.patch
>
>
> llap-server/src/main/resources/package.py restricts the keytab URI to support
> HDFS only and hence fails for other FileSystem API conforming FSs like s3a,
> wasb, gs, etc.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20838) Timestamps with timezone are set to null when using the streaming API
[
https://issues.apache.org/jira/browse/HIVE-20838?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jaume M updated HIVE-20838:
---
Status: Open (was: Patch Available)
> Timestamps with timezone are set to null when using the streaming API
> -
>
> Key: HIVE-20838
> URL: https://issues.apache.org/jira/browse/HIVE-20838
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 3.1.0
>Reporter: Jaume M
>Assignee: Jaume M
>Priority: Major
> Attachments: HIVE-20838.1.patch, HIVE-20838.10.patch,
> HIVE-20838.2.patch, HIVE-20838.3.patch, HIVE-20838.3.patch,
> HIVE-20838.4.patch, HIVE-20838.5.patch, HIVE-20838.6.patch,
> HIVE-20838.7.patch, HIVE-20838.8.patch, HIVE-20838.9.patch
>
>
> For example:
> {code}
> beeline> create table default.timest (a TIMESTAMP) stored as orc " +
> "TBLPROPERTIES('transactional'='true')
> # And then:
> connection.write("2018-10-19 10:35:00 America/Los_Angeles".getBytes());
> {code}
> inserts NULL.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20907) TestGetPartitionsUsingProjectionAndFilterSpecs is flaky
[
https://issues.apache.org/jira/browse/HIVE-20907?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16684265#comment-16684265
]
Hive QA commented on HIVE-20907:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12947854/HIVE-20907.02.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 15539 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/14891/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/14891/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-14891/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12947854 - PreCommit-HIVE-Build
> TestGetPartitionsUsingProjectionAndFilterSpecs is flaky
> ---
>
> Key: HIVE-20907
> URL: https://issues.apache.org/jira/browse/HIVE-20907
> Project: Hive
> Issue Type: Bug
>Affects Versions: 4.0.0
>Reporter: Antal Sinkovits
>Assignee: Antal Sinkovits
>Priority: Major
> Attachments: HIVE-20907.01.patch, HIVE-20907.02.patch
>
>
> private void verifyLocations(List origPartitions,
> StorageDescriptor sharedSD,
> List partitionWithoutSDS)
> method expects, that the order of the two list are the same.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20838) Timestamps with timezone are set to null when using the streaming API
[
https://issues.apache.org/jira/browse/HIVE-20838?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jaume M updated HIVE-20838:
---
Attachment: HIVE-20838.10.patch
Status: Patch Available (was: Open)
> Timestamps with timezone are set to null when using the streaming API
> -
>
> Key: HIVE-20838
> URL: https://issues.apache.org/jira/browse/HIVE-20838
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 3.1.0
>Reporter: Jaume M
>Assignee: Jaume M
>Priority: Major
> Attachments: HIVE-20838.1.patch, HIVE-20838.10.patch,
> HIVE-20838.2.patch, HIVE-20838.3.patch, HIVE-20838.3.patch,
> HIVE-20838.4.patch, HIVE-20838.5.patch, HIVE-20838.6.patch,
> HIVE-20838.7.patch, HIVE-20838.8.patch, HIVE-20838.9.patch
>
>
> For example:
> {code}
> beeline> create table default.timest (a TIMESTAMP) stored as orc " +
> "TBLPROPERTIES('transactional'='true')
> # And then:
> connection.write("2018-10-19 10:35:00 America/Los_Angeles".getBytes());
> {code}
> inserts NULL.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19026) Configurable serde for druid kafka indexing
[ https://issues.apache.org/jira/browse/HIVE-19026?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16684235#comment-16684235 ] Ashutosh Chauhan commented on HIVE-19026: - +1 > Configurable serde for druid kafka indexing > > > Key: HIVE-19026 > URL: https://issues.apache.org/jira/browse/HIVE-19026 > Project: Hive > Issue Type: Task >Reporter: Nishant Bangarwa >Assignee: Nishant Bangarwa >Priority: Major > Attachments: HIVE-19026.1.patch, HIVE-19026.2.patch, > HIVE-19026.3.patch, HIVE-19026.4.patch, HIVE-19026.patch > > > https://issues.apache.org/jira/browse/HIVE-18976 introduces support for > setting up druid kafka-indexing service. > Input serialization should be configurable. for now we can say we only > support json, but there should be a mechanism to support other formats. > Perhaps, we can make use of Hive's serde library like LazySimpleSerde etc. > Also add support to ingest timestamp column when the input timestamp column > name in input is not `__time`. > e.g. > CREATE EXTERNAL TABLE druid_kafka_test_avro(__time timestamp , other > columns...) > STORED BY 'org.apache.hadoop.hive.druid.DruidStorageHandler' > TBLPROPERTIES ( > "druid.timestamp.column" = "myinputColumnTimestamp" > other ppts > ) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20907) TestGetPartitionsUsingProjectionAndFilterSpecs is flaky
[
https://issues.apache.org/jira/browse/HIVE-20907?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16684209#comment-16684209
]
Hive QA commented on HIVE-20907:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
39s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
24s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
6s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 1m
4s{color} | {color:blue} standalone-metastore/metastore-server in master has
185 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
17s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
17s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 12m 34s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-14891/dev-support/hive-personality.sh
|
| git revision | master / 5588764 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: standalone-metastore/metastore-server U:
standalone-metastore/metastore-server |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-14891/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> TestGetPartitionsUsingProjectionAndFilterSpecs is flaky
> ---
>
> Key: HIVE-20907
> URL: https://issues.apache.org/jira/browse/HIVE-20907
> Project: Hive
> Issue Type: Bug
>Affects Versions: 4.0.0
>Reporter: Antal Sinkovits
>Assignee: Antal Sinkovits
>Priority: Major
> Attachments: HIVE-20907.01.patch, HIVE-20907.02.patch
>
>
> private void verifyLocations(List origPartitions,
> StorageDescriptor sharedSD,
> List partitionWithoutSDS)
> method expects, that the order of the two list are the same.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20905) querying streaming table fails with out of memory exception
[ https://issues.apache.org/jira/browse/HIVE-20905?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16684189#comment-16684189 ] Eugene Koifman commented on HIVE-20905: --- Is modifying "conf" issue? This is likely shared with other parts of the metastore code. If't setting uris to "" is appropriate perhaps it should be done for the whole metastore > querying streaming table fails with out of memory exception > --- > > Key: HIVE-20905 > URL: https://issues.apache.org/jira/browse/HIVE-20905 > Project: Hive > Issue Type: Bug > Components: Metastore, Transactions >Affects Versions: 4.0.0 >Reporter: mahesh kumar behera >Assignee: mahesh kumar behera >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-20905.01.patch, HIVE-20905.02.patch > > > Streaming app was ran for 24hrs post which it went down due authentication > issue . The table was accessible for 12hrs into the run, however currently > querying the table fails with OOM exception. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20905) querying streaming table fails with out of memory exception
[ https://issues.apache.org/jira/browse/HIVE-20905?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman updated HIVE-20905: -- Component/s: (was: Hive) Transactions Metastore > querying streaming table fails with out of memory exception > --- > > Key: HIVE-20905 > URL: https://issues.apache.org/jira/browse/HIVE-20905 > Project: Hive > Issue Type: Bug > Components: Metastore, Transactions >Affects Versions: 4.0.0 >Reporter: mahesh kumar behera >Assignee: mahesh kumar behera >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-20905.01.patch, HIVE-20905.02.patch > > > Streaming app was ran for 24hrs post which it went down due authentication > issue . The table was accessible for 12hrs into the run, however currently > querying the table fails with OOM exception. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20850) Push case conditional from projections to dimension tables if possible
[
https://issues.apache.org/jira/browse/HIVE-20850?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jesus Camacho Rodriguez updated HIVE-20850:
---
Resolution: Fixed
Fix Version/s: 4.0.0
Status: Resolved (was: Patch Available)
Pushed to master, thanks [~kgyrtkirk]!
> Push case conditional from projections to dimension tables if possible
> --
>
> Key: HIVE-20850
> URL: https://issues.apache.org/jira/browse/HIVE-20850
> Project: Hive
> Issue Type: Improvement
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-20850.01.patch, HIVE-20850.01.patch,
> HIVE-20850.01.patch, HIVE-20850.02.patch, HIVE-20850.03.patch,
> HIVE-20850.04.patch, HIVE-20850.05.patch, HIVE-20850.06.patch,
> HIVE-20850.07.patch, HIVE-20850.07.patch
>
>
> noticed by [~gopalv]: If there is a project which could be only evaluated
> after the join; but the condition references only a single column from a
> small dimension table; hive will end up evaluating the same thing over and
> over again...
> {code}
> explain
> select s_store_name, s_store_id,
> sum(case when (d_day_name='Sunday') then ss_sales_price else null
> end) sun_sales,
> sum(case when (d_day_name='Monday') then ss_sales_price else null
> end) mon_sales,
> sum(case when (d_day_name='Tuesday') then ss_sales_price else null
> end) tue_sales,
> sum(case when (d_day_name='Wednesday') then ss_sales_price else null
> end) wed_sales,
> sum(case when (d_day_name='Thursday') then ss_sales_price else null
> end) thu_sales,
> sum(case when (d_day_name='Friday') then ss_sales_price else null
> end) fri_sales,
> sum(case when (d_day_name='Saturday') then ss_sales_price else null
> end) sat_sales
> from date_dim, store_sales, store
> where d_date_sk = ss_sold_date_sk and
>s_store_sk = ss_store_sk and
>s_gmt_offset = -6 and
>d_year = 1998
> group by s_store_name, s_store_id
> order by s_store_name,
> s_store_id,sun_sales,mon_sales,tue_sales,wed_sales,thu_sales,fri_sales,sat_sales
> limit 100;
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20906) Hive locks are not really ephemeral
[ https://issues.apache.org/jira/browse/HIVE-20906?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajesh KSV updated HIVE-20906: -- Description: I am using Hive 2.3.3. According to docs ([https://cwiki.apache.org/confluence/display/Hive/Locking]), they are ephemeral. But in the Zookeeper and Hive source Code, they seemed to be non-ephemeral. I ran the following query in Hive. *lock table discovery.defaultns* When I see the lock in zookeeper, it doesn't look like ephemeral node. [zk: localhost:2181(CONNECTED) 55] get /hive_zookeeper_namespace/discovery/defaultns/LOCK-EXCLUSIVE-00 hive_20181112191555_d6640955-2806-4fa7-9e52-30d04c377a04:1542030355565:EXPLICIT:lock table discovery.defaultns exclusive:10.33.222.139 cZxid = 0x2400296808 ctime = Mon Nov 12 19:15:55 IST 2018 mZxid = 0x2400296808 mtime = Mon Nov 12 19:15:55 IST 2018 pZxid = 0x2400296808 cversion = 0 dataVersion = 0 aclVersion = 0 *ephemeralOwner = 0x0* dataLength = 134 numChildren = 0 Also in the code - [https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/lockmgr/HiveTxnManagerImpl.java], the lock is KeepAlive/PERSISTENT and not EPHIMERAL. Current code is *HiveLock lck = lockMgr.lock(new HiveLockObject(tbl, lockData), mode, true);* Instead it should be *HiveLock lck = lockMgr.lock(new HiveLockObject(tbl, lockData), mode, false);* The patch is the same. Please correct me if I am wrong. This is my first ever Jira/Patch in Hive was: I am using Hive 2.3.3. According to docs ([https://cwiki.apache.org/confluence/display/Hive/Locking]), they are ephemeral. But in the Zookeeper and Hive source Code, they seemed to be non-ephemeral. I ran the following query in Hive. *lock table discovery.defaultns* When I see the lock in zookeeper, it doesn't look like ephemeral node. [zk: localhost:2181(CONNECTED) 55] get /hive_zookeeper_namespace/discovery/defaultns/LOCK-EXCLUSIVE-00 hive_20181112191555_d6640955-2806-4fa7-9e52-30d04c377a04:1542030355565:EXPLICIT:lock table discovery.defaultns exclusive:10.33.222.139 cZxid = 0x2400296808 ctime = Mon Nov 12 19:15:55 IST 2018 mZxid = 0x2400296808 mtime = Mon Nov 12 19:15:55 IST 2018 pZxid = 0x2400296808 cversion = 0 dataVersion = 0 aclVersion = 0 *ephemeralOwner = 0x0* dataLength = 134 numChildren = 0 Also in the code - [https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/lockmgr/HiveTxnManagerImpl.java], the lock is KeepAlive/PERSISTENT and not EPHIMERAL. Current code is *HiveLock lck = lockMgr.lock(new HiveLockObject(tbl, lockData), mode, true);* Instead it should be *HiveLock lck = lockMgr.lock(new HiveLockObject(tbl, lockData), mode, false);* The patch is the same. > Hive locks are not really ephemeral > --- > > Key: HIVE-20906 > URL: https://issues.apache.org/jira/browse/HIVE-20906 > Project: Hive > Issue Type: Bug >Affects Versions: 2.3.0, 2.3.3, 3.1.1 >Reporter: Rajesh KSV >Priority: Major > Fix For: 3.1.1 > > Attachments: ephemeral.patch > > > I am using Hive 2.3.3. > > According to docs > ([https://cwiki.apache.org/confluence/display/Hive/Locking]), they are > ephemeral. But in the Zookeeper and Hive source Code, they seemed to be > non-ephemeral. > > I ran the following query in Hive. > > *lock table discovery.defaultns* > > When I see the lock in zookeeper, it doesn't look like ephemeral node. > > [zk: localhost:2181(CONNECTED) 55] get > /hive_zookeeper_namespace/discovery/defaultns/LOCK-EXCLUSIVE-00 > > hive_20181112191555_d6640955-2806-4fa7-9e52-30d04c377a04:1542030355565:EXPLICIT:lock > table discovery.defaultns exclusive:10.33.222.139 > cZxid = 0x2400296808 > ctime = Mon Nov 12 19:15:55 IST 2018 > mZxid = 0x2400296808 > mtime = Mon Nov 12 19:15:55 IST 2018 > pZxid = 0x2400296808 > cversion = 0 > dataVersion = 0 > aclVersion = 0 > *ephemeralOwner = 0x0* > dataLength = 134 > numChildren = 0 > > Also in the code - > [https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/lockmgr/HiveTxnManagerImpl.java], > the lock is KeepAlive/PERSISTENT and not EPHIMERAL. Current code is > > *HiveLock lck = lockMgr.lock(new HiveLockObject(tbl, lockData), mode, > true);* > Instead it should be > *HiveLock lck = lockMgr.lock(new HiveLockObject(tbl, lockData), mode, > false);* > The patch is the same. > Please correct me if I am wrong. This is my first ever Jira/Patch in Hive -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20905) querying streaming table fails with out of memory exception
[
https://issues.apache.org/jira/browse/HIVE-20905?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16684180#comment-16684180
]
Hive QA commented on HIVE-20905:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12947852/HIVE-20905.02.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 15537 tests
executed
*Failed tests:*
{noformat}
TestMiniDruidCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=196)
[druidmini_masking.q,druidmini_test1.q,druidkafkamini_basic.q,druidmini_joins.q,druid_timestamptz.q]
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/14890/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/14890/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-14890/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12947852 - PreCommit-HIVE-Build
> querying streaming table fails with out of memory exception
> ---
>
> Key: HIVE-20905
> URL: https://issues.apache.org/jira/browse/HIVE-20905
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 4.0.0
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-20905.01.patch, HIVE-20905.02.patch
>
>
> Streaming app was ran for 24hrs post which it went down due authentication
> issue . The table was accessible for 12hrs into the run, however currently
> querying the table fails with OOM exception.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20850) Push case conditional from projections to dimension tables if possible
[
https://issues.apache.org/jira/browse/HIVE-20850?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16684177#comment-16684177
]
Jesus Camacho Rodriguez commented on HIVE-20850:
+1
> Push case conditional from projections to dimension tables if possible
> --
>
> Key: HIVE-20850
> URL: https://issues.apache.org/jira/browse/HIVE-20850
> Project: Hive
> Issue Type: Improvement
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Attachments: HIVE-20850.01.patch, HIVE-20850.01.patch,
> HIVE-20850.01.patch, HIVE-20850.02.patch, HIVE-20850.03.patch,
> HIVE-20850.04.patch, HIVE-20850.05.patch, HIVE-20850.06.patch,
> HIVE-20850.07.patch, HIVE-20850.07.patch
>
>
> noticed by [~gopalv]: If there is a project which could be only evaluated
> after the join; but the condition references only a single column from a
> small dimension table; hive will end up evaluating the same thing over and
> over again...
> {code}
> explain
> select s_store_name, s_store_id,
> sum(case when (d_day_name='Sunday') then ss_sales_price else null
> end) sun_sales,
> sum(case when (d_day_name='Monday') then ss_sales_price else null
> end) mon_sales,
> sum(case when (d_day_name='Tuesday') then ss_sales_price else null
> end) tue_sales,
> sum(case when (d_day_name='Wednesday') then ss_sales_price else null
> end) wed_sales,
> sum(case when (d_day_name='Thursday') then ss_sales_price else null
> end) thu_sales,
> sum(case when (d_day_name='Friday') then ss_sales_price else null
> end) fri_sales,
> sum(case when (d_day_name='Saturday') then ss_sales_price else null
> end) sat_sales
> from date_dim, store_sales, store
> where d_date_sk = ss_sold_date_sk and
>s_store_sk = ss_store_sk and
>s_gmt_offset = -6 and
>d_year = 1998
> group by s_store_name, s_store_id
> order by s_store_name,
> s_store_id,sun_sales,mon_sales,tue_sales,wed_sales,thu_sales,fri_sales,sat_sales
> limit 100;
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20906) Hive locks are not really ephemeral
[ https://issues.apache.org/jira/browse/HIVE-20906?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajesh KSV updated HIVE-20906: -- Description: I am using Hive 2.3.3. According to docs ([https://cwiki.apache.org/confluence/display/Hive/Locking]), they are ephemeral. But in the Zookeeper and Hive source Code, they seemed to be non-ephemeral. I ran the following query in Hive. *lock table discovery.defaultns* When I see the lock in zookeeper, it doesn't look like ephemeral node. [zk: localhost:2181(CONNECTED) 55] get /hive_zookeeper_namespace/discovery/defaultns/LOCK-EXCLUSIVE-00 hive_20181112191555_d6640955-2806-4fa7-9e52-30d04c377a04:1542030355565:EXPLICIT:lock table discovery.defaultns exclusive:10.33.222.139 cZxid = 0x2400296808 ctime = Mon Nov 12 19:15:55 IST 2018 mZxid = 0x2400296808 mtime = Mon Nov 12 19:15:55 IST 2018 pZxid = 0x2400296808 cversion = 0 dataVersion = 0 aclVersion = 0 *ephemeralOwner = 0x0* dataLength = 134 numChildren = 0 Also in the code - [https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/lockmgr/HiveTxnManagerImpl.java], the lock is KeepAlive/PERSISTENT and not EPHIMERAL. Current code is *HiveLock lck = lockMgr.lock(new HiveLockObject(tbl, lockData), mode, true);* Instead it should be *HiveLock lck = lockMgr.lock(new HiveLockObject(tbl, lockData), mode, false);* The patch is the same. was: I am using Hive 2.3.3. According to docs ([https://cwiki.apache.org/confluence/display/Hive/Locking]), they are ephemeral. But in the Zookeeper and Hive source Code, they seemed to be non-ephemeral. I ran the following query in Hive. *lock table discovery.defaultns* When I see the lock in zookeeper, it doesn't look like ephemeral node. [zk: localhost:2181(CONNECTED) 55] get /hive_zookeeper_namespace/discovery/defaultns/LOCK-EXCLUSIVE-00 hive_20181112191555_d6640955-2806-4fa7-9e52-30d04c377a04:1542030355565:EXPLICIT:lock table discovery.defaultns exclusive:10.33.222.139 cZxid = 0x2400296808 ctime = Mon Nov 12 19:15:55 IST 2018 mZxid = 0x2400296808 mtime = Mon Nov 12 19:15:55 IST 2018 pZxid = 0x2400296808 cversion = 0 dataVersion = 0 aclVersion = 0 *ephemeralOwner = 0x0* dataLength = 134 numChildren = 0 Also in the code - [https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/lockmgr/HiveTxnManagerImpl.java], the lock is KeepAlive/PERSISTENT and not EPHIMERAL . *HiveLock lck = lockMgr.lock(new HiveLockObject(tbl, lockData), mode, true);* Instead it should be *HiveLock lck = lockMgr.lock(new HiveLockObject(tbl, lockData), mode, false);* Is it a bug or am I missing something? Thanks! > Hive locks are not really ephemeral > --- > > Key: HIVE-20906 > URL: https://issues.apache.org/jira/browse/HIVE-20906 > Project: Hive > Issue Type: Bug >Affects Versions: 2.3.0, 2.3.3, 3.1.1 >Reporter: Rajesh KSV >Priority: Major > Fix For: 3.1.1 > > Attachments: ephemeral.patch > > > I am using Hive 2.3.3. > > According to docs > ([https://cwiki.apache.org/confluence/display/Hive/Locking]), they are > ephemeral. But in the Zookeeper and Hive source Code, they seemed to be > non-ephemeral. > > I ran the following query in Hive. > > *lock table discovery.defaultns* > > When I see the lock in zookeeper, it doesn't look like ephemeral node. > > [zk: localhost:2181(CONNECTED) 55] get > /hive_zookeeper_namespace/discovery/defaultns/LOCK-EXCLUSIVE-00 > > hive_20181112191555_d6640955-2806-4fa7-9e52-30d04c377a04:1542030355565:EXPLICIT:lock > table discovery.defaultns exclusive:10.33.222.139 > cZxid = 0x2400296808 > ctime = Mon Nov 12 19:15:55 IST 2018 > mZxid = 0x2400296808 > mtime = Mon Nov 12 19:15:55 IST 2018 > pZxid = 0x2400296808 > cversion = 0 > dataVersion = 0 > aclVersion = 0 > *ephemeralOwner = 0x0* > dataLength = 134 > numChildren = 0 > > Also in the code - > [https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/lockmgr/HiveTxnManagerImpl.java], > the lock is KeepAlive/PERSISTENT and not EPHIMERAL. Current code is > > *HiveLock lck = lockMgr.lock(new HiveLockObject(tbl, lockData), mode, > true);* > Instead it should be > *HiveLock lck = lockMgr.lock(new HiveLockObject(tbl, lockData), mode, > false);* > The patch is the same. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20906) Hive locks are not really ephemeral
[ https://issues.apache.org/jira/browse/HIVE-20906?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajesh KSV updated HIVE-20906: -- Summary: Hive locks are not really ephemeral (was: Are Hive locks really ephemeral ?) > Hive locks are not really ephemeral > --- > > Key: HIVE-20906 > URL: https://issues.apache.org/jira/browse/HIVE-20906 > Project: Hive > Issue Type: Bug >Affects Versions: 2.3.0, 2.3.3, 3.1.1 >Reporter: Rajesh KSV >Priority: Major > Fix For: 3.1.1 > > Attachments: ephemeral.patch > > > I am using Hive 2.3.3. > > According to docs > ([https://cwiki.apache.org/confluence/display/Hive/Locking]), they are > ephemeral. But in the Zookeeper and Hive source Code, they seemed to be > non-ephemeral. > > I ran the following query in Hive. > > *lock table discovery.defaultns* > > When I see the lock in zookeeper, it doesn't look like ephemeral node. > > [zk: localhost:2181(CONNECTED) 55] get > /hive_zookeeper_namespace/discovery/defaultns/LOCK-EXCLUSIVE-00 > > hive_20181112191555_d6640955-2806-4fa7-9e52-30d04c377a04:1542030355565:EXPLICIT:lock > table discovery.defaultns exclusive:10.33.222.139 > cZxid = 0x2400296808 > ctime = Mon Nov 12 19:15:55 IST 2018 > mZxid = 0x2400296808 > mtime = Mon Nov 12 19:15:55 IST 2018 > pZxid = 0x2400296808 > cversion = 0 > dataVersion = 0 > aclVersion = 0 > *ephemeralOwner = 0x0* > dataLength = 134 > numChildren = 0 > > Also in the code - > [https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/lockmgr/HiveTxnManagerImpl.java], > the lock is KeepAlive/PERSISTENT and not EPHIMERAL . > > *HiveLock lck = lockMgr.lock(new HiveLockObject(tbl, lockData), mode, > true);* > Instead it should be > *HiveLock lck = lockMgr.lock(new HiveLockObject(tbl, lockData), mode, > false);* > Is it a bug or am I missing something? Thanks! -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20906) Are Hive locks really ephemeral ?
[ https://issues.apache.org/jira/browse/HIVE-20906?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajesh KSV updated HIVE-20906: -- Affects Version/s: 2.3.3 > Are Hive locks really ephemeral ? > - > > Key: HIVE-20906 > URL: https://issues.apache.org/jira/browse/HIVE-20906 > Project: Hive > Issue Type: Bug >Affects Versions: 2.3.0, 2.3.3, 3.1.1 >Reporter: Rajesh KSV >Priority: Major > Fix For: 3.1.1 > > Attachments: ephemeral.patch > > > I am using Hive 2.3.3. > > According to docs > ([https://cwiki.apache.org/confluence/display/Hive/Locking]), they are > ephemeral. But in the Zookeeper and Hive source Code, they seemed to be > non-ephemeral. > > I ran the following query in Hive. > > *lock table discovery.defaultns* > > When I see the lock in zookeeper, it doesn't look like ephemeral node. > > [zk: localhost:2181(CONNECTED) 55] get > /hive_zookeeper_namespace/discovery/defaultns/LOCK-EXCLUSIVE-00 > > hive_20181112191555_d6640955-2806-4fa7-9e52-30d04c377a04:1542030355565:EXPLICIT:lock > table discovery.defaultns exclusive:10.33.222.139 > cZxid = 0x2400296808 > ctime = Mon Nov 12 19:15:55 IST 2018 > mZxid = 0x2400296808 > mtime = Mon Nov 12 19:15:55 IST 2018 > pZxid = 0x2400296808 > cversion = 0 > dataVersion = 0 > aclVersion = 0 > *ephemeralOwner = 0x0* > dataLength = 134 > numChildren = 0 > > Also in the code - > [https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/lockmgr/HiveTxnManagerImpl.java], > the lock is KeepAlive/PERSISTENT and not EPHIMERAL . > > *HiveLock lck = lockMgr.lock(new HiveLockObject(tbl, lockData), mode, > true);* > Instead it should be > *HiveLock lck = lockMgr.lock(new HiveLockObject(tbl, lockData), mode, > false);* > Is it a bug or am I missing something? Thanks! -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20906) Are Hive locks really ephemeral ?
[ https://issues.apache.org/jira/browse/HIVE-20906?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajesh KSV updated HIVE-20906: -- Fix Version/s: 3.1.1 Affects Version/s: 2.3.0 3.1.1 Attachment: ephemeral.patch Status: Patch Available (was: Open) > Are Hive locks really ephemeral ? > - > > Key: HIVE-20906 > URL: https://issues.apache.org/jira/browse/HIVE-20906 > Project: Hive > Issue Type: Bug >Affects Versions: 3.1.1, 2.3.0 >Reporter: Rajesh KSV >Priority: Major > Fix For: 3.1.1 > > Attachments: ephemeral.patch > > > I am using Hive 2.3.3. > > According to docs > ([https://cwiki.apache.org/confluence/display/Hive/Locking]), they are > ephemeral. But in the Zookeeper and Hive source Code, they seemed to be > non-ephemeral. > > I ran the following query in Hive. > > *lock table discovery.defaultns* > > When I see the lock in zookeeper, it doesn't look like ephemeral node. > > [zk: localhost:2181(CONNECTED) 55] get > /hive_zookeeper_namespace/discovery/defaultns/LOCK-EXCLUSIVE-00 > > hive_20181112191555_d6640955-2806-4fa7-9e52-30d04c377a04:1542030355565:EXPLICIT:lock > table discovery.defaultns exclusive:10.33.222.139 > cZxid = 0x2400296808 > ctime = Mon Nov 12 19:15:55 IST 2018 > mZxid = 0x2400296808 > mtime = Mon Nov 12 19:15:55 IST 2018 > pZxid = 0x2400296808 > cversion = 0 > dataVersion = 0 > aclVersion = 0 > *ephemeralOwner = 0x0* > dataLength = 134 > numChildren = 0 > > Also in the code - > [https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/lockmgr/HiveTxnManagerImpl.java], > the lock is KeepAlive/PERSISTENT and not EPHIMERAL . > > *HiveLock lck = lockMgr.lock(new HiveLockObject(tbl, lockData), mode, > true);* > Instead it should be > *HiveLock lck = lockMgr.lock(new HiveLockObject(tbl, lockData), mode, > false);* > Is it a bug or am I missing something? Thanks! -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20906) Are Hive locks really ephemeral ?
[ https://issues.apache.org/jira/browse/HIVE-20906?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajesh KSV updated HIVE-20906: -- Description: I am using Hive 2.3.3. According to docs ([https://cwiki.apache.org/confluence/display/Hive/Locking]), they are ephemeral. But in the Zookeeper and Hive source Code, they seemed to be non-ephemeral. I ran the following query in Hive. *lock table discovery.defaultns* When I see the lock in zookeeper, it doesn't look like ephemeral node. [zk: localhost:2181(CONNECTED) 55] get /hive_zookeeper_namespace/discovery/defaultns/LOCK-EXCLUSIVE-00 hive_20181112191555_d6640955-2806-4fa7-9e52-30d04c377a04:1542030355565:EXPLICIT:lock table discovery.defaultns exclusive:10.33.222.139 cZxid = 0x2400296808 ctime = Mon Nov 12 19:15:55 IST 2018 mZxid = 0x2400296808 mtime = Mon Nov 12 19:15:55 IST 2018 pZxid = 0x2400296808 cversion = 0 dataVersion = 0 aclVersion = 0 *ephemeralOwner = 0x0* dataLength = 134 numChildren = 0 Also in the code - [https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/lockmgr/HiveTxnManagerImpl.java], the lock is KeepAlive/PERSISTENT and not EPHIMERAL . *HiveLock lck = lockMgr.lock(new HiveLockObject(tbl, lockData), mode, true);* Instead it should be *HiveLock lck = lockMgr.lock(new HiveLockObject(tbl, lockData), mode, false);* Is it a bug or am I missing something? Thanks! was: I am using Hive 2.3.3. According to docs ([https://cwiki.apache.org/confluence/display/Hive/Locking]), they are ephemeral. But in the Zookeeper and Hive source Code, they seemed to be non-ephemeral. I ran the following query in Hive. *lock table discovery.defaultns* When I see the lock in zookeeper, it doesn't look like ephemeral node. [zk: localhost:2181(CONNECTED) 55] get /hive_zookeeper_namespace/discovery/defaultns/LOCK-EXCLUSIVE-00 hive_20181112191555_d6640955-2806-4fa7-9e52-30d04c377a04:1542030355565:EXPLICIT:lock table discovery.defaultns exclusive:10.33.222.139 cZxid = 0x2400296808 ctime = Mon Nov 12 19:15:55 IST 2018 mZxid = 0x2400296808 mtime = Mon Nov 12 19:15:55 IST 2018 pZxid = 0x2400296808 cversion = 0 dataVersion = 0 aclVersion = 0 *ephemeralOwner = 0x0* dataLength = 134 numChildren = 0 Also in the code - [https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/lockmgr/HiveTxnManagerImpl.java], the lock is KeepAlive/PERSISTENT and not EPHIMERAL . *HiveLock lck = lockMgr.lock(new HiveLockObject(tbl, lockData), mode, true);* Is it a bug or am I missing something? Thanks! > Are Hive locks really ephemeral ? > - > > Key: HIVE-20906 > URL: https://issues.apache.org/jira/browse/HIVE-20906 > Project: Hive > Issue Type: Bug >Reporter: Rajesh KSV >Priority: Major > > I am using Hive 2.3.3. > > According to docs > ([https://cwiki.apache.org/confluence/display/Hive/Locking]), they are > ephemeral. But in the Zookeeper and Hive source Code, they seemed to be > non-ephemeral. > > I ran the following query in Hive. > > *lock table discovery.defaultns* > > When I see the lock in zookeeper, it doesn't look like ephemeral node. > > [zk: localhost:2181(CONNECTED) 55] get > /hive_zookeeper_namespace/discovery/defaultns/LOCK-EXCLUSIVE-00 > > hive_20181112191555_d6640955-2806-4fa7-9e52-30d04c377a04:1542030355565:EXPLICIT:lock > table discovery.defaultns exclusive:10.33.222.139 > cZxid = 0x2400296808 > ctime = Mon Nov 12 19:15:55 IST 2018 > mZxid = 0x2400296808 > mtime = Mon Nov 12 19:15:55 IST 2018 > pZxid = 0x2400296808 > cversion = 0 > dataVersion = 0 > aclVersion = 0 > *ephemeralOwner = 0x0* > dataLength = 134 > numChildren = 0 > > Also in the code - > [https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/lockmgr/HiveTxnManagerImpl.java], > the lock is KeepAlive/PERSISTENT and not EPHIMERAL . > > *HiveLock lck = lockMgr.lock(new HiveLockObject(tbl, lockData), mode, > true);* > Instead it should be > *HiveLock lck = lockMgr.lock(new HiveLockObject(tbl, lockData), mode, > false);* > Is it a bug or am I missing something? Thanks! -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20905) querying streaming table fails with out of memory exception
[
https://issues.apache.org/jira/browse/HIVE-20905?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16684118#comment-16684118
]
Hive QA commented on HIVE-20905:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
41s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
0s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
35s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
44s{color} | {color:blue} ql in master has 2316 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
54s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
36s{color} | {color:red} ql: The patch generated 1 new + 8 unchanged - 0 fixed
= 9 total (was 8) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 22m 18s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-14890/dev-support/hive-personality.sh
|
| git revision | master / dca389b |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-14890/yetus/diff-checkstyle-ql.txt
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-14890/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> querying streaming table fails with out of memory exception
> ---
>
> Key: HIVE-20905
> URL: https://issues.apache.org/jira/browse/HIVE-20905
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 4.0.0
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-20905.01.patch, HIVE-20905.02.patch
>
>
> Streaming app was ran for 24hrs post which it went down due authentication
> issue . The table was accessible for 12hrs into the run, however currently
> querying the table fails with OOM exception.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (HIVE-20899) Keytab URI for LLAP YARN Service is restrictive to support HDFS only
[ https://issues.apache.org/jira/browse/HIVE-20899?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran reassigned HIVE-20899: Assignee: Prasanth Jayachandran (was: Gour Saha) > Keytab URI for LLAP YARN Service is restrictive to support HDFS only > > > Key: HIVE-20899 > URL: https://issues.apache.org/jira/browse/HIVE-20899 > Project: Hive > Issue Type: Bug > Components: llap >Affects Versions: 3.1.1 >Reporter: Gour Saha >Assignee: Prasanth Jayachandran >Priority: Major > Attachments: HIVE-20899.01.patch, HIVE-20899.02.patch, > HIVE-20899.03.patch, HIVE-20899.04.patch > > > llap-server/src/main/resources/package.py restricts the keytab URI to support > HDFS only and hence fails for other FileSystem API conforming FSs like s3a, > wasb, gs, etc. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20899) Keytab URI for LLAP YARN Service is restrictive to support HDFS only
[ https://issues.apache.org/jira/browse/HIVE-20899?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16684105#comment-16684105 ] Prasanth Jayachandran commented on HIVE-20899: -- Yes. The failures are unrelated. But we need a green run to commit as per the new rules. > Keytab URI for LLAP YARN Service is restrictive to support HDFS only > > > Key: HIVE-20899 > URL: https://issues.apache.org/jira/browse/HIVE-20899 > Project: Hive > Issue Type: Bug > Components: llap >Affects Versions: 3.1.1 >Reporter: Gour Saha >Assignee: Gour Saha >Priority: Major > Attachments: HIVE-20899.01.patch, HIVE-20899.02.patch, > HIVE-20899.03.patch, HIVE-20899.04.patch > > > llap-server/src/main/resources/package.py restricts the keytab URI to support > HDFS only and hence fails for other FileSystem API conforming FSs like s3a, > wasb, gs, etc. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20899) Keytab URI for LLAP YARN Service is restrictive to support HDFS only
[ https://issues.apache.org/jira/browse/HIVE-20899?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-20899: - Attachment: HIVE-20899.04.patch > Keytab URI for LLAP YARN Service is restrictive to support HDFS only > > > Key: HIVE-20899 > URL: https://issues.apache.org/jira/browse/HIVE-20899 > Project: Hive > Issue Type: Bug > Components: llap >Affects Versions: 3.1.1 >Reporter: Gour Saha >Assignee: Prasanth Jayachandran >Priority: Major > Attachments: HIVE-20899.01.patch, HIVE-20899.02.patch, > HIVE-20899.03.patch, HIVE-20899.04.patch > > > llap-server/src/main/resources/package.py restricts the keytab URI to support > HDFS only and hence fails for other FileSystem API conforming FSs like s3a, > wasb, gs, etc. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-20899) Keytab URI for LLAP YARN Service is restrictive to support HDFS only
[ https://issues.apache.org/jira/browse/HIVE-20899?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran reassigned HIVE-20899: Assignee: Gour Saha (was: Prasanth Jayachandran) > Keytab URI for LLAP YARN Service is restrictive to support HDFS only > > > Key: HIVE-20899 > URL: https://issues.apache.org/jira/browse/HIVE-20899 > Project: Hive > Issue Type: Bug > Components: llap >Affects Versions: 3.1.1 >Reporter: Gour Saha >Assignee: Gour Saha >Priority: Major > Attachments: HIVE-20899.01.patch, HIVE-20899.02.patch, > HIVE-20899.03.patch, HIVE-20899.04.patch > > > llap-server/src/main/resources/package.py restricts the keytab URI to support > HDFS only and hence fails for other FileSystem API conforming FSs like s3a, > wasb, gs, etc. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20905) querying streaming table fails with out of memory exception
[ https://issues.apache.org/jira/browse/HIVE-20905?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] mahesh kumar behera updated HIVE-20905: --- Status: Patch Available (was: Open) > querying streaming table fails with out of memory exception > --- > > Key: HIVE-20905 > URL: https://issues.apache.org/jira/browse/HIVE-20905 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 4.0.0 >Reporter: mahesh kumar behera >Assignee: mahesh kumar behera >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-20905.01.patch, HIVE-20905.02.patch > > > Streaming app was ran for 24hrs post which it went down due authentication > issue . The table was accessible for 12hrs into the run, however currently > querying the table fails with OOM exception. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20907) TestGetPartitionsUsingProjectionAndFilterSpecs is flaky
[ https://issues.apache.org/jira/browse/HIVE-20907?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Antal Sinkovits updated HIVE-20907: --- Attachment: HIVE-20907.02.patch > TestGetPartitionsUsingProjectionAndFilterSpecs is flaky > --- > > Key: HIVE-20907 > URL: https://issues.apache.org/jira/browse/HIVE-20907 > Project: Hive > Issue Type: Bug >Affects Versions: 4.0.0 >Reporter: Antal Sinkovits >Assignee: Antal Sinkovits >Priority: Major > Attachments: HIVE-20907.01.patch, HIVE-20907.02.patch > > > private void verifyLocations(List origPartitions, > StorageDescriptor sharedSD, > List partitionWithoutSDS) > method expects, that the order of the two list are the same. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20907) TestGetPartitionsUsingProjectionAndFilterSpecs is flaky
[ https://issues.apache.org/jira/browse/HIVE-20907?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16684061#comment-16684061 ] Antal Sinkovits commented on HIVE-20907: rebase > TestGetPartitionsUsingProjectionAndFilterSpecs is flaky > --- > > Key: HIVE-20907 > URL: https://issues.apache.org/jira/browse/HIVE-20907 > Project: Hive > Issue Type: Bug >Affects Versions: 4.0.0 >Reporter: Antal Sinkovits >Assignee: Antal Sinkovits >Priority: Major > Attachments: HIVE-20907.01.patch, HIVE-20907.02.patch > > > private void verifyLocations(List origPartitions, > StorageDescriptor sharedSD, > List partitionWithoutSDS) > method expects, that the order of the two list are the same. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20905) querying streaming table fails with out of memory exception
[ https://issues.apache.org/jira/browse/HIVE-20905?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] mahesh kumar behera updated HIVE-20905: --- Attachment: HIVE-20905.02.patch > querying streaming table fails with out of memory exception > --- > > Key: HIVE-20905 > URL: https://issues.apache.org/jira/browse/HIVE-20905 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 4.0.0 >Reporter: mahesh kumar behera >Assignee: mahesh kumar behera >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-20905.01.patch, HIVE-20905.02.patch > > > Streaming app was ran for 24hrs post which it went down due authentication > issue . The table was accessible for 12hrs into the run, however currently > querying the table fails with OOM exception. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
