[jira] [Commented] (HIVE-20897) TestJdbcDriver2#testSelectExecAsync2 fails with result set not present error
[
https://issues.apache.org/jira/browse/HIVE-20897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16682265#comment-16682265
]
Hive QA commented on HIVE-20897:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12947566/HIVE-20897.01.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 6 failed/errored test(s), 15537 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.ql.lockmgr.TestDbTxnManager2.testWriteSetTracking3
(batchId=322)
org.apache.hive.jdbc.TestJdbcWithMiniHS2.testJoinThriftSerializeInTasks
(batchId=261)
org.apache.hive.jdbc.TestJdbcWithMiniHS2.testParallelCompilation (batchId=261)
org.apache.hive.jdbc.TestJdbcWithMiniHS2.testParallelCompilation2 (batchId=261)
org.apache.hive.jdbc.TestJdbcWithMiniHS2.testParallelCompilation3 (batchId=261)
org.apache.hive.jdbc.TestJdbcWithMiniHS2.testParallelCompilation4 (batchId=261)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/14852/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/14852/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-14852/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 6 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12947566 - PreCommit-HIVE-Build
> TestJdbcDriver2#testSelectExecAsync2 fails with result set not present error
>
>
> Key: HIVE-20897
> URL: https://issues.apache.org/jira/browse/HIVE-20897
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
> Attachments: HIVE-20897.01.patch
>
>
> if async prepare is enabled, control will be returned to the client before
> driver could set of the query has a result set or not. But in current code,
> while generating the response for the query, it is not checked if the result
> set field is set or not.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20903) Cleanup code inspection issue on the druid adapter.
[ https://issues.apache.org/jira/browse/HIVE-20903?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] slim bouguerra updated HIVE-20903: -- Component/s: Druid integration > Cleanup code inspection issue on the druid adapter. > --- > > Key: HIVE-20903 > URL: https://issues.apache.org/jira/browse/HIVE-20903 > Project: Hive > Issue Type: Improvement > Components: Druid integration >Reporter: slim bouguerra >Assignee: slim bouguerra >Priority: Major > > This is a simple cleanup of the code and minor refactor. > I did not change any of the behavior. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20903) Cleanup code inspection issue on the druid adapter.
[ https://issues.apache.org/jira/browse/HIVE-20903?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] slim bouguerra updated HIVE-20903: -- Status: Patch Available (was: Open) > Cleanup code inspection issue on the druid adapter. > --- > > Key: HIVE-20903 > URL: https://issues.apache.org/jira/browse/HIVE-20903 > Project: Hive > Issue Type: Improvement > Components: Druid integration >Reporter: slim bouguerra >Assignee: slim bouguerra >Priority: Major > Attachments: HIVE-20903.patch > > > This is a simple cleanup of the code and minor refactor. > I did not change any of the behavior. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20903) Cleanup code inspection issue on the druid adapter.
[ https://issues.apache.org/jira/browse/HIVE-20903?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] slim bouguerra updated HIVE-20903: -- Attachment: HIVE-20903.patch > Cleanup code inspection issue on the druid adapter. > --- > > Key: HIVE-20903 > URL: https://issues.apache.org/jira/browse/HIVE-20903 > Project: Hive > Issue Type: Improvement > Components: Druid integration >Reporter: slim bouguerra >Assignee: slim bouguerra >Priority: Major > Attachments: HIVE-20903.patch > > > This is a simple cleanup of the code and minor refactor. > I did not change any of the behavior. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-20903) Cleanup code inspection issue on the druid adapter.
[ https://issues.apache.org/jira/browse/HIVE-20903?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] slim bouguerra reassigned HIVE-20903: - > Cleanup code inspection issue on the druid adapter. > --- > > Key: HIVE-20903 > URL: https://issues.apache.org/jira/browse/HIVE-20903 > Project: Hive > Issue Type: Improvement >Reporter: slim bouguerra >Assignee: slim bouguerra >Priority: Major > > This is a simple cleanup of the code and minor refactor. > I did not change any of the behavior. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20897) TestJdbcDriver2#testSelectExecAsync2 fails with result set not present error
[
https://issues.apache.org/jira/browse/HIVE-20897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16682247#comment-16682247
]
Hive QA commented on HIVE-20897:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
39s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
21s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
11s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
36s{color} | {color:blue} service in master has 48 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
16s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
12s{color} | {color:green} service: The patch generated 0 new + 60 unchanged -
1 fixed = 60 total (was 61) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
16s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
12s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 11m 28s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-14852/dev-support/hive-personality.sh
|
| git revision | master / 7ae4a2c |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: service U: service |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-14852/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> TestJdbcDriver2#testSelectExecAsync2 fails with result set not present error
>
>
> Key: HIVE-20897
> URL: https://issues.apache.org/jira/browse/HIVE-20897
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
> Attachments: HIVE-20897.01.patch
>
>
> if async prepare is enabled, control will be returned to the client before
> driver could set of the query has a result set or not. But in current code,
> while generating the response for the query, it is not checked if the result
> set field is set or not.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20850) Push case conditional from projections to dimension tables if possible
[
https://issues.apache.org/jira/browse/HIVE-20850?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16682240#comment-16682240
]
Hive QA commented on HIVE-20850:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12947547/HIVE-20850.06.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/14851/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/14851/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-14851/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Tests exited with: Exception: Patch URL
https://issues.apache.org/jira/secure/attachment/12947547/HIVE-20850.06.patch
was found in seen patch url's cache and a test was probably run already on it.
Aborting...
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12947547 - PreCommit-HIVE-Build
> Push case conditional from projections to dimension tables if possible
> --
>
> Key: HIVE-20850
> URL: https://issues.apache.org/jira/browse/HIVE-20850
> Project: Hive
> Issue Type: Improvement
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Attachments: HIVE-20850.01.patch, HIVE-20850.01.patch,
> HIVE-20850.01.patch, HIVE-20850.02.patch, HIVE-20850.03.patch,
> HIVE-20850.04.patch, HIVE-20850.05.patch, HIVE-20850.06.patch
>
>
> noticed by [~gopalv]: If there is a project which could be only evaluated
> after the join; but the condition references only a single column from a
> small dimension table; hive will end up evaluating the same thing over and
> over again...
> {code}
> explain
> select s_store_name, s_store_id,
> sum(case when (d_day_name='Sunday') then ss_sales_price else null
> end) sun_sales,
> sum(case when (d_day_name='Monday') then ss_sales_price else null
> end) mon_sales,
> sum(case when (d_day_name='Tuesday') then ss_sales_price else null
> end) tue_sales,
> sum(case when (d_day_name='Wednesday') then ss_sales_price else null
> end) wed_sales,
> sum(case when (d_day_name='Thursday') then ss_sales_price else null
> end) thu_sales,
> sum(case when (d_day_name='Friday') then ss_sales_price else null
> end) fri_sales,
> sum(case when (d_day_name='Saturday') then ss_sales_price else null
> end) sat_sales
> from date_dim, store_sales, store
> where d_date_sk = ss_sold_date_sk and
>s_store_sk = ss_store_sk and
>s_gmt_offset = -6 and
>d_year = 1998
> group by s_store_name, s_store_id
> order by s_store_name,
> s_store_id,sun_sales,mon_sales,tue_sales,wed_sales,thu_sales,fri_sales,sat_sales
> limit 100;
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20896) CachedStore fail to cache stats in multiple code paths
[
https://issues.apache.org/jira/browse/HIVE-20896?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16682239#comment-16682239
]
Hive QA commented on HIVE-20896:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12947655/HIVE-20896.1.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 15535 tests
executed
*Failed tests:*
{noformat}
TestMiniDruidCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=196)
[druidmini_masking.q,druidmini_test1.q,druidkafkamini_basic.q,druidmini_joins.q,druid_timestamptz.q]
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/14850/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/14850/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-14850/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12947655 - PreCommit-HIVE-Build
> CachedStore fail to cache stats in multiple code paths
> --
>
> Key: HIVE-20896
> URL: https://issues.apache.org/jira/browse/HIVE-20896
> Project: Hive
> Issue Type: Bug
> Components: Standalone Metastore
>Reporter: Daniel Dai
>Assignee: Daniel Dai
>Priority: Major
> Attachments: HIVE-20896.0.patch, HIVE-20896.1.patch
>
>
> Bunch of issues discovered in CachedStore to keep up column statistics:
> 1. The criteria for partition/non-partition is wrong
> (table.isSetPartitionKeys() is always true)
> 2. In update(), partition column stats are removed when populate table basic
> stats
> 3. Dirty flags are true right after prewarm(), so the first update() does not
> do anything
> 4. Could invoke cacheLock without holding the lock, which results a freeze in
> update()
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20896) CachedStore fail to cache stats in multiple code paths
[
https://issues.apache.org/jira/browse/HIVE-20896?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16682218#comment-16682218
]
Hive QA commented on HIVE-20896:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
38s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
24s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
6s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 1m
1s{color} | {color:blue} standalone-metastore/metastore-server in master has
185 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
18s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
8s{color} | {color:red} standalone-metastore/metastore-server generated 1 new +
185 unchanged - 0 fixed = 186 total (was 185) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
18s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 12m 28s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:standalone-metastore/metastore-server |
| | Dead store to partitionColStats in
org.apache.hadoop.hive.metastore.cache.CachedStore$CacheUpdateMasterWork.updateTablePartitionColStats(RawStore,
String, String, String) At
CachedStore.java:org.apache.hadoop.hive.metastore.cache.CachedStore$CacheUpdateMasterWork.updateTablePartitionColStats(RawStore,
String, String, String) At CachedStore.java:[line 602] |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-14850/dev-support/hive-personality.sh
|
| git revision | master / 7ae4a2c |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-14850/yetus/new-findbugs-standalone-metastore_metastore-server.html
|
| modules | C: standalone-metastore/metastore-server U:
standalone-metastore/metastore-server |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-14850/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> CachedStore fail to cache stats in multiple code paths
> --
>
> Key: HIVE-20896
> URL: https://issues.apache.org/jira/browse/HIVE-20896
> Project: Hive
> Issue Type: Bug
> Components: Standalone Metastore
>Reporter: Daniel Dai
>Assignee: Daniel Dai
>Priority: Major
> Attachments: HIVE-20896.0.patch, HIVE-20896.1.patch
>
>
> Bunch of issues discovered in CachedStore to keep up column statistics:
> 1. The criteria for partition/non-partition is wrong
> (table.isSetPartitionKeys() is always true)
> 2. In update(), partition column stats are removed when populate table basic
> stats
> 3. Dirty flags are true right after prewarm(), so the first update() does not
> do anything
> 4. Could invoke cacheLock without holding the lock, which results a freeze in
> update()
--
This
[jira] [Commented] (HIVE-20845) Fix TestJdbcWithDBTokenStoreNoDoAs flakiness
[
https://issues.apache.org/jira/browse/HIVE-20845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16682211#comment-16682211
]
Hive QA commented on HIVE-20845:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12947526/HIVE-20845.2.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 14 failed/errored test(s), 15529 tests
executed
*Failed tests:*
{noformat}
TestSSL - did not produce a TEST-*.xml file (likely timed out) (batchId=257)
org.apache.hive.beeline.hs2connection.TestBeelineConnectionUsingHiveSite.testBeelineConnectionSSL
(batchId=253)
org.apache.hive.beeline.hs2connection.TestBeelineWithUserHs2ConnectionFile.testBeelineConnectionSSL
(batchId=253)
org.apache.hive.jdbc.TestActivePassiveHA.testActivePassiveHA (batchId=258)
org.apache.hive.jdbc.TestActivePassiveHA.testClientConnectionsOnFailover
(batchId=258)
org.apache.hive.jdbc.TestActivePassiveHA.testConnectionActivePassiveHAServiceDiscovery
(batchId=258)
org.apache.hive.jdbc.TestActivePassiveHA.testManualFailover (batchId=258)
org.apache.hive.jdbc.TestActivePassiveHA.testManualFailoverUnauthorized
(batchId=258)
org.apache.hive.jdbc.TestActivePassiveHA.testNoConnectionOnPassive (batchId=258)
org.apache.hive.jdbc.TestNoSaslAuth.org.apache.hive.jdbc.TestNoSaslAuth
(batchId=261)
org.apache.hive.jdbc.TestXSRFFilter.testFilterEnabledNoInjection (batchId=256)
org.apache.hive.jdbc.miniHS2.TestHs2ConnectionMetricsBinary.testOpenConnectionMetrics
(batchId=262)
org.apache.hive.jdbc.miniHS2.TestHs2ConnectionMetricsHttp.testOpenConnectionMetrics
(batchId=262)
org.apache.hive.minikdc.TestSSLWithMiniKdc.org.apache.hive.minikdc.TestSSLWithMiniKdc
(batchId=273)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/14849/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/14849/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-14849/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 14 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12947526 - PreCommit-HIVE-Build
> Fix TestJdbcWithDBTokenStoreNoDoAs flakiness
>
>
> Key: HIVE-20845
> URL: https://issues.apache.org/jira/browse/HIVE-20845
> Project: Hive
> Issue Type: Test
>Reporter: Peter Vary
>Assignee: Peter Vary
>Priority: Major
> Attachments: HIVE-20845.2.patch, HIVE-20845.patch
>
>
> Previously did a dirty fix for TestJdbcWithDBTokenStoreNoDoAs and
> TestJdbcWithDBTokenStore
> Found out the issue is that we do not wait enough for HS2 to come up.
> Need to fix in MiniHS2.waitForStartup()
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20845) Fix TestJdbcWithDBTokenStoreNoDoAs flakiness
[
https://issues.apache.org/jira/browse/HIVE-20845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16682198#comment-16682198
]
Hive QA commented on HIVE-20845:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
33s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
27s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
53s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
24s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
43s{color} | {color:blue} itests/util in master has 48 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
35s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
23s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
51s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
12s{color} | {color:red} itests/util: The patch generated 2 new + 16 unchanged
- 1 fixed = 18 total (was 17) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
35s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
12s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 15m 7s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-14849/dev-support/hive-personality.sh
|
| git revision | master / 7ae4a2c |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-14849/yetus/diff-checkstyle-itests_util.txt
|
| modules | C: itests/util itests/hive-minikdc U: itests |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-14849/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Fix TestJdbcWithDBTokenStoreNoDoAs flakiness
>
>
> Key: HIVE-20845
> URL: https://issues.apache.org/jira/browse/HIVE-20845
> Project: Hive
> Issue Type: Test
>Reporter: Peter Vary
>Assignee: Peter Vary
>Priority: Major
> Attachments: HIVE-20845.2.patch, HIVE-20845.patch
>
>
> Previously did a dirty fix for TestJdbcWithDBTokenStoreNoDoAs and
> TestJdbcWithDBTokenStore
> Found out the issue is that we do not wait enough for HS2 to come up.
> Need to fix in MiniHS2.waitForStartup()
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20880) Update default value for hive.stats.filter.in.min.ratio
[
https://issues.apache.org/jira/browse/HIVE-20880?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16682191#comment-16682191
]
Hive QA commented on HIVE-20880:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12947523/HIVE-20880.2.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 9 failed/errored test(s), 15521 tests
executed
*Failed tests:*
{noformat}
TestAlterTableMetadata - did not produce a TEST-*.xml file (likely timed out)
(batchId=250)
TestAutoPurgeTables - did not produce a TEST-*.xml file (likely timed out)
(batchId=250)
TestLocationQueries - did not produce a TEST-*.xml file (likely timed out)
(batchId=250)
TestReplAcidTablesWithJsonMessage - did not produce a TEST-*.xml file (likely
timed out) (batchId=250)
TestSemanticAnalyzerHookLoading - did not produce a TEST-*.xml file (likely
timed out) (batchId=250)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[acid_no_buckets]
(batchId=176)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vectorization_0]
(batchId=181)
org.apache.hadoop.hive.cli.TestSparkPerfCliDriver.testCliDriver[query34]
(batchId=274)
org.apache.hadoop.hive.cli.TestSparkPerfCliDriver.testCliDriver[query73]
(batchId=274)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/14848/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/14848/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-14848/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 9 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12947523 - PreCommit-HIVE-Build
> Update default value for hive.stats.filter.in.min.ratio
> ---
>
> Key: HIVE-20880
> URL: https://issues.apache.org/jira/browse/HIVE-20880
> Project: Hive
> Issue Type: Improvement
> Components: Configuration
>Reporter: Ashutosh Chauhan
>Assignee: Ashutosh Chauhan
>Priority: Major
> Attachments: HIVE-20880.2.patch, HIVE-20880.patch
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20899) Keytab URI for LLAP YARN Service is restrictive to support HDFS only
[ https://issues.apache.org/jira/browse/HIVE-20899?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16682179#comment-16682179 ] Gour Saha commented on HIVE-20899: -- /cc [~thejas]. Post review it should be committed to branch-3 and master. > Keytab URI for LLAP YARN Service is restrictive to support HDFS only > > > Key: HIVE-20899 > URL: https://issues.apache.org/jira/browse/HIVE-20899 > Project: Hive > Issue Type: Bug > Components: llap >Affects Versions: 3.1.1 >Reporter: Gour Saha >Assignee: Gour Saha >Priority: Major > Attachments: HIVE-20899.01.patch > > > llap-server/src/main/resources/package.py restricts the keytab URI to support > HDFS only and hence fails for other FileSystem API conforming FSs like s3a, > wasb, gs, etc. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20899) Keytab URI for LLAP YARN Service is restrictive to support HDFS only
[ https://issues.apache.org/jira/browse/HIVE-20899?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16682174#comment-16682174 ] Gour Saha commented on HIVE-20899: -- Tested this patch in a hadoop cluster based on branch-3.1 > Keytab URI for LLAP YARN Service is restrictive to support HDFS only > > > Key: HIVE-20899 > URL: https://issues.apache.org/jira/browse/HIVE-20899 > Project: Hive > Issue Type: Bug > Components: llap >Affects Versions: 3.1.1 >Reporter: Gour Saha >Assignee: Gour Saha >Priority: Major > Attachments: HIVE-20899.01.patch > > > llap-server/src/main/resources/package.py restricts the keytab URI to support > HDFS only and hence fails for other FileSystem API conforming FSs like s3a, > wasb, gs, etc. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20880) Update default value for hive.stats.filter.in.min.ratio
[
https://issues.apache.org/jira/browse/HIVE-20880?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16682173#comment-16682173
]
Hive QA commented on HIVE-20880:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
30s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
6s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
15s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
48s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
30s{color} | {color:blue} common in master has 65 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
37s{color} | {color:blue} ql in master has 2315 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
6s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
10s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
5s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 25m 8s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-14848/dev-support/hive-personality.sh
|
| git revision | master / 7ae4a2c |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: common ql U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-14848/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Update default value for hive.stats.filter.in.min.ratio
> ---
>
> Key: HIVE-20880
> URL: https://issues.apache.org/jira/browse/HIVE-20880
> Project: Hive
> Issue Type: Improvement
> Components: Configuration
>Reporter: Ashutosh Chauhan
>Assignee: Ashutosh Chauhan
>Priority: Major
> Attachments: HIVE-20880.2.patch, HIVE-20880.patch
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20899) Keytab URI for LLAP YARN Service is restrictive to support HDFS only
[ https://issues.apache.org/jira/browse/HIVE-20899?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gour Saha updated HIVE-20899: - Status: Patch Available (was: In Progress) > Keytab URI for LLAP YARN Service is restrictive to support HDFS only > > > Key: HIVE-20899 > URL: https://issues.apache.org/jira/browse/HIVE-20899 > Project: Hive > Issue Type: Bug > Components: llap >Affects Versions: 3.1.1 >Reporter: Gour Saha >Assignee: Gour Saha >Priority: Major > Attachments: HIVE-20899.01.patch > > > llap-server/src/main/resources/package.py restricts the keytab URI to support > HDFS only and hence fails for other FileSystem API conforming FSs like s3a, > wasb, gs, etc. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20899) Keytab URI for LLAP YARN Service is restrictive to support HDFS only
[ https://issues.apache.org/jira/browse/HIVE-20899?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gour Saha updated HIVE-20899: - Attachment: HIVE-20899.01.patch > Keytab URI for LLAP YARN Service is restrictive to support HDFS only > > > Key: HIVE-20899 > URL: https://issues.apache.org/jira/browse/HIVE-20899 > Project: Hive > Issue Type: Bug > Components: llap >Affects Versions: 3.1.1 >Reporter: Gour Saha >Assignee: Gour Saha >Priority: Major > Attachments: HIVE-20899.01.patch > > > llap-server/src/main/resources/package.py restricts the keytab URI to support > HDFS only and hence fails for other FileSystem API conforming FSs like s3a, > wasb, gs, etc. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20740) Remove global lock in ObjectStore.setConf method
[
https://issues.apache.org/jira/browse/HIVE-20740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16682159#comment-16682159
]
Hive QA commented on HIVE-20740:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12947502/HIVE-20740.11.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/14847/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/14847/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-14847/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2018-11-10 02:42:25.172
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-14847/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2018-11-10 02:42:25.175
+ cd apache-github-source-source
+ git fetch origin
+ git reset --hard HEAD
HEAD is now at 7ae4a2c HIVE-20833: package.jdo needs to be updated to conform
with HIVE-20221 changes (Vineet Garg, reviewed by Jason Dere)
+ git clean -f -d
Removing standalone-metastore/metastore-server/src/gen/
+ git checkout master
Already on 'master'
Your branch is up-to-date with 'origin/master'.
+ git reset --hard origin/master
HEAD is now at 7ae4a2c HIVE-20833: package.jdo needs to be updated to conform
with HIVE-20221 changes (Vineet Garg, reviewed by Jason Dere)
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2018-11-10 02:42:25.935
+ rm -rf ../yetus_PreCommit-HIVE-Build-14847
+ mkdir ../yetus_PreCommit-HIVE-Build-14847
+ git gc
+ cp -R . ../yetus_PreCommit-HIVE-Build-14847
+ mkdir /data/hiveptest/logs/PreCommit-HIVE-Build-14847/yetus
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh
/data/hiveptest/working/scratch/build.patch
error:
a/itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/parse/TestReplicationScenarios.java:
does not exist in index
error:
a/itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestJdbcWithMiniHS2.java:
does not exist in index
error: a/ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java: does
not exist in index
error:
a/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java:
does not exist in index
error:
a/standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/TestObjectStore.java:
does not exist in index
error: patch failed:
standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/TestObjectStore.java:86
Falling back to three-way merge...
Applied patch to
'standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/TestObjectStore.java'
cleanly.
Going to apply patch with: git apply -p1
error: patch failed:
standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/TestObjectStore.java:86
Falling back to three-way merge...
Applied patch to
'standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/TestObjectStore.java'
cleanly.
+ [[ maven == \m\a\v\e\n ]]
+ rm -rf /data/hiveptest/working/maven/org/apache/hive
+ mvn -B clean install -DskipTests -T 4 -q
-Dmaven.repo.local=/data/hiveptest/working/maven
protoc-jar: executing: [/tmp/protoc7010054971666294437.exe, --version]
protoc-jar: executing: [/tmp/protoc7010054971666294437.exe,
-I/data/hiveptest/working/apache-github-source-source/standalone-metastore/metastore-common/src/main/protobuf/org/apache/hadoop/hive/metastore,
--java_out=/data/hiveptest/working/apache-github-source-source/standalone-metastore/metastore-common/target/generated-sources,
[jira] [Commented] (HIVE-18902) Lower Logging Level for Cleaning Up "local RawStore"
[
https://issues.apache.org/jira/browse/HIVE-18902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16682157#comment-16682157
]
Hive QA commented on HIVE-18902:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12947490/HIVE-18902.5.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 15537 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/14846/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/14846/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-14846/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12947490 - PreCommit-HIVE-Build
> Lower Logging Level for Cleaning Up "local RawStore"
>
>
> Key: HIVE-18902
> URL: https://issues.apache.org/jira/browse/HIVE-18902
> Project: Hive
> Issue Type: Improvement
> Components: Standalone Metastore
>Affects Versions: 3.0.0, 2.4.0
>Reporter: BELUGA BEHR
>Assignee: Bohdan Chupika
>Priority: Trivial
> Labels: noob
> Attachments: HIVE-18902.01.patch, HIVE-18902.02.patch,
> HIVE-18902.03.patch, HIVE-18902.04.patch, HIVE-18902.5.patch
>
>
> [https://github.com/apache/hive/blob/7c22d74c8d0eb0650adf6e84e0536127c103e46c/standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java#L7756-L7768]
>
> {code:java}
> private static void cleanupRawStore() {
> try {
> RawStore rs = HMSHandler.getRawStore();
> if (rs != null) {
> HMSHandler.logInfo("Cleaning up thread local RawStore...");
> rs.shutdown();
> }
> } finally {
> HMSHandler handler = HMSHandler.threadLocalHMSHandler.get();
> if (handler != null) {
> handler.notifyMetaListenersOnShutDown();
> }
> HMSHandler.threadLocalHMSHandler.remove();
> HMSHandler.threadLocalConf.remove();
> HMSHandler.threadLocalModifiedConfig.remove();
> HMSHandler.removeRawStore();
> HMSHandler.logInfo("Done cleaning up thread local RawStore");
> }
> }
> {code}
> {code}
> 2018-03-03 17:21:49,832 INFO
> org.apache.hadoop.hive.metastore.HiveMetaStore: [pool-4-thread-21]: 19:
> Cleaning up thread local RawStore...
> 2018-03-03 17:21:49,834 INFO
> org.apache.hadoop.hive.metastore.HiveMetaStore: [pool-4-thread-21]: 19: Done
> cleaning up thread local RawStore
> {code}
> Not very helpful logging. Please change logging levels to _debug_ or even
> _trace_
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18902) Lower Logging Level for Cleaning Up "local RawStore"
[
https://issues.apache.org/jira/browse/HIVE-18902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16682140#comment-16682140
]
Hive QA commented on HIVE-18902:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
41s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
23s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
7s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 1m
3s{color} | {color:blue} standalone-metastore/metastore-server in master has
185 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
17s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
17s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
12s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 12m 34s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-14846/dev-support/hive-personality.sh
|
| git revision | master / 7ae4a2c |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: standalone-metastore/metastore-server U:
standalone-metastore/metastore-server |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-14846/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Lower Logging Level for Cleaning Up "local RawStore"
>
>
> Key: HIVE-18902
> URL: https://issues.apache.org/jira/browse/HIVE-18902
> Project: Hive
> Issue Type: Improvement
> Components: Standalone Metastore
>Affects Versions: 3.0.0, 2.4.0
>Reporter: BELUGA BEHR
>Assignee: Bohdan Chupika
>Priority: Trivial
> Labels: noob
> Attachments: HIVE-18902.01.patch, HIVE-18902.02.patch,
> HIVE-18902.03.patch, HIVE-18902.04.patch, HIVE-18902.5.patch
>
>
> [https://github.com/apache/hive/blob/7c22d74c8d0eb0650adf6e84e0536127c103e46c/standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java#L7756-L7768]
>
> {code:java}
> private static void cleanupRawStore() {
> try {
> RawStore rs = HMSHandler.getRawStore();
> if (rs != null) {
> HMSHandler.logInfo("Cleaning up thread local RawStore...");
> rs.shutdown();
> }
> } finally {
> HMSHandler handler = HMSHandler.threadLocalHMSHandler.get();
> if (handler != null) {
> handler.notifyMetaListenersOnShutDown();
> }
> HMSHandler.threadLocalHMSHandler.remove();
> HMSHandler.threadLocalConf.remove();
> HMSHandler.threadLocalModifiedConfig.remove();
> HMSHandler.removeRawStore();
> HMSHandler.logInfo("Done cleaning up thread local RawStore");
> }
> }
> {code}
> {code}
> 2018-03-03 17:21:49,832 INFO
> org.apache.hadoop.hive.metastore.HiveMetaStore: [pool-4-thread-21]: 19:
> Cleaning up thread
[jira] [Commented] (HIVE-20847) Review of NullScan Code
[
https://issues.apache.org/jira/browse/HIVE-20847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16682135#comment-16682135
]
Hive QA commented on HIVE-20847:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12947486/HIVE-20847.1.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 15532 tests
executed
*Failed tests:*
{noformat}
TestMiniDruidCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=195)
[druidmini_dynamic_partition.q,druidmini_test_ts.q,druidmini_expressions.q,druidmini_test_alter.q,druidmini_test_insert.q]
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/14845/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/14845/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-14845/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12947486 - PreCommit-HIVE-Build
> Review of NullScan Code
> ---
>
> Key: HIVE-20847
> URL: https://issues.apache.org/jira/browse/HIVE-20847
> Project: Hive
> Issue Type: Improvement
> Components: Physical Optimizer
>Affects Versions: 3.1.0, 4.0.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Minor
> Attachments: HIVE-20847.1.patch, HIVE-20847.1.patch,
> HIVE-20847.1.patch
>
>
> What got me looking at this class was the verboseness of some of the logging.
> I would like to request that we DEBUG the logging since this level of detail
> means nothing to a cluster admin.
> Also... this {{contains}} call would be better applied onto a {{HashSet}}
> instead of an {{ArrayList}}.
> {code:java|title=NullScanTaskDispatcher.java}
> private void processAlias(MapWork work, Path path, ArrayList
> aliasesAffected, ArrayList aliases) {
> // the aliases that are allowed to map to a null scan.
> ArrayList allowed = new ArrayList();
> for (String alias : aliasesAffected) {
> if (aliases.contains(alias)) {
> allowed.add(alias);
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (HIVE-20901) running compactor when there is nothing to do produces duplicate data
[
https://issues.apache.org/jira/browse/HIVE-20901?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eugene Koifman reassigned HIVE-20901:
-
> running compactor when there is nothing to do produces duplicate data
> -
>
> Key: HIVE-20901
> URL: https://issues.apache.org/jira/browse/HIVE-20901
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 4.0.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
>Priority: Major
>
> suppose we run minor compaction 2 times, via alter table
> The 2nd request to compaction should have nothing to do but I don't think
> there is a check for that. It's visible in the context of HIVE-20823, where
> each compactor run produces a delta with new visibility suffix so we end up
> with something like
> {noformat}
> target/tmp/org.apache.hadoop.hive.ql.TestTxnCommands3-1541810844849/warehouse/t/
> ├── delete_delta_001_002_v019
> │ ├── _orc_acid_version
> │ └── bucket_0
> ├── delete_delta_001_002_v021
> │ ├── _orc_acid_version
> │ └── bucket_0
> ├── delta_001_001_
> │ ├── _orc_acid_version
> │ └── bucket_0
> ├── delta_001_002_v019
> │ ├── _orc_acid_version
> │ └── bucket_0
> ├── delta_001_002_v021
> │ ├── _orc_acid_version
> │ └── bucket_0
> └── delta_002_002_
> ├── _orc_acid_version
> └── bucket_0{noformat}
> i.e. 2 deltas with the same write ID range
> this is bad. Probably happens today as well but new run produces a delta
> with the same name and clobbers the previous one, which may interfere with
> writers
>
> need to investigate
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (HIVE-20902) Math.abs(rand.nextInt())) or Math.abs(rand.nexLong())) can return a negative number
[ https://issues.apache.org/jira/browse/HIVE-20902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] slim bouguerra reassigned HIVE-20902: - > Math.abs(rand.nextInt())) or Math.abs(rand.nexLong())) can return a negative > number > > > Key: HIVE-20902 > URL: https://issues.apache.org/jira/browse/HIVE-20902 > Project: Hive > Issue Type: Bug >Reporter: slim bouguerra >Assignee: slim bouguerra >Priority: Major > > i see a lot of Math.abs(rand.nextInt())) in the code base and this can > return a negative number. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20893) BloomK Filter probing method is not thread safe
[ https://issues.apache.org/jira/browse/HIVE-20893?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16682121#comment-16682121 ] slim bouguerra commented on HIVE-20893: --- I did test this locally and added some tests for float case. [^HIVE-20893.patch] Will run some Benchmark soon on our internal Cluster while tests will run on the patch. > BloomK Filter probing method is not thread safe > --- > > Key: HIVE-20893 > URL: https://issues.apache.org/jira/browse/HIVE-20893 > Project: Hive > Issue Type: Bug > Components: storage-api >Reporter: slim bouguerra >Assignee: slim bouguerra >Priority: Major > Attachments: HIVE-20893.WIP.patch, HIVE-20893.patch > > > As far i can tell this is not an issue for Hive yet (most of the usage of > probing seems to be done by one thread at a time) but it is an issue of other > users like Druid as per the following > issue.[https://github.com/apache/incubator-druid/issues/6546] > The fix is proposed by the author of > [https://github.com/apache/incubator-druid/pull/6584] is to make couple of > local fields as ThreadLocals. > Idea looks good to me and doesn't have any perf drawbacks. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20893) BloomK Filter probing method is not thread safe
[ https://issues.apache.org/jira/browse/HIVE-20893?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] slim bouguerra updated HIVE-20893: -- Attachment: HIVE-20893.patch > BloomK Filter probing method is not thread safe > --- > > Key: HIVE-20893 > URL: https://issues.apache.org/jira/browse/HIVE-20893 > Project: Hive > Issue Type: Bug > Components: storage-api >Reporter: slim bouguerra >Assignee: slim bouguerra >Priority: Major > Attachments: HIVE-20893.WIP.patch, HIVE-20893.patch > > > As far i can tell this is not an issue for Hive yet (most of the usage of > probing seems to be done by one thread at a time) but it is an issue of other > users like Druid as per the following > issue.[https://github.com/apache/incubator-druid/issues/6546] > The fix is proposed by the author of > [https://github.com/apache/incubator-druid/pull/6584] is to make couple of > local fields as ThreadLocals. > Idea looks good to me and doesn't have any perf drawbacks. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20893) BloomK Filter probing method is not thread safe
[ https://issues.apache.org/jira/browse/HIVE-20893?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] slim bouguerra updated HIVE-20893: -- Assignee: slim bouguerra Status: Patch Available (was: Open) > BloomK Filter probing method is not thread safe > --- > > Key: HIVE-20893 > URL: https://issues.apache.org/jira/browse/HIVE-20893 > Project: Hive > Issue Type: Bug > Components: storage-api >Reporter: slim bouguerra >Assignee: slim bouguerra >Priority: Major > Attachments: HIVE-20893.WIP.patch, HIVE-20893.patch > > > As far i can tell this is not an issue for Hive yet (most of the usage of > probing seems to be done by one thread at a time) but it is an issue of other > users like Druid as per the following > issue.[https://github.com/apache/incubator-druid/issues/6546] > The fix is proposed by the author of > [https://github.com/apache/incubator-druid/pull/6584] is to make couple of > local fields as ThreadLocals. > Idea looks good to me and doesn't have any perf drawbacks. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20847) Review of NullScan Code
[
https://issues.apache.org/jira/browse/HIVE-20847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16682115#comment-16682115
]
Hive QA commented on HIVE-20847:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
12s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
57s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
36s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
38s{color} | {color:blue} ql in master has 2315 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
54s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
0s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
36s{color} | {color:red} ql: The patch generated 1 new + 11 unchanged - 8 fixed
= 12 total (was 19) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
56s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 21m 51s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-14845/dev-support/hive-personality.sh
|
| git revision | master / 7ae4a2c |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-14845/yetus/diff-checkstyle-ql.txt
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-14845/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Review of NullScan Code
> ---
>
> Key: HIVE-20847
> URL: https://issues.apache.org/jira/browse/HIVE-20847
> Project: Hive
> Issue Type: Improvement
> Components: Physical Optimizer
>Affects Versions: 3.1.0, 4.0.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Minor
> Attachments: HIVE-20847.1.patch, HIVE-20847.1.patch,
> HIVE-20847.1.patch
>
>
> What got me looking at this class was the verboseness of some of the logging.
> I would like to request that we DEBUG the logging since this level of detail
> means nothing to a cluster admin.
> Also... this {{contains}} call would be better applied onto a {{HashSet}}
> instead of an {{ArrayList}}.
> {code:java|title=NullScanTaskDispatcher.java}
> private void processAlias(MapWork work, Path path, ArrayList
> aliasesAffected, ArrayList aliases) {
> // the aliases that are allowed to map to a null scan.
> ArrayList allowed = new ArrayList();
> for (String alias : aliasesAffected) {
> if (aliases.contains(alias)) {
> allowed.add(alias);
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20900) serde2.JsonSerDe no longer supports timestamp.formats
[ https://issues.apache.org/jira/browse/HIVE-20900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16682113#comment-16682113 ] slim bouguerra commented on HIVE-20900: --- did add this https://issues.apache.org/jira/browse/HIVE-20375 couple of month ago, seems linked > serde2.JsonSerDe no longer supports timestamp.formats > - > > Key: HIVE-20900 > URL: https://issues.apache.org/jira/browse/HIVE-20900 > Project: Hive > Issue Type: Bug > Components: Serializers/Deserializers >Reporter: Jason Dere >Assignee: Jason Dere >Priority: Major > > Looks like HIVE-18545 broke this. > Also json_serde_tsformat.q only tested the hcat version of JsonSerde, and the > format in that test used the ISO timestamp format which apparently is now > parsed by the default timestamp parsing, so the test was too simple. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20896) CachedStore fail to cache stats in multiple code paths
[ https://issues.apache.org/jira/browse/HIVE-20896?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Dai updated HIVE-20896: -- Attachment: HIVE-20896.1.patch > CachedStore fail to cache stats in multiple code paths > -- > > Key: HIVE-20896 > URL: https://issues.apache.org/jira/browse/HIVE-20896 > Project: Hive > Issue Type: Bug > Components: Standalone Metastore >Reporter: Daniel Dai >Assignee: Daniel Dai >Priority: Major > Attachments: HIVE-20896.0.patch, HIVE-20896.1.patch > > > Bunch of issues discovered in CachedStore to keep up column statistics: > 1. The criteria for partition/non-partition is wrong > (table.isSetPartitionKeys() is always true) > 2. In update(), partition column stats are removed when populate table basic > stats > 3. Dirty flags are true right after prewarm(), so the first update() does not > do anything > 4. Could invoke cacheLock without holding the lock, which results a freeze in > update() -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20665) Hive Parallel Tasks - Hive Configuration ConcurrentModificationException
[
https://issues.apache.org/jira/browse/HIVE-20665?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16682102#comment-16682102
]
Hive QA commented on HIVE-20665:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12947487/HIVE-20665.1.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 9 failed/errored test(s), 15537 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver[hbase_handler_bulk]
(batchId=106)
org.apache.hadoop.hive.cli.TestMiniDruidCliDriver.testCliDriver[druid_timestamptz2]
(batchId=194)
org.apache.hadoop.hive.cli.TestMiniDruidCliDriver.testCliDriver[druidmini_dynamic_partition]
(batchId=195)
org.apache.hadoop.hive.cli.TestMiniDruidCliDriver.testCliDriver[druidmini_mv]
(batchId=194)
org.apache.hadoop.hive.cli.TestMiniDruidCliDriver.testCliDriver[druidmini_test_alter]
(batchId=195)
org.apache.hadoop.hive.cli.TestMiniDruidCliDriver.testCliDriver[druidmini_test_insert]
(batchId=195)
org.apache.hadoop.hive.cli.TestMiniDruidCliDriver.testCliDriver[kafka_storage_handler]
(batchId=194)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[cbo_limit]
(batchId=182)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[druid_address]
(batchId=98)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/14844/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/14844/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-14844/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 9 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12947487 - PreCommit-HIVE-Build
> Hive Parallel Tasks - Hive Configuration ConcurrentModificationException
>
>
> Key: HIVE-20665
> URL: https://issues.apache.org/jira/browse/HIVE-20665
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 2.3.2, 3.1.0, 4.0.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Major
> Attachments: HIVE-20665.1.patch, HIVE-20665.1.patch
>
>
> When parallel tasks are enabled in Hive, all of the resulting queries share
> the same Hive configuration. This is problematic as each query will modify
> the same {{HiveConf}} object with things like query ID and query text. This
> will overwrite each other and cause {{ConcurrentModificationException}}
> issues.
> {code:java|title=SQLOperation.java}
> public Object run() throws HiveSQLException {
> Hive.set(parentHive, false);
> // TODO: can this result in cross-thread reuse of session state?
> SessionState.setCurrentSessionState(parentSessionState);
> PerfLogger.setPerfLogger(SessionState.getPerfLogger());
> LogUtils.registerLoggingContext(queryState.getConf());
> try {
> if (asyncPrepare) {
> prepare(queryState);
> }
> runQuery();
> } catch (HiveSQLException e) {
> // ...
> {code}
> [Code
> Here|https://github.com/apache/hive/blob/6e27a5315a44c55ef3b178e7212c9068de322d01/service/src/java/org/apache/hive/service/cli/operation/SQLOperation.java#L308-L319]
> From this code it can be seen that for every thread launched, it is all
> calling {{setCurrentSessionState}}.
> {code:java|title=SessionStates.java}
> /**
>* Sets the given session state in the thread local var for sessions.
>*/
> public static void setCurrentSessionState(SessionState startSs) {
> tss.get().attach(startSs);
> }
> // SessionState is not available in runtime and Hive.get().getConf() is not
> safe to call
> private static class SessionStates {
> private SessionState state;
> private HiveConf conf;
> private void attach(SessionState state) {
> this.state = state;
> attach(state.getConf());
> }
> private void attach(HiveConf conf) {
> // -- SHALLOW COPY HERE, ALL THREADS SHARING SAME REFERENCE -- //
> this.conf = conf;
> ClassLoader classLoader = conf.getClassLoader();
> if (classLoader != null) {
> Thread.currentThread().setContextClassLoader(classLoader);
> }
> }
> }
> {code}
> [Code
> Here|https://github.com/apache/hive/blob/7795c0a7dc59941671f8845d78b16d9e5ddc9ea3/ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java#L540-L556]
> Ensure that all threads get
[jira] [Assigned] (HIVE-20900) serde2.JsonSerDe no longer supports timestamp.formats
[ https://issues.apache.org/jira/browse/HIVE-20900?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Dere reassigned HIVE-20900: - Assignee: Jason Dere > serde2.JsonSerDe no longer supports timestamp.formats > - > > Key: HIVE-20900 > URL: https://issues.apache.org/jira/browse/HIVE-20900 > Project: Hive > Issue Type: Bug > Components: Serializers/Deserializers >Reporter: Jason Dere >Assignee: Jason Dere >Priority: Major > > Looks like HIVE-18545 broke this. > Also json_serde_tsformat.q only tested the hcat version of JsonSerde, and the > format in that test used the ISO timestamp format which apparently is now > parsed by the default timestamp parsing, so the test was too simple. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20665) Hive Parallel Tasks - Hive Configuration ConcurrentModificationException
[
https://issues.apache.org/jira/browse/HIVE-20665?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16682067#comment-16682067
]
Hive QA commented on HIVE-20665:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
1s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
45s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
0s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
37s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
45s{color} | {color:blue} ql in master has 2315 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
53s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 22m 26s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-14844/dev-support/hive-personality.sh
|
| git revision | master / 7ae4a2c |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-14844/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Hive Parallel Tasks - Hive Configuration ConcurrentModificationException
>
>
> Key: HIVE-20665
> URL: https://issues.apache.org/jira/browse/HIVE-20665
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 2.3.2, 3.1.0, 4.0.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Major
> Attachments: HIVE-20665.1.patch, HIVE-20665.1.patch
>
>
> When parallel tasks are enabled in Hive, all of the resulting queries share
> the same Hive configuration. This is problematic as each query will modify
> the same {{HiveConf}} object with things like query ID and query text. This
> will overwrite each other and cause {{ConcurrentModificationException}}
> issues.
> {code:java|title=SQLOperation.java}
> public Object run() throws HiveSQLException {
> Hive.set(parentHive, false);
> // TODO: can this result in cross-thread reuse of session state?
> SessionState.setCurrentSessionState(parentSessionState);
> PerfLogger.setPerfLogger(SessionState.getPerfLogger());
> LogUtils.registerLoggingContext(queryState.getConf());
> try {
> if (asyncPrepare) {
> prepare(queryState);
> }
> runQuery();
> } catch (HiveSQLException e) {
> // ...
> {code}
> [Code
> Here|https://github.com/apache/hive/blob/6e27a5315a44c55ef3b178e7212c9068de322d01/service/src/java/org/apache/hive/service/cli/operation/SQLOperation.java#L308-L319]
> From this code it can be seen that for every thread
[jira] [Commented] (HIVE-19354) from_utc_timestamp returns incorrect results for datetime values with timezone
[
https://issues.apache.org/jira/browse/HIVE-19354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16682045#comment-16682045
]
Hive QA commented on HIVE-19354:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12922083/HIVE-19354.01.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/14843/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/14843/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-14843/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2018-11-09 23:02:07.171
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-14843/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2018-11-09 23:02:07.174
+ cd apache-github-source-source
+ git fetch origin
+ git reset --hard HEAD
HEAD is now at 7ae4a2c HIVE-20833: package.jdo needs to be updated to conform
with HIVE-20221 changes (Vineet Garg, reviewed by Jason Dere)
+ git clean -f -d
+ git checkout master
Already on 'master'
Your branch is up-to-date with 'origin/master'.
+ git reset --hard origin/master
HEAD is now at 7ae4a2c HIVE-20833: package.jdo needs to be updated to conform
with HIVE-20221 changes (Vineet Garg, reviewed by Jason Dere)
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2018-11-09 23:02:07.737
+ rm -rf ../yetus_PreCommit-HIVE-Build-14843
+ mkdir ../yetus_PreCommit-HIVE-Build-14843
+ git gc
+ cp -R . ../yetus_PreCommit-HIVE-Build-14843
+ mkdir /data/hiveptest/logs/PreCommit-HIVE-Build-14843/yetus
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh
/data/hiveptest/working/scratch/build.patch
error: patch failed:
ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFFromUtcTimestamp.java:20
Falling back to three-way merge...
Applied patch to
'ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFFromUtcTimestamp.java'
with conflicts.
error: patch failed:
serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/PrimitiveObjectInspectorUtils.java:27
Falling back to three-way merge...
Applied patch to
'serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/PrimitiveObjectInspectorUtils.java'
with conflicts.
Going to apply patch with: git apply -p0
error: patch failed:
ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFFromUtcTimestamp.java:20
Falling back to three-way merge...
Applied patch to
'ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFFromUtcTimestamp.java'
with conflicts.
error: patch failed:
serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/PrimitiveObjectInspectorUtils.java:27
Falling back to three-way merge...
Applied patch to
'serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/PrimitiveObjectInspectorUtils.java'
with conflicts.
U
ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFFromUtcTimestamp.java
U
serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/PrimitiveObjectInspectorUtils.java
+ result=1
+ '[' 1 -ne 0 ']'
+ rm -rf yetus_PreCommit-HIVE-Build-14843
+ exit 1
'
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12922083 - PreCommit-HIVE-Build
> from_utc_timestamp returns incorrect results for datetime values with timezone
> --
>
> Key: HIVE-19354
> URL: https://issues.apache.org/jira/browse/HIVE-19354
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 3.1.0
>Reporter:
[jira] [Commented] (HIVE-20855) Clean up checkstyle & other warnings for CachedStore
[
https://issues.apache.org/jira/browse/HIVE-20855?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16682043#comment-16682043
]
Hive QA commented on HIVE-20855:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12947478/HIVE-20855.3.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/14842/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/14842/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-14842/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2018-11-09 23:00:58.351
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-14842/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2018-11-09 23:00:58.354
+ cd apache-github-source-source
+ git fetch origin
>From https://github.com/apache/hive
10f4ead..7ae4a2c master -> origin/master
+ git reset --hard HEAD
HEAD is now at 10f4ead HIVE-20853 : Expose ShuffleHandler.registerDag in the
llap daemon API (Jaume Marhuenda, reviewed by Sergey Shelukhin)
+ git clean -f -d
Removing standalone-metastore/metastore-server/src/gen/
+ git checkout master
Already on 'master'
Your branch is behind 'origin/master' by 1 commit, and can be fast-forwarded.
(use "git pull" to update your local branch)
+ git reset --hard origin/master
HEAD is now at 7ae4a2c HIVE-20833: package.jdo needs to be updated to conform
with HIVE-20221 changes (Vineet Garg, reviewed by Jason Dere)
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2018-11-09 23:00:59.929
+ rm -rf ../yetus_PreCommit-HIVE-Build-14842
+ mkdir ../yetus_PreCommit-HIVE-Build-14842
+ git gc
+ cp -R . ../yetus_PreCommit-HIVE-Build-14842
+ mkdir /data/hiveptest/logs/PreCommit-HIVE-Build-14842/yetus
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh
/data/hiveptest/working/scratch/build.patch
error:
a/itests/hcatalog-unit/src/test/java/org/apache/hive/hcatalog/listener/DummyRawStoreFailEvent.java:
does not exist in index
error:
a/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java:
does not exist in index
error:
a/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java:
does not exist in index
error:
a/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/RawStore.java:
does not exist in index
error:
a/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/cache/ByteArrayWrapper.java:
does not exist in index
error:
a/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/cache/CacheUtils.java:
does not exist in index
error:
a/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java:
does not exist in index
error:
a/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/cache/SharedCache.java:
does not exist in index
error:
a/standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/DummyRawStoreControlledCommit.java:
does not exist in index
error:
a/standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/DummyRawStoreForJdoConnection.java:
does not exist in index
error:
a/standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/InjectableBehaviourObjectStore.java:
does not exist in index
error:
a/standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/TestOldSchema.java:
does not exist in index
error:
[jira] [Commented] (HIVE-20850) Push case conditional from projections to dimension tables if possible
[
https://issues.apache.org/jira/browse/HIVE-20850?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16682041#comment-16682041
]
Hive QA commented on HIVE-20850:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12947547/HIVE-20850.06.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/14841/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/14841/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-14841/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Tests exited with: Exception: Patch URL
https://issues.apache.org/jira/secure/attachment/12947547/HIVE-20850.06.patch
was found in seen patch url's cache and a test was probably run already on it.
Aborting...
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12947547 - PreCommit-HIVE-Build
> Push case conditional from projections to dimension tables if possible
> --
>
> Key: HIVE-20850
> URL: https://issues.apache.org/jira/browse/HIVE-20850
> Project: Hive
> Issue Type: Improvement
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Attachments: HIVE-20850.01.patch, HIVE-20850.01.patch,
> HIVE-20850.01.patch, HIVE-20850.02.patch, HIVE-20850.03.patch,
> HIVE-20850.04.patch, HIVE-20850.05.patch, HIVE-20850.06.patch
>
>
> noticed by [~gopalv]: If there is a project which could be only evaluated
> after the join; but the condition references only a single column from a
> small dimension table; hive will end up evaluating the same thing over and
> over again...
> {code}
> explain
> select s_store_name, s_store_id,
> sum(case when (d_day_name='Sunday') then ss_sales_price else null
> end) sun_sales,
> sum(case when (d_day_name='Monday') then ss_sales_price else null
> end) mon_sales,
> sum(case when (d_day_name='Tuesday') then ss_sales_price else null
> end) tue_sales,
> sum(case when (d_day_name='Wednesday') then ss_sales_price else null
> end) wed_sales,
> sum(case when (d_day_name='Thursday') then ss_sales_price else null
> end) thu_sales,
> sum(case when (d_day_name='Friday') then ss_sales_price else null
> end) fri_sales,
> sum(case when (d_day_name='Saturday') then ss_sales_price else null
> end) sat_sales
> from date_dim, store_sales, store
> where d_date_sk = ss_sold_date_sk and
>s_store_sk = ss_store_sk and
>s_gmt_offset = -6 and
>d_year = 1998
> group by s_store_name, s_store_id
> order by s_store_name,
> s_store_id,sun_sales,mon_sales,tue_sales,wed_sales,thu_sales,fri_sales,sat_sales
> limit 100;
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20895) Cleanup JdbcColumn Class
[
https://issues.apache.org/jira/browse/HIVE-20895?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16682040#comment-16682040
]
Hive QA commented on HIVE-20895:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12947471/HIVE-20895.1.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 15537 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_groupby_reduce]
(batchId=61)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/14840/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/14840/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-14840/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12947471 - PreCommit-HIVE-Build
> Cleanup JdbcColumn Class
>
>
> Key: HIVE-20895
> URL: https://issues.apache.org/jira/browse/HIVE-20895
> Project: Hive
> Issue Type: Improvement
> Components: JDBC
>Affects Versions: 4.0.0, 3.1.1
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Minor
> Attachments: HIVE-20895.1.patch
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20826) Enhance HiveSemiJoin rule to convert join + group by on left side to Left Semi Join
[
https://issues.apache.org/jira/browse/HIVE-20826?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vineet Garg updated HIVE-20826:
---
Status: Patch Available (was: Open)
> Enhance HiveSemiJoin rule to convert join + group by on left side to Left
> Semi Join
> ---
>
> Key: HIVE-20826
> URL: https://issues.apache.org/jira/browse/HIVE-20826
> Project: Hive
> Issue Type: Improvement
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
> Attachments: HIVE-20826.1.patch, HIVE-20826.2.patch,
> HIVE-20826.3.patch, HIVE-20826.4.patch
>
>
> Currently HiveSemiJoin rule looks for pattern where group by is on right side.
> We can convert joins which have group by on left side (assuming group by keys
> are same as join keys and none of the columns are being projected from left
> side) to LEFT SEMI JOIN by swapping the inputs. e.g. queries such as:
> {code:sql}
> explain select pp.p_partkey from (select distinct p_name from part) p join
> part pp on pp.p_name = p.p_name;
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20826) Enhance HiveSemiJoin rule to convert join + group by on left side to Left Semi Join
[
https://issues.apache.org/jira/browse/HIVE-20826?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vineet Garg updated HIVE-20826:
---
Status: Open (was: Patch Available)
> Enhance HiveSemiJoin rule to convert join + group by on left side to Left
> Semi Join
> ---
>
> Key: HIVE-20826
> URL: https://issues.apache.org/jira/browse/HIVE-20826
> Project: Hive
> Issue Type: Improvement
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
> Attachments: HIVE-20826.1.patch, HIVE-20826.2.patch,
> HIVE-20826.3.patch, HIVE-20826.4.patch
>
>
> Currently HiveSemiJoin rule looks for pattern where group by is on right side.
> We can convert joins which have group by on left side (assuming group by keys
> are same as join keys and none of the columns are being projected from left
> side) to LEFT SEMI JOIN by swapping the inputs. e.g. queries such as:
> {code:sql}
> explain select pp.p_partkey from (select distinct p_name from part) p join
> part pp on pp.p_name = p.p_name;
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20826) Enhance HiveSemiJoin rule to convert join + group by on left side to Left Semi Join
[
https://issues.apache.org/jira/browse/HIVE-20826?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vineet Garg updated HIVE-20826:
---
Attachment: HIVE-20826.4.patch
> Enhance HiveSemiJoin rule to convert join + group by on left side to Left
> Semi Join
> ---
>
> Key: HIVE-20826
> URL: https://issues.apache.org/jira/browse/HIVE-20826
> Project: Hive
> Issue Type: Improvement
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
> Attachments: HIVE-20826.1.patch, HIVE-20826.2.patch,
> HIVE-20826.3.patch, HIVE-20826.4.patch
>
>
> Currently HiveSemiJoin rule looks for pattern where group by is on right side.
> We can convert joins which have group by on left side (assuming group by keys
> are same as join keys and none of the columns are being projected from left
> side) to LEFT SEMI JOIN by swapping the inputs. e.g. queries such as:
> {code:sql}
> explain select pp.p_partkey from (select distinct p_name from part) p join
> part pp on pp.p_name = p.p_name;
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20744) Use SQL constraints to improve join reordering algorithm
[ https://issues.apache.org/jira/browse/HIVE-20744?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16682010#comment-16682010 ] Ashutosh Chauhan commented on HIVE-20744: - +1 > Use SQL constraints to improve join reordering algorithm > > > Key: HIVE-20744 > URL: https://issues.apache.org/jira/browse/HIVE-20744 > Project: Hive > Issue Type: Bug > Components: CBO >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Major > Attachments: HIVE-20744.01.patch, HIVE-20744.02.patch, > HIVE-20744.03.patch, HIVE-20744.04.patch, HIVE-20744.05.patch, > HIVE-20744.patch > > > Till now, it was all based on stats stored for the base tables and their > columns. Now the optimizer can rely on constraints. Hence, this patch is for > the join reordering costing to use constraints, and if it does not find any, > rely on old code path. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20842) Fix logic introduced in HIVE-20660 to estimate statistics for group by
[ https://issues.apache.org/jira/browse/HIVE-20842?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-20842: --- Status: Open (was: Patch Available) > Fix logic introduced in HIVE-20660 to estimate statistics for group by > -- > > Key: HIVE-20842 > URL: https://issues.apache.org/jira/browse/HIVE-20842 > Project: Hive > Issue Type: Bug > Components: Statistics >Affects Versions: 4.0.0 >Reporter: Vineet Garg >Assignee: Vineet Garg >Priority: Major > Attachments: HIVE-20842.1.patch, HIVE-20842.2.patch, > HIVE-20842.3.patch, HIVE-20842.4.patch, HIVE-20842.5.patch, > HIVE-20842.6.patch, HIVE-20842.7.patch, HIVE-20842.8.patch > > > HIVE-20660 introduced better estimation for group by operator. But the logic > did not account for Partial and Full group by separately. > For partial group by parallelism (i.e. number of tasks) should be taken into > account. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20842) Fix logic introduced in HIVE-20660 to estimate statistics for group by
[ https://issues.apache.org/jira/browse/HIVE-20842?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-20842: --- Status: Patch Available (was: Open) > Fix logic introduced in HIVE-20660 to estimate statistics for group by > -- > > Key: HIVE-20842 > URL: https://issues.apache.org/jira/browse/HIVE-20842 > Project: Hive > Issue Type: Bug > Components: Statistics >Affects Versions: 4.0.0 >Reporter: Vineet Garg >Assignee: Vineet Garg >Priority: Major > Attachments: HIVE-20842.1.patch, HIVE-20842.2.patch, > HIVE-20842.3.patch, HIVE-20842.4.patch, HIVE-20842.5.patch, > HIVE-20842.6.patch, HIVE-20842.7.patch, HIVE-20842.8.patch > > > HIVE-20660 introduced better estimation for group by operator. But the logic > did not account for Partial and Full group by separately. > For partial group by parallelism (i.e. number of tasks) should be taken into > account. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20842) Fix logic introduced in HIVE-20660 to estimate statistics for group by
[ https://issues.apache.org/jira/browse/HIVE-20842?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-20842: --- Attachment: HIVE-20842.8.patch > Fix logic introduced in HIVE-20660 to estimate statistics for group by > -- > > Key: HIVE-20842 > URL: https://issues.apache.org/jira/browse/HIVE-20842 > Project: Hive > Issue Type: Bug > Components: Statistics >Affects Versions: 4.0.0 >Reporter: Vineet Garg >Assignee: Vineet Garg >Priority: Major > Attachments: HIVE-20842.1.patch, HIVE-20842.2.patch, > HIVE-20842.3.patch, HIVE-20842.4.patch, HIVE-20842.5.patch, > HIVE-20842.6.patch, HIVE-20842.7.patch, HIVE-20842.8.patch > > > HIVE-20660 introduced better estimation for group by operator. But the logic > did not account for Partial and Full group by separately. > For partial group by parallelism (i.e. number of tasks) should be taken into > account. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20833) package.jdo needs to be updated to conform with HIVE-20221 changes
[
https://issues.apache.org/jira/browse/HIVE-20833?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vineet Garg updated HIVE-20833:
---
Resolution: Fixed
Fix Version/s: 4.0.0
Status: Resolved (was: Patch Available)
Pushed to master
> package.jdo needs to be updated to conform with HIVE-20221 changes
> --
>
> Key: HIVE-20833
> URL: https://issues.apache.org/jira/browse/HIVE-20833
> Project: Hive
> Issue Type: Bug
>Affects Versions: 4.0.0
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-20833.1.patch, HIVE-20833.2.patch,
> HIVE-20833.3.patch, HIVE-20833.4.patch, HIVE-20833.5.patch,
> HIVE-20833.6.patch, HIVE-20833.7.patch, HIVE-20833.8.patch
>
>
> Following test if run with TestMiniLlapLocalCliDriver will fail:
> {code:sql}
> CREATE TABLE `alterPartTbl`(
>`po_header_id` bigint,
>`vendor_num` string,
>`requester_name` string,
>`approver_name` string,
>`buyer_name` string,
>`preparer_name` string,
>`po_requisition_number` string,
>`po_requisition_id` bigint,
>`po_requisition_desc` string,
>`rate_type` string,
>`rate_date` date,
>`rate` double,
>`blanket_total_amount` double,
>`authorization_status` string,
>`revision_num` bigint,
>`revised_date` date,
>`approved_flag` string,
>`approved_date` timestamp,
>`amount_limit` double,
>`note_to_authorizer` string,
>`note_to_vendor` string,
>`note_to_receiver` string,
>`vendor_order_num` string,
>`comments` string,
>`acceptance_required_flag` string,
>`acceptance_due_date` date,
>`closed_date` timestamp,
>`user_hold_flag` string,
>`approval_required_flag` string,
>`cancel_flag` string,
>`firm_status_lookup_code` string,
>`firm_date` date,
>`frozen_flag` string,
>`closed_code` string,
>`org_id` bigint,
>`reference_num` string,
>`wf_item_type` string,
>`wf_item_key` string,
>`submit_date` date,
>`sap_company_code` string,
>`sap_fiscal_year` bigint,
>`po_number` string,
>`sap_line_item` bigint,
>`closed_status_flag` string,
>`balancing_segment` string,
>`cost_center_segment` string,
>`base_amount_limit` double,
>`base_blanket_total_amount` double,
>`base_open_amount` double,
>`base_ordered_amount` double,
>`cancel_date` timestamp,
>`cbc_accounting_date` date,
>`change_requested_by` string,
>`change_summary` string,
>`confirming_order_flag` string,
>`document_creation_method` string,
>`edi_processed_flag` string,
>`edi_processed_status` string,
>`enabled_flag` string,
>`encumbrance_required_flag` string,
>`end_date` date,
>`end_date_active` date,
>`from_header_id` bigint,
>`from_type_lookup_code` string,
>`global_agreement_flag` string,
>`government_context` string,
>`interface_source_code` string,
>`ledger_currency_code` string,
>`open_amount` double,
>`ordered_amount` double,
>`pay_on_code` string,
>`payment_term_name` string,
>`pending_signature_flag` string,
>`po_revision_num` double,
>`preparer_id` bigint,
>`price_update_tolerance` double,
>`print_count` double,
>`printed_date` date,
>`reply_date` date,
>`reply_method_lookup_code` string,
>`rfq_close_date` date,
>`segment2` string,
>`segment3` string,
>`segment4` string,
>`segment5` string,
>`shipping_control` string,
>`start_date` date,
>`start_date_active` date,
>`summary_flag` string,
>`supply_agreement_flag` string,
>`usd_amount_limit` double,
>`usd_blanket_total_amount` double,
>`usd_exchange_rate` double,
>`usd_open_amount` double,
>`usd_order_amount` double,
>`ussgl_transaction_code` string,
>`xml_flag` string,
>`purchasing_organization_id` bigint,
>`purchasing_group_code` string,
>`last_updated_by_name` string,
>`created_by_name` string,
>`incoterms_1` string,
>`incoterms_2` string,
>`ame_approval_id` double,
>`ame_transaction_type` string,
>`auto_sourcing_flag` string,
>`cat_admin_auth_enabled_flag` string,
>`clm_document_number` string,
>`comm_rev_num` double,
>`consigned_consumption_flag` string,
>`consume_req_demand_flag` string,
>`conterms_articles_upd_date` timestamp,
>`conterms_deliv_upd_date` timestamp,
>`conterms_exist_flag` string,
>`cpa_reference` double,
>`created_language` string,
>`email_address` string,
>`enable_all_sites` string,
>`fax` string,
>`lock_owner_role` string,
>`lock_owner_user_id` double,
>`min_release_amount` double,
>`mrc_rate` string,
>`mrc_rate_date` string,
>`mrc_rate_type`
[jira] [Commented] (HIVE-20895) Cleanup JdbcColumn Class
[
https://issues.apache.org/jira/browse/HIVE-20895?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16681991#comment-16681991
]
Hive QA commented on HIVE-20895:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
35s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
17s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
11s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
28s{color} | {color:blue} jdbc in master has 17 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
12s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
17s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
10s{color} | {color:red} jdbc: The patch generated 1 new + 8 unchanged - 57
fixed = 9 total (was 65) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
12s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
12s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 11m 11s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-14840/dev-support/hive-personality.sh
|
| git revision | master / 10f4ead |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-14840/yetus/diff-checkstyle-jdbc.txt
|
| modules | C: jdbc U: jdbc |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-14840/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Cleanup JdbcColumn Class
>
>
> Key: HIVE-20895
> URL: https://issues.apache.org/jira/browse/HIVE-20895
> Project: Hive
> Issue Type: Improvement
> Components: JDBC
>Affects Versions: 4.0.0, 3.1.1
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Minor
> Attachments: HIVE-20895.1.patch
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20833) package.jdo needs to be updated to conform with HIVE-20221 changes
[
https://issues.apache.org/jira/browse/HIVE-20833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16681968#comment-16681968
]
Hive QA commented on HIVE-20833:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12947463/HIVE-20833.8.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 15536 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/14839/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/14839/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-14839/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12947463 - PreCommit-HIVE-Build
> package.jdo needs to be updated to conform with HIVE-20221 changes
> --
>
> Key: HIVE-20833
> URL: https://issues.apache.org/jira/browse/HIVE-20833
> Project: Hive
> Issue Type: Bug
>Affects Versions: 4.0.0
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
> Attachments: HIVE-20833.1.patch, HIVE-20833.2.patch,
> HIVE-20833.3.patch, HIVE-20833.4.patch, HIVE-20833.5.patch,
> HIVE-20833.6.patch, HIVE-20833.7.patch, HIVE-20833.8.patch
>
>
> Following test if run with TestMiniLlapLocalCliDriver will fail:
> {code:sql}
> CREATE TABLE `alterPartTbl`(
>`po_header_id` bigint,
>`vendor_num` string,
>`requester_name` string,
>`approver_name` string,
>`buyer_name` string,
>`preparer_name` string,
>`po_requisition_number` string,
>`po_requisition_id` bigint,
>`po_requisition_desc` string,
>`rate_type` string,
>`rate_date` date,
>`rate` double,
>`blanket_total_amount` double,
>`authorization_status` string,
>`revision_num` bigint,
>`revised_date` date,
>`approved_flag` string,
>`approved_date` timestamp,
>`amount_limit` double,
>`note_to_authorizer` string,
>`note_to_vendor` string,
>`note_to_receiver` string,
>`vendor_order_num` string,
>`comments` string,
>`acceptance_required_flag` string,
>`acceptance_due_date` date,
>`closed_date` timestamp,
>`user_hold_flag` string,
>`approval_required_flag` string,
>`cancel_flag` string,
>`firm_status_lookup_code` string,
>`firm_date` date,
>`frozen_flag` string,
>`closed_code` string,
>`org_id` bigint,
>`reference_num` string,
>`wf_item_type` string,
>`wf_item_key` string,
>`submit_date` date,
>`sap_company_code` string,
>`sap_fiscal_year` bigint,
>`po_number` string,
>`sap_line_item` bigint,
>`closed_status_flag` string,
>`balancing_segment` string,
>`cost_center_segment` string,
>`base_amount_limit` double,
>`base_blanket_total_amount` double,
>`base_open_amount` double,
>`base_ordered_amount` double,
>`cancel_date` timestamp,
>`cbc_accounting_date` date,
>`change_requested_by` string,
>`change_summary` string,
>`confirming_order_flag` string,
>`document_creation_method` string,
>`edi_processed_flag` string,
>`edi_processed_status` string,
>`enabled_flag` string,
>`encumbrance_required_flag` string,
>`end_date` date,
>`end_date_active` date,
>`from_header_id` bigint,
>`from_type_lookup_code` string,
>`global_agreement_flag` string,
>`government_context` string,
>`interface_source_code` string,
>`ledger_currency_code` string,
>`open_amount` double,
>`ordered_amount` double,
>`pay_on_code` string,
>`payment_term_name` string,
>`pending_signature_flag` string,
>`po_revision_num` double,
>`preparer_id` bigint,
>`price_update_tolerance` double,
>`print_count` double,
>`printed_date` date,
>`reply_date` date,
>`reply_method_lookup_code` string,
>`rfq_close_date` date,
>`segment2` string,
>`segment3` string,
>`segment4` string,
>`segment5` string,
>`shipping_control` string,
>`start_date` date,
>`start_date_active` date,
>`summary_flag` string,
>`supply_agreement_flag` string,
>`usd_amount_limit` double,
>`usd_blanket_total_amount` double,
>`usd_exchange_rate` double,
>`usd_open_amount` double,
>`usd_order_amount` double,
>`ussgl_transaction_code` string,
>`xml_flag` string,
>`purchasing_organization_id` bigint,
>`purchasing_group_code` string,
>`last_updated_by_name` string,
>`created_by_name` string,
>
[jira] [Commented] (HIVE-20833) package.jdo needs to be updated to conform with HIVE-20221 changes
[
https://issues.apache.org/jira/browse/HIVE-20833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16681937#comment-16681937
]
Hive QA commented on HIVE-20833:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
29s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
17s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
24s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
43s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
39s{color} | {color:blue} ql in master has 2315 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 1m
6s{color} | {color:blue} standalone-metastore/metastore-server in master has
185 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
12s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
9s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
12s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
12s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 27m 2s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-14839/dev-support/hive-personality.sh
|
| git revision | master / 01cef92 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: itests ql standalone-metastore/metastore-server U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-14839/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> package.jdo needs to be updated to conform with HIVE-20221 changes
> --
>
> Key: HIVE-20833
> URL: https://issues.apache.org/jira/browse/HIVE-20833
> Project: Hive
> Issue Type: Bug
>Affects Versions: 4.0.0
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
> Attachments: HIVE-20833.1.patch, HIVE-20833.2.patch,
> HIVE-20833.3.patch, HIVE-20833.4.patch, HIVE-20833.5.patch,
> HIVE-20833.6.patch, HIVE-20833.7.patch, HIVE-20833.8.patch
>
>
> Following test if run with TestMiniLlapLocalCliDriver will fail:
> {code:sql}
> CREATE TABLE `alterPartTbl`(
>`po_header_id` bigint,
>`vendor_num` string,
>`requester_name` string,
>`approver_name` string,
>`buyer_name` string,
>`preparer_name` string,
>`po_requisition_number` string,
>`po_requisition_id` bigint,
>`po_requisition_desc` string,
>`rate_type` string,
>`rate_date` date,
>`rate` double,
>`blanket_total_amount` double,
>`authorization_status` string,
>`revision_num` bigint,
>`revised_date`
[jira] [Updated] (HIVE-20853) Expose ShuffleHandler.registerDag in the llap daemon API
[ https://issues.apache.org/jira/browse/HIVE-20853?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-20853: Resolution: Fixed Fix Version/s: 4.0.0 Status: Resolved (was: Patch Available) Committed to master. Thanks for the patch! > Expose ShuffleHandler.registerDag in the llap daemon API > > > Key: HIVE-20853 > URL: https://issues.apache.org/jira/browse/HIVE-20853 > Project: Hive > Issue Type: Improvement > Components: llap >Affects Versions: 3.1.0 >Reporter: Jaume M >Assignee: Jaume M >Priority: Critical > Fix For: 4.0.0 > > Attachments: HIVE-20853.1.patch, HIVE-20853.2.patch, > HIVE-20853.3.patch, HIVE-20853.4.patch, HIVE-20853.5.patch > > > Currently DAGs are only registered when a submitWork is called for that DAG. > At this point the crendentials are added to the ShuffleHandler and it can > start serving. > However Tez might (and will) schedule tasks to fetch from the ShuffleHandler > before anything of this happens and all this tasks will fail which may > results in the query failing. > This happens in the scenario in which a LlapDaemon just comes up and tez > fetchers try to open a connection before a DAG has been registered. > Adding this API will allow to register the DAG against the Daemon when the AM > notices that a new Daemon is up. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work started] (HIVE-20899) Keytab URI for LLAP YARN Service is restrictive to support HDFS only
[ https://issues.apache.org/jira/browse/HIVE-20899?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HIVE-20899 started by Gour Saha. > Keytab URI for LLAP YARN Service is restrictive to support HDFS only > > > Key: HIVE-20899 > URL: https://issues.apache.org/jira/browse/HIVE-20899 > Project: Hive > Issue Type: Bug > Components: llap >Affects Versions: 3.1.1 >Reporter: Gour Saha >Assignee: Gour Saha >Priority: Major > > llap-server/src/main/resources/package.py restricts the keytab URI to support > HDFS only and hence fails for other FileSystem API conforming FSs like s3a, > wasb, gs, etc. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-20899) Keytab URI for LLAP YARN Service is restrictive to support HDFS only
[ https://issues.apache.org/jira/browse/HIVE-20899?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gour Saha reassigned HIVE-20899: Assignee: Gour Saha > Keytab URI for LLAP YARN Service is restrictive to support HDFS only > > > Key: HIVE-20899 > URL: https://issues.apache.org/jira/browse/HIVE-20899 > Project: Hive > Issue Type: Bug > Components: llap >Affects Versions: 3.1.1 >Reporter: Gour Saha >Assignee: Gour Saha >Priority: Major > > llap-server/src/main/resources/package.py restricts the keytab URI to support > HDFS only and hence fails for other FileSystem API conforming FSs like s3a, > wasb, gs, etc. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20894) Clean Up JDBC HiveQueryResultSet
[
https://issues.apache.org/jira/browse/HIVE-20894?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16681917#comment-16681917
]
Hive QA commented on HIVE-20894:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12947460/HIVE-20894.1.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 15536 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/14838/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/14838/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-14838/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12947460 - PreCommit-HIVE-Build
> Clean Up JDBC HiveQueryResultSet
>
>
> Key: HIVE-20894
> URL: https://issues.apache.org/jira/browse/HIVE-20894
> Project: Hive
> Issue Type: Improvement
> Components: JDBC
>Affects Versions: 4.0.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Minor
> Attachments: HIVE-20894.1.patch
>
>
> Removed unused code and unused imports.
> Simplify the code for readability.
> Remove superfluous calls to {{printStackTrace();}} - let the client code
> handle it
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20894) Clean Up JDBC HiveQueryResultSet
[
https://issues.apache.org/jira/browse/HIVE-20894?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16681874#comment-16681874
]
Hive QA commented on HIVE-20894:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
57s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
18s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
10s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
26s{color} | {color:blue} jdbc in master has 17 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
12s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
32s{color} | {color:green} jdbc generated 0 new + 16 unchanged - 1 fixed = 16
total (was 17) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
12s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 11m 34s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-14838/dev-support/hive-personality.sh
|
| git revision | master / 01cef92 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: jdbc U: jdbc |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-14838/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Clean Up JDBC HiveQueryResultSet
>
>
> Key: HIVE-20894
> URL: https://issues.apache.org/jira/browse/HIVE-20894
> Project: Hive
> Issue Type: Improvement
> Components: JDBC
>Affects Versions: 4.0.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Minor
> Attachments: HIVE-20894.1.patch
>
>
> Removed unused code and unused imports.
> Simplify the code for readability.
> Remove superfluous calls to {{printStackTrace();}} - let the client code
> handle it
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20838) Timestamps with timezone are set to null when using the streaming API
[
https://issues.apache.org/jira/browse/HIVE-20838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16681851#comment-16681851
]
Hive QA commented on HIVE-20838:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12947461/HIVE-20838.9.patch
{color:green}SUCCESS:{color} +1 due to 3 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 5 failed/errored test(s), 15537 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[schema_evol_text_nonvec_part_all_primitive]
(batchId=172)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[schema_evol_text_nonvec_part_all_primitive_llap_io]
(batchId=182)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[schema_evol_text_vec_part_all_primitive]
(batchId=174)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[schema_evol_text_vec_part_all_primitive_llap_io]
(batchId=179)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[schema_evol_text_vecrow_part_all_primitive]
(batchId=175)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/14837/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/14837/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-14837/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 5 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12947461 - PreCommit-HIVE-Build
> Timestamps with timezone are set to null when using the streaming API
> -
>
> Key: HIVE-20838
> URL: https://issues.apache.org/jira/browse/HIVE-20838
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 3.1.0
>Reporter: Jaume M
>Assignee: Jaume M
>Priority: Major
> Attachments: HIVE-20838.1.patch, HIVE-20838.2.patch,
> HIVE-20838.3.patch, HIVE-20838.3.patch, HIVE-20838.4.patch,
> HIVE-20838.5.patch, HIVE-20838.6.patch, HIVE-20838.7.patch,
> HIVE-20838.8.patch, HIVE-20838.9.patch
>
>
> For example:
> {code}
> beeline> create table default.timest (a TIMESTAMP) stored as orc " +
> "TBLPROPERTIES('transactional'='true')
> # And then:
> connection.write("2018-10-19 10:35:00 America/Los_Angeles".getBytes());
> {code}
> inserts NULL.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20838) Timestamps with timezone are set to null when using the streaming API
[
https://issues.apache.org/jira/browse/HIVE-20838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16681797#comment-16681797
]
Hive QA commented on HIVE-20838:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
35s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
14s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
51s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
37s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
29s{color} | {color:blue} common in master has 65 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
37s{color} | {color:blue} serde in master has 198 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
24s{color} | {color:blue} streaming in master has 2 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
37s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
9s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
38s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 17m 21s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-14837/dev-support/hive-personality.sh
|
| git revision | master / 01cef92 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: common serde streaming U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-14837/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Timestamps with timezone are set to null when using the streaming API
> -
>
> Key: HIVE-20838
> URL: https://issues.apache.org/jira/browse/HIVE-20838
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 3.1.0
>Reporter: Jaume M
>Assignee: Jaume M
>Priority: Major
> Attachments: HIVE-20838.1.patch, HIVE-20838.2.patch,
> HIVE-20838.3.patch, HIVE-20838.3.patch, HIVE-20838.4.patch,
> HIVE-20838.5.patch, HIVE-20838.6.patch, HIVE-20838.7.patch,
> HIVE-20838.8.patch, HIVE-20838.9.patch
>
>
> For example:
> {code}
> beeline> create table default.timest (a TIMESTAMP) stored as orc " +
> "TBLPROPERTIES('transactional'='true')
> # And then:
> connection.write("2018-10-19 10:35:00 America/Los_Angeles".getBytes());
> {code}
> inserts NULL.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20512) Improve record and memory usage logging in SparkRecordHandler
[
https://issues.apache.org/jira/browse/HIVE-20512?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bharathkrishna Guruvayoor Murali updated HIVE-20512:
Attachment: (was: HIVE-20512.9.patch)
> Improve record and memory usage logging in SparkRecordHandler
> -
>
> Key: HIVE-20512
> URL: https://issues.apache.org/jira/browse/HIVE-20512
> Project: Hive
> Issue Type: Sub-task
> Components: Spark
>Reporter: Sahil Takiar
>Assignee: Bharathkrishna Guruvayoor Murali
>Priority: Major
> Attachments: HIVE-20512.1.patch, HIVE-20512.2.patch,
> HIVE-20512.3.patch, HIVE-20512.4.patch, HIVE-20512.5.patch,
> HIVE-20512.6.patch, HIVE-20512.7.patch, HIVE-20512.8.patch, HIVE-20512.9.patch
>
>
> We currently log memory usage and # of records processed in Spark tasks, but
> we should improve the methodology for how frequently we log this info.
> Currently we use the following code:
> {code:java}
> private long getNextLogThreshold(long currentThreshold) {
> // A very simple counter to keep track of number of rows processed by the
> // reducer. It dumps
> // every 1 million times, and quickly before that
> if (currentThreshold >= 100) {
> return currentThreshold + 100;
> }
> return 10 * currentThreshold;
> }
> {code}
> The issue is that after a while, the increase by 10x factor means that you
> have to process a huge # of records before this gets triggered.
> A better approach would be to log this info at a given interval. This would
> help in debugging tasks that are seemingly hung.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20512) Improve record and memory usage logging in SparkRecordHandler
[
https://issues.apache.org/jira/browse/HIVE-20512?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bharathkrishna Guruvayoor Murali updated HIVE-20512:
Attachment: HIVE-20512.9.patch
> Improve record and memory usage logging in SparkRecordHandler
> -
>
> Key: HIVE-20512
> URL: https://issues.apache.org/jira/browse/HIVE-20512
> Project: Hive
> Issue Type: Sub-task
> Components: Spark
>Reporter: Sahil Takiar
>Assignee: Bharathkrishna Guruvayoor Murali
>Priority: Major
> Attachments: HIVE-20512.1.patch, HIVE-20512.2.patch,
> HIVE-20512.3.patch, HIVE-20512.4.patch, HIVE-20512.5.patch,
> HIVE-20512.6.patch, HIVE-20512.7.patch, HIVE-20512.8.patch, HIVE-20512.9.patch
>
>
> We currently log memory usage and # of records processed in Spark tasks, but
> we should improve the methodology for how frequently we log this info.
> Currently we use the following code:
> {code:java}
> private long getNextLogThreshold(long currentThreshold) {
> // A very simple counter to keep track of number of rows processed by the
> // reducer. It dumps
> // every 1 million times, and quickly before that
> if (currentThreshold >= 100) {
> return currentThreshold + 100;
> }
> return 10 * currentThreshold;
> }
> {code}
> The issue is that after a while, the increase by 10x factor means that you
> have to process a huge # of records before this gets triggered.
> A better approach would be to log this info at a given interval. This would
> help in debugging tasks that are seemingly hung.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20512) Improve record and memory usage logging in SparkRecordHandler
[
https://issues.apache.org/jira/browse/HIVE-20512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16681769#comment-16681769
]
Hive QA commented on HIVE-20512:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12947442/HIVE-20512.9.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 15501 tests
executed
*Failed tests:*
{noformat}
TestMiniSparkOnYarnCliDriver - did not produce a TEST-*.xml file (likely timed
out) (batchId=189)
[infer_bucket_sort_reducers_power_two.q,list_bucket_dml_10.q,orc_merge9.q,leftsemijoin_mr.q,bucket6.q,bucketmapjoin7.q,uber_reduce.q,empty_dir_in_table.q,vector_outer_join2.q,spark_explain_groupbyshuffle.q,spark_dynamic_partition_pruning.q,spark_combine_equivalent_work.q,orc_merge1.q,spark_use_op_stats.q,orc_merge_diff_fs.q,quotedid_smb.q,truncate_column_buckets.q,spark_vectorized_dynamic_partition_pruning.q,spark_in_process_launcher.q,orc_merge3.q]
TestSparkCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=128)
[load_dyn_part15.q,explaindenpendencydiffengs.q,transform2.q,groupby5.q,cbo_semijoin.q,bucketmapjoin13.q,alter_merge_stats_orc.q,subquery_scalar.q,union_remove_2.q,groupby_position.q,join12.q,smb_mapjoin_8.q,subquery_select.q,join21.q,auto_join16.q]
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/14836/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/14836/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-14836/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12947442 - PreCommit-HIVE-Build
> Improve record and memory usage logging in SparkRecordHandler
> -
>
> Key: HIVE-20512
> URL: https://issues.apache.org/jira/browse/HIVE-20512
> Project: Hive
> Issue Type: Sub-task
> Components: Spark
>Reporter: Sahil Takiar
>Assignee: Bharathkrishna Guruvayoor Murali
>Priority: Major
> Attachments: HIVE-20512.1.patch, HIVE-20512.2.patch,
> HIVE-20512.3.patch, HIVE-20512.4.patch, HIVE-20512.5.patch,
> HIVE-20512.6.patch, HIVE-20512.7.patch, HIVE-20512.8.patch, HIVE-20512.9.patch
>
>
> We currently log memory usage and # of records processed in Spark tasks, but
> we should improve the methodology for how frequently we log this info.
> Currently we use the following code:
> {code:java}
> private long getNextLogThreshold(long currentThreshold) {
> // A very simple counter to keep track of number of rows processed by the
> // reducer. It dumps
> // every 1 million times, and quickly before that
> if (currentThreshold >= 100) {
> return currentThreshold + 100;
> }
> return 10 * currentThreshold;
> }
> {code}
> The issue is that after a while, the increase by 10x factor means that you
> have to process a huge # of records before this gets triggered.
> A better approach would be to log this info at a given interval. This would
> help in debugging tasks that are seemingly hung.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20703) Put dynamic sort partition optimization under cost based decision
[ https://issues.apache.org/jira/browse/HIVE-20703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16681766#comment-16681766 ] Vineet Garg commented on HIVE-20703: [~ychena] You are right this was moved for tez engine only and will not work for MR and spark anymore. Since MR is deprecated now I don't think it make sense to support this for it but if you think this is an issue with Hive on Spark and this optimization should be supported we can probably add this for spark. > Put dynamic sort partition optimization under cost based decision > - > > Key: HIVE-20703 > URL: https://issues.apache.org/jira/browse/HIVE-20703 > Project: Hive > Issue Type: Improvement > Components: Query Planning >Reporter: Vineet Garg >Assignee: Vineet Garg >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-20703.1.patch, HIVE-20703.10.patch, > HIVE-20703.11.patch, HIVE-20703.12.patch, HIVE-20703.2.patch, > HIVE-20703.3.patch, HIVE-20703.4.patch, HIVE-20703.5.patch, > HIVE-20703.6.patch, HIVE-20703.7.patch, HIVE-20703.8.patch, HIVE-20703.9.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20730) Do delete event filtering even if hive.acid.index is not there
[
https://issues.apache.org/jira/browse/HIVE-20730?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Saurabh Seth updated HIVE-20730:
Attachment: HIVE-20730.3.patch
Status: Patch Available (was: In Progress)
Had uploaded an incorrect patch accidentally.
Also fixed the TestOrcRawRecordMerger.testNewBase test failure and uploaded new
patch.
> Do delete event filtering even if hive.acid.index is not there
> --
>
> Key: HIVE-20730
> URL: https://issues.apache.org/jira/browse/HIVE-20730
> Project: Hive
> Issue Type: Improvement
> Components: Transactions
>Affects Versions: 4.0.0
>Reporter: Eugene Koifman
>Assignee: Saurabh Seth
>Priority: Major
> Attachments: HIVE-20730.2.patch, HIVE-20730.3.patch, HIVE-20730.patch
>
>
> since HIVE-16812 {{VectorizedOrcAcidRowBatchReader}} filters delete events
> based on min/max ROW__ID in the split which relies on {{hive.acid.index}} to
> be in the ORC footer.
> There is no way to generate {{hive.acid.index}} from a plain query as in
> HIVE-20699 and so we need to make sure that we generate a SARG into
> delete_delta/bucket_x based on stripe stats even the index is missing
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20730) Do delete event filtering even if hive.acid.index is not there
[
https://issues.apache.org/jira/browse/HIVE-20730?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Saurabh Seth updated HIVE-20730:
Status: In Progress (was: Patch Available)
> Do delete event filtering even if hive.acid.index is not there
> --
>
> Key: HIVE-20730
> URL: https://issues.apache.org/jira/browse/HIVE-20730
> Project: Hive
> Issue Type: Improvement
> Components: Transactions
>Affects Versions: 4.0.0
>Reporter: Eugene Koifman
>Assignee: Saurabh Seth
>Priority: Major
> Attachments: HIVE-20730.2.patch, HIVE-20730.patch
>
>
> since HIVE-16812 {{VectorizedOrcAcidRowBatchReader}} filters delete events
> based on min/max ROW__ID in the split which relies on {{hive.acid.index}} to
> be in the ORC footer.
> There is no way to generate {{hive.acid.index}} from a plain query as in
> HIVE-20699 and so we need to make sure that we generate a SARG into
> delete_delta/bucket_x based on stripe stats even the index is missing
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (HIVE-20392) make compaction atomic on S3
[ https://issues.apache.org/jira/browse/HIVE-20392?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman resolved HIVE-20392. --- Resolution: Won't Fix will be handled in HIVE-20823 > make compaction atomic on S3 > > > Key: HIVE-20392 > URL: https://issues.apache.org/jira/browse/HIVE-20392 > Project: Hive > Issue Type: Bug > Components: Transactions >Reporter: Eugene Koifman >Assignee: Eugene Koifman >Priority: Major > Attachments: HIVE-20392.01.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20512) Improve record and memory usage logging in SparkRecordHandler
[
https://issues.apache.org/jira/browse/HIVE-20512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16681698#comment-16681698
]
Hive QA commented on HIVE-20512:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
34s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
1s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
36s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
40s{color} | {color:blue} ql in master has 2315 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
52s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
36s{color} | {color:green} ql: The patch generated 0 new + 4 unchanged - 5
fixed = 4 total (was 9) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 3m
51s{color} | {color:red} ql generated 1 new + 2315 unchanged - 0 fixed = 2316
total (was 2315) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
12s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 22m 9s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:ql |
| | Increment of volatile field
org.apache.hadoop.hive.ql.exec.spark.SparkRecordHandler.rowNumber in
org.apache.hadoop.hive.ql.exec.spark.SparkRecordHandler.incrementRowNumber()
At SparkRecordHandler.java:in
org.apache.hadoop.hive.ql.exec.spark.SparkRecordHandler.incrementRowNumber()
At SparkRecordHandler.java:[line 113] |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-14836/dev-support/hive-personality.sh
|
| git revision | master / 01cef92 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-14836/yetus/new-findbugs-ql.html
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-14836/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Improve record and memory usage logging in SparkRecordHandler
> -
>
> Key: HIVE-20512
> URL: https://issues.apache.org/jira/browse/HIVE-20512
> Project: Hive
> Issue Type: Sub-task
> Components: Spark
>Reporter: Sahil Takiar
>Assignee: Bharathkrishna Guruvayoor Murali
>Priority: Major
> Attachments: HIVE-20512.1.patch, HIVE-20512.2.patch,
> HIVE-20512.3.patch, HIVE-20512.4.patch, HIVE-20512.5.patch,
> HIVE-20512.6.patch, HIVE-20512.7.patch, HIVE-20512.8.patch, HIVE-20512.9.patch
>
>
> We currently log memory usage and # of records processed in Spark tasks, but
> we should improve the methodology for how frequently we log this info.
> Currently we use the following code:
> {code:java}
> private long getNextLogThreshold(long currentThreshold) {
> // A very simple counter to keep track of number of rows processed by the
> // reducer. It dumps
> // every 1 million times, and quickly before that
> if (currentThreshold >= 100) {
>
[jira] [Commented] (HIVE-20849) Review of ConstantPropagateProcFactory
[
https://issues.apache.org/jira/browse/HIVE-20849?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16681666#comment-16681666
]
Hive QA commented on HIVE-20849:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12947443/HIVE-20849.1.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/14835/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/14835/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-14835/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2018-11-09 16:24:52.582
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-14835/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2018-11-09 16:24:52.585
+ cd apache-github-source-source
+ git fetch origin
+ git reset --hard HEAD
HEAD is now at 01cef92 HIVE-20881: Constant propagation oversimplifies
projections (Zoltan Haindrich reviewed by Ashutosh Chauhan)
+ git clean -f -d
Removing standalone-metastore/metastore-server/src/gen/
+ git checkout master
Already on 'master'
Your branch is up-to-date with 'origin/master'.
+ git reset --hard origin/master
HEAD is now at 01cef92 HIVE-20881: Constant propagation oversimplifies
projections (Zoltan Haindrich reviewed by Ashutosh Chauhan)
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2018-11-09 16:24:53.149
+ rm -rf ../yetus_PreCommit-HIVE-Build-14835
+ mkdir ../yetus_PreCommit-HIVE-Build-14835
+ git gc
+ cp -R . ../yetus_PreCommit-HIVE-Build-14835
+ mkdir /data/hiveptest/logs/PreCommit-HIVE-Build-14835/yetus
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh
/data/hiveptest/working/scratch/build.patch
error:
a/ql/src/java/org/apache/hadoop/hive/ql/optimizer/ConstantPropagateProcFactory.java:
does not exist in index
error: patch failed:
ql/src/java/org/apache/hadoop/hive/ql/optimizer/ConstantPropagateProcFactory.java:63
Falling back to three-way merge...
Applied patch to
'ql/src/java/org/apache/hadoop/hive/ql/optimizer/ConstantPropagateProcFactory.java'
with conflicts.
Going to apply patch with: git apply -p1
/data/hiveptest/working/scratch/build.patch:234: trailing whitespace.
return (udf.getClass().equals(GenericUDFUnixTimeStamp.class)
error: patch failed:
ql/src/java/org/apache/hadoop/hive/ql/optimizer/ConstantPropagateProcFactory.java:63
Falling back to three-way merge...
Applied patch to
'ql/src/java/org/apache/hadoop/hive/ql/optimizer/ConstantPropagateProcFactory.java'
with conflicts.
U
ql/src/java/org/apache/hadoop/hive/ql/optimizer/ConstantPropagateProcFactory.java
warning: 1 line adds whitespace errors.
+ result=1
+ '[' 1 -ne 0 ']'
+ rm -rf yetus_PreCommit-HIVE-Build-14835
+ exit 1
'
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12947443 - PreCommit-HIVE-Build
> Review of ConstantPropagateProcFactory
> --
>
> Key: HIVE-20849
> URL: https://issues.apache.org/jira/browse/HIVE-20849
> Project: Hive
> Issue Type: Improvement
> Components: Logical Optimizer
>Affects Versions: 3.1.0, 4.0.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Minor
> Attachments: HIVE-20849.1.patch, HIVE-20849.1.patch
>
>
> I was looking at this class because it blasts a lot of useless (to an admin)
> information to the logs. Especially if the table has a lot of columns, I see
> big blocks of logging that are meaningless to me. I request that the logging
> is toned down to debug, and some other improvements to the
[jira] [Commented] (HIVE-20730) Do delete event filtering even if hive.acid.index is not there
[
https://issues.apache.org/jira/browse/HIVE-20730?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16681664#comment-16681664
]
Hive QA commented on HIVE-20730:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12947392/HIVE-20730.2.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 15537 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.ql.io.orc.TestOrcRawRecordMerger.testNewBase
(batchId=298)
org.apache.hadoop.hive.ql.io.orc.TestVectorizedOrcAcidRowBatchReader.testDeleteEventFilteringOnWithoutIdx2
(batchId=297)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/14834/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/14834/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-14834/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12947392 - PreCommit-HIVE-Build
> Do delete event filtering even if hive.acid.index is not there
> --
>
> Key: HIVE-20730
> URL: https://issues.apache.org/jira/browse/HIVE-20730
> Project: Hive
> Issue Type: Improvement
> Components: Transactions
>Affects Versions: 4.0.0
>Reporter: Eugene Koifman
>Assignee: Saurabh Seth
>Priority: Major
> Attachments: HIVE-20730.2.patch, HIVE-20730.patch
>
>
> since HIVE-16812 {{VectorizedOrcAcidRowBatchReader}} filters delete events
> based on min/max ROW__ID in the split which relies on {{hive.acid.index}} to
> be in the ORC footer.
> There is no way to generate {{hive.acid.index}} from a plain query as in
> HIVE-20699 and so we need to make sure that we generate a SARG into
> delete_delta/bucket_x based on stripe stats even the index is missing
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work started] (HIVE-20888) TxnHandler: sort() called on immutable lists
[
https://issues.apache.org/jira/browse/HIVE-20888?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on HIVE-20888 started by Igor Kryvenko.
> TxnHandler: sort() called on immutable lists
>
>
> Key: HIVE-20888
> URL: https://issues.apache.org/jira/browse/HIVE-20888
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Reporter: Gopal V
>Assignee: Igor Kryvenko
>Priority: Major
>
> {code}
> } else {
> assert (!rqst.isSetSrcTxnToWriteIdList());
> assert (rqst.isSetTxnIds());
> txnIds = rqst.getTxnIds();
> }
> Collections.sort(txnIds); //easier to read logs and for assumption
> done in replication flow
> {code}
> when the input comes from
> {code}
> @Override
> public long allocateTableWriteId(long txnId, String dbName, String
> tableName) throws TException {
> return allocateTableWriteIdsBatch(Collections.singletonList(txnId),
> dbName, tableName).get(0).getWriteId();
> }
> {code}
> {code}
> java.lang.UnsupportedOperationException: null
> at java.util.AbstractList.set(AbstractList.java:132) ~[?:1.8.0]
> at java.util.AbstractList$ListItr.set(AbstractList.java:426) ~[?:1.8.0]
> at java.util.Collections.sort(Collections.java:170) ~[?:1.8.0]
> at
> org.apache.hadoop.hive.metastore.txn.TxnHandler.allocateTableWriteIds(TxnHandler.java:1523)
> ~[hive-standalone-metastore-server-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.allocate_table_write_ids(HiveMetaStore.java:7349)
> ~[hive-standalone-metastore-server-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (HIVE-20888) TxnHandler: sort() called on immutable lists
[
https://issues.apache.org/jira/browse/HIVE-20888?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Igor Kryvenko reassigned HIVE-20888:
Assignee: Igor Kryvenko
> TxnHandler: sort() called on immutable lists
>
>
> Key: HIVE-20888
> URL: https://issues.apache.org/jira/browse/HIVE-20888
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Reporter: Gopal V
>Assignee: Igor Kryvenko
>Priority: Major
>
> {code}
> } else {
> assert (!rqst.isSetSrcTxnToWriteIdList());
> assert (rqst.isSetTxnIds());
> txnIds = rqst.getTxnIds();
> }
> Collections.sort(txnIds); //easier to read logs and for assumption
> done in replication flow
> {code}
> when the input comes from
> {code}
> @Override
> public long allocateTableWriteId(long txnId, String dbName, String
> tableName) throws TException {
> return allocateTableWriteIdsBatch(Collections.singletonList(txnId),
> dbName, tableName).get(0).getWriteId();
> }
> {code}
> {code}
> java.lang.UnsupportedOperationException: null
> at java.util.AbstractList.set(AbstractList.java:132) ~[?:1.8.0]
> at java.util.AbstractList$ListItr.set(AbstractList.java:426) ~[?:1.8.0]
> at java.util.Collections.sort(Collections.java:170) ~[?:1.8.0]
> at
> org.apache.hadoop.hive.metastore.txn.TxnHandler.allocateTableWriteIds(TxnHandler.java:1523)
> ~[hive-standalone-metastore-server-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.allocate_table_write_ids(HiveMetaStore.java:7349)
> ~[hive-standalone-metastore-server-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20730) Do delete event filtering even if hive.acid.index is not there
[
https://issues.apache.org/jira/browse/HIVE-20730?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16681603#comment-16681603
]
Hive QA commented on HIVE-20730:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
34s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
9s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
17s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
52s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
28s{color} | {color:blue} common in master has 65 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
42s{color} | {color:blue} ql in master has 2315 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
5s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
9s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
15s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
37s{color} | {color:red} ql: The patch generated 1 new + 104 unchanged - 0
fixed = 105 total (was 104) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
35s{color} | {color:green} common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
44s{color} | {color:green} ql generated 0 new + 2314 unchanged - 1 fixed = 2314
total (was 2315) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
5s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 25m 22s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-14834/dev-support/hive-personality.sh
|
| git revision | master / 01cef92 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-14834/yetus/diff-checkstyle-ql.txt
|
| modules | C: common ql U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-14834/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Do delete event filtering even if hive.acid.index is not there
> --
>
> Key: HIVE-20730
> URL: https://issues.apache.org/jira/browse/HIVE-20730
> Project: Hive
> Issue Type: Improvement
> Components: Transactions
>Affects Versions: 4.0.0
>Reporter: Eugene Koifman
>Assignee: Saurabh Seth
>Priority: Major
> Attachments: HIVE-20730.2.patch, HIVE-20730.patch
>
>
> since HIVE-16812 {{VectorizedOrcAcidRowBatchReader}} filters delete events
> based on min/max ROW__ID in the split which relies on {{hive.acid.index}} to
> be in the ORC footer.
> There is no way to generate {{hive.acid.index}} from a plain query as in
> HIVE-20699 and so we need to make sure that we generate a SARG into
> delete_delta/bucket_x based on stripe stats
[jira] [Commented] (HIVE-19647) use bitvectors in IN operators
[
https://issues.apache.org/jira/browse/HIVE-19647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16681568#comment-16681568
]
Hive QA commented on HIVE-19647:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12947389/HIVE-19647.03.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 10 failed/errored test(s), 15537 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.llap.security.TestLlapSignerImpl.testSigning
(batchId=333)
org.apache.hadoop.hive.metastore.client.TestDefaultConstraint.addNoSuchTable[Remote]
(batchId=223)
org.apache.hadoop.hive.metastore.client.TestDefaultConstraint.createGetDrop[Remote]
(batchId=223)
org.apache.hadoop.hive.metastore.client.TestDefaultConstraint.createTableWithConstraintsPkInOtherCatalog[Remote]
(batchId=223)
org.apache.hadoop.hive.metastore.client.TestDefaultConstraint.createTableWithConstraintsPk[Remote]
(batchId=223)
org.apache.hadoop.hive.metastore.client.TestDefaultConstraint.doubleAddUniqueConstraint[Remote]
(batchId=223)
org.apache.hadoop.hive.metastore.client.TestDefaultConstraint.getNoSuchCatalog[Remote]
(batchId=223)
org.apache.hadoop.hive.metastore.client.TestDefaultConstraint.getNoSuchDb[Remote]
(batchId=223)
org.apache.hadoop.hive.metastore.client.TestDefaultConstraint.getNoSuchTable[Remote]
(batchId=223)
org.apache.hadoop.hive.metastore.client.TestDefaultConstraint.inOtherCatalog[Remote]
(batchId=223)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/14833/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/14833/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-14833/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 10 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12947389 - PreCommit-HIVE-Build
> use bitvectors in IN operators
> --
>
> Key: HIVE-19647
> URL: https://issues.apache.org/jira/browse/HIVE-19647
> Project: Hive
> Issue Type: Improvement
> Components: Statistics
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Attachments: HIVE-19647.01.patch, HIVE-19647.02.patch,
> HIVE-19647.03.patch, partial.patch
>
>
> for strings there is no easy way to filter things out ; even min/max is null;
> but the bitvectors could be used if they are available
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20440) Create better cache eviction policy for SmallTableCache
[ https://issues.apache.org/jira/browse/HIVE-20440?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Antal Sinkovits updated HIVE-20440: --- Attachment: HIVE-20440.11.patch > Create better cache eviction policy for SmallTableCache > --- > > Key: HIVE-20440 > URL: https://issues.apache.org/jira/browse/HIVE-20440 > Project: Hive > Issue Type: Improvement > Components: Spark >Reporter: Antal Sinkovits >Assignee: Antal Sinkovits >Priority: Major > Attachments: HIVE-20440.01.patch, HIVE-20440.02.patch, > HIVE-20440.03.patch, HIVE-20440.04.patch, HIVE-20440.05.patch, > HIVE-20440.06.patch, HIVE-20440.07.patch, HIVE-20440.08.patch, > HIVE-20440.09.patch, HIVE-20440.10.patch, HIVE-20440.11.patch > > > Enhance the SmallTableCache, to use guava cache with soft references, so that > we evict when there is memory pressure. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-18045) can VectorizedOrcAcidRowBatchReader be used all the time
[ https://issues.apache.org/jira/browse/HIVE-18045?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saurabh Seth reassigned HIVE-18045: --- Assignee: Saurabh Seth > can VectorizedOrcAcidRowBatchReader be used all the time > > > Key: HIVE-18045 > URL: https://issues.apache.org/jira/browse/HIVE-18045 > Project: Hive > Issue Type: Improvement > Components: Transactions >Reporter: Eugene Koifman >Assignee: Saurabh Seth >Priority: Blocker > > Can we use VectorizedOrcAcidRowBatchReader for non-vectorized queries? > It would just need a wrapper on top of it to turn VRBs into rows. > This would mean there is just 1 acid reader to maintain - not 2. > Would this be an issue for sorted reader/SMB support? -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19647) use bitvectors in IN operators
[
https://issues.apache.org/jira/browse/HIVE-19647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16681507#comment-16681507
]
Hive QA commented on HIVE-19647:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
29s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
2s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
15s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
52s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
30s{color} | {color:blue} common in master has 65 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
41s{color} | {color:blue} ql in master has 2315 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
6s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
9s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
14s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
37s{color} | {color:red} ql: The patch generated 5 new + 101 unchanged - 1
fixed = 106 total (was 102) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 3m
47s{color} | {color:red} ql generated 3 new + 2315 unchanged - 0 fixed = 2318
total (was 2315) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
6s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
12s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 25m 14s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:ql |
| | Found reliance on default encoding in
org.apache.hadoop.hive.ql.optimizer.stats.annotation.HiveMurmur3Adapter.murmur3(Object):in
org.apache.hadoop.hive.ql.optimizer.stats.annotation.HiveMurmur3Adapter.murmur3(Object):
String.getBytes() At HiveMurmur3Adapter.java:[line 68] |
| | org.apache.hadoop.hive.ql.plan.ColStatistics.getBitVectors() may expose
internal representation by returning ColStatistics.bitVectors At
ColStatistics.java:by returning ColStatistics.bitVectors At
ColStatistics.java:[line 206] |
| | org.apache.hadoop.hive.ql.plan.ColStatistics.setBitVectors(byte[]) may
expose internal representation by storing an externally mutable object into
ColStatistics.bitVectors At ColStatistics.java:by storing an externally
mutable object into ColStatistics.bitVectors At ColStatistics.java:[line 202] |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-14833/dev-support/hive-personality.sh
|
| git revision | master / dc3cbf2 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-14833/yetus/diff-checkstyle-ql.txt
|
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-14833/yetus/new-findbugs-ql.html
|
| modules | C: common ql U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-14833/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> use bitvectors in IN operators
> --
>
>
[jira] [Commented] (HIVE-20891) Call alter_partition in batch when dynamically loading partitions
[
https://issues.apache.org/jira/browse/HIVE-20891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16681450#comment-16681450
]
Hive QA commented on HIVE-20891:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12947386/HIVE-20891.01.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 8 failed/errored test(s), 15527 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.ql.TestTxnExIm.testExportPart (batchId=305)
org.apache.hadoop.hive.ql.TestTxnExIm.testExportPartPartial (batchId=305)
org.apache.hadoop.hive.ql.TestTxnExIm.testExportPartPartial2 (batchId=305)
org.apache.hadoop.hive.ql.TestTxnExIm.testExportPartPartial3 (batchId=305)
org.apache.hadoop.hive.ql.TestTxnExIm.testImportPartitionedCreate (batchId=305)
org.apache.hadoop.hive.ql.TestTxnExIm.testImportPartitionedCreate2 (batchId=305)
org.apache.hadoop.hive.ql.parse.TestReplIncrementalLoadAcidTablesWithJsonMessage.testAcidTableIncrementalReplication
(batchId=250)
org.apache.hadoop.hive.ql.parse.TestReplicationScenariosIncrementalLoadAcidTables.testAcidTableIncrementalReplication
(batchId=248)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/14832/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/14832/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-14832/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 8 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12947386 - PreCommit-HIVE-Build
> Call alter_partition in batch when dynamically loading partitions
> -
>
> Key: HIVE-20891
> URL: https://issues.apache.org/jira/browse/HIVE-20891
> Project: Hive
> Issue Type: Improvement
> Components: Hive
>Affects Versions: 4.0.0
>Reporter: Laszlo Pinter
>Assignee: Laszlo Pinter
>Priority: Major
> Attachments: HIVE-20891.01.patch
>
>
> When dynamically loading partitions, the setStatsPropAndAlterPartition() is
> called for each partition one by one, resulting in unnecessary calls to the
> metastore client. This whole logic can be changed to just one call.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20795) Rename CompileLockTest.java to follow the naming convention for the tests.
[
https://issues.apache.org/jira/browse/HIVE-20795?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Vary updated HIVE-20795:
--
Resolution: Fixed
Fix Version/s: 4.0.0
Status: Resolved (was: Patch Available)
Pushed to master.
Thanks for the patch [~dkuzmenko]!
> Rename CompileLockTest.java to follow the naming convention for the tests.
> --
>
> Key: HIVE-20795
> URL: https://issues.apache.org/jira/browse/HIVE-20795
> Project: Hive
> Issue Type: Bug
> Components: Test
>Reporter: Denys Kuzmenko
>Assignee: Denys Kuzmenko
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-20795.1.patch
>
>
> Tests without "Test" prefix are ignored by
> UnitTestPropertiesParser.generateFullTestSet()
> -> if (className.startsWith("Test") && !className.contains("$")) \{ }
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20891) Call alter_partition in batch when dynamically loading partitions
[
https://issues.apache.org/jira/browse/HIVE-20891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16681410#comment-16681410
]
Hive QA commented on HIVE-20891:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
1s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
34s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
58s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
37s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
42s{color} | {color:blue} ql in master has 2315 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
53s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
58s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
38s{color} | {color:red} ql: The patch generated 2 new + 231 unchanged - 0
fixed = 233 total (was 231) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 22m 3s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-14832/dev-support/hive-personality.sh
|
| git revision | master / 961224c |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-14832/yetus/diff-checkstyle-ql.txt
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-14832/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Call alter_partition in batch when dynamically loading partitions
> -
>
> Key: HIVE-20891
> URL: https://issues.apache.org/jira/browse/HIVE-20891
> Project: Hive
> Issue Type: Improvement
> Components: Hive
>Affects Versions: 4.0.0
>Reporter: Laszlo Pinter
>Assignee: Laszlo Pinter
>Priority: Major
> Attachments: HIVE-20891.01.patch
>
>
> When dynamically loading partitions, the setStatsPropAndAlterPartition() is
> called for each partition one by one, resulting in unnecessary calls to the
> metastore client. This whole logic can be changed to just one call.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20898) For time related functions arguments may not be casted to a non nullable type
[
https://issues.apache.org/jira/browse/HIVE-20898?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zoltan Haindrich updated HIVE-20898:
Attachment: HIVE-20898.01.patch
> For time related functions arguments may not be casted to a non nullable type
> -
>
> Key: HIVE-20898
> URL: https://issues.apache.org/jira/browse/HIVE-20898
> Project: Hive
> Issue Type: Bug
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Attachments: HIVE-20898.01.patch
>
>
> {code}
> create table t (a string); insert into t values (null),('1988-11-11');
> set hive.cbo.enable=true;
> select 'expected 1 (second)', count(1) from t where second(a) is null;
> {code}
> this may only cause trouble if Calcite is exploiting the datatype
> nullability; it will be need after CALCITE-2469 (1.18.0)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20898) For time related functions arguments may not be casted to a non nullable type
[
https://issues.apache.org/jira/browse/HIVE-20898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16681404#comment-16681404
]
Zoltan Haindrich commented on HIVE-20898:
-
checked with a locally built Calcite 1.18.0-SNAPSHOT
[~jcamachorodriguez] Could you please take a look?
> For time related functions arguments may not be casted to a non nullable type
> -
>
> Key: HIVE-20898
> URL: https://issues.apache.org/jira/browse/HIVE-20898
> Project: Hive
> Issue Type: Bug
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Attachments: HIVE-20898.01.patch
>
>
> {code}
> create table t (a string); insert into t values (null),('1988-11-11');
> set hive.cbo.enable=true;
> select 'expected 1 (second)', count(1) from t where second(a) is null;
> {code}
> this may only cause trouble if Calcite is exploiting the datatype
> nullability; it will be need after CALCITE-2469 (1.18.0)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20898) For time related functions arguments may not be casted to a non nullable type
[
https://issues.apache.org/jira/browse/HIVE-20898?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zoltan Haindrich updated HIVE-20898:
Status: Patch Available (was: Open)
> For time related functions arguments may not be casted to a non nullable type
> -
>
> Key: HIVE-20898
> URL: https://issues.apache.org/jira/browse/HIVE-20898
> Project: Hive
> Issue Type: Bug
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Attachments: HIVE-20898.01.patch
>
>
> {code}
> create table t (a string); insert into t values (null),('1988-11-11');
> set hive.cbo.enable=true;
> select 'expected 1 (second)', count(1) from t where second(a) is null;
> {code}
> this may only cause trouble if Calcite is exploiting the datatype
> nullability; it will be need after CALCITE-2469 (1.18.0)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20898) For time related functions arguments may not be casted to a non nullable type
[
https://issues.apache.org/jira/browse/HIVE-20898?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zoltan Haindrich updated HIVE-20898:
Description:
{code}
create table t (a string); insert into t values (null),('1988-11-11');
set hive.cbo.enable=true;
select 'expected 1 (second)', count(1) from t where second(a) is null;
{code}
this may only cause trouble if Calcite is exploiting the datatype nullability;
it will be need after CALCITE-2469 (1.18.0)
was:
{code}
create table t (a string); insert into t values (null),('1988-11-11');
set hive.cbo.enable=true;
select 'expected 1 (second)', count(1) from t where second(a) is null;
{code}
this may only cause trouble if Calcite is exploiting the datatype nullability.
> For time related functions arguments may not be casted to a non nullable type
> -
>
> Key: HIVE-20898
> URL: https://issues.apache.org/jira/browse/HIVE-20898
> Project: Hive
> Issue Type: Bug
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
>
> {code}
> create table t (a string); insert into t values (null),('1988-11-11');
> set hive.cbo.enable=true;
> select 'expected 1 (second)', count(1) from t where second(a) is null;
> {code}
> this may only cause trouble if Calcite is exploiting the datatype
> nullability; it will be need after CALCITE-2469 (1.18.0)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-14557) Nullpointer When both SkewJoin and Mapjoin Enabled
[
https://issues.apache.org/jira/browse/HIVE-14557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16681401#comment-16681401
]
Rui Li commented on HIVE-14557:
---
Sorry about the late response. Attach an updated patch.
> Nullpointer When both SkewJoin and Mapjoin Enabled
> ---
>
> Key: HIVE-14557
> URL: https://issues.apache.org/jira/browse/HIVE-14557
> Project: Hive
> Issue Type: Bug
> Components: Physical Optimizer
>Affects Versions: 1.1.0, 2.1.0
>Reporter: Nemon Lou
>Assignee: Rui Li
>Priority: Major
> Attachments: HIVE-14557.2.patch, HIVE-14557.3.patch, HIVE-14557.patch
>
>
> The following sql failed with return code 2 on mr.
> {noformat}
> create table a(id int,id1 int);
> create table b(id int,id1 int);
> create table c(id int,id1 int);
> set hive.optimize.skewjoin=true;
> select a.id,b.id,c.id1 from a,b,c where a.id=b.id and a.id1=c.id1;
> {noformat}
> Error log as follows:
> {noformat}
> 2016-08-17 21:13:42,081 INFO [main]
> org.apache.hadoop.hive.ql.exec.mr.ExecMapper:
> Id =0
>
> Id =21
>
> Id =28
>
> Id =16
>
> <\Children>
> Id = 28 null<\Parent>
> <\FS>
> <\Children>
> Id = 21 nullId = 33
> Id =33
> null
> <\Children>
> <\Parent>
> <\HASHTABLEDUMMY><\Parent>
> <\MAPJOIN>
> <\Children>
> Id = 0 null<\Parent>
> <\TS>
> <\Children>
> <\Parent>
> <\MAP>
> 2016-08-17 21:13:42,084 INFO [main]
> org.apache.hadoop.hive.ql.exec.TableScanOperator: Initializing operator TS[21]
> 2016-08-17 21:13:42,084 INFO [main]
> org.apache.hadoop.hive.ql.exec.mr.ExecMapper: Initializing dummy operator
> 2016-08-17 21:13:42,086 INFO [main]
> org.apache.hadoop.hive.ql.exec.MapOperator: DESERIALIZE_ERRORS:0,
> RECORDS_IN:0,
> 2016-08-17 21:13:42,087 ERROR [main]
> org.apache.hadoop.hive.ql.exec.mr.ExecMapper: Hit error while closing
> operators - failing tree
> 2016-08-17 21:13:42,088 WARN [main] org.apache.hadoop.mapred.YarnChild:
> Exception running child : java.lang.RuntimeException: Hive Runtime Error
> while closing operators
> at
> org.apache.hadoop.hive.ql.exec.mr.ExecMapper.close(ExecMapper.java:207)
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:61)
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
> Caused by: java.lang.NullPointerException
> at
> org.apache.hadoop.hive.ql.exec.MapJoinOperator.closeOp(MapJoinOperator.java:474)
> at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:682)
> at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:696)
> at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:696)
> at
> org.apache.hadoop.hive.ql.exec.mr.ExecMapper.close(ExecMapper.java:189)
> ... 8 more
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-14557) Nullpointer When both SkewJoin and Mapjoin Enabled
[
https://issues.apache.org/jira/browse/HIVE-14557?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rui Li updated HIVE-14557:
--
Attachment: HIVE-14557.3.patch
> Nullpointer When both SkewJoin and Mapjoin Enabled
> ---
>
> Key: HIVE-14557
> URL: https://issues.apache.org/jira/browse/HIVE-14557
> Project: Hive
> Issue Type: Bug
> Components: Physical Optimizer
>Affects Versions: 1.1.0, 2.1.0
>Reporter: Nemon Lou
>Assignee: Rui Li
>Priority: Major
> Attachments: HIVE-14557.2.patch, HIVE-14557.3.patch, HIVE-14557.patch
>
>
> The following sql failed with return code 2 on mr.
> {noformat}
> create table a(id int,id1 int);
> create table b(id int,id1 int);
> create table c(id int,id1 int);
> set hive.optimize.skewjoin=true;
> select a.id,b.id,c.id1 from a,b,c where a.id=b.id and a.id1=c.id1;
> {noformat}
> Error log as follows:
> {noformat}
> 2016-08-17 21:13:42,081 INFO [main]
> org.apache.hadoop.hive.ql.exec.mr.ExecMapper:
> Id =0
>
> Id =21
>
> Id =28
>
> Id =16
>
> <\Children>
> Id = 28 null<\Parent>
> <\FS>
> <\Children>
> Id = 21 nullId = 33
> Id =33
> null
> <\Children>
> <\Parent>
> <\HASHTABLEDUMMY><\Parent>
> <\MAPJOIN>
> <\Children>
> Id = 0 null<\Parent>
> <\TS>
> <\Children>
> <\Parent>
> <\MAP>
> 2016-08-17 21:13:42,084 INFO [main]
> org.apache.hadoop.hive.ql.exec.TableScanOperator: Initializing operator TS[21]
> 2016-08-17 21:13:42,084 INFO [main]
> org.apache.hadoop.hive.ql.exec.mr.ExecMapper: Initializing dummy operator
> 2016-08-17 21:13:42,086 INFO [main]
> org.apache.hadoop.hive.ql.exec.MapOperator: DESERIALIZE_ERRORS:0,
> RECORDS_IN:0,
> 2016-08-17 21:13:42,087 ERROR [main]
> org.apache.hadoop.hive.ql.exec.mr.ExecMapper: Hit error while closing
> operators - failing tree
> 2016-08-17 21:13:42,088 WARN [main] org.apache.hadoop.mapred.YarnChild:
> Exception running child : java.lang.RuntimeException: Hive Runtime Error
> while closing operators
> at
> org.apache.hadoop.hive.ql.exec.mr.ExecMapper.close(ExecMapper.java:207)
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:61)
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
> Caused by: java.lang.NullPointerException
> at
> org.apache.hadoop.hive.ql.exec.MapJoinOperator.closeOp(MapJoinOperator.java:474)
> at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:682)
> at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:696)
> at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:696)
> at
> org.apache.hadoop.hive.ql.exec.mr.ExecMapper.close(ExecMapper.java:189)
> ... 8 more
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20881) Constant propagation oversimplifies projections
[
https://issues.apache.org/jira/browse/HIVE-20881?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16681393#comment-16681393
]
Hive QA commented on HIVE-20881:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12947371/HIVE-20881.01.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 15528 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/14831/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/14831/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-14831/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12947371 - PreCommit-HIVE-Build
> Constant propagation oversimplifies projections
> ---
>
> Key: HIVE-20881
> URL: https://issues.apache.org/jira/browse/HIVE-20881
> Project: Hive
> Issue Type: Bug
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Attachments: HIVE-20881.01.patch
>
>
> {code:java}
> create table cx2(bool1 boolean);
> insert into cx2 values (true),(false),(null);
> set hive.cbo.enable=true;
> select bool1 IS TRUE OR (cast(NULL as boolean) AND bool1 IS NOT TRUE AND
> bool1 IS NOT FALSE) from cx2;
> ++
> | _c0 |
> ++
> | true |
> | false |
> | NULL |
> ++
> set hive.cbo.enable=false;
> select bool1 IS TRUE OR (cast(NULL as boolean) AND bool1 IS NOT TRUE AND
> bool1 IS NOT FALSE) from cx2;
> +---+
> | _c0 |
> +---+
> | true |
> | NULL |
> | NULL |
> +---+
> {code}
> from explain it seems the expression was simplified to: {{(_col0 is true or
> null)}}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (HIVE-20898) For time related functions arguments may not be casted to a non nullable type
[
https://issues.apache.org/jira/browse/HIVE-20898?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zoltan Haindrich reassigned HIVE-20898:
---
> For time related functions arguments may not be casted to a non nullable type
> -
>
> Key: HIVE-20898
> URL: https://issues.apache.org/jira/browse/HIVE-20898
> Project: Hive
> Issue Type: Bug
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
>
> {code}
> create table t (a string); insert into t values (null),('1988-11-11');
> set hive.cbo.enable=true;
> select 'expected 1 (second)', count(1) from t where second(a) is null;
> {code}
> this may only cause trouble if Calcite is exploiting the datatype nullability.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-17477) PidFilePatternConverter not rolling log according to their pid
[
https://issues.apache.org/jira/browse/HIVE-17477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16681343#comment-16681343
]
Bohdan Chupika commented on HIVE-17477:
---
Reattached the patch for jenkins rebuild
> PidFilePatternConverter not rolling log according to their pid
> ---
>
> Key: HIVE-17477
> URL: https://issues.apache.org/jira/browse/HIVE-17477
> Project: Hive
> Issue Type: Bug
> Components: Logging
>Reporter: Vlad Gudikov
>Assignee: Bohdan Chupika
>Priority: Major
> Attachments: HIVE-17477.1.patch, HIVE-17477.2.patch,
> HIVE-17477.3.patch, HIVE-17477.4.patch, HIVE-17477.5.patch, HIVE-17477.6.patch
>
>
> We can use pid in th filePattern:
> appender.DRFA.filePattern =
> ${sys:hive.log.dir}/${sys:hive.log.file}.%d{-MM-dd}.%pid
> But when its time roll the logs at the end of the RollingAppender just
> renames hive.log by applying the pattern described above and keeps logs that
> are not related to the process this log was named by. The issue is that all
> processes are writing to the same log and we cannot separate them by pid.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-17477) PidFilePatternConverter not rolling log according to their pid
[
https://issues.apache.org/jira/browse/HIVE-17477?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bohdan Chupika updated HIVE-17477:
--
Attachment: HIVE-17477.6.patch
> PidFilePatternConverter not rolling log according to their pid
> ---
>
> Key: HIVE-17477
> URL: https://issues.apache.org/jira/browse/HIVE-17477
> Project: Hive
> Issue Type: Bug
> Components: Logging
>Reporter: Vlad Gudikov
>Assignee: Bohdan Chupika
>Priority: Major
> Attachments: HIVE-17477.1.patch, HIVE-17477.2.patch,
> HIVE-17477.3.patch, HIVE-17477.4.patch, HIVE-17477.5.patch, HIVE-17477.6.patch
>
>
> We can use pid in th filePattern:
> appender.DRFA.filePattern =
> ${sys:hive.log.dir}/${sys:hive.log.file}.%d{-MM-dd}.%pid
> But when its time roll the logs at the end of the RollingAppender just
> renames hive.log by applying the pattern described above and keeps logs that
> are not related to the process this log was named by. The issue is that all
> processes are writing to the same log and we cannot separate them by pid.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20881) Constant propagation oversimplifies projections
[
https://issues.apache.org/jira/browse/HIVE-20881?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16681342#comment-16681342
]
Hive QA commented on HIVE-20881:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
32s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
0s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
35s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
42s{color} | {color:blue} ql in master has 2315 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
54s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
37s{color} | {color:green} ql: The patch generated 0 new + 94 unchanged - 1
fixed = 94 total (was 95) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 22m 8s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-14831/dev-support/hive-personality.sh
|
| git revision | master / 961224c |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-14831/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Constant propagation oversimplifies projections
> ---
>
> Key: HIVE-20881
> URL: https://issues.apache.org/jira/browse/HIVE-20881
> Project: Hive
> Issue Type: Bug
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Attachments: HIVE-20881.01.patch
>
>
> {code:java}
> create table cx2(bool1 boolean);
> insert into cx2 values (true),(false),(null);
> set hive.cbo.enable=true;
> select bool1 IS TRUE OR (cast(NULL as boolean) AND bool1 IS NOT TRUE AND
> bool1 IS NOT FALSE) from cx2;
> ++
> | _c0 |
> ++
> | true |
> | false |
> | NULL |
> ++
> set hive.cbo.enable=false;
> select bool1 IS TRUE OR (cast(NULL as boolean) AND bool1 IS NOT TRUE AND
> bool1 IS NOT FALSE) from cx2;
> +---+
> | _c0 |
> +---+
> | true |
> | NULL |
> | NULL |
> +---+
> {code}
> from explain it seems the expression was simplified to: {{(_col0 is true or
> null)}}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18661) CachedStore: Use metastore notification log events to update cache
[ https://issues.apache.org/jira/browse/HIVE-18661?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] mahesh kumar behera updated HIVE-18661: --- Status: Patch Available (was: Open) # Removed cache invalidation during raw store operation if event based cache update is enabled. 2. Added check to read stats directly from raw store if event based update is enabled as events are not generated for stats update. 3. Added code to trigger cache update during commit txn time. This may cause invalid cache read if same transaction tried to read the value from cache. 4. Need to check how to handle config change to disable/enable cache update using events during run time after prewarm is done. 5. For catalog operation cache will not be used , if event based update is used. [~thejas] [~daijy] Can you please review the patch > CachedStore: Use metastore notification log events to update cache > -- > > Key: HIVE-18661 > URL: https://issues.apache.org/jira/browse/HIVE-18661 > Project: Hive > Issue Type: Sub-task > Components: Metastore >Reporter: Vaibhav Gumashta >Assignee: mahesh kumar behera >Priority: Major > Labels: pull-request-available > Attachments: HIVE-18661.02.patch, HIVE-18661.03.patch, > HIVE-18661.04.patch, HIVE-18661.05.patch > > > Currently, a background thread updates the entire cache which is pretty > inefficient. We capture the updates to metadata in NOTIFICATION_LOG table > which is getting used in the Replication work. We should have the background > thread apply these notifications to incrementally update the cache. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18661) CachedStore: Use metastore notification log events to update cache
[ https://issues.apache.org/jira/browse/HIVE-18661?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] mahesh kumar behera updated HIVE-18661: --- Attachment: HIVE-18661.05.patch > CachedStore: Use metastore notification log events to update cache > -- > > Key: HIVE-18661 > URL: https://issues.apache.org/jira/browse/HIVE-18661 > Project: Hive > Issue Type: Sub-task > Components: Metastore >Reporter: Vaibhav Gumashta >Assignee: mahesh kumar behera >Priority: Major > Labels: pull-request-available > Attachments: HIVE-18661.02.patch, HIVE-18661.03.patch, > HIVE-18661.04.patch, HIVE-18661.05.patch > > > Currently, a background thread updates the entire cache which is pretty > inefficient. We capture the updates to metadata in NOTIFICATION_LOG table > which is getting used in the Replication work. We should have the background > thread apply these notifications to incrementally update the cache. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-20897) TestJdbcDriver2#testSelectExecAsync2 fails with result set not present error
[ https://issues.apache.org/jira/browse/HIVE-20897?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] mahesh kumar behera reassigned HIVE-20897: -- > TestJdbcDriver2#testSelectExecAsync2 fails with result set not present error > > > Key: HIVE-20897 > URL: https://issues.apache.org/jira/browse/HIVE-20897 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: mahesh kumar behera >Assignee: mahesh kumar behera >Priority: Major > > if async prepare is enabled, control will be returned to the client before > driver could set of the query has a result set or not. But in current code, > while generating the response for the query, it is not checked if the result > set field is set or not. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20897) TestJdbcDriver2#testSelectExecAsync2 fails with result set not present error
[ https://issues.apache.org/jira/browse/HIVE-20897?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] mahesh kumar behera updated HIVE-20897: --- Attachment: HIVE-20897.01.patch > TestJdbcDriver2#testSelectExecAsync2 fails with result set not present error > > > Key: HIVE-20897 > URL: https://issues.apache.org/jira/browse/HIVE-20897 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: mahesh kumar behera >Assignee: mahesh kumar behera >Priority: Major > Attachments: HIVE-20897.01.patch > > > if async prepare is enabled, control will be returned to the client before > driver could set of the query has a result set or not. But in current code, > while generating the response for the query, it is not checked if the result > set field is set or not. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20897) TestJdbcDriver2#testSelectExecAsync2 fails with result set not present error
[ https://issues.apache.org/jira/browse/HIVE-20897?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] mahesh kumar behera updated HIVE-20897: --- Status: Patch Available (was: Open) > TestJdbcDriver2#testSelectExecAsync2 fails with result set not present error > > > Key: HIVE-20897 > URL: https://issues.apache.org/jira/browse/HIVE-20897 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: mahesh kumar behera >Assignee: mahesh kumar behera >Priority: Major > Attachments: HIVE-20897.01.patch > > > if async prepare is enabled, control will be returned to the client before > driver could set of the query has a result set or not. But in current code, > while generating the response for the query, it is not checked if the result > set field is set or not. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20734) Beeline: When beeline-site.xml is and hive CLI redirects to beeline, it should use the system username/dummy password instead of prompting for one
[ https://issues.apache.org/jira/browse/HIVE-20734?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16681302#comment-16681302 ] mahesh kumar behera commented on HIVE-20734: [~thejas] can please commit it > Beeline: When beeline-site.xml is and hive CLI redirects to beeline, it > should use the system username/dummy password instead of prompting for one > -- > > Key: HIVE-20734 > URL: https://issues.apache.org/jira/browse/HIVE-20734 > Project: Hive > Issue Type: Bug > Components: Beeline >Affects Versions: 3.1.0 >Reporter: Vaibhav Gumashta >Assignee: mahesh kumar behera >Priority: Major > Attachments: HIVE-20734.1.patch, HIVE-20734.2.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20850) Push case conditional from projections to dimension tables if possible
[
https://issues.apache.org/jira/browse/HIVE-20850?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16681298#comment-16681298
]
Hive QA commented on HIVE-20850:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12947547/HIVE-20850.06.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 10 failed/errored test(s), 15527 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[unionDistinct_1]
(batchId=154)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_smb_ptf]
(batchId=164)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[constprog_semijoin]
(batchId=191)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[dynamic_rdd_cache]
(batchId=190)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_explainuser_1]
(batchId=190)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_use_ts_stats_for_mapjoin]
(batchId=190)
org.apache.hive.jdbc.TestTriggersMoveWorkloadManager.testTriggerMoveAndKill
(batchId=258)
org.apache.hive.jdbc.TestTriggersMoveWorkloadManager.testTriggerMoveBackKill
(batchId=258)
org.apache.hive.jdbc.TestTriggersTezSessionPoolManager.testTriggerCustomReadOps
(batchId=258)
org.apache.hive.jdbc.TestTriggersTezSessionPoolManager.testTriggerDagTotalTasks
(batchId=258)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/14830/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/14830/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-14830/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 10 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12947547 - PreCommit-HIVE-Build
> Push case conditional from projections to dimension tables if possible
> --
>
> Key: HIVE-20850
> URL: https://issues.apache.org/jira/browse/HIVE-20850
> Project: Hive
> Issue Type: Improvement
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Attachments: HIVE-20850.01.patch, HIVE-20850.01.patch,
> HIVE-20850.01.patch, HIVE-20850.02.patch, HIVE-20850.03.patch,
> HIVE-20850.04.patch, HIVE-20850.05.patch, HIVE-20850.06.patch
>
>
> noticed by [~gopalv]: If there is a project which could be only evaluated
> after the join; but the condition references only a single column from a
> small dimension table; hive will end up evaluating the same thing over and
> over again...
> {code}
> explain
> select s_store_name, s_store_id,
> sum(case when (d_day_name='Sunday') then ss_sales_price else null
> end) sun_sales,
> sum(case when (d_day_name='Monday') then ss_sales_price else null
> end) mon_sales,
> sum(case when (d_day_name='Tuesday') then ss_sales_price else null
> end) tue_sales,
> sum(case when (d_day_name='Wednesday') then ss_sales_price else null
> end) wed_sales,
> sum(case when (d_day_name='Thursday') then ss_sales_price else null
> end) thu_sales,
> sum(case when (d_day_name='Friday') then ss_sales_price else null
> end) fri_sales,
> sum(case when (d_day_name='Saturday') then ss_sales_price else null
> end) sat_sales
> from date_dim, store_sales, store
> where d_date_sk = ss_sold_date_sk and
>s_store_sk = ss_store_sk and
>s_gmt_offset = -6 and
>d_year = 1998
> group by s_store_name, s_store_id
> order by s_store_name,
> s_store_id,sun_sales,mon_sales,tue_sales,wed_sales,thu_sales,fri_sales,sat_sales
> limit 100;
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20850) Push case conditional from projections to dimension tables if possible
[
https://issues.apache.org/jira/browse/HIVE-20850?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16681244#comment-16681244
]
Hive QA commented on HIVE-20850:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
38s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
35s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
35s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
1s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
26s{color} | {color:blue} accumulo-handler in master has 21 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
27s{color} | {color:blue} hbase-handler in master has 15 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
38s{color} | {color:blue} ql in master has 2315 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
15s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
9s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
15s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
12s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 29m 28s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-14830/dev-support/hive-personality.sh
|
| git revision | master / 961224c |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: accumulo-handler hbase-handler ql U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-14830/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Push case conditional from projections to dimension tables if possible
> --
>
> Key: HIVE-20850
> URL: https://issues.apache.org/jira/browse/HIVE-20850
> Project: Hive
> Issue Type: Improvement
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Attachments: HIVE-20850.01.patch, HIVE-20850.01.patch,
> HIVE-20850.01.patch, HIVE-20850.02.patch, HIVE-20850.03.patch,
> HIVE-20850.04.patch, HIVE-20850.05.patch, HIVE-20850.06.patch
>
>
> noticed by [~gopalv]: If there is a project which could be only evaluated
> after the join; but the condition references only a single column from a
> small dimension table; hive will end up evaluating the same thing over and
> over again...
> {code}
> explain
> select s_store_name, s_store_id,
> sum(case when (d_day_name='Sunday') then ss_sales_price else null
> end) sun_sales,
> sum(case when
[jira] [Commented] (HIVE-20823) Make Compactor run in a transaction
[
https://issues.apache.org/jira/browse/HIVE-20823?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16681203#comment-16681203
]
Abhishek Somani commented on HIVE-20823:
[~ekoifman] is this expected to solve
https://issues.apache.org/jira/browse/HIVE-20392 as well?
> Make Compactor run in a transaction
> ---
>
> Key: HIVE-20823
> URL: https://issues.apache.org/jira/browse/HIVE-20823
> Project: Hive
> Issue Type: Improvement
> Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
>Priority: Major
> Attachments: HIVE-20823.01.patch, HIVE-20823.03.patch,
> HIVE-20823.04.patch, HIVE-20823.05.patch, HIVE-20823.07.patch
>
>
> Have compactor open a transaction and run the job in that transaction.
> # make compactor produced base/delta include this txn id in the folder name,
> e.g. base_7_c17 where 17 is the txnid.
> # add {{CQ_TXN_ID bigint}} to COMPACTION_QUEUE and COMPLETED_COMPACTIONS to
> record this txn id
> # make sure {{AcidUtils.getAcidState()}} pays attention to this transaction
> on read and ignores this dir if this txn id is not committed in the current
> snapshot
> ## this means not only validWriteIdList but ValidTxnIdList should be passed
> along in config (if it isn't yet)
> # once this is done, {{CompactorMR.createCompactorMarker()}} can be
> eliminated and {{AcidUtils.isValidBase}} modified accordingly
> # modify Cleaner so that it doesn't clean old files until new file is visible
> to all readers
> #
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20682) Async query execution can potentially fail if shared sessionHive is closed by master thread.
[
https://issues.apache.org/jira/browse/HIVE-20682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16681200#comment-16681200
]
Hive QA commented on HIVE-20682:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12947357/HIVE-20682.06.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 15527 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_groupby_reduce]
(batchId=61)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/14829/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/14829/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-14829/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12947357 - PreCommit-HIVE-Build
> Async query execution can potentially fail if shared sessionHive is closed by
> master thread.
>
>
> Key: HIVE-20682
> URL: https://issues.apache.org/jira/browse/HIVE-20682
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 3.1.0, 4.0.0
>Reporter: Sankar Hariappan
>Assignee: Sankar Hariappan
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-20682.01.patch, HIVE-20682.02.patch,
> HIVE-20682.03.patch, HIVE-20682.04.patch, HIVE-20682.05.patch,
> HIVE-20682.06.patch
>
>
> *Problem description:*
> The master thread initializes the *sessionHive* object in *HiveSessionImpl*
> class when we open a new session for a client connection and by default all
> queries from this connection shares the same sessionHive object.
> If the master thread executes a *synchronous* query, it closes the
> sessionHive object (referred via thread local hiveDb) if
> {{Hive.isCompatible}} returns false and sets new Hive object in thread local
> HiveDb but doesn't change the sessionHive object in the session. Whereas,
> *asynchronous* query execution via async threads never closes the sessionHive
> object and it just creates a new one if needed and sets it as their thread
> local hiveDb.
> So, the problem can happen in the case where an *asynchronous* query is being
> executed by async threads refers to sessionHive object and the master thread
> receives a *synchronous* query that closes the same sessionHive object.
> Also, each query execution overwrites the thread local hiveDb object to
> sessionHive object which potentially leaks a metastore connection if the
> previous synchronous query execution re-created the Hive object.
> *Possible Fix:*
> The *sessionHive* object could be shared my multiple threads and so it
> shouldn't be allowed to be closed by any query execution threads when they
> re-create the Hive object due to changes in Hive configurations. But the Hive
> objects created by query execution threads should be closed when the thread
> exits.
> So, it is proposed to have an *isAllowClose* flag (default: *true*) in Hive
> object which should be set to *false* for *sessionHive* and would be
> forcefully closed when the session is closed or released.
> Also, when we reset *sessionHive* object with new one due to changes in
> *sessionConf*, the old one should be closed when no async thread is referring
> to it. This can be done using "*finalize*" method of Hive object where we can
> close HMS connection when Hive object is garbage collected.
> cc [~pvary]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20682) Async query execution can potentially fail if shared sessionHive is closed by master thread.
[
https://issues.apache.org/jira/browse/HIVE-20682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16681136#comment-16681136
]
Hive QA commented on HIVE-20682:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
39s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
7s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
57s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
9s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
36s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
43s{color} | {color:blue} ql in master has 2315 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
36s{color} | {color:blue} service in master has 48 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
32s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
9s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
19s{color} | {color:green} The patch hive-unit passed checkstyle {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
37s{color} | {color:green} ql: The patch generated 0 new + 218 unchanged - 2
fixed = 218 total (was 220) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
12s{color} | {color:green} The patch service passed checkstyle {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
28s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 30m 43s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-14829/dev-support/hive-personality.sh
|
| git revision | master / 961224c |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: itests/hive-unit ql service U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-14829/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Async query execution can potentially fail if shared sessionHive is closed by
> master thread.
>
>
> Key: HIVE-20682
> URL: https://issues.apache.org/jira/browse/HIVE-20682
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 3.1.0, 4.0.0
>Reporter: Sankar Hariappan
>Assignee: Sankar Hariappan
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-20682.01.patch, HIVE-20682.02.patch,
> HIVE-20682.03.patch, HIVE-20682.04.patch,
[jira] [Commented] (HIVE-20893) BloomK Filter probing method is not thread safe
[ https://issues.apache.org/jira/browse/HIVE-20893?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16681121#comment-16681121 ] Clint Wylie commented on HIVE-20893: Ah, this one works, thanks! I'll apply these changes to the version I've copied into Druid until this is properly released. > BloomK Filter probing method is not thread safe > --- > > Key: HIVE-20893 > URL: https://issues.apache.org/jira/browse/HIVE-20893 > Project: Hive > Issue Type: Bug > Components: storage-api >Reporter: slim bouguerra >Priority: Major > Attachments: HIVE-20893.WIP.patch > > > As far i can tell this is not an issue for Hive yet (most of the usage of > probing seems to be done by one thread at a time) but it is an issue of other > users like Druid as per the following > issue.[https://github.com/apache/incubator-druid/issues/6546] > The fix is proposed by the author of > [https://github.com/apache/incubator-druid/pull/6584] is to make couple of > local fields as ThreadLocals. > Idea looks good to me and doesn't have any perf drawbacks. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20801) ACID: Allow DbTxnManager to ignore non-ACID table locking
[
https://issues.apache.org/jira/browse/HIVE-20801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16681079#comment-16681079
]
Hive QA commented on HIVE-20801:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12947348/HIVE-20801.2.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 9 failed/errored test(s), 15528 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[show_conf] (batchId=27)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[timestamptz_2]
(batchId=85)
org.apache.hadoop.hive.metastore.TestObjectStore.testDirectSQLDropParitionsCleanup
(batchId=231)
org.apache.hadoop.hive.metastore.TestObjectStore.testDirectSQLDropPartitionsCacheCrossSession
(batchId=231)
org.apache.hadoop.hive.metastore.TestObjectStore.testDirectSqlErrorMetrics
(batchId=231)
org.apache.hadoop.hive.metastore.TestObjectStore.testMaxEventResponse
(batchId=231)
org.apache.hadoop.hive.metastore.TestObjectStore.testPartitionOps (batchId=231)
org.apache.hadoop.hive.metastore.TestObjectStore.testQueryCloseOnError
(batchId=231)
org.apache.hadoop.hive.metastore.TestObjectStore.testRoleOps (batchId=231)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/14828/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/14828/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-14828/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 9 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12947348 - PreCommit-HIVE-Build
> ACID: Allow DbTxnManager to ignore non-ACID table locking
> -
>
> Key: HIVE-20801
> URL: https://issues.apache.org/jira/browse/HIVE-20801
> Project: Hive
> Issue Type: Bug
> Components: Locking, Transactions
>Affects Versions: 4.0.0
>Reporter: Gopal V
>Assignee: Gopal V
>Priority: Major
> Labels: Branch3Candidate, TODOC
> Attachments: HIVE-20801.1.patch, HIVE-20801.2.patch
>
>
> Enabling ACIDv1 on a cluster produces a central locking bottleneck for all
> table types, which is not always the intention.
> The Hive locking for non-acid tables are advisory (i.e a client can
> write/read without locking), which means that the implementation does not
> offer strong consistency despite the lock manager consuming resources
> centrally.
> Disabling this lock acquisition would improve the performance of non-ACID
> tables co-existing with a globally configured DbTxnManager implementation.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20850) Push case conditional from projections to dimension tables if possible
[
https://issues.apache.org/jira/browse/HIVE-20850?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zoltan Haindrich updated HIVE-20850:
Attachment: HIVE-20850.06.patch
> Push case conditional from projections to dimension tables if possible
> --
>
> Key: HIVE-20850
> URL: https://issues.apache.org/jira/browse/HIVE-20850
> Project: Hive
> Issue Type: Improvement
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Attachments: HIVE-20850.01.patch, HIVE-20850.01.patch,
> HIVE-20850.01.patch, HIVE-20850.02.patch, HIVE-20850.03.patch,
> HIVE-20850.04.patch, HIVE-20850.05.patch, HIVE-20850.06.patch
>
>
> noticed by [~gopalv]: If there is a project which could be only evaluated
> after the join; but the condition references only a single column from a
> small dimension table; hive will end up evaluating the same thing over and
> over again...
> {code}
> explain
> select s_store_name, s_store_id,
> sum(case when (d_day_name='Sunday') then ss_sales_price else null
> end) sun_sales,
> sum(case when (d_day_name='Monday') then ss_sales_price else null
> end) mon_sales,
> sum(case when (d_day_name='Tuesday') then ss_sales_price else null
> end) tue_sales,
> sum(case when (d_day_name='Wednesday') then ss_sales_price else null
> end) wed_sales,
> sum(case when (d_day_name='Thursday') then ss_sales_price else null
> end) thu_sales,
> sum(case when (d_day_name='Friday') then ss_sales_price else null
> end) fri_sales,
> sum(case when (d_day_name='Saturday') then ss_sales_price else null
> end) sat_sales
> from date_dim, store_sales, store
> where d_date_sk = ss_sold_date_sk and
>s_store_sk = ss_store_sk and
>s_gmt_offset = -6 and
>d_year = 1998
> group by s_store_name, s_store_id
> order by s_store_name,
> s_store_id,sun_sales,mon_sales,tue_sales,wed_sales,thu_sales,fri_sales,sat_sales
> limit 100;
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20801) ACID: Allow DbTxnManager to ignore non-ACID table locking
[
https://issues.apache.org/jira/browse/HIVE-20801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16681021#comment-16681021
]
Hive QA commented on HIVE-20801:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
27s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
4s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
16s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
55s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
31s{color} | {color:blue} common in master has 65 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
38s{color} | {color:blue} ql in master has 2315 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
4s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
9s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
12s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
41s{color} | {color:red} ql: The patch generated 6 new + 537 unchanged - 0
fixed = 543 total (was 537) {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 2 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
6s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 25m 14s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-14828/dev-support/hive-personality.sh
|
| git revision | master / 5aac805 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-14828/yetus/diff-checkstyle-ql.txt
|
| whitespace |
http://104.198.109.242/logs//PreCommit-HIVE-Build-14828/yetus/whitespace-eol.txt
|
| modules | C: common ql U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-14828/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> ACID: Allow DbTxnManager to ignore non-ACID table locking
> -
>
> Key: HIVE-20801
> URL: https://issues.apache.org/jira/browse/HIVE-20801
> Project: Hive
> Issue Type: Bug
> Components: Locking, Transactions
>Affects Versions: 4.0.0
>Reporter: Gopal V
>Assignee: Gopal V
>Priority: Major
> Labels: Branch3Candidate, TODOC
> Attachments: HIVE-20801.1.patch, HIVE-20801.2.patch
>
>
> Enabling ACIDv1 on a cluster produces a central locking bottleneck for all
> table types, which is not always the intention.
> The Hive locking for non-acid tables are advisory (i.e a client can
> write/read without locking), which means that the implementation does not
> offer strong consistency despite the lock manager consuming resources
> centrally.
>
