[jira] [Commented] (HIVE-20488) SparkSubmitSparkClient#launchDriver should parse exceptions, not just errors
[
https://issues.apache.org/jira/browse/HIVE-20488?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16652993#comment-16652993
]
Bharathkrishna Guruvayoor Murali commented on HIVE-20488:
-
Attached same patch again to run tests. All tests passing now.
> SparkSubmitSparkClient#launchDriver should parse exceptions, not just errors
>
>
> Key: HIVE-20488
> URL: https://issues.apache.org/jira/browse/HIVE-20488
> Project: Hive
> Issue Type: Sub-task
> Components: Spark
>Reporter: Sahil Takiar
>Assignee: Bharathkrishna Guruvayoor Murali
>Priority: Major
> Attachments: HIVE-20488.1.patch
>
>
> In {{SparkSubmitSparkClient#launchDriver}} we parse the stdout / stderr of
> {{bin/spark-submit}} for strings that contain "Error", but we should also
> look for "Exception".
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19647) use bitvectors in IN operators
[ https://issues.apache.org/jira/browse/HIVE-19647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16652992#comment-16652992 ] Zoltan Haindrich commented on HIVE-19647: - opened rb: https://reviews.apache.org/r/69058/ [~ashutoshc] Could you please take a look? > use bitvectors in IN operators > -- > > Key: HIVE-19647 > URL: https://issues.apache.org/jira/browse/HIVE-19647 > Project: Hive > Issue Type: Improvement > Components: Statistics >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Major > Attachments: HIVE-19647.01.patch, HIVE-19647.02.patch, partial.patch > > > for strings there is no easy way to filter things out ; even min/max is null; > but the bitvectors could be used if they are available -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20617) Fix type of constants in IN expressions to have correct type
[
https://issues.apache.org/jira/browse/HIVE-20617?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zoltan Haindrich updated HIVE-20617:
Attachment: HIVE-20617.08.patch
> Fix type of constants in IN expressions to have correct type
>
>
> Key: HIVE-20617
> URL: https://issues.apache.org/jira/browse/HIVE-20617
> Project: Hive
> Issue Type: Bug
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Attachments: HIVE-20617.01.patch, HIVE-20617.02.patch,
> HIVE-20617.03.patch, HIVE-20617.05.patch, HIVE-20617.06.patch,
> HIVE-20617.07.patch, HIVE-20617.08.patch, HIVE-20617.08.patch,
> HIVE-20617.08.patch, HIVE-20617.08.patch, HIVE-20617.08.patch
>
>
> In statements like {{struct(a,b) IN (const struct('x','y'), ... )}} the
> comparision in UDFIn may fail because if a or b is of char/varchar type the
> constants will retain string type - especially after PointlookupOptimizer
> compaction.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20724) Add blobstore tests for MSCK REPAIR TABLE
[
https://issues.apache.org/jira/browse/HIVE-20724?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16652983#comment-16652983
]
Hive QA commented on HIVE-20724:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
44s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 1m 16s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-14515/dev-support/hive-personality.sh
|
| git revision | master / fa97c67 |
| modules | C: itests/hive-blobstore U: itests/hive-blobstore |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-14515/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Add blobstore tests for MSCK REPAIR TABLE
> -
>
> Key: HIVE-20724
> URL: https://issues.apache.org/jira/browse/HIVE-20724
> Project: Hive
> Issue Type: Test
> Components: Tests
>Reporter: Junyang Li
>Assignee: Junyang Li
>Priority: Major
> Fix For: 2.4.0
>
> Attachments: HIVE-20724.patch
>
>
> This patch introduces a regression test into the hive-blobstore qtest module:
> recover_partitions.q.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-16882) Improvements For Avro SerDe Package
[
https://issues.apache.org/jira/browse/HIVE-16882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16652976#comment-16652976
]
Hive QA commented on HIVE-16882:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12944187/HIVE-16882.13.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 15096 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/14514/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/14514/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-14514/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12944187 - PreCommit-HIVE-Build
> Improvements For Avro SerDe Package
> ---
>
> Key: HIVE-16882
> URL: https://issues.apache.org/jira/browse/HIVE-16882
> Project: Hive
> Issue Type: Improvement
> Components: Serializers/Deserializers
>Affects Versions: 3.0.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Minor
> Attachments: HIVE-16882.1.patch, HIVE-16882.10.patch,
> HIVE-16882.11.patch, HIVE-16882.12.patch, HIVE-16882.13.patch,
> HIVE-16882.2.patch, HIVE-16882.3.patch, HIVE-16882.4.patch,
> HIVE-16882.5.patch, HIVE-16882.6.patch, HIVE-16882.7.patch,
> HIVE-16882.8.patch, HIVE-16882.9.patch
>
>
> # Use SLF4J parameter DEBUG logging
> # Use re-usable libraries where appropriate
> # Use enhanced for loops where appropriate
> # Fix several minor check-style error
> # Small performance enhancements in InstanceCache
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20758) Constraints: Show create table does not show constraints
[
https://issues.apache.org/jira/browse/HIVE-20758?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16652937#comment-16652937
]
Gopal V commented on HIVE-20758:
bq. This will require calling metastore for all constraints (Default, pk/fk,
Check, Not null, Unique).
This is not a particularly performance sensitive operation, so that would still
be fine (because "desc formatted" wasn't very slow).
For me, this is was filed out of just sheer laziness on copying out schema
between clusters :)
> Constraints: Show create table does not show constraints
>
>
> Key: HIVE-20758
> URL: https://issues.apache.org/jira/browse/HIVE-20758
> Project: Hive
> Issue Type: Bug
>Affects Versions: 4.0.0
>Reporter: Gopal V
>Priority: Major
>
> Even though the desc formatted shows the constraints, the show create table
> does not
> {code}
> | # Primary Key | NULL
> | NULL
>|
> | Table: |
> tpcds_bin_partitioned_orc_1.inventory | NULL
> |
> | Constraint Name: | pk_in
> | NULL
>|
> | Column Names: | inv_date_sk
> | inv_item_sk
>|
> || NULL
> | NULL
>|
> | # Foreign Keys | NULL
> | NULL
>|
> | Table: |
> tpcds_bin_partitioned_orc_1.inventory | NULL
> |
> | Constraint Name: | inv_d
> | NULL
>|
> | Parent Column Name:tpcds_bin_partitioned_orc_1.date_dim.d_date_sk |
> Column Name:inv_date_sk| Key Sequence:1
> |
> || NULL
> | NULL
>|
> | Constraint Name: | inv_i
> | NULL
>|
> | Parent Column Name:tpcds_bin_partitioned_orc_1.item.i_item_sk | Column
> Name:inv_item_sk| Key Sequence:1
>|
> || NULL
> | NULL
>|
> | Constraint Name: | inv_w
> | NULL
>|
> | Parent Column Name:tpcds_bin_partitioned_orc_1.warehouse.w_warehouse_sk
> | Column Name:inv_warehouse_sk | Key Sequence:1
> |
> || NULL
> | NULL
>|
> {code}
> But
> {code}
> ++
> | createtab_stmt |
> ++
> | CREATE TABLE `inventory`( |
> | `inv_item_sk` bigint,|
> | `inv_warehouse_sk` bigint, |
> | `inv_quantity_on_hand` int, |
> | `inv_date_sk` bigint)|
> | ROW FORMAT SERDE |
> | 'org.apache.hadoop.hive.ql.io.orc.OrcSerde' |
> | STORED AS INPUTFORMAT |
> | 'org.apache.hadoop.hive.ql.io.orc.OrcInputFormat' |
> | OUTPUTFORMAT |
> | 'org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat' |
> | LOCATION |
> |
> 'hdfs:///warehouse/tablespace/managed/hive/tpcds_bin_partitioned_orc_1.db/inventory'
> |
> | TBLPROPERTIES (|
> | 'bucketing_version'='2', |
> | 'transactional'='true', |
> | 'transactional_properties'='default',
[jira] [Commented] (HIVE-16882) Improvements For Avro SerDe Package
[
https://issues.apache.org/jira/browse/HIVE-16882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16652932#comment-16652932
]
Hive QA commented on HIVE-16882:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
40s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
16s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
14s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
36s{color} | {color:blue} serde in master has 198 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
14s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
13s{color} | {color:green} serde: The patch generated 0 new + 91 unchanged - 7
fixed = 91 total (was 98) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
42s{color} | {color:green} serde generated 0 new + 197 unchanged - 1 fixed =
197 total (was 198) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
13s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
12s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 11m 18s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-14514/dev-support/hive-personality.sh
|
| git revision | master / fa97c67 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: serde U: serde |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-14514/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Improvements For Avro SerDe Package
> ---
>
> Key: HIVE-16882
> URL: https://issues.apache.org/jira/browse/HIVE-16882
> Project: Hive
> Issue Type: Improvement
> Components: Serializers/Deserializers
>Affects Versions: 3.0.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Minor
> Attachments: HIVE-16882.1.patch, HIVE-16882.10.patch,
> HIVE-16882.11.patch, HIVE-16882.12.patch, HIVE-16882.13.patch,
> HIVE-16882.2.patch, HIVE-16882.3.patch, HIVE-16882.4.patch,
> HIVE-16882.5.patch, HIVE-16882.6.patch, HIVE-16882.7.patch,
> HIVE-16882.8.patch, HIVE-16882.9.patch
>
>
> # Use SLF4J parameter DEBUG logging
> # Use re-usable libraries where appropriate
> # Use enhanced for loops where appropriate
> # Fix several minor check-style error
> # Small performance enhancements in InstanceCache
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20488) SparkSubmitSparkClient#launchDriver should parse exceptions, not just errors
[
https://issues.apache.org/jira/browse/HIVE-20488?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16652918#comment-16652918
]
Hive QA commented on HIVE-20488:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12944178/HIVE-20488.1.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 15097 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/14513/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/14513/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-14513/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12944178 - PreCommit-HIVE-Build
> SparkSubmitSparkClient#launchDriver should parse exceptions, not just errors
>
>
> Key: HIVE-20488
> URL: https://issues.apache.org/jira/browse/HIVE-20488
> Project: Hive
> Issue Type: Sub-task
> Components: Spark
>Reporter: Sahil Takiar
>Assignee: Bharathkrishna Guruvayoor Murali
>Priority: Major
> Attachments: HIVE-20488.1.patch
>
>
> In {{SparkSubmitSparkClient#launchDriver}} we parse the stdout / stderr of
> {{bin/spark-submit}} for strings that contain "Error", but we should also
> look for "Exception".
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20758) Constraints: Show create table does not show constraints
[
https://issues.apache.org/jira/browse/HIVE-20758?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16652903#comment-16652903
]
Vineet Garg commented on HIVE-20758:
This will require calling metastore for all constraints (Default, pk/fk, Check,
Not null, Unique). Instead of calling each and every time it will be good to
add metastore api to return all constraints given a table
> Constraints: Show create table does not show constraints
>
>
> Key: HIVE-20758
> URL: https://issues.apache.org/jira/browse/HIVE-20758
> Project: Hive
> Issue Type: Bug
>Affects Versions: 4.0.0
>Reporter: Gopal V
>Priority: Major
>
> Even though the desc formatted shows the constraints, the show create table
> does not
> {code}
> | # Primary Key | NULL
> | NULL
>|
> | Table: |
> tpcds_bin_partitioned_orc_1.inventory | NULL
> |
> | Constraint Name: | pk_in
> | NULL
>|
> | Column Names: | inv_date_sk
> | inv_item_sk
>|
> || NULL
> | NULL
>|
> | # Foreign Keys | NULL
> | NULL
>|
> | Table: |
> tpcds_bin_partitioned_orc_1.inventory | NULL
> |
> | Constraint Name: | inv_d
> | NULL
>|
> | Parent Column Name:tpcds_bin_partitioned_orc_1.date_dim.d_date_sk |
> Column Name:inv_date_sk| Key Sequence:1
> |
> || NULL
> | NULL
>|
> | Constraint Name: | inv_i
> | NULL
>|
> | Parent Column Name:tpcds_bin_partitioned_orc_1.item.i_item_sk | Column
> Name:inv_item_sk| Key Sequence:1
>|
> || NULL
> | NULL
>|
> | Constraint Name: | inv_w
> | NULL
>|
> | Parent Column Name:tpcds_bin_partitioned_orc_1.warehouse.w_warehouse_sk
> | Column Name:inv_warehouse_sk | Key Sequence:1
> |
> || NULL
> | NULL
>|
> {code}
> But
> {code}
> ++
> | createtab_stmt |
> ++
> | CREATE TABLE `inventory`( |
> | `inv_item_sk` bigint,|
> | `inv_warehouse_sk` bigint, |
> | `inv_quantity_on_hand` int, |
> | `inv_date_sk` bigint)|
> | ROW FORMAT SERDE |
> | 'org.apache.hadoop.hive.ql.io.orc.OrcSerde' |
> | STORED AS INPUTFORMAT |
> | 'org.apache.hadoop.hive.ql.io.orc.OrcInputFormat' |
> | OUTPUTFORMAT |
> | 'org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat' |
> | LOCATION |
> |
> 'hdfs:///warehouse/tablespace/managed/hive/tpcds_bin_partitioned_orc_1.db/inventory'
> |
> | TBLPROPERTIES (|
> | 'bucketing_version'='2', |
> | 'transactional'='true', |
> | 'transactional_properties'='default',|
> | 'transient_lastDdlTime'='1539710410')|
>
[jira] [Commented] (HIVE-20488) SparkSubmitSparkClient#launchDriver should parse exceptions, not just errors
[
https://issues.apache.org/jira/browse/HIVE-20488?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16652892#comment-16652892
]
Hive QA commented on HIVE-20488:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
33s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
14s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
9s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
23s{color} | {color:blue} spark-client in master has 10 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
10s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
9s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 10m 25s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-14513/dev-support/hive-personality.sh
|
| git revision | master / fa97c67 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: spark-client U: spark-client |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-14513/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> SparkSubmitSparkClient#launchDriver should parse exceptions, not just errors
>
>
> Key: HIVE-20488
> URL: https://issues.apache.org/jira/browse/HIVE-20488
> Project: Hive
> Issue Type: Sub-task
> Components: Spark
>Reporter: Sahil Takiar
>Assignee: Bharathkrishna Guruvayoor Murali
>Priority: Major
> Attachments: HIVE-20488.1.patch
>
>
> In {{SparkSubmitSparkClient#launchDriver}} we parse the stdout / stderr of
> {{bin/spark-submit}} for strings that contain "Error", but we should also
> look for "Exception".
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20512) Improve record and memory usage logging in SparkRecordHandler
[
https://issues.apache.org/jira/browse/HIVE-20512?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bharathkrishna Guruvayoor Murali updated HIVE-20512:
Attachment: HIVE-20512.1.patch
> Improve record and memory usage logging in SparkRecordHandler
> -
>
> Key: HIVE-20512
> URL: https://issues.apache.org/jira/browse/HIVE-20512
> Project: Hive
> Issue Type: Sub-task
> Components: Spark
>Reporter: Sahil Takiar
>Assignee: Bharathkrishna Guruvayoor Murali
>Priority: Major
> Attachments: HIVE-20512.1.patch
>
>
> We currently log memory usage and # of records processed in Spark tasks, but
> we should improve the methodology for how frequently we log this info.
> Currently we use the following code:
> {code:java}
> private long getNextLogThreshold(long currentThreshold) {
> // A very simple counter to keep track of number of rows processed by the
> // reducer. It dumps
> // every 1 million times, and quickly before that
> if (currentThreshold >= 100) {
> return currentThreshold + 100;
> }
> return 10 * currentThreshold;
> }
> {code}
> The issue is that after a while, the increase by 10x factor means that you
> have to process a huge # of records before this gets triggered.
> A better approach would be to log this info at a given interval. This would
> help in debugging tasks that are seemingly hung.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20740) Remove global lock in ObjectStore.setConf method
[
https://issues.apache.org/jira/browse/HIVE-20740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16652889#comment-16652889
]
Alan Gates commented on HIVE-20740:
---
Will be a couple of days before I have time to look at it.
cc [~thejas]
> Remove global lock in ObjectStore.setConf method
>
>
> Key: HIVE-20740
> URL: https://issues.apache.org/jira/browse/HIVE-20740
> Project: Hive
> Issue Type: Improvement
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
>Priority: Major
> Attachments: HIVE-20740.01.patch, HIVE-20740.02.patch
>
>
> The ObjectStore#setConf method has a global lock which can block other
> clients in concurrent workloads.
> {code}
> @Override
> @SuppressWarnings("nls")
> public void setConf(Configuration conf) {
> // Although an instance of ObjectStore is accessed by one thread, there
> may
> // be many threads with ObjectStore instances. So the static variables
> // pmf and prop need to be protected with locks.
> pmfPropLock.lock();
> try {
> isInitialized = false;
> this.conf = conf;
> this.areTxnStatsSupported = MetastoreConf.getBoolVar(conf,
> ConfVars.HIVE_TXN_STATS_ENABLED);
> configureSSL(conf);
> Properties propsFromConf = getDataSourceProps(conf);
> boolean propsChanged = !propsFromConf.equals(prop);
> if (propsChanged) {
> if (pmf != null){
> clearOutPmfClassLoaderCache(pmf);
> if (!forTwoMetastoreTesting) {
> // close the underlying connection pool to avoid leaks
> pmf.close();
> }
> }
> pmf = null;
> prop = null;
> }
> assert(!isActiveTransaction());
> shutdown();
> // Always want to re-create pm as we don't know if it were created by
> the
> // most recent instance of the pmf
> pm = null;
> directSql = null;
> expressionProxy = null;
> openTrasactionCalls = 0;
> currentTransaction = null;
> transactionStatus = TXN_STATUS.NO_STATE;
> initialize(propsFromConf);
> String partitionValidationRegex =
> MetastoreConf.getVar(this.conf,
> ConfVars.PARTITION_NAME_WHITELIST_PATTERN);
> if (partitionValidationRegex != null &&
> !partitionValidationRegex.isEmpty()) {
> partitionValidationPattern =
> Pattern.compile(partitionValidationRegex);
> } else {
> partitionValidationPattern = null;
> }
> // Note, if metrics have not been initialized this will return null,
> which means we aren't
> // using metrics. Thus we should always check whether this is non-null
> before using.
> MetricRegistry registry = Metrics.getRegistry();
> if (registry != null) {
> directSqlErrors =
> Metrics.getOrCreateCounter(MetricsConstants.DIRECTSQL_ERRORS);
> }
> this.batchSize = MetastoreConf.getIntVar(conf,
> ConfVars.RAWSTORE_PARTITION_BATCH_SIZE);
> if (!isInitialized) {
> throw new RuntimeException(
> "Unable to create persistence manager. Check dss.log for details");
> } else {
> LOG.debug("Initialized ObjectStore");
> }
> } finally {
> pmfPropLock.unlock();
> }
> }
> {code}
> The {{pmfPropLock}} is a static object and it disallows any other new
> connection to HMS which is trying to instantiate ObjectStore. We should
> either remove the lock or reduce the scope of the lock so that it is held for
> a very small amount of time.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20512) Improve record and memory usage logging in SparkRecordHandler
[
https://issues.apache.org/jira/browse/HIVE-20512?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bharathkrishna Guruvayoor Murali updated HIVE-20512:
Attachment: (was: HIVE-20512.1.patch)
> Improve record and memory usage logging in SparkRecordHandler
> -
>
> Key: HIVE-20512
> URL: https://issues.apache.org/jira/browse/HIVE-20512
> Project: Hive
> Issue Type: Sub-task
> Components: Spark
>Reporter: Sahil Takiar
>Assignee: Bharathkrishna Guruvayoor Murali
>Priority: Major
> Attachments: HIVE-20512.1.patch
>
>
> We currently log memory usage and # of records processed in Spark tasks, but
> we should improve the methodology for how frequently we log this info.
> Currently we use the following code:
> {code:java}
> private long getNextLogThreshold(long currentThreshold) {
> // A very simple counter to keep track of number of rows processed by the
> // reducer. It dumps
> // every 1 million times, and quickly before that
> if (currentThreshold >= 100) {
> return currentThreshold + 100;
> }
> return 10 * currentThreshold;
> }
> {code}
> The issue is that after a while, the increase by 10x factor means that you
> have to process a huge # of records before this gets triggered.
> A better approach would be to log this info at a given interval. This would
> help in debugging tasks that are seemingly hung.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20512) Improve record and memory usage logging in SparkRecordHandler
[
https://issues.apache.org/jira/browse/HIVE-20512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16652888#comment-16652888
]
Hive QA commented on HIVE-20512:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12944186/HIVE-20512.1.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 15096 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.ql.TestTxnCommands.testMergeOnTezEdges (batchId=318)
org.apache.hive.jdbc.TestJdbcWithMiniLlapArrow.testComplexQuery (batchId=253)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/14512/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/14512/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-14512/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12944186 - PreCommit-HIVE-Build
> Improve record and memory usage logging in SparkRecordHandler
> -
>
> Key: HIVE-20512
> URL: https://issues.apache.org/jira/browse/HIVE-20512
> Project: Hive
> Issue Type: Sub-task
> Components: Spark
>Reporter: Sahil Takiar
>Assignee: Bharathkrishna Guruvayoor Murali
>Priority: Major
> Attachments: HIVE-20512.1.patch
>
>
> We currently log memory usage and # of records processed in Spark tasks, but
> we should improve the methodology for how frequently we log this info.
> Currently we use the following code:
> {code:java}
> private long getNextLogThreshold(long currentThreshold) {
> // A very simple counter to keep track of number of rows processed by the
> // reducer. It dumps
> // every 1 million times, and quickly before that
> if (currentThreshold >= 100) {
> return currentThreshold + 100;
> }
> return 10 * currentThreshold;
> }
> {code}
> The issue is that after a while, the increase by 10x factor means that you
> have to process a huge # of records before this gets triggered.
> A better approach would be to log this info at a given interval. This would
> help in debugging tasks that are seemingly hung.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20512) Improve record and memory usage logging in SparkRecordHandler
[
https://issues.apache.org/jira/browse/HIVE-20512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16652858#comment-16652858
]
Hive QA commented on HIVE-20512:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
33s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
58s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
35s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
38s{color} | {color:blue} ql in master has 2318 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
54s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
35s{color} | {color:green} ql: The patch generated 0 new + 4 unchanged - 2
fixed = 4 total (was 6) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
52s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
12s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 21m 59s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-14512/dev-support/hive-personality.sh
|
| git revision | master / fa97c67 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-14512/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Improve record and memory usage logging in SparkRecordHandler
> -
>
> Key: HIVE-20512
> URL: https://issues.apache.org/jira/browse/HIVE-20512
> Project: Hive
> Issue Type: Sub-task
> Components: Spark
>Reporter: Sahil Takiar
>Assignee: Bharathkrishna Guruvayoor Murali
>Priority: Major
> Attachments: HIVE-20512.1.patch
>
>
> We currently log memory usage and # of records processed in Spark tasks, but
> we should improve the methodology for how frequently we log this info.
> Currently we use the following code:
> {code:java}
> private long getNextLogThreshold(long currentThreshold) {
> // A very simple counter to keep track of number of rows processed by the
> // reducer. It dumps
> // every 1 million times, and quickly before that
> if (currentThreshold >= 100) {
> return currentThreshold + 100;
> }
> return 10 * currentThreshold;
> }
> {code}
> The issue is that after a while, the increase by 10x factor means that you
> have to process a huge # of records before this gets triggered.
> A better approach would be to log this info at a given interval. This would
> help in debugging tasks that are seemingly hung.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20754) JDBC: Add some missing classes to jdbc standalone jar (follow up to HIVE-19801)
[
https://issues.apache.org/jira/browse/HIVE-20754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16652823#comment-16652823
]
Hive QA commented on HIVE-20754:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12944176/HIVE-20754.1.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 15091 tests
executed
*Failed tests:*
{noformat}
TestMiniDruidCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=195)
[druidmini_dynamic_partition.q,druidmini_test_ts.q,druidmini_expressions.q,druidmini_test_alter.q,druidmini_test_insert.q]
org.apache.hive.jdbc.TestJdbcWithMiniLlapArrow.testComplexQuery (batchId=253)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/14511/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/14511/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-14511/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12944176 - PreCommit-HIVE-Build
> JDBC: Add some missing classes to jdbc standalone jar (follow up to
> HIVE-19801)
> ---
>
> Key: HIVE-20754
> URL: https://issues.apache.org/jira/browse/HIVE-20754
> Project: Hive
> Issue Type: Bug
> Components: JDBC
>Affects Versions: 3.1.0
>Reporter: Vaibhav Gumashta
>Assignee: Vaibhav Gumashta
>Priority: Major
> Attachments: HIVE-20754.1.patch
>
>
> Some more classes are needed in a secure cluster
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20754) JDBC: Add some missing classes to jdbc standalone jar (follow up to HIVE-19801)
[
https://issues.apache.org/jira/browse/HIVE-20754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16652759#comment-16652759
]
Hive QA commented on HIVE-20754:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
1s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
44s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
15s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
12s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
16s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch 14 line(s) with tabs. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
12s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
12s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 9m 48s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc xml compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-14511/dev-support/hive-personality.sh
|
| git revision | master / fa97c67 |
| Default Java | 1.8.0_111 |
| whitespace |
http://104.198.109.242/logs//PreCommit-HIVE-Build-14511/yetus/whitespace-tabs.txt
|
| modules | C: jdbc U: jdbc |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-14511/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> JDBC: Add some missing classes to jdbc standalone jar (follow up to
> HIVE-19801)
> ---
>
> Key: HIVE-20754
> URL: https://issues.apache.org/jira/browse/HIVE-20754
> Project: Hive
> Issue Type: Bug
> Components: JDBC
>Affects Versions: 3.1.0
>Reporter: Vaibhav Gumashta
>Assignee: Vaibhav Gumashta
>Priority: Major
> Attachments: HIVE-20754.1.patch
>
>
> Some more classes are needed in a secure cluster
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20737) Local SparkContext is shared between user sessions and should be closed only when there is no active
[
https://issues.apache.org/jira/browse/HIVE-20737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16652738#comment-16652738
]
Hive QA commented on HIVE-20737:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12944170/HIVE-20737.10.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 15097 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_groupby_reduce]
(batchId=60)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[authorization_jdbc_keystore]
(batchId=98)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/14510/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/14510/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-14510/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12944170 - PreCommit-HIVE-Build
> Local SparkContext is shared between user sessions and should be closed only
> when there is no active
>
>
> Key: HIVE-20737
> URL: https://issues.apache.org/jira/browse/HIVE-20737
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Reporter: Denys Kuzmenko
>Assignee: Denys Kuzmenko
>Priority: Major
> Attachments: HIVE-20737.1.patch, HIVE-20737.10.patch,
> HIVE-20737.2.patch, HIVE-20737.5.patch, HIVE-20737.6.patch,
> HIVE-20737.7.patch, HIVE-20737.8.patch, HIVE-20737.9.patch
>
>
> 1. Local SparkContext is shared between user sessions and should be closed
> only when there is no active.
> 2. Possible race condition in SparkSession.open() in case when user queries
> run in parallel within the same session.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20737) Local SparkContext is shared between user sessions and should be closed only when there is no active
[
https://issues.apache.org/jira/browse/HIVE-20737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16652682#comment-16652682
]
Hive QA commented on HIVE-20737:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
39s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
57s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
34s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
50s{color} | {color:blue} ql in master has 2318 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
53s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
36s{color} | {color:green} ql: The patch generated 0 new + 6 unchanged - 1
fixed = 6 total (was 7) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
12s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 22m 20s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-14510/dev-support/hive-personality.sh
|
| git revision | master / e39a198 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-14510/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Local SparkContext is shared between user sessions and should be closed only
> when there is no active
>
>
> Key: HIVE-20737
> URL: https://issues.apache.org/jira/browse/HIVE-20737
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Reporter: Denys Kuzmenko
>Assignee: Denys Kuzmenko
>Priority: Major
> Attachments: HIVE-20737.1.patch, HIVE-20737.10.patch,
> HIVE-20737.2.patch, HIVE-20737.5.patch, HIVE-20737.6.patch,
> HIVE-20737.7.patch, HIVE-20737.8.patch, HIVE-20737.9.patch
>
>
> 1. Local SparkContext is shared between user sessions and should be closed
> only when there is no active.
> 2. Possible race condition in SparkSession.open() in case when user queries
> run in parallel within the same session.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20758) Constraints: Show create table does not show constraints
[
https://issues.apache.org/jira/browse/HIVE-20758?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gopal V updated HIVE-20758:
---
Affects Version/s: 4.0.0
> Constraints: Show create table does not show constraints
>
>
> Key: HIVE-20758
> URL: https://issues.apache.org/jira/browse/HIVE-20758
> Project: Hive
> Issue Type: Bug
>Affects Versions: 4.0.0
>Reporter: Gopal V
>Priority: Major
>
> Even though the desc formatted shows the constraints, the show create table
> does not
> {code}
> | # Primary Key | NULL
> | NULL
>|
> | Table: |
> tpcds_bin_partitioned_orc_1.inventory | NULL
> |
> | Constraint Name: | pk_in
> | NULL
>|
> | Column Names: | inv_date_sk
> | inv_item_sk
>|
> || NULL
> | NULL
>|
> | # Foreign Keys | NULL
> | NULL
>|
> | Table: |
> tpcds_bin_partitioned_orc_1.inventory | NULL
> |
> | Constraint Name: | inv_d
> | NULL
>|
> | Parent Column Name:tpcds_bin_partitioned_orc_1.date_dim.d_date_sk |
> Column Name:inv_date_sk| Key Sequence:1
> |
> || NULL
> | NULL
>|
> | Constraint Name: | inv_i
> | NULL
>|
> | Parent Column Name:tpcds_bin_partitioned_orc_1.item.i_item_sk | Column
> Name:inv_item_sk| Key Sequence:1
>|
> || NULL
> | NULL
>|
> | Constraint Name: | inv_w
> | NULL
>|
> | Parent Column Name:tpcds_bin_partitioned_orc_1.warehouse.w_warehouse_sk
> | Column Name:inv_warehouse_sk | Key Sequence:1
> |
> || NULL
> | NULL
>|
> {code}
> But
> {code}
> ++
> | createtab_stmt |
> ++
> | CREATE TABLE `inventory`( |
> | `inv_item_sk` bigint,|
> | `inv_warehouse_sk` bigint, |
> | `inv_quantity_on_hand` int, |
> | `inv_date_sk` bigint)|
> | ROW FORMAT SERDE |
> | 'org.apache.hadoop.hive.ql.io.orc.OrcSerde' |
> | STORED AS INPUTFORMAT |
> | 'org.apache.hadoop.hive.ql.io.orc.OrcInputFormat' |
> | OUTPUTFORMAT |
> | 'org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat' |
> | LOCATION |
> |
> 'hdfs:///warehouse/tablespace/managed/hive/tpcds_bin_partitioned_orc_1.db/inventory'
> |
> | TBLPROPERTIES (|
> | 'bucketing_version'='2', |
> | 'transactional'='true', |
> | 'transactional_properties'='default',|
> | 'transient_lastDdlTime'='1539710410')|
> ++
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20703) Put dynamic sort partition optimization under cost based decision
[ https://issues.apache.org/jira/browse/HIVE-20703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16652680#comment-16652680 ] Vineet Garg commented on HIVE-20703: [~sershe] Waiting for HIVE-20713's patch. Once that is in I'll update the hardcoded number. > Put dynamic sort partition optimization under cost based decision > - > > Key: HIVE-20703 > URL: https://issues.apache.org/jira/browse/HIVE-20703 > Project: Hive > Issue Type: Improvement > Components: Query Planning >Reporter: Vineet Garg >Assignee: Vineet Garg >Priority: Major > Attachments: HIVE-20703.1.patch, HIVE-20703.2.patch, > HIVE-20703.3.patch, HIVE-20703.4.patch, HIVE-20703.5.patch, > HIVE-20703.6.patch, HIVE-20703.7.patch, HIVE-20703.8.patch, HIVE-20703.9.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20716) Set default value for hive.cbo.stats.correlated.multi.key.joins to true
[ https://issues.apache.org/jira/browse/HIVE-20716?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-20716: --- Attachment: HIVE-20716.03.patch > Set default value for hive.cbo.stats.correlated.multi.key.joins to true > --- > > Key: HIVE-20716 > URL: https://issues.apache.org/jira/browse/HIVE-20716 > Project: Hive > Issue Type: Bug >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Major > Attachments: HIVE-20716.01.patch, HIVE-20716.02.patch, > HIVE-20716.03.patch, HIVE-20716.03.patch, HIVE-20716.03.patch, > HIVE-20716.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20716) Set default value for hive.cbo.stats.correlated.multi.key.joins to true
[
https://issues.apache.org/jira/browse/HIVE-20716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16652646#comment-16652646
]
Hive QA commented on HIVE-20716:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12944155/HIVE-20716.03.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 15096 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[authorization_jdbc_keystore]
(batchId=98)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/14509/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/14509/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-14509/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12944155 - PreCommit-HIVE-Build
> Set default value for hive.cbo.stats.correlated.multi.key.joins to true
> ---
>
> Key: HIVE-20716
> URL: https://issues.apache.org/jira/browse/HIVE-20716
> Project: Hive
> Issue Type: Bug
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-20716.01.patch, HIVE-20716.02.patch,
> HIVE-20716.03.patch, HIVE-20716.03.patch, HIVE-20716.patch
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20752) In case of LLAP start failure add info how to find YARN logs
[ https://issues.apache.org/jira/browse/HIVE-20752?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Gergely updated HIVE-20752: -- Attachment: HIVE-20752.02.patch > In case of LLAP start failure add info how to find YARN logs > > > Key: HIVE-20752 > URL: https://issues.apache.org/jira/browse/HIVE-20752 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Affects Versions: 4.0.0 >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-20752.01.patch, HIVE-20752.02.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20752) In case of LLAP start failure add info how to find YARN logs
[ https://issues.apache.org/jira/browse/HIVE-20752?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Gergely updated HIVE-20752: -- Status: Patch Available (was: Open) > In case of LLAP start failure add info how to find YARN logs > > > Key: HIVE-20752 > URL: https://issues.apache.org/jira/browse/HIVE-20752 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Affects Versions: 4.0.0 >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-20752.01.patch, HIVE-20752.02.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20752) In case of LLAP start failure add info how to find YARN logs
[ https://issues.apache.org/jira/browse/HIVE-20752?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Gergely updated HIVE-20752: -- Status: Open (was: Patch Available) > In case of LLAP start failure add info how to find YARN logs > > > Key: HIVE-20752 > URL: https://issues.apache.org/jira/browse/HIVE-20752 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Affects Versions: 4.0.0 >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-20752.01.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20756) Disable SARG leaf creation for date column until ORC-135
[ https://issues.apache.org/jira/browse/HIVE-20756?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-20756: Attachment: HIVE-20756.01-branch-1.patch > Disable SARG leaf creation for date column until ORC-135 > > > Key: HIVE-20756 > URL: https://issues.apache.org/jira/browse/HIVE-20756 > Project: Hive > Issue Type: Bug >Affects Versions: 2.1.1 >Reporter: Chiran Ravani >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-20756.01-branch-1.patch, HIVE-20756.1.patch > > > Until ORC-135 is committed and orc version is updated in hive, disable SARG > creation for date columns in hive. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-20756) Disable SARG leaf creation for date column until ORC-135
[ https://issues.apache.org/jira/browse/HIVE-20756?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin reassigned HIVE-20756: --- Assignee: Sergey Shelukhin (was: Chiran Ravani) > Disable SARG leaf creation for date column until ORC-135 > > > Key: HIVE-20756 > URL: https://issues.apache.org/jira/browse/HIVE-20756 > Project: Hive > Issue Type: Bug >Affects Versions: 2.1.1 >Reporter: Chiran Ravani >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-20756.01-branch-1.patch, HIVE-20756.1.patch > > > Until ORC-135 is committed and orc version is updated in hive, disable SARG > creation for date columns in hive. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20756) Disable SARG leaf creation for date column until ORC-135
[ https://issues.apache.org/jira/browse/HIVE-20756?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-20756: Assignee: Chiran Ravani (was: Sergey Shelukhin) Status: Patch Available (was: Open) > Disable SARG leaf creation for date column until ORC-135 > > > Key: HIVE-20756 > URL: https://issues.apache.org/jira/browse/HIVE-20756 > Project: Hive > Issue Type: Bug >Affects Versions: 2.1.1 >Reporter: Chiran Ravani >Assignee: Chiran Ravani >Priority: Major > Attachments: HIVE-20756.01-branch-1.patch, HIVE-20756.1.patch > > > Until ORC-135 is committed and orc version is updated in hive, disable SARG > creation for date columns in hive. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20756) Disable SARG leaf creation for date column until ORC-135
[ https://issues.apache.org/jira/browse/HIVE-20756?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16652619#comment-16652619 ] Sergey Shelukhin commented on HIVE-20756: - branch-1 tests are all screwed up anyway :) > Disable SARG leaf creation for date column until ORC-135 > > > Key: HIVE-20756 > URL: https://issues.apache.org/jira/browse/HIVE-20756 > Project: Hive > Issue Type: Bug >Affects Versions: 2.1.1 >Reporter: Chiran Ravani >Assignee: Chiran Ravani >Priority: Major > Attachments: HIVE-20756.1.patch > > > Until ORC-135 is committed and orc version is updated in hive, disable SARG > creation for date columns in hive. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20703) Put dynamic sort partition optimization under cost based decision
[ https://issues.apache.org/jira/browse/HIVE-20703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16652616#comment-16652616 ] Sergey Shelukhin commented on HIVE-20703: - Looks good other than the 40 number remaining hardcoded. > Put dynamic sort partition optimization under cost based decision > - > > Key: HIVE-20703 > URL: https://issues.apache.org/jira/browse/HIVE-20703 > Project: Hive > Issue Type: Improvement > Components: Query Planning >Reporter: Vineet Garg >Assignee: Vineet Garg >Priority: Major > Attachments: HIVE-20703.1.patch, HIVE-20703.2.patch, > HIVE-20703.3.patch, HIVE-20703.4.patch, HIVE-20703.5.patch, > HIVE-20703.6.patch, HIVE-20703.7.patch, HIVE-20703.8.patch, HIVE-20703.9.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20740) Remove global lock in ObjectStore.setConf method
[
https://issues.apache.org/jira/browse/HIVE-20740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16652579#comment-16652579
]
Vihang Karajgaonkar commented on HIVE-20740:
v2 patch includes suggested changes from [~asherman] on review board
> Remove global lock in ObjectStore.setConf method
>
>
> Key: HIVE-20740
> URL: https://issues.apache.org/jira/browse/HIVE-20740
> Project: Hive
> Issue Type: Improvement
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
>Priority: Major
> Attachments: HIVE-20740.01.patch, HIVE-20740.02.patch
>
>
> The ObjectStore#setConf method has a global lock which can block other
> clients in concurrent workloads.
> {code}
> @Override
> @SuppressWarnings("nls")
> public void setConf(Configuration conf) {
> // Although an instance of ObjectStore is accessed by one thread, there
> may
> // be many threads with ObjectStore instances. So the static variables
> // pmf and prop need to be protected with locks.
> pmfPropLock.lock();
> try {
> isInitialized = false;
> this.conf = conf;
> this.areTxnStatsSupported = MetastoreConf.getBoolVar(conf,
> ConfVars.HIVE_TXN_STATS_ENABLED);
> configureSSL(conf);
> Properties propsFromConf = getDataSourceProps(conf);
> boolean propsChanged = !propsFromConf.equals(prop);
> if (propsChanged) {
> if (pmf != null){
> clearOutPmfClassLoaderCache(pmf);
> if (!forTwoMetastoreTesting) {
> // close the underlying connection pool to avoid leaks
> pmf.close();
> }
> }
> pmf = null;
> prop = null;
> }
> assert(!isActiveTransaction());
> shutdown();
> // Always want to re-create pm as we don't know if it were created by
> the
> // most recent instance of the pmf
> pm = null;
> directSql = null;
> expressionProxy = null;
> openTrasactionCalls = 0;
> currentTransaction = null;
> transactionStatus = TXN_STATUS.NO_STATE;
> initialize(propsFromConf);
> String partitionValidationRegex =
> MetastoreConf.getVar(this.conf,
> ConfVars.PARTITION_NAME_WHITELIST_PATTERN);
> if (partitionValidationRegex != null &&
> !partitionValidationRegex.isEmpty()) {
> partitionValidationPattern =
> Pattern.compile(partitionValidationRegex);
> } else {
> partitionValidationPattern = null;
> }
> // Note, if metrics have not been initialized this will return null,
> which means we aren't
> // using metrics. Thus we should always check whether this is non-null
> before using.
> MetricRegistry registry = Metrics.getRegistry();
> if (registry != null) {
> directSqlErrors =
> Metrics.getOrCreateCounter(MetricsConstants.DIRECTSQL_ERRORS);
> }
> this.batchSize = MetastoreConf.getIntVar(conf,
> ConfVars.RAWSTORE_PARTITION_BATCH_SIZE);
> if (!isInitialized) {
> throw new RuntimeException(
> "Unable to create persistence manager. Check dss.log for details");
> } else {
> LOG.debug("Initialized ObjectStore");
> }
> } finally {
> pmfPropLock.unlock();
> }
> }
> {code}
> The {{pmfPropLock}} is a static object and it disallows any other new
> connection to HMS which is trying to instantiate ObjectStore. We should
> either remove the lock or reduce the scope of the lock so that it is held for
> a very small amount of time.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20740) Remove global lock in ObjectStore.setConf method
[
https://issues.apache.org/jira/browse/HIVE-20740?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vihang Karajgaonkar updated HIVE-20740:
---
Attachment: HIVE-20740.02.patch
> Remove global lock in ObjectStore.setConf method
>
>
> Key: HIVE-20740
> URL: https://issues.apache.org/jira/browse/HIVE-20740
> Project: Hive
> Issue Type: Improvement
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
>Priority: Major
> Attachments: HIVE-20740.01.patch, HIVE-20740.02.patch
>
>
> The ObjectStore#setConf method has a global lock which can block other
> clients in concurrent workloads.
> {code}
> @Override
> @SuppressWarnings("nls")
> public void setConf(Configuration conf) {
> // Although an instance of ObjectStore is accessed by one thread, there
> may
> // be many threads with ObjectStore instances. So the static variables
> // pmf and prop need to be protected with locks.
> pmfPropLock.lock();
> try {
> isInitialized = false;
> this.conf = conf;
> this.areTxnStatsSupported = MetastoreConf.getBoolVar(conf,
> ConfVars.HIVE_TXN_STATS_ENABLED);
> configureSSL(conf);
> Properties propsFromConf = getDataSourceProps(conf);
> boolean propsChanged = !propsFromConf.equals(prop);
> if (propsChanged) {
> if (pmf != null){
> clearOutPmfClassLoaderCache(pmf);
> if (!forTwoMetastoreTesting) {
> // close the underlying connection pool to avoid leaks
> pmf.close();
> }
> }
> pmf = null;
> prop = null;
> }
> assert(!isActiveTransaction());
> shutdown();
> // Always want to re-create pm as we don't know if it were created by
> the
> // most recent instance of the pmf
> pm = null;
> directSql = null;
> expressionProxy = null;
> openTrasactionCalls = 0;
> currentTransaction = null;
> transactionStatus = TXN_STATUS.NO_STATE;
> initialize(propsFromConf);
> String partitionValidationRegex =
> MetastoreConf.getVar(this.conf,
> ConfVars.PARTITION_NAME_WHITELIST_PATTERN);
> if (partitionValidationRegex != null &&
> !partitionValidationRegex.isEmpty()) {
> partitionValidationPattern =
> Pattern.compile(partitionValidationRegex);
> } else {
> partitionValidationPattern = null;
> }
> // Note, if metrics have not been initialized this will return null,
> which means we aren't
> // using metrics. Thus we should always check whether this is non-null
> before using.
> MetricRegistry registry = Metrics.getRegistry();
> if (registry != null) {
> directSqlErrors =
> Metrics.getOrCreateCounter(MetricsConstants.DIRECTSQL_ERRORS);
> }
> this.batchSize = MetastoreConf.getIntVar(conf,
> ConfVars.RAWSTORE_PARTITION_BATCH_SIZE);
> if (!isInitialized) {
> throw new RuntimeException(
> "Unable to create persistence manager. Check dss.log for details");
> } else {
> LOG.debug("Initialized ObjectStore");
> }
> } finally {
> pmfPropLock.unlock();
> }
> }
> {code}
> The {{pmfPropLock}} is a static object and it disallows any other new
> connection to HMS which is trying to instantiate ObjectStore. We should
> either remove the lock or reduce the scope of the lock so that it is held for
> a very small amount of time.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20740) Remove global lock in ObjectStore.setConf method
[
https://issues.apache.org/jira/browse/HIVE-20740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16652576#comment-16652576
]
Vihang Karajgaonkar commented on HIVE-20740:
Added review board link.
> Remove global lock in ObjectStore.setConf method
>
>
> Key: HIVE-20740
> URL: https://issues.apache.org/jira/browse/HIVE-20740
> Project: Hive
> Issue Type: Improvement
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
>Priority: Major
> Attachments: HIVE-20740.01.patch, HIVE-20740.02.patch
>
>
> The ObjectStore#setConf method has a global lock which can block other
> clients in concurrent workloads.
> {code}
> @Override
> @SuppressWarnings("nls")
> public void setConf(Configuration conf) {
> // Although an instance of ObjectStore is accessed by one thread, there
> may
> // be many threads with ObjectStore instances. So the static variables
> // pmf and prop need to be protected with locks.
> pmfPropLock.lock();
> try {
> isInitialized = false;
> this.conf = conf;
> this.areTxnStatsSupported = MetastoreConf.getBoolVar(conf,
> ConfVars.HIVE_TXN_STATS_ENABLED);
> configureSSL(conf);
> Properties propsFromConf = getDataSourceProps(conf);
> boolean propsChanged = !propsFromConf.equals(prop);
> if (propsChanged) {
> if (pmf != null){
> clearOutPmfClassLoaderCache(pmf);
> if (!forTwoMetastoreTesting) {
> // close the underlying connection pool to avoid leaks
> pmf.close();
> }
> }
> pmf = null;
> prop = null;
> }
> assert(!isActiveTransaction());
> shutdown();
> // Always want to re-create pm as we don't know if it were created by
> the
> // most recent instance of the pmf
> pm = null;
> directSql = null;
> expressionProxy = null;
> openTrasactionCalls = 0;
> currentTransaction = null;
> transactionStatus = TXN_STATUS.NO_STATE;
> initialize(propsFromConf);
> String partitionValidationRegex =
> MetastoreConf.getVar(this.conf,
> ConfVars.PARTITION_NAME_WHITELIST_PATTERN);
> if (partitionValidationRegex != null &&
> !partitionValidationRegex.isEmpty()) {
> partitionValidationPattern =
> Pattern.compile(partitionValidationRegex);
> } else {
> partitionValidationPattern = null;
> }
> // Note, if metrics have not been initialized this will return null,
> which means we aren't
> // using metrics. Thus we should always check whether this is non-null
> before using.
> MetricRegistry registry = Metrics.getRegistry();
> if (registry != null) {
> directSqlErrors =
> Metrics.getOrCreateCounter(MetricsConstants.DIRECTSQL_ERRORS);
> }
> this.batchSize = MetastoreConf.getIntVar(conf,
> ConfVars.RAWSTORE_PARTITION_BATCH_SIZE);
> if (!isInitialized) {
> throw new RuntimeException(
> "Unable to create persistence manager. Check dss.log for details");
> } else {
> LOG.debug("Initialized ObjectStore");
> }
> } finally {
> pmfPropLock.unlock();
> }
> }
> {code}
> The {{pmfPropLock}} is a static object and it disallows any other new
> connection to HMS which is trying to instantiate ObjectStore. We should
> either remove the lock or reduce the scope of the lock so that it is held for
> a very small amount of time.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20716) Set default value for hive.cbo.stats.correlated.multi.key.joins to true
[
https://issues.apache.org/jira/browse/HIVE-20716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16652573#comment-16652573
]
Hive QA commented on HIVE-20716:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
24s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
31s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
21s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
56s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
35s{color} | {color:blue} common in master has 65 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
6s{color} | {color:blue} ql in master has 2318 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
11s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
10s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
7s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 27m 1s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-14509/dev-support/hive-personality.sh
|
| git revision | master / e39a198 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: common ql U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-14509/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Set default value for hive.cbo.stats.correlated.multi.key.joins to true
> ---
>
> Key: HIVE-20716
> URL: https://issues.apache.org/jira/browse/HIVE-20716
> Project: Hive
> Issue Type: Bug
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-20716.01.patch, HIVE-20716.02.patch,
> HIVE-20716.03.patch, HIVE-20716.03.patch, HIVE-20716.patch
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20701) Allow HiveStreaming to receive a key value to commit atomically together with the transaction
[ https://issues.apache.org/jira/browse/HIVE-20701?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jaume M updated HIVE-20701: --- Status: Open (was: Patch Available) > Allow HiveStreaming to receive a key value to commit atomically together with > the transaction > - > > Key: HIVE-20701 > URL: https://issues.apache.org/jira/browse/HIVE-20701 > Project: Hive > Issue Type: Improvement >Reporter: Jaume M >Assignee: Jaume M >Priority: Major > Attachments: HIVE-20701.1.patch, HIVE-20701.2.patch, > HIVE-20701.3.patch, HIVE-20701.4.patch, HIVE-20701.5.patch > > > Following up with HIVE-20538 it'd be nice to be able to use this feature with > hive streaming -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20701) Allow HiveStreaming to receive a key value to commit atomically together with the transaction
[ https://issues.apache.org/jira/browse/HIVE-20701?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jaume M updated HIVE-20701: --- Attachment: HIVE-20701.5.patch Status: Patch Available (was: Open) > Allow HiveStreaming to receive a key value to commit atomically together with > the transaction > - > > Key: HIVE-20701 > URL: https://issues.apache.org/jira/browse/HIVE-20701 > Project: Hive > Issue Type: Improvement >Reporter: Jaume M >Assignee: Jaume M >Priority: Major > Attachments: HIVE-20701.1.patch, HIVE-20701.2.patch, > HIVE-20701.3.patch, HIVE-20701.4.patch, HIVE-20701.5.patch > > > Following up with HIVE-20538 it'd be nice to be able to use this feature with > hive streaming -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18767) Some alterPartitions invocations throw 'NumberFormatException: null'
[
https://issues.apache.org/jira/browse/HIVE-18767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16652537#comment-16652537
]
Hive QA commented on HIVE-18767:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12944156/HIVE-18767.4-branch-3.1.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 104 failed/errored test(s), 14385 tests
executed
*Failed tests:*
{noformat}
TestAddPartitions - did not produce a TEST-*.xml file (likely timed out)
(batchId=226)
TestAddPartitionsFromPartSpec - did not produce a TEST-*.xml file (likely timed
out) (batchId=228)
TestAdminUser - did not produce a TEST-*.xml file (likely timed out)
(batchId=234)
TestAggregateStatsCache - did not produce a TEST-*.xml file (likely timed out)
(batchId=228)
TestAlterPartitions - did not produce a TEST-*.xml file (likely timed out)
(batchId=228)
TestAppendPartitions - did not produce a TEST-*.xml file (likely timed out)
(batchId=228)
TestBeeLineDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=271)
TestCachedStore - did not produce a TEST-*.xml file (likely timed out)
(batchId=234)
TestCatalogCaching - did not produce a TEST-*.xml file (likely timed out)
(batchId=234)
TestCatalogNonDefaultClient - did not produce a TEST-*.xml file (likely timed
out) (batchId=226)
TestCatalogNonDefaultSvr - did not produce a TEST-*.xml file (likely timed out)
(batchId=234)
TestCatalogOldClient - did not produce a TEST-*.xml file (likely timed out)
(batchId=226)
TestCatalogs - did not produce a TEST-*.xml file (likely timed out)
(batchId=228)
TestCheckConstraint - did not produce a TEST-*.xml file (likely timed out)
(batchId=226)
TestDataSourceProviderFactory - did not produce a TEST-*.xml file (likely timed
out) (batchId=236)
TestDatabaseName - did not produce a TEST-*.xml file (likely timed out)
(batchId=195)
TestDatabases - did not produce a TEST-*.xml file (likely timed out)
(batchId=228)
TestDeadline - did not produce a TEST-*.xml file (likely timed out)
(batchId=234)
TestDefaultConstraint - did not produce a TEST-*.xml file (likely timed out)
(batchId=228)
TestDropPartitions - did not produce a TEST-*.xml file (likely timed out)
(batchId=226)
TestDummy - did not produce a TEST-*.xml file (likely timed out) (batchId=271)
TestEmbeddedHiveMetaStore - did not produce a TEST-*.xml file (likely timed
out) (batchId=229)
TestExchangePartitions - did not produce a TEST-*.xml file (likely timed out)
(batchId=228)
TestFMSketchSerialization - did not produce a TEST-*.xml file (likely timed
out) (batchId=236)
TestFilterHooks - did not produce a TEST-*.xml file (likely timed out)
(batchId=226)
TestForeignKey - did not produce a TEST-*.xml file (likely timed out)
(batchId=228)
TestFunctions - did not produce a TEST-*.xml file (likely timed out)
(batchId=226)
TestGetPartitions - did not produce a TEST-*.xml file (likely timed out)
(batchId=226)
TestGetPartitionsUsingProjection - did not produce a TEST-*.xml file (likely
timed out) (batchId=228)
TestGetTableMeta - did not produce a TEST-*.xml file (likely timed out)
(batchId=226)
TestHLLNoBias - did not produce a TEST-*.xml file (likely timed out)
(batchId=236)
TestHLLSerialization - did not produce a TEST-*.xml file (likely timed out)
(batchId=236)
TestHdfsUtils - did not produce a TEST-*.xml file (likely timed out)
(batchId=234)
TestHiveAlterHandler - did not produce a TEST-*.xml file (likely timed out)
(batchId=226)
TestHiveMetaStoreGetMetaConf - did not produce a TEST-*.xml file (likely timed
out) (batchId=236)
TestHiveMetaStorePartitionSpecs - did not produce a TEST-*.xml file (likely
timed out) (batchId=228)
TestHiveMetaStoreSchemaMethods - did not produce a TEST-*.xml file (likely
timed out) (batchId=234)
TestHiveMetaStoreTimeout - did not produce a TEST-*.xml file (likely timed out)
(batchId=236)
TestHiveMetaStoreTxns - did not produce a TEST-*.xml file (likely timed out)
(batchId=236)
TestHiveMetaStoreWithEnvironmentContext - did not produce a TEST-*.xml file
(likely timed out) (batchId=231)
TestHiveMetastoreCli - did not produce a TEST-*.xml file (likely timed out)
(batchId=226)
TestHyperLogLog - did not produce a TEST-*.xml file (likely timed out)
(batchId=236)
TestHyperLogLogDense - did not produce a TEST-*.xml file (likely timed out)
(batchId=236)
TestHyperLogLogMerge - did not produce a TEST-*.xml file (likely timed out)
(batchId=236)
TestHyperLogLogSparse - did not produce a TEST-*.xml file (likely timed out)
(batchId=236)
TestJSONMessageDeserializer - did not produce a TEST-*.xml file (likely timed
out) (batchId=234)
TestListPartitions - did not produce a TEST-*.xml file (likely timed out)
(batchId=226)
TestLockRequestBuilder - did not produce a TEST-*.xml file (likely timed out)
(batchId=226)
TestMarkPartition - did not produce a TEST-*.xml file
[jira] [Commented] (HIVE-20756) Disable SARG leaf creation for date column until ORC-135
[ https://issues.apache.org/jira/browse/HIVE-20756?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16652458#comment-16652458 ] Gopal V commented on HIVE-20756: The patch will fail tests unless it is renamed to have branch-1.patch. > Disable SARG leaf creation for date column until ORC-135 > > > Key: HIVE-20756 > URL: https://issues.apache.org/jira/browse/HIVE-20756 > Project: Hive > Issue Type: Bug >Affects Versions: 2.1.1 >Reporter: Chiran Ravani >Assignee: Chiran Ravani >Priority: Major > Attachments: HIVE-20756.1.patch > > > Until ORC-135 is committed and orc version is updated in hive, disable SARG > creation for date columns in hive. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20307) Add support for filterspec to the getPartitions with projection API
[ https://issues.apache.org/jira/browse/HIVE-20307?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vihang Karajgaonkar updated HIVE-20307: --- Resolution: Fixed Fix Version/s: 4.0.0 Status: Resolved (was: Patch Available) Patch merged into master branch. Thanks for the review [~asherman] > Add support for filterspec to the getPartitions with projection API > --- > > Key: HIVE-20307 > URL: https://issues.apache.org/jira/browse/HIVE-20307 > Project: Hive > Issue Type: Sub-task >Reporter: Vihang Karajgaonkar >Assignee: Vihang Karajgaonkar >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-20307.01.patch, HIVE-20307.02.patch, > HIVE-20307.03.patch, HIVE-20307.04.patch, HIVE-20307.05.patch > > > Implement the BY_EXPR, BY_NAMES and BY_VALUES filter modes for the projection > API to filter the partitions returned as discussed in the design doc -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (HIVE-20740) Remove global lock in ObjectStore.setConf method
[
https://issues.apache.org/jira/browse/HIVE-20740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16652282#comment-16652282
]
Vihang Karajgaonkar edited comment on HIVE-20740 at 10/16/18 9:07 PM:
--
Turns out there was a race condition as well in the code. The {{getPMF}} is a
{{static synchronized}} method (takes a lock on ObjectStore.class) while the
{{setConf}} method takes a lock on {{pmfPropLock}} object. So in theory it is
possible that some thread which calls {{getPMF}} and is updating {{pmf}}
instance while another thread calls {{setConf}} also tries to update {{pmf}} as
can be seen from the code above. The patch moves all the updates to the {{pmf}}
and {{prop}} static variables into one method where they are protected using
read and write locks to optimize for the common case of datasource properties
not getting updated often.
was (Author: vihangk1):
Turns out there was a race condition as well in the code. The {{getPMF}} is a
{{static synchronized}} method (takes a lock on ObjectStore.class) while the
{{setConf}} method takes a lock on {{pmfPropLock}} method. So in theory it is
possible that some thread which calls {{getPMF}} is updating {{pmf}} instance
while another thread which calls {{setConf}} is also updating {{pmf}} as can be
seen from the code above. The patch moves all the updates to the {{pmf}} and
{{prop}} static variables into one method where they are protected using read
locks and write locks to optimize for the common case of datasource properties
not getting updated often.
> Remove global lock in ObjectStore.setConf method
>
>
> Key: HIVE-20740
> URL: https://issues.apache.org/jira/browse/HIVE-20740
> Project: Hive
> Issue Type: Improvement
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
>Priority: Major
> Attachments: HIVE-20740.01.patch
>
>
> The ObjectStore#setConf method has a global lock which can block other
> clients in concurrent workloads.
> {code}
> @Override
> @SuppressWarnings("nls")
> public void setConf(Configuration conf) {
> // Although an instance of ObjectStore is accessed by one thread, there
> may
> // be many threads with ObjectStore instances. So the static variables
> // pmf and prop need to be protected with locks.
> pmfPropLock.lock();
> try {
> isInitialized = false;
> this.conf = conf;
> this.areTxnStatsSupported = MetastoreConf.getBoolVar(conf,
> ConfVars.HIVE_TXN_STATS_ENABLED);
> configureSSL(conf);
> Properties propsFromConf = getDataSourceProps(conf);
> boolean propsChanged = !propsFromConf.equals(prop);
> if (propsChanged) {
> if (pmf != null){
> clearOutPmfClassLoaderCache(pmf);
> if (!forTwoMetastoreTesting) {
> // close the underlying connection pool to avoid leaks
> pmf.close();
> }
> }
> pmf = null;
> prop = null;
> }
> assert(!isActiveTransaction());
> shutdown();
> // Always want to re-create pm as we don't know if it were created by
> the
> // most recent instance of the pmf
> pm = null;
> directSql = null;
> expressionProxy = null;
> openTrasactionCalls = 0;
> currentTransaction = null;
> transactionStatus = TXN_STATUS.NO_STATE;
> initialize(propsFromConf);
> String partitionValidationRegex =
> MetastoreConf.getVar(this.conf,
> ConfVars.PARTITION_NAME_WHITELIST_PATTERN);
> if (partitionValidationRegex != null &&
> !partitionValidationRegex.isEmpty()) {
> partitionValidationPattern =
> Pattern.compile(partitionValidationRegex);
> } else {
> partitionValidationPattern = null;
> }
> // Note, if metrics have not been initialized this will return null,
> which means we aren't
> // using metrics. Thus we should always check whether this is non-null
> before using.
> MetricRegistry registry = Metrics.getRegistry();
> if (registry != null) {
> directSqlErrors =
> Metrics.getOrCreateCounter(MetricsConstants.DIRECTSQL_ERRORS);
> }
> this.batchSize = MetastoreConf.getIntVar(conf,
> ConfVars.RAWSTORE_PARTITION_BATCH_SIZE);
> if (!isInitialized) {
> throw new RuntimeException(
> "Unable to create persistence manager. Check dss.log for details");
> } else {
> LOG.debug("Initialized ObjectStore");
> }
> } finally {
> pmfPropLock.unlock();
> }
> }
> {code}
> The {{pmfPropLock}} is a static object and it disallows any other new
> connection to HMS which is trying to instantiate ObjectStore. We should
> either remove the lock or reduce the scope of the lock so that it is held for
> a very small
[jira] [Commented] (HIVE-20307) Add support for filterspec to the getPartitions with projection API
[ https://issues.apache.org/jira/browse/HIVE-20307?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16652438#comment-16652438 ] Andrew Sherman commented on HIVE-20307: --- +1 LGTM > Add support for filterspec to the getPartitions with projection API > --- > > Key: HIVE-20307 > URL: https://issues.apache.org/jira/browse/HIVE-20307 > Project: Hive > Issue Type: Sub-task >Reporter: Vihang Karajgaonkar >Assignee: Vihang Karajgaonkar >Priority: Major > Attachments: HIVE-20307.01.patch, HIVE-20307.02.patch, > HIVE-20307.03.patch, HIVE-20307.04.patch, HIVE-20307.05.patch > > > Implement the BY_EXPR, BY_NAMES and BY_VALUES filter modes for the projection > API to filter the partitions returned as discussed in the design doc -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18767) Some alterPartitions invocations throw 'NumberFormatException: null'
[
https://issues.apache.org/jira/browse/HIVE-18767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16652425#comment-16652425
]
Hive QA commented on HIVE-18767:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 14s{color}
| {color:red}
/data/hiveptest/logs/PreCommit-HIVE-Build-14508/patches/PreCommit-HIVE-Build-14508.patch
does not apply to master. Rebase required? Wrong Branch? See
http://cwiki.apache.org/confluence/display/Hive/HowToContribute for help.
{color} |
\\
\\
|| Subsystem || Report/Notes ||
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-14508/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Some alterPartitions invocations throw 'NumberFormatException: null'
>
>
> Key: HIVE-18767
> URL: https://issues.apache.org/jira/browse/HIVE-18767
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 2.3.3, 3.1.0, 4.0.0, 3.2.0
>Reporter: Yuming Wang
>Assignee: Mass Dosage
>Priority: Major
> Fix For: 2.3.3, 4.0.0
>
> Attachments: HIVE-18767-branch-2.3.patch, HIVE-18767-branch-2.patch,
> HIVE-18767-branch-3.1.patch, HIVE-18767-branch-3.patch, HIVE-18767.1.patch,
> HIVE-18767.2-branch-2.3.patch, HIVE-18767.2-branch-2.patch,
> HIVE-18767.2-branch-3.1.patch, HIVE-18767.2.patch,
> HIVE-18767.3-branch-3.1.patch, HIVE-18767.3.patch,
> HIVE-18767.4-branch-3.1.patch, HIVE-18767.4.patch, HIVE-18767.5.patch,
> HIVE-18767.6.patch
>
>
> Error messages:
> {noformat}
> [info] Cause: java.lang.NumberFormatException: null
> [info] at java.lang.Long.parseLong(Long.java:552)
> [info] at java.lang.Long.parseLong(Long.java:631)
> [info] at
> org.apache.hadoop.hive.metastore.MetaStoreUtils.isFastStatsSame(MetaStoreUtils.java:315)
> [info] at
> org.apache.hadoop.hive.metastore.HiveAlterHandler.alterPartitions(HiveAlterHandler.java:605)
> [info] at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.alter_partitions_with_environment_context(HiveMetaStore.java:3837)
> [info] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> [info] at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> [info] at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> [info] at java.lang.reflect.Method.invoke(Method.java:498)
> [info] at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:148)
> [info] at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:107)
> [info] at
> com.sun.proxy.$Proxy23.alter_partitions_with_environment_context(Unknown
> Source)
> [info] at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.alter_partitions(HiveMetaStoreClient.java:1527)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20731) keystore file in JdbcStorageHandler should be authorized
[ https://issues.apache.org/jira/browse/HIVE-20731?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Dai updated HIVE-20731: -- Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Patch pushed to master. Thanks Thejas for review! > keystore file in JdbcStorageHandler should be authorized > > > Key: HIVE-20731 > URL: https://issues.apache.org/jira/browse/HIVE-20731 > Project: Hive > Issue Type: Improvement > Components: StorageHandler >Reporter: Daniel Dai >Assignee: Daniel Dai >Priority: Major > Attachments: HIVE-20731.1.patch, HIVE-20731.2.patch, > HIVE-20731.3.patch, HIVE-20731.4.patch > > > The keystore file introduced in HIVE-20651 shall be authorized with > configured authorizer. Otherwise any user knows the keystore file location > can access the password. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20617) Fix type of constants in IN expressions to have correct type
[
https://issues.apache.org/jira/browse/HIVE-20617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16652422#comment-16652422
]
Hive QA commented on HIVE-20617:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12944154/HIVE-20617.08.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/14507/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/14507/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-14507/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Tests exited with: Exception: Patch URL
https://issues.apache.org/jira/secure/attachment/12944154/HIVE-20617.08.patch
was found in seen patch url's cache and a test was probably run already on it.
Aborting...
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12944154 - PreCommit-HIVE-Build
> Fix type of constants in IN expressions to have correct type
>
>
> Key: HIVE-20617
> URL: https://issues.apache.org/jira/browse/HIVE-20617
> Project: Hive
> Issue Type: Bug
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Attachments: HIVE-20617.01.patch, HIVE-20617.02.patch,
> HIVE-20617.03.patch, HIVE-20617.05.patch, HIVE-20617.06.patch,
> HIVE-20617.07.patch, HIVE-20617.08.patch, HIVE-20617.08.patch,
> HIVE-20617.08.patch, HIVE-20617.08.patch
>
>
> In statements like {{struct(a,b) IN (const struct('x','y'), ... )}} the
> comparision in UDFIn may fail because if a or b is of char/varchar type the
> constants will retain string type - especially after PointlookupOptimizer
> compaction.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20752) In case of LLAP start failure add info how to find YARN logs
[
https://issues.apache.org/jira/browse/HIVE-20752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16652421#comment-16652421
]
Hive QA commented on HIVE-20752:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12944139/HIVE-20752.01.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/14506/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/14506/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-14506/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2018-10-16 20:48:20.569
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-14506/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2018-10-16 20:48:20.573
+ cd apache-github-source-source
+ git fetch origin
+ git reset --hard HEAD
HEAD is now at d7be4b9 HIVE-17231:
ColumnizedDeleteEventRegistry.DeleteReaderValue optimization (Eugene Koifman,
reviewed by Gopal V)
+ git clean -f -d
Removing standalone-metastore/metastore-server/src/gen/
+ git checkout master
Already on 'master'
Your branch is up-to-date with 'origin/master'.
+ git reset --hard origin/master
HEAD is now at d7be4b9 HIVE-17231:
ColumnizedDeleteEventRegistry.DeleteReaderValue optimization (Eugene Koifman,
reviewed by Gopal V)
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2018-10-16 20:48:21.660
+ rm -rf ../yetus_PreCommit-HIVE-Build-14506
+ mkdir ../yetus_PreCommit-HIVE-Build-14506
+ git gc
+ cp -R . ../yetus_PreCommit-HIVE-Build-14506
+ mkdir /data/hiveptest/logs/PreCommit-HIVE-Build-14506/yetus
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh
/data/hiveptest/working/scratch/build.patch
error:
a/llap-server/src/java/org/apache/hadoop/hive/llap/cli/LlapStatusServiceDriver.java:
does not exist in index
Going to apply patch with: git apply -p1
+ [[ maven == \m\a\v\e\n ]]
+ rm -rf /data/hiveptest/working/maven/org/apache/hive
+ mvn -B clean install -DskipTests -T 4 -q
-Dmaven.repo.local=/data/hiveptest/working/maven
protoc-jar: executing: [/tmp/protoc8057290364756599564.exe, --version]
libprotoc 2.5.0
protoc-jar: executing: [/tmp/protoc8057290364756599564.exe,
-I/data/hiveptest/working/apache-github-source-source/standalone-metastore/metastore-common/src/main/protobuf/org/apache/hadoop/hive/metastore,
--java_out=/data/hiveptest/working/apache-github-source-source/standalone-metastore/metastore-common/target/generated-sources,
/data/hiveptest/working/apache-github-source-source/standalone-metastore/metastore-common/src/main/protobuf/org/apache/hadoop/hive/metastore/metastore.proto]
ANTLR Parser Generator Version 3.5.2
[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-remote-resources-plugin:1.5:process
(process-resource-bundles) on project hive-shims: Execution
process-resource-bundles of goal
org.apache.maven.plugins:maven-remote-resources-plugin:1.5:process failed.
ConcurrentModificationException -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e
switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please
read the following articles:
[ERROR] [Help 1]
http://cwiki.apache.org/confluence/display/MAVEN/PluginExecutionException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR] mvn -rf :hive-shims
+ result=1
+ '[' 1 -ne 0 ']'
+ rm -rf yetus_PreCommit-HIVE-Build-14506
+ exit 1
'
{noformat}
This message is automatically generated.
ATTACHMENT ID:
[jira] [Commented] (HIVE-20617) Fix type of constants in IN expressions to have correct type
[
https://issues.apache.org/jira/browse/HIVE-20617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16652411#comment-16652411
]
Hive QA commented on HIVE-20617:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12944154/HIVE-20617.08.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 5 failed/errored test(s), 15085 tests
executed
*Failed tests:*
{noformat}
TestMiniDruidCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=196)
[druidmini_masking.q,druidmini_test1.q,druidkafkamini_basic.q,druidmini_joins.q,druid_timestamptz.q]
org.apache.hadoop.hive.metastore.TestMetaStoreMetrics.testConnections
(batchId=239)
org.apache.hadoop.hive.metastore.TestMetaStoreMetrics.testMetaDataCounts
(batchId=239)
org.apache.hadoop.hive.metastore.TestMetaStoreMetrics.testMethodCounts
(batchId=239)
org.apache.hive.jdbc.TestJdbcWithMiniLlapArrow.testComplexQuery (batchId=253)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/14505/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/14505/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-14505/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 5 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12944154 - PreCommit-HIVE-Build
> Fix type of constants in IN expressions to have correct type
>
>
> Key: HIVE-20617
> URL: https://issues.apache.org/jira/browse/HIVE-20617
> Project: Hive
> Issue Type: Bug
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Attachments: HIVE-20617.01.patch, HIVE-20617.02.patch,
> HIVE-20617.03.patch, HIVE-20617.05.patch, HIVE-20617.06.patch,
> HIVE-20617.07.patch, HIVE-20617.08.patch, HIVE-20617.08.patch,
> HIVE-20617.08.patch, HIVE-20617.08.patch
>
>
> In statements like {{struct(a,b) IN (const struct('x','y'), ... )}} the
> comparision in UDFIn may fail because if a or b is of char/varchar type the
> constants will retain string type - especially after PointlookupOptimizer
> compaction.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (HIVE-20740) Remove global lock in ObjectStore.setConf method
[
https://issues.apache.org/jira/browse/HIVE-20740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16652401#comment-16652401
]
Vihang Karajgaonkar edited comment on HIVE-20740 at 10/16/18 8:38 PM:
--
[~asherman] [~pvary] [~alangates] Can you please help review?
was (Author: vihangk1):
[~asherman] [~pvary] Can you please help review?
> Remove global lock in ObjectStore.setConf method
>
>
> Key: HIVE-20740
> URL: https://issues.apache.org/jira/browse/HIVE-20740
> Project: Hive
> Issue Type: Improvement
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
>Priority: Major
> Attachments: HIVE-20740.01.patch
>
>
> The ObjectStore#setConf method has a global lock which can block other
> clients in concurrent workloads.
> {code}
> @Override
> @SuppressWarnings("nls")
> public void setConf(Configuration conf) {
> // Although an instance of ObjectStore is accessed by one thread, there
> may
> // be many threads with ObjectStore instances. So the static variables
> // pmf and prop need to be protected with locks.
> pmfPropLock.lock();
> try {
> isInitialized = false;
> this.conf = conf;
> this.areTxnStatsSupported = MetastoreConf.getBoolVar(conf,
> ConfVars.HIVE_TXN_STATS_ENABLED);
> configureSSL(conf);
> Properties propsFromConf = getDataSourceProps(conf);
> boolean propsChanged = !propsFromConf.equals(prop);
> if (propsChanged) {
> if (pmf != null){
> clearOutPmfClassLoaderCache(pmf);
> if (!forTwoMetastoreTesting) {
> // close the underlying connection pool to avoid leaks
> pmf.close();
> }
> }
> pmf = null;
> prop = null;
> }
> assert(!isActiveTransaction());
> shutdown();
> // Always want to re-create pm as we don't know if it were created by
> the
> // most recent instance of the pmf
> pm = null;
> directSql = null;
> expressionProxy = null;
> openTrasactionCalls = 0;
> currentTransaction = null;
> transactionStatus = TXN_STATUS.NO_STATE;
> initialize(propsFromConf);
> String partitionValidationRegex =
> MetastoreConf.getVar(this.conf,
> ConfVars.PARTITION_NAME_WHITELIST_PATTERN);
> if (partitionValidationRegex != null &&
> !partitionValidationRegex.isEmpty()) {
> partitionValidationPattern =
> Pattern.compile(partitionValidationRegex);
> } else {
> partitionValidationPattern = null;
> }
> // Note, if metrics have not been initialized this will return null,
> which means we aren't
> // using metrics. Thus we should always check whether this is non-null
> before using.
> MetricRegistry registry = Metrics.getRegistry();
> if (registry != null) {
> directSqlErrors =
> Metrics.getOrCreateCounter(MetricsConstants.DIRECTSQL_ERRORS);
> }
> this.batchSize = MetastoreConf.getIntVar(conf,
> ConfVars.RAWSTORE_PARTITION_BATCH_SIZE);
> if (!isInitialized) {
> throw new RuntimeException(
> "Unable to create persistence manager. Check dss.log for details");
> } else {
> LOG.debug("Initialized ObjectStore");
> }
> } finally {
> pmfPropLock.unlock();
> }
> }
> {code}
> The {{pmfPropLock}} is a static object and it disallows any other new
> connection to HMS which is trying to instantiate ObjectStore. We should
> either remove the lock or reduce the scope of the lock so that it is held for
> a very small amount of time.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20740) Remove global lock in ObjectStore.setConf method
[
https://issues.apache.org/jira/browse/HIVE-20740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16652401#comment-16652401
]
Vihang Karajgaonkar commented on HIVE-20740:
[~asherman] [~pvary] Can you please help review?
> Remove global lock in ObjectStore.setConf method
>
>
> Key: HIVE-20740
> URL: https://issues.apache.org/jira/browse/HIVE-20740
> Project: Hive
> Issue Type: Improvement
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
>Priority: Major
> Attachments: HIVE-20740.01.patch
>
>
> The ObjectStore#setConf method has a global lock which can block other
> clients in concurrent workloads.
> {code}
> @Override
> @SuppressWarnings("nls")
> public void setConf(Configuration conf) {
> // Although an instance of ObjectStore is accessed by one thread, there
> may
> // be many threads with ObjectStore instances. So the static variables
> // pmf and prop need to be protected with locks.
> pmfPropLock.lock();
> try {
> isInitialized = false;
> this.conf = conf;
> this.areTxnStatsSupported = MetastoreConf.getBoolVar(conf,
> ConfVars.HIVE_TXN_STATS_ENABLED);
> configureSSL(conf);
> Properties propsFromConf = getDataSourceProps(conf);
> boolean propsChanged = !propsFromConf.equals(prop);
> if (propsChanged) {
> if (pmf != null){
> clearOutPmfClassLoaderCache(pmf);
> if (!forTwoMetastoreTesting) {
> // close the underlying connection pool to avoid leaks
> pmf.close();
> }
> }
> pmf = null;
> prop = null;
> }
> assert(!isActiveTransaction());
> shutdown();
> // Always want to re-create pm as we don't know if it were created by
> the
> // most recent instance of the pmf
> pm = null;
> directSql = null;
> expressionProxy = null;
> openTrasactionCalls = 0;
> currentTransaction = null;
> transactionStatus = TXN_STATUS.NO_STATE;
> initialize(propsFromConf);
> String partitionValidationRegex =
> MetastoreConf.getVar(this.conf,
> ConfVars.PARTITION_NAME_WHITELIST_PATTERN);
> if (partitionValidationRegex != null &&
> !partitionValidationRegex.isEmpty()) {
> partitionValidationPattern =
> Pattern.compile(partitionValidationRegex);
> } else {
> partitionValidationPattern = null;
> }
> // Note, if metrics have not been initialized this will return null,
> which means we aren't
> // using metrics. Thus we should always check whether this is non-null
> before using.
> MetricRegistry registry = Metrics.getRegistry();
> if (registry != null) {
> directSqlErrors =
> Metrics.getOrCreateCounter(MetricsConstants.DIRECTSQL_ERRORS);
> }
> this.batchSize = MetastoreConf.getIntVar(conf,
> ConfVars.RAWSTORE_PARTITION_BATCH_SIZE);
> if (!isInitialized) {
> throw new RuntimeException(
> "Unable to create persistence manager. Check dss.log for details");
> } else {
> LOG.debug("Initialized ObjectStore");
> }
> } finally {
> pmfPropLock.unlock();
> }
> }
> {code}
> The {{pmfPropLock}} is a static object and it disallows any other new
> connection to HMS which is trying to instantiate ObjectStore. We should
> either remove the lock or reduce the scope of the lock so that it is held for
> a very small amount of time.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20617) Fix type of constants in IN expressions to have correct type
[
https://issues.apache.org/jira/browse/HIVE-20617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16652375#comment-16652375
]
Hive QA commented on HIVE-20617:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
47s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
8s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
40s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
1s{color} | {color:blue} ql in master has 2318 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
2s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
7s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
42s{color} | {color:red} ql: The patch generated 5 new + 85 unchanged - 11
fixed = 90 total (was 96) {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 2 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 4m
24s{color} | {color:red} ql generated 4 new + 2314 unchanged - 4 fixed = 2318
total (was 2318) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
57s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 25m 10s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:ql |
| |
org.apache.hadoop.hive.ql.parse.TypeCheckProcFactory$DefaultExprProcessor.interpretConstantAsPrimitive(PrimitiveTypeInfo,
Object, TypeInfo) invokes inefficient new Byte(String) constructor; use
Byte.valueOf(String) instead At TypeCheckProcFactory.java:new Byte(String)
constructor; use Byte.valueOf(String) instead At
TypeCheckProcFactory.java:[line 1391] |
| |
org.apache.hadoop.hive.ql.parse.TypeCheckProcFactory$DefaultExprProcessor.interpretConstantAsPrimitive(PrimitiveTypeInfo,
Object, TypeInfo) invokes inefficient new Integer(String) constructor; use
Integer.valueOf(String) instead At TypeCheckProcFactory.java:new
Integer(String) constructor; use Integer.valueOf(String) instead At
TypeCheckProcFactory.java:[line 1383] |
| |
org.apache.hadoop.hive.ql.parse.TypeCheckProcFactory$DefaultExprProcessor.interpretConstantAsPrimitive(PrimitiveTypeInfo,
Object, TypeInfo) invokes inefficient new Long(String) constructor; use
Long.valueOf(String) instead At TypeCheckProcFactory.java:new Long(String)
constructor; use Long.valueOf(String) instead At
TypeCheckProcFactory.java:[line 1385] |
| |
org.apache.hadoop.hive.ql.parse.TypeCheckProcFactory$DefaultExprProcessor.interpretConstantAsPrimitive(PrimitiveTypeInfo,
Object, TypeInfo) invokes inefficient new Short(String) constructor; use
Short.valueOf(String) instead At TypeCheckProcFactory.java:new Short(String)
constructor; use Short.valueOf(String) instead At
TypeCheckProcFactory.java:[line 1393] |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-14505/dev-support/hive-personality.sh
|
| git revision | master / d7be4b9 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-14505/yetus/diff-checkstyle-ql.txt
|
| whitespace |
http://104.198.109.242/logs//PreCommit-HIVE-Build-14505/yetus/whitespace-eol.txt
|
|
[jira] [Updated] (HIVE-20748) Disable materialized view rewriting when plan pattern is not allowed
[ https://issues.apache.org/jira/browse/HIVE-20748?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-20748: --- Attachment: HIVE-20748.01.patch > Disable materialized view rewriting when plan pattern is not allowed > > > Key: HIVE-20748 > URL: https://issues.apache.org/jira/browse/HIVE-20748 > Project: Hive > Issue Type: Bug > Components: Materialized views >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Major > Attachments: HIVE-20748.01.patch, HIVE-20748.patch > > > For instance, currently rewriting algorithm does not support some operators. > Or we cannot have non-deterministic function in the MV definition. In those > cases, we should fail either when we try to create the MV with rewriting > enabled, or when when we enable the rewriting for a MV already created. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20659) Update commons-compress to 1.18 due to security issues
[ https://issues.apache.org/jira/browse/HIVE-20659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16652368#comment-16652368 ] Andrew Sherman commented on HIVE-20659: --- +1 LGTM > Update commons-compress to 1.18 due to security issues > -- > > Key: HIVE-20659 > URL: https://issues.apache.org/jira/browse/HIVE-20659 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 1.2.1, 3.0.0, 2.3.2, 3.1.0 >Reporter: Jörn Franke >Assignee: Bharathkrishna Guruvayoor Murali >Priority: Critical > Attachments: HIVE-20659.1.patch > > > Currently most Hive version depends on commons-compress 1.9 or 1.4. Those > versions have several security issues: > [https://commons.apache.org/proper/commons-compress/security-reports.html] > I propose to upgrade all commons-compress dependencies in all Hive > (sub-)projects to at least 1.18. This will also make it easier for future > extensions to Hive (serde, udfs, etc.) that have dependencies to > commons-compress (e.g. [https://github.com/zuinnote/hadoopoffice/wiki)] to > integrate into Hive without upgrading the commons-compress library manually > in the Hive lib folder. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20703) Put dynamic sort partition optimization under cost based decision
[ https://issues.apache.org/jira/browse/HIVE-20703?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-20703: --- Status: Patch Available (was: Open) > Put dynamic sort partition optimization under cost based decision > - > > Key: HIVE-20703 > URL: https://issues.apache.org/jira/browse/HIVE-20703 > Project: Hive > Issue Type: Improvement > Components: Query Planning >Reporter: Vineet Garg >Assignee: Vineet Garg >Priority: Major > Attachments: HIVE-20703.1.patch, HIVE-20703.2.patch, > HIVE-20703.3.patch, HIVE-20703.4.patch, HIVE-20703.5.patch, > HIVE-20703.6.patch, HIVE-20703.7.patch, HIVE-20703.8.patch, HIVE-20703.9.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20703) Put dynamic sort partition optimization under cost based decision
[ https://issues.apache.org/jira/browse/HIVE-20703?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-20703: --- Status: Open (was: Patch Available) > Put dynamic sort partition optimization under cost based decision > - > > Key: HIVE-20703 > URL: https://issues.apache.org/jira/browse/HIVE-20703 > Project: Hive > Issue Type: Improvement > Components: Query Planning >Reporter: Vineet Garg >Assignee: Vineet Garg >Priority: Major > Attachments: HIVE-20703.1.patch, HIVE-20703.2.patch, > HIVE-20703.3.patch, HIVE-20703.4.patch, HIVE-20703.5.patch, > HIVE-20703.6.patch, HIVE-20703.7.patch, HIVE-20703.8.patch, HIVE-20703.9.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20703) Put dynamic sort partition optimization under cost based decision
[ https://issues.apache.org/jira/browse/HIVE-20703?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-20703: --- Attachment: HIVE-20703.9.patch > Put dynamic sort partition optimization under cost based decision > - > > Key: HIVE-20703 > URL: https://issues.apache.org/jira/browse/HIVE-20703 > Project: Hive > Issue Type: Improvement > Components: Query Planning >Reporter: Vineet Garg >Assignee: Vineet Garg >Priority: Major > Attachments: HIVE-20703.1.patch, HIVE-20703.2.patch, > HIVE-20703.3.patch, HIVE-20703.4.patch, HIVE-20703.5.patch, > HIVE-20703.6.patch, HIVE-20703.7.patch, HIVE-20703.8.patch, HIVE-20703.9.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20708) Load (dumped) an external table as an external table on target with the same location as on the source
[
https://issues.apache.org/jira/browse/HIVE-20708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16652300#comment-16652300
]
Hive QA commented on HIVE-20708:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12944081/HIVE-20708.01
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 15085 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[repl_2_exim_basic]
(batchId=84)
org.apache.hadoop.hive.ql.parse.TestReplicationScenariosAcrossInstances.testDumpExternalTableSetFalse
(batchId=245)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/14504/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/14504/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-14504/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12944081 - PreCommit-HIVE-Build
> Load (dumped) an external table as an external table on target with the same
> location as on the source
> --
>
> Key: HIVE-20708
> URL: https://issues.apache.org/jira/browse/HIVE-20708
> Project: Hive
> Issue Type: Improvement
> Components: repl
>Reporter: Ashutosh Bapat
>Assignee: Ashutosh Bapat
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-20708.01
>
>
> External tables are currently mapped to managed tables on target. A lot of
> jobs in user environment are dependent upon locations specified in external
> table definitions to run, hence, the path for external tables on the target
> and on the source are expected to be the same. An external table being loaded
> as a managed table makes it difficult for failover (Controlled Failover) /
> failback since there is no option of moving data from managed to external
> table. So the external table replicated to target cluster needs to be kept as
> external table with same location as on the source.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20740) Remove global lock in ObjectStore.setConf method
[
https://issues.apache.org/jira/browse/HIVE-20740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16652282#comment-16652282
]
Vihang Karajgaonkar commented on HIVE-20740:
Turns out there was a race condition as well in the code. The {{getPMF}} is a
{{static synchronized}} method (takes a lock on ObjectStore.class) while the
{{setConf}} method takes a lock on {{pmfPropLock}} method. So in theory it is
possible that some thread which calls {{getPMF}} is updating {{pmf}} instance
while another thread which calls {{setConf}} is also updating {{pmf}} as can be
seen from the code above. The patch moves all the updates to the {{pmf}} and
{{prop}} static variables into one method where they are protected using read
locks and write locks to optimize for the common case of datasource properties
not getting updated often.
> Remove global lock in ObjectStore.setConf method
>
>
> Key: HIVE-20740
> URL: https://issues.apache.org/jira/browse/HIVE-20740
> Project: Hive
> Issue Type: Improvement
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
>Priority: Major
> Attachments: HIVE-20740.01.patch
>
>
> The ObjectStore#setConf method has a global lock which can block other
> clients in concurrent workloads.
> {code}
> @Override
> @SuppressWarnings("nls")
> public void setConf(Configuration conf) {
> // Although an instance of ObjectStore is accessed by one thread, there
> may
> // be many threads with ObjectStore instances. So the static variables
> // pmf and prop need to be protected with locks.
> pmfPropLock.lock();
> try {
> isInitialized = false;
> this.conf = conf;
> this.areTxnStatsSupported = MetastoreConf.getBoolVar(conf,
> ConfVars.HIVE_TXN_STATS_ENABLED);
> configureSSL(conf);
> Properties propsFromConf = getDataSourceProps(conf);
> boolean propsChanged = !propsFromConf.equals(prop);
> if (propsChanged) {
> if (pmf != null){
> clearOutPmfClassLoaderCache(pmf);
> if (!forTwoMetastoreTesting) {
> // close the underlying connection pool to avoid leaks
> pmf.close();
> }
> }
> pmf = null;
> prop = null;
> }
> assert(!isActiveTransaction());
> shutdown();
> // Always want to re-create pm as we don't know if it were created by
> the
> // most recent instance of the pmf
> pm = null;
> directSql = null;
> expressionProxy = null;
> openTrasactionCalls = 0;
> currentTransaction = null;
> transactionStatus = TXN_STATUS.NO_STATE;
> initialize(propsFromConf);
> String partitionValidationRegex =
> MetastoreConf.getVar(this.conf,
> ConfVars.PARTITION_NAME_WHITELIST_PATTERN);
> if (partitionValidationRegex != null &&
> !partitionValidationRegex.isEmpty()) {
> partitionValidationPattern =
> Pattern.compile(partitionValidationRegex);
> } else {
> partitionValidationPattern = null;
> }
> // Note, if metrics have not been initialized this will return null,
> which means we aren't
> // using metrics. Thus we should always check whether this is non-null
> before using.
> MetricRegistry registry = Metrics.getRegistry();
> if (registry != null) {
> directSqlErrors =
> Metrics.getOrCreateCounter(MetricsConstants.DIRECTSQL_ERRORS);
> }
> this.batchSize = MetastoreConf.getIntVar(conf,
> ConfVars.RAWSTORE_PARTITION_BATCH_SIZE);
> if (!isInitialized) {
> throw new RuntimeException(
> "Unable to create persistence manager. Check dss.log for details");
> } else {
> LOG.debug("Initialized ObjectStore");
> }
> } finally {
> pmfPropLock.unlock();
> }
> }
> {code}
> The {{pmfPropLock}} is a static object and it disallows any other new
> connection to HMS which is trying to instantiate ObjectStore. We should
> either remove the lock or reduce the scope of the lock so that it is held for
> a very small amount of time.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20740) Remove global lock in ObjectStore.setConf method
[
https://issues.apache.org/jira/browse/HIVE-20740?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vihang Karajgaonkar updated HIVE-20740:
---
Status: Patch Available (was: Open)
> Remove global lock in ObjectStore.setConf method
>
>
> Key: HIVE-20740
> URL: https://issues.apache.org/jira/browse/HIVE-20740
> Project: Hive
> Issue Type: Improvement
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
>Priority: Major
> Attachments: HIVE-20740.01.patch
>
>
> The ObjectStore#setConf method has a global lock which can block other
> clients in concurrent workloads.
> {code}
> @Override
> @SuppressWarnings("nls")
> public void setConf(Configuration conf) {
> // Although an instance of ObjectStore is accessed by one thread, there
> may
> // be many threads with ObjectStore instances. So the static variables
> // pmf and prop need to be protected with locks.
> pmfPropLock.lock();
> try {
> isInitialized = false;
> this.conf = conf;
> this.areTxnStatsSupported = MetastoreConf.getBoolVar(conf,
> ConfVars.HIVE_TXN_STATS_ENABLED);
> configureSSL(conf);
> Properties propsFromConf = getDataSourceProps(conf);
> boolean propsChanged = !propsFromConf.equals(prop);
> if (propsChanged) {
> if (pmf != null){
> clearOutPmfClassLoaderCache(pmf);
> if (!forTwoMetastoreTesting) {
> // close the underlying connection pool to avoid leaks
> pmf.close();
> }
> }
> pmf = null;
> prop = null;
> }
> assert(!isActiveTransaction());
> shutdown();
> // Always want to re-create pm as we don't know if it were created by
> the
> // most recent instance of the pmf
> pm = null;
> directSql = null;
> expressionProxy = null;
> openTrasactionCalls = 0;
> currentTransaction = null;
> transactionStatus = TXN_STATUS.NO_STATE;
> initialize(propsFromConf);
> String partitionValidationRegex =
> MetastoreConf.getVar(this.conf,
> ConfVars.PARTITION_NAME_WHITELIST_PATTERN);
> if (partitionValidationRegex != null &&
> !partitionValidationRegex.isEmpty()) {
> partitionValidationPattern =
> Pattern.compile(partitionValidationRegex);
> } else {
> partitionValidationPattern = null;
> }
> // Note, if metrics have not been initialized this will return null,
> which means we aren't
> // using metrics. Thus we should always check whether this is non-null
> before using.
> MetricRegistry registry = Metrics.getRegistry();
> if (registry != null) {
> directSqlErrors =
> Metrics.getOrCreateCounter(MetricsConstants.DIRECTSQL_ERRORS);
> }
> this.batchSize = MetastoreConf.getIntVar(conf,
> ConfVars.RAWSTORE_PARTITION_BATCH_SIZE);
> if (!isInitialized) {
> throw new RuntimeException(
> "Unable to create persistence manager. Check dss.log for details");
> } else {
> LOG.debug("Initialized ObjectStore");
> }
> } finally {
> pmfPropLock.unlock();
> }
> }
> {code}
> The {{pmfPropLock}} is a static object and it disallows any other new
> connection to HMS which is trying to instantiate ObjectStore. We should
> either remove the lock or reduce the scope of the lock so that it is held for
> a very small amount of time.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20740) Remove global lock in ObjectStore.setConf method
[
https://issues.apache.org/jira/browse/HIVE-20740?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vihang Karajgaonkar updated HIVE-20740:
---
Attachment: HIVE-20740.01.patch
> Remove global lock in ObjectStore.setConf method
>
>
> Key: HIVE-20740
> URL: https://issues.apache.org/jira/browse/HIVE-20740
> Project: Hive
> Issue Type: Improvement
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
>Priority: Major
> Attachments: HIVE-20740.01.patch
>
>
> The ObjectStore#setConf method has a global lock which can block other
> clients in concurrent workloads.
> {code}
> @Override
> @SuppressWarnings("nls")
> public void setConf(Configuration conf) {
> // Although an instance of ObjectStore is accessed by one thread, there
> may
> // be many threads with ObjectStore instances. So the static variables
> // pmf and prop need to be protected with locks.
> pmfPropLock.lock();
> try {
> isInitialized = false;
> this.conf = conf;
> this.areTxnStatsSupported = MetastoreConf.getBoolVar(conf,
> ConfVars.HIVE_TXN_STATS_ENABLED);
> configureSSL(conf);
> Properties propsFromConf = getDataSourceProps(conf);
> boolean propsChanged = !propsFromConf.equals(prop);
> if (propsChanged) {
> if (pmf != null){
> clearOutPmfClassLoaderCache(pmf);
> if (!forTwoMetastoreTesting) {
> // close the underlying connection pool to avoid leaks
> pmf.close();
> }
> }
> pmf = null;
> prop = null;
> }
> assert(!isActiveTransaction());
> shutdown();
> // Always want to re-create pm as we don't know if it were created by
> the
> // most recent instance of the pmf
> pm = null;
> directSql = null;
> expressionProxy = null;
> openTrasactionCalls = 0;
> currentTransaction = null;
> transactionStatus = TXN_STATUS.NO_STATE;
> initialize(propsFromConf);
> String partitionValidationRegex =
> MetastoreConf.getVar(this.conf,
> ConfVars.PARTITION_NAME_WHITELIST_PATTERN);
> if (partitionValidationRegex != null &&
> !partitionValidationRegex.isEmpty()) {
> partitionValidationPattern =
> Pattern.compile(partitionValidationRegex);
> } else {
> partitionValidationPattern = null;
> }
> // Note, if metrics have not been initialized this will return null,
> which means we aren't
> // using metrics. Thus we should always check whether this is non-null
> before using.
> MetricRegistry registry = Metrics.getRegistry();
> if (registry != null) {
> directSqlErrors =
> Metrics.getOrCreateCounter(MetricsConstants.DIRECTSQL_ERRORS);
> }
> this.batchSize = MetastoreConf.getIntVar(conf,
> ConfVars.RAWSTORE_PARTITION_BATCH_SIZE);
> if (!isInitialized) {
> throw new RuntimeException(
> "Unable to create persistence manager. Check dss.log for details");
> } else {
> LOG.debug("Initialized ObjectStore");
> }
> } finally {
> pmfPropLock.unlock();
> }
> }
> {code}
> The {{pmfPropLock}} is a static object and it disallows any other new
> connection to HMS which is trying to instantiate ObjectStore. We should
> either remove the lock or reduce the scope of the lock so that it is held for
> a very small amount of time.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20756) Disable SARG leaf creation for date column until ORC-135
[ https://issues.apache.org/jira/browse/HIVE-20756?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16652267#comment-16652267 ] Prasanth Jayachandran commented on HIVE-20756: -- branch-1 has different type conversions for PPD.. this may still be applicable. the DATE/timestamp ppd is subject to timezone issues anyway. so safer to disable it. +1 > Disable SARG leaf creation for date column until ORC-135 > > > Key: HIVE-20756 > URL: https://issues.apache.org/jira/browse/HIVE-20756 > Project: Hive > Issue Type: Bug >Affects Versions: 2.1.1 >Reporter: Chiran Ravani >Assignee: Chiran Ravani >Priority: Major > Attachments: HIVE-20756.1.patch > > > Until ORC-135 is committed and orc version is updated in hive, disable SARG > creation for date columns in hive. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20756) Disable SARG leaf creation for date column until ORC-135
[ https://issues.apache.org/jira/browse/HIVE-20756?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16652253#comment-16652253 ] Sergey Shelukhin commented on HIVE-20756: - Makes sense to me +1 This is applicable to branch-1 only, ORC is separate after that. [~prasanth_j] does this make sense? > Disable SARG leaf creation for date column until ORC-135 > > > Key: HIVE-20756 > URL: https://issues.apache.org/jira/browse/HIVE-20756 > Project: Hive > Issue Type: Bug >Affects Versions: 2.1.1 >Reporter: Chiran Ravani >Assignee: Chiran Ravani >Priority: Major > Attachments: HIVE-20756.1.patch > > > Until ORC-135 is committed and orc version is updated in hive, disable SARG > creation for date columns in hive. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20724) Add blobstore tests for MSCK REPAIR TABLE
[ https://issues.apache.org/jira/browse/HIVE-20724?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Junyang Li updated HIVE-20724: -- Attachment: (was: HIVE-20724.patch) > Add blobstore tests for MSCK REPAIR TABLE > - > > Key: HIVE-20724 > URL: https://issues.apache.org/jira/browse/HIVE-20724 > Project: Hive > Issue Type: Test > Components: Tests >Reporter: Junyang Li >Assignee: Junyang Li >Priority: Major > Fix For: 2.4.0 > > Attachments: HIVE-20724.patch > > > This patch introduces a regression test into the hive-blobstore qtest module: > recover_partitions.q. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20724) Add blobstore tests for MSCK REPAIR TABLE
[ https://issues.apache.org/jira/browse/HIVE-20724?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Junyang Li updated HIVE-20724: -- Attachment: HIVE-20724.patch Status: Patch Available (was: Open) > Add blobstore tests for MSCK REPAIR TABLE > - > > Key: HIVE-20724 > URL: https://issues.apache.org/jira/browse/HIVE-20724 > Project: Hive > Issue Type: Test > Components: Tests >Reporter: Junyang Li >Assignee: Junyang Li >Priority: Major > Fix For: 2.4.0 > > Attachments: HIVE-20724.patch, HIVE-20724.patch > > > This patch introduces a regression test into the hive-blobstore qtest module: > recover_partitions.q. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20724) Add blobstore tests for MSCK REPAIR TABLE
[ https://issues.apache.org/jira/browse/HIVE-20724?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Junyang Li updated HIVE-20724: -- Status: Open (was: Patch Available) > Add blobstore tests for MSCK REPAIR TABLE > - > > Key: HIVE-20724 > URL: https://issues.apache.org/jira/browse/HIVE-20724 > Project: Hive > Issue Type: Test > Components: Tests >Reporter: Junyang Li >Assignee: Junyang Li >Priority: Major > Fix For: 2.4.0 > > Attachments: HIVE-20724.patch > > > This patch introduces a regression test into the hive-blobstore qtest module: > recover_partitions.q. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20724) Add blobstore tests for MSCK REPAIR TABLE
[ https://issues.apache.org/jira/browse/HIVE-20724?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Junyang Li updated HIVE-20724: -- Component/s: Test > Add blobstore tests for MSCK REPAIR TABLE > - > > Key: HIVE-20724 > URL: https://issues.apache.org/jira/browse/HIVE-20724 > Project: Hive > Issue Type: Test > Components: Tests >Reporter: Junyang Li >Assignee: Junyang Li >Priority: Major > Fix For: 2.4.0 > > Attachments: HIVE-20724.patch > > > This patch introduces a regression test into the hive-blobstore qtest module: > recover_partitions.q. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20724) Add blobstore tests for MSCK REPAIR TABLE
[ https://issues.apache.org/jira/browse/HIVE-20724?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Junyang Li updated HIVE-20724: -- Component/s: (was: Test) Tests > Add blobstore tests for MSCK REPAIR TABLE > - > > Key: HIVE-20724 > URL: https://issues.apache.org/jira/browse/HIVE-20724 > Project: Hive > Issue Type: Test > Components: Tests >Reporter: Junyang Li >Assignee: Junyang Li >Priority: Major > Fix For: 2.4.0 > > Attachments: HIVE-20724.patch > > > This patch introduces a regression test into the hive-blobstore qtest module: > recover_partitions.q. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20731) keystore file in JdbcStorageHandler should be authorized
[ https://issues.apache.org/jira/browse/HIVE-20731?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16652242#comment-16652242 ] Thejas M Nair commented on HIVE-20731: -- +1 > keystore file in JdbcStorageHandler should be authorized > > > Key: HIVE-20731 > URL: https://issues.apache.org/jira/browse/HIVE-20731 > Project: Hive > Issue Type: Improvement > Components: StorageHandler >Reporter: Daniel Dai >Assignee: Daniel Dai >Priority: Major > Attachments: HIVE-20731.1.patch, HIVE-20731.2.patch, > HIVE-20731.3.patch, HIVE-20731.4.patch > > > The keystore file introduced in HIVE-20651 shall be authorized with > configured authorizer. Otherwise any user knows the keystore file location > can access the password. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20724) Add blobstore tests for MSCK REPAIR TABLE
[ https://issues.apache.org/jira/browse/HIVE-20724?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Junyang Li updated HIVE-20724: -- Description: This patch introduces a regression test into the hive-blobstore qtest module: recover_partitions.q. (was: This patch introduces a regression test into the hive-blobstore qtest module: alter_table_concatenate.q and recover_partitions.q.) > Add blobstore tests for MSCK REPAIR TABLE > - > > Key: HIVE-20724 > URL: https://issues.apache.org/jira/browse/HIVE-20724 > Project: Hive > Issue Type: Test >Reporter: Junyang Li >Assignee: Junyang Li >Priority: Major > Fix For: 2.4.0 > > Attachments: HIVE-20724.patch > > > This patch introduces a regression test into the hive-blobstore qtest module: > recover_partitions.q. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20708) Load (dumped) an external table as an external table on target with the same location as on the source
[
https://issues.apache.org/jira/browse/HIVE-20708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16652236#comment-16652236
]
Hive QA commented on HIVE-20708:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
36s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
2s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
46s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
59s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
41s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
7s{color} | {color:blue} ql in master has 2318 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
24s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
11s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
46s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
17s{color} | {color:red} itests/hive-unit: The patch generated 5 new + 134
unchanged - 0 fixed = 139 total (was 134) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
41s{color} | {color:red} ql: The patch generated 3 new + 204 unchanged - 0
fixed = 207 total (was 204) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
25s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 30m 11s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-14504/dev-support/hive-personality.sh
|
| git revision | master / 0f2f999 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-14504/yetus/diff-checkstyle-itests_hive-unit.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-14504/yetus/diff-checkstyle-ql.txt
|
| modules | C: itests/hive-unit ql U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-14504/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Load (dumped) an external table as an external table on target with the same
> location as on the source
> --
>
> Key: HIVE-20708
> URL: https://issues.apache.org/jira/browse/HIVE-20708
> Project: Hive
> Issue Type: Improvement
> Components: repl
>Reporter: Ashutosh Bapat
>Assignee: Ashutosh Bapat
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-20708.01
>
>
> External tables are currently mapped to managed tables on target. A lot of
> jobs in user environment are dependent upon locations specified in external
> table
[jira] [Assigned] (HIVE-20756) Disable SARG leaf creation for date column until ORC-135
[ https://issues.apache.org/jira/browse/HIVE-20756?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chiran Ravani reassigned HIVE-20756: Assignee: Chiran Ravani (was: Prasanth Jayachandran) > Disable SARG leaf creation for date column until ORC-135 > > > Key: HIVE-20756 > URL: https://issues.apache.org/jira/browse/HIVE-20756 > Project: Hive > Issue Type: Bug >Affects Versions: 2.1.1 >Reporter: Chiran Ravani >Assignee: Chiran Ravani >Priority: Major > Attachments: HIVE-20756.1.patch > > > Until ORC-135 is committed and orc version is updated in hive, disable SARG > creation for date columns in hive. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20756) Disable SARG leaf creation for date column until ORC-135
[ https://issues.apache.org/jira/browse/HIVE-20756?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chiran Ravani updated HIVE-20756: - Attachment: HIVE-20756.1.patch > Disable SARG leaf creation for date column until ORC-135 > > > Key: HIVE-20756 > URL: https://issues.apache.org/jira/browse/HIVE-20756 > Project: Hive > Issue Type: Bug >Affects Versions: 2.1.1 >Reporter: Chiran Ravani >Assignee: Prasanth Jayachandran >Priority: Major > Attachments: HIVE-20756.1.patch > > > Until ORC-135 is committed and orc version is updated in hive, disable SARG > creation for date columns in hive. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-20206) Hive Union all query with two views on the same hbase external table producing incorrect results
[
https://issues.apache.org/jira/browse/HIVE-20206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alice Fan reassigned HIVE-20206:
Assignee: Alice Fan
> Hive Union all query with two views on the same hbase external table
> producing incorrect results
>
>
> Key: HIVE-20206
> URL: https://issues.apache.org/jira/browse/HIVE-20206
> Project: Hive
> Issue Type: Bug
> Components: Beeline
>Affects Versions: 1.1.0
>Reporter: Venkata
>Assignee: Alice Fan
>Priority: Major
> Attachments: hive_view_union_all.txt
>
>
> We are running this in Cloudera cdh5.13.3 version.
> Hive version - 1.1.0-cdh5.13.3
> Hbase version - 1.2.0-cdh5.13.3
> When running hive union all query with two views created on the same hive
> *external* hbase table, the query is returning incorrect results.
> The query is returning correct results if *set hive.optimize.ppd=false;*
> We don't want to turn off the ppd as it will greatly impact performance.
> Note: The union all is working fine for the views created on the same Hive
> *Managed* table.
> The results marked in {color:#FF}red{color} are wrong results.
>
> The below are the queries ( i have attached the same queries as an
> attachment):
> *HBASE:*
> create '*test*','default'
> put 'test','111','default:name','john1'
> put 'test','111','default:dept','hr1'
> put 'test','111','default:type','a'
> put 'test','112','default:name','rambo1'
> put 'test','112','default:dept','eng1'
> put 'test','112','default:type','a'
> put 'test','113','default:name','alex1'
> put 'test','113','default:dept','dev1'
> put 'test','113','default:type','a'
> put 'test','211','default:name','john2'
> put 'test','211','default:dept','hr2'
> put 'test','211','default:type','b'
> put 'test','212','default:name','rambo2'
> put 'test','212','default:dept','eng2'
> put 'test','212','default:type','b'
> put 'test','213','default:name','alex2'
> put 'test','213','default:dept','dev2'
> put 'test','213','default:type','b'
> put 'test','311','default:name','john3'
> put 'test','311','default:dept','hr3'
> put 'test','311','default:type','c'
> put 'test','312','default:name','rambo3'
> put 'test','312','default:dept','eng3'
> put 'test','312','default:type','c'
> put 'test','313','default:name','alex3'
> put 'test','313','default:dept','dev3'
> put 'test','313','default:type','c'
>
> *BEELINE*
> CREATE external TABLE *hbase_ext_table*(id string, name string,dept
> string,type string) STORED BY
> 'org.apache.hadoop.hive.hbase.HBaseStorageHandler' WITH SERDEPROPERTIES
> ("hbase.columns.mapping" = ":key,default:name,default:dept,default:type")
> TBLPROPERTIES ("hbase.table.name" = "*test*");
> create view *view1* as select id ,name,dept,type from *hbase_ext_table* where
> id>='1' and id<'2' and type is not null;
> create view *view2* as select id,name,dept,type from *hbase_ext_table* where
> id>='2' and id<'3' and type is not null;
> > select * from view1;
>
> +--+--++--++--
> |view1.id|view1.name|view1.dept|view1.type|
> +--+--++--++--
> |111|john1|hr1|a|
> |112|rambo1|eng1|a|
> |113|alex1|dev1|a|
> +--+--++--++--
> > select * from view2;
>
> +--+--++--++--
> |view2.id|view2.name|view2.dept|view2.type|
> +--+--++--++--
> |211|john2|hr2|b|
> |212|rambo2|eng2|b|
> |213|alex2|dev2|b|
> +--+--++--++--
> > select id,name,dept,type from *view1* union all select id,name,dept,type
> > from *view2*;
> +--+--++--++--
> |_u1.id|_u1.name|_u1.dept|_u1.type|
> +--+--++--++--
> |111|john1|hr1|a|
> |111|john1|hr1|a|
> |112|rambo1|eng1|a|
> |112|rambo1|eng1|a|
> |113|alex1|dev1|a|
> |113|alex1|dev1|a|
> |211|john2|hr2|b|
> |211|john2|hr2|b|
> |212|rambo2|eng2|b|
> |212|rambo2|eng2|b|
> |213|alex2|dev2|b|
> |213|alex2|dev2|b|
> |{color:#ff}*311 \| john3 \| hr3 \| c \|*{color}
> {color:#ff}*\| 311 \| john3 \| hr3 \| c \|*{color}
> {color:#ff}*\| 312 \| rambo3 \| eng3 \| c \|*{color}
> {color:#ff}*\| 312 \| rambo3 \| eng3 \| c \|*{color}
> {color:#ff}*\| 313 \| alex3 \| dev3 \| c \|*{color}
> {color:#ff}*\| 313 \| alex3 \| dev3 \| c \|*{color}
> +--+--++--++--|
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20756) Disable SARG leaf creation for date column until ORC-135
[ https://issues.apache.org/jira/browse/HIVE-20756?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chiran Ravani updated HIVE-20756: - Description: Until ORC-135 is committed and orc version is updated in hive, disable SARG creation for date columns in hive. (was: Until ORC-135 is committed and orc version is updated in hive, disable SARG creation for timestamp columns in hive. ) > Disable SARG leaf creation for date column until ORC-135 > > > Key: HIVE-20756 > URL: https://issues.apache.org/jira/browse/HIVE-20756 > Project: Hive > Issue Type: Bug >Affects Versions: 2.1.1 >Reporter: Chiran Ravani >Assignee: Prasanth Jayachandran >Priority: Major > > Until ORC-135 is committed and orc version is updated in hive, disable SARG > creation for date columns in hive. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-20756) Disable SARG leaf creation for date column until ORC-135
[ https://issues.apache.org/jira/browse/HIVE-20756?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chiran Ravani reassigned HIVE-20756: > Disable SARG leaf creation for date column until ORC-135 > > > Key: HIVE-20756 > URL: https://issues.apache.org/jira/browse/HIVE-20756 > Project: Hive > Issue Type: Bug >Affects Versions: 2.1.1 >Reporter: Chiran Ravani >Assignee: Prasanth Jayachandran >Priority: Major > > Until ORC-135 is committed and orc version is updated in hive, disable SARG > creation for timestamp columns in hive. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (HIVE-17231) ColumnizedDeleteEventRegistry.DeleteReaderValue optimization
[
https://issues.apache.org/jira/browse/HIVE-17231?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eugene Koifman resolved HIVE-17231.
---
Resolution: Fixed
Fix Version/s: 4.0.0
Release Note: n/a
committed to master
thanks Gopal for the review
> ColumnizedDeleteEventRegistry.DeleteReaderValue optimization
>
>
> Key: HIVE-17231
> URL: https://issues.apache.org/jira/browse/HIVE-17231
> Project: Hive
> Issue Type: Sub-task
> Components: Transactions
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-17231.01.patch, HIVE-17231.02.patch
>
>
> For unbucketed tables DeleteReaderValue will currently return all delete
> events. Once we trust that
> the N in bucketN for "base" spit is reliable, all delete events not
> matching N can be skipped.
> This is useful to protect against extreme cases where someone runs an
> update/delete on a partition that matches 10 billion rows thus generates very
> many delete events.
> Since HIVE-19890 all acid data files must have bucketid/writerid in the file
> name match bucketid/writerid in ROW__ID in the data.
> {{OrcRawRecrodMerger.getDeltaFiles()}} should only return files representing
> the right {{bucket}}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-16882) Improvements For Avro SerDe Package
[ https://issues.apache.org/jira/browse/HIVE-16882?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] BELUGA BEHR updated HIVE-16882: --- Status: Patch Available (was: Open) Re-attaching latest patch to test build > Improvements For Avro SerDe Package > --- > > Key: HIVE-16882 > URL: https://issues.apache.org/jira/browse/HIVE-16882 > Project: Hive > Issue Type: Improvement > Components: Serializers/Deserializers >Affects Versions: 3.0.0 >Reporter: BELUGA BEHR >Assignee: BELUGA BEHR >Priority: Minor > Attachments: HIVE-16882.1.patch, HIVE-16882.10.patch, > HIVE-16882.11.patch, HIVE-16882.12.patch, HIVE-16882.13.patch, > HIVE-16882.2.patch, HIVE-16882.3.patch, HIVE-16882.4.patch, > HIVE-16882.5.patch, HIVE-16882.6.patch, HIVE-16882.7.patch, > HIVE-16882.8.patch, HIVE-16882.9.patch > > > # Use SLF4J parameter DEBUG logging > # Use re-usable libraries where appropriate > # Use enhanced for loops where appropriate > # Fix several minor check-style error > # Small performance enhancements in InstanceCache -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-16882) Improvements For Avro SerDe Package
[ https://issues.apache.org/jira/browse/HIVE-16882?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] BELUGA BEHR updated HIVE-16882: --- Attachment: HIVE-16882.13.patch > Improvements For Avro SerDe Package > --- > > Key: HIVE-16882 > URL: https://issues.apache.org/jira/browse/HIVE-16882 > Project: Hive > Issue Type: Improvement > Components: Serializers/Deserializers >Affects Versions: 3.0.0 >Reporter: BELUGA BEHR >Assignee: BELUGA BEHR >Priority: Minor > Attachments: HIVE-16882.1.patch, HIVE-16882.10.patch, > HIVE-16882.11.patch, HIVE-16882.12.patch, HIVE-16882.13.patch, > HIVE-16882.2.patch, HIVE-16882.3.patch, HIVE-16882.4.patch, > HIVE-16882.5.patch, HIVE-16882.6.patch, HIVE-16882.7.patch, > HIVE-16882.8.patch, HIVE-16882.9.patch > > > # Use SLF4J parameter DEBUG logging > # Use re-usable libraries where appropriate > # Use enhanced for loops where appropriate > # Fix several minor check-style error > # Small performance enhancements in InstanceCache -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-16882) Improvements For Avro SerDe Package
[ https://issues.apache.org/jira/browse/HIVE-16882?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] BELUGA BEHR updated HIVE-16882: --- Status: Open (was: Patch Available) > Improvements For Avro SerDe Package > --- > > Key: HIVE-16882 > URL: https://issues.apache.org/jira/browse/HIVE-16882 > Project: Hive > Issue Type: Improvement > Components: Serializers/Deserializers >Affects Versions: 3.0.0 >Reporter: BELUGA BEHR >Assignee: BELUGA BEHR >Priority: Minor > Attachments: HIVE-16882.1.patch, HIVE-16882.10.patch, > HIVE-16882.11.patch, HIVE-16882.12.patch, HIVE-16882.13.patch, > HIVE-16882.2.patch, HIVE-16882.3.patch, HIVE-16882.4.patch, > HIVE-16882.5.patch, HIVE-16882.6.patch, HIVE-16882.7.patch, > HIVE-16882.8.patch, HIVE-16882.9.patch > > > # Use SLF4J parameter DEBUG logging > # Use re-usable libraries where appropriate > # Use enhanced for loops where appropriate > # Fix several minor check-style error > # Small performance enhancements in InstanceCache -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20189) Separate metastore client code into its own module
[
https://issues.apache.org/jira/browse/HIVE-20189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16652180#comment-16652180
]
Hive QA commented on HIVE-20189:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12944082/HIVE-20189.05.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 5 failed/errored test(s), 15085 tests
executed
*Failed tests:*
{noformat}
org.apache.hive.jdbc.TestJdbcWithMiniLlapArrow.testComplexQuery (batchId=253)
org.apache.hive.jdbc.TestJdbcWithMiniLlapArrow.testDataTypes (batchId=253)
org.apache.hive.jdbc.TestJdbcWithMiniLlapArrow.testEscapedStrings (batchId=253)
org.apache.hive.jdbc.TestJdbcWithMiniLlapArrow.testLlapInputFormatEndToEnd
(batchId=253)
org.apache.hive.jdbc.TestJdbcWithMiniLlapArrow.testNonAsciiStrings (batchId=253)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/14503/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/14503/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-14503/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 5 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12944082 - PreCommit-HIVE-Build
> Separate metastore client code into its own module
> --
>
> Key: HIVE-20189
> URL: https://issues.apache.org/jira/browse/HIVE-20189
> Project: Hive
> Issue Type: Sub-task
> Components: Standalone Metastore
>Affects Versions: 4.0.0, 3.2.0
>Reporter: Alexander Kolbasov
>Assignee: Alexander Kolbasov
>Priority: Major
> Attachments: HIVE-20189.01.patch, HIVE-20189.03.patch,
> HIVE-20189.04.patch, HIVE-20189.05.patch
>
>
> The goal of this JIRA is to split HiveMetastoreClient code out of
> metastore-common. This is a pom-only change that does not require any changes
> in the code.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20512) Improve record and memory usage logging in SparkRecordHandler
[
https://issues.apache.org/jira/browse/HIVE-20512?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bharathkrishna Guruvayoor Murali updated HIVE-20512:
Attachment: HIVE-20512.1.patch
Status: Patch Available (was: Open)
[~stakiar] Can you please check the attached patch if this approach looks good.
I tried adding unit tests, but not sure if it is needed, as I should parse the
log in a hard-coded way to check the String "processed " + rowNumber + " rows:
used memory ="
and probably sleep() till the threshold to check if logs are printed again.
What do you suggest?
> Improve record and memory usage logging in SparkRecordHandler
> -
>
> Key: HIVE-20512
> URL: https://issues.apache.org/jira/browse/HIVE-20512
> Project: Hive
> Issue Type: Sub-task
> Components: Spark
>Reporter: Sahil Takiar
>Assignee: Bharathkrishna Guruvayoor Murali
>Priority: Major
> Attachments: HIVE-20512.1.patch
>
>
> We currently log memory usage and # of records processed in Spark tasks, but
> we should improve the methodology for how frequently we log this info.
> Currently we use the following code:
> {code:java}
> private long getNextLogThreshold(long currentThreshold) {
> // A very simple counter to keep track of number of rows processed by the
> // reducer. It dumps
> // every 1 million times, and quickly before that
> if (currentThreshold >= 100) {
> return currentThreshold + 100;
> }
> return 10 * currentThreshold;
> }
> {code}
> The issue is that after a while, the increase by 10x factor means that you
> have to process a huge # of records before this gets triggered.
> A better approach would be to log this info at a given interval. This would
> help in debugging tasks that are seemingly hung.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20735) Address some of the review comments plus Kerberos support
[ https://issues.apache.org/jira/browse/HIVE-20735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16652139#comment-16652139 ] slim bouguerra commented on HIVE-20735: --- [~ashutoshc] can you please push this? > Address some of the review comments plus Kerberos support > - > > Key: HIVE-20735 > URL: https://issues.apache.org/jira/browse/HIVE-20735 > Project: Hive > Issue Type: Sub-task > Components: kafka integration >Reporter: slim bouguerra >Assignee: slim bouguerra >Priority: Major > Attachments: HIVE-20735.patch > > > As part of the review comments we agreed to: > # remove start and end offsets columns > # remove the best effort mode > # make the 2pc as default protocol for EOS > Also this patch will include an additional enhancement to add kerberos > support. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20189) Separate metastore client code into its own module
[
https://issues.apache.org/jira/browse/HIVE-20189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16652133#comment-16652133
]
Hive QA commented on HIVE-20189:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
41s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
13s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
33s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
44s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 2m
40s{color} | {color:blue} standalone-metastore/metastore-common in master has
28 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
42s{color} | {color:blue} llap-server in master has 84 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 1m
8s{color} | {color:blue} standalone-metastore/metastore-server in master has
182 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
49s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
11s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
7s{color} | {color:green} The patch standalone-metastore passed checkstyle
{color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
7s{color} | {color:green} The patch metastore-common passed checkstyle {color}
|
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
32s{color} | {color:red} standalone-metastore/metastore-client: The patch
generated 2180 new + 0 unchanged - 0 fixed = 2180 total (was 0) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
9s{color} | {color:green} The patch metastore passed checkstyle {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
13s{color} | {color:green} llap-server: The patch generated 0 new + 8 unchanged
- 1 fixed = 8 total (was 9) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
7s{color} | {color:green} The patch metastore-server passed checkstyle {color}
|
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch 1 line(s) with tabs. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
4s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m
42s{color} | {color:red} standalone-metastore/metastore-common generated 4 new
+ 24 unchanged - 4 fixed = 28 total (was 28) {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 1m
15s{color} | {color:red} standalone-metastore generated 1 new + 66 unchanged -
1 fixed = 67 total (was 67) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
52s{color} | {color:green} standalone-metastore_metastore-common generated 0
new + 16 unchanged - 1 fixed = 16 total (was 17) {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
10s{color} | {color:red} standalone-metastore_metastore-client generated 1 new
+ 0 unchanged - 0 fixed = 1 total (was 0) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
11s{color} | {color:green} metastore in the patch passed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
14s{color} | {color:green} llap-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
22s{color} | {color:green}
[jira] [Updated] (HIVE-20755) Add the ability to push Dynamic Between and IN clauses to JDBC handler
[ https://issues.apache.org/jira/browse/HIVE-20755?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-20755: --- Summary: Add the ability to push Dynamic Between and IN clauses to JDBC handler (was: Add the ability to push Dynamic Between and Bloom filters to JDBC handler) > Add the ability to push Dynamic Between and IN clauses to JDBC handler > -- > > Key: HIVE-20755 > URL: https://issues.apache.org/jira/browse/HIVE-20755 > Project: Hive > Issue Type: New Feature > Components: StorageHandler >Reporter: Jesus Camacho Rodriguez >Priority: Major > > HIVE-20683 has done some work to push semijoin reduction to Druid. We could > use similar model to push them to JDBC sources, which would be quite useful, > e.g., for joins between Hive and JDBC sources. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20488) SparkSubmitSparkClient#launchDriver should parse exceptions, not just errors
[
https://issues.apache.org/jira/browse/HIVE-20488?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bharathkrishna Guruvayoor Murali updated HIVE-20488:
Attachment: (was: HIVE-20488.1.patch)
> SparkSubmitSparkClient#launchDriver should parse exceptions, not just errors
>
>
> Key: HIVE-20488
> URL: https://issues.apache.org/jira/browse/HIVE-20488
> Project: Hive
> Issue Type: Sub-task
> Components: Spark
>Reporter: Sahil Takiar
>Assignee: Bharathkrishna Guruvayoor Murali
>Priority: Major
> Attachments: HIVE-20488.1.patch
>
>
> In {{SparkSubmitSparkClient#launchDriver}} we parse the stdout / stderr of
> {{bin/spark-submit}} for strings that contain "Error", but we should also
> look for "Exception".
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20488) SparkSubmitSparkClient#launchDriver should parse exceptions, not just errors
[
https://issues.apache.org/jira/browse/HIVE-20488?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bharathkrishna Guruvayoor Murali updated HIVE-20488:
Attachment: HIVE-20488.1.patch
> SparkSubmitSparkClient#launchDriver should parse exceptions, not just errors
>
>
> Key: HIVE-20488
> URL: https://issues.apache.org/jira/browse/HIVE-20488
> Project: Hive
> Issue Type: Sub-task
> Components: Spark
>Reporter: Sahil Takiar
>Assignee: Bharathkrishna Guruvayoor Murali
>Priority: Major
> Attachments: HIVE-20488.1.patch
>
>
> In {{SparkSubmitSparkClient#launchDriver}} we parse the stdout / stderr of
> {{bin/spark-submit}} for strings that contain "Error", but we should also
> look for "Exception".
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-8123) Support parquet ACID
[ https://issues.apache.org/jira/browse/HIVE-8123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16652125#comment-16652125 ] Imtiaz Ahmed commented on HIVE-8123: Is there any continued work being done for supporting Acid Transactions with parquet? > Support parquet ACID > > > Key: HIVE-8123 > URL: https://issues.apache.org/jira/browse/HIVE-8123 > Project: Hive > Issue Type: Task >Reporter: Brock Noland >Assignee: Ferdinand Xu >Priority: Major > > Hive "ACID" work currently only works with ORC. It should work with Parquet > as well. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20755) Add the ability to push Dynamic Between and Bloom filters to JDBC handler
[ https://issues.apache.org/jira/browse/HIVE-20755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16652100#comment-16652100 ] Jesus Camacho Rodriguez commented on HIVE-20755: Cc [~daijy] [~thejas] > Add the ability to push Dynamic Between and Bloom filters to JDBC handler > - > > Key: HIVE-20755 > URL: https://issues.apache.org/jira/browse/HIVE-20755 > Project: Hive > Issue Type: New Feature > Components: StorageHandler >Reporter: Jesus Camacho Rodriguez >Priority: Major > > HIVE-20683 has done some work to push semijoin reduction to Druid. We could > use similar model to push them to JDBC sources, which would be quite useful, > e.g., for joins between Hive and JDBC sources. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20754) JDBC: Add some missing classes to jdbc standalone jar (follow up to HIVE-19801)
[ https://issues.apache.org/jira/browse/HIVE-20754?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vaibhav Gumashta updated HIVE-20754: Status: Patch Available (was: Open) > JDBC: Add some missing classes to jdbc standalone jar (follow up to > HIVE-19801) > --- > > Key: HIVE-20754 > URL: https://issues.apache.org/jira/browse/HIVE-20754 > Project: Hive > Issue Type: Bug > Components: JDBC >Affects Versions: 3.1.0 >Reporter: Vaibhav Gumashta >Assignee: Vaibhav Gumashta >Priority: Major > Attachments: HIVE-20754.1.patch > > > Some more classes are needed in a secure cluster -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20754) JDBC: Add some missing classes to jdbc standalone jar (follow up to HIVE-19801)
[ https://issues.apache.org/jira/browse/HIVE-20754?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vaibhav Gumashta updated HIVE-20754: Attachment: HIVE-20754.1.patch > JDBC: Add some missing classes to jdbc standalone jar (follow up to > HIVE-19801) > --- > > Key: HIVE-20754 > URL: https://issues.apache.org/jira/browse/HIVE-20754 > Project: Hive > Issue Type: Bug > Components: JDBC >Affects Versions: 3.1.0 >Reporter: Vaibhav Gumashta >Assignee: Vaibhav Gumashta >Priority: Major > Attachments: HIVE-20754.1.patch > > > Some more classes are needed in a secure cluster -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20720) Add partition column option to JDBC handler
[ https://issues.apache.org/jira/browse/HIVE-20720?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16652081#comment-16652081 ] Daniel Dai commented on HIVE-20720: --- Yes, it is ready for review. RB link attached. > Add partition column option to JDBC handler > --- > > Key: HIVE-20720 > URL: https://issues.apache.org/jira/browse/HIVE-20720 > Project: Hive > Issue Type: New Feature > Components: StorageHandler >Reporter: Daniel Dai >Assignee: Daniel Dai >Priority: Major > Attachments: HIVE-20720.1.patch, HIVE-20720.2.patch, > HIVE-20720.3.patch, HIVE-20720.4.patch, HIVE-20720.5.patch, > HIVE-20720.6.patch, HIVE-20720.7.patch > > > Currently JdbcStorageHandler does not split input in Tez. The reason is > numSplit of JdbcInputFormat.getSplits can only pass via "mapreduce.job.maps" > in Tez. And "mapreduce.job.maps" is not a valid param if authorizer(eg. > SQLStdAuth) is in use. User ends up always use 1 split. > We need to rely on this new feature if we want to support multi-splits. Here > is my proposal: > 1. Specify partitionColumn/numPartitions, and optional lowerBound/upperBound > in tblproperties if user want to split jdbc data source. In case > lowerBound/upperBound is not specified, JdbcStorageHandler will run max/min > query to get this in planner. We can currently limit partitionColumn to only > numeric/date/timestamp column for simplicity > 2. If partitionColumn/numPartitions are not specified, don't split input > 3. Splits are equal intervals without respect to data distribution > 4. There is also a "hive.sql.query.split" flag vetos the split (can be set > manually or automatically by calcite) > 5. If partitionColumn is not defined, but numPartitions is defined, use > original limit/offset logic (however, don't rely on numSplit). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-20754) JDBC: Add some missing classes to jdbc standalone jar (follow up to HIVE-19801)
[ https://issues.apache.org/jira/browse/HIVE-20754?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vaibhav Gumashta reassigned HIVE-20754: --- Assignee: Vaibhav Gumashta > JDBC: Add some missing classes to jdbc standalone jar (follow up to > HIVE-19801) > --- > > Key: HIVE-20754 > URL: https://issues.apache.org/jira/browse/HIVE-20754 > Project: Hive > Issue Type: Bug > Components: JDBC >Affects Versions: 3.1.0 >Reporter: Vaibhav Gumashta >Assignee: Vaibhav Gumashta >Priority: Major > > Some more classes are needed in a secure cluster -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20707) Automatic partition management
[
https://issues.apache.org/jira/browse/HIVE-20707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16652067#comment-16652067
]
Hive QA commented on HIVE-20707:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12944066/HIVE-20707.4.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/14502/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/14502/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-14502/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Tests exited with: Exception: Patch URL
https://issues.apache.org/jira/secure/attachment/12944066/HIVE-20707.4.patch
was found in seen patch url's cache and a test was probably run already on it.
Aborting...
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12944066 - PreCommit-HIVE-Build
> Automatic partition management
> --
>
> Key: HIVE-20707
> URL: https://issues.apache.org/jira/browse/HIVE-20707
> Project: Hive
> Issue Type: New Feature
>Affects Versions: 4.0.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
>Priority: Major
> Attachments: HIVE-20702.3.patch, HIVE-20707.1.patch,
> HIVE-20707.2.patch, HIVE-20707.4.patch
>
>
> In current scenario, to add partitions for external tables to metastore, MSCK
> REPAIR command has to be executed manually. To avoid this manual step,
> external tables can be specified a table property based on which a background
> metastore thread can sync partitions periodically. Tables can also be
> specified with partition retention period. Any partition whose age exceeds
> the retention period will be dropped automatically.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20746) HiveProtoHookLogger does not close file at end of day.
[
https://issues.apache.org/jira/browse/HIVE-20746?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16652066#comment-16652066
]
Hive QA commented on HIVE-20746:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12944058/HIVE-20746.02.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 15086 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[schemeAuthority]
(batchId=190)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/14501/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/14501/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-14501/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12944058 - PreCommit-HIVE-Build
> HiveProtoHookLogger does not close file at end of day.
> --
>
> Key: HIVE-20746
> URL: https://issues.apache.org/jira/browse/HIVE-20746
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Reporter: Harish Jaiprakash
>Assignee: Harish Jaiprakash
>Priority: Major
> Attachments: HIVE-20746.01.patch, HIVE-20746.02.patch
>
>
> The file rotation happens with an event currently. If there are no queries
> fired for a long time, then the file rotation does not happen and we do not
> close the file. This causes the clients to poll for the file for an
> indeterminate amount of time. If there are multiple hiveservers there is no
> way to tell which file will get more data. Fix this to close at end of day.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20737) Local SparkContext is shared between user sessions and should be closed only when there is no active
[ https://issues.apache.org/jira/browse/HIVE-20737?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Denys Kuzmenko updated HIVE-20737: -- Attachment: HIVE-20737.10.patch > Local SparkContext is shared between user sessions and should be closed only > when there is no active > > > Key: HIVE-20737 > URL: https://issues.apache.org/jira/browse/HIVE-20737 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Denys Kuzmenko >Assignee: Denys Kuzmenko >Priority: Major > Attachments: HIVE-20737.1.patch, HIVE-20737.10.patch, > HIVE-20737.2.patch, HIVE-20737.5.patch, HIVE-20737.6.patch, > HIVE-20737.7.patch, HIVE-20737.8.patch, HIVE-20737.9.patch > > > 1. Local SparkContext is shared between user sessions and should be closed > only when there is no active. > 2. Possible race condition in SparkSession.open() in case when user queries > run in parallel within the same session. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-17231) ColumnizedDeleteEventRegistry.DeleteReaderValue optimization
[
https://issues.apache.org/jira/browse/HIVE-17231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16652037#comment-16652037
]
Gopal V commented on HIVE-17231:
LGTM - +1
> ColumnizedDeleteEventRegistry.DeleteReaderValue optimization
>
>
> Key: HIVE-17231
> URL: https://issues.apache.org/jira/browse/HIVE-17231
> Project: Hive
> Issue Type: Sub-task
> Components: Transactions
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
>Priority: Major
> Attachments: HIVE-17231.01.patch, HIVE-17231.02.patch
>
>
> For unbucketed tables DeleteReaderValue will currently return all delete
> events. Once we trust that
> the N in bucketN for "base" spit is reliable, all delete events not
> matching N can be skipped.
> This is useful to protect against extreme cases where someone runs an
> update/delete on a partition that matches 10 billion rows thus generates very
> many delete events.
> Since HIVE-19890 all acid data files must have bucketid/writerid in the file
> name match bucketid/writerid in ROW__ID in the data.
> {{OrcRawRecrodMerger.getDeltaFiles()}} should only return files representing
> the right {{bucket}}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20720) Add partition column option to JDBC handler
[ https://issues.apache.org/jira/browse/HIVE-20720?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16652031#comment-16652031 ] Jesus Camacho Rodriguez commented on HIVE-20720: [~daijy], I see you got a clean run, is it ready to be reviewed? Do you have a PR or RB link? Thanks > Add partition column option to JDBC handler > --- > > Key: HIVE-20720 > URL: https://issues.apache.org/jira/browse/HIVE-20720 > Project: Hive > Issue Type: New Feature > Components: StorageHandler >Reporter: Daniel Dai >Assignee: Daniel Dai >Priority: Major > Attachments: HIVE-20720.1.patch, HIVE-20720.2.patch, > HIVE-20720.3.patch, HIVE-20720.4.patch, HIVE-20720.5.patch, > HIVE-20720.6.patch, HIVE-20720.7.patch > > > Currently JdbcStorageHandler does not split input in Tez. The reason is > numSplit of JdbcInputFormat.getSplits can only pass via "mapreduce.job.maps" > in Tez. And "mapreduce.job.maps" is not a valid param if authorizer(eg. > SQLStdAuth) is in use. User ends up always use 1 split. > We need to rely on this new feature if we want to support multi-splits. Here > is my proposal: > 1. Specify partitionColumn/numPartitions, and optional lowerBound/upperBound > in tblproperties if user want to split jdbc data source. In case > lowerBound/upperBound is not specified, JdbcStorageHandler will run max/min > query to get this in planner. We can currently limit partitionColumn to only > numeric/date/timestamp column for simplicity > 2. If partitionColumn/numPartitions are not specified, don't split input > 3. Splits are equal intervals without respect to data distribution > 4. There is also a "hive.sql.query.split" flag vetos the split (can be set > manually or automatically by calcite) > 5. If partitionColumn is not defined, but numPartitions is defined, use > original limit/offset logic (however, don't rely on numSplit). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20753) Derby thread interrupt during ptest
[
https://issues.apache.org/jira/browse/HIVE-20753?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16652021#comment-16652021
]
Eugene Koifman commented on HIVE-20753:
---
derby can't handle concurrent threads. In the past I've mostly seen it get
wedged.
The hearbeater thread is normally started automatically by the client
(DbTxnManager) to let the system know the txn is alive.
By default, the txn timeout is 5min and the 1st hearbeat for a given query
should happen after 2.5 minutes.
So if your query is running longer than that (which is unusual in UTs), it's
possible that the heartbeater is the 2nd thread...
> Derby thread interrupt during ptest
> ---
>
> Key: HIVE-20753
> URL: https://issues.apache.org/jira/browse/HIVE-20753
> Project: Hive
> Issue Type: Bug
>Reporter: Zoltan Haindrich
>Priority: Major
> Attachments: derby.log
>
>
> I've had another failed ptest executionit seems like derby have caught an
> unexpected interrupt call ; which have hanged the execution; after that
> nothing happend for about half an hour - after which batch timeout have
> happened...
> {code}
> Caused by: ERROR XSDG9: Derby thread received an interrupt during a disk I/O
> operation, please check your application for the source of the interrupt.
> at org.apache.derby.iapi.error.StandardException.newException(Unknown
> Source)
> at
> org.apache.derby.impl.jdbc.SQLExceptionFactory.wrapArgsForTransportAcrossDRDA(Unknown
> Source)
> ... 42 more
> {code}
> full stacktrack:
> {code}
> 2018-10-16T06:47:29,355 ERROR [Heartbeater-3] lockmgr.DbTxnManager: Failed
> trying to heartbeat queryId=null, currentUser: hiveptest (auth:SIMPLE): null
> java.lang.reflect.UndeclaredThrowableException: null
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1700)
> ~[hadoop-common-3.1.0.jar:?]
> at
> org.apache.hadoop.hive.ql.lockmgr.DbTxnManager$Heartbeater.run(DbTxnManager.java:955)
> [hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> [?:1.8.0_102]
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> [?:1.8.0_102]
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> [?:1.8.0_102]
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> [?:1.8.0_102]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> [?:1.8.0_102]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> [?:1.8.0_102]
> at java.lang.Thread.run(Thread.java:745) [?:1.8.0_102]
> Caused by: org.apache.hadoop.hive.ql.lockmgr.LockException: Error
> communicating with the metastore(txnid:15,lockid:0 queryId=null txnid:0)
> at
> org.apache.hadoop.hive.ql.lockmgr.DbTxnManager.heartbeat(DbTxnManager.java:590)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.lockmgr.DbTxnManager$Heartbeater.lambda$run$0(DbTxnManager.java:956)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at java.security.AccessController.doPrivileged(Native Method)
> ~[?:1.8.0_102]
> at javax.security.auth.Subject.doAs(Subject.java:422) ~[?:1.8.0_102]
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1682)
> ~[hadoop-common-3.1.0.jar:?]
> ... 8 more
> Caused by: org.apache.hadoop.hive.metastore.api.MetaException: Unable to
> select from transaction database java.sql.SQLException: Derby thread received
> an interrupt during a disk I/O operation, please check your application for
> the source of the interrupt.
> at
> org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(Unknown Source)
> at
> org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(Unknown Source)
> at org.apache.derby.impl.jdbc.Util.seeNextException(Unknown Source)
> at
> org.apache.derby.impl.jdbc.TransactionResourceImpl.wrapInSQLException(Unknown
> Source)
> at
> org.apache.derby.impl.jdbc.TransactionResourceImpl.handleException(Unknown
> Source)
> at org.apache.derby.impl.jdbc.EmbedConnection.handleException(Unknown
> Source)
> at org.apache.derby.impl.jdbc.ConnectionChild.handleException(Unknown
> Source)
> at org.apache.derby.impl.jdbc.EmbedStatement.execute(Unknown Source)
> at org.apache.derby.impl.jdbc.EmbedStatement.executeLargeUpdate(Unknown
> Source)
> at org.apache.derby.impl.jdbc.EmbedStatement.executeUpdate(Unknown
> Source)
> at
> com.zaxxer.hikari.pool.ProxyStatement.executeUpdate(ProxyStatement.java:117)
> at
>
[jira] [Commented] (HIVE-20746) HiveProtoHookLogger does not close file at end of day.
[
https://issues.apache.org/jira/browse/HIVE-20746?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16652024#comment-16652024
]
Hive QA commented on HIVE-20746:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
43s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
45s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
23s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
53s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
34s{color} | {color:blue} common in master has 65 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
57s{color} | {color:blue} ql in master has 2318 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
11s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
10s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
10s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 27m 28s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-14501/dev-support/hive-personality.sh
|
| git revision | master / 0f2f999 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: common ql U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-14501/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> HiveProtoHookLogger does not close file at end of day.
> --
>
> Key: HIVE-20746
> URL: https://issues.apache.org/jira/browse/HIVE-20746
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Reporter: Harish Jaiprakash
>Assignee: Harish Jaiprakash
>Priority: Major
> Attachments: HIVE-20746.01.patch, HIVE-20746.02.patch
>
>
> The file rotation happens with an event currently. If there are no queries
> fired for a long time, then the file rotation does not happen and we do not
> close the file. This causes the clients to poll for the file for an
> indeterminate amount of time. If there are multiple hiveservers there is no
> way to tell which file will get more data. Fix this to close at end of day.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20753) Derby thread interrupt during ptest
[
https://issues.apache.org/jira/browse/HIVE-20753?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16651993#comment-16651993
]
Zoltan Haindrich commented on HIVE-20753:
-
we seem to be using a pretty recent derby...10.14.1.0 - 10.14.2.0 doesn't seem
to bring any bug fixes relating to this...
> Derby thread interrupt during ptest
> ---
>
> Key: HIVE-20753
> URL: https://issues.apache.org/jira/browse/HIVE-20753
> Project: Hive
> Issue Type: Bug
>Reporter: Zoltan Haindrich
>Priority: Major
> Attachments: derby.log
>
>
> I've had another failed ptest executionit seems like derby have caught an
> unexpected interrupt call ; which have hanged the execution; after that
> nothing happend for about half an hour - after which batch timeout have
> happened...
> {code}
> Caused by: ERROR XSDG9: Derby thread received an interrupt during a disk I/O
> operation, please check your application for the source of the interrupt.
> at org.apache.derby.iapi.error.StandardException.newException(Unknown
> Source)
> at
> org.apache.derby.impl.jdbc.SQLExceptionFactory.wrapArgsForTransportAcrossDRDA(Unknown
> Source)
> ... 42 more
> {code}
> full stacktrack:
> {code}
> 2018-10-16T06:47:29,355 ERROR [Heartbeater-3] lockmgr.DbTxnManager: Failed
> trying to heartbeat queryId=null, currentUser: hiveptest (auth:SIMPLE): null
> java.lang.reflect.UndeclaredThrowableException: null
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1700)
> ~[hadoop-common-3.1.0.jar:?]
> at
> org.apache.hadoop.hive.ql.lockmgr.DbTxnManager$Heartbeater.run(DbTxnManager.java:955)
> [hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> [?:1.8.0_102]
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> [?:1.8.0_102]
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> [?:1.8.0_102]
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> [?:1.8.0_102]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> [?:1.8.0_102]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> [?:1.8.0_102]
> at java.lang.Thread.run(Thread.java:745) [?:1.8.0_102]
> Caused by: org.apache.hadoop.hive.ql.lockmgr.LockException: Error
> communicating with the metastore(txnid:15,lockid:0 queryId=null txnid:0)
> at
> org.apache.hadoop.hive.ql.lockmgr.DbTxnManager.heartbeat(DbTxnManager.java:590)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.lockmgr.DbTxnManager$Heartbeater.lambda$run$0(DbTxnManager.java:956)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at java.security.AccessController.doPrivileged(Native Method)
> ~[?:1.8.0_102]
> at javax.security.auth.Subject.doAs(Subject.java:422) ~[?:1.8.0_102]
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1682)
> ~[hadoop-common-3.1.0.jar:?]
> ... 8 more
> Caused by: org.apache.hadoop.hive.metastore.api.MetaException: Unable to
> select from transaction database java.sql.SQLException: Derby thread received
> an interrupt during a disk I/O operation, please check your application for
> the source of the interrupt.
> at
> org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(Unknown Source)
> at
> org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(Unknown Source)
> at org.apache.derby.impl.jdbc.Util.seeNextException(Unknown Source)
> at
> org.apache.derby.impl.jdbc.TransactionResourceImpl.wrapInSQLException(Unknown
> Source)
> at
> org.apache.derby.impl.jdbc.TransactionResourceImpl.handleException(Unknown
> Source)
> at org.apache.derby.impl.jdbc.EmbedConnection.handleException(Unknown
> Source)
> at org.apache.derby.impl.jdbc.ConnectionChild.handleException(Unknown
> Source)
> at org.apache.derby.impl.jdbc.EmbedStatement.execute(Unknown Source)
> at org.apache.derby.impl.jdbc.EmbedStatement.executeLargeUpdate(Unknown
> Source)
> at org.apache.derby.impl.jdbc.EmbedStatement.executeUpdate(Unknown
> Source)
> at
> com.zaxxer.hikari.pool.ProxyStatement.executeUpdate(ProxyStatement.java:117)
> at
> com.zaxxer.hikari.pool.HikariProxyStatement.executeUpdate(HikariProxyStatement.java)
> at
> org.apache.hadoop.hive.metastore.txn.TxnHandler.heartbeatTxn(TxnHandler.java:4405)
> at
> org.apache.hadoop.hive.metastore.txn.TxnHandler.heartbeat(TxnHandler.java:2659)
> at
>
[jira] [Commented] (HIVE-17231) ColumnizedDeleteEventRegistry.DeleteReaderValue optimization
[
https://issues.apache.org/jira/browse/HIVE-17231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16651986#comment-16651986
]
Eugene Koifman commented on HIVE-17231:
---
[~gopalv], ping
> ColumnizedDeleteEventRegistry.DeleteReaderValue optimization
>
>
> Key: HIVE-17231
> URL: https://issues.apache.org/jira/browse/HIVE-17231
> Project: Hive
> Issue Type: Sub-task
> Components: Transactions
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
>Priority: Major
> Attachments: HIVE-17231.01.patch, HIVE-17231.02.patch
>
>
> For unbucketed tables DeleteReaderValue will currently return all delete
> events. Once we trust that
> the N in bucketN for "base" spit is reliable, all delete events not
> matching N can be skipped.
> This is useful to protect against extreme cases where someone runs an
> update/delete on a partition that matches 10 billion rows thus generates very
> many delete events.
> Since HIVE-19890 all acid data files must have bucketid/writerid in the file
> name match bucketid/writerid in ROW__ID in the data.
> {{OrcRawRecrodMerger.getDeltaFiles()}} should only return files representing
> the right {{bucket}}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
