[jira] [Commented] (HIVE-11282) CBO (Calcite Return Path): Inferring Hive type char/varchar of length zero which is not allowed
[ https://issues.apache.org/jira/browse/HIVE-11282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14630050#comment-14630050 ] Ashutosh Chauhan commented on HIVE-11282: - +1 CBO (Calcite Return Path): Inferring Hive type char/varchar of length zero which is not allowed --- Key: HIVE-11282 URL: https://issues.apache.org/jira/browse/HIVE-11282 Project: Hive Issue Type: Sub-task Components: CBO Reporter: Jesus Camacho Rodriguez Assignee: Jesus Camacho Rodriguez Attachments: HIVE-11282.patch When RT is on, we try to infer the Hive type from the Calcite type for the value '’ e.g. in udf3.q, and we end up with char (length=0) as a result. The min length of char/varchar in Hive is 1, thus an Exception is thrown. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11244) Beeline prompt info improvement for cluster mode
[
https://issues.apache.org/jira/browse/HIVE-11244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14630051#comment-14630051
]
Hive QA commented on HIVE-11244:
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12745623/HIVE-11244.1.patch
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 9220 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cbo_rp_join0
org.apache.hive.jdbc.TestJdbcWithLocalClusterSpark.testTempTable
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/4629/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/4629/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-4629/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12745623 - PreCommit-HIVE-TRUNK-Build
Beeline prompt info improvement for cluster mode
Key: HIVE-11244
URL: https://issues.apache.org/jira/browse/HIVE-11244
Project: Hive
Issue Type: Improvement
Components: Beeline
Affects Versions: 1.2.0
Reporter: Nemon Lou
Assignee: Nemon Lou
Priority: Minor
Labels: TODOC2.0
Attachments: HIVE-11244.1.patch, HIVE-11244.patch, HIVE-11244.patch,
HIVE-11244.patch
Currently Beeline prompt info for Cluster mode is like this:
{noformat}
0: jdbc:hive2://192.168.115.1:24002,192.168.1
{noformat}
Using the very HiveServer2's IP that this beeline connect to is more helpful
for users.
Like this:
{noformat}
0: jdbc:hive2://192.168.115.1:24002
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-11281) Memoize toStringTree() for ASTNode
[
https://issues.apache.org/jira/browse/HIVE-11281?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14630078#comment-14630078
]
Prasanth Jayachandran commented on HIVE-11281:
--
nits:
Shouldn't the following code come after buf.append(t.toStringTree()); to avoid
double space appends
{code}
buf.append(t.toStringTree());
{code}
Rename str to astStr or something more relevant?
Memoize toStringTree() for ASTNode
--
Key: HIVE-11281
URL: https://issues.apache.org/jira/browse/HIVE-11281
Project: Hive
Issue Type: Bug
Components: Query Planning
Affects Versions: 1.0.0, 1.2.0, 1.1.0
Reporter: Ashutosh Chauhan
Assignee: Ashutosh Chauhan
Attachments: HIVE-11281.patch
Currently it defaults to BaseTree::toStringTree() which always computes it
recursively. For very large trees it becomes expensive. On one particular
test query compilation time went down from 55 seconds to 2 seconds.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (HIVE-11281) Memoize toStringTree() for ASTNode
[
https://issues.apache.org/jira/browse/HIVE-11281?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14630078#comment-14630078
]
Prasanth Jayachandran edited comment on HIVE-11281 at 7/16/15 5:53 PM:
---
nits:
Shouldn't the following code come after buf.append(t.toStringTree()); to avoid
double space appends
{code}
if ( i0 ) {
buf.append(' ');
}
{code}
Rename str to astStr or something more relevant?
was (Author: prasanth_j):
nits:
Shouldn't the following code come after buf.append(t.toStringTree()); to avoid
double space appends
{code}
buf.append(t.toStringTree());
{code}
Rename str to astStr or something more relevant?
Memoize toStringTree() for ASTNode
--
Key: HIVE-11281
URL: https://issues.apache.org/jira/browse/HIVE-11281
Project: Hive
Issue Type: Bug
Components: Query Planning
Affects Versions: 1.0.0, 1.2.0, 1.1.0
Reporter: Ashutosh Chauhan
Assignee: Ashutosh Chauhan
Attachments: HIVE-11281.patch
Currently it defaults to BaseTree::toStringTree() which always computes it
recursively. For very large trees it becomes expensive. On one particular
test query compilation time went down from 55 seconds to 2 seconds.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-11279) Hive should emit lineage information in json compact format
[ https://issues.apache.org/jira/browse/HIVE-11279?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szehon Ho updated HIVE-11279: - Attachment: HIVE-11279.2.patch Hive should emit lineage information in json compact format --- Key: HIVE-11279 URL: https://issues.apache.org/jira/browse/HIVE-11279 Project: Hive Issue Type: Bug Components: Logging Affects Versions: 1.3.0 Reporter: Lenni Kuff Assignee: Lenni Kuff Attachments: HIVE-11279.1.patch, HIVE-11279.2.patch Hive should emit lineage information in json compact format. Currently, Hive prints this in human readable format which makes it harder to consume (identify record boundaries) and makes the output files very long. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-11281) Memoize toStringTree() for ASTNode
[ https://issues.apache.org/jira/browse/HIVE-11281?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-11281: Attachment: HIVE-11281.2.patch Patch with review feedback addressed. Memoize toStringTree() for ASTNode -- Key: HIVE-11281 URL: https://issues.apache.org/jira/browse/HIVE-11281 Project: Hive Issue Type: Bug Components: Query Planning Affects Versions: 1.0.0, 1.2.0, 1.1.0 Reporter: Ashutosh Chauhan Assignee: Ashutosh Chauhan Attachments: HIVE-11281.2.patch, HIVE-11281.patch Currently it defaults to BaseTree::toStringTree() which always computes it recursively. For very large trees it becomes expensive. On one particular test query compilation time went down from 55 seconds to 2 seconds. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (HIVE-11263) LLAP: TaskExecutorService state is not cleaned up
[
https://issues.apache.org/jira/browse/HIVE-11263?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siddharth Seth reassigned HIVE-11263:
-
Assignee: Siddharth Seth (was: Gunther Hagleitner)
LLAP: TaskExecutorService state is not cleaned up
-
Key: HIVE-11263
URL: https://issues.apache.org/jira/browse/HIVE-11263
Project: Hive
Issue Type: Sub-task
Reporter: Sergey Shelukhin
Assignee: Siddharth Seth
See TaskExecutorService::getExecutorsStatus, this is used to report on
queue/etc. status in JMX.
Currently, it reports 100s of bogus tasks in queue:
{noformat}
ExecutorsStatus : [ attempt_1435700346116_1888_1_04_000205_22
(sershe_20150714174105_0d013941-1f0e-4f74-9387-a2f29279a185:3/Map 1, in
queue), attempt_1435700346116_1889_1_05_000101_1
(sershe_20150714174104_b0b9f300-667e-4370-bb01-f9cb7da331e0:4/Map 1, in
queue), attempt_1435700346116_1889_1_05_000191_3
(sershe_20150714174104_b0b9f300-667e-4370-bb01-f9cb7da331e0:4/Map 1, in
queue), attempt_1435700346116_1887_7_00_000202_3
(sershe_20150714174737_bea682d1-fa0f-4281-a1cb-439d85bb2016:22/Map 5, in
queue), attempt_1435700346116_1886_1_04_73_15
(sershe_20150714174108_f9483d76-8fd9-4f82-96ee-17231b6f9b2c:1/Reducer 2, in
queue), attempt_1435700346116_1887_8_04_000166_15
(sershe_20150714174900_710d7d69-3d66-45e9-865b-cd0f87bb0d98:27/Map 1, in
queue), attempt_1435700346116_1888_11_05_000140_3
(sershe_20150714174903_50359459-5342-4d1b-852c-622a3fa92a27:28/Map 3, in
queue), attempt_1435700346116_1886_1_04_42_29
(sershe_20150714174108_f9483d76-8fd9-4f82-96ee-17231b6f9b2c:1/Reducer 2, in
queue), attempt_1435700346116_1888_2_03_000169_12
(sershe_20150714174310_97ce1d4b-8029-4ef6-a823-46e29f09718a:5/Map 1, in
queue), attempt_1435700346116_1887_1_04_000197_18
(sershe_20150714174107_8fcfe954-4eeb-46e5-bad5-42a47327b26c:2/Map 1, in
queue), attempt_1435700346116_1887_1_04_000218_21
(sershe_20150714174107_8fcfe954-4eeb-46e5-bad5-42a47327b26c:2/Map 1, in
queue), attempt_1435700346116_1886_7_09_84_1
(sershe_20150714174841_462b9bdb-c017-47c2-9fa7-7edfbfc09e60:24/Map 1, in
queue), attempt_1435700346116_1887_5_04_78_0
(sershe_20150714174509_9a5cd476-b3c8-4679-af8e-1188922713a2:14/Map 3, in
queue), attempt_1435700346116_1887_7_04_000162_6
(sershe_20150714174737_bea682d1-fa0f-4281-a1cb-439d85bb2016:22/Map 3, in
queue), attempt_1435700346116_1887_7_04_000180_0
(sershe_20150714174737_bea682d1-fa0f-4281-a1cb-439d85bb2016:22/Map 3, in
queue), attempt_1435700346116_1886_3_04_000144_0
(sershe_20150714174435_fe3077dd-a97f-4582-995b-5f723170b02f:12/Reducer 2, in
queue), attempt_1435700346116_1887_5_00_000153_1
(sershe_20150714174509_9a5cd476-b3c8-4679-af8e-1188922713a2:14/Map 5, in
queue), attempt_1435700346116_1887_7_04_000141_7
(sershe_20150714174737_bea682d1-fa0f-4281-a1cb-439d85bb2016:22/Map 3, in
queue), attempt_1435700346116_1887_1_04_24_7
(sershe_20150714174107_8fcfe954-4eeb-46e5-bad5-42a47327b26c:2/Map 1, in
queue), attempt_1435700346116_1887_5_04_000130_1
(sershe_20150714174509_9a5cd476-b3c8-4679-af8e-1188922713a2:14/Map 3, in
queue), attempt_1435700346116_1888_1_04_000200_1
(sershe_20150714174105_0d013941-1f0e-4f74-9387-a2f29279a185:3/Map 1, in
queue), attempt_1435700346116_1886_15_04_000180_0
(sershe_20150714175411_bda950b7-8aa5-417f-84f6-dd646247dca8:43/Map 1, in
queue), attempt_1435700346116_1887_7_00_000205_1
(sershe_20150714174737_bea682d1-fa0f-4281-a1cb-439d85bb2016:22/Map 5, in
queue), attempt_1435700346116_1888_4_04_000183_4
(sershe_20150714174407_f0924540-f69f-45c2-831a-9d2d1f66a124:10/Map 1, in
queue), attempt_1435700346116_1887_1_04_81_6
(sershe_20150714174107_8fcfe954-4eeb-46e5-bad5-42a47327b26c:2/Map 1, in
queue), attempt_1435700346116_1888_1_04_80_4
(sershe_20150714174105_0d013941-1f0e-4f74-9387-a2f29279a185:3/Map 1, in
queue), attempt_1435700346116_1887_7_04_05_3
(sershe_20150714174737_bea682d1-fa0f-4281-a1cb-439d85bb2016:22/Map 3, in
queue), attempt_1435700346116_1887_7_00_000169_2
(sershe_20150714174737_bea682d1-fa0f-4281-a1cb-439d85bb2016:22/Map 5, in
queue), attempt_1435700346116_1888_8_04_37_2
(sershe_20150714174731_261f2d52-8c47-4db6-8f17-8098efe144a2:20/Reducer 3, in
queue), attempt_1435700346116_1887_9_00_96_6
(sershe_20150714175015_cc1b6647-8479-4c5f-918c-00935bff7232:30/Map 5, in
queue), attempt_1435700346116_1888_11_01_01_2
(sershe_20150714174903_50359459-5342-4d1b-852c-622a3fa92a27:28/Map 7, in
queue), attempt_1435700346116_1889_1_05_000206_8
(sershe_20150714174104_b0b9f300-667e-4370-bb01-f9cb7da331e0:4/Map 1, in
queue), attempt_1435700346116_1887_5_04_54_0
(sershe_20150714174509_9a5cd476-b3c8-4679-af8e-1188922713a2:14/Map 3, in
queue),
[jira] [Commented] (HIVE-11279) Hive should emit lineage information in json compact format
[
https://issues.apache.org/jira/browse/HIVE-11279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14629499#comment-14629499
]
Hive QA commented on HIVE-11279:
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12745589/HIVE-11279.1.patch
{color:red}ERROR:{color} -1 due to 5 failed/errored test(s), 9189 tests executed
*Failed tests:*
{noformat}
TestCliDriver-timestamp_null.q-udf_xpath.q-groupby2_map_skew.q-and-12-more -
did not produce a TEST-*.xml file
TestCliDriver-udf_bitmap_empty.q-index_auto_update.q-quotedid_basic.q-and-12-more
- did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cbo_rp_join0
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_lineage2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_lineage3
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/4627/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/4627/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-4627/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 5 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12745589 - PreCommit-HIVE-TRUNK-Build
Hive should emit lineage information in json compact format
---
Key: HIVE-11279
URL: https://issues.apache.org/jira/browse/HIVE-11279
Project: Hive
Issue Type: Bug
Components: Logging
Affects Versions: 1.3.0
Reporter: Lenni Kuff
Assignee: Lenni Kuff
Attachments: HIVE-11279.1.patch
Hive should emit lineage information in json compact format. Currently, Hive
prints this in human readable format which makes it harder to consume
(identify record boundaries) and makes the output files very long.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (HIVE-11277) Merge master to parquet 06/16/2015 [Parquet branch]
[ https://issues.apache.org/jira/browse/HIVE-11277?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ferdinand Xu resolved HIVE-11277. - Resolution: Fixed Merge the master to the parquet branch commit 68fd78ddb15a78ee5b68f78e84ea11af435d7b94 Merge: 5a0ae3c e61a1a9 Author: Ferdinand Xu cheng.a...@intel.com Date: Thu Jul 16 03:51:03 2015 -0400 HIVE-11277: Merge master to parquet 06/16/2015 [Parquet branch] Merge master to parquet 06/16/2015 [Parquet branch] --- Key: HIVE-11277 URL: https://issues.apache.org/jira/browse/HIVE-11277 Project: Hive Issue Type: Sub-task Reporter: Ferdinand Xu Assignee: Ferdinand Xu -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-11279) Hive should emit lineage information in json compact format
[ https://issues.apache.org/jira/browse/HIVE-11279?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lenni Kuff updated HIVE-11279: -- Attachment: HIVE-11279.1.patch Hive should emit lineage information in json compact format --- Key: HIVE-11279 URL: https://issues.apache.org/jira/browse/HIVE-11279 Project: Hive Issue Type: Bug Components: Logging Affects Versions: 1.3.0 Reporter: Lenni Kuff Assignee: Lenni Kuff Attachments: HIVE-11279.1.patch Hive should emit lineage information in json compact format. Currently, Hive prints this in human readable format which makes it harder to consume (identify record boundaries) and makes the output files very long. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-10884) Enable some beeline tests and turn on HIVE-4239 by default
[ https://issues.apache.org/jira/browse/HIVE-10884?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergio Peña updated HIVE-10884: --- Attachment: HIVE-10884.07.patch Enable some beeline tests and turn on HIVE-4239 by default -- Key: HIVE-10884 URL: https://issues.apache.org/jira/browse/HIVE-10884 Project: Hive Issue Type: Bug Reporter: Sergey Shelukhin Assignee: Sergey Shelukhin Attachments: HIVE-10884.01.patch, HIVE-10884.02.patch, HIVE-10884.03.patch, HIVE-10884.04.patch, HIVE-10884.05.patch, HIVE-10884.06.patch, HIVE-10884.07.patch, HIVE-10884.07.patch, HIVE-10884.patch See comments in HIVE-4239. Beeline tests with parallelism need to be enabled to turn compilation parallelism on by default. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11227) Kryo exception during table creation in Hive
[ https://issues.apache.org/jira/browse/HIVE-11227?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14629601#comment-14629601 ] Akamai commented on HIVE-11227: --- Hi Xuefu Thank for the response, Please find the attached queries and error logs. Kryo exception during table creation in Hive Key: HIVE-11227 URL: https://issues.apache.org/jira/browse/HIVE-11227 Project: Hive Issue Type: Bug Components: Configuration, Database/Schema, Hive, HiveServer2, Indexing, Locking Affects Versions: 0.13.1 Environment: CentOS 6.5, jdk 1.7, cpu: 2x1.9 GHz 6-core Xeon (24 cores), Ram: 64GB-128GB Reporter: Akamai Fix For: 0.14.1 Attachments: Kryo Exception.txt, init_load_hdpextract_user.tlog.clean.log, tlog_detail.20150710.log.clean, trsm_tlog_detail.20150714.log.clean Exception is getting thorwn during table creation in Hive Error: java.lang.RuntimeException: org.apache.hive.com/esotericsoftware.kryo.KryoException: Encountered unregistered class ID: 380 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-11227) Kryo exception during table creation in Hive
[ https://issues.apache.org/jira/browse/HIVE-11227?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akamai updated HIVE-11227: -- Attachment: trsm_tlog_detail.20150714.log.clean tlog_detail.20150710.log.clean init_load_hdpextract_user.tlog.clean.log Queries and error logs. Kryo exception during table creation in Hive Key: HIVE-11227 URL: https://issues.apache.org/jira/browse/HIVE-11227 Project: Hive Issue Type: Bug Components: Configuration, Database/Schema, Hive, HiveServer2, Indexing, Locking Affects Versions: 0.13.1 Environment: CentOS 6.5, jdk 1.7, cpu: 2x1.9 GHz 6-core Xeon (24 cores), Ram: 64GB-128GB Reporter: Akamai Fix For: 0.14.1 Attachments: Kryo Exception.txt, init_load_hdpextract_user.tlog.clean.log, tlog_detail.20150710.log.clean, trsm_tlog_detail.20150714.log.clean Exception is getting thorwn during table creation in Hive Error: java.lang.RuntimeException: org.apache.hive.com/esotericsoftware.kryo.KryoException: Encountered unregistered class ID: 380 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-11244) Beeline prompt info improvement for cluster mode
[
https://issues.apache.org/jira/browse/HIVE-11244?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nemon Lou updated HIVE-11244:
-
Attachment: HIVE-11244.1.patch
This patch has been tested on my cluster ,and the parameter works.
{noformat}
beeline --showConnectedUrl
{noformat}
And only the connected HiveServer2's IP shows up.
{noformat}
0: jdbc:hive2://192.168.115.1:21066/
{noformat}
Patch name has been updated following Hive community's convention .
Thanks [~Ferd] for your guide.
Beeline prompt info improvement for cluster mode
Key: HIVE-11244
URL: https://issues.apache.org/jira/browse/HIVE-11244
Project: Hive
Issue Type: Improvement
Components: Beeline
Affects Versions: 1.2.0
Reporter: Nemon Lou
Assignee: Nemon Lou
Priority: Minor
Labels: TODOC2.0
Attachments: HIVE-11244.1.patch, HIVE-11244.patch, HIVE-11244.patch,
HIVE-11244.patch
Currently Beeline prompt info for Cluster mode is like this:
{noformat}
0: jdbc:hive2://192.168.115.1:24002,192.168.1
{noformat}
Using the very HiveServer2's IP that this beeline connect to is more helpful
for users.
Like this:
{noformat}
0: jdbc:hive2://192.168.115.1:24002
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-11271) java.lang.IndexOutOfBoundsException when union all with if function
[
https://issues.apache.org/jira/browse/HIVE-11271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14629820#comment-14629820
]
Yongzhi Chen commented on HIVE-11271:
-
This is related to ppd, when set hive.optimize.ppd=false; the query works fine.
java.lang.IndexOutOfBoundsException when union all with if function
---
Key: HIVE-11271
URL: https://issues.apache.org/jira/browse/HIVE-11271
Project: Hive
Issue Type: Bug
Components: Logical Optimizer
Affects Versions: 0.14.0, 1.0.0, 1.2.0
Reporter: Yongzhi Chen
Assignee: Yongzhi Chen
Some queries with Union all as subquery fail in MapReduce task with
stacktrace:
{noformat}
15/07/15 14:19:30 [pool-13-thread-1]: INFO exec.UnionOperator: Initializing
operator UNION[104]
15/07/15 14:19:30 [Thread-72]: INFO mapred.LocalJobRunner: Map task executor
complete.
15/07/15 14:19:30 [Thread-72]: WARN mapred.LocalJobRunner:
job_local826862759_0005
java.lang.Exception: java.lang.RuntimeException: Error in configuring object
at
org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:354)
Caused by: java.lang.RuntimeException: Error in configuring object
at
org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:93)
at
org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:64)
at
org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:426)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:366)
at
org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:223)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.GeneratedMethodAccessor53.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:88)
... 10 more
Caused by: java.lang.RuntimeException: Error in configuring object
at
org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:93)
at
org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:64)
at
org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117)
at org.apache.hadoop.mapred.MapRunner.configure(MapRunner.java:34)
... 14 more
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.GeneratedMethodAccessor53.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:88)
... 17 more
Caused by: java.lang.RuntimeException: Map operator initialization failed
at
org.apache.hadoop.hive.ql.exec.mr.ExecMapper.configure(ExecMapper.java:140)
... 21 more
Caused by: java.lang.IndexOutOfBoundsException: Index: 1, Size: 1
at java.util.ArrayList.rangeCheck(ArrayList.java:635)
at java.util.ArrayList.get(ArrayList.java:411)
at
org.apache.hadoop.hive.ql.exec.UnionOperator.initializeOp(UnionOperator.java:86)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:362)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:481)
at
org.apache.hadoop.hive.ql.exec.Operator.initializeChildren(Operator.java:438)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:375)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:481)
at
org.apache.hadoop.hive.ql.exec.Operator.initializeChildren(Operator.java:438)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:375)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:481)
at
org.apache.hadoop.hive.ql.exec.Operator.initializeChildren(Operator.java:438)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:375)
at
org.apache.hadoop.hive.ql.exec.MapOperator.initializeMapOperator(MapOperator.java:442)
at
org.apache.hadoop.hive.ql.exec.mr.ExecMapper.configure(ExecMapper.java:119)
... 21 more
{noformat}
Reproduce:
{noformat}
[jira] [Commented] (HIVE-11254) Process result sets returned by a stored procedure
[
https://issues.apache.org/jira/browse/HIVE-11254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14629814#comment-14629814
]
Dmitry Tolpeko commented on HIVE-11254:
---
DB2 approach:
{code}
DECLARE loc RESULT_SET_LOCATOR VARYING;
CALL spResultSet1;
ASSOCIATE RESULT SET LOCATOR (loc) WITH PROCEDURE spResultSet1;
ALLOCATE c1 CURSOR FOR RESULT SET loc1;
FETCH c1 INTO v1, v2;
{code}
Process result sets returned by a stored procedure
--
Key: HIVE-11254
URL: https://issues.apache.org/jira/browse/HIVE-11254
Project: Hive
Issue Type: Improvement
Components: hpl/sql
Reporter: Dmitry Tolpeko
Assignee: Dmitry Tolpeko
Stored procedure can return one or more result sets. A caller should be able
to process them.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-11271) java.lang.IndexOutOfBoundsException when union all with if function
[
https://issues.apache.org/jira/browse/HIVE-11271?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yongzhi Chen updated HIVE-11271:
Component/s: Logical Optimizer
java.lang.IndexOutOfBoundsException when union all with if function
---
Key: HIVE-11271
URL: https://issues.apache.org/jira/browse/HIVE-11271
Project: Hive
Issue Type: Bug
Components: Logical Optimizer
Affects Versions: 0.14.0, 1.0.0, 1.2.0
Reporter: Yongzhi Chen
Assignee: Yongzhi Chen
Some queries with Union all as subquery fail in MapReduce task with
stacktrace:
{noformat}
15/07/15 14:19:30 [pool-13-thread-1]: INFO exec.UnionOperator: Initializing
operator UNION[104]
15/07/15 14:19:30 [Thread-72]: INFO mapred.LocalJobRunner: Map task executor
complete.
15/07/15 14:19:30 [Thread-72]: WARN mapred.LocalJobRunner:
job_local826862759_0005
java.lang.Exception: java.lang.RuntimeException: Error in configuring object
at
org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:354)
Caused by: java.lang.RuntimeException: Error in configuring object
at
org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:93)
at
org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:64)
at
org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:426)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:366)
at
org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:223)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.GeneratedMethodAccessor53.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:88)
... 10 more
Caused by: java.lang.RuntimeException: Error in configuring object
at
org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:93)
at
org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:64)
at
org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117)
at org.apache.hadoop.mapred.MapRunner.configure(MapRunner.java:34)
... 14 more
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.GeneratedMethodAccessor53.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:88)
... 17 more
Caused by: java.lang.RuntimeException: Map operator initialization failed
at
org.apache.hadoop.hive.ql.exec.mr.ExecMapper.configure(ExecMapper.java:140)
... 21 more
Caused by: java.lang.IndexOutOfBoundsException: Index: 1, Size: 1
at java.util.ArrayList.rangeCheck(ArrayList.java:635)
at java.util.ArrayList.get(ArrayList.java:411)
at
org.apache.hadoop.hive.ql.exec.UnionOperator.initializeOp(UnionOperator.java:86)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:362)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:481)
at
org.apache.hadoop.hive.ql.exec.Operator.initializeChildren(Operator.java:438)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:375)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:481)
at
org.apache.hadoop.hive.ql.exec.Operator.initializeChildren(Operator.java:438)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:375)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:481)
at
org.apache.hadoop.hive.ql.exec.Operator.initializeChildren(Operator.java:438)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:375)
at
org.apache.hadoop.hive.ql.exec.MapOperator.initializeMapOperator(MapOperator.java:442)
at
org.apache.hadoop.hive.ql.exec.mr.ExecMapper.configure(ExecMapper.java:119)
... 21 more
{noformat}
Reproduce:
{noformat}
create table if not exists union_all_bug_test_1
(
f1 int,
f2 int
);
create table if not exists
[jira] [Updated] (HIVE-11281) Memoize toStringTree() for ASTNode
[ https://issues.apache.org/jira/browse/HIVE-11281?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-11281: Attachment: HIVE-11281.patch Memoize toStringTree() for ASTNode -- Key: HIVE-11281 URL: https://issues.apache.org/jira/browse/HIVE-11281 Project: Hive Issue Type: Bug Components: Query Planning Affects Versions: 1.0.0, 1.2.0, 1.1.0 Reporter: Ashutosh Chauhan Assignee: Ashutosh Chauhan Attachments: HIVE-11281.patch Currently it defaults to BaseTree::toStringTree() which always computes it recursively. For very large trees it becomes expensive. On one particular test query compilation time went down from 55 seconds to 2 seconds. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-11282) CBO (Calcite Return Path): Inferring Hive type char/varchar of length zero which is not allowed
[ https://issues.apache.org/jira/browse/HIVE-11282?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-11282: --- Attachment: HIVE-11282.patch CBO (Calcite Return Path): Inferring Hive type char/varchar of length zero which is not allowed --- Key: HIVE-11282 URL: https://issues.apache.org/jira/browse/HIVE-11282 Project: Hive Issue Type: Sub-task Components: CBO Reporter: Jesus Camacho Rodriguez Assignee: Jesus Camacho Rodriguez Attachments: HIVE-11282.patch When RT is on, we try to infer the Hive type from the Calcite type for the value '’ e.g. in udf3.q, and we end up with char (length=0) as a result. The min length of char/varchar in Hive is 1, thus an Exception is thrown. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11279) Hive should emit lineage information in json compact format
[ https://issues.apache.org/jira/browse/HIVE-11279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14630009#comment-14630009 ] Szehon Ho commented on HIVE-11279: -- +1. Looks like the two test outputs need to be re-generated with the new format, can take care of that on commit. Hive should emit lineage information in json compact format --- Key: HIVE-11279 URL: https://issues.apache.org/jira/browse/HIVE-11279 Project: Hive Issue Type: Bug Components: Logging Affects Versions: 1.3.0 Reporter: Lenni Kuff Assignee: Lenni Kuff Attachments: HIVE-11279.1.patch Hive should emit lineage information in json compact format. Currently, Hive prints this in human readable format which makes it harder to consume (identify record boundaries) and makes the output files very long. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11271) java.lang.IndexOutOfBoundsException when union all with if function
[
https://issues.apache.org/jira/browse/HIVE-11271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14629978#comment-14629978
]
Pengcheng Xiong commented on HIVE-11271:
I tried to reproduce the problem too. If I remove WHERE (filter = 1), it also
works. [~ashutoshc], could you please take a brief look and see if it is
related to current improvement of PPD? Thanks.
java.lang.IndexOutOfBoundsException when union all with if function
---
Key: HIVE-11271
URL: https://issues.apache.org/jira/browse/HIVE-11271
Project: Hive
Issue Type: Bug
Components: Logical Optimizer
Affects Versions: 0.14.0, 1.0.0, 1.2.0
Reporter: Yongzhi Chen
Assignee: Yongzhi Chen
Some queries with Union all as subquery fail in MapReduce task with
stacktrace:
{noformat}
15/07/15 14:19:30 [pool-13-thread-1]: INFO exec.UnionOperator: Initializing
operator UNION[104]
15/07/15 14:19:30 [Thread-72]: INFO mapred.LocalJobRunner: Map task executor
complete.
15/07/15 14:19:30 [Thread-72]: WARN mapred.LocalJobRunner:
job_local826862759_0005
java.lang.Exception: java.lang.RuntimeException: Error in configuring object
at
org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:354)
Caused by: java.lang.RuntimeException: Error in configuring object
at
org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:93)
at
org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:64)
at
org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:426)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:366)
at
org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:223)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.GeneratedMethodAccessor53.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:88)
... 10 more
Caused by: java.lang.RuntimeException: Error in configuring object
at
org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:93)
at
org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:64)
at
org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117)
at org.apache.hadoop.mapred.MapRunner.configure(MapRunner.java:34)
... 14 more
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.GeneratedMethodAccessor53.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:88)
... 17 more
Caused by: java.lang.RuntimeException: Map operator initialization failed
at
org.apache.hadoop.hive.ql.exec.mr.ExecMapper.configure(ExecMapper.java:140)
... 21 more
Caused by: java.lang.IndexOutOfBoundsException: Index: 1, Size: 1
at java.util.ArrayList.rangeCheck(ArrayList.java:635)
at java.util.ArrayList.get(ArrayList.java:411)
at
org.apache.hadoop.hive.ql.exec.UnionOperator.initializeOp(UnionOperator.java:86)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:362)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:481)
at
org.apache.hadoop.hive.ql.exec.Operator.initializeChildren(Operator.java:438)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:375)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:481)
at
org.apache.hadoop.hive.ql.exec.Operator.initializeChildren(Operator.java:438)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:375)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:481)
at
org.apache.hadoop.hive.ql.exec.Operator.initializeChildren(Operator.java:438)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:375)
at
org.apache.hadoop.hive.ql.exec.MapOperator.initializeMapOperator(MapOperator.java:442)
at
[jira] [Commented] (HIVE-10884) Enable some beeline tests and turn on HIVE-4239 by default
[
https://issues.apache.org/jira/browse/HIVE-10884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14629926#comment-14629926
]
Hive QA commented on HIVE-10884:
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12745618/HIVE-10884.07.patch
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 9220 tests executed
*Failed tests:*
{noformat}
TestBeeLineDriver - did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cbo_rp_join0
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/4628/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/4628/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-4628/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12745618 - PreCommit-HIVE-TRUNK-Build
Enable some beeline tests and turn on HIVE-4239 by default
--
Key: HIVE-10884
URL: https://issues.apache.org/jira/browse/HIVE-10884
Project: Hive
Issue Type: Bug
Reporter: Sergey Shelukhin
Assignee: Sergey Shelukhin
Attachments: HIVE-10884.01.patch, HIVE-10884.02.patch,
HIVE-10884.03.patch, HIVE-10884.04.patch, HIVE-10884.05.patch,
HIVE-10884.06.patch, HIVE-10884.07.patch, HIVE-10884.07.patch,
HIVE-10884.patch
See comments in HIVE-4239.
Beeline tests with parallelism need to be enabled to turn compilation
parallelism on by default.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-11283) DROP table command with schema name broken in HIVE 14 release
[ https://issues.apache.org/jira/browse/HIVE-11283?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Khaja Hussain updated HIVE-11283: - Description: With Hive 14 when we run drop command with the schema.table it fails. This feature was working in Hive 13. Hive Version: * Hive 0.14.0.2.2.4.2-2 Subversion git://ip-10-0-0-54.ec2.internal/grid/0/jenkins/workspace/HDP-2.2.4.1-centos5/bigtop/build/hive/rpm/BUILD/hive-0.14.0.2.2.4.2 -r 115d99896f5a4a81e7d91e052e8d38d7436b78d4 Compiled by jenkins on Tue Mar 31 16:03:30 EDT 2015 From source with checksum 1f34a1d4e566c3e801582862ed85ee93 Commands Execution: * hive (dev_sft_pos_usf_c09013_ps799_bre000_pfc0_spr104_pfs0) DROP TABLE IF EXISTS dev_sft_pos_usf_c09013_ps799_bre000_pfc0_spr104_pfs0.aggr_store_aiv_mon; OK Time taken: 0.608 seconds hive (dev_sft_pos_usf_c09013_ps799_bre000_pfc0_spr104_pfs0) use default; OK Time taken: 0.222 seconds hive (default) DROP TABLE IF EXISTS dev_sft_pos_usf_c09013_ps799_bre000_pfc0_spr104_pfs0.basedondata; FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. org.apache.thrift.TApplicationException: Internal error processing get_partition_names hive (default) was: With Hive 14 when we run drop command with the schema.table it fails. Where this feature was working in Hive 13. Hive Version: * Hive 0.14.0.2.2.4.2-2 Subversion git://ip-10-0-0-54.ec2.internal/grid/0/jenkins/workspace/HDP-2.2.4.1-centos5/bigtop/build/hive/rpm/BUILD/hive-0.14.0.2.2.4.2 -r 115d99896f5a4a81e7d91e052e8d38d7436b78d4 Compiled by jenkins on Tue Mar 31 16:03:30 EDT 2015 From source with checksum 1f34a1d4e566c3e801582862ed85ee93 Commands Execution: * hive (dev_sft_pos_usf_c09013_ps799_bre000_pfc0_spr104_pfs0) DROP TABLE IF EXISTS dev_sft_pos_usf_c09013_ps799_bre000_pfc0_spr104_pfs0.aggr_store_aiv_mon; OK Time taken: 0.608 seconds hive (dev_sft_pos_usf_c09013_ps799_bre000_pfc0_spr104_pfs0) use default; OK Time taken: 0.222 seconds hive (default) DROP TABLE IF EXISTS dev_sft_pos_usf_c09013_ps799_bre000_pfc0_spr104_pfs0.basedondata; FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. org.apache.thrift.TApplicationException: Internal error processing get_partition_names hive (default) DROP table command with schema name broken in HIVE 14 release - Key: HIVE-11283 URL: https://issues.apache.org/jira/browse/HIVE-11283 Project: Hive Issue Type: Bug Components: Database/Schema Affects Versions: 0.14.0 Reporter: Khaja Hussain Priority: Minor With Hive 14 when we run drop command with the schema.table it fails. This feature was working in Hive 13. Hive Version: * Hive 0.14.0.2.2.4.2-2 Subversion git://ip-10-0-0-54.ec2.internal/grid/0/jenkins/workspace/HDP-2.2.4.1-centos5/bigtop/build/hive/rpm/BUILD/hive-0.14.0.2.2.4.2 -r 115d99896f5a4a81e7d91e052e8d38d7436b78d4 Compiled by jenkins on Tue Mar 31 16:03:30 EDT 2015 From source with checksum 1f34a1d4e566c3e801582862ed85ee93 Commands Execution: * hive (dev_sft_pos_usf_c09013_ps799_bre000_pfc0_spr104_pfs0) DROP TABLE IF EXISTS dev_sft_pos_usf_c09013_ps799_bre000_pfc0_spr104_pfs0.aggr_store_aiv_mon; OK Time taken: 0.608 seconds hive (dev_sft_pos_usf_c09013_ps799_bre000_pfc0_spr104_pfs0) use default; OK Time taken: 0.222 seconds hive (default) DROP TABLE IF EXISTS dev_sft_pos_usf_c09013_ps799_bre000_pfc0_spr104_pfs0.basedondata; FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. org.apache.thrift.TApplicationException: Internal error processing get_partition_names hive (default) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-11263) LLAP: TaskExecutorService state is not cleaned up
[
https://issues.apache.org/jira/browse/HIVE-11263?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siddharth Seth updated HIVE-11263:
--
Attachment: HIVE-11263.1.txt
Patch to clean up knownTasks in case of waitQueue preemption. That was not
happening earlier.
[~prasanth_j] - please review. This also changes schedule to actually throw a
RejectedExecutionException.
LLAP: TaskExecutorService state is not cleaned up
-
Key: HIVE-11263
URL: https://issues.apache.org/jira/browse/HIVE-11263
Project: Hive
Issue Type: Sub-task
Reporter: Sergey Shelukhin
Assignee: Siddharth Seth
Attachments: HIVE-11263.1.txt
See TaskExecutorService::getExecutorsStatus, this is used to report on
queue/etc. status in JMX.
Currently, it reports 100s of bogus tasks in queue:
{noformat}
ExecutorsStatus : [ attempt_1435700346116_1888_1_04_000205_22
(sershe_20150714174105_0d013941-1f0e-4f74-9387-a2f29279a185:3/Map 1, in
queue), attempt_1435700346116_1889_1_05_000101_1
(sershe_20150714174104_b0b9f300-667e-4370-bb01-f9cb7da331e0:4/Map 1, in
queue), attempt_1435700346116_1889_1_05_000191_3
(sershe_20150714174104_b0b9f300-667e-4370-bb01-f9cb7da331e0:4/Map 1, in
queue), attempt_1435700346116_1887_7_00_000202_3
(sershe_20150714174737_bea682d1-fa0f-4281-a1cb-439d85bb2016:22/Map 5, in
queue), attempt_1435700346116_1886_1_04_73_15
(sershe_20150714174108_f9483d76-8fd9-4f82-96ee-17231b6f9b2c:1/Reducer 2, in
queue), attempt_1435700346116_1887_8_04_000166_15
(sershe_20150714174900_710d7d69-3d66-45e9-865b-cd0f87bb0d98:27/Map 1, in
queue), attempt_1435700346116_1888_11_05_000140_3
(sershe_20150714174903_50359459-5342-4d1b-852c-622a3fa92a27:28/Map 3, in
queue), attempt_1435700346116_1886_1_04_42_29
(sershe_20150714174108_f9483d76-8fd9-4f82-96ee-17231b6f9b2c:1/Reducer 2, in
queue), attempt_1435700346116_1888_2_03_000169_12
(sershe_20150714174310_97ce1d4b-8029-4ef6-a823-46e29f09718a:5/Map 1, in
queue), attempt_1435700346116_1887_1_04_000197_18
(sershe_20150714174107_8fcfe954-4eeb-46e5-bad5-42a47327b26c:2/Map 1, in
queue), attempt_1435700346116_1887_1_04_000218_21
(sershe_20150714174107_8fcfe954-4eeb-46e5-bad5-42a47327b26c:2/Map 1, in
queue), attempt_1435700346116_1886_7_09_84_1
(sershe_20150714174841_462b9bdb-c017-47c2-9fa7-7edfbfc09e60:24/Map 1, in
queue), attempt_1435700346116_1887_5_04_78_0
(sershe_20150714174509_9a5cd476-b3c8-4679-af8e-1188922713a2:14/Map 3, in
queue), attempt_1435700346116_1887_7_04_000162_6
(sershe_20150714174737_bea682d1-fa0f-4281-a1cb-439d85bb2016:22/Map 3, in
queue), attempt_1435700346116_1887_7_04_000180_0
(sershe_20150714174737_bea682d1-fa0f-4281-a1cb-439d85bb2016:22/Map 3, in
queue), attempt_1435700346116_1886_3_04_000144_0
(sershe_20150714174435_fe3077dd-a97f-4582-995b-5f723170b02f:12/Reducer 2, in
queue), attempt_1435700346116_1887_5_00_000153_1
(sershe_20150714174509_9a5cd476-b3c8-4679-af8e-1188922713a2:14/Map 5, in
queue), attempt_1435700346116_1887_7_04_000141_7
(sershe_20150714174737_bea682d1-fa0f-4281-a1cb-439d85bb2016:22/Map 3, in
queue), attempt_1435700346116_1887_1_04_24_7
(sershe_20150714174107_8fcfe954-4eeb-46e5-bad5-42a47327b26c:2/Map 1, in
queue), attempt_1435700346116_1887_5_04_000130_1
(sershe_20150714174509_9a5cd476-b3c8-4679-af8e-1188922713a2:14/Map 3, in
queue), attempt_1435700346116_1888_1_04_000200_1
(sershe_20150714174105_0d013941-1f0e-4f74-9387-a2f29279a185:3/Map 1, in
queue), attempt_1435700346116_1886_15_04_000180_0
(sershe_20150714175411_bda950b7-8aa5-417f-84f6-dd646247dca8:43/Map 1, in
queue), attempt_1435700346116_1887_7_00_000205_1
(sershe_20150714174737_bea682d1-fa0f-4281-a1cb-439d85bb2016:22/Map 5, in
queue), attempt_1435700346116_1888_4_04_000183_4
(sershe_20150714174407_f0924540-f69f-45c2-831a-9d2d1f66a124:10/Map 1, in
queue), attempt_1435700346116_1887_1_04_81_6
(sershe_20150714174107_8fcfe954-4eeb-46e5-bad5-42a47327b26c:2/Map 1, in
queue), attempt_1435700346116_1888_1_04_80_4
(sershe_20150714174105_0d013941-1f0e-4f74-9387-a2f29279a185:3/Map 1, in
queue), attempt_1435700346116_1887_7_04_05_3
(sershe_20150714174737_bea682d1-fa0f-4281-a1cb-439d85bb2016:22/Map 3, in
queue), attempt_1435700346116_1887_7_00_000169_2
(sershe_20150714174737_bea682d1-fa0f-4281-a1cb-439d85bb2016:22/Map 5, in
queue), attempt_1435700346116_1888_8_04_37_2
(sershe_20150714174731_261f2d52-8c47-4db6-8f17-8098efe144a2:20/Reducer 3, in
queue), attempt_1435700346116_1887_9_00_96_6
(sershe_20150714175015_cc1b6647-8479-4c5f-918c-00935bff7232:30/Map 5, in
queue), attempt_1435700346116_1888_11_01_01_2
(sershe_20150714174903_50359459-5342-4d1b-852c-622a3fa92a27:28/Map 7, in
queue), attempt_1435700346116_1889_1_05_000206_8
[jira] [Commented] (HIVE-11281) Memoize toStringTree() for ASTNode
[ https://issues.apache.org/jira/browse/HIVE-11281?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14630249#comment-14630249 ] Ashutosh Chauhan commented on HIVE-11281: - Didn't follow you about double spacing. Also, just trying to be consistent with [BaseTree::toStringTree()|https://github.com/antlr/antlr3/blob/master/runtime/Java/src/main/java/org/antlr/runtime/tree/BaseTree.java#L375] in actual string representation, which does same thing. Will rename the field. Test failures are unrelated. Memoize toStringTree() for ASTNode -- Key: HIVE-11281 URL: https://issues.apache.org/jira/browse/HIVE-11281 Project: Hive Issue Type: Bug Components: Query Planning Affects Versions: 1.0.0, 1.2.0, 1.1.0 Reporter: Ashutosh Chauhan Assignee: Ashutosh Chauhan Attachments: HIVE-11281.patch Currently it defaults to BaseTree::toStringTree() which always computes it recursively. For very large trees it becomes expensive. On one particular test query compilation time went down from 55 seconds to 2 seconds. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-11285) ObjectInspector for partition columns in FetchOperator in SMBJoin causes exception
[
https://issues.apache.org/jira/browse/HIVE-11285?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pengcheng Xiong updated HIVE-11285:
---
Attachment: HIVE-11285.01.patch
[~ashutoshc], could you please review the patch? Thanks.
ObjectInspector for partition columns in FetchOperator in SMBJoin causes
exception
--
Key: HIVE-11285
URL: https://issues.apache.org/jira/browse/HIVE-11285
Project: Hive
Issue Type: Bug
Reporter: Pengcheng Xiong
Assignee: Pengcheng Xiong
Attachments: HIVE-11285.01.patch
{code}
STEPS TO REPRODUCE:
*$ cat data.out
1|One
2|Two
hql
CREATE TABLE data_table (key INT, value STRING) ROW FORMAT DELIMITED FIELDS
TERMINATED BY '|';
LOAD DATA LOCAL INPATH '$
{system:user.dir}
/data.out' INTO TABLE data_table;
CREATE TABLE smb_table (key INT, value STRING)
CLUSTERED BY (key)
SORTED BY (key) INTO 1 BUCKETS
STORED AS ORC;
CREATE TABLE smb_table_part (key INT, value STRING)
PARTITIONED BY (p1 DECIMAL)
CLUSTERED BY (key)
SORTED BY (key) INTO 1 BUCKETS
STORED AS ORC;
INSERT OVERWRITE TABLE smb_table SELECT * FROM data_table;
INSERT OVERWRITE TABLE smb_table_part PARTITION (p1) SELECT key, value, 100
as p1 FROM data_table;
SET hive.execution.engine=mr;
SET hive.enforce.sortmergebucketmapjoin=false;
SET hive.auto.convert.sortmerge.join=true;
SET hive.optimize.bucketmapjoin = true;
SET hive.optimize.bucketmapjoin.sortedmerge = true;
SET hive.input.format =
org.apache.hadoop.hive.ql.io.BucketizedHiveInputFormat;
SELECT s1.key, s2.p1
FROM smb_table s1
INNER JOIN smb_table_part s2
ON s1.key = s2.key
ORDER BY s1.key;
ERROR:
2015-07-15 13:39:04,333 WARN main org.apache.hadoop.mapred.YarnChild:
Exception running child : java.lang.RuntimeException:
org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while
processing row
{key:1,value:One}
at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:185)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:450)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime
Error while processing row
{key:1,value:One}
at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:503)
at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:176)
... 8 more
Caused by: java.lang.RuntimeException: Map local work failed
at
org.apache.hadoop.hive.ql.exec.SMBMapJoinOperator.fetchOneRow(SMBMapJoinOperator.java:569)
at
org.apache.hadoop.hive.ql.exec.SMBMapJoinOperator.fetchNextGroup(SMBMapJoinOperator.java:429)
at
org.apache.hadoop.hive.ql.exec.SMBMapJoinOperator.processOp(SMBMapJoinOperator.java:260)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:815)
at
org.apache.hadoop.hive.ql.exec.FilterOperator.processOp(FilterOperator.java:120)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:815)
at
org.apache.hadoop.hive.ql.exec.TableScanOperator.processOp(TableScanOperator.java:95)
at
org.apache.hadoop.hive.ql.exec.MapOperator$MapOpCtx.forward(MapOperator.java:157)
at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:493)
... 9 more
Caused by: java.lang.ClassCastException: org.apache.hadoop.io.IntWritable
cannot be cast to java.lang.Integer
at
org.apache.hadoop.hive.serde2.objectinspector.primitive.JavaIntObjectInspector.getPrimitiveWritableObject(JavaIntObjectInspector.java:35)
at
org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorUtils.copyToStandardObject(ObjectInspectorUtils.java:305)
at org.apache.hadoop.hive.ql.exec.JoinUtil.computeValues(JoinUtil.java:193)
at
org.apache.hadoop.hive.ql.exec.CommonJoinOperator.getFilteredValue(CommonJoinOperator.java:408)
at
org.apache.hadoop.hive.ql.exec.SMBMapJoinOperator.processOp(SMBMapJoinOperator.java:270)
at
org.apache.hadoop.hive.ql.exec.SMBMapJoinOperator.fetchOneRow(SMBMapJoinOperator.java:558)
... 17 more
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-11281) Memoize toStringTree() for ASTNode
[ https://issues.apache.org/jira/browse/HIVE-11281?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14630255#comment-14630255 ] Prasanth Jayachandran commented on HIVE-11281: -- the white space is not an issue. my bad. never mind. Memoize toStringTree() for ASTNode -- Key: HIVE-11281 URL: https://issues.apache.org/jira/browse/HIVE-11281 Project: Hive Issue Type: Bug Components: Query Planning Affects Versions: 1.0.0, 1.2.0, 1.1.0 Reporter: Ashutosh Chauhan Assignee: Ashutosh Chauhan Attachments: HIVE-11281.patch Currently it defaults to BaseTree::toStringTree() which always computes it recursively. For very large trees it becomes expensive. On one particular test query compilation time went down from 55 seconds to 2 seconds. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11284) Fix cbo_rp_join0 failure on master
[ https://issues.apache.org/jira/browse/HIVE-11284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14630257#comment-14630257 ] Ashutosh Chauhan commented on HIVE-11284: - [~jcamachorodriguez] Would you like to take a look? Seems like either HIVE-11251 or HIVE-11223 broke it? Fix cbo_rp_join0 failure on master -- Key: HIVE-11284 URL: https://issues.apache.org/jira/browse/HIVE-11284 Project: Hive Issue Type: Bug Components: CBO, Query Planning Affects Versions: 2.0.0 Reporter: Ashutosh Chauhan It first failed in this build : http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/4621/ -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11281) Memoize toStringTree() for ASTNode
[
https://issues.apache.org/jira/browse/HIVE-11281?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14630280#comment-14630280
]
Hari Sankar Sivarama Subramaniyan commented on HIVE-11281:
--
[~ashutoshc]
Cant we just do this to simplify this inside ASTNode
{code}
@Override
public String toStringTree() {
if (null != str) {
return str;
}
return str=super.toStringTree();
}
{code}
Memoize toStringTree() for ASTNode
--
Key: HIVE-11281
URL: https://issues.apache.org/jira/browse/HIVE-11281
Project: Hive
Issue Type: Bug
Components: Query Planning
Affects Versions: 1.0.0, 1.2.0, 1.1.0
Reporter: Ashutosh Chauhan
Assignee: Ashutosh Chauhan
Attachments: HIVE-11281.2.patch, HIVE-11281.patch
Currently it defaults to BaseTree::toStringTree() which always computes it
recursively. For very large trees it becomes expensive. On one particular
test query compilation time went down from 55 seconds to 2 seconds.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-11262) Skip MapJoin processing if the join hash table is empty
[ https://issues.apache.org/jira/browse/HIVE-11262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14630113#comment-14630113 ] Vikram Dixit K commented on HIVE-11262: --- +1 LGTM. Skip MapJoin processing if the join hash table is empty --- Key: HIVE-11262 URL: https://issues.apache.org/jira/browse/HIVE-11262 Project: Hive Issue Type: Improvement Components: Query Processor Reporter: Jason Dere Assignee: Jason Dere Attachments: HIVE-11262.1.patch, HIVE-11262.2.patch Currently the map join processor processes all rows of the big table, even when the hash table is empty. If it is an inner join, we should be able to skip the join processing, since the result should be empty. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-11273) LLAP: Register for finishable state change notifications when adding a task instead of when scheduling it
[ https://issues.apache.org/jira/browse/HIVE-11273?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated HIVE-11273: -- Attachment: HIVE-11273.1.txt Patch to fix this to ensure a task sitting in the wait queue is re-ordered when it moves to finishable state. [~prasanth_j] - could you please take a quick look. There's a test for the condition. LLAP: Register for finishable state change notifications when adding a task instead of when scheduling it - Key: HIVE-11273 URL: https://issues.apache.org/jira/browse/HIVE-11273 Project: Hive Issue Type: Sub-task Reporter: Siddharth Seth Assignee: Siddharth Seth Fix For: llap Attachments: HIVE-11273.1.txt Registering when trying to execute is far too late. The task won't be considered for execution (queue may not be re-oredered) without the notification coming in. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11281) Memoize toStringTree() for ASTNode
[
https://issues.apache.org/jira/browse/HIVE-11281?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14630200#comment-14630200
]
Hive QA commented on HIVE-11281:
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12745636/HIVE-11281.patch
{color:red}ERROR:{color} -1 due to 3 failed/errored test(s), 9215 tests executed
*Failed tests:*
{noformat}
TestContribNegativeCliDriver - did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cbo_rp_join0
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_schemeAuthority
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/4630/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/4630/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-4630/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 3 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12745636 - PreCommit-HIVE-TRUNK-Build
Memoize toStringTree() for ASTNode
--
Key: HIVE-11281
URL: https://issues.apache.org/jira/browse/HIVE-11281
Project: Hive
Issue Type: Bug
Components: Query Planning
Affects Versions: 1.0.0, 1.2.0, 1.1.0
Reporter: Ashutosh Chauhan
Assignee: Ashutosh Chauhan
Attachments: HIVE-11281.patch
Currently it defaults to BaseTree::toStringTree() which always computes it
recursively. For very large trees it becomes expensive. On one particular
test query compilation time went down from 55 seconds to 2 seconds.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-11285) ObjectInspector for partition columns in FetchOperator in SMBJoin causes exception
[
https://issues.apache.org/jira/browse/HIVE-11285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14630356#comment-14630356
]
Ashutosh Chauhan commented on HIVE-11285:
-
+1 pending tests
ObjectInspector for partition columns in FetchOperator in SMBJoin causes
exception
--
Key: HIVE-11285
URL: https://issues.apache.org/jira/browse/HIVE-11285
Project: Hive
Issue Type: Bug
Reporter: Pengcheng Xiong
Assignee: Pengcheng Xiong
Attachments: HIVE-11285.01.patch
{code}
STEPS TO REPRODUCE:
*$ cat data.out
1|One
2|Two
hql
CREATE TABLE data_table (key INT, value STRING) ROW FORMAT DELIMITED FIELDS
TERMINATED BY '|';
LOAD DATA LOCAL INPATH '$
{system:user.dir}
/data.out' INTO TABLE data_table;
CREATE TABLE smb_table (key INT, value STRING)
CLUSTERED BY (key)
SORTED BY (key) INTO 1 BUCKETS
STORED AS ORC;
CREATE TABLE smb_table_part (key INT, value STRING)
PARTITIONED BY (p1 DECIMAL)
CLUSTERED BY (key)
SORTED BY (key) INTO 1 BUCKETS
STORED AS ORC;
INSERT OVERWRITE TABLE smb_table SELECT * FROM data_table;
INSERT OVERWRITE TABLE smb_table_part PARTITION (p1) SELECT key, value, 100
as p1 FROM data_table;
SET hive.execution.engine=mr;
SET hive.enforce.sortmergebucketmapjoin=false;
SET hive.auto.convert.sortmerge.join=true;
SET hive.optimize.bucketmapjoin = true;
SET hive.optimize.bucketmapjoin.sortedmerge = true;
SET hive.input.format =
org.apache.hadoop.hive.ql.io.BucketizedHiveInputFormat;
SELECT s1.key, s2.p1
FROM smb_table s1
INNER JOIN smb_table_part s2
ON s1.key = s2.key
ORDER BY s1.key;
ERROR:
2015-07-15 13:39:04,333 WARN main org.apache.hadoop.mapred.YarnChild:
Exception running child : java.lang.RuntimeException:
org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while
processing row
{key:1,value:One}
at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:185)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:450)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime
Error while processing row
{key:1,value:One}
at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:503)
at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:176)
... 8 more
Caused by: java.lang.RuntimeException: Map local work failed
at
org.apache.hadoop.hive.ql.exec.SMBMapJoinOperator.fetchOneRow(SMBMapJoinOperator.java:569)
at
org.apache.hadoop.hive.ql.exec.SMBMapJoinOperator.fetchNextGroup(SMBMapJoinOperator.java:429)
at
org.apache.hadoop.hive.ql.exec.SMBMapJoinOperator.processOp(SMBMapJoinOperator.java:260)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:815)
at
org.apache.hadoop.hive.ql.exec.FilterOperator.processOp(FilterOperator.java:120)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:815)
at
org.apache.hadoop.hive.ql.exec.TableScanOperator.processOp(TableScanOperator.java:95)
at
org.apache.hadoop.hive.ql.exec.MapOperator$MapOpCtx.forward(MapOperator.java:157)
at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:493)
... 9 more
Caused by: java.lang.ClassCastException: org.apache.hadoop.io.IntWritable
cannot be cast to java.lang.Integer
at
org.apache.hadoop.hive.serde2.objectinspector.primitive.JavaIntObjectInspector.getPrimitiveWritableObject(JavaIntObjectInspector.java:35)
at
org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorUtils.copyToStandardObject(ObjectInspectorUtils.java:305)
at org.apache.hadoop.hive.ql.exec.JoinUtil.computeValues(JoinUtil.java:193)
at
org.apache.hadoop.hive.ql.exec.CommonJoinOperator.getFilteredValue(CommonJoinOperator.java:408)
at
org.apache.hadoop.hive.ql.exec.SMBMapJoinOperator.processOp(SMBMapJoinOperator.java:270)
at
org.apache.hadoop.hive.ql.exec.SMBMapJoinOperator.fetchOneRow(SMBMapJoinOperator.java:558)
... 17 more
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-11244) Beeline prompt info improvement for cluster mode
[
https://issues.apache.org/jira/browse/HIVE-11244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14630664#comment-14630664
]
Ferdinand Xu commented on HIVE-11244:
-
Could you take a look at the failed test case
org.apache.hive.jdbc.TestJdbcWithLocalClusterSpark.testTempTable?
Also some minor issues about code style, you can refer the wiki for details.
https://cwiki.apache.org/confluence/display/Hive/HowToContribute#HowToContribute-CodingConventions
{noformat}
1236 return getPrompt(getDatabaseConnections().getIndex() 1238
return getPrompt(getDatabaseConnections().getIndex()
1237 + : + getDatabaseConnection().getUrl()) + printClosed +
; 1239 + : + url) + printClosed + ;
{noformat}
No need for breaking line.
{noformat}
181 \ --showConnectedUrl=[true/false] prompt HiveServer2's
IP to which this beeline connected.Only works for HiveServer2 cluster mode.\n \
{noformat}
Extra space needed after the first sentence and IP should be URI.
Beeline prompt info improvement for cluster mode
Key: HIVE-11244
URL: https://issues.apache.org/jira/browse/HIVE-11244
Project: Hive
Issue Type: Improvement
Components: Beeline
Affects Versions: 1.2.0
Reporter: Nemon Lou
Assignee: Nemon Lou
Priority: Minor
Labels: TODOC2.0
Attachments: HIVE-11244.1.patch, HIVE-11244.patch, HIVE-11244.patch,
HIVE-11244.patch
Currently Beeline prompt info for Cluster mode is like this:
{noformat}
0: jdbc:hive2://192.168.115.1:24002,192.168.1
{noformat}
Using the very HiveServer2's IP that this beeline connect to is more helpful
for users.
Like this:
{noformat}
0: jdbc:hive2://192.168.115.1:24002
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-11276) Optimization around job submission and adding jars [Spark Branch]
[ https://issues.apache.org/jira/browse/HIVE-11276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14630682#comment-14630682 ] Chengxiang Li commented on HIVE-11276: -- [~xuefuz], I review the the code in RemoteHiveSparkClient, the reason why it need to invoke refreshLocalResources() for every job submission is that Hive user may use ADD \[FILE|JAR|ARCHIVE\] value command to add resources on runtime, so spark client need to upload these resources to spark cluster before job execution. RemoteHiveSparkClient have a list which records all the resources it has uploaded to spark cluster, and use it to filter out already uploaded jars during refreshLocalResources(), only new added jars would be uploaded to spark cluster, and the list should have a quite small size at most time, so i think it should not has performance issue here. Optimization around job submission and adding jars [Spark Branch] - Key: HIVE-11276 URL: https://issues.apache.org/jira/browse/HIVE-11276 Project: Hive Issue Type: Sub-task Components: Spark Affects Versions: 1.1.0 Reporter: Xuefu Zhang Assignee: Chengxiang Li It seems that Hive on Spark has some room for performance improvement on job submission. Specifically, we are calling refreshLocalResources() for every job submission despite there is are no changes in the jar list. Since Hive on Spark is reusing the containers in the whole user session, we might be able to optimize that. We do need to take into consideration the case of dynamic allocation, in which new executors might be added. This task is some RD in this area. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11285) ObjectInspector for partition columns in FetchOperator in SMBJoin causes exception
[
https://issues.apache.org/jira/browse/HIVE-11285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14630751#comment-14630751
]
Pengcheng Xiong commented on HIVE-11285:
[~ashutoshc], the test failure is unrelated. Thus, i think the patch is ready
to go. Thanks.
ObjectInspector for partition columns in FetchOperator in SMBJoin causes
exception
--
Key: HIVE-11285
URL: https://issues.apache.org/jira/browse/HIVE-11285
Project: Hive
Issue Type: Bug
Reporter: Pengcheng Xiong
Assignee: Pengcheng Xiong
Attachments: HIVE-11285.01.patch
{code}
STEPS TO REPRODUCE:
*$ cat data.out
1|One
2|Two
hql
CREATE TABLE data_table (key INT, value STRING) ROW FORMAT DELIMITED FIELDS
TERMINATED BY '|';
LOAD DATA LOCAL INPATH '$
{system:user.dir}
/data.out' INTO TABLE data_table;
CREATE TABLE smb_table (key INT, value STRING)
CLUSTERED BY (key)
SORTED BY (key) INTO 1 BUCKETS
STORED AS ORC;
CREATE TABLE smb_table_part (key INT, value STRING)
PARTITIONED BY (p1 DECIMAL)
CLUSTERED BY (key)
SORTED BY (key) INTO 1 BUCKETS
STORED AS ORC;
INSERT OVERWRITE TABLE smb_table SELECT * FROM data_table;
INSERT OVERWRITE TABLE smb_table_part PARTITION (p1) SELECT key, value, 100
as p1 FROM data_table;
SET hive.execution.engine=mr;
SET hive.enforce.sortmergebucketmapjoin=false;
SET hive.auto.convert.sortmerge.join=true;
SET hive.optimize.bucketmapjoin = true;
SET hive.optimize.bucketmapjoin.sortedmerge = true;
SET hive.input.format =
org.apache.hadoop.hive.ql.io.BucketizedHiveInputFormat;
SELECT s1.key, s2.p1
FROM smb_table s1
INNER JOIN smb_table_part s2
ON s1.key = s2.key
ORDER BY s1.key;
ERROR:
2015-07-15 13:39:04,333 WARN main org.apache.hadoop.mapred.YarnChild:
Exception running child : java.lang.RuntimeException:
org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while
processing row
{key:1,value:One}
at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:185)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:450)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime
Error while processing row
{key:1,value:One}
at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:503)
at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:176)
... 8 more
Caused by: java.lang.RuntimeException: Map local work failed
at
org.apache.hadoop.hive.ql.exec.SMBMapJoinOperator.fetchOneRow(SMBMapJoinOperator.java:569)
at
org.apache.hadoop.hive.ql.exec.SMBMapJoinOperator.fetchNextGroup(SMBMapJoinOperator.java:429)
at
org.apache.hadoop.hive.ql.exec.SMBMapJoinOperator.processOp(SMBMapJoinOperator.java:260)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:815)
at
org.apache.hadoop.hive.ql.exec.FilterOperator.processOp(FilterOperator.java:120)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:815)
at
org.apache.hadoop.hive.ql.exec.TableScanOperator.processOp(TableScanOperator.java:95)
at
org.apache.hadoop.hive.ql.exec.MapOperator$MapOpCtx.forward(MapOperator.java:157)
at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:493)
... 9 more
Caused by: java.lang.ClassCastException: org.apache.hadoop.io.IntWritable
cannot be cast to java.lang.Integer
at
org.apache.hadoop.hive.serde2.objectinspector.primitive.JavaIntObjectInspector.getPrimitiveWritableObject(JavaIntObjectInspector.java:35)
at
org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorUtils.copyToStandardObject(ObjectInspectorUtils.java:305)
at org.apache.hadoop.hive.ql.exec.JoinUtil.computeValues(JoinUtil.java:193)
at
org.apache.hadoop.hive.ql.exec.CommonJoinOperator.getFilteredValue(CommonJoinOperator.java:408)
at
org.apache.hadoop.hive.ql.exec.SMBMapJoinOperator.processOp(SMBMapJoinOperator.java:270)
at
org.apache.hadoop.hive.ql.exec.SMBMapJoinOperator.fetchOneRow(SMBMapJoinOperator.java:558)
... 17 more
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-11276) Optimization around job submission and adding jars [Spark Branch]
[ https://issues.apache.org/jira/browse/HIVE-11276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14630791#comment-14630791 ] Chengxiang Li commented on HIVE-11276: -- That make sense to me, launch the spark cluster during first query execution would mislead the user that Hive on Spark is slower than it actually does. Besides, we may also open spark session while user set hive.execution.engine to spark. Optimization around job submission and adding jars [Spark Branch] - Key: HIVE-11276 URL: https://issues.apache.org/jira/browse/HIVE-11276 Project: Hive Issue Type: Sub-task Components: Spark Affects Versions: 1.1.0 Reporter: Xuefu Zhang Assignee: Chengxiang Li It seems that Hive on Spark has some room for performance improvement on job submission. Specifically, we are calling refreshLocalResources() for every job submission despite there is are no changes in the jar list. Since Hive on Spark is reusing the containers in the whole user session, we might be able to optimize that. We do need to take into consideration the case of dynamic allocation, in which new executors might be added. This task is some RD in this area. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (HIVE-11276) Optimization around job submission and adding jars [Spark Branch]
[ https://issues.apache.org/jira/browse/HIVE-11276?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chengxiang Li reassigned HIVE-11276: Assignee: Chengxiang Li Optimization around job submission and adding jars [Spark Branch] - Key: HIVE-11276 URL: https://issues.apache.org/jira/browse/HIVE-11276 Project: Hive Issue Type: Sub-task Components: Spark Affects Versions: 1.1.0 Reporter: Xuefu Zhang Assignee: Chengxiang Li It seems that Hive on Spark has some room for performance improvement on job submission. Specifically, we are calling refreshLocalResources() for every job submission despite there is are no changes in the jar list. Since Hive on Spark is reusing the containers in the whole user session, we might be able to optimize that. We do need to take into consideration the case of dynamic allocation, in which new executors might be added. This task is some RD in this area. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11276) Optimization around job submission and adding jars [Spark Branch]
[ https://issues.apache.org/jira/browse/HIVE-11276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14630689#comment-14630689 ] Chengxiang Li commented on HIVE-11276: -- Besides, for the case of dynamic allocation, i'm not sure whether it would be influenced by this. From my point of view, as we use Spark API like SparkContext::addJar()/addFile() to upload resources to SparkCluster, after that, it should be Spark's responsibility to make sure it's executor JVM load these resources. From the experience of my previous test of dynamic allocation, everything works well. Optimization around job submission and adding jars [Spark Branch] - Key: HIVE-11276 URL: https://issues.apache.org/jira/browse/HIVE-11276 Project: Hive Issue Type: Sub-task Components: Spark Affects Versions: 1.1.0 Reporter: Xuefu Zhang Assignee: Chengxiang Li It seems that Hive on Spark has some room for performance improvement on job submission. Specifically, we are calling refreshLocalResources() for every job submission despite there is are no changes in the jar list. Since Hive on Spark is reusing the containers in the whole user session, we might be able to optimize that. We do need to take into consideration the case of dynamic allocation, in which new executors might be added. This task is some RD in this area. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-10982) Customizable the value of java.sql.statement.setFetchSize in Hive JDBC Driver
[ https://issues.apache.org/jira/browse/HIVE-10982?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14630732#comment-14630732 ] Pengcheng Xiong commented on HIVE-10982: [~libing], I saw you assigned the JIRA to yourself month ago. Sorry for pushing but do you still want to continue to work on this? Thanks. Customizable the value of java.sql.statement.setFetchSize in Hive JDBC Driver -- Key: HIVE-10982 URL: https://issues.apache.org/jira/browse/HIVE-10982 Project: Hive Issue Type: Improvement Components: JDBC Affects Versions: 1.2.0 Reporter: Bing Li Assignee: Bing Li Priority: Critical The current JDBC driver for Hive hard-code the value of setFetchSize to 50, which will be a bottleneck for performance. Pentaho filed this issue as http://jira.pentaho.com/browse/PDI-11511, whose status is open. Also it has discussion in http://forums.pentaho.com/showthread.php?158381-Hive-JDBC-Query-too-slow-too-many-fetches-after-query-execution-Kettle-Xform http://mail-archives.apache.org/mod_mbox/hive-user/201307.mbox/%3ccacq46vevgrfqg5rwxnr1psgyz7dcf07mvlo8mm2qit3anm1...@mail.gmail.com%3E -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11276) Optimization around job submission and adding jars [Spark Branch]
[ https://issues.apache.org/jira/browse/HIVE-11276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14630702#comment-14630702 ] Xuefu Zhang commented on HIVE-11276: Hi [~chengxiang li], your analysis is correct. I realized after I created this JIRA that we are not uploading the jars every time even though refreshLocalResources() is called. This is fine. Also, dynamic allocation worked well with the existing implementation. Therefore, the JIRA is not a problem. I'm going to close this one. I think we need a pre-warm containers for user sessions that executes only one query and then exits, such as those issued by Oozie. Spark session can be created right after user connects to Hive and the execution engine is Spark. This way, the remote driver and the executors will be up when the query comes. As part of that, some jars, such as hive-exec.jar, can be also uploaded to HDSF. Of course, connection will be slower. Thus, we need a configuration to turn on this. What do you think? Optimization around job submission and adding jars [Spark Branch] - Key: HIVE-11276 URL: https://issues.apache.org/jira/browse/HIVE-11276 Project: Hive Issue Type: Sub-task Components: Spark Affects Versions: 1.1.0 Reporter: Xuefu Zhang Assignee: Chengxiang Li It seems that Hive on Spark has some room for performance improvement on job submission. Specifically, we are calling refreshLocalResources() for every job submission despite there is are no changes in the jar list. Since Hive on Spark is reusing the containers in the whole user session, we might be able to optimize that. We do need to take into consideration the case of dynamic allocation, in which new executors might be added. This task is some RD in this area. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-11259) LLAP: clean up ORC dependencies part 1
[ https://issues.apache.org/jira/browse/HIVE-11259?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-11259: Attachment: HIVE-11259.patch A patch. Mostly moves code around. Removes all ORC dependencies on LLAP and moves 4 classes/interfaces to temporary location in hive-common, to be moved to storage-api when it's there. There's one more problem of what to do with hive-common pool dependency, I will look at it separately LLAP: clean up ORC dependencies part 1 -- Key: HIVE-11259 URL: https://issues.apache.org/jira/browse/HIVE-11259 Project: Hive Issue Type: Sub-task Reporter: Sergey Shelukhin Assignee: Sergey Shelukhin Attachments: HIVE-11259.patch Before there's storage handler module, we can clean some things up -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-11287) Hive Metastore does not tolerate leading spaces in JDBC url
[
https://issues.apache.org/jira/browse/HIVE-11287?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gopal V updated HIVE-11287:
---
Affects Version/s: 2.0.0
Hive Metastore does not tolerate leading spaces in JDBC url
---
Key: HIVE-11287
URL: https://issues.apache.org/jira/browse/HIVE-11287
Project: Hive
Issue Type: Bug
Components: Metastore
Affects Versions: 1.2.0, 1.3.0, 2.0.0
Reporter: Gopal V
Priority: Minor
Labels: newbie
The hive metastore is configured with
{code}
property
name
javax.jdo.option.ConnectionURL
/name
value
jdbc:mysql://hostname/hive
/value
/property
{code}
The initialization fails with an error
{code}
java.sql.SQLException: No suitable driver found for
jdbc:mysql://hostname/hive
at java.sql.DriverManager.getConnection(DriverManager.java:689)
at java.sql.DriverManager.getConnection(DriverManager.java:208)
at
com.jolbox.bonecp.BoneCP.obtainRawInternalConnection(BoneCP.java:361)
at com.jolbox.bonecp.BoneCP.obtainInternalConnection(BoneCP.java:269)
at
com.jolbox.bonecp.ConnectionHandle.init(ConnectionHandle.java:242)
at
com.jolbox.bonecp.PoolWatchThread.fillConnections(PoolWatchThread.java:115)
at com.jolbox.bonecp.PoolWatchThread.run(PoolWatchThread.java:85)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-11287) Hive Metastore does not tolerate leading spaces in JDBC url
[
https://issues.apache.org/jira/browse/HIVE-11287?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gopal V updated HIVE-11287:
---
Component/s: Configuration
Hive Metastore does not tolerate leading spaces in JDBC url
---
Key: HIVE-11287
URL: https://issues.apache.org/jira/browse/HIVE-11287
Project: Hive
Issue Type: Bug
Components: Configuration, Metastore
Affects Versions: 1.2.0, 1.3.0, 2.0.0
Reporter: Gopal V
Priority: Minor
Labels: newbie
The hive metastore is configured with
{code}
property
name
javax.jdo.option.ConnectionURL
/name
value
jdbc:mysql://hostname/hive
/value
/property
{code}
The initialization fails with an error
{code}
java.sql.SQLException: No suitable driver found for
jdbc:mysql://hostname/hive
at java.sql.DriverManager.getConnection(DriverManager.java:689)
at java.sql.DriverManager.getConnection(DriverManager.java:208)
at
com.jolbox.bonecp.BoneCP.obtainRawInternalConnection(BoneCP.java:361)
at com.jolbox.bonecp.BoneCP.obtainInternalConnection(BoneCP.java:269)
at
com.jolbox.bonecp.ConnectionHandle.init(ConnectionHandle.java:242)
at
com.jolbox.bonecp.PoolWatchThread.fillConnections(PoolWatchThread.java:115)
at com.jolbox.bonecp.PoolWatchThread.run(PoolWatchThread.java:85)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-11273) LLAP: Register for finishable state change notifications when adding a task instead of when scheduling it
[ https://issues.apache.org/jira/browse/HIVE-11273?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14630565#comment-14630565 ] Prasanth Jayachandran commented on HIVE-11273: -- Patch looks good to me +1. Although it will be better to have both of the patch together HIVE-11263 and this patch. Can this be combined into a single patch? LLAP: Register for finishable state change notifications when adding a task instead of when scheduling it - Key: HIVE-11273 URL: https://issues.apache.org/jira/browse/HIVE-11273 Project: Hive Issue Type: Sub-task Reporter: Siddharth Seth Assignee: Siddharth Seth Fix For: llap Attachments: HIVE-11273.1.txt Registering when trying to execute is far too late. The task won't be considered for execution (queue may not be re-oredered) without the notification coming in. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11281) Memoize toStringTree() for ASTNode
[ https://issues.apache.org/jira/browse/HIVE-11281?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14630584#comment-14630584 ] Ashutosh Chauhan commented on HIVE-11281: - I checked UpdateDeleteSemanticAnalyzer and that doesn't make use of StringTree. We can document in StringTree() method about this gotcha to warn future devs about this. Memoize toStringTree() for ASTNode -- Key: HIVE-11281 URL: https://issues.apache.org/jira/browse/HIVE-11281 Project: Hive Issue Type: Bug Components: Query Planning Affects Versions: 1.0.0, 1.2.0, 1.1.0 Reporter: Ashutosh Chauhan Assignee: Ashutosh Chauhan Attachments: HIVE-11281.2.patch, HIVE-11281.3.patch, HIVE-11281.patch Currently it defaults to BaseTree::toStringTree() which always computes it recursively. For very large trees it becomes expensive. On one particular test query compilation time went down from 55 seconds to 2 seconds. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HIVE-10174) LLAP: ORC MemoryManager is singleton synchronized
[ https://issues.apache.org/jira/browse/HIVE-10174?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gopal V resolved HIVE-10174. Resolution: Done Fixed in HIVE-10191 LLAP: ORC MemoryManager is singleton synchronized - Key: HIVE-10174 URL: https://issues.apache.org/jira/browse/HIVE-10174 Project: Hive Issue Type: Sub-task Components: File Formats Affects Versions: llap Reporter: Gopal V Assignee: Gopal V Attachments: orc-memorymanager-1.png, orc-memorymanager-2.png ORC MemoryManager::addedRow() checks are bad for LLAP multi-threaded performance. !orc-memorymanager-1.png! !orc-memorymanager-2.png! -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11263) LLAP: TaskExecutorService state is not cleaned up
[
https://issues.apache.org/jira/browse/HIVE-11263?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14630550#comment-14630550
]
Prasanth Jayachandran commented on HIVE-11263:
--
lgtm, +1
LLAP: TaskExecutorService state is not cleaned up
-
Key: HIVE-11263
URL: https://issues.apache.org/jira/browse/HIVE-11263
Project: Hive
Issue Type: Sub-task
Reporter: Sergey Shelukhin
Assignee: Siddharth Seth
Attachments: HIVE-11263.1.txt, HIVE-11263.2.txt
See TaskExecutorService::getExecutorsStatus, this is used to report on
queue/etc. status in JMX.
Currently, it reports 100s of bogus tasks in queue:
{noformat}
ExecutorsStatus : [ attempt_1435700346116_1888_1_04_000205_22
(sershe_20150714174105_0d013941-1f0e-4f74-9387-a2f29279a185:3/Map 1, in
queue), attempt_1435700346116_1889_1_05_000101_1
(sershe_20150714174104_b0b9f300-667e-4370-bb01-f9cb7da331e0:4/Map 1, in
queue), attempt_1435700346116_1889_1_05_000191_3
(sershe_20150714174104_b0b9f300-667e-4370-bb01-f9cb7da331e0:4/Map 1, in
queue), attempt_1435700346116_1887_7_00_000202_3
(sershe_20150714174737_bea682d1-fa0f-4281-a1cb-439d85bb2016:22/Map 5, in
queue), attempt_1435700346116_1886_1_04_73_15
(sershe_20150714174108_f9483d76-8fd9-4f82-96ee-17231b6f9b2c:1/Reducer 2, in
queue), attempt_1435700346116_1887_8_04_000166_15
(sershe_20150714174900_710d7d69-3d66-45e9-865b-cd0f87bb0d98:27/Map 1, in
queue), attempt_1435700346116_1888_11_05_000140_3
(sershe_20150714174903_50359459-5342-4d1b-852c-622a3fa92a27:28/Map 3, in
queue), attempt_1435700346116_1886_1_04_42_29
(sershe_20150714174108_f9483d76-8fd9-4f82-96ee-17231b6f9b2c:1/Reducer 2, in
queue), attempt_1435700346116_1888_2_03_000169_12
(sershe_20150714174310_97ce1d4b-8029-4ef6-a823-46e29f09718a:5/Map 1, in
queue), attempt_1435700346116_1887_1_04_000197_18
(sershe_20150714174107_8fcfe954-4eeb-46e5-bad5-42a47327b26c:2/Map 1, in
queue), attempt_1435700346116_1887_1_04_000218_21
(sershe_20150714174107_8fcfe954-4eeb-46e5-bad5-42a47327b26c:2/Map 1, in
queue), attempt_1435700346116_1886_7_09_84_1
(sershe_20150714174841_462b9bdb-c017-47c2-9fa7-7edfbfc09e60:24/Map 1, in
queue), attempt_1435700346116_1887_5_04_78_0
(sershe_20150714174509_9a5cd476-b3c8-4679-af8e-1188922713a2:14/Map 3, in
queue), attempt_1435700346116_1887_7_04_000162_6
(sershe_20150714174737_bea682d1-fa0f-4281-a1cb-439d85bb2016:22/Map 3, in
queue), attempt_1435700346116_1887_7_04_000180_0
(sershe_20150714174737_bea682d1-fa0f-4281-a1cb-439d85bb2016:22/Map 3, in
queue), attempt_1435700346116_1886_3_04_000144_0
(sershe_20150714174435_fe3077dd-a97f-4582-995b-5f723170b02f:12/Reducer 2, in
queue), attempt_1435700346116_1887_5_00_000153_1
(sershe_20150714174509_9a5cd476-b3c8-4679-af8e-1188922713a2:14/Map 5, in
queue), attempt_1435700346116_1887_7_04_000141_7
(sershe_20150714174737_bea682d1-fa0f-4281-a1cb-439d85bb2016:22/Map 3, in
queue), attempt_1435700346116_1887_1_04_24_7
(sershe_20150714174107_8fcfe954-4eeb-46e5-bad5-42a47327b26c:2/Map 1, in
queue), attempt_1435700346116_1887_5_04_000130_1
(sershe_20150714174509_9a5cd476-b3c8-4679-af8e-1188922713a2:14/Map 3, in
queue), attempt_1435700346116_1888_1_04_000200_1
(sershe_20150714174105_0d013941-1f0e-4f74-9387-a2f29279a185:3/Map 1, in
queue), attempt_1435700346116_1886_15_04_000180_0
(sershe_20150714175411_bda950b7-8aa5-417f-84f6-dd646247dca8:43/Map 1, in
queue), attempt_1435700346116_1887_7_00_000205_1
(sershe_20150714174737_bea682d1-fa0f-4281-a1cb-439d85bb2016:22/Map 5, in
queue), attempt_1435700346116_1888_4_04_000183_4
(sershe_20150714174407_f0924540-f69f-45c2-831a-9d2d1f66a124:10/Map 1, in
queue), attempt_1435700346116_1887_1_04_81_6
(sershe_20150714174107_8fcfe954-4eeb-46e5-bad5-42a47327b26c:2/Map 1, in
queue), attempt_1435700346116_1888_1_04_80_4
(sershe_20150714174105_0d013941-1f0e-4f74-9387-a2f29279a185:3/Map 1, in
queue), attempt_1435700346116_1887_7_04_05_3
(sershe_20150714174737_bea682d1-fa0f-4281-a1cb-439d85bb2016:22/Map 3, in
queue), attempt_1435700346116_1887_7_00_000169_2
(sershe_20150714174737_bea682d1-fa0f-4281-a1cb-439d85bb2016:22/Map 5, in
queue), attempt_1435700346116_1888_8_04_37_2
(sershe_20150714174731_261f2d52-8c47-4db6-8f17-8098efe144a2:20/Reducer 3, in
queue), attempt_1435700346116_1887_9_00_96_6
(sershe_20150714175015_cc1b6647-8479-4c5f-918c-00935bff7232:30/Map 5, in
queue), attempt_1435700346116_1888_11_01_01_2
(sershe_20150714174903_50359459-5342-4d1b-852c-622a3fa92a27:28/Map 7, in
queue), attempt_1435700346116_1889_1_05_000206_8
(sershe_20150714174104_b0b9f300-667e-4370-bb01-f9cb7da331e0:4/Map 1, in
queue), attempt_1435700346116_1887_5_04_54_0
[jira] [Commented] (HIVE-11281) Memoize toStringTree() for ASTNode
[ https://issues.apache.org/jira/browse/HIVE-11281?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14630562#comment-14630562 ] Eugene Koifman commented on HIVE-11281: --- UpdateDeleteSemanticAnalyzer does mutate AST. There may be others, but even if we currently didn't modify AST this change could created very subtle bugs (especially when trying to debug something based on logs) Memoize toStringTree() for ASTNode -- Key: HIVE-11281 URL: https://issues.apache.org/jira/browse/HIVE-11281 Project: Hive Issue Type: Bug Components: Query Planning Affects Versions: 1.0.0, 1.2.0, 1.1.0 Reporter: Ashutosh Chauhan Assignee: Ashutosh Chauhan Attachments: HIVE-11281.2.patch, HIVE-11281.3.patch, HIVE-11281.patch Currently it defaults to BaseTree::toStringTree() which always computes it recursively. For very large trees it becomes expensive. On one particular test query compilation time went down from 55 seconds to 2 seconds. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-11281) Memoize toStringTree() for ASTNode
[ https://issues.apache.org/jira/browse/HIVE-11281?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-11281: Attachment: HIVE-11281.3.patch Patch with Hari's suggestion. Memoize toStringTree() for ASTNode -- Key: HIVE-11281 URL: https://issues.apache.org/jira/browse/HIVE-11281 Project: Hive Issue Type: Bug Components: Query Planning Affects Versions: 1.0.0, 1.2.0, 1.1.0 Reporter: Ashutosh Chauhan Assignee: Ashutosh Chauhan Attachments: HIVE-11281.2.patch, HIVE-11281.3.patch, HIVE-11281.patch Currently it defaults to BaseTree::toStringTree() which always computes it recursively. For very large trees it becomes expensive. On one particular test query compilation time went down from 55 seconds to 2 seconds. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11263) LLAP: TaskExecutorService state is not cleaned up
[
https://issues.apache.org/jira/browse/HIVE-11263?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14630515#comment-14630515
]
Prasanth Jayachandran commented on HIVE-11263:
--
YOu can move the offer code to top of the schedule method. If the returned
object is same as the one offered we can immediately throw
RejectedExecutionException else add it to knownTasks and do the usual stuffs.
This can remove addedToQueue bool as well
LLAP: TaskExecutorService state is not cleaned up
-
Key: HIVE-11263
URL: https://issues.apache.org/jira/browse/HIVE-11263
Project: Hive
Issue Type: Sub-task
Reporter: Sergey Shelukhin
Assignee: Siddharth Seth
Attachments: HIVE-11263.1.txt
See TaskExecutorService::getExecutorsStatus, this is used to report on
queue/etc. status in JMX.
Currently, it reports 100s of bogus tasks in queue:
{noformat}
ExecutorsStatus : [ attempt_1435700346116_1888_1_04_000205_22
(sershe_20150714174105_0d013941-1f0e-4f74-9387-a2f29279a185:3/Map 1, in
queue), attempt_1435700346116_1889_1_05_000101_1
(sershe_20150714174104_b0b9f300-667e-4370-bb01-f9cb7da331e0:4/Map 1, in
queue), attempt_1435700346116_1889_1_05_000191_3
(sershe_20150714174104_b0b9f300-667e-4370-bb01-f9cb7da331e0:4/Map 1, in
queue), attempt_1435700346116_1887_7_00_000202_3
(sershe_20150714174737_bea682d1-fa0f-4281-a1cb-439d85bb2016:22/Map 5, in
queue), attempt_1435700346116_1886_1_04_73_15
(sershe_20150714174108_f9483d76-8fd9-4f82-96ee-17231b6f9b2c:1/Reducer 2, in
queue), attempt_1435700346116_1887_8_04_000166_15
(sershe_20150714174900_710d7d69-3d66-45e9-865b-cd0f87bb0d98:27/Map 1, in
queue), attempt_1435700346116_1888_11_05_000140_3
(sershe_20150714174903_50359459-5342-4d1b-852c-622a3fa92a27:28/Map 3, in
queue), attempt_1435700346116_1886_1_04_42_29
(sershe_20150714174108_f9483d76-8fd9-4f82-96ee-17231b6f9b2c:1/Reducer 2, in
queue), attempt_1435700346116_1888_2_03_000169_12
(sershe_20150714174310_97ce1d4b-8029-4ef6-a823-46e29f09718a:5/Map 1, in
queue), attempt_1435700346116_1887_1_04_000197_18
(sershe_20150714174107_8fcfe954-4eeb-46e5-bad5-42a47327b26c:2/Map 1, in
queue), attempt_1435700346116_1887_1_04_000218_21
(sershe_20150714174107_8fcfe954-4eeb-46e5-bad5-42a47327b26c:2/Map 1, in
queue), attempt_1435700346116_1886_7_09_84_1
(sershe_20150714174841_462b9bdb-c017-47c2-9fa7-7edfbfc09e60:24/Map 1, in
queue), attempt_1435700346116_1887_5_04_78_0
(sershe_20150714174509_9a5cd476-b3c8-4679-af8e-1188922713a2:14/Map 3, in
queue), attempt_1435700346116_1887_7_04_000162_6
(sershe_20150714174737_bea682d1-fa0f-4281-a1cb-439d85bb2016:22/Map 3, in
queue), attempt_1435700346116_1887_7_04_000180_0
(sershe_20150714174737_bea682d1-fa0f-4281-a1cb-439d85bb2016:22/Map 3, in
queue), attempt_1435700346116_1886_3_04_000144_0
(sershe_20150714174435_fe3077dd-a97f-4582-995b-5f723170b02f:12/Reducer 2, in
queue), attempt_1435700346116_1887_5_00_000153_1
(sershe_20150714174509_9a5cd476-b3c8-4679-af8e-1188922713a2:14/Map 5, in
queue), attempt_1435700346116_1887_7_04_000141_7
(sershe_20150714174737_bea682d1-fa0f-4281-a1cb-439d85bb2016:22/Map 3, in
queue), attempt_1435700346116_1887_1_04_24_7
(sershe_20150714174107_8fcfe954-4eeb-46e5-bad5-42a47327b26c:2/Map 1, in
queue), attempt_1435700346116_1887_5_04_000130_1
(sershe_20150714174509_9a5cd476-b3c8-4679-af8e-1188922713a2:14/Map 3, in
queue), attempt_1435700346116_1888_1_04_000200_1
(sershe_20150714174105_0d013941-1f0e-4f74-9387-a2f29279a185:3/Map 1, in
queue), attempt_1435700346116_1886_15_04_000180_0
(sershe_20150714175411_bda950b7-8aa5-417f-84f6-dd646247dca8:43/Map 1, in
queue), attempt_1435700346116_1887_7_00_000205_1
(sershe_20150714174737_bea682d1-fa0f-4281-a1cb-439d85bb2016:22/Map 5, in
queue), attempt_1435700346116_1888_4_04_000183_4
(sershe_20150714174407_f0924540-f69f-45c2-831a-9d2d1f66a124:10/Map 1, in
queue), attempt_1435700346116_1887_1_04_81_6
(sershe_20150714174107_8fcfe954-4eeb-46e5-bad5-42a47327b26c:2/Map 1, in
queue), attempt_1435700346116_1888_1_04_80_4
(sershe_20150714174105_0d013941-1f0e-4f74-9387-a2f29279a185:3/Map 1, in
queue), attempt_1435700346116_1887_7_04_05_3
(sershe_20150714174737_bea682d1-fa0f-4281-a1cb-439d85bb2016:22/Map 3, in
queue), attempt_1435700346116_1887_7_00_000169_2
(sershe_20150714174737_bea682d1-fa0f-4281-a1cb-439d85bb2016:22/Map 5, in
queue), attempt_1435700346116_1888_8_04_37_2
(sershe_20150714174731_261f2d52-8c47-4db6-8f17-8098efe144a2:20/Reducer 3, in
queue), attempt_1435700346116_1887_9_00_96_6
(sershe_20150714175015_cc1b6647-8479-4c5f-918c-00935bff7232:30/Map 5, in
queue), attempt_1435700346116_1888_11_01_01_2
[jira] [Commented] (HIVE-11281) Memoize toStringTree() for ASTNode
[ https://issues.apache.org/jira/browse/HIVE-11281?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14630531#comment-14630531 ] Ashutosh Chauhan commented on HIVE-11281: - Thats a valid concern. But AFAIK we dont mutate our ASTNode in a way which will impact this. Do you know any place where we do that? Memoize toStringTree() for ASTNode -- Key: HIVE-11281 URL: https://issues.apache.org/jira/browse/HIVE-11281 Project: Hive Issue Type: Bug Components: Query Planning Affects Versions: 1.0.0, 1.2.0, 1.1.0 Reporter: Ashutosh Chauhan Assignee: Ashutosh Chauhan Attachments: HIVE-11281.2.patch, HIVE-11281.3.patch, HIVE-11281.patch Currently it defaults to BaseTree::toStringTree() which always computes it recursively. For very large trees it becomes expensive. On one particular test query compilation time went down from 55 seconds to 2 seconds. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-11263) LLAP: TaskExecutorService state is not cleaned up
[
https://issues.apache.org/jira/browse/HIVE-11263?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siddharth Seth updated HIVE-11263:
--
Attachment: HIVE-11263.2.txt
Updated patch to address feedback.
LLAP: TaskExecutorService state is not cleaned up
-
Key: HIVE-11263
URL: https://issues.apache.org/jira/browse/HIVE-11263
Project: Hive
Issue Type: Sub-task
Reporter: Sergey Shelukhin
Assignee: Siddharth Seth
Attachments: HIVE-11263.1.txt, HIVE-11263.2.txt
See TaskExecutorService::getExecutorsStatus, this is used to report on
queue/etc. status in JMX.
Currently, it reports 100s of bogus tasks in queue:
{noformat}
ExecutorsStatus : [ attempt_1435700346116_1888_1_04_000205_22
(sershe_20150714174105_0d013941-1f0e-4f74-9387-a2f29279a185:3/Map 1, in
queue), attempt_1435700346116_1889_1_05_000101_1
(sershe_20150714174104_b0b9f300-667e-4370-bb01-f9cb7da331e0:4/Map 1, in
queue), attempt_1435700346116_1889_1_05_000191_3
(sershe_20150714174104_b0b9f300-667e-4370-bb01-f9cb7da331e0:4/Map 1, in
queue), attempt_1435700346116_1887_7_00_000202_3
(sershe_20150714174737_bea682d1-fa0f-4281-a1cb-439d85bb2016:22/Map 5, in
queue), attempt_1435700346116_1886_1_04_73_15
(sershe_20150714174108_f9483d76-8fd9-4f82-96ee-17231b6f9b2c:1/Reducer 2, in
queue), attempt_1435700346116_1887_8_04_000166_15
(sershe_20150714174900_710d7d69-3d66-45e9-865b-cd0f87bb0d98:27/Map 1, in
queue), attempt_1435700346116_1888_11_05_000140_3
(sershe_20150714174903_50359459-5342-4d1b-852c-622a3fa92a27:28/Map 3, in
queue), attempt_1435700346116_1886_1_04_42_29
(sershe_20150714174108_f9483d76-8fd9-4f82-96ee-17231b6f9b2c:1/Reducer 2, in
queue), attempt_1435700346116_1888_2_03_000169_12
(sershe_20150714174310_97ce1d4b-8029-4ef6-a823-46e29f09718a:5/Map 1, in
queue), attempt_1435700346116_1887_1_04_000197_18
(sershe_20150714174107_8fcfe954-4eeb-46e5-bad5-42a47327b26c:2/Map 1, in
queue), attempt_1435700346116_1887_1_04_000218_21
(sershe_20150714174107_8fcfe954-4eeb-46e5-bad5-42a47327b26c:2/Map 1, in
queue), attempt_1435700346116_1886_7_09_84_1
(sershe_20150714174841_462b9bdb-c017-47c2-9fa7-7edfbfc09e60:24/Map 1, in
queue), attempt_1435700346116_1887_5_04_78_0
(sershe_20150714174509_9a5cd476-b3c8-4679-af8e-1188922713a2:14/Map 3, in
queue), attempt_1435700346116_1887_7_04_000162_6
(sershe_20150714174737_bea682d1-fa0f-4281-a1cb-439d85bb2016:22/Map 3, in
queue), attempt_1435700346116_1887_7_04_000180_0
(sershe_20150714174737_bea682d1-fa0f-4281-a1cb-439d85bb2016:22/Map 3, in
queue), attempt_1435700346116_1886_3_04_000144_0
(sershe_20150714174435_fe3077dd-a97f-4582-995b-5f723170b02f:12/Reducer 2, in
queue), attempt_1435700346116_1887_5_00_000153_1
(sershe_20150714174509_9a5cd476-b3c8-4679-af8e-1188922713a2:14/Map 5, in
queue), attempt_1435700346116_1887_7_04_000141_7
(sershe_20150714174737_bea682d1-fa0f-4281-a1cb-439d85bb2016:22/Map 3, in
queue), attempt_1435700346116_1887_1_04_24_7
(sershe_20150714174107_8fcfe954-4eeb-46e5-bad5-42a47327b26c:2/Map 1, in
queue), attempt_1435700346116_1887_5_04_000130_1
(sershe_20150714174509_9a5cd476-b3c8-4679-af8e-1188922713a2:14/Map 3, in
queue), attempt_1435700346116_1888_1_04_000200_1
(sershe_20150714174105_0d013941-1f0e-4f74-9387-a2f29279a185:3/Map 1, in
queue), attempt_1435700346116_1886_15_04_000180_0
(sershe_20150714175411_bda950b7-8aa5-417f-84f6-dd646247dca8:43/Map 1, in
queue), attempt_1435700346116_1887_7_00_000205_1
(sershe_20150714174737_bea682d1-fa0f-4281-a1cb-439d85bb2016:22/Map 5, in
queue), attempt_1435700346116_1888_4_04_000183_4
(sershe_20150714174407_f0924540-f69f-45c2-831a-9d2d1f66a124:10/Map 1, in
queue), attempt_1435700346116_1887_1_04_81_6
(sershe_20150714174107_8fcfe954-4eeb-46e5-bad5-42a47327b26c:2/Map 1, in
queue), attempt_1435700346116_1888_1_04_80_4
(sershe_20150714174105_0d013941-1f0e-4f74-9387-a2f29279a185:3/Map 1, in
queue), attempt_1435700346116_1887_7_04_05_3
(sershe_20150714174737_bea682d1-fa0f-4281-a1cb-439d85bb2016:22/Map 3, in
queue), attempt_1435700346116_1887_7_00_000169_2
(sershe_20150714174737_bea682d1-fa0f-4281-a1cb-439d85bb2016:22/Map 5, in
queue), attempt_1435700346116_1888_8_04_37_2
(sershe_20150714174731_261f2d52-8c47-4db6-8f17-8098efe144a2:20/Reducer 3, in
queue), attempt_1435700346116_1887_9_00_96_6
(sershe_20150714175015_cc1b6647-8479-4c5f-918c-00935bff7232:30/Map 5, in
queue), attempt_1435700346116_1888_11_01_01_2
(sershe_20150714174903_50359459-5342-4d1b-852c-622a3fa92a27:28/Map 7, in
queue), attempt_1435700346116_1889_1_05_000206_8
(sershe_20150714174104_b0b9f300-667e-4370-bb01-f9cb7da331e0:4/Map 1, in
queue), attempt_1435700346116_1887_5_04_54_0
[jira] [Commented] (HIVE-11281) Memoize toStringTree() for ASTNode
[
https://issues.apache.org/jira/browse/HIVE-11281?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14630566#comment-14630566
]
Hive QA commented on HIVE-11281:
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12745702/HIVE-11281.3.patch
{color:red}ERROR:{color} -1 due to 3 failed/errored test(s), 9197 tests executed
*Failed tests:*
{noformat}
TestParseNegative - did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cbo_rp_join0
org.apache.hive.hcatalog.streaming.TestStreaming.testTransactionBatchEmptyCommit
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/4633/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/4633/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-4633/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 3 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12745702 - PreCommit-HIVE-TRUNK-Build
Memoize toStringTree() for ASTNode
--
Key: HIVE-11281
URL: https://issues.apache.org/jira/browse/HIVE-11281
Project: Hive
Issue Type: Bug
Components: Query Planning
Affects Versions: 1.0.0, 1.2.0, 1.1.0
Reporter: Ashutosh Chauhan
Assignee: Ashutosh Chauhan
Attachments: HIVE-11281.2.patch, HIVE-11281.3.patch, HIVE-11281.patch
Currently it defaults to BaseTree::toStringTree() which always computes it
recursively. For very large trees it becomes expensive. On one particular
test query compilation time went down from 55 seconds to 2 seconds.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-11281) Memoize toStringTree() for ASTNode
[ https://issues.apache.org/jira/browse/HIVE-11281?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14630640#comment-14630640 ] Eugene Koifman commented on HIVE-11281: --- perhaps another overloaded toStringTree(boolean refreshCache) can be added to be used in whichever context this is important and leave the old method as it was. Is that workable? Memoize toStringTree() for ASTNode -- Key: HIVE-11281 URL: https://issues.apache.org/jira/browse/HIVE-11281 Project: Hive Issue Type: Bug Components: Query Planning Affects Versions: 1.0.0, 1.2.0, 1.1.0 Reporter: Ashutosh Chauhan Assignee: Ashutosh Chauhan Attachments: HIVE-11281.2.patch, HIVE-11281.3.patch, HIVE-11281.patch Currently it defaults to BaseTree::toStringTree() which always computes it recursively. For very large trees it becomes expensive. On one particular test query compilation time went down from 55 seconds to 2 seconds. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-10632) Make sure TXN_COMPONENTS gets cleaned up if table is dropped before compaction.
[ https://issues.apache.org/jira/browse/HIVE-10632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14630443#comment-14630443 ] Kristopher Kane commented on HIVE-10632: In Hive .14 I am seeing this with the dropped table scenario. All existing transactions are of state 'aborted' and 'select txn_id from TXNS where txn_id not in (select tc_txnid from TXN_COMPONENTS) and txn_state = ‘a';' returns 0 rows since both tables contain the same IDs. Recreating the table had no effect on trigger a clean up process. Is the only way to fix this to remove the entries from both transaction tables in the RDBMS? Make sure TXN_COMPONENTS gets cleaned up if table is dropped before compaction. --- Key: HIVE-10632 URL: https://issues.apache.org/jira/browse/HIVE-10632 Project: Hive Issue Type: Bug Components: Metastore, Transactions Affects Versions: 1.0.0 Reporter: Eugene Koifman Assignee: Alan Gates The compaction process will clean up entries in TXNS, COMPLETED_TXN_COMPONENTS, TXN_COMPONENTS. If the table/partition is dropped before compaction is complete there will be data left in these tables. Need to investigate if there are other situations where this may happen and address it. see HIVE-10595 for additional info -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (HIVE-11284) Fix cbo_rp_join0 failure on master
[ https://issues.apache.org/jira/browse/HIVE-11284?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez reassigned HIVE-11284: -- Assignee: Jesus Camacho Rodriguez Fix cbo_rp_join0 failure on master -- Key: HIVE-11284 URL: https://issues.apache.org/jira/browse/HIVE-11284 Project: Hive Issue Type: Bug Components: CBO, Query Planning Affects Versions: 2.0.0 Reporter: Ashutosh Chauhan Assignee: Jesus Camacho Rodriguez It first failed in this build : http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/4621/ -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11281) Memoize toStringTree() for ASTNode
[ https://issues.apache.org/jira/browse/HIVE-11281?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14630402#comment-14630402 ] Hari Sankar Sivarama Subramaniyan commented on HIVE-11281: -- +1 Memoize toStringTree() for ASTNode -- Key: HIVE-11281 URL: https://issues.apache.org/jira/browse/HIVE-11281 Project: Hive Issue Type: Bug Components: Query Planning Affects Versions: 1.0.0, 1.2.0, 1.1.0 Reporter: Ashutosh Chauhan Assignee: Ashutosh Chauhan Attachments: HIVE-11281.2.patch, HIVE-11281.3.patch, HIVE-11281.patch Currently it defaults to BaseTree::toStringTree() which always computes it recursively. For very large trees it becomes expensive. On one particular test query compilation time went down from 55 seconds to 2 seconds. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11279) Hive should emit lineage information in json compact format
[
https://issues.apache.org/jira/browse/HIVE-11279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14630448#comment-14630448
]
Hive QA commented on HIVE-11279:
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12745657/HIVE-11279.2.patch
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 9219 tests executed
*Failed tests:*
{noformat}
TestCustomAuthentication - did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cbo_rp_join0
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/4632/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/4632/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-4632/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12745657 - PreCommit-HIVE-TRUNK-Build
Hive should emit lineage information in json compact format
---
Key: HIVE-11279
URL: https://issues.apache.org/jira/browse/HIVE-11279
Project: Hive
Issue Type: Bug
Components: Logging
Affects Versions: 1.3.0
Reporter: Lenni Kuff
Assignee: Lenni Kuff
Attachments: HIVE-11279.1.patch, HIVE-11279.2.patch
Hive should emit lineage information in json compact format. Currently, Hive
prints this in human readable format which makes it harder to consume
(identify record boundaries) and makes the output files very long.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-11227) Kryo exception during table creation in Hive
[ https://issues.apache.org/jira/browse/HIVE-11227?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14630601#comment-14630601 ] Xuefu Zhang commented on HIVE-11227: From the log it seems that Kryo is serializing PerfLogger.starttimes field originated from MapJoinOperator. While I don't know why it complains that the type (HashMap) is not registered, there is no reason to serialize PerfLogger itself. This is certainly be done, but the repro case is too complicated to verify the fix. Could you try to one of the two things? 1. use latest Hive with your query to see if you can reproduce. If so, I can provide a patch to see if it addresses the problem. 2. provide a simpler repro case where I can try and verify the fix. Thanks. Kryo exception during table creation in Hive Key: HIVE-11227 URL: https://issues.apache.org/jira/browse/HIVE-11227 Project: Hive Issue Type: Bug Components: Configuration, Database/Schema, Hive, HiveServer2, Indexing, Locking Affects Versions: 0.13.1 Environment: CentOS 6.5, jdk 1.7, cpu: 2x1.9 GHz 6-core Xeon (24 cores), Ram: 64GB-128GB Reporter: Akamai Fix For: 0.14.1 Attachments: Kryo Exception.txt, init_load_hdpextract_user.tlog.clean.log, tlog_detail.20150710.log.clean, trsm_tlog_detail.20150714.log.clean Exception is getting thorwn during table creation in Hive Error: java.lang.RuntimeException: org.apache.hive.com/esotericsoftware.kryo.KryoException: Encountered unregistered class ID: 380 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11281) Memoize toStringTree() for ASTNode
[ https://issues.apache.org/jira/browse/HIVE-11281?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14630599#comment-14630599 ] Eugene Koifman commented on HIVE-11281: --- Imagine someone implemented HashMap.toString() to cache the value the first time it's called and documented this method about this gotcha to warn future devs about this. Memoize toStringTree() for ASTNode -- Key: HIVE-11281 URL: https://issues.apache.org/jira/browse/HIVE-11281 Project: Hive Issue Type: Bug Components: Query Planning Affects Versions: 1.0.0, 1.2.0, 1.1.0 Reporter: Ashutosh Chauhan Assignee: Ashutosh Chauhan Attachments: HIVE-11281.2.patch, HIVE-11281.3.patch, HIVE-11281.patch Currently it defaults to BaseTree::toStringTree() which always computes it recursively. For very large trees it becomes expensive. On one particular test query compilation time went down from 55 seconds to 2 seconds. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11285) ObjectInspector for partition columns in FetchOperator in SMBJoin causes exception
[
https://issues.apache.org/jira/browse/HIVE-11285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14630647#comment-14630647
]
Hive QA commented on HIVE-11285:
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12745694/HIVE-11285.01.patch
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 9226 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cbo_rp_join0
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/4634/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/4634/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-4634/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12745694 - PreCommit-HIVE-TRUNK-Build
ObjectInspector for partition columns in FetchOperator in SMBJoin causes
exception
--
Key: HIVE-11285
URL: https://issues.apache.org/jira/browse/HIVE-11285
Project: Hive
Issue Type: Bug
Reporter: Pengcheng Xiong
Assignee: Pengcheng Xiong
Attachments: HIVE-11285.01.patch
{code}
STEPS TO REPRODUCE:
*$ cat data.out
1|One
2|Two
hql
CREATE TABLE data_table (key INT, value STRING) ROW FORMAT DELIMITED FIELDS
TERMINATED BY '|';
LOAD DATA LOCAL INPATH '$
{system:user.dir}
/data.out' INTO TABLE data_table;
CREATE TABLE smb_table (key INT, value STRING)
CLUSTERED BY (key)
SORTED BY (key) INTO 1 BUCKETS
STORED AS ORC;
CREATE TABLE smb_table_part (key INT, value STRING)
PARTITIONED BY (p1 DECIMAL)
CLUSTERED BY (key)
SORTED BY (key) INTO 1 BUCKETS
STORED AS ORC;
INSERT OVERWRITE TABLE smb_table SELECT * FROM data_table;
INSERT OVERWRITE TABLE smb_table_part PARTITION (p1) SELECT key, value, 100
as p1 FROM data_table;
SET hive.execution.engine=mr;
SET hive.enforce.sortmergebucketmapjoin=false;
SET hive.auto.convert.sortmerge.join=true;
SET hive.optimize.bucketmapjoin = true;
SET hive.optimize.bucketmapjoin.sortedmerge = true;
SET hive.input.format =
org.apache.hadoop.hive.ql.io.BucketizedHiveInputFormat;
SELECT s1.key, s2.p1
FROM smb_table s1
INNER JOIN smb_table_part s2
ON s1.key = s2.key
ORDER BY s1.key;
ERROR:
2015-07-15 13:39:04,333 WARN main org.apache.hadoop.mapred.YarnChild:
Exception running child : java.lang.RuntimeException:
org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while
processing row
{key:1,value:One}
at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:185)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:450)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime
Error while processing row
{key:1,value:One}
at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:503)
at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:176)
... 8 more
Caused by: java.lang.RuntimeException: Map local work failed
at
org.apache.hadoop.hive.ql.exec.SMBMapJoinOperator.fetchOneRow(SMBMapJoinOperator.java:569)
at
org.apache.hadoop.hive.ql.exec.SMBMapJoinOperator.fetchNextGroup(SMBMapJoinOperator.java:429)
at
org.apache.hadoop.hive.ql.exec.SMBMapJoinOperator.processOp(SMBMapJoinOperator.java:260)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:815)
at
org.apache.hadoop.hive.ql.exec.FilterOperator.processOp(FilterOperator.java:120)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:815)
at
org.apache.hadoop.hive.ql.exec.TableScanOperator.processOp(TableScanOperator.java:95)
at
org.apache.hadoop.hive.ql.exec.MapOperator$MapOpCtx.forward(MapOperator.java:157)
at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:493)
... 9 more
Caused by: java.lang.ClassCastException: org.apache.hadoop.io.IntWritable
cannot be cast to
