[jira] [Commented] (HIVE-16550) Semijoin Hints should be able to skip the optimization if needed.
[ https://issues.apache.org/jira/browse/HIVE-16550?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15992365#comment-15992365 ] Deepak Jaiswal commented on HIVE-16550: --- [~jdere][~sershe] Can you please review? https://reviews.apache.org/r/58914/ > Semijoin Hints should be able to skip the optimization if needed. > - > > Key: HIVE-16550 > URL: https://issues.apache.org/jira/browse/HIVE-16550 > Project: Hive > Issue Type: Improvement >Reporter: Deepak Jaiswal >Assignee: Deepak Jaiswal > Attachments: HIVE-16550.1.patch > > > Currently semi join hints are designed to enforce a particular semi join, > however, it should also be able to skip the optimization all together in a > query using hints. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-11133) Support hive.explain.user for Spark
[
https://issues.apache.org/jira/browse/HIVE-11133?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sahil Takiar updated HIVE-11133:
Attachment: HIVE-11133.10.patch
Hey [~xuefuz], sorry, not sure why it isn't working, but I attached a new patch
that is generated in a slightly different way. Running {{patch -p1 <
patch-file}} should work with it. I tested it and it successfully builds.
> Support hive.explain.user for Spark
> ---
>
> Key: HIVE-11133
> URL: https://issues.apache.org/jira/browse/HIVE-11133
> Project: Hive
> Issue Type: Sub-task
> Components: Spark
>Reporter: Mohit Sabharwal
>Assignee: Sahil Takiar
> Labels: TODOC3.0
> Fix For: 3.0.0
>
> Attachments: HIVE-11133.10.patch, HIVE-11133.1.patch,
> HIVE-11133.2.patch, HIVE-11133.3.patch, HIVE-11133.4.patch,
> HIVE-11133.5.patch, HIVE-11133.6.patch, HIVE-11133.7.patch,
> HIVE-11133.8.patch, HIVE-11133.9.patch
>
>
> User friendly explain output ({{set hive.explain.user=true}}) should support
> Spark as well.
> Once supported, we should also enable related q-tests like {{explainuser_1.q}}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-11133) Support hive.explain.user for Spark
[
https://issues.apache.org/jira/browse/HIVE-11133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15992341#comment-15992341
]
Xuefu Zhang commented on HIVE-11133:
It didn't work. Please play it and make sure the patch works for you and then
provide instructions. Thanks.
> Support hive.explain.user for Spark
> ---
>
> Key: HIVE-11133
> URL: https://issues.apache.org/jira/browse/HIVE-11133
> Project: Hive
> Issue Type: Sub-task
> Components: Spark
>Reporter: Mohit Sabharwal
>Assignee: Sahil Takiar
> Labels: TODOC3.0
> Fix For: 3.0.0
>
> Attachments: HIVE-11133.1.patch, HIVE-11133.2.patch,
> HIVE-11133.3.patch, HIVE-11133.4.patch, HIVE-11133.5.patch,
> HIVE-11133.6.patch, HIVE-11133.7.patch, HIVE-11133.8.patch, HIVE-11133.9.patch
>
>
> User friendly explain output ({{set hive.explain.user=true}}) should support
> Spark as well.
> Once supported, we should also enable related q-tests like {{explainuser_1.q}}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-16552) Limit the number of tasks a Spark job may contain
[ https://issues.apache.org/jira/browse/HIVE-16552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15992334#comment-15992334 ] Xuefu Zhang commented on HIVE-16552: [~lirui], while you have a point, but the resource usage doesn't necessarily limit to the number of concurrent tasks. For instance, if a query scanning lots of partitions can create a spike in NN calls at compile time. On the other hand, the large number of tasks usually means more total resource consumption, which is also important a resource queue is shared in a team. I can certainly understand this is rather a poor man's choice when it's desirable to block a large or bad query. The debate can also go on for the similar configurations for MR. I'm open to better ideas if there are any. Plus, for those who don't care or need this, the default value would just work as if the configuration didn't exist. > Limit the number of tasks a Spark job may contain > - > > Key: HIVE-16552 > URL: https://issues.apache.org/jira/browse/HIVE-16552 > Project: Hive > Issue Type: Improvement > Components: Spark >Affects Versions: 1.0.0, 2.0.0 >Reporter: Xuefu Zhang >Assignee: Xuefu Zhang > Attachments: HIVE-16552.1.patch, HIVE-16552.patch > > > It's commonly desirable to block bad and big queries that takes a lot of YARN > resources. One approach, similar to mapreduce.job.max.map in MapReduce, is to > stop a query that invokes a Spark job that contains too many tasks. The > proposal here is to introduce hive.spark.job.max.tasks with a default value > of -1 (no limit), which an admin can set to block queries that trigger too > many spark tasks. > Please note that this control knob applies to a spark job, though it's > possible that one query can trigger multiple Spark jobs (such as in case of > map-join). Nevertheless, the proposed approach is still helpful. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16550) Semijoin Hints should be able to skip the optimization if needed.
[ https://issues.apache.org/jira/browse/HIVE-16550?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Deepak Jaiswal updated HIVE-16550: -- Attachment: HIVE-16550.1.patch Initial patch. > Semijoin Hints should be able to skip the optimization if needed. > - > > Key: HIVE-16550 > URL: https://issues.apache.org/jira/browse/HIVE-16550 > Project: Hive > Issue Type: Improvement >Reporter: Deepak Jaiswal >Assignee: Deepak Jaiswal > Attachments: HIVE-16550.1.patch > > > Currently semi join hints are designed to enforce a particular semi join, > however, it should also be able to skip the optimization all together in a > query using hints. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16550) Semijoin Hints should be able to skip the optimization if needed.
[ https://issues.apache.org/jira/browse/HIVE-16550?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Deepak Jaiswal updated HIVE-16550: -- Status: Patch Available (was: In Progress) > Semijoin Hints should be able to skip the optimization if needed. > - > > Key: HIVE-16550 > URL: https://issues.apache.org/jira/browse/HIVE-16550 > Project: Hive > Issue Type: Improvement >Reporter: Deepak Jaiswal >Assignee: Deepak Jaiswal > > Currently semi join hints are designed to enforce a particular semi join, > however, it should also be able to skip the optimization all together in a > query using hints. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16566) Set column stats default as true when creating new tables/partitions
[ https://issues.apache.org/jira/browse/HIVE-16566?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong updated HIVE-16566: --- Status: Patch Available (was: Open) > Set column stats default as true when creating new tables/partitions > > > Key: HIVE-16566 > URL: https://issues.apache.org/jira/browse/HIVE-16566 > Project: Hive > Issue Type: Sub-task >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong > Attachments: HIVE-16566.01.patch, HIVE-16566.02.patch > > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16566) Set column stats default as true when creating new tables/partitions
[ https://issues.apache.org/jira/browse/HIVE-16566?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong updated HIVE-16566: --- Attachment: HIVE-16566.02.patch > Set column stats default as true when creating new tables/partitions > > > Key: HIVE-16566 > URL: https://issues.apache.org/jira/browse/HIVE-16566 > Project: Hive > Issue Type: Sub-task >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong > Attachments: HIVE-16566.01.patch, HIVE-16566.02.patch > > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16566) Set column stats default as true when creating new tables/partitions
[ https://issues.apache.org/jira/browse/HIVE-16566?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong updated HIVE-16566: --- Status: Open (was: Patch Available) > Set column stats default as true when creating new tables/partitions > > > Key: HIVE-16566 > URL: https://issues.apache.org/jira/browse/HIVE-16566 > Project: Hive > Issue Type: Sub-task >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong > Attachments: HIVE-16566.01.patch, HIVE-16566.02.patch > > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-11133) Support hive.explain.user for Spark
[
https://issues.apache.org/jira/browse/HIVE-11133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15992320#comment-15992320
]
Sahil Takiar commented on HIVE-11133:
-
Hey [~xuefuz] sorry about that, {{git apply [path-to-patch-file]}} should work
> Support hive.explain.user for Spark
> ---
>
> Key: HIVE-11133
> URL: https://issues.apache.org/jira/browse/HIVE-11133
> Project: Hive
> Issue Type: Sub-task
> Components: Spark
>Reporter: Mohit Sabharwal
>Assignee: Sahil Takiar
> Labels: TODOC3.0
> Fix For: 3.0.0
>
> Attachments: HIVE-11133.1.patch, HIVE-11133.2.patch,
> HIVE-11133.3.patch, HIVE-11133.4.patch, HIVE-11133.5.patch,
> HIVE-11133.6.patch, HIVE-11133.7.patch, HIVE-11133.8.patch, HIVE-11133.9.patch
>
>
> User friendly explain output ({{set hive.explain.user=true}}) should support
> Spark as well.
> Once supported, we should also enable related q-tests like {{explainuser_1.q}}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HIVE-16488) Support replicating into existing db if the db is empty
[ https://issues.apache.org/jira/browse/HIVE-16488?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sushanth Sowmyan updated HIVE-16488: Resolution: Fixed Fix Version/s: 3.0.0 Status: Resolved (was: Patch Available) > Support replicating into existing db if the db is empty > --- > > Key: HIVE-16488 > URL: https://issues.apache.org/jira/browse/HIVE-16488 > Project: Hive > Issue Type: Sub-task > Components: repl >Affects Versions: 2.1.0 >Reporter: Sankar Hariappan >Assignee: Sankar Hariappan > Labels: DR, Replication > Fix For: 3.0.0 > > Attachments: HIVE-16488.01.patch, HIVE-16488.02.patch > > > This is a potential usecase where a user may want to manually create a db on > destination to make sure it goes to a certain dir root, or they may have > cases where the db (default, for instance) was automatically created. We > should still allow replicating into this without failing if the db is empty. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16488) Support replicating into existing db if the db is empty
[ https://issues.apache.org/jira/browse/HIVE-16488?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15992318#comment-15992318 ] Sushanth Sowmyan commented on HIVE-16488: - +1, LGTM, and the test failures are not related. Thanks, [~sankarh]! Committing to master. > Support replicating into existing db if the db is empty > --- > > Key: HIVE-16488 > URL: https://issues.apache.org/jira/browse/HIVE-16488 > Project: Hive > Issue Type: Sub-task > Components: repl >Affects Versions: 2.1.0 >Reporter: Sankar Hariappan >Assignee: Sankar Hariappan > Labels: DR, Replication > Attachments: HIVE-16488.01.patch, HIVE-16488.02.patch > > > This is a potential usecase where a user may want to manually create a db on > destination to make sure it goes to a certain dir root, or they may have > cases where the db (default, for instance) was automatically created. We > should still allow replicating into this without failing if the db is empty. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16552) Limit the number of tasks a Spark job may contain
[ https://issues.apache.org/jira/browse/HIVE-16552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15992308#comment-15992308 ] Rui Li commented on HIVE-16552: --- [~xuefuz], my point is a large number of tasks doesn't mean a lot of resources. For example, a user can request only 1 container with 1 slot, and submit a job containing 1000 tasks. At any point there'll be no more than 1 task running simultaneously. On the other hand, a user can also request lots of containers but only run small jobs with them. The latter user of course takes up more resources and has bigger impact on other users. > Limit the number of tasks a Spark job may contain > - > > Key: HIVE-16552 > URL: https://issues.apache.org/jira/browse/HIVE-16552 > Project: Hive > Issue Type: Improvement > Components: Spark >Affects Versions: 1.0.0, 2.0.0 >Reporter: Xuefu Zhang >Assignee: Xuefu Zhang > Attachments: HIVE-16552.1.patch, HIVE-16552.patch > > > It's commonly desirable to block bad and big queries that takes a lot of YARN > resources. One approach, similar to mapreduce.job.max.map in MapReduce, is to > stop a query that invokes a Spark job that contains too many tasks. The > proposal here is to introduce hive.spark.job.max.tasks with a default value > of -1 (no limit), which an admin can set to block queries that trigger too > many spark tasks. > Please note that this control knob applies to a spark job, though it's > possible that one query can trigger multiple Spark jobs (such as in case of > map-join). Nevertheless, the proposed approach is still helpful. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-11133) Support hive.explain.user for Spark
[
https://issues.apache.org/jira/browse/HIVE-11133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15992303#comment-15992303
]
Xuefu Zhang commented on HIVE-11133:
Sorry for breaking the build. I should have built locally before push it.
[~stakiar], could you please provide instructions on how to commit your patch?
I used "patch -p1 < HIVE-11133.9.patch" and it appeared not working. Thanks.
> Support hive.explain.user for Spark
> ---
>
> Key: HIVE-11133
> URL: https://issues.apache.org/jira/browse/HIVE-11133
> Project: Hive
> Issue Type: Sub-task
> Components: Spark
>Reporter: Mohit Sabharwal
>Assignee: Sahil Takiar
> Labels: TODOC3.0
> Fix For: 3.0.0
>
> Attachments: HIVE-11133.1.patch, HIVE-11133.2.patch,
> HIVE-11133.3.patch, HIVE-11133.4.patch, HIVE-11133.5.patch,
> HIVE-11133.6.patch, HIVE-11133.7.patch, HIVE-11133.8.patch, HIVE-11133.9.patch
>
>
> User friendly explain output ({{set hive.explain.user=true}}) should support
> Spark as well.
> Once supported, we should also enable related q-tests like {{explainuser_1.q}}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-15859) HoS: Write RPC messages in event loop
[
https://issues.apache.org/jira/browse/HIVE-15859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15992302#comment-15992302
]
Yi Yao commented on HIVE-15859:
---
[~lirui], thanks for your support!
> HoS: Write RPC messages in event loop
> -
>
> Key: HIVE-15859
> URL: https://issues.apache.org/jira/browse/HIVE-15859
> Project: Hive
> Issue Type: Bug
> Components: Hive, Spark
>Affects Versions: 2.1.1
> Environment: hadoop2.7.1
> spark1.6.2
> hive2.2
>Reporter: KaiXu
>Assignee: Rui Li
> Fix For: 1.3.0, 2.2.0
>
> Attachments: HIVE-15859.1.patch, HIVE-15859.2.patch,
> HIVE-15859.3.patch
>
>
> Hive on Spark, failed with error:
> {noformat}
> 2017-02-08 09:50:59,331 Stage-2_0: 1039(+2)/1041 Stage-3_0: 796(+456)/1520
> Stage-4_0: 0/2021 Stage-5_0: 0/1009 Stage-6_0: 0/1
> 2017-02-08 09:51:00,335 Stage-2_0: 1040(+1)/1041 Stage-3_0: 914(+398)/1520
> Stage-4_0: 0/2021 Stage-5_0: 0/1009 Stage-6_0: 0/1
> 2017-02-08 09:51:01,338 Stage-2_0: 1041/1041 Finished Stage-3_0:
> 961(+383)/1520 Stage-4_0: 0/2021 Stage-5_0: 0/1009 Stage-6_0: 0/1
> Failed to monitor Job[ 2] with exception 'java.lang.IllegalStateException(RPC
> channel is closed.)'
> FAILED: Execution Error, return code 1 from
> org.apache.hadoop.hive.ql.exec.spark.SparkTask
> {noformat}
> application log shows the driver commanded a shutdown with some unknown
> reason, but hive's log shows Driver could not get RPC header( Expected RPC
> header, got org.apache.hive.spark.client.rpc.Rpc$NullMessage instead).
> {noformat}
> 17/02/08 09:51:04 INFO exec.Utilities: PLAN PATH =
> hdfs://hsx-node1:8020/tmp/hive/root/b723c85d-2a7b-469e-bab1-9c165b25e656/hive_2017-02-08_09-49-37_890_6267025825539539056-1/-mr-10006/71a9dacb-a463-40ef-9e86-78d3b8e3738d/map.xml
> 17/02/08 09:51:04 INFO executor.Executor: Executor killed task 1169.0 in
> stage 3.0 (TID 2519)
> 17/02/08 09:51:04 INFO executor.CoarseGrainedExecutorBackend: Driver
> commanded a shutdown
> 17/02/08 09:51:04 INFO storage.MemoryStore: MemoryStore cleared
> 17/02/08 09:51:04 INFO storage.BlockManager: BlockManager stopped
> 17/02/08 09:51:04 INFO exec.Utilities: PLAN PATH =
> hdfs://hsx-node1:8020/tmp/hive/root/b723c85d-2a7b-469e-bab1-9c165b25e656/hive_2017-02-08_09-49-37_890_6267025825539539056-1/-mr-10006/71a9dacb-a463-40ef-9e86-78d3b8e3738d/map.xml
> 17/02/08 09:51:04 WARN executor.CoarseGrainedExecutorBackend: An unknown
> (hsx-node1:42777) driver disconnected.
> 17/02/08 09:51:04 ERROR executor.CoarseGrainedExecutorBackend: Driver
> 192.168.1.1:42777 disassociated! Shutting down.
> 17/02/08 09:51:04 INFO executor.Executor: Executor killed task 1105.0 in
> stage 3.0 (TID 2511)
> 17/02/08 09:51:04 INFO util.ShutdownHookManager: Shutdown hook called
> 17/02/08 09:51:04 INFO remote.RemoteActorRefProvider$RemotingTerminator:
> Shutting down remote daemon.
> 17/02/08 09:51:04 INFO util.ShutdownHookManager: Deleting directory
> /mnt/disk6/yarn/nm/usercache/root/appcache/application_1486453422616_0150/spark-71da1dfc-99bd-4687-bc2f-33452db8de3d
> 17/02/08 09:51:04 INFO util.ShutdownHookManager: Deleting directory
> /mnt/disk2/yarn/nm/usercache/root/appcache/application_1486453422616_0150/spark-7f134d81-e77e-4b92-bd99-0a51d0962c14
> 17/02/08 09:51:04 INFO util.ShutdownHookManager: Deleting directory
> /mnt/disk5/yarn/nm/usercache/root/appcache/application_1486453422616_0150/spark-77a90d63-fb05-4bc6-8d5e-1562cc502e6c
> 17/02/08 09:51:04 INFO remote.RemoteActorRefProvider$RemotingTerminator:
> Remote daemon shut down; proceeding with flushing remote transports.
> 17/02/08 09:51:04 INFO util.ShutdownHookManager: Deleting directory
> /mnt/disk4/yarn/nm/usercache/root/appcache/application_1486453422616_0150/spark-91f8b91a-114d-4340-8560-d3cd085c1cd4
> 17/02/08 09:51:04 INFO util.ShutdownHookManager: Deleting directory
> /mnt/disk1/yarn/nm/usercache/root/appcache/application_1486453422616_0150/spark-a3c24f9e-8609-48f0-9d37-0de7ae06682a
> 17/02/08 09:51:04 INFO remote.RemoteActorRefProvider$RemotingTerminator:
> Remoting shut down.
> 17/02/08 09:51:04 INFO util.ShutdownHookManager: Deleting directory
> /mnt/disk7/yarn/nm/usercache/root/appcache/application_1486453422616_0150/spark-f6120a43-2158-4780-927c-c5786b78f53e
> 17/02/08 09:51:04 INFO util.ShutdownHookManager: Deleting directory
> /mnt/disk3/yarn/nm/usercache/root/appcache/application_1486453422616_0150/spark-e17931ad-9e8a-45da-86f8-9a0fdca0fad1
> 17/02/08 09:51:04 INFO util.ShutdownHookManager: Deleting directory
> /mnt/disk8/yarn/nm/usercache/root/appcache/application_1486453422616_0150/spark-4de34175-f871-4c28-8ec0-d2fc0020c5c3
> 17/02/08 09:51:04 INFO executor.Executor: Executor killed task 1137.0 in
> stage 3.0 (TID 2515)
> 17/02/08 09:51:04 INFO executor.Executor: Executor killed task 897.0 in stage
> 3.0 (TID 2417)
> 17/02/08
[jira] [Updated] (HIVE-15859) HoS: Write RPC messages in event loop
[
https://issues.apache.org/jira/browse/HIVE-15859?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rui Li updated HIVE-15859:
--
Fix Version/s: 1.3.0
> HoS: Write RPC messages in event loop
> -
>
> Key: HIVE-15859
> URL: https://issues.apache.org/jira/browse/HIVE-15859
> Project: Hive
> Issue Type: Bug
> Components: Hive, Spark
>Affects Versions: 2.1.1
> Environment: hadoop2.7.1
> spark1.6.2
> hive2.2
>Reporter: KaiXu
>Assignee: Rui Li
> Fix For: 1.3.0, 2.2.0
>
> Attachments: HIVE-15859.1.patch, HIVE-15859.2.patch,
> HIVE-15859.3.patch
>
>
> Hive on Spark, failed with error:
> {noformat}
> 2017-02-08 09:50:59,331 Stage-2_0: 1039(+2)/1041 Stage-3_0: 796(+456)/1520
> Stage-4_0: 0/2021 Stage-5_0: 0/1009 Stage-6_0: 0/1
> 2017-02-08 09:51:00,335 Stage-2_0: 1040(+1)/1041 Stage-3_0: 914(+398)/1520
> Stage-4_0: 0/2021 Stage-5_0: 0/1009 Stage-6_0: 0/1
> 2017-02-08 09:51:01,338 Stage-2_0: 1041/1041 Finished Stage-3_0:
> 961(+383)/1520 Stage-4_0: 0/2021 Stage-5_0: 0/1009 Stage-6_0: 0/1
> Failed to monitor Job[ 2] with exception 'java.lang.IllegalStateException(RPC
> channel is closed.)'
> FAILED: Execution Error, return code 1 from
> org.apache.hadoop.hive.ql.exec.spark.SparkTask
> {noformat}
> application log shows the driver commanded a shutdown with some unknown
> reason, but hive's log shows Driver could not get RPC header( Expected RPC
> header, got org.apache.hive.spark.client.rpc.Rpc$NullMessage instead).
> {noformat}
> 17/02/08 09:51:04 INFO exec.Utilities: PLAN PATH =
> hdfs://hsx-node1:8020/tmp/hive/root/b723c85d-2a7b-469e-bab1-9c165b25e656/hive_2017-02-08_09-49-37_890_6267025825539539056-1/-mr-10006/71a9dacb-a463-40ef-9e86-78d3b8e3738d/map.xml
> 17/02/08 09:51:04 INFO executor.Executor: Executor killed task 1169.0 in
> stage 3.0 (TID 2519)
> 17/02/08 09:51:04 INFO executor.CoarseGrainedExecutorBackend: Driver
> commanded a shutdown
> 17/02/08 09:51:04 INFO storage.MemoryStore: MemoryStore cleared
> 17/02/08 09:51:04 INFO storage.BlockManager: BlockManager stopped
> 17/02/08 09:51:04 INFO exec.Utilities: PLAN PATH =
> hdfs://hsx-node1:8020/tmp/hive/root/b723c85d-2a7b-469e-bab1-9c165b25e656/hive_2017-02-08_09-49-37_890_6267025825539539056-1/-mr-10006/71a9dacb-a463-40ef-9e86-78d3b8e3738d/map.xml
> 17/02/08 09:51:04 WARN executor.CoarseGrainedExecutorBackend: An unknown
> (hsx-node1:42777) driver disconnected.
> 17/02/08 09:51:04 ERROR executor.CoarseGrainedExecutorBackend: Driver
> 192.168.1.1:42777 disassociated! Shutting down.
> 17/02/08 09:51:04 INFO executor.Executor: Executor killed task 1105.0 in
> stage 3.0 (TID 2511)
> 17/02/08 09:51:04 INFO util.ShutdownHookManager: Shutdown hook called
> 17/02/08 09:51:04 INFO remote.RemoteActorRefProvider$RemotingTerminator:
> Shutting down remote daemon.
> 17/02/08 09:51:04 INFO util.ShutdownHookManager: Deleting directory
> /mnt/disk6/yarn/nm/usercache/root/appcache/application_1486453422616_0150/spark-71da1dfc-99bd-4687-bc2f-33452db8de3d
> 17/02/08 09:51:04 INFO util.ShutdownHookManager: Deleting directory
> /mnt/disk2/yarn/nm/usercache/root/appcache/application_1486453422616_0150/spark-7f134d81-e77e-4b92-bd99-0a51d0962c14
> 17/02/08 09:51:04 INFO util.ShutdownHookManager: Deleting directory
> /mnt/disk5/yarn/nm/usercache/root/appcache/application_1486453422616_0150/spark-77a90d63-fb05-4bc6-8d5e-1562cc502e6c
> 17/02/08 09:51:04 INFO remote.RemoteActorRefProvider$RemotingTerminator:
> Remote daemon shut down; proceeding with flushing remote transports.
> 17/02/08 09:51:04 INFO util.ShutdownHookManager: Deleting directory
> /mnt/disk4/yarn/nm/usercache/root/appcache/application_1486453422616_0150/spark-91f8b91a-114d-4340-8560-d3cd085c1cd4
> 17/02/08 09:51:04 INFO util.ShutdownHookManager: Deleting directory
> /mnt/disk1/yarn/nm/usercache/root/appcache/application_1486453422616_0150/spark-a3c24f9e-8609-48f0-9d37-0de7ae06682a
> 17/02/08 09:51:04 INFO remote.RemoteActorRefProvider$RemotingTerminator:
> Remoting shut down.
> 17/02/08 09:51:04 INFO util.ShutdownHookManager: Deleting directory
> /mnt/disk7/yarn/nm/usercache/root/appcache/application_1486453422616_0150/spark-f6120a43-2158-4780-927c-c5786b78f53e
> 17/02/08 09:51:04 INFO util.ShutdownHookManager: Deleting directory
> /mnt/disk3/yarn/nm/usercache/root/appcache/application_1486453422616_0150/spark-e17931ad-9e8a-45da-86f8-9a0fdca0fad1
> 17/02/08 09:51:04 INFO util.ShutdownHookManager: Deleting directory
> /mnt/disk8/yarn/nm/usercache/root/appcache/application_1486453422616_0150/spark-4de34175-f871-4c28-8ec0-d2fc0020c5c3
> 17/02/08 09:51:04 INFO executor.Executor: Executor killed task 1137.0 in
> stage 3.0 (TID 2515)
> 17/02/08 09:51:04 INFO executor.Executor: Executor killed task 897.0 in stage
> 3.0 (TID 2417)
> 17/02/08 09:51:04 INFO executor.Executor: Executor killed
[jira] [Commented] (HIVE-15859) HoS: Write RPC messages in event loop
[
https://issues.apache.org/jira/browse/HIVE-15859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15992299#comment-15992299
]
Rui Li commented on HIVE-15859:
---
Pushed to branch-1. Sorry about the delay.
> HoS: Write RPC messages in event loop
> -
>
> Key: HIVE-15859
> URL: https://issues.apache.org/jira/browse/HIVE-15859
> Project: Hive
> Issue Type: Bug
> Components: Hive, Spark
>Affects Versions: 2.1.1
> Environment: hadoop2.7.1
> spark1.6.2
> hive2.2
>Reporter: KaiXu
>Assignee: Rui Li
> Fix For: 2.2.0
>
> Attachments: HIVE-15859.1.patch, HIVE-15859.2.patch,
> HIVE-15859.3.patch
>
>
> Hive on Spark, failed with error:
> {noformat}
> 2017-02-08 09:50:59,331 Stage-2_0: 1039(+2)/1041 Stage-3_0: 796(+456)/1520
> Stage-4_0: 0/2021 Stage-5_0: 0/1009 Stage-6_0: 0/1
> 2017-02-08 09:51:00,335 Stage-2_0: 1040(+1)/1041 Stage-3_0: 914(+398)/1520
> Stage-4_0: 0/2021 Stage-5_0: 0/1009 Stage-6_0: 0/1
> 2017-02-08 09:51:01,338 Stage-2_0: 1041/1041 Finished Stage-3_0:
> 961(+383)/1520 Stage-4_0: 0/2021 Stage-5_0: 0/1009 Stage-6_0: 0/1
> Failed to monitor Job[ 2] with exception 'java.lang.IllegalStateException(RPC
> channel is closed.)'
> FAILED: Execution Error, return code 1 from
> org.apache.hadoop.hive.ql.exec.spark.SparkTask
> {noformat}
> application log shows the driver commanded a shutdown with some unknown
> reason, but hive's log shows Driver could not get RPC header( Expected RPC
> header, got org.apache.hive.spark.client.rpc.Rpc$NullMessage instead).
> {noformat}
> 17/02/08 09:51:04 INFO exec.Utilities: PLAN PATH =
> hdfs://hsx-node1:8020/tmp/hive/root/b723c85d-2a7b-469e-bab1-9c165b25e656/hive_2017-02-08_09-49-37_890_6267025825539539056-1/-mr-10006/71a9dacb-a463-40ef-9e86-78d3b8e3738d/map.xml
> 17/02/08 09:51:04 INFO executor.Executor: Executor killed task 1169.0 in
> stage 3.0 (TID 2519)
> 17/02/08 09:51:04 INFO executor.CoarseGrainedExecutorBackend: Driver
> commanded a shutdown
> 17/02/08 09:51:04 INFO storage.MemoryStore: MemoryStore cleared
> 17/02/08 09:51:04 INFO storage.BlockManager: BlockManager stopped
> 17/02/08 09:51:04 INFO exec.Utilities: PLAN PATH =
> hdfs://hsx-node1:8020/tmp/hive/root/b723c85d-2a7b-469e-bab1-9c165b25e656/hive_2017-02-08_09-49-37_890_6267025825539539056-1/-mr-10006/71a9dacb-a463-40ef-9e86-78d3b8e3738d/map.xml
> 17/02/08 09:51:04 WARN executor.CoarseGrainedExecutorBackend: An unknown
> (hsx-node1:42777) driver disconnected.
> 17/02/08 09:51:04 ERROR executor.CoarseGrainedExecutorBackend: Driver
> 192.168.1.1:42777 disassociated! Shutting down.
> 17/02/08 09:51:04 INFO executor.Executor: Executor killed task 1105.0 in
> stage 3.0 (TID 2511)
> 17/02/08 09:51:04 INFO util.ShutdownHookManager: Shutdown hook called
> 17/02/08 09:51:04 INFO remote.RemoteActorRefProvider$RemotingTerminator:
> Shutting down remote daemon.
> 17/02/08 09:51:04 INFO util.ShutdownHookManager: Deleting directory
> /mnt/disk6/yarn/nm/usercache/root/appcache/application_1486453422616_0150/spark-71da1dfc-99bd-4687-bc2f-33452db8de3d
> 17/02/08 09:51:04 INFO util.ShutdownHookManager: Deleting directory
> /mnt/disk2/yarn/nm/usercache/root/appcache/application_1486453422616_0150/spark-7f134d81-e77e-4b92-bd99-0a51d0962c14
> 17/02/08 09:51:04 INFO util.ShutdownHookManager: Deleting directory
> /mnt/disk5/yarn/nm/usercache/root/appcache/application_1486453422616_0150/spark-77a90d63-fb05-4bc6-8d5e-1562cc502e6c
> 17/02/08 09:51:04 INFO remote.RemoteActorRefProvider$RemotingTerminator:
> Remote daemon shut down; proceeding with flushing remote transports.
> 17/02/08 09:51:04 INFO util.ShutdownHookManager: Deleting directory
> /mnt/disk4/yarn/nm/usercache/root/appcache/application_1486453422616_0150/spark-91f8b91a-114d-4340-8560-d3cd085c1cd4
> 17/02/08 09:51:04 INFO util.ShutdownHookManager: Deleting directory
> /mnt/disk1/yarn/nm/usercache/root/appcache/application_1486453422616_0150/spark-a3c24f9e-8609-48f0-9d37-0de7ae06682a
> 17/02/08 09:51:04 INFO remote.RemoteActorRefProvider$RemotingTerminator:
> Remoting shut down.
> 17/02/08 09:51:04 INFO util.ShutdownHookManager: Deleting directory
> /mnt/disk7/yarn/nm/usercache/root/appcache/application_1486453422616_0150/spark-f6120a43-2158-4780-927c-c5786b78f53e
> 17/02/08 09:51:04 INFO util.ShutdownHookManager: Deleting directory
> /mnt/disk3/yarn/nm/usercache/root/appcache/application_1486453422616_0150/spark-e17931ad-9e8a-45da-86f8-9a0fdca0fad1
> 17/02/08 09:51:04 INFO util.ShutdownHookManager: Deleting directory
> /mnt/disk8/yarn/nm/usercache/root/appcache/application_1486453422616_0150/spark-4de34175-f871-4c28-8ec0-d2fc0020c5c3
> 17/02/08 09:51:04 INFO executor.Executor: Executor killed task 1137.0 in
> stage 3.0 (TID 2515)
> 17/02/08 09:51:04 INFO executor.Executor: Executor killed task 897.0 in stage
> 3.0 (TID 2417)
> 17/02/08
[jira] [Commented] (HIVE-16552) Limit the number of tasks a Spark job may contain
[ https://issues.apache.org/jira/browse/HIVE-16552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15992296#comment-15992296 ] Xuefu Zhang commented on HIVE-16552: [~lirui], as mentioned in the description, the main use of this property is to block large/bad queries that taking a lot of resources, such as scanning a lot of partitions. YARN resource settings doesn't prevent users from submitting such a large query. MR has things like mapreduce.job.max.map, whereas Spark doesn't provide such options. Large/bad queries not just run longer but also creates huge load on HS2 and HDFS. This option provides an admin to control such queries. Regular users don't have to worry about this configuration. They just need to rewrite their blocked queries. It's advisable for an admin to blacklist this configuration. Also, for admins or regular users who don't have a such problem, the default value will just do for them. Make sense? > Limit the number of tasks a Spark job may contain > - > > Key: HIVE-16552 > URL: https://issues.apache.org/jira/browse/HIVE-16552 > Project: Hive > Issue Type: Improvement > Components: Spark >Affects Versions: 1.0.0, 2.0.0 >Reporter: Xuefu Zhang >Assignee: Xuefu Zhang > Attachments: HIVE-16552.1.patch, HIVE-16552.patch > > > It's commonly desirable to block bad and big queries that takes a lot of YARN > resources. One approach, similar to mapreduce.job.max.map in MapReduce, is to > stop a query that invokes a Spark job that contains too many tasks. The > proposal here is to introduce hive.spark.job.max.tasks with a default value > of -1 (no limit), which an admin can set to block queries that trigger too > many spark tasks. > Please note that this control knob applies to a spark job, though it's > possible that one query can trigger multiple Spark jobs (such as in case of > map-join). Nevertheless, the proposed approach is still helpful. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16566) Set column stats default as true when creating new tables/partitions
[
https://issues.apache.org/jira/browse/HIVE-16566?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15992293#comment-15992293
]
Hive QA commented on HIVE-16566:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12865868/HIVE-16566.01.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/4983/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/4983/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-4983/
Messages:
{noformat}
This message was trimmed, see log for full details
patching file
ql/src/test/results/clientpositive/groupby_ppr_multi_distinct.q.out
patching file ql/src/test/results/clientpositive/groupby_sort_1_23.q.out
patching file ql/src/test/results/clientpositive/groupby_sort_6.q.out
patching file ql/src/test/results/clientpositive/groupby_sort_skew_1_23.q.out
patching file ql/src/test/results/clientpositive/input_part1.q.out
patching file ql/src/test/results/clientpositive/input_part2.q.out
patching file
ql/src/test/results/clientpositive/insert_values_orig_table_use_metadata.q.out
patching file ql/src/test/results/clientpositive/join17.q.out
patching file ql/src/test/results/clientpositive/join26.q.out
patching file ql/src/test/results/clientpositive/join32.q.out
patching file ql/src/test/results/clientpositive/join33.q.out
patching file ql/src/test/results/clientpositive/join34.q.out
patching file ql/src/test/results/clientpositive/join35.q.out
patching file ql/src/test/results/clientpositive/join9.q.out
patching file ql/src/test/results/clientpositive/join_map_ppr.q.out
patching file ql/src/test/results/clientpositive/list_bucket_dml_14.q.out
patching file
ql/src/test/results/clientpositive/list_bucket_query_oneskew_1.q.out
patching file
ql/src/test/results/clientpositive/list_bucket_query_oneskew_2.q.out
patching file
ql/src/test/results/clientpositive/list_bucket_query_oneskew_3.q.out
patching file ql/src/test/results/clientpositive/llap/autoColumnStats_1.q.out
patching file ql/src/test/results/clientpositive/llap/autoColumnStats_2.q.out
patching file ql/src/test/results/clientpositive/llap/bucket2.q.out
patching file ql/src/test/results/clientpositive/llap/bucket4.q.out
patching file ql/src/test/results/clientpositive/llap/bucket5.q.out
patching file ql/src/test/results/clientpositive/llap/bucket_many.q.out
patching file ql/src/test/results/clientpositive/llap/bucketmapjoin1.q.out
patching file ql/src/test/results/clientpositive/llap/bucketmapjoin2.q.out
patching file ql/src/test/results/clientpositive/llap/bucketmapjoin3.q.out
patching file ql/src/test/results/clientpositive/llap/bucketmapjoin4.q.out
patching file
ql/src/test/results/clientpositive/llap/column_names_with_leading_and_trailing_spaces.q.out
patching file
ql/src/test/results/clientpositive/llap/disable_merge_for_bucketing.q.out
patching file
ql/src/test/results/clientpositive/llap/dynamic_semijoin_reduction.q.out
patching file
ql/src/test/results/clientpositive/llap/dynamic_semijoin_user_level.q.out
patching file
ql/src/test/results/clientpositive/llap/external_table_with_space_in_location_path.q.out
patching file ql/src/test/results/clientpositive/llap/join32_lessSize.q.out
patching file ql/src/test/results/clientpositive/llap/metadataonly1.q.out
patching file ql/src/test/results/clientpositive/llap/orc_create.q.out
patching file ql/src/test/results/clientpositive/llap/reduce_deduplicate.q.out
patching file ql/src/test/results/clientpositive/llap/sample1.q.out
patching file ql/src/test/results/clientpositive/llap/schemeAuthority.q.out
patching file ql/src/test/results/clientpositive/llap/stats11.q.out
patching file ql/src/test/results/clientpositive/llap/stats_noscan_1.q.out
patching file
ql/src/test/results/clientpositive/parquet_array_null_element.q.out
patching file ql/src/test/results/clientpositive/parquet_create.q.out
patching file ql/src/test/results/clientpositive/parquet_int96_timestamp.q.out
patching file ql/src/test/results/clientpositive/part_inherit_tbl_props.q.out
patching file
ql/src/test/results/clientpositive/part_inherit_tbl_props_empty.q.out
patching file
ql/src/test/results/clientpositive/part_inherit_tbl_props_with_star.q.out
patching file ql/src/test/results/clientpositive/partial_column_stats.q.out
patching file
ql/src/test/results/clientpositive/partition_coltype_literals.q.out
patching file ql/src/test/results/clientpositive/pcr.q.out
patching file ql/src/test/results/clientpositive/quotedid_stats.q.out
patching file ql/src/test/results/clientpositive/rand_partitionpruner2.q.out
patching file ql/src/test/results/clientpositive/rcfile_default_format.q.out
patching file ql/src/test/results/clientpositive/recursive_dir.q.out
patching file ql/src/test/results/clientpositive/sample1.q.out
patching file
[jira] [Updated] (HIVE-16566) Set column stats default as true when creating new tables/partitions
[ https://issues.apache.org/jira/browse/HIVE-16566?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong updated HIVE-16566: --- Attachment: (was: HIVE-16566.01.patch) > Set column stats default as true when creating new tables/partitions > > > Key: HIVE-16566 > URL: https://issues.apache.org/jira/browse/HIVE-16566 > Project: Hive > Issue Type: Sub-task >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong > Attachments: HIVE-16566.01.patch > > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16566) Set column stats default as true when creating new tables/partitions
[ https://issues.apache.org/jira/browse/HIVE-16566?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong updated HIVE-16566: --- Attachment: HIVE-16566.01.patch > Set column stats default as true when creating new tables/partitions > > > Key: HIVE-16566 > URL: https://issues.apache.org/jira/browse/HIVE-16566 > Project: Hive > Issue Type: Sub-task >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong > Attachments: HIVE-16566.01.patch > > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16566) Set column stats default as true when creating new tables/partitions
[ https://issues.apache.org/jira/browse/HIVE-16566?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong updated HIVE-16566: --- Status: Open (was: Patch Available) > Set column stats default as true when creating new tables/partitions > > > Key: HIVE-16566 > URL: https://issues.apache.org/jira/browse/HIVE-16566 > Project: Hive > Issue Type: Sub-task >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong > Attachments: HIVE-16566.01.patch > > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16566) Set column stats default as true when creating new tables/partitions
[ https://issues.apache.org/jira/browse/HIVE-16566?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong updated HIVE-16566: --- Status: Patch Available (was: Open) > Set column stats default as true when creating new tables/partitions > > > Key: HIVE-16566 > URL: https://issues.apache.org/jira/browse/HIVE-16566 > Project: Hive > Issue Type: Sub-task >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong > Attachments: HIVE-16566.01.patch > > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16552) Limit the number of tasks a Spark job may contain
[
https://issues.apache.org/jira/browse/HIVE-16552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15992287#comment-15992287
]
Rui Li commented on HIVE-16552:
---
Hi [~xuefuz], could you give some example when we want to put a limit on the
number of tasks? Because in my opinion, a user's share of YARN resources is
determined by the number of slots instead of number of tasks (although more
tasks means the slots will be held for longer time). And the number of slots
can be controlled by things like {{spark.executor.memory}} and
{{spark.executor.instances}}. Besides, what should we recommend to user for
this new config, given that the #reducers is automatically set by Hive?
> Limit the number of tasks a Spark job may contain
> -
>
> Key: HIVE-16552
> URL: https://issues.apache.org/jira/browse/HIVE-16552
> Project: Hive
> Issue Type: Improvement
> Components: Spark
>Affects Versions: 1.0.0, 2.0.0
>Reporter: Xuefu Zhang
>Assignee: Xuefu Zhang
> Attachments: HIVE-16552.1.patch, HIVE-16552.patch
>
>
> It's commonly desirable to block bad and big queries that takes a lot of YARN
> resources. One approach, similar to mapreduce.job.max.map in MapReduce, is to
> stop a query that invokes a Spark job that contains too many tasks. The
> proposal here is to introduce hive.spark.job.max.tasks with a default value
> of -1 (no limit), which an admin can set to block queries that trigger too
> many spark tasks.
> Please note that this control knob applies to a spark job, though it's
> possible that one query can trigger multiple Spark jobs (such as in case of
> map-join). Nevertheless, the proposed approach is still helpful.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-16534) Add capability to tell aborted transactions apart from open transactions in ValidTxnList
[
https://issues.apache.org/jira/browse/HIVE-16534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15992283#comment-15992283
]
Hive QA commented on HIVE-16534:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12865844/HIVE-16534.5.patch
{color:green}SUCCESS:{color} +1 due to 3 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 257 failed/errored test(s), 10634 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestAccumuloCliDriver.testCliDriver[accumulo_index]
(batchId=225)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_join] (batchId=15)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_mapjoin]
(batchId=10)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_subquery]
(batchId=37)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_table_stats]
(batchId=50)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_vectorization]
(batchId=61)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_vectorization_partition]
(batchId=68)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_vectorization_project]
(batchId=18)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[authorization_delete]
(batchId=77)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[authorization_delete_own_table]
(batchId=62)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[authorization_update]
(batchId=8)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[authorization_update_own_table]
(batchId=73)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[autoColumnStats_4]
(batchId=12)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[dbtxnmgr_showlocks]
(batchId=74)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[delete_all_non_partitioned]
(batchId=27)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[delete_all_partitioned]
(batchId=27)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[delete_orig_table]
(batchId=38)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[delete_tmp_table]
(batchId=48)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[delete_where_no_match]
(batchId=27)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[delete_where_non_partitioned]
(batchId=37)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[delete_where_partitioned]
(batchId=38)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[delete_whole_partition]
(batchId=9)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[insert_acid_dynamic_partition]
(batchId=19)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[insert_nonacid_from_acid]
(batchId=69)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[insert_orig_table]
(batchId=58)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[insert_update_delete]
(batchId=80)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[insert_values_dynamic_partitioned]
(batchId=71)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[insert_values_non_partitioned]
(batchId=19)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[insert_values_orig_table]
(batchId=54)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
(batchId=59)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[insert_values_partitioned]
(batchId=71)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[insert_values_tmp_table]
(batchId=4)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[llap_acid] (batchId=75)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[llap_reader] (batchId=7)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[masking_acid_no_masking]
(batchId=22)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[orc_ppd_exception]
(batchId=32)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[row__id] (batchId=73)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[transform_acid]
(batchId=19)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[update_after_multiple_inserts]
(batchId=64)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[update_after_multiple_inserts_special_characters]
(batchId=68)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[update_all_non_partitioned]
(batchId=7)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[update_all_partitioned]
(batchId=49)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[update_all_types]
(batchId=17)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[update_orig_table]
(batchId=58)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[update_tmp_table]
(batchId=34)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[update_two_cols]
(batchId=20)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[update_where_no_match]
(batchId=19)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[update_where_non_partitioned]
(batchId=15)
[jira] [Updated] (HIVE-16465) NullPointer Exception when enable vectorization for Parquet file format
[ https://issues.apache.org/jira/browse/HIVE-16465?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ferdinand Xu updated HIVE-16465: Resolution: Fixed Fix Version/s: (was: 2.3.0) Status: Resolved (was: Patch Available) Push to the master. Thanks [~colin_mjj] for the contribution. > NullPointer Exception when enable vectorization for Parquet file format > --- > > Key: HIVE-16465 > URL: https://issues.apache.org/jira/browse/HIVE-16465 > Project: Hive > Issue Type: Bug >Affects Versions: 2.2.0 >Reporter: Colin Ma >Assignee: Colin Ma >Priority: Critical > Fix For: 3.0.0 > > Attachments: HIVE-16465.001.patch, HIVE-16465-branch-2.3.001.patch, > HIVE-16465-branch-2.3.patch > > > NullPointer Exception when enable vectorization for Parquet file format. It > is caused by the null value of the InputSplit. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16456) Kill spark job when InterruptedException happens or driverContext.isShutdown is true.
[
https://issues.apache.org/jira/browse/HIVE-16456?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15992120#comment-15992120
]
Rui Li commented on HIVE-16456:
---
Thanks [~zxu] for working on this. One question is could you explain in what
situation will InterruptedException happen in the monitors? I think one case is
the sleep during check intervals. In other cases however, e.g.
{{sparkJobStatus.getState()}}, the InterruptedException may be wrapped and
thrown as a HiveException, which your patch doesn't handle.
And a minor improvement is we can check {{if (jobRef != null && !jobKilled)}}
before we enter the synchronized block right?
{code}
private void killJob() {
boolean needToKillJob = false;
synchronized(this) {
if (jobRef != null && !jobKilled) {
jobKilled = true;
needToKillJob = true;
}
}
{code}
> Kill spark job when InterruptedException happens or driverContext.isShutdown
> is true.
> -
>
> Key: HIVE-16456
> URL: https://issues.apache.org/jira/browse/HIVE-16456
> Project: Hive
> Issue Type: Improvement
>Reporter: zhihai xu
>Assignee: zhihai xu
>Priority: Minor
> Attachments: HIVE-16456.000.patch
>
>
> Kill spark job when InterruptedException happens or driverContext.isShutdown
> is true. If the InterruptedException happened in RemoteSparkJobMonitor and
> LocalSparkJobMonitor, it will be better to kill the job. Also there is a race
> condition between submit the spark job and query/operation cancellation, it
> will be better to check driverContext.isShutdown right after submit the spark
> job. This will guarantee the job being killed no matter when shutdown is
> called. It is similar as HIVE-15997.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-11133) Support hive.explain.user for Spark
[
https://issues.apache.org/jira/browse/HIVE-11133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15992118#comment-15992118
]
Sahil Takiar commented on HIVE-11133:
-
Hey [~xuefuz] I think the the file rename in the patch wasn't applied properly:
{code}
diff --git
a/common/src/java/org/apache/hadoop/hive/common/jsonexplain/tez/Connection.java
b/common/src/java/org/apache/hadoop/hive/common/jsonexplain/Connection.java
similarity index 95%
rename from
common/src/java/org/apache/hadoop/hive/common/jsonexplain/tez/Connection.java
rename to
common/src/java/org/apache/hadoop/hive/common/jsonexplain/Connection.java
diff --git
a/common/src/java/org/apache/hadoop/hive/common/jsonexplain/tez/TezJsonParserUtils.java
b/common/src/java/org/apache/hadoop/hive/common/jsonexplain/DagJsonParserUtils.java
similarity index 92%
rename from
common/src/java/org/apache/hadoop/hive/common/jsonexplain/tez/TezJsonParserUtils.java
rename to
common/src/java/org/apache/hadoop/hive/common/jsonexplain/DagJsonParserUtils.java
diff --git
a/common/src/java/org/apache/hadoop/hive/common/jsonexplain/tez/Op.java
b/common/src/java/org/apache/hadoop/hive/common/jsonexplain/Op.java
similarity index 90%
rename from
common/src/java/org/apache/hadoop/hive/common/jsonexplain/tez/Op.java
rename to common/src/java/org/apache/hadoop/hive/common/jsonexplain/Op.java
diff --git
a/common/src/java/org/apache/hadoop/hive/common/jsonexplain/tez/Printer.java
b/common/src/java/org/apache/hadoop/hive/common/jsonexplain/Printer.java
similarity index 95%
rename from
common/src/java/org/apache/hadoop/hive/common/jsonexplain/tez/Printer.java
rename to common/src/java/org/apache/hadoop/hive/common/jsonexplain/Printer.java
diff --git
a/common/src/java/org/apache/hadoop/hive/common/jsonexplain/tez/Stage.java
b/common/src/java/org/apache/hadoop/hive/common/jsonexplain/Stage.java
similarity index 94%
rename from
common/src/java/org/apache/hadoop/hive/common/jsonexplain/tez/Stage.java
rename to common/src/java/org/apache/hadoop/hive/common/jsonexplain/Stage.java
diff --git
a/common/src/java/org/apache/hadoop/hive/common/jsonexplain/tez/Vertex.java
b/common/src/java/org/apache/hadoop/hive/common/jsonexplain/Vertex.java
similarity index 81%
rename from
common/src/java/org/apache/hadoop/hive/common/jsonexplain/tez/Vertex.java
rename to common/src/java/org/apache/hadoop/hive/common/jsonexplain/Vertex.java
{code}
> Support hive.explain.user for Spark
> ---
>
> Key: HIVE-11133
> URL: https://issues.apache.org/jira/browse/HIVE-11133
> Project: Hive
> Issue Type: Sub-task
> Components: Spark
>Reporter: Mohit Sabharwal
>Assignee: Sahil Takiar
> Labels: TODOC3.0
> Fix For: 3.0.0
>
> Attachments: HIVE-11133.1.patch, HIVE-11133.2.patch,
> HIVE-11133.3.patch, HIVE-11133.4.patch, HIVE-11133.5.patch,
> HIVE-11133.6.patch, HIVE-11133.7.patch, HIVE-11133.8.patch, HIVE-11133.9.patch
>
>
> User friendly explain output ({{set hive.explain.user=true}}) should support
> Spark as well.
> Once supported, we should also enable related q-tests like {{explainuser_1.q}}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Comment Edited] (HIVE-11133) Support hive.explain.user for Spark
[
https://issues.apache.org/jira/browse/HIVE-11133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15992118#comment-15992118
]
Sahil Takiar edited comment on HIVE-11133 at 5/2/17 2:59 AM:
-
Hey [~xuefuz] I think the the file renames in the patch weren't applied
properly:
{code}
diff --git
a/common/src/java/org/apache/hadoop/hive/common/jsonexplain/tez/Connection.java
b/common/src/java/org/apache/hadoop/hive/common/jsonexplain/Connection.java
similarity index 95%
rename from
common/src/java/org/apache/hadoop/hive/common/jsonexplain/tez/Connection.java
rename to
common/src/java/org/apache/hadoop/hive/common/jsonexplain/Connection.java
diff --git
a/common/src/java/org/apache/hadoop/hive/common/jsonexplain/tez/TezJsonParserUtils.java
b/common/src/java/org/apache/hadoop/hive/common/jsonexplain/DagJsonParserUtils.java
similarity index 92%
rename from
common/src/java/org/apache/hadoop/hive/common/jsonexplain/tez/TezJsonParserUtils.java
rename to
common/src/java/org/apache/hadoop/hive/common/jsonexplain/DagJsonParserUtils.java
diff --git
a/common/src/java/org/apache/hadoop/hive/common/jsonexplain/tez/Op.java
b/common/src/java/org/apache/hadoop/hive/common/jsonexplain/Op.java
similarity index 90%
rename from
common/src/java/org/apache/hadoop/hive/common/jsonexplain/tez/Op.java
rename to common/src/java/org/apache/hadoop/hive/common/jsonexplain/Op.java
diff --git
a/common/src/java/org/apache/hadoop/hive/common/jsonexplain/tez/Printer.java
b/common/src/java/org/apache/hadoop/hive/common/jsonexplain/Printer.java
similarity index 95%
rename from
common/src/java/org/apache/hadoop/hive/common/jsonexplain/tez/Printer.java
rename to common/src/java/org/apache/hadoop/hive/common/jsonexplain/Printer.java
diff --git
a/common/src/java/org/apache/hadoop/hive/common/jsonexplain/tez/Stage.java
b/common/src/java/org/apache/hadoop/hive/common/jsonexplain/Stage.java
similarity index 94%
rename from
common/src/java/org/apache/hadoop/hive/common/jsonexplain/tez/Stage.java
rename to common/src/java/org/apache/hadoop/hive/common/jsonexplain/Stage.java
diff --git
a/common/src/java/org/apache/hadoop/hive/common/jsonexplain/tez/Vertex.java
b/common/src/java/org/apache/hadoop/hive/common/jsonexplain/Vertex.java
similarity index 81%
rename from
common/src/java/org/apache/hadoop/hive/common/jsonexplain/tez/Vertex.java
rename to common/src/java/org/apache/hadoop/hive/common/jsonexplain/Vertex.java
{code}
was (Author: stakiar):
Hey [~xuefuz] I think the the file rename in the patch wasn't applied properly:
{code}
diff --git
a/common/src/java/org/apache/hadoop/hive/common/jsonexplain/tez/Connection.java
b/common/src/java/org/apache/hadoop/hive/common/jsonexplain/Connection.java
similarity index 95%
rename from
common/src/java/org/apache/hadoop/hive/common/jsonexplain/tez/Connection.java
rename to
common/src/java/org/apache/hadoop/hive/common/jsonexplain/Connection.java
diff --git
a/common/src/java/org/apache/hadoop/hive/common/jsonexplain/tez/TezJsonParserUtils.java
b/common/src/java/org/apache/hadoop/hive/common/jsonexplain/DagJsonParserUtils.java
similarity index 92%
rename from
common/src/java/org/apache/hadoop/hive/common/jsonexplain/tez/TezJsonParserUtils.java
rename to
common/src/java/org/apache/hadoop/hive/common/jsonexplain/DagJsonParserUtils.java
diff --git
a/common/src/java/org/apache/hadoop/hive/common/jsonexplain/tez/Op.java
b/common/src/java/org/apache/hadoop/hive/common/jsonexplain/Op.java
similarity index 90%
rename from
common/src/java/org/apache/hadoop/hive/common/jsonexplain/tez/Op.java
rename to common/src/java/org/apache/hadoop/hive/common/jsonexplain/Op.java
diff --git
a/common/src/java/org/apache/hadoop/hive/common/jsonexplain/tez/Printer.java
b/common/src/java/org/apache/hadoop/hive/common/jsonexplain/Printer.java
similarity index 95%
rename from
common/src/java/org/apache/hadoop/hive/common/jsonexplain/tez/Printer.java
rename to common/src/java/org/apache/hadoop/hive/common/jsonexplain/Printer.java
diff --git
a/common/src/java/org/apache/hadoop/hive/common/jsonexplain/tez/Stage.java
b/common/src/java/org/apache/hadoop/hive/common/jsonexplain/Stage.java
similarity index 94%
rename from
common/src/java/org/apache/hadoop/hive/common/jsonexplain/tez/Stage.java
rename to common/src/java/org/apache/hadoop/hive/common/jsonexplain/Stage.java
diff --git
a/common/src/java/org/apache/hadoop/hive/common/jsonexplain/tez/Vertex.java
b/common/src/java/org/apache/hadoop/hive/common/jsonexplain/Vertex.java
similarity index 81%
rename from
common/src/java/org/apache/hadoop/hive/common/jsonexplain/tez/Vertex.java
rename to common/src/java/org/apache/hadoop/hive/common/jsonexplain/Vertex.java
{code}
> Support hive.explain.user for Spark
> ---
>
> Key: HIVE-11133
> URL: https://issues.apache.org/jira/browse/HIVE-11133
> Project:
[jira] [Updated] (HIVE-16566) Set column stats default as true when creating new tables/partitions
[ https://issues.apache.org/jira/browse/HIVE-16566?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong updated HIVE-16566: --- Status: Open (was: Patch Available) > Set column stats default as true when creating new tables/partitions > > > Key: HIVE-16566 > URL: https://issues.apache.org/jira/browse/HIVE-16566 > Project: Hive > Issue Type: Sub-task >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong > Attachments: HIVE-16566.01.patch > > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16566) Set column stats default as true when creating new tables/partitions
[ https://issues.apache.org/jira/browse/HIVE-16566?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong updated HIVE-16566: --- Status: Patch Available (was: Open) > Set column stats default as true when creating new tables/partitions > > > Key: HIVE-16566 > URL: https://issues.apache.org/jira/browse/HIVE-16566 > Project: Hive > Issue Type: Sub-task >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong > Attachments: HIVE-16566.01.patch > > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16399) create an index for tc_txnid in TXN_COMPONENTS
[ https://issues.apache.org/jira/browse/HIVE-16399?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Zheng updated HIVE-16399: - Resolution: Fixed Fix Version/s: 2.4.0 3.0.0 2.3.0 Status: Resolved (was: Patch Available) Committed to master, branch-2 and branch-2.3. Thanks Eugene for the review. > create an index for tc_txnid in TXN_COMPONENTS > -- > > Key: HIVE-16399 > URL: https://issues.apache.org/jira/browse/HIVE-16399 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 1.0.0 >Reporter: Eugene Koifman >Assignee: Wei Zheng > Fix For: 2.3.0, 3.0.0, 2.4.0 > > Attachments: HIVE-16399.branch-2.3.patch, HIVE-16399.branch-2.patch, > HIVE-16399.master.patch > > > w/o this TxnStore.cleanEmptyAbortedTxns() can be very slow -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16558) In the hiveserver2.jsp Closed Queries table under the data click Drilldown Link view details, the Chinese show garbled
[ https://issues.apache.org/jira/browse/HIVE-16558?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ZhangBing Lin updated HIVE-16558: - Description: In QueryProfileImpl.jamon,We see the following settings： HiveServer2 So we should set the response code to utf-8, which can avoid Chinese garbled or other languages garbled,Please check it! was: In QueryProfileImpl.jamon,We see the following settings： HiveServer2 So we should set the response code to utf-8, which can avoid Chinese garbled or other languages,Please check it! > In the hiveserver2.jsp Closed Queries table under the data click Drilldown > Link view details, the Chinese show garbled > -- > > Key: HIVE-16558 > URL: https://issues.apache.org/jira/browse/HIVE-16558 > Project: Hive > Issue Type: Bug >Affects Versions: 2.1.0 >Reporter: ZhangBing Lin >Assignee: ZhangBing Lin > Fix For: 3.0.0 > > Attachments: HIVE-16558.1.patch > > > In QueryProfileImpl.jamon,We see the following settings： > > > > > HiveServer2 > > > > > > > So we should set the response code to utf-8, which can avoid Chinese garbled > or other languages garbled,Please check it! -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16546) LLAP: Fail map join tasks if hash table memory exceeds threshold
[ https://issues.apache.org/jira/browse/HIVE-16546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15992015#comment-15992015 ] Prasanth Jayachandran commented on HIVE-16546: -- There is a new issue showed up in large scale test run. Will post a fix for it shortly. > LLAP: Fail map join tasks if hash table memory exceeds threshold > > > Key: HIVE-16546 > URL: https://issues.apache.org/jira/browse/HIVE-16546 > Project: Hive > Issue Type: Bug > Components: llap >Affects Versions: 3.0.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran > Attachments: HIVE-16546.1.patch, HIVE-16546.2.patch, > HIVE-16546.3.patch, HIVE-16546.4.patch, HIVE-16546.WIP.patch > > > When map join task is running in llap, it can potentially use lot more memory > than its limit which could be memory per executor or no conditional task > size. If it uses more memory, it can adversely affect other query performance > or it can even bring down the daemon. In such cases, it is better to fail the > query than to bring down the daemon. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Reopened] (HIVE-11133) Support hive.explain.user for Spark
[
https://issues.apache.org/jira/browse/HIVE-11133?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Prasanth Jayachandran reopened HIVE-11133:
--
This is breaking build in master
https://travis-ci.org/apache/hive/builds/227787298?utm_source=github_status_medium=notification
Reverted this patch to unblock build failures.
> Support hive.explain.user for Spark
> ---
>
> Key: HIVE-11133
> URL: https://issues.apache.org/jira/browse/HIVE-11133
> Project: Hive
> Issue Type: Sub-task
> Components: Spark
>Reporter: Mohit Sabharwal
>Assignee: Sahil Takiar
> Labels: TODOC3.0
> Fix For: 3.0.0
>
> Attachments: HIVE-11133.1.patch, HIVE-11133.2.patch,
> HIVE-11133.3.patch, HIVE-11133.4.patch, HIVE-11133.5.patch,
> HIVE-11133.6.patch, HIVE-11133.7.patch, HIVE-11133.8.patch, HIVE-11133.9.patch
>
>
> User friendly explain output ({{set hive.explain.user=true}}) should support
> Spark as well.
> Once supported, we should also enable related q-tests like {{explainuser_1.q}}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HIVE-16399) create an index for tc_txnid in TXN_COMPONENTS
[ https://issues.apache.org/jira/browse/HIVE-16399?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Zheng updated HIVE-16399: - Target Version/s: 2.3.0, 3.0.0, 2.4.0 (was: 3.0.0, 2.4.0) > create an index for tc_txnid in TXN_COMPONENTS > -- > > Key: HIVE-16399 > URL: https://issues.apache.org/jira/browse/HIVE-16399 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 1.0.0 >Reporter: Eugene Koifman >Assignee: Wei Zheng > Attachments: HIVE-16399.branch-2.3.patch, HIVE-16399.branch-2.patch, > HIVE-16399.master.patch > > > w/o this TxnStore.cleanEmptyAbortedTxns() can be very slow -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16548) LLAP: EncodedReaderImpl.addOneCompressionBuffer throws NPE
[
https://issues.apache.org/jira/browse/HIVE-16548?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15992006#comment-15992006
]
Sergey Shelukhin commented on HIVE-16548:
-
I can repro this on the same data... looks like the determination of what to
read is incorrect, it tries to read from the wrong place because the range from
disk is missing.
E.g. we try to get CB (or CBs, due to ORC index related safety margin at the
end) at [30615403, 30781251) from the stream at [30572092, 30781251), but the
corresponding two chunks from disk and/or cache are [30517488, 30571555) and
[30745405, 30781251) - the start of the range, and probably the entire CB we
want, is missing from the data. This can lead to all kinds of errors. Need to
investigate why.
> LLAP: EncodedReaderImpl.addOneCompressionBuffer throws NPE
> --
>
> Key: HIVE-16548
> URL: https://issues.apache.org/jira/browse/HIVE-16548
> Project: Hive
> Issue Type: Bug
> Components: llap
>Reporter: Rajesh Balamohan
>
> Env: Based on apr-25 apache master codebase.
> {noformat}
> Caused by: java.io.IOException: java.lang.IllegalArgumentException: Buffer
> size too small. size = 65536 needed = 3762509
> at
> org.apache.hadoop.hive.ql.io.orc.encoded.EncodedReaderImpl.readEncodedStream(EncodedReaderImpl.java:695)
> at
> org.apache.hadoop.hive.ql.io.orc.encoded.EncodedReaderImpl.readEncodedColumns(EncodedReaderImpl.java:454)
> at
> org.apache.hadoop.hive.llap.io.encoded.OrcEncodedDataReader.performDataRead(OrcEncodedDataReader.java:420)
> at
> org.apache.hadoop.hive.llap.io.encoded.OrcEncodedDataReader$4.run(OrcEncodedDataReader.java:242)
> at
> org.apache.hadoop.hive.llap.io.encoded.OrcEncodedDataReader$4.run(OrcEncodedDataReader.java:239)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1724)
> at
> org.apache.hadoop.hive.llap.io.encoded.OrcEncodedDataReader.callInternal(OrcEncodedDataReader.java:239)
> at
> org.apache.hadoop.hive.llap.io.encoded.OrcEncodedDataReader.callInternal(OrcEncodedDataReader.java:93)
> ... 6 more
> Caused by: java.lang.IllegalArgumentException: Buffer size too small. size =
> 65536 needed = 3762509
> at
> org.apache.hadoop.hive.ql.io.orc.encoded.EncodedReaderImpl.addOneCompressionBuffer(EncodedReaderImpl.java:1223)
> at
> org.apache.hadoop.hive.ql.io.orc.encoded.EncodedReaderImpl.prepareRangesForCompressedRead(EncodedReaderImpl.java:813)
> at
> org.apache.hadoop.hive.ql.io.orc.encoded.EncodedReaderImpl.readEncodedStream(EncodedReaderImpl.java:685)
> ... 15 more
> Caused by: java.io.IOException: java.io.IOException:
> java.lang.NullPointerException
> at
> org.apache.hadoop.hive.io.HiveIOExceptionHandlerChain.handleRecordReaderNextException(HiveIOExceptionHandlerChain.java:121)
> at
> org.apache.hadoop.hive.io.HiveIOExceptionHandlerUtil.handleRecordReaderNextException(HiveIOExceptionHandlerUtil.java:77)
> at
> org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.doNext(HiveContextAwareRecordReader.java:365)
> at
> org.apache.hadoop.hive.ql.io.HiveRecordReader.doNext(HiveRecordReader.java:79)
> at
> org.apache.hadoop.hive.ql.io.HiveRecordReader.doNext(HiveRecordReader.java:33)
> at
> org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.next(HiveContextAwareRecordReader.java:116)
> at
> org.apache.hadoop.mapred.split.TezGroupedSplitsInputFormat$TezGroupedSplitsRecordReader.next(TezGroupedSplitsInputFormat.java:151)
> at
> org.apache.tez.mapreduce.lib.MRReaderMapred.next(MRReaderMapred.java:116)
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.pushRecord(MapRecordSource.java:62)
> ... 17 more
> Caused by: java.io.IOException: java.lang.NullPointerException
> at
> org.apache.hadoop.hive.ql.io.orc.encoded.EncodedReaderImpl.readEncodedStream(EncodedReaderImpl.java:695)
> at
> org.apache.hadoop.hive.ql.io.orc.encoded.EncodedReaderImpl.readEncodedColumns(EncodedReaderImpl.java:454)
> at
> org.apache.hadoop.hive.llap.io.encoded.OrcEncodedDataReader.performDataRead(OrcEncodedDataReader.java:420)
> at
> org.apache.hadoop.hive.llap.io.encoded.OrcEncodedDataReader$4.run(OrcEncodedDataReader.java:242)
> at
> org.apache.hadoop.hive.llap.io.encoded.OrcEncodedDataReader$4.run(OrcEncodedDataReader.java:239)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
>
[jira] [Commented] (HIVE-16546) LLAP: Fail map join tasks if hash table memory exceeds threshold
[
https://issues.apache.org/jira/browse/HIVE-16546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15992002#comment-15992002
]
Hive QA commented on HIVE-16546:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12865854/HIVE-16546.4.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/4981/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/4981/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-4981/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2017-05-02 02:11:50.529
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-4981/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2017-05-02 02:11:50.532
+ cd apache-github-source-source
+ git fetch origin
+ git reset --hard HEAD
HEAD is now at 5ab03cb HIVE-16524: Remove the redundant item type in
hiveserver2.jsp and QueryProfileTmpl.jamon (ZhangBing via Xuefu)
+ git clean -f -d
Removing common/src/java/org/apache/hadoop/hive/common/MemoryEstimate.java
Removing
ql/src/java/org/apache/hadoop/hive/ql/exec/mapjoin/MapJoinMemoryExhaustionError.java
+ git checkout master
Already on 'master'
Your branch is up-to-date with 'origin/master'.
+ git reset --hard origin/master
HEAD is now at 5ab03cb HIVE-16524: Remove the redundant item type in
hiveserver2.jsp and QueryProfileTmpl.jamon (ZhangBing via Xuefu)
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2017-05-02 02:11:51.193
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh
/data/hiveptest/working/scratch/build.patch
Going to apply patch with: patch -p1
patching file common/src/java/org/apache/hadoop/hive/common/MemoryEstimate.java
patching file common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
Hunk #1 succeeded at 3161 (offset 9 lines).
patching file
ql/src/java/org/apache/hadoop/hive/ql/exec/mapjoin/MapJoinMemoryExhaustionError.java
patching file
ql/src/java/org/apache/hadoop/hive/ql/exec/mapjoin/MapJoinMemoryExhaustionException.java
patching file
ql/src/java/org/apache/hadoop/hive/ql/exec/mapjoin/MapJoinMemoryExhaustionHandler.java
patching file ql/src/java/org/apache/hadoop/hive/ql/exec/mr/MapredLocalTask.java
patching file
ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/BytesBytesMultiHashMap.java
patching file
ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/HashMapWrapper.java
patching file
ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/HybridHashTableContainer.java
patching file
ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinBytesTableContainer.java
patching file
ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinTableContainer.java
patching file
ql/src/java/org/apache/hadoop/hive/ql/exec/tez/HashTableLoader.java
patching file ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezProcessor.java
patching file
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/fast/VectorMapJoinFastBytesHashMap.java
patching file
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/fast/VectorMapJoinFastBytesHashMultiSet.java
patching file
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/fast/VectorMapJoinFastBytesHashSet.java
patching file
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/fast/VectorMapJoinFastBytesHashTable.java
patching file
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/fast/VectorMapJoinFastHashTable.java
patching file
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/fast/VectorMapJoinFastHashTableLoader.java
patching file
[jira] [Commented] (HIVE-16546) LLAP: Fail map join tasks if hash table memory exceeds threshold
[
https://issues.apache.org/jira/browse/HIVE-16546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15992000#comment-15992000
]
Hive QA commented on HIVE-16546:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12865854/HIVE-16546.4.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/4980/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/4980/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-4980/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2017-05-02 02:10:39.514
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-4980/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2017-05-02 02:10:39.516
+ cd apache-github-source-source
+ git fetch origin
+ git reset --hard HEAD
HEAD is now at 5ab03cb HIVE-16524: Remove the redundant item type in
hiveserver2.jsp and QueryProfileTmpl.jamon (ZhangBing via Xuefu)
+ git clean -f -d
+ git checkout master
Already on 'master'
Your branch is up-to-date with 'origin/master'.
+ git reset --hard origin/master
HEAD is now at 5ab03cb HIVE-16524: Remove the redundant item type in
hiveserver2.jsp and QueryProfileTmpl.jamon (ZhangBing via Xuefu)
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2017-05-02 02:10:39.982
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh
/data/hiveptest/working/scratch/build.patch
Going to apply patch with: patch -p1
patching file common/src/java/org/apache/hadoop/hive/common/MemoryEstimate.java
patching file common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
Hunk #1 succeeded at 3161 (offset 9 lines).
patching file
ql/src/java/org/apache/hadoop/hive/ql/exec/mapjoin/MapJoinMemoryExhaustionError.java
patching file
ql/src/java/org/apache/hadoop/hive/ql/exec/mapjoin/MapJoinMemoryExhaustionException.java
patching file
ql/src/java/org/apache/hadoop/hive/ql/exec/mapjoin/MapJoinMemoryExhaustionHandler.java
patching file ql/src/java/org/apache/hadoop/hive/ql/exec/mr/MapredLocalTask.java
patching file
ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/BytesBytesMultiHashMap.java
patching file
ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/HashMapWrapper.java
patching file
ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/HybridHashTableContainer.java
patching file
ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinBytesTableContainer.java
patching file
ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinTableContainer.java
patching file
ql/src/java/org/apache/hadoop/hive/ql/exec/tez/HashTableLoader.java
patching file ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezProcessor.java
patching file
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/fast/VectorMapJoinFastBytesHashMap.java
patching file
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/fast/VectorMapJoinFastBytesHashMultiSet.java
patching file
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/fast/VectorMapJoinFastBytesHashSet.java
patching file
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/fast/VectorMapJoinFastBytesHashTable.java
patching file
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/fast/VectorMapJoinFastHashTable.java
patching file
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/fast/VectorMapJoinFastHashTableLoader.java
patching file
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/fast/VectorMapJoinFastKeyStore.java
patching file
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/fast/VectorMapJoinFastLongHashMap.java
patching file
[jira] [Commented] (HIVE-5831) filter input files for bucketed tables
[
https://issues.apache.org/jira/browse/HIVE-5831?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15991997#comment-15991997
]
Hive QA commented on HIVE-5831:
---
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12614033/hive-5831.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/4979/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/4979/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-4979/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2017-05-02 02:09:44.763
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-4979/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2017-05-02 02:09:44.766
+ cd apache-github-source-source
+ git fetch origin
+ git reset --hard HEAD
HEAD is now at 5ab03cb HIVE-16524: Remove the redundant item type in
hiveserver2.jsp and QueryProfileTmpl.jamon (ZhangBing via Xuefu)
+ git clean -f -d
Removing ql/src/test/queries/clientpositive/alterColumnStats.q
Removing ql/src/test/queries/clientpositive/alterColumnStatsPart.q
Removing ql/src/test/results/clientpositive/alterColumnStats.q.out
Removing ql/src/test/results/clientpositive/alterColumnStatsPart.q.out
+ git checkout master
Already on 'master'
Your branch is up-to-date with 'origin/master'.
+ git reset --hard origin/master
HEAD is now at 5ab03cb HIVE-16524: Remove the redundant item type in
hiveserver2.jsp and QueryProfileTmpl.jamon (ZhangBing via Xuefu)
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2017-05-02 02:09:46.163
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh
/data/hiveptest/working/scratch/build.patch
error: src/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java: No such
file or directory
error: src/RELEASE_NOTES.txt: No such file or directory
error:
src/ql/src/java/org/apache/hadoop/hive/ql/optimizer/ppr/OpProcFactory.java: No
such file or directory
error: src/ql/src/java/org/apache/hadoop/hive/ql/optimizer/Optimizer.java: No
such file or directory
error: src/ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java:
No such file or directory
error: src/ql/src/java/org/apache/hadoop/hive/ql/plan/MapWork.java: No such
file or directory
error: src/ql/src/java/org/apache/hadoop/hive/ql/parse/MapReduceCompiler.java:
No such file or directory
error: src/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java:
No such file or directory
error: src/ql/src/java/org/apache/hadoop/hive/ql/parse/ParseContext.java: No
such file or directory
error: src/ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java: No such
file or directory
error: src/ql/src/java/org/apache/hadoop/hive/ql/io/HiveInputFormat.java: No
such file or directory
The patch does not appear to apply with p0, p1, or p2
+ exit 1
'
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12614033 - PreCommit-HIVE-Build
> filter input files for bucketed tables
> --
>
> Key: HIVE-5831
> URL: https://issues.apache.org/jira/browse/HIVE-5831
> Project: Hive
> Issue Type: Improvement
> Components: Query Processor
>Reporter: Rui Li
> Attachments: hive-5831.patch
>
>
> When the users query a bucketed table and use the bucketed column in the
> predicate, only the buckets that satisfy the predicate need to be scanned,
> thus improving the performance.
> Given a table test:
> CREATE TABLE test (x INT, y STRING) CLUSTERED BY ( x ) INTO 10 BUCKETS;
> The following query only has to scan
[jira] [Commented] (HIVE-16566) Set column stats default as true when creating new tables/partitions
[
https://issues.apache.org/jira/browse/HIVE-16566?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15991996#comment-15991996
]
Hive QA commented on HIVE-16566:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12865845/HIVE-16566.01.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/4978/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/4978/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-4978/
Messages:
{noformat}
This message was trimmed, see log for full details
patching file common/src/java/org/apache/hadoop/hive/common/StatsSetupConst.java
patching file hbase-handler/src/test/results/positive/external_table_ppd.q.out
patching file
hbase-handler/src/test/results/positive/hbase_binary_storage_queries.q.out
patching file hbase-handler/src/test/results/positive/hbasestats.q.out
patching file
itests/hive-blobstore/src/test/results/clientpositive/write_final_output_blobstore.q.out
patching file
metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java
patching file ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java
patching file ql/src/java/org/apache/hadoop/hive/ql/plan/CreateTableDesc.java
patching file ql/src/test/queries/clientpositive/alterColumnStats.q
patching file ql/src/test/queries/clientpositive/alterColumnStatsPart.q
patching file ql/src/test/results/clientnegative/alter_file_format.q.out
patching file
ql/src/test/results/clientnegative/stats_partialscan_autogether.q.out
patching file ql/src/test/results/clientnegative/unset_table_property.q.out
patching file ql/src/test/results/clientpositive/alterColumnStats.q.out
patching file ql/src/test/results/clientpositive/alterColumnStatsPart.q.out
patching file ql/src/test/results/clientpositive/alter_file_format.q.out
patching file
ql/src/test/results/clientpositive/alter_partition_clusterby_sortby.q.out
patching file ql/src/test/results/clientpositive/alter_skewed_table.q.out
patching file ql/src/test/results/clientpositive/alter_table_add_partition.q.out
patching file ql/src/test/results/clientpositive/alter_table_not_sorted.q.out
patching file ql/src/test/results/clientpositive/authorization_parts.q.out
patching file ql/src/test/results/clientpositive/autoColumnStats_3.q.out
patching file ql/src/test/results/clientpositive/autoColumnStats_4.q.out
patching file ql/src/test/results/clientpositive/autoColumnStats_5.q.out
patching file ql/src/test/results/clientpositive/binary_output_format.q.out
patching file ql/src/test/results/clientpositive/bucket1.q.out
patching file ql/src/test/results/clientpositive/bucket2.q.out
patching file ql/src/test/results/clientpositive/bucket_map_join_spark1.q.out
patching file ql/src/test/results/clientpositive/bucket_map_join_spark2.q.out
patching file ql/src/test/results/clientpositive/bucket_map_join_spark3.q.out
patching file ql/src/test/results/clientpositive/bucketmapjoin5.q.out
patching file ql/src/test/results/clientpositive/bucketmapjoin_negative.q.out
patching file ql/src/test/results/clientpositive/bucketmapjoin_negative2.q.out
patching file
ql/src/test/results/clientpositive/column_names_with_leading_and_trailing_spaces.q.out
patching file ql/src/test/results/clientpositive/columnstats_infinity.q.out
patching file ql/src/test/results/clientpositive/compustat_avro.q.out
patching file
ql/src/test/results/clientpositive/create_alter_list_bucketing_table1.q.out
patching file ql/src/test/results/clientpositive/create_like.q.out
patching file ql/src/test/results/clientpositive/create_like2.q.out
patching file ql/src/test/results/clientpositive/create_like_tbl_props.q.out
patching file ql/src/test/results/clientpositive/create_like_view.q.out
patching file ql/src/test/results/clientpositive/create_skewed_table1.q.out
patching file ql/src/test/results/clientpositive/create_table_like_stats.q.out
patching file ql/src/test/results/clientpositive/create_with_constraints.q.out
patching file ql/src/test/results/clientpositive/database_location.q.out
patching file ql/src/test/results/clientpositive/default_file_format.q.out
patching file ql/src/test/results/clientpositive/describe_comment_indent.q.out
patching file ql/src/test/results/clientpositive/describe_comment_nonascii.q.out
patching file
ql/src/test/results/clientpositive/describe_formatted_view_partitioned.q.out
patching file ql/src/test/results/clientpositive/describe_syntax.q.out
patching file ql/src/test/results/clientpositive/describe_table.q.out
patching file ql/src/test/results/clientpositive/display_colstats_tbllvl.q.out
patching file ql/src/test/results/clientpositive/druid_basic1.q.out
patching file ql/src/test/results/clientpositive/druid_basic2.q.out
patching file ql/src/test/results/clientpositive/druid_intervals.q.out
patching file
[jira] [Commented] (HIVE-16534) Add capability to tell aborted transactions apart from open transactions in ValidTxnList
[
https://issues.apache.org/jira/browse/HIVE-16534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15991995#comment-15991995
]
Hive QA commented on HIVE-16534:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12865844/HIVE-16534.5.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/4977/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/4977/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-4977/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2017-05-02 02:07:19.555
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-4977/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2017-05-02 02:07:19.557
+ cd apache-github-source-source
+ git fetch origin
+ git reset --hard HEAD
HEAD is now at 5ab03cb HIVE-16524: Remove the redundant item type in
hiveserver2.jsp and QueryProfileTmpl.jamon (ZhangBing via Xuefu)
+ git clean -f -d
Removing metastore/scripts/upgrade/hive/
Removing ql/src/test/queries/clientpositive/sysdb.q
Removing ql/src/test/results/clientpositive/llap/sysdb.q.out
+ git checkout master
Already on 'master'
Your branch is up-to-date with 'origin/master'.
+ git reset --hard origin/master
HEAD is now at 5ab03cb HIVE-16524: Remove the redundant item type in
hiveserver2.jsp and QueryProfileTmpl.jamon (ZhangBing via Xuefu)
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2017-05-02 02:07:20.082
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh
/data/hiveptest/working/scratch/build.patch
Going to apply patch with: patch -p0
patching file
common/src/java/org/apache/hadoop/hive/common/ValidCompactorTxnList.java
patching file
common/src/java/org/apache/hadoop/hive/common/ValidReadTxnList.java
patching file common/src/java/org/apache/hadoop/hive/common/ValidTxnList.java
patching file
common/src/test/org/apache/hadoop/hive/common/TestValidReadTxnList.java
patching file
itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/txn/compactor/TestCompactor.java
patching file metastore/if/hive_metastore.thrift
patching file metastore/src/gen/thrift/gen-cpp/ThriftHiveMetastore.cpp
patching file metastore/src/gen/thrift/gen-cpp/hive_metastore_types.cpp
patching file metastore/src/gen/thrift/gen-cpp/hive_metastore_types.h
patching file
metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/GetOpenTxnsResponse.java
patching file metastore/src/gen/thrift/gen-php/metastore/ThriftHiveMetastore.php
patching file metastore/src/gen/thrift/gen-php/metastore/Types.php
patching file metastore/src/gen/thrift/gen-py/hive_metastore/ttypes.py
patching file metastore/src/gen/thrift/gen-rb/hive_metastore_types.rb
patching file
metastore/src/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java
patching file
metastore/src/java/org/apache/hadoop/hive/metastore/txn/TxnUtils.java
patching file
metastore/src/test/org/apache/hadoop/hive/metastore/txn/TestValidCompactorTxnList.java
patching file ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Cleaner.java
+ [[ maven == \m\a\v\e\n ]]
+ rm -rf /data/hiveptest/working/maven/org/apache/hive
+ mvn -B clean install -DskipTests -T 4 -q
-Dmaven.repo.local=/data/hiveptest/working/maven
[ERROR] COMPILATION ERROR :
[ERROR]
/data/hiveptest/working/apache-github-source-source/common/src/java/org/apache/hadoop/hive/common/jsonexplain/tez/TezJsonParserUtils.java:[27,8]
class DagJsonParserUtils is public, should be declared in a file named
DagJsonParserUtils.java
[ERROR] Failed to execute goal
[jira] [Commented] (HIVE-1010) Implement INFORMATION_SCHEMA in Hive
[
https://issues.apache.org/jira/browse/HIVE-1010?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15991991#comment-15991991
]
Hive QA commented on HIVE-1010:
---
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12865843/HIVE-1010.12.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/4976/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/4976/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-4976/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2017-05-02 02:05:58.395
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-4976/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2017-05-02 02:05:58.397
+ cd apache-github-source-source
+ git fetch origin
+ git reset --hard HEAD
HEAD is now at 5ab03cb HIVE-16524: Remove the redundant item type in
hiveserver2.jsp and QueryProfileTmpl.jamon (ZhangBing via Xuefu)
+ git clean -f -d
+ git checkout master
Already on 'master'
Your branch is up-to-date with 'origin/master'.
+ git reset --hard origin/master
HEAD is now at 5ab03cb HIVE-16524: Remove the redundant item type in
hiveserver2.jsp and QueryProfileTmpl.jamon (ZhangBing via Xuefu)
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2017-05-02 02:05:58.872
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh
/data/hiveptest/working/scratch/build.patch
Going to apply patch with: patch -p0
patching file beeline/src/java/org/apache/hive/beeline/HiveSchemaHelper.java
patching file beeline/src/java/org/apache/hive/beeline/HiveSchemaTool.java
patching file beeline/src/test/org/apache/hive/beeline/TestHiveSchemaTool.java
patching file itests/src/test/resources/testconfiguration.properties
patching file
jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/JdbcInputFormat.java
patching file
jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/JdbcRecordReader.java
patching file
jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/JdbcSerDe.java
patching file
jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/JdbcStorageHandler.java
patching file
jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/conf/DatabaseType.java
patching file
jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/conf/JdbcStorageConfigManager.java
patching file
jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/dao/DatabaseAccessorFactory.java
patching file
jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/dao/GenericJdbcDatabaseAccessor.java
patching file
jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/dao/JdbcRecordIterator.java
patching file
jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/dao/MySqlDatabaseAccessor.java
patching file
jdbc-handler/src/test/java/org/apache/hive/config/JdbcStorageConfigManagerTest.java
patching file
llap-server/src/java/org/apache/hadoop/hive/llap/cli/LlapServiceDriver.java
patching file metastore/scripts/upgrade/hive/hive-schema-2.2.0-hive.sql
patching file ql/src/java/org/apache/hadoop/hive/ql/parse/UnparseTranslator.java
patching file ql/src/java/org/apache/hadoop/hive/ql/plan/PartitionDesc.java
patching file ql/src/test/queries/clientpositive/jdbc_handler.q
patching file ql/src/test/queries/clientpositive/sysdb.q
patching file ql/src/test/results/clientpositive/llap/jdbc_handler.q.out
patching file ql/src/test/results/clientpositive/llap/sysdb.q.out
+ [[ maven == \m\a\v\e\n ]]
+ rm -rf /data/hiveptest/working/maven/org/apache/hive
+ mvn -B clean install -DskipTests -T 4 -q
-Dmaven.repo.local=/data/hiveptest/working/maven
[ERROR] COMPILATION
[jira] [Commented] (HIVE-16558) In the hiveserver2.jsp Closed Queries table under the data click Drilldown Link view details, the Chinese show garbled
[ https://issues.apache.org/jira/browse/HIVE-16558?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15991992#comment-15991992 ] ZhangBing Lin commented on HIVE-16558: -- [~xuefuz],can you plz take a quick review? > In the hiveserver2.jsp Closed Queries table under the data click Drilldown > Link view details, the Chinese show garbled > -- > > Key: HIVE-16558 > URL: https://issues.apache.org/jira/browse/HIVE-16558 > Project: Hive > Issue Type: Bug >Affects Versions: 2.1.0 >Reporter: ZhangBing Lin >Assignee: ZhangBing Lin > Fix For: 3.0.0 > > Attachments: HIVE-16558.1.patch > > > In QueryProfileImpl.jamon,We see the following settings： > > > > > HiveServer2 > > > > > > > So we should set the response code to utf-8, which can avoid Chinese garbled > or other languages,Please check it! -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16534) Add capability to tell aborted transactions apart from open transactions in ValidTxnList
[
https://issues.apache.org/jira/browse/HIVE-16534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15991989#comment-15991989
]
Hive QA commented on HIVE-16534:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12865844/HIVE-16534.5.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/4974/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/4974/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-4974/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2017-05-02 02:03:35.278
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-4974/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2017-05-02 02:03:35.280
+ cd apache-github-source-source
+ git fetch origin
>From https://github.com/apache/hive
62fbdd8..5ab03cb master -> origin/master
+ git reset --hard HEAD
HEAD is now at 62fbdd8 Add license and notice file for storage-api
+ git clean -f -d
Removing metastore/scripts/upgrade/derby/040-HIVE-16399.derby.sql
Removing metastore/scripts/upgrade/mssql/025-HIVE-16399.mssql.sql
Removing metastore/scripts/upgrade/mysql/040-HIVE-16399.mysql.sql
Removing metastore/scripts/upgrade/oracle/040-HIVE-16399.oracle.sql
Removing metastore/scripts/upgrade/postgres/039-HIVE-16399.postgres.sql
+ git checkout master
Already on 'master'
Your branch is behind 'origin/master' by 2 commits, and can be fast-forwarded.
(use "git pull" to update your local branch)
+ git reset --hard origin/master
HEAD is now at 5ab03cb HIVE-16524: Remove the redundant item type in
hiveserver2.jsp and QueryProfileTmpl.jamon (ZhangBing via Xuefu)
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2017-05-02 02:03:40.406
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh
/data/hiveptest/working/scratch/build.patch
Going to apply patch with: patch -p0
patching file
common/src/java/org/apache/hadoop/hive/common/ValidCompactorTxnList.java
patching file
common/src/java/org/apache/hadoop/hive/common/ValidReadTxnList.java
patching file common/src/java/org/apache/hadoop/hive/common/ValidTxnList.java
patching file
common/src/test/org/apache/hadoop/hive/common/TestValidReadTxnList.java
patching file
itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/txn/compactor/TestCompactor.java
patching file metastore/if/hive_metastore.thrift
patching file metastore/src/gen/thrift/gen-cpp/ThriftHiveMetastore.cpp
patching file metastore/src/gen/thrift/gen-cpp/hive_metastore_types.cpp
patching file metastore/src/gen/thrift/gen-cpp/hive_metastore_types.h
patching file
metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/GetOpenTxnsResponse.java
patching file metastore/src/gen/thrift/gen-php/metastore/ThriftHiveMetastore.php
patching file metastore/src/gen/thrift/gen-php/metastore/Types.php
patching file metastore/src/gen/thrift/gen-py/hive_metastore/ttypes.py
patching file metastore/src/gen/thrift/gen-rb/hive_metastore_types.rb
patching file
metastore/src/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java
patching file
metastore/src/java/org/apache/hadoop/hive/metastore/txn/TxnUtils.java
patching file
metastore/src/test/org/apache/hadoop/hive/metastore/txn/TestValidCompactorTxnList.java
patching file ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Cleaner.java
+ [[ maven == \m\a\v\e\n ]]
+ rm -rf /data/hiveptest/working/maven/org/apache/hive
+ mvn -B clean install -DskipTests -T 4 -q
-Dmaven.repo.local=/data/hiveptest/working/maven
[ERROR] COMPILATION ERROR :
[ERROR]
[jira] [Commented] (HIVE-16555) Add a new thrift API call for get_metastore_uuid
[
https://issues.apache.org/jira/browse/HIVE-16555?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15991990#comment-15991990
]
Hive QA commented on HIVE-16555:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12865840/HIVE-16555.01.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/4975/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/4975/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-4975/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2017-05-02 02:05:01.546
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-4975/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2017-05-02 02:05:01.548
+ cd apache-github-source-source
+ git fetch origin
+ git reset --hard HEAD
HEAD is now at 5ab03cb HIVE-16524: Remove the redundant item type in
hiveserver2.jsp and QueryProfileTmpl.jamon (ZhangBing via Xuefu)
+ git clean -f -d
+ git checkout master
Already on 'master'
Your branch is up-to-date with 'origin/master'.
+ git reset --hard origin/master
HEAD is now at 5ab03cb HIVE-16524: Remove the redundant item type in
hiveserver2.jsp and QueryProfileTmpl.jamon (ZhangBing via Xuefu)
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2017-05-02 02:05:02.193
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh
/data/hiveptest/working/scratch/build.patch
error:
a/itests/hcatalog-unit/src/test/java/org/apache/hive/hcatalog/listener/DummyRawStoreFailEvent.java:
No such file or directory
error:
a/itests/hive-unit/src/test/java/org/apache/hadoop/hive/metastore/TestEmbeddedHiveMetaStore.java:
No such file or directory
error:
a/itests/hive-unit/src/test/java/org/apache/hadoop/hive/metastore/TestHiveMetaStore.java:
No such file or directory
error:
a/itests/hive-unit/src/test/java/org/apache/hadoop/hive/metastore/TestRemoteHiveMetaStore.java:
No such file or directory
error:
a/itests/hive-unit/src/test/java/org/apache/hadoop/hive/metastore/TestSetUGIOnOnlyClient.java:
No such file or directory
error:
a/itests/hive-unit/src/test/java/org/apache/hadoop/hive/metastore/TestSetUGIOnOnlyServer.java:
No such file or directory
error: a/metastore/if/hive_metastore.thrift: No such file or directory
error: a/metastore/src/gen/thrift/gen-cpp/ThriftHiveMetastore.cpp: No such file
or directory
error: a/metastore/src/gen/thrift/gen-cpp/ThriftHiveMetastore.h: No such file
or directory
error:
a/metastore/src/gen/thrift/gen-cpp/ThriftHiveMetastore_server.skeleton.cpp: No
such file or directory
error: a/metastore/src/gen/thrift/gen-cpp/hive_metastore_types.cpp: No such
file or directory
error: a/metastore/src/gen/thrift/gen-cpp/hive_metastore_types.h: No such file
or directory
error:
a/metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/ThriftHiveMetastore.java:
No such file or directory
error: a/metastore/src/gen/thrift/gen-php/metastore/ThriftHiveMetastore.php: No
such file or directory
error: a/metastore/src/gen/thrift/gen-php/metastore/Types.php: No such file or
directory
error:
a/metastore/src/gen/thrift/gen-py/hive_metastore/ThriftHiveMetastore-remote: No
such file or directory
error: a/metastore/src/gen/thrift/gen-py/hive_metastore/ThriftHiveMetastore.py:
No such file or directory
error: a/metastore/src/gen/thrift/gen-py/hive_metastore/ttypes.py: No such file
or directory
error: a/metastore/src/gen/thrift/gen-rb/hive_metastore_types.rb: No such file
or directory
error: a/metastore/src/gen/thrift/gen-rb/thrift_hive_metastore.rb: No such file
or directory
error:
[jira] [Commented] (HIVE-16399) create an index for tc_txnid in TXN_COMPONENTS
[
https://issues.apache.org/jira/browse/HIVE-16399?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15991986#comment-15991986
]
Hive QA commented on HIVE-16399:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12865836/HIVE-16399.master.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 3 failed/errored test(s), 10631 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestAccumuloCliDriver.testCliDriver[accumulo_index]
(batchId=225)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_if_expr]
(batchId=143)
org.apache.hadoop.hive.ql.parse.TestParseNegativeDriver.testCliDriver[wrong_distinct2]
(batchId=233)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/4973/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/4973/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-4973/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 3 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12865836 - PreCommit-HIVE-Build
> create an index for tc_txnid in TXN_COMPONENTS
> --
>
> Key: HIVE-16399
> URL: https://issues.apache.org/jira/browse/HIVE-16399
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 1.0.0
>Reporter: Eugene Koifman
>Assignee: Wei Zheng
> Attachments: HIVE-16399.branch-2.3.patch, HIVE-16399.branch-2.patch,
> HIVE-16399.master.patch
>
>
> w/o this TxnStore.cleanEmptyAbortedTxns() can be very slow
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HIVE-16524) Remove the redundant item type in hiveserver2.jsp and QueryProfileTmpl.jamon
[ https://issues.apache.org/jira/browse/HIVE-16524?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuefu Zhang updated HIVE-16524: --- Resolution: Fixed Fix Version/s: 3.0.0 Status: Resolved (was: Patch Available) Committed to master. Thanks, Zhangbing! > Remove the redundant item type in hiveserver2.jsp and QueryProfileTmpl.jamon > > > Key: HIVE-16524 > URL: https://issues.apache.org/jira/browse/HIVE-16524 > Project: Hive > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: ZhangBing Lin >Assignee: ZhangBing Lin >Priority: Minor > Fix For: 3.0.0 > > Attachments: HIVE-16524.1.patch > > > The Id attribute is defined in w3c as follows: > 1.The id attribute specifies the unique id of the HTML element. > 2.Id must be unique in the HTML document. > 3.The id attribute can be used as a link anchor, by JavaScript (HTML DOM) or > by CSS to change or add a style to an element with the specified id. > But,the "id='attributes_table'" in hiveserver2.jsp and > QueryProfileTmpl.jamon： > 1.Not quoted by any css and js > 2.It has the same id attribute name on the same page > So I suggest removing this id attribute definition，Please Check It. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-1010) Implement INFORMATION_SCHEMA in Hive
[ https://issues.apache.org/jira/browse/HIVE-1010?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15991941#comment-15991941 ] Thejas M Nair commented on HIVE-1010: - Some questions/comments - - Does the create table commands work against other databases ? - If you need to customize queries against other db, how would you detect the underlying db type - Version table under sys should not be a mirror of underlying rdbms versions table. Since upgrade of 'sys' db would be a separate invocation, it won't work with rdbms schema upgrade. You can create a separate table db_version or so that mirrors underlying version table. - upgrade scripts - you can keep that simple by just doing drop and re-create via new 'init' schema file. - secure db password being sent to tasks via job.credentials apis - Looks like this would go into hive 3.0.0, so schema file name would need to be updated. > Implement INFORMATION_SCHEMA in Hive > > > Key: HIVE-1010 > URL: https://issues.apache.org/jira/browse/HIVE-1010 > Project: Hive > Issue Type: New Feature > Components: Metastore, Query Processor, Server Infrastructure >Reporter: Jeff Hammerbacher >Assignee: Gunther Hagleitner > Attachments: HIVE-1010.10.patch, HIVE-1010.11.patch, > HIVE-1010.12.patch, HIVE-1010.7.patch, HIVE-1010.8.patch, HIVE-1010.9.patch > > > INFORMATION_SCHEMA is part of the SQL92 standard and would be useful to > implement using our metastore. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-11133) Support hive.explain.user for Spark
[
https://issues.apache.org/jira/browse/HIVE-11133?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xuefu Zhang updated HIVE-11133:
---
Labels: TODOC3.0 (was: )
> Support hive.explain.user for Spark
> ---
>
> Key: HIVE-11133
> URL: https://issues.apache.org/jira/browse/HIVE-11133
> Project: Hive
> Issue Type: Sub-task
> Components: Spark
>Reporter: Mohit Sabharwal
>Assignee: Sahil Takiar
> Labels: TODOC3.0
> Fix For: 3.0.0
>
> Attachments: HIVE-11133.1.patch, HIVE-11133.2.patch,
> HIVE-11133.3.patch, HIVE-11133.4.patch, HIVE-11133.5.patch,
> HIVE-11133.6.patch, HIVE-11133.7.patch, HIVE-11133.8.patch, HIVE-11133.9.patch
>
>
> User friendly explain output ({{set hive.explain.user=true}}) should support
> Spark as well.
> Once supported, we should also enable related q-tests like {{explainuser_1.q}}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HIVE-11133) Support hive.explain.user for Spark
[
https://issues.apache.org/jira/browse/HIVE-11133?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xuefu Zhang updated HIVE-11133:
---
Resolution: Fixed
Fix Version/s: 3.0.0
Status: Resolved (was: Patch Available)
Committed to master. Thanks, Sahil!
> Support hive.explain.user for Spark
> ---
>
> Key: HIVE-11133
> URL: https://issues.apache.org/jira/browse/HIVE-11133
> Project: Hive
> Issue Type: Sub-task
> Components: Spark
>Reporter: Mohit Sabharwal
>Assignee: Sahil Takiar
> Fix For: 3.0.0
>
> Attachments: HIVE-11133.1.patch, HIVE-11133.2.patch,
> HIVE-11133.3.patch, HIVE-11133.4.patch, HIVE-11133.5.patch,
> HIVE-11133.6.patch, HIVE-11133.7.patch, HIVE-11133.8.patch, HIVE-11133.9.patch
>
>
> User friendly explain output ({{set hive.explain.user=true}}) should support
> Spark as well.
> Once supported, we should also enable related q-tests like {{explainuser_1.q}}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-16556) Modify schematool scripts to initialize and create METASTORE_DB_PROPERTIES table
[
https://issues.apache.org/jira/browse/HIVE-16556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15991928#comment-15991928
]
Hive QA commented on HIVE-16556:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12865831/HIVE-16556.01.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 3 failed/errored test(s), 10631 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestAccumuloCliDriver.testCliDriver[accumulo_index]
(batchId=225)
org.apache.hadoop.hive.cli.TestAccumuloCliDriver.testCliDriver[accumulo_queries]
(batchId=225)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[columnstats_part_coltype]
(batchId=155)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/4972/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/4972/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-4972/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 3 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12865831 - PreCommit-HIVE-Build
> Modify schematool scripts to initialize and create METASTORE_DB_PROPERTIES
> table
>
>
> Key: HIVE-16556
> URL: https://issues.apache.org/jira/browse/HIVE-16556
> Project: Hive
> Issue Type: Sub-task
> Components: Metastore
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
> Attachments: HIVE-16556.01.patch
>
>
> sub-task to modify schema tool and its related changes so that the new table
> is added to the schema when schematool initializes or upgrades the schema.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-16524) Remove the redundant item type in hiveserver2.jsp and QueryProfileTmpl.jamon
[ https://issues.apache.org/jira/browse/HIVE-16524?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15991915#comment-15991915 ] ZhangBing Lin commented on HIVE-16524: -- Test failures are unrelated.Please check it. > Remove the redundant item type in hiveserver2.jsp and QueryProfileTmpl.jamon > > > Key: HIVE-16524 > URL: https://issues.apache.org/jira/browse/HIVE-16524 > Project: Hive > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: ZhangBing Lin >Assignee: ZhangBing Lin >Priority: Minor > Attachments: HIVE-16524.1.patch > > > The Id attribute is defined in w3c as follows: > 1.The id attribute specifies the unique id of the HTML element. > 2.Id must be unique in the HTML document. > 3.The id attribute can be used as a link anchor, by JavaScript (HTML DOM) or > by CSS to change or add a style to an element with the specified id. > But,the "id='attributes_table'" in hiveserver2.jsp and > QueryProfileTmpl.jamon： > 1.Not quoted by any css and js > 2.It has the same id attribute name on the same page > So I suggest removing this id attribute definition，Please Check It. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16558) In the hiveserver2.jsp Closed Queries table under the data click Drilldown Link view details, the Chinese show garbled
[ https://issues.apache.org/jira/browse/HIVE-16558?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15991912#comment-15991912 ] ZhangBing Lin commented on HIVE-16558: -- Test failures are unrelated. > In the hiveserver2.jsp Closed Queries table under the data click Drilldown > Link view details, the Chinese show garbled > -- > > Key: HIVE-16558 > URL: https://issues.apache.org/jira/browse/HIVE-16558 > Project: Hive > Issue Type: Bug >Affects Versions: 2.1.0 >Reporter: ZhangBing Lin >Assignee: ZhangBing Lin > Fix For: 3.0.0 > > Attachments: HIVE-16558.1.patch > > > In QueryProfileImpl.jamon,We see the following settings： > > > > > HiveServer2 > > > > > > > So we should set the response code to utf-8, which can avoid Chinese garbled > or other languages,Please check it! -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16546) LLAP: Fail map join tasks if hash table memory exceeds threshold
[ https://issues.apache.org/jira/browse/HIVE-16546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15991904#comment-15991904 ] Sergey Shelukhin commented on HIVE-16546: - +1 > LLAP: Fail map join tasks if hash table memory exceeds threshold > > > Key: HIVE-16546 > URL: https://issues.apache.org/jira/browse/HIVE-16546 > Project: Hive > Issue Type: Bug > Components: llap >Affects Versions: 3.0.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran > Attachments: HIVE-16546.1.patch, HIVE-16546.2.patch, > HIVE-16546.3.patch, HIVE-16546.4.patch, HIVE-16546.WIP.patch > > > When map join task is running in llap, it can potentially use lot more memory > than its limit which could be memory per executor or no conditional task > size. If it uses more memory, it can adversely affect other query performance > or it can even bring down the daemon. In such cases, it is better to fail the > query than to bring down the daemon. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-5831) filter input files for bucketed tables
[ https://issues.apache.org/jira/browse/HIVE-5831?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuefu Zhang updated HIVE-5831: -- Resolution: Duplicate Status: Resolved (was: Patch Available) > filter input files for bucketed tables > -- > > Key: HIVE-5831 > URL: https://issues.apache.org/jira/browse/HIVE-5831 > Project: Hive > Issue Type: Improvement > Components: Query Processor >Reporter: Rui Li > Attachments: hive-5831.patch > > > When the users query a bucketed table and use the bucketed column in the > predicate, only the buckets that satisfy the predicate need to be scanned, > thus improving the performance. > Given a table test: > CREATE TABLE test (x INT, y STRING) CLUSTERED BY ( x ) INTO 10 BUCKETS; > The following query only has to scan bucket 5: > SELECT * FROM test WHERE x=5; -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-5831) filter input files for bucketed tables
[ https://issues.apache.org/jira/browse/HIVE-5831?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15991898#comment-15991898 ] Xuefu Zhang commented on HIVE-5831: --- Great! Thanks for the info, [~gopalv]! > filter input files for bucketed tables > -- > > Key: HIVE-5831 > URL: https://issues.apache.org/jira/browse/HIVE-5831 > Project: Hive > Issue Type: Improvement > Components: Query Processor >Reporter: Rui Li > Attachments: hive-5831.patch > > > When the users query a bucketed table and use the bucketed column in the > predicate, only the buckets that satisfy the predicate need to be scanned, > thus improving the performance. > Given a table test: > CREATE TABLE test (x INT, y STRING) CLUSTERED BY ( x ) INTO 10 BUCKETS; > The following query only has to scan bucket 5: > SELECT * FROM test WHERE x=5; -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16546) LLAP: Fail map join tasks if hash table memory exceeds threshold
[ https://issues.apache.org/jira/browse/HIVE-16546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15991897#comment-15991897 ] Prasanth Jayachandran commented on HIVE-16546: -- [~sershe] could you please take another look? Thanks! > LLAP: Fail map join tasks if hash table memory exceeds threshold > > > Key: HIVE-16546 > URL: https://issues.apache.org/jira/browse/HIVE-16546 > Project: Hive > Issue Type: Bug > Components: llap >Affects Versions: 3.0.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran > Attachments: HIVE-16546.1.patch, HIVE-16546.2.patch, > HIVE-16546.3.patch, HIVE-16546.4.patch, HIVE-16546.WIP.patch > > > When map join task is running in llap, it can potentially use lot more memory > than its limit which could be memory per executor or no conditional task > size. If it uses more memory, it can adversely affect other query performance > or it can even bring down the daemon. In such cases, it is better to fail the > query than to bring down the daemon. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-5831) filter input files for bucketed tables
[ https://issues.apache.org/jira/browse/HIVE-5831?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15991892#comment-15991892 ] Gopal V commented on HIVE-5831: --- [~xuefuz]: HIVE-11525 obsoletes this approach - see TableScanDesc::includedBuckets in Apache master > filter input files for bucketed tables > -- > > Key: HIVE-5831 > URL: https://issues.apache.org/jira/browse/HIVE-5831 > Project: Hive > Issue Type: Improvement > Components: Query Processor >Reporter: Rui Li > Attachments: hive-5831.patch > > > When the users query a bucketed table and use the bucketed column in the > predicate, only the buckets that satisfy the predicate need to be scanned, > thus improving the performance. > Given a table test: > CREATE TABLE test (x INT, y STRING) CLUSTERED BY ( x ) INTO 10 BUCKETS; > The following query only has to scan bucket 5: > SELECT * FROM test WHERE x=5; -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16546) LLAP: Fail map join tasks if hash table memory exceeds threshold
[ https://issues.apache.org/jira/browse/HIVE-16546?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-16546: - Attachment: HIVE-16546.4.patch Minor change related to logging. > LLAP: Fail map join tasks if hash table memory exceeds threshold > > > Key: HIVE-16546 > URL: https://issues.apache.org/jira/browse/HIVE-16546 > Project: Hive > Issue Type: Bug > Components: llap >Affects Versions: 3.0.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran > Attachments: HIVE-16546.1.patch, HIVE-16546.2.patch, > HIVE-16546.3.patch, HIVE-16546.4.patch, HIVE-16546.WIP.patch > > > When map join task is running in llap, it can potentially use lot more memory > than its limit which could be memory per executor or no conditional task > size. If it uses more memory, it can adversely affect other query performance > or it can even bring down the daemon. In such cases, it is better to fail the > query than to bring down the daemon. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16399) create an index for tc_txnid in TXN_COMPONENTS
[ https://issues.apache.org/jira/browse/HIVE-16399?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15991889#comment-15991889 ] Eugene Koifman commented on HIVE-16399: --- +1 > create an index for tc_txnid in TXN_COMPONENTS > -- > > Key: HIVE-16399 > URL: https://issues.apache.org/jira/browse/HIVE-16399 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 1.0.0 >Reporter: Eugene Koifman >Assignee: Wei Zheng > Attachments: HIVE-16399.branch-2.3.patch, HIVE-16399.branch-2.patch, > HIVE-16399.master.patch > > > w/o this TxnStore.cleanEmptyAbortedTxns() can be very slow -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Issue Comment Deleted] (HIVE-16399) create an index for tc_txnid in TXN_COMPONENTS
[ https://issues.apache.org/jira/browse/HIVE-16399?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman updated HIVE-16399: -- Comment: was deleted (was: why is it upgrade-2.2.0-to-2.3.0.derby.sql rather than 2.3.0-3.0.0? ) > create an index for tc_txnid in TXN_COMPONENTS > -- > > Key: HIVE-16399 > URL: https://issues.apache.org/jira/browse/HIVE-16399 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 1.0.0 >Reporter: Eugene Koifman >Assignee: Wei Zheng > Attachments: HIVE-16399.branch-2.3.patch, HIVE-16399.branch-2.patch, > HIVE-16399.master.patch > > > w/o this TxnStore.cleanEmptyAbortedTxns() can be very slow -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16546) LLAP: Fail map join tasks if hash table memory exceeds threshold
[ https://issues.apache.org/jira/browse/HIVE-16546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15991885#comment-15991885 ] Prasanth Jayachandran commented on HIVE-16546: -- Fixed review comments in the patch. Regarding the inflation factor. It is already accounted in during statistics annotation. Most file formats provide rawDataSize which is deserialized and decompressed data size which is stored in the metastore. When rawDataSize is available, the map join decision is already taken based on that. So inflation factor is already accounted in most cases. Example ORC file could be 10MB on disk which corresponds to totalSize in metastore whose rawDataSize could be several 100MB which is also stored in metastore. The inflation factor in the context of this patch is we allow the hash table to expand by that many factors in-memory before killing it. Say if container size is 4GB, noconditional task size is configured to 1GB. We will wait until the estimated memory size to reach 2GB before killing the task. Addressed the grace size review comment with Math.max(threshold, (2/3)*maxMemoryAvailable).. maxMemoryAvailable will be container size in case of tez and memory per executor in case of LLAP. This guards against poorly configured noconditional task size. > LLAP: Fail map join tasks if hash table memory exceeds threshold > > > Key: HIVE-16546 > URL: https://issues.apache.org/jira/browse/HIVE-16546 > Project: Hive > Issue Type: Bug > Components: llap >Affects Versions: 3.0.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran > Attachments: HIVE-16546.1.patch, HIVE-16546.2.patch, > HIVE-16546.3.patch, HIVE-16546.WIP.patch > > > When map join task is running in llap, it can potentially use lot more memory > than its limit which could be memory per executor or no conditional task > size. If it uses more memory, it can adversely affect other query performance > or it can even bring down the daemon. In such cases, it is better to fail the > query than to bring down the daemon. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16399) create an index for tc_txnid in TXN_COMPONENTS
[ https://issues.apache.org/jira/browse/HIVE-16399?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15991884#comment-15991884 ] Eugene Koifman commented on HIVE-16399: --- why is it upgrade-2.2.0-to-2.3.0.derby.sql rather than 2.3.0-3.0.0? > create an index for tc_txnid in TXN_COMPONENTS > -- > > Key: HIVE-16399 > URL: https://issues.apache.org/jira/browse/HIVE-16399 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 1.0.0 >Reporter: Eugene Koifman >Assignee: Wei Zheng > Attachments: HIVE-16399.branch-2.3.patch, HIVE-16399.branch-2.patch, > HIVE-16399.master.patch > > > w/o this TxnStore.cleanEmptyAbortedTxns() can be very slow -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16546) LLAP: Fail map join tasks if hash table memory exceeds threshold
[ https://issues.apache.org/jira/browse/HIVE-16546?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-16546: - Attachment: HIVE-16546.3.patch > LLAP: Fail map join tasks if hash table memory exceeds threshold > > > Key: HIVE-16546 > URL: https://issues.apache.org/jira/browse/HIVE-16546 > Project: Hive > Issue Type: Bug > Components: llap >Affects Versions: 3.0.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran > Attachments: HIVE-16546.1.patch, HIVE-16546.2.patch, > HIVE-16546.3.patch, HIVE-16546.WIP.patch > > > When map join task is running in llap, it can potentially use lot more memory > than its limit which could be memory per executor or no conditional task > size. If it uses more memory, it can adversely affect other query performance > or it can even bring down the daemon. In such cases, it is better to fail the > query than to bring down the daemon. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-11133) Support hive.explain.user for Spark
[
https://issues.apache.org/jira/browse/HIVE-11133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15991861#comment-15991861
]
Hive QA commented on HIVE-11133:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12865832/HIVE-11133.9.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 3 failed/errored test(s), 10631 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestAccumuloCliDriver.testCliDriver[accumulo_index]
(batchId=225)
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[smb_mapjoin_12]
(batchId=234)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_if_expr]
(batchId=143)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/4971/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/4971/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-4971/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 3 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12865832 - PreCommit-HIVE-Build
> Support hive.explain.user for Spark
> ---
>
> Key: HIVE-11133
> URL: https://issues.apache.org/jira/browse/HIVE-11133
> Project: Hive
> Issue Type: Sub-task
> Components: Spark
>Reporter: Mohit Sabharwal
>Assignee: Sahil Takiar
> Attachments: HIVE-11133.1.patch, HIVE-11133.2.patch,
> HIVE-11133.3.patch, HIVE-11133.4.patch, HIVE-11133.5.patch,
> HIVE-11133.6.patch, HIVE-11133.7.patch, HIVE-11133.8.patch, HIVE-11133.9.patch
>
>
> User friendly explain output ({{set hive.explain.user=true}}) should support
> Spark as well.
> Once supported, we should also enable related q-tests like {{explainuser_1.q}}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-5831) filter input files for bucketed tables
[ https://issues.apache.org/jira/browse/HIVE-5831?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15991859#comment-15991859 ] Xuefu Zhang commented on HIVE-5831: --- Any interest in moving this forward? > filter input files for bucketed tables > -- > > Key: HIVE-5831 > URL: https://issues.apache.org/jira/browse/HIVE-5831 > Project: Hive > Issue Type: Improvement > Components: Query Processor >Reporter: Rui Li > Attachments: hive-5831.patch > > > When the users query a bucketed table and use the bucketed column in the > predicate, only the buckets that satisfy the predicate need to be scanned, > thus improving the performance. > Given a table test: > CREATE TABLE test (x INT, y STRING) CLUSTERED BY ( x ) INTO 10 BUCKETS; > The following query only has to scan bucket 5: > SELECT * FROM test WHERE x=5; -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16566) Set column stats default as true when creating new tables/partitions
[ https://issues.apache.org/jira/browse/HIVE-16566?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong updated HIVE-16566: --- Status: Patch Available (was: Open) > Set column stats default as true when creating new tables/partitions > > > Key: HIVE-16566 > URL: https://issues.apache.org/jira/browse/HIVE-16566 > Project: Hive > Issue Type: Sub-task >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong > Attachments: HIVE-16566.01.patch > > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Assigned] (HIVE-16566) Set column stats default as true when creating new tables/partitions
[ https://issues.apache.org/jira/browse/HIVE-16566?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong reassigned HIVE-16566: -- > Set column stats default as true when creating new tables/partitions > > > Key: HIVE-16566 > URL: https://issues.apache.org/jira/browse/HIVE-16566 > Project: Hive > Issue Type: Sub-task >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16534) Add capability to tell aborted transactions apart from open transactions in ValidTxnList
[ https://issues.apache.org/jira/browse/HIVE-16534?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Zheng updated HIVE-16534: - Attachment: HIVE-16534.5.patch > Add capability to tell aborted transactions apart from open transactions in > ValidTxnList > > > Key: HIVE-16534 > URL: https://issues.apache.org/jira/browse/HIVE-16534 > Project: Hive > Issue Type: Bug > Components: Transactions >Reporter: Wei Zheng >Assignee: Wei Zheng > Attachments: HIVE-16534.1.patch, HIVE-16534.2.patch, > HIVE-16534.3.patch, HIVE-16534.4.patch, HIVE-16534.5.patch > > > Currently in ValidReadTxnList, open transactions and aborted transactions are > stored together in one array. That makes it impossible to extract just > aborted transactions or open transactions. > For ValidCompactorTxnList this is fine, since we only store aborted > transactions but no open transactions. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-1010) Implement INFORMATION_SCHEMA in Hive
[ https://issues.apache.org/jira/browse/HIVE-1010?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15991803#comment-15991803 ] Gunther Hagleitner commented on HIVE-1010: -- .12 is rebased. > Implement INFORMATION_SCHEMA in Hive > > > Key: HIVE-1010 > URL: https://issues.apache.org/jira/browse/HIVE-1010 > Project: Hive > Issue Type: New Feature > Components: Metastore, Query Processor, Server Infrastructure >Reporter: Jeff Hammerbacher >Assignee: Gunther Hagleitner > Attachments: HIVE-1010.10.patch, HIVE-1010.11.patch, > HIVE-1010.12.patch, HIVE-1010.7.patch, HIVE-1010.8.patch, HIVE-1010.9.patch > > > INFORMATION_SCHEMA is part of the SQL92 standard and would be useful to > implement using our metastore. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-1010) Implement INFORMATION_SCHEMA in Hive
[ https://issues.apache.org/jira/browse/HIVE-1010?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gunther Hagleitner updated HIVE-1010: - Attachment: HIVE-1010.12.patch > Implement INFORMATION_SCHEMA in Hive > > > Key: HIVE-1010 > URL: https://issues.apache.org/jira/browse/HIVE-1010 > Project: Hive > Issue Type: New Feature > Components: Metastore, Query Processor, Server Infrastructure >Reporter: Jeff Hammerbacher >Assignee: Gunther Hagleitner > Attachments: HIVE-1010.10.patch, HIVE-1010.11.patch, > HIVE-1010.12.patch, HIVE-1010.7.patch, HIVE-1010.8.patch, HIVE-1010.9.patch > > > INFORMATION_SCHEMA is part of the SQL92 standard and would be useful to > implement using our metastore. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16555) Add a new thrift API call for get_metastore_uuid
[ https://issues.apache.org/jira/browse/HIVE-16555?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vihang Karajgaonkar updated HIVE-16555: --- Attachment: HIVE-16555.01.patch > Add a new thrift API call for get_metastore_uuid > > > Key: HIVE-16555 > URL: https://issues.apache.org/jira/browse/HIVE-16555 > Project: Hive > Issue Type: Sub-task > Components: Metastore >Reporter: Vihang Karajgaonkar >Assignee: Vihang Karajgaonkar > Attachments: HIVE-16555.01.patch > > > Sub-task of the main JIRA to add the new thrift API -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16513) width_bucket issues
[ https://issues.apache.org/jira/browse/HIVE-16513?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15991784#comment-15991784 ] Sahil Takiar commented on HIVE-16513: - [~ashutoshc], addressed comments. Test failures are flaky. > width_bucket issues > --- > > Key: HIVE-16513 > URL: https://issues.apache.org/jira/browse/HIVE-16513 > Project: Hive > Issue Type: Bug >Reporter: Carter Shanklin >Assignee: Sahil Takiar > Attachments: HIVE-16513.1.patch, HIVE-16513.2.patch, > HIVE-16513.3.patch > > > width_bucket was recently added with HIVE-15982. This ticket notes a few > issues. > Usability issue: > Currently only accepts integral numeric types. Decimals, floats and doubles > are not supported. > Runtime failures: This query will cause a runtime divide-by-zero in the > reduce stage. > select width_bucket(c1, 0, c1*2, 10) from e011_01 group by c1; > The divide-by-zero seems to trigger any time I use a group-by. Here's another > example (that actually requires the group-by): > select width_bucket(c1, 0, max(c1), 10) from e011_01 group by c1; > Advanced Usage Issues: > Suppose you have a table e011_01 as follows: > create table e011_01 (c1 integer, c2 smallint); > insert into e011_01 values (1, 1), (2, 2); > Compile-time problems: > You cannot use simple case expressions, searched case expressions or grouping > sets. These queries fail: > select width_bucket(5, c2, case c1 when 1 then c1 * 2 else c1 * 3 end, 10) > from e011_01; > select width_bucket(5, c2, case when c1 < 2 then c1 * 2 else c1 * 3 end, 10) > from e011_01; > select width_bucket(5, c2, max(c1)*10, cast(grouping(c1, c2)*20+1 as > integer)) from e011_02 group by cube(c1, c2); > I'll admit the grouping one is pretty contrived but the case ones seem > straightforward, valid, and it's strange that they don't work. Similar > queries work with other UDFs like sum. Why wouldn't they "just work"? Maybe > [~ashutoshc] can lend some perspective on that? > Interestingly, you can use window functions in width_bucket, example: > select width_bucket(rank() over (order by c2), 0, 10, 10) from e011_01; > works just fine. Hopefully we can get to a place where people implementing > functions like this don't need to think about value expression support but we > don't seem to be there yet. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16555) Add a new thrift API call for get_metastore_uuid
[ https://issues.apache.org/jira/browse/HIVE-16555?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vihang Karajgaonkar updated HIVE-16555: --- Status: Patch Available (was: Open) > Add a new thrift API call for get_metastore_uuid > > > Key: HIVE-16555 > URL: https://issues.apache.org/jira/browse/HIVE-16555 > Project: Hive > Issue Type: Sub-task > Components: Metastore >Reporter: Vihang Karajgaonkar >Assignee: Vihang Karajgaonkar > Attachments: HIVE-16555.01.patch > > > Sub-task of the main JIRA to add the new thrift API -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16534) Add capability to tell aborted transactions apart from open transactions in ValidTxnList
[
https://issues.apache.org/jira/browse/HIVE-16534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15991764#comment-15991764
]
Hive QA commented on HIVE-16534:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12865830/HIVE-16534.4.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/4970/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/4970/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-4970/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2017-05-01 23:20:05.963
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-4970/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2017-05-01 23:20:05.966
+ cd apache-github-source-source
+ git fetch origin
+ git reset --hard HEAD
HEAD is now at 2f79bd6 HIVE-16520: Cache hive metadata in metastore (Daniel
Dai, Vaibhav Gumashta, reviewed by Thejas Nair)
+ git clean -f -d
+ git checkout master
Already on 'master'
Your branch is up-to-date with 'origin/master'.
+ git reset --hard origin/master
HEAD is now at 2f79bd6 HIVE-16520: Cache hive metadata in metastore (Daniel
Dai, Vaibhav Gumashta, reviewed by Thejas Nair)
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2017-05-01 23:20:11.973
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh
/data/hiveptest/working/scratch/build.patch
error: patch failed:
metastore/src/gen/thrift/gen-cpp/ThriftHiveMetastore.cpp:6055
error: metastore/src/gen/thrift/gen-cpp/ThriftHiveMetastore.cpp: patch does not
apply
error: patch failed:
metastore/src/gen/thrift/gen-php/metastore/ThriftHiveMetastore.php:15489
error: metastore/src/gen/thrift/gen-php/metastore/ThriftHiveMetastore.php:
patch does not apply
error: patch failed:
metastore/src/java/org/apache/hadoop/hive/metastore/txn/TxnUtils.java:51
error: metastore/src/java/org/apache/hadoop/hive/metastore/txn/TxnUtils.java:
patch does not apply
The patch does not appear to apply with p0, p1, or p2
+ exit 1
'
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12865830 - PreCommit-HIVE-Build
> Add capability to tell aborted transactions apart from open transactions in
> ValidTxnList
>
>
> Key: HIVE-16534
> URL: https://issues.apache.org/jira/browse/HIVE-16534
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Reporter: Wei Zheng
>Assignee: Wei Zheng
> Attachments: HIVE-16534.1.patch, HIVE-16534.2.patch,
> HIVE-16534.3.patch, HIVE-16534.4.patch
>
>
> Currently in ValidReadTxnList, open transactions and aborted transactions are
> stored together in one array. That makes it impossible to extract just
> aborted transactions or open transactions.
> For ValidCompactorTxnList this is fine, since we only store aborted
> transactions but no open transactions.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-16513) width_bucket issues
[
https://issues.apache.org/jira/browse/HIVE-16513?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15991759#comment-15991759
]
Hive QA commented on HIVE-16513:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12865814/HIVE-16513.3.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 10631 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestAccumuloCliDriver.testCliDriver[accumulo_index]
(batchId=225)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/4969/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/4969/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-4969/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12865814 - PreCommit-HIVE-Build
> width_bucket issues
> ---
>
> Key: HIVE-16513
> URL: https://issues.apache.org/jira/browse/HIVE-16513
> Project: Hive
> Issue Type: Bug
>Reporter: Carter Shanklin
>Assignee: Sahil Takiar
> Attachments: HIVE-16513.1.patch, HIVE-16513.2.patch,
> HIVE-16513.3.patch
>
>
> width_bucket was recently added with HIVE-15982. This ticket notes a few
> issues.
> Usability issue:
> Currently only accepts integral numeric types. Decimals, floats and doubles
> are not supported.
> Runtime failures: This query will cause a runtime divide-by-zero in the
> reduce stage.
> select width_bucket(c1, 0, c1*2, 10) from e011_01 group by c1;
> The divide-by-zero seems to trigger any time I use a group-by. Here's another
> example (that actually requires the group-by):
> select width_bucket(c1, 0, max(c1), 10) from e011_01 group by c1;
> Advanced Usage Issues:
> Suppose you have a table e011_01 as follows:
> create table e011_01 (c1 integer, c2 smallint);
> insert into e011_01 values (1, 1), (2, 2);
> Compile-time problems:
> You cannot use simple case expressions, searched case expressions or grouping
> sets. These queries fail:
> select width_bucket(5, c2, case c1 when 1 then c1 * 2 else c1 * 3 end, 10)
> from e011_01;
> select width_bucket(5, c2, case when c1 < 2 then c1 * 2 else c1 * 3 end, 10)
> from e011_01;
> select width_bucket(5, c2, max(c1)*10, cast(grouping(c1, c2)*20+1 as
> integer)) from e011_02 group by cube(c1, c2);
> I'll admit the grouping one is pretty contrived but the case ones seem
> straightforward, valid, and it's strange that they don't work. Similar
> queries work with other UDFs like sum. Why wouldn't they "just work"? Maybe
> [~ashutoshc] can lend some perspective on that?
> Interestingly, you can use window functions in width_bucket, example:
> select width_bucket(rank() over (order by c2), 0, 10, 10) from e011_01;
> works just fine. Hopefully we can get to a place where people implementing
> functions like this don't need to think about value expression support but we
> don't seem to be there yet.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-16399) create an index for tc_txnid in TXN_COMPONENTS
[ https://issues.apache.org/jira/browse/HIVE-16399?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15991748#comment-15991748 ] Wei Zheng commented on HIVE-16399: -- Yeah I shouldn't have done that. But here're the deleted attachments I found from history: https://issues.apache.org/jira/secure/attachment/12863176/HIVE-16399.branch-2.3.patch https://issues.apache.org/jira/secure/attachment/12863177/HIVE-16399.branch-2.patch https://issues.apache.org/jira/secure/attachment/12863178/HIVE-16399.master.patch > create an index for tc_txnid in TXN_COMPONENTS > -- > > Key: HIVE-16399 > URL: https://issues.apache.org/jira/browse/HIVE-16399 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 1.0.0 >Reporter: Eugene Koifman >Assignee: Wei Zheng > Attachments: HIVE-16399.branch-2.3.patch, HIVE-16399.branch-2.patch, > HIVE-16399.master.patch > > > w/o this TxnStore.cleanEmptyAbortedTxns() can be very slow -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-15997) Resource leaks when query is cancelled
[
https://issues.apache.org/jira/browse/HIVE-15997?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15991745#comment-15991745
]
Yongzhi Chen commented on HIVE-15997:
-
657 if (lDrvState.driverState == DriverState.INTERRUPT) { 657
if (lDrvState.driverState == DriverState.INTERRUPT) {
658 Thread.currentThread().interrupt();
659 return true;658 return true;
Remove Thread.currentThread().interrupt(); // can solve some of the
resource leaks (depend on the cancel time), it let the query closed gracefully
instead of being interrupted during files clean up.
And
The fixes in zookeeper code can fix the lock leaks in my test case.
> Resource leaks when query is cancelled
> ---
>
> Key: HIVE-15997
> URL: https://issues.apache.org/jira/browse/HIVE-15997
> Project: Hive
> Issue Type: Bug
>Reporter: Yongzhi Chen
>Assignee: Yongzhi Chen
> Fix For: 2.2.0
>
> Attachments: HIVE-15997.1.patch
>
>
> There may some resource leaks when query is cancelled.
> We see following stacks in the log:
> Possible files and folder leak:
> {noformat}
> 2017-02-02 06:23:25,410 WARN hive.ql.Context: [HiveServer2-Background-Pool:
> Thread-61]: Error Removing Scratch: java.io.IOException: Failed on local
> exception: java.nio.channels.ClosedByInterruptException; Host Details : local
> host is: "ychencdh511t-1.vpc.cloudera.com/172.26.11.50"; destination host is:
> "ychencdh511t-1.vpc.cloudera.com":8020;
> at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:772)
> at org.apache.hadoop.ipc.Client.call(Client.java:1476)
> at org.apache.hadoop.ipc.Client.call(Client.java:1409)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:230)
>
> at com.sun.proxy.$Proxy25.delete(Unknown Source)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.delete(ClientNamenodeProtocolTranslatorPB.java:535)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:256)
>
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:104)
>
> at com.sun.proxy.$Proxy26.delete(Unknown Source)
> at org.apache.hadoop.hdfs.DFSClient.delete(DFSClient.java:2059)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$13.doCall(DistributedFileSystem.java:675)
>
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$13.doCall(DistributedFileSystem.java:671)
>
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
>
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.delete(DistributedFileSystem.java:671)
>
> at org.apache.hadoop.hive.ql.Context.removeScratchDir(Context.java:405)
> at org.apache.hadoop.hive.ql.Context.clear(Context.java:541)
> at org.apache.hadoop.hive.ql.Driver.releaseContext(Driver.java:2109)
> at org.apache.hadoop.hive.ql.Driver.closeInProcess(Driver.java:2150)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1472)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1212)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1207)
> at
> org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:237)
>
> at
> org.apache.hive.service.cli.operation.SQLOperation.access$300(SQLOperation.java:88)
>
> at
> org.apache.hive.service.cli.operation.SQLOperation$3$1.run(SQLOperation.java:293)
>
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1796)
>
> at
> org.apache.hive.service.cli.operation.SQLOperation$3.run(SQLOperation.java:306)
>
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.nio.channels.ClosedByInterruptException
> at
> java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:202)
>
> at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:681)
> at
> org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:192)
>
> at
[jira] [Commented] (HIVE-16399) create an index for tc_txnid in TXN_COMPONENTS
[ https://issues.apache.org/jira/browse/HIVE-16399?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15991731#comment-15991731 ] Eugene Koifman commented on HIVE-16399: --- since you delete the old files I can't diff the patches any more ... > create an index for tc_txnid in TXN_COMPONENTS > -- > > Key: HIVE-16399 > URL: https://issues.apache.org/jira/browse/HIVE-16399 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 1.0.0 >Reporter: Eugene Koifman >Assignee: Wei Zheng > Attachments: HIVE-16399.branch-2.3.patch, HIVE-16399.branch-2.patch, > HIVE-16399.master.patch > > > w/o this TxnStore.cleanEmptyAbortedTxns() can be very slow -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16399) create an index for tc_txnid in TXN_COMPONENTS
[ https://issues.apache.org/jira/browse/HIVE-16399?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15991704#comment-15991704 ] Wei Zheng commented on HIVE-16399: -- I reverted the changes to upgrade scripts 2.1.0-to-2.2.0, and put the changes in upgrade scripts 2.2.0-to-2.3.0. Also reverted changes to schema-2.2 scripts, and put the changes to schema-2.3 scripts. Basically branch-2.2 is irrelevant to this change. It will only be effective from branch-2.3 and above. > create an index for tc_txnid in TXN_COMPONENTS > -- > > Key: HIVE-16399 > URL: https://issues.apache.org/jira/browse/HIVE-16399 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 1.0.0 >Reporter: Eugene Koifman >Assignee: Wei Zheng > Attachments: HIVE-16399.branch-2.3.patch, HIVE-16399.branch-2.patch, > HIVE-16399.master.patch > > > w/o this TxnStore.cleanEmptyAbortedTxns() can be very slow -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16399) create an index for tc_txnid in TXN_COMPONENTS
[ https://issues.apache.org/jira/browse/HIVE-16399?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15991699#comment-15991699 ] Eugene Koifman commented on HIVE-16399: --- what exactly did you change? > create an index for tc_txnid in TXN_COMPONENTS > -- > > Key: HIVE-16399 > URL: https://issues.apache.org/jira/browse/HIVE-16399 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 1.0.0 >Reporter: Eugene Koifman >Assignee: Wei Zheng > Attachments: HIVE-16399.branch-2.3.patch, HIVE-16399.branch-2.patch, > HIVE-16399.master.patch > > > w/o this TxnStore.cleanEmptyAbortedTxns() can be very slow -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16399) create an index for tc_txnid in TXN_COMPONENTS
[ https://issues.apache.org/jira/browse/HIVE-16399?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Zheng updated HIVE-16399: - Status: Patch Available (was: Open) > create an index for tc_txnid in TXN_COMPONENTS > -- > > Key: HIVE-16399 > URL: https://issues.apache.org/jira/browse/HIVE-16399 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 1.0.0 >Reporter: Eugene Koifman >Assignee: Wei Zheng > Attachments: HIVE-16399.branch-2.3.patch, HIVE-16399.branch-2.patch, > HIVE-16399.master.patch > > > w/o this TxnStore.cleanEmptyAbortedTxns() can be very slow -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16399) create an index for tc_txnid in TXN_COMPONENTS
[ https://issues.apache.org/jira/browse/HIVE-16399?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15991696#comment-15991696 ] Wei Zheng commented on HIVE-16399: -- Re-attach patches for branch-2.3, branch-2 and master. At this moment we don't make any change to branch-2.2 as the details for that release is still on the fly. [~ekoifman] Can you take another look? > create an index for tc_txnid in TXN_COMPONENTS > -- > > Key: HIVE-16399 > URL: https://issues.apache.org/jira/browse/HIVE-16399 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 1.0.0 >Reporter: Eugene Koifman >Assignee: Wei Zheng > Attachments: HIVE-16399.branch-2.3.patch, HIVE-16399.branch-2.patch, > HIVE-16399.master.patch > > > w/o this TxnStore.cleanEmptyAbortedTxns() can be very slow -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16399) create an index for tc_txnid in TXN_COMPONENTS
[ https://issues.apache.org/jira/browse/HIVE-16399?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Zheng updated HIVE-16399: - Attachment: (was: HIVE-16399.master.patch) > create an index for tc_txnid in TXN_COMPONENTS > -- > > Key: HIVE-16399 > URL: https://issues.apache.org/jira/browse/HIVE-16399 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 1.0.0 >Reporter: Eugene Koifman >Assignee: Wei Zheng > Attachments: HIVE-16399.branch-2.3.patch, HIVE-16399.branch-2.patch, > HIVE-16399.master.patch > > > w/o this TxnStore.cleanEmptyAbortedTxns() can be very slow -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16399) create an index for tc_txnid in TXN_COMPONENTS
[ https://issues.apache.org/jira/browse/HIVE-16399?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Zheng updated HIVE-16399: - Attachment: (was: HIVE-16399.branch-2.3.patch) > create an index for tc_txnid in TXN_COMPONENTS > -- > > Key: HIVE-16399 > URL: https://issues.apache.org/jira/browse/HIVE-16399 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 1.0.0 >Reporter: Eugene Koifman >Assignee: Wei Zheng > Attachments: HIVE-16399.branch-2.3.patch, HIVE-16399.branch-2.patch, > HIVE-16399.master.patch > > > w/o this TxnStore.cleanEmptyAbortedTxns() can be very slow -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16399) create an index for tc_txnid in TXN_COMPONENTS
[ https://issues.apache.org/jira/browse/HIVE-16399?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Zheng updated HIVE-16399: - Attachment: HIVE-16399.master.patch HIVE-16399.branch-2.patch HIVE-16399.branch-2.3.patch > create an index for tc_txnid in TXN_COMPONENTS > -- > > Key: HIVE-16399 > URL: https://issues.apache.org/jira/browse/HIVE-16399 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 1.0.0 >Reporter: Eugene Koifman >Assignee: Wei Zheng > Attachments: HIVE-16399.branch-2.3.patch, HIVE-16399.branch-2.patch, > HIVE-16399.master.patch > > > w/o this TxnStore.cleanEmptyAbortedTxns() can be very slow -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16399) create an index for tc_txnid in TXN_COMPONENTS
[ https://issues.apache.org/jira/browse/HIVE-16399?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Zheng updated HIVE-16399: - Attachment: (was: HIVE-16399.branch-2.patch) > create an index for tc_txnid in TXN_COMPONENTS > -- > > Key: HIVE-16399 > URL: https://issues.apache.org/jira/browse/HIVE-16399 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 1.0.0 >Reporter: Eugene Koifman >Assignee: Wei Zheng > Attachments: HIVE-16399.branch-2.3.patch, HIVE-16399.branch-2.patch, > HIVE-16399.master.patch > > > w/o this TxnStore.cleanEmptyAbortedTxns() can be very slow -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-11133) Support hive.explain.user for Spark
[
https://issues.apache.org/jira/browse/HIVE-11133?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sahil Takiar updated HIVE-11133:
Attachment: HIVE-11133.9.patch
Rebased patch.
> Support hive.explain.user for Spark
> ---
>
> Key: HIVE-11133
> URL: https://issues.apache.org/jira/browse/HIVE-11133
> Project: Hive
> Issue Type: Sub-task
> Components: Spark
>Reporter: Mohit Sabharwal
>Assignee: Sahil Takiar
> Attachments: HIVE-11133.1.patch, HIVE-11133.2.patch,
> HIVE-11133.3.patch, HIVE-11133.4.patch, HIVE-11133.5.patch,
> HIVE-11133.6.patch, HIVE-11133.7.patch, HIVE-11133.8.patch, HIVE-11133.9.patch
>
>
> User friendly explain output ({{set hive.explain.user=true}}) should support
> Spark as well.
> Once supported, we should also enable related q-tests like {{explainuser_1.q}}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HIVE-16556) Modify schematool scripts to initialize and create METASTORE_DB_PROPERTIES table
[ https://issues.apache.org/jira/browse/HIVE-16556?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vihang Karajgaonkar updated HIVE-16556: --- Status: Patch Available (was: Open) Attaching the initial version of scripts to trigger the HiveQA > Modify schematool scripts to initialize and create METASTORE_DB_PROPERTIES > table > > > Key: HIVE-16556 > URL: https://issues.apache.org/jira/browse/HIVE-16556 > Project: Hive > Issue Type: Sub-task > Components: Metastore >Reporter: Vihang Karajgaonkar >Assignee: Vihang Karajgaonkar > Attachments: HIVE-16556.01.patch > > > sub-task to modify schema tool and its related changes so that the new table > is added to the schema when schematool initializes or upgrades the schema. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16563) Alter table partition set location should use fully qualified path for non-default FS
[
https://issues.apache.org/jira/browse/HIVE-16563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15991666#comment-15991666
]
Hive QA commented on HIVE-16563:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12865813/HIVE-16563.1.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 10632 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestAccumuloCliDriver.testCliDriver[accumulo_index]
(batchId=225)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_if_expr]
(batchId=143)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/4968/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/4968/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-4968/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12865813 - PreCommit-HIVE-Build
> Alter table partition set location should use fully qualified path for
> non-default FS
> -
>
> Key: HIVE-16563
> URL: https://issues.apache.org/jira/browse/HIVE-16563
> Project: Hive
> Issue Type: Bug
>Reporter: Chao Sun
>Assignee: Chao Sun
> Attachments: HIVE-16563.1.patch
>
>
> Similar to HIVE-6374, for command {{ALTER TABLE .. PARTITION(..) SET LOCATION
> ..}}, if location path is not a fully qualified path and Hive is not using
> default namenode, it should use fully qualified path for the partition.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HIVE-16556) Modify schematool scripts to initialize and create METASTORE_DB_PROPERTIES table
[ https://issues.apache.org/jira/browse/HIVE-16556?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vihang Karajgaonkar updated HIVE-16556: --- Attachment: HIVE-16556.01.patch > Modify schematool scripts to initialize and create METASTORE_DB_PROPERTIES > table > > > Key: HIVE-16556 > URL: https://issues.apache.org/jira/browse/HIVE-16556 > Project: Hive > Issue Type: Sub-task > Components: Metastore >Reporter: Vihang Karajgaonkar >Assignee: Vihang Karajgaonkar > Attachments: HIVE-16556.01.patch > > > sub-task to modify schema tool and its related changes so that the new table > is added to the schema when schematool initializes or upgrades the schema. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16534) Add capability to tell aborted transactions apart from open transactions in ValidTxnList
[ https://issues.apache.org/jira/browse/HIVE-16534?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Zheng updated HIVE-16534: - Attachment: HIVE-16534.4.patch Added the comment for ValidTxnList.readFromString > Add capability to tell aborted transactions apart from open transactions in > ValidTxnList > > > Key: HIVE-16534 > URL: https://issues.apache.org/jira/browse/HIVE-16534 > Project: Hive > Issue Type: Bug > Components: Transactions >Reporter: Wei Zheng >Assignee: Wei Zheng > Attachments: HIVE-16534.1.patch, HIVE-16534.2.patch, > HIVE-16534.3.patch, HIVE-16534.4.patch > > > Currently in ValidReadTxnList, open transactions and aborted transactions are > stored together in one array. That makes it impossible to extract just > aborted transactions or open transactions. > For ValidCompactorTxnList this is fine, since we only store aborted > transactions but no open transactions. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16534) Add capability to tell aborted transactions apart from open transactions in ValidTxnList
[ https://issues.apache.org/jira/browse/HIVE-16534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15991638#comment-15991638 ] Eugene Koifman commented on HIVE-16534: --- +1 patch 3 pending tests Could you add a comment that the exception list is excepted the sorted when passed in ValidReadTxnList > Add capability to tell aborted transactions apart from open transactions in > ValidTxnList > > > Key: HIVE-16534 > URL: https://issues.apache.org/jira/browse/HIVE-16534 > Project: Hive > Issue Type: Bug > Components: Transactions >Reporter: Wei Zheng >Assignee: Wei Zheng > Attachments: HIVE-16534.1.patch, HIVE-16534.2.patch, > HIVE-16534.3.patch > > > Currently in ValidReadTxnList, open transactions and aborted transactions are > stored together in one array. That makes it impossible to extract just > aborted transactions or open transactions. > For ValidCompactorTxnList this is fine, since we only store aborted > transactions but no open transactions. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16565) Improve how the open transactions and aborted transactions are deserialized in ValidReadTxnList.readFromString
[ https://issues.apache.org/jira/browse/HIVE-16565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Zheng updated HIVE-16565: - Description: This is a follow-up of HIVE-16534. In ValidReadTxnList.writeToString, we write out open and aborted transactions as two sorted lists. We can take advantage of that and merge sort them together when reading them back in readFromString. Note that the aborted bits should also be marked properly during the merge sort. was: This is a follow-up of HIVE-16534. In ValidReadTxnList.writeToString, we write out open and aborted transactions as two sorted lists. We can take advantage of that and perform merge sort them together when reading them back in readFromString. Note that the aborted bits should also be handled properly during the merge sort. > Improve how the open transactions and aborted transactions are deserialized > in ValidReadTxnList.readFromString > -- > > Key: HIVE-16565 > URL: https://issues.apache.org/jira/browse/HIVE-16565 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 3.0.0 >Reporter: Wei Zheng >Assignee: Wei Zheng > > This is a follow-up of HIVE-16534. > In ValidReadTxnList.writeToString, we write out open and aborted transactions > as two sorted lists. We can take advantage of that and merge sort them > together when reading them back in readFromString. Note that the aborted bits > should also be marked properly during the merge sort. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16565) Improve how the open transactions and aborted transactions are deserialized in ValidReadTxnList.readFromString
[ https://issues.apache.org/jira/browse/HIVE-16565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Zheng updated HIVE-16565: - Description: This is a follow-up of HIVE-16534. In ValidReadTxnList.writeToString, we write out open and aborted transactions as two sorted lists. We can take advantage of that and perform merge sort them together when reading them back in readFromString. Note that the aborted bits should also be handled properly during the merge sort. was: This is a follow-up of HIVE-16534. In ValidReadTxnList.writeToString, we write out two open and aborted transactions as two sorted lists. We can take advantage of that and perform merge sort them together when reading them back in readFromString. Note that the aborted bits should also be handled properly during the merge sort. > Improve how the open transactions and aborted transactions are deserialized > in ValidReadTxnList.readFromString > -- > > Key: HIVE-16565 > URL: https://issues.apache.org/jira/browse/HIVE-16565 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 3.0.0 >Reporter: Wei Zheng >Assignee: Wei Zheng > > This is a follow-up of HIVE-16534. > In ValidReadTxnList.writeToString, we write out open and aborted transactions > as two sorted lists. We can take advantage of that and perform merge sort > them together when reading them back in readFromString. Note that the aborted > bits should also be handled properly during the merge sort. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Assigned] (HIVE-16565) Improve how the open transactions and aborted transactions are deserialized in ValidReadTxnList.readFromString
[ https://issues.apache.org/jira/browse/HIVE-16565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Zheng reassigned HIVE-16565: > Improve how the open transactions and aborted transactions are deserialized > in ValidReadTxnList.readFromString > -- > > Key: HIVE-16565 > URL: https://issues.apache.org/jira/browse/HIVE-16565 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 3.0.0 >Reporter: Wei Zheng >Assignee: Wei Zheng > > This is a follow-up of HIVE-16534. > In ValidReadTxnList.writeToString, we write out two open and aborted > transactions as two sorted lists. We can take advantage of that and perform > merge sort them together when reading them back in readFromString. Note that > the aborted bits should also be handled properly during the merge sort. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16534) Add capability to tell aborted transactions apart from open transactions in ValidTxnList
[ https://issues.apache.org/jira/browse/HIVE-16534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15991589#comment-15991589 ] Wei Zheng commented on HIVE-16534: -- patch 3 removes the binary search logic in isTxnRangeAborted as commented above. > Add capability to tell aborted transactions apart from open transactions in > ValidTxnList > > > Key: HIVE-16534 > URL: https://issues.apache.org/jira/browse/HIVE-16534 > Project: Hive > Issue Type: Bug > Components: Transactions >Reporter: Wei Zheng >Assignee: Wei Zheng > Attachments: HIVE-16534.1.patch, HIVE-16534.2.patch, > HIVE-16534.3.patch > > > Currently in ValidReadTxnList, open transactions and aborted transactions are > stored together in one array. That makes it impossible to extract just > aborted transactions or open transactions. > For ValidCompactorTxnList this is fine, since we only store aborted > transactions but no open transactions. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16143) Improve msck repair batching
[
https://issues.apache.org/jira/browse/HIVE-16143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15991584#comment-15991584
]
Hive QA commented on HIVE-16143:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12865799/HIVE-16143.04.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 4 failed/errored test(s), 10640 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestAccumuloCliDriver.testCliDriver[accumulo_index]
(batchId=225)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[create_like]
(batchId=237)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[msck_repair_batchsize]
(batchId=64)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_if_expr]
(batchId=143)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/4966/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/4966/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-4966/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 4 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12865799 - PreCommit-HIVE-Build
> Improve msck repair batching
>
>
> Key: HIVE-16143
> URL: https://issues.apache.org/jira/browse/HIVE-16143
> Project: Hive
> Issue Type: Improvement
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
> Attachments: HIVE-16143.01.patch, HIVE-16143.02.patch,
> HIVE-16143.03.patch, HIVE-16143.04.patch
>
>
> Currently, the {{msck repair table}} command batches the number of partitions
> created in the metastore using the config {{HIVE_MSCK_REPAIR_BATCH_SIZE}}.
> Following snippet shows the batching logic. There can be couple of
> improvements to this batching logic:
> {noformat}
> int batch_size = conf.getIntVar(ConfVars.HIVE_MSCK_REPAIR_BATCH_SIZE);
> if (batch_size > 0 && partsNotInMs.size() > batch_size) {
> int counter = 0;
> for (CheckResult.PartitionResult part : partsNotInMs) {
> counter++;
>
> apd.addPartition(Warehouse.makeSpecFromName(part.getPartitionName()), null);
> repairOutput.add("Repair: Added partition to metastore " +
> msckDesc.getTableName()
> + ':' + part.getPartitionName());
> if (counter % batch_size == 0 || counter ==
> partsNotInMs.size()) {
> db.createPartitions(apd);
> apd = new AddPartitionDesc(table.getDbName(),
> table.getTableName(), false);
> }
> }
> } else {
> for (CheckResult.PartitionResult part : partsNotInMs) {
>
> apd.addPartition(Warehouse.makeSpecFromName(part.getPartitionName()), null);
> repairOutput.add("Repair: Added partition to metastore " +
> msckDesc.getTableName()
> + ':' + part.getPartitionName());
> }
> db.createPartitions(apd);
> }
> } catch (Exception e) {
> LOG.info("Could not bulk-add partitions to metastore; trying one by
> one", e);
> repairOutput.clear();
> msckAddPartitionsOneByOne(db, table, partsNotInMs, repairOutput);
> }
> {noformat}
> 1. If the batch size is too aggressive the code falls back to adding
> partitions one by one which is almost always very slow. It is easily possible
> that users increase the batch size to higher value to make the command run
> faster but end up with a worse performance because code falls back to adding
> one by one. Users are then expected to determine the tuned value of batch
> size which works well for their environment. I think the code could handle
> this situation better by exponentially decaying the batch size instead of
> falling back to one by one.
> 2. The other issue with this implementation is if lets say first batch
> succeeds and the second one fails, the code tries to add all the partitions
> one by one irrespective of whether some of the were successfully added or
> not. If we need to fall back to one by one we should atleast remove the ones
> which we know for sure are already added successfully.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HIVE-16534) Add capability to tell aborted transactions apart from open transactions in ValidTxnList
[ https://issues.apache.org/jira/browse/HIVE-16534?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Zheng updated HIVE-16534: - Attachment: HIVE-16534.3.patch > Add capability to tell aborted transactions apart from open transactions in > ValidTxnList > > > Key: HIVE-16534 > URL: https://issues.apache.org/jira/browse/HIVE-16534 > Project: Hive > Issue Type: Bug > Components: Transactions >Reporter: Wei Zheng >Assignee: Wei Zheng > Attachments: HIVE-16534.1.patch, HIVE-16534.2.patch, > HIVE-16534.3.patch > > > Currently in ValidReadTxnList, open transactions and aborted transactions are > stored together in one array. That makes it impossible to extract just > aborted transactions or open transactions. > For ValidCompactorTxnList this is fine, since we only store aborted > transactions but no open transactions. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
