[jira] [Commented] (HIVE-16370) Avro data type null not supported on partitioned tables
[
https://issues.apache.org/jira/browse/HIVE-16370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16603942#comment-16603942
]
Alice Fan commented on HIVE-16370:
--
Hi [~iamwrong], Do you still work on this one? Do you mind I can take this?
Thanks.
> Avro data type null not supported on partitioned tables
> ---
>
> Key: HIVE-16370
> URL: https://issues.apache.org/jira/browse/HIVE-16370
> Project: Hive
> Issue Type: Bug
>Affects Versions: 1.1.0, 2.1.1
>Reporter: rui miranda
>Assignee: Alice Fan
>Priority: Minor
>
> I was attempting to create hive tables over some partitioned Avro files. It
> seems the void data type (Avro null) is not supported on partitioned tables
> (i could not replicate the bug on an un-partitioned table).
> ---
> i managed to replicate the bug on two different hive versions.
> Hive 1.1.0-cdh5.10.0
> Hive 2.1.1-amzn-0
>
> how to replicate (avro tools are required to create the avro files):
> $ wget
> http://mirror.serversupportforum.de/apache/avro/avro-1.8.1/java/avro-tools-1.8.1.jar
> $ mkdir /tmp/avro
> $ mkdir /tmp/avro/null
> $ echo "{ \
> \"type\" : \"record\", \
> \"name\" : \"null_failure\", \
> \"namespace\" : \"org.apache.avro.null_failure\", \
> \"doc\":\"the purpose of this schema is to replicate the hive avro null
> failure\", \
> \"fields\" : [{\"name\":\"one\", \"type\":\"null\",\"default\":null}] \
> } " > /tmp/avro/null/schema.avsc
> $ echo "{\"one\":null}" > /tmp/avro/null/data.json
> $ java -jar avro-tools-1.8.1.jar fromjson --schema-file
> /tmp/avro/null/schema.avsc /tmp/avro/null/data.json > /tmp/avro/null/data.avro
> $ hdfs dfs -mkdir /tmp/avro
> $ hdfs dfs -mkdir /tmp/avro/null
> $ hdfs dfs -mkdir /tmp/avro/null/schema
> $ hdfs dfs -mkdir /tmp/avro/null/data
> $ hdfs dfs -mkdir /tmp/avro/null/data/foo=bar
> $ hdfs dfs -copyFromLocal /tmp/avro/null/schema.avsc
> /tmp/avro/null/schema/schema.avsc
> $ hdfs dfs -copyFromLocal /tmp/avro/null/data.avro
> /tmp/avro/null/data/foo=bar/data.avro
> $ hive
> hive> CREATE EXTERNAL TABLE avro_null
> PARTITIONED BY (foo string)
> ROW FORMAT SERDE
> 'org.apache.hadoop.hive.serde2.avro.AvroSerDe'
> STORED as INPUTFORMAT
> 'org.apache.hadoop.hive.ql.io.avro.AvroContainerInputFormat'
> OUTPUTFORMAT
> 'org.apache.hadoop.hive.ql.io.avro.AvroContainerOutputFormat'
> LOCATION
> '/tmp/avro/null/data/'
> TBLPROPERTIES (
> 'avro.schema.url'='/tmp/avro/null/schema/schema.avsc')
> ;
> OK
> Time taken: 3.127 seconds
> hive> msck repair table avro_null;
> OK
> Partitions not in metastore: avro_null:foo=bar
> Repair: Added partition to metastore avro_null:foo=bar
> Time taken: 0.712 seconds, Fetched: 2 row(s)
> hive> select * from avro_null;
> FAILED: RuntimeException org.apache.hadoop.hive.ql.metadata.HiveException:
> Failed with exception Hive internal error inside
> isAssignableFromSettablePrimitiveOI void not supported
> yet.java.lang.RuntimeException: Hive internal error inside
> isAssignableFromSettablePrimitiveOI void not supported yet.
> hive> select foo, count(1) from avro_null group by foo;
> OK
> bar 1
> Time taken: 29.806 seconds, Fetched: 1 row(s)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (HIVE-16370) Avro data type null not supported on partitioned tables
[
https://issues.apache.org/jira/browse/HIVE-16370?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alice Fan reassigned HIVE-16370:
Assignee: Alice Fan
> Avro data type null not supported on partitioned tables
> ---
>
> Key: HIVE-16370
> URL: https://issues.apache.org/jira/browse/HIVE-16370
> Project: Hive
> Issue Type: Bug
>Affects Versions: 1.1.0, 2.1.1
>Reporter: rui miranda
>Assignee: Alice Fan
>Priority: Minor
>
> I was attempting to create hive tables over some partitioned Avro files. It
> seems the void data type (Avro null) is not supported on partitioned tables
> (i could not replicate the bug on an un-partitioned table).
> ---
> i managed to replicate the bug on two different hive versions.
> Hive 1.1.0-cdh5.10.0
> Hive 2.1.1-amzn-0
>
> how to replicate (avro tools are required to create the avro files):
> $ wget
> http://mirror.serversupportforum.de/apache/avro/avro-1.8.1/java/avro-tools-1.8.1.jar
> $ mkdir /tmp/avro
> $ mkdir /tmp/avro/null
> $ echo "{ \
> \"type\" : \"record\", \
> \"name\" : \"null_failure\", \
> \"namespace\" : \"org.apache.avro.null_failure\", \
> \"doc\":\"the purpose of this schema is to replicate the hive avro null
> failure\", \
> \"fields\" : [{\"name\":\"one\", \"type\":\"null\",\"default\":null}] \
> } " > /tmp/avro/null/schema.avsc
> $ echo "{\"one\":null}" > /tmp/avro/null/data.json
> $ java -jar avro-tools-1.8.1.jar fromjson --schema-file
> /tmp/avro/null/schema.avsc /tmp/avro/null/data.json > /tmp/avro/null/data.avro
> $ hdfs dfs -mkdir /tmp/avro
> $ hdfs dfs -mkdir /tmp/avro/null
> $ hdfs dfs -mkdir /tmp/avro/null/schema
> $ hdfs dfs -mkdir /tmp/avro/null/data
> $ hdfs dfs -mkdir /tmp/avro/null/data/foo=bar
> $ hdfs dfs -copyFromLocal /tmp/avro/null/schema.avsc
> /tmp/avro/null/schema/schema.avsc
> $ hdfs dfs -copyFromLocal /tmp/avro/null/data.avro
> /tmp/avro/null/data/foo=bar/data.avro
> $ hive
> hive> CREATE EXTERNAL TABLE avro_null
> PARTITIONED BY (foo string)
> ROW FORMAT SERDE
> 'org.apache.hadoop.hive.serde2.avro.AvroSerDe'
> STORED as INPUTFORMAT
> 'org.apache.hadoop.hive.ql.io.avro.AvroContainerInputFormat'
> OUTPUTFORMAT
> 'org.apache.hadoop.hive.ql.io.avro.AvroContainerOutputFormat'
> LOCATION
> '/tmp/avro/null/data/'
> TBLPROPERTIES (
> 'avro.schema.url'='/tmp/avro/null/schema/schema.avsc')
> ;
> OK
> Time taken: 3.127 seconds
> hive> msck repair table avro_null;
> OK
> Partitions not in metastore: avro_null:foo=bar
> Repair: Added partition to metastore avro_null:foo=bar
> Time taken: 0.712 seconds, Fetched: 2 row(s)
> hive> select * from avro_null;
> FAILED: RuntimeException org.apache.hadoop.hive.ql.metadata.HiveException:
> Failed with exception Hive internal error inside
> isAssignableFromSettablePrimitiveOI void not supported
> yet.java.lang.RuntimeException: Hive internal error inside
> isAssignableFromSettablePrimitiveOI void not supported yet.
> hive> select foo, count(1) from avro_null group by foo;
> OK
> bar 1
> Time taken: 29.806 seconds, Fetched: 1 row(s)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20432) Rewrite BETWEEN to IN for integer types for stats estimation
[
https://issues.apache.org/jira/browse/HIVE-20432?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16603938#comment-16603938
]
Ashutosh Chauhan commented on HIVE-20432:
-
1. Instead of if (leftValue instanceof Integer) {..} else if (leftValue
instanceof Long) {...} else {} you may simply do
long leftVal = ((Number) leftValue).longValue() concise and simpler to read.
2. In isIntegerType() you also need to add || primitiveCategory ==
PrimitiveCategory.BYTE in if () for tinyint.
+1 pending these changes.
> Rewrite BETWEEN to IN for integer types for stats estimation
>
>
> Key: HIVE-20432
> URL: https://issues.apache.org/jira/browse/HIVE-20432
> Project: Hive
> Issue Type: Improvement
>Affects Versions: 4.0.0
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
> Attachments: HIVE-20432.1.patch, HIVE-20432.2.patch,
> HIVE-20432.3.patch
>
>
> IN clause provides better statistics estimation since predicates are equality
> based and it keeps into account selectivity of all values. BETWEEN is
> rewritten into {{>= AND <=}} i.e. non-equality predicates which cause worse
> estimation. For integer types BETWEEN could be rewritten into IN e.g. BETWEEN
> 4 and 7 is equivalent to IN (4,5,6,7).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18873) Skipping predicate pushdown for MR silently at HiveInputFormat can cause storage handlers to produce erroneous result
[
https://issues.apache.org/jira/browse/HIVE-18873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16603936#comment-16603936
]
Hive QA commented on HIVE-18873:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
38s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
53s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
36s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
54s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
45s{color} | {color:blue} itests/util in master has 52 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
3s{color} | {color:blue} ql in master has 2310 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
20s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
8s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
14s{color} | {color:green} The patch util passed checkstyle {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
41s{color} | {color:green} ql: The patch generated 0 new + 20 unchanged - 1
fixed = 20 total (was 21) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
19s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 29m 2s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-13594/dev-support/hive-personality.sh
|
| git revision | master / 33fa62f |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: itests/util ql U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-13594/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Skipping predicate pushdown for MR silently at HiveInputFormat can cause
> storage handlers to produce erroneous result
> -
>
> Key: HIVE-18873
> URL: https://issues.apache.org/jira/browse/HIVE-18873
> Project: Hive
> Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Ankit Singhal
>Assignee: Ankit Singhal
>Priority: Major
> Attachments: HIVE-18873.2.patch, HIVE-18873.2_reattach.patch,
> HIVE-18873.3.patch, HIVE-18873.patch
>
>
> {code:java}
> // disable filter pushdown for mapreduce when there are more than one table
> aliases,
> // since we don't clone jobConf per alias
> if (mrwork != null && mrwork.getAliases() != null &&
> mrwork.getAliases().size() > 1 &&
> jobConf.get(ConfVars.HIVE_EXECUTION_ENGINE.varname).equals("mr")) {
> return;
> }
> {code}
> I believe this needs to be
[jira] [Updated] (HIVE-20377) Hive Kafka Storage Handler
[
https://issues.apache.org/jira/browse/HIVE-20377?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gopal V updated HIVE-20377:
---
Resolution: Fixed
Status: Resolved (was: Patch Available)
Pushed to master, thanks [~bslim]!
> Hive Kafka Storage Handler
> --
>
> Key: HIVE-20377
> URL: https://issues.apache.org/jira/browse/HIVE-20377
> Project: Hive
> Issue Type: New Feature
>Affects Versions: 4.0.0
>Reporter: slim bouguerra
>Assignee: slim bouguerra
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-20377.10.patch, HIVE-20377.11.patch,
> HIVE-20377.12.patch, HIVE-20377.15.patch, HIVE-20377.18.patch,
> HIVE-20377.18.patch, HIVE-20377.19.patch, HIVE-20377.19.patch,
> HIVE-20377.19.patch, HIVE-20377.4.patch, HIVE-20377.5.patch,
> HIVE-20377.6.patch, HIVE-20377.8.patch, HIVE-20377.8.patch, HIVE-20377.patch
>
>
> h1. Goal
> * Read streaming data form Kafka queue as an external table.
> * Allow streaming navigation by pushing down filters on Kafka record
> partition id, offset and timestamp.
> * Insert streaming data form Kafka to an actual Hive internal table, using
> CTAS statement.
> h1. Example
> h2. Create the external table
> {code}
> CREATE EXTERNAL TABLE kafka_table (`timestamp` timestamp, page string, `user`
> string, language string, added int, deleted int, flags string,comment string,
> namespace string)
> STORED BY 'org.apache.hadoop.hive.kafka.KafkaStorageHandler'
> TBLPROPERTIES
> ("kafka.topic" = "wikipedia",
> "kafka.bootstrap.servers"="brokeraddress:9092",
> "kafka.serde.class"="org.apache.hadoop.hive.serde2.JsonSerDe");
> {code}
> h2. Kafka Metadata
> In order to keep track of Kafka records the storage handler will add
> automatically the Kafka row metadata eg partition id, record offset and
> record timestamp.
> {code}
> DESCRIBE EXTENDED kafka_table
> timestamp timestamp from deserializer
> page string from deserializer
> user string from deserializer
> language string from deserializer
> country string from deserializer
> continent string from deserializer
> namespace string from deserializer
> newpage boolean from deserializer
> unpatrolled boolean from deserializer
> anonymous boolean from deserializer
> robot boolean from deserializer
> added int from deserializer
> deleted int from deserializer
> delta bigint from deserializer
> __partition int from deserializer
> __offset bigint from deserializer
> __timestamp bigint from deserializer
> {code}
> h2. Filter push down.
> Newer Kafka consumers 0.11.0 and higher allow seeking on the stream based on
> a given offset. The proposed storage handler will be able to leverage such
> API by pushing down filters over metadata columns, namely __partition (int),
> __offset(long) and __timestamp(long)
> For instance Query like
> {code}
> select `__offset` from kafka_table where (`__offset` < 10 and `__offset`>3
> and `__partition` = 0) or (`__partition` = 0 and `__offset` < 105 and
> `__offset` > 99) or (`__offset` = 109);
> {code}
> Will result on a scan of partition 0 only then read only records between
> offset 4 and 109.
> h2. With timestamp seeks
> The seeking based on the internal timestamps allows the handler to run on
> recently arrived data, by doing
> {code}
> select count(*) from kafka_table where `__timestamp` > 1000 *
> to_unix_timestamp(CURRENT_TIMESTAMP - interval '20' hours) ;
> {code}
> This allows for implicit relationships between event timestamps and kafka
> timestamps to be expressed in queries (i.e event_timestamp is always < than
> kafka __timestamp and kafka __timestamp is never > 15 minutes from event etc).
> h2. More examples with Avro
> {code}
> CREATE EXTERNAL TABLE wiki_kafka_avro_table
> STORED BY 'org.apache.hadoop.hive.kafka.KafkaStorageHandler'
> TBLPROPERTIES
> ("kafka.topic" = "wiki_kafka_avro_table",
> "kafka.bootstrap.servers"="localhost:9092",
> "kafka.serde.class"="org.apache.hadoop.hive.serde2.avro.AvroSerDe",
> 'avro.schema.literal'='{
> "type" : "record",
> "name" : "Wikipedia",
> "namespace" : "org.apache.hive.kafka",
> "version": "1",
> "fields" : [ {
> "name" : "isrobot",
> "type" : "boolean"
> }, {
> "name" : "channel",
> "type" : "string"
> }, {
> "name" : "timestamp",
[jira] [Updated] (HIVE-20377) Hive Kafka Storage Handler
[
https://issues.apache.org/jira/browse/HIVE-20377?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gopal V updated HIVE-20377:
---
Fix Version/s: 4.0.0
> Hive Kafka Storage Handler
> --
>
> Key: HIVE-20377
> URL: https://issues.apache.org/jira/browse/HIVE-20377
> Project: Hive
> Issue Type: New Feature
>Affects Versions: 4.0.0
>Reporter: slim bouguerra
>Assignee: slim bouguerra
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-20377.10.patch, HIVE-20377.11.patch,
> HIVE-20377.12.patch, HIVE-20377.15.patch, HIVE-20377.18.patch,
> HIVE-20377.18.patch, HIVE-20377.19.patch, HIVE-20377.19.patch,
> HIVE-20377.19.patch, HIVE-20377.4.patch, HIVE-20377.5.patch,
> HIVE-20377.6.patch, HIVE-20377.8.patch, HIVE-20377.8.patch, HIVE-20377.patch
>
>
> h1. Goal
> * Read streaming data form Kafka queue as an external table.
> * Allow streaming navigation by pushing down filters on Kafka record
> partition id, offset and timestamp.
> * Insert streaming data form Kafka to an actual Hive internal table, using
> CTAS statement.
> h1. Example
> h2. Create the external table
> {code}
> CREATE EXTERNAL TABLE kafka_table (`timestamp` timestamp, page string, `user`
> string, language string, added int, deleted int, flags string,comment string,
> namespace string)
> STORED BY 'org.apache.hadoop.hive.kafka.KafkaStorageHandler'
> TBLPROPERTIES
> ("kafka.topic" = "wikipedia",
> "kafka.bootstrap.servers"="brokeraddress:9092",
> "kafka.serde.class"="org.apache.hadoop.hive.serde2.JsonSerDe");
> {code}
> h2. Kafka Metadata
> In order to keep track of Kafka records the storage handler will add
> automatically the Kafka row metadata eg partition id, record offset and
> record timestamp.
> {code}
> DESCRIBE EXTENDED kafka_table
> timestamp timestamp from deserializer
> page string from deserializer
> user string from deserializer
> language string from deserializer
> country string from deserializer
> continent string from deserializer
> namespace string from deserializer
> newpage boolean from deserializer
> unpatrolled boolean from deserializer
> anonymous boolean from deserializer
> robot boolean from deserializer
> added int from deserializer
> deleted int from deserializer
> delta bigint from deserializer
> __partition int from deserializer
> __offset bigint from deserializer
> __timestamp bigint from deserializer
> {code}
> h2. Filter push down.
> Newer Kafka consumers 0.11.0 and higher allow seeking on the stream based on
> a given offset. The proposed storage handler will be able to leverage such
> API by pushing down filters over metadata columns, namely __partition (int),
> __offset(long) and __timestamp(long)
> For instance Query like
> {code}
> select `__offset` from kafka_table where (`__offset` < 10 and `__offset`>3
> and `__partition` = 0) or (`__partition` = 0 and `__offset` < 105 and
> `__offset` > 99) or (`__offset` = 109);
> {code}
> Will result on a scan of partition 0 only then read only records between
> offset 4 and 109.
> h2. With timestamp seeks
> The seeking based on the internal timestamps allows the handler to run on
> recently arrived data, by doing
> {code}
> select count(*) from kafka_table where `__timestamp` > 1000 *
> to_unix_timestamp(CURRENT_TIMESTAMP - interval '20' hours) ;
> {code}
> This allows for implicit relationships between event timestamps and kafka
> timestamps to be expressed in queries (i.e event_timestamp is always < than
> kafka __timestamp and kafka __timestamp is never > 15 minutes from event etc).
> h2. More examples with Avro
> {code}
> CREATE EXTERNAL TABLE wiki_kafka_avro_table
> STORED BY 'org.apache.hadoop.hive.kafka.KafkaStorageHandler'
> TBLPROPERTIES
> ("kafka.topic" = "wiki_kafka_avro_table",
> "kafka.bootstrap.servers"="localhost:9092",
> "kafka.serde.class"="org.apache.hadoop.hive.serde2.avro.AvroSerDe",
> 'avro.schema.literal'='{
> "type" : "record",
> "name" : "Wikipedia",
> "namespace" : "org.apache.hive.kafka",
> "version": "1",
> "fields" : [ {
> "name" : "isrobot",
> "type" : "boolean"
> }, {
> "name" : "channel",
> "type" : "string"
> }, {
> "name" : "timestamp",
> "type" : "string"
> }, {
> "name" : "flags",
> "type" : "string"
>
[jira] [Commented] (HIVE-17684) HoS memory issues with MapJoinMemoryExhaustionHandler
[
https://issues.apache.org/jira/browse/HIVE-17684?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16603896#comment-16603896
]
Hive QA commented on HIVE-17684:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12938303/HIVE-17684.05.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 34 failed/errored test(s), 14924 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_mapjoin]
(batchId=11)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[autoColumnStats_1]
(batchId=23)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join10] (batchId=37)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join14] (batchId=15)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join15] (batchId=16)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join25] (batchId=77)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join26] (batchId=14)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join33] (batchId=12)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join_without_localtask]
(batchId=1)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[bucketcontext_1]
(batchId=34)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[bucketmapjoin12]
(batchId=37)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[bucketmapjoin8]
(batchId=13)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[bucketmapjoin9]
(batchId=17)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[convert_decimal64_to_decimal]
(batchId=52)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[correlationoptimizer7]
(batchId=22)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[infer_bucket_sort_convert_join]
(batchId=57)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[join33] (batchId=16)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[join_empty] (batchId=37)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[mapjoin_hook]
(batchId=13)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[nonblock_op_deduplicate]
(batchId=23)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[skewjoin_mapjoin10]
(batchId=35)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[union22] (batchId=14)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[union34] (batchId=13)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[union_remove_14]
(batchId=13)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_char_mapjoin1]
(batchId=34)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_include_no_sel]
(batchId=4)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_left_outer_join]
(batchId=23)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_outer_join2]
(batchId=32)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_outer_join3]
(batchId=35)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_reduce_groupby_duplicate_cols]
(batchId=37)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vectorized_context]
(batchId=34)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vectorized_mapjoin3]
(batchId=12)
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver[hbase_ppd_join]
(batchId=103)
org.apache.hadoop.hive.cli.TestSparkPerfCliDriver.testCliDriver[query39]
(batchId=266)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/13593/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/13593/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-13593/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 34 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12938303 - PreCommit-HIVE-Build
> HoS memory issues with MapJoinMemoryExhaustionHandler
> -
>
> Key: HIVE-17684
> URL: https://issues.apache.org/jira/browse/HIVE-17684
> Project: Hive
> Issue Type: Bug
> Components: Spark
>Reporter: Sahil Takiar
>Assignee: Misha Dmitriev
>Priority: Major
> Attachments: HIVE-17684.01.patch, HIVE-17684.02.patch,
> HIVE-17684.03.patch, HIVE-17684.04.patch, HIVE-17684.05.patch
>
>
> We have seen a number of memory issues due the {{HashSinkOperator}} use of
> the {{MapJoinMemoryExhaustionHandler}}. This handler is meant to detect
> scenarios where the small table is taking too much space in memory, in which
> case a
[jira] [Commented] (HIVE-17684) HoS memory issues with MapJoinMemoryExhaustionHandler
[
https://issues.apache.org/jira/browse/HIVE-17684?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16603897#comment-16603897
]
Hive QA commented on HIVE-17684:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
2s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 10m
39s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m
48s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
3s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
13s{color} | {color:blue} ql in master has 2310 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 7m
57s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
8s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 7m
28s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
41s{color} | {color:red} ql: The patch generated 1 new + 31 unchanged - 0 fixed
= 32 total (was 31) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 4m
23s{color} | {color:red} ql generated 4 new + 2309 unchanged - 1 fixed = 2313
total (was 2310) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 8m
9s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 64m 15s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:ql |
| | Incorrect lazy initialization and update of static field
org.apache.hadoop.hive.ql.exec.Operator.hiveGcTimeMonitor in
org.apache.hadoop.hive.ql.exec.Operator.initialize(Configuration,
ObjectInspector[]) At Operator.java:of static field
org.apache.hadoop.hive.ql.exec.Operator.hiveGcTimeMonitor in
org.apache.hadoop.hive.ql.exec.Operator.initialize(Configuration,
ObjectInspector[]) At Operator.java:[lines 430-434] |
| | Write to static field
org.apache.hadoop.hive.ql.exec.Operator.criticalGcTimePercentage from instance
method org.apache.hadoop.hive.ql.exec.Operator.initialize(Configuration,
ObjectInspector[]) At Operator.java:from instance method
org.apache.hadoop.hive.ql.exec.Operator.initialize(Configuration,
ObjectInspector[]) At Operator.java:[line 432] |
| | Write to static field
org.apache.hadoop.hive.ql.exec.Operator.hiveGcTimeMonitor from instance method
org.apache.hadoop.hive.ql.exec.Operator.initialize(Configuration,
ObjectInspector[]) At Operator.java:from instance method
org.apache.hadoop.hive.ql.exec.Operator.initialize(Configuration,
ObjectInspector[]) At Operator.java:[line 434] |
| | Write to static field
org.apache.hadoop.hive.ql.exec.Operator.lastAlertGcTimePercentage from instance
method
org.apache.hadoop.hive.ql.exec.Operator$HiveGcTimeMonitor$1.alert(GcTimeMonitor$GcData)
At Operator.java:from instance method
org.apache.hadoop.hive.ql.exec.Operator$HiveGcTimeMonitor$1.alert(GcTimeMonitor$GcData)
At Operator.java:[line 141] |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense xml javac javadoc compile findbugs
checkstyle |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
[jira] [Commented] (HIVE-20395) Parallelize files move in the ql.metadata.Hive#replaceFiles
[ https://issues.apache.org/jira/browse/HIVE-20395?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16603885#comment-16603885 ] ASF GitHub Bot commented on HIVE-20395: --- Github user medb closed the pull request at: https://github.com/apache/hive/pull/420 > Parallelize files move in the ql.metadata.Hive#replaceFiles > --- > > Key: HIVE-20395 > URL: https://issues.apache.org/jira/browse/HIVE-20395 > Project: Hive > Issue Type: Improvement > Components: Hive >Affects Versions: 3.1.0 >Reporter: Igor Dvorzhak >Assignee: Igor Dvorzhak >Priority: Major > Labels: pull-request-available > Attachments: HIVE-20395.10-branch-2.patch, HIVE-20395.10.patch > > > Files move is not parallelized in `replaceFiles` method and could be very > slow on Cloud Object Stores. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20377) Hive Kafka Storage Handler
[
https://issues.apache.org/jira/browse/HIVE-20377?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16603873#comment-16603873
]
Hive QA commented on HIVE-20377:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
41s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
11s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 9m
43s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
17s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
41s{color} | {color:blue} serde in master has 195 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
22s{color} | {color:blue} itests/qtest-druid in master has 6 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
46s{color} | {color:blue} itests/util in master has 52 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
42s{color} | {color:blue} llap-server in master has 84 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
3s{color} | {color:blue} ql in master has 2310 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 9m
30s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
7s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 11m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 10m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 10m
9s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
11s{color} | {color:red} itests/qtest-druid: The patch generated 37 new + 3
unchanged - 0 fixed = 40 total (was 3) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
10s{color} | {color:red} kafka-handler: The patch generated 33 new + 0
unchanged - 0 fixed = 33 total (was 0) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
13s{color} | {color:red} llap-server: The patch generated 1 new + 26 unchanged
- 4 fixed = 27 total (was 30) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
5s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
39s{color} | {color:red} patch/serde cannot run setBugDatabaseInfo from
findbugs {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
21s{color} | {color:red} patch/itests/qtest-druid cannot run setBugDatabaseInfo
from findbugs {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
47s{color} | {color:red} patch/itests/util cannot run setBugDatabaseInfo from
findbugs {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
24s{color} | {color:red} patch/kafka-handler cannot run setBugDatabaseInfo from
findbugs {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
11s{color} | {color:red} patch/llap-server cannot run setBugDatabaseInfo from
findbugs {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 7m
22s{color} | {color:red} patch/ql cannot run setBugDatabaseInfo from findbugs
{color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
35s{color} | {color:red} itests_util generated 1 new + 9 unchanged - 0 fixed =
10 total (was 9) {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
17s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
|
[jira] [Commented] (HIVE-20377) Hive Kafka Storage Handler
[
https://issues.apache.org/jira/browse/HIVE-20377?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16603866#comment-16603866
]
Hive QA commented on HIVE-20377:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12938298/HIVE-20377.19.patch
{color:green}SUCCESS:{color} +1 due to 8 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 14925 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/13592/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/13592/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-13592/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12938298 - PreCommit-HIVE-Build
> Hive Kafka Storage Handler
> --
>
> Key: HIVE-20377
> URL: https://issues.apache.org/jira/browse/HIVE-20377
> Project: Hive
> Issue Type: New Feature
>Affects Versions: 4.0.0
>Reporter: slim bouguerra
>Assignee: slim bouguerra
>Priority: Major
> Attachments: HIVE-20377.10.patch, HIVE-20377.11.patch,
> HIVE-20377.12.patch, HIVE-20377.15.patch, HIVE-20377.18.patch,
> HIVE-20377.18.patch, HIVE-20377.19.patch, HIVE-20377.19.patch,
> HIVE-20377.19.patch, HIVE-20377.4.patch, HIVE-20377.5.patch,
> HIVE-20377.6.patch, HIVE-20377.8.patch, HIVE-20377.8.patch, HIVE-20377.patch
>
>
> h1. Goal
> * Read streaming data form Kafka queue as an external table.
> * Allow streaming navigation by pushing down filters on Kafka record
> partition id, offset and timestamp.
> * Insert streaming data form Kafka to an actual Hive internal table, using
> CTAS statement.
> h1. Example
> h2. Create the external table
> {code}
> CREATE EXTERNAL TABLE kafka_table (`timestamp` timestamp, page string, `user`
> string, language string, added int, deleted int, flags string,comment string,
> namespace string)
> STORED BY 'org.apache.hadoop.hive.kafka.KafkaStorageHandler'
> TBLPROPERTIES
> ("kafka.topic" = "wikipedia",
> "kafka.bootstrap.servers"="brokeraddress:9092",
> "kafka.serde.class"="org.apache.hadoop.hive.serde2.JsonSerDe");
> {code}
> h2. Kafka Metadata
> In order to keep track of Kafka records the storage handler will add
> automatically the Kafka row metadata eg partition id, record offset and
> record timestamp.
> {code}
> DESCRIBE EXTENDED kafka_table
> timestamp timestamp from deserializer
> page string from deserializer
> user string from deserializer
> language string from deserializer
> country string from deserializer
> continent string from deserializer
> namespace string from deserializer
> newpage boolean from deserializer
> unpatrolled boolean from deserializer
> anonymous boolean from deserializer
> robot boolean from deserializer
> added int from deserializer
> deleted int from deserializer
> delta bigint from deserializer
> __partition int from deserializer
> __offset bigint from deserializer
> __timestamp bigint from deserializer
> {code}
> h2. Filter push down.
> Newer Kafka consumers 0.11.0 and higher allow seeking on the stream based on
> a given offset. The proposed storage handler will be able to leverage such
> API by pushing down filters over metadata columns, namely __partition (int),
> __offset(long) and __timestamp(long)
> For instance Query like
> {code}
> select `__offset` from kafka_table where (`__offset` < 10 and `__offset`>3
> and `__partition` = 0) or (`__partition` = 0 and `__offset` < 105 and
> `__offset` > 99) or (`__offset` = 109);
> {code}
> Will result on a scan of partition 0 only then read only records between
> offset 4 and 109.
> h2. With timestamp seeks
> The seeking based on the internal timestamps allows the handler to run on
> recently arrived data, by doing
> {code}
> select count(*) from kafka_table where `__timestamp` > 1000 *
> to_unix_timestamp(CURRENT_TIMESTAMP - interval '20' hours) ;
> {code}
> This allows for implicit relationships between event
[jira] [Updated] (HIVE-20306) Implement projection spec for fetching only requested fields from partitions
[ https://issues.apache.org/jira/browse/HIVE-20306?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexander Kolbasov updated HIVE-20306: -- Attachment: HIVE-20306.13.patch > Implement projection spec for fetching only requested fields from partitions > > > Key: HIVE-20306 > URL: https://issues.apache.org/jira/browse/HIVE-20306 > Project: Hive > Issue Type: Sub-task >Reporter: Vihang Karajgaonkar >Assignee: Alexander Kolbasov >Priority: Major > Attachments: HIVE-20306.02.patch, HIVE-20306.03.patch, > HIVE-20306.04.patch, HIVE-20306.05.patch, HIVE-20306.06.patch, > HIVE-20306.07.patch, HIVE-20306.08.patch, HIVE-20306.09.patch, > HIVE-20306.10.patch, HIVE-20306.11.patch, HIVE-20306.12.patch, > HIVE-20306.13.patch, HIVE-20306.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20306) Implement projection spec for fetching only requested fields from partitions
[
https://issues.apache.org/jira/browse/HIVE-20306?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16603826#comment-16603826
]
Hive QA commented on HIVE-20306:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12938293/HIVE-20306.12.patch
{color:green}SUCCESS:{color} +1 due to 3 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 14947 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.metastore.TestGetPartitionsUsingProjection.testInvalidProjectFieldNames
(batchId=221)
org.apache.hadoop.hive.metastore.TestGetPartitionsUsingProjection.testInvalidProjectFieldNames2
(batchId=221)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/13591/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/13591/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-13591/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12938293 - PreCommit-HIVE-Build
> Implement projection spec for fetching only requested fields from partitions
>
>
> Key: HIVE-20306
> URL: https://issues.apache.org/jira/browse/HIVE-20306
> Project: Hive
> Issue Type: Sub-task
>Reporter: Vihang Karajgaonkar
>Assignee: Alexander Kolbasov
>Priority: Major
> Attachments: HIVE-20306.02.patch, HIVE-20306.03.patch,
> HIVE-20306.04.patch, HIVE-20306.05.patch, HIVE-20306.06.patch,
> HIVE-20306.07.patch, HIVE-20306.08.patch, HIVE-20306.09.patch,
> HIVE-20306.10.patch, HIVE-20306.11.patch, HIVE-20306.12.patch,
> HIVE-20306.patch
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18772) Make Acid Cleaner use MIN_HISTORY_LEVEL
[ https://issues.apache.org/jira/browse/HIVE-18772?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman updated HIVE-18772: -- Attachment: HIVE-18772.02.patch > Make Acid Cleaner use MIN_HISTORY_LEVEL > --- > > Key: HIVE-18772 > URL: https://issues.apache.org/jira/browse/HIVE-18772 > Project: Hive > Issue Type: Improvement > Components: Transactions >Affects Versions: 3.0.0 >Reporter: Eugene Koifman >Assignee: Eugene Koifman >Priority: Major > Attachments: HIVE-18772.01.patch, HIVE-18772.02.patch, > HIVE-18772.02.patch > > > Instead of using Lock Manager state as it currently does. > This will eliminate possible race conditions > See this > [comment|https://issues.apache.org/jira/browse/HIVE-18192?focusedCommentId=16338208=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16338208] > Suppose A is the set of all ValidTxnList across all active readers. Each > ValidTxnList has minOpenTxnId. > MIN_HISTORY_LEVEL allows us to determine X = min(minOpenTxnId) across all > currently active readers > This means that no active transaction in the system sees any txn with txnid < > X as open. > This means if construct ValidTxnIdList with HWM=X-1 and use that in > getAcidState(), any files determined by this call as 'obsolete', will be seen > as obsolete by any existing/future reader, i.e. can be physically deleted. > This is also necessary for multi-statement transactions where relying on the > state of Lock Manager is not sufficient. For example > Suppose txn 17 starts at t1 and sees txnid 13 with writeID 13 open. > 13 commits (via it's parent txn) at t2 > t1. (17 is still running). > Compaction runs at t3 >t2 to produce base_14 (or delta_10_14 for example) on > Table1/Part1 (17 is still running) > Now delta_13 may be cleaned since it can be seen as obsolete and there may be > no locks on it, i.e. no one is reading it. > Now at t4 > t3 17 may (multi stmt txn) needs to read Table1/Part1. It cannot > use base_14 is that may have absorbed delete events from delete_delta_14. > Using MIN_HISTORY_LEVEL solves this. > See description of HIVE-18747 for more details on MIN_HISTORY_LEVEL -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18772) Make Acid Cleaner use MIN_HISTORY_LEVEL
[ https://issues.apache.org/jira/browse/HIVE-18772?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman updated HIVE-18772: -- Attachment: HIVE-18772.02.patch > Make Acid Cleaner use MIN_HISTORY_LEVEL > --- > > Key: HIVE-18772 > URL: https://issues.apache.org/jira/browse/HIVE-18772 > Project: Hive > Issue Type: Improvement > Components: Transactions >Affects Versions: 3.0.0 >Reporter: Eugene Koifman >Assignee: Eugene Koifman >Priority: Major > Attachments: HIVE-18772.01.patch, HIVE-18772.02.patch > > > Instead of using Lock Manager state as it currently does. > This will eliminate possible race conditions > See this > [comment|https://issues.apache.org/jira/browse/HIVE-18192?focusedCommentId=16338208=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16338208] > Suppose A is the set of all ValidTxnList across all active readers. Each > ValidTxnList has minOpenTxnId. > MIN_HISTORY_LEVEL allows us to determine X = min(minOpenTxnId) across all > currently active readers > This means that no active transaction in the system sees any txn with txnid < > X as open. > This means if construct ValidTxnIdList with HWM=X-1 and use that in > getAcidState(), any files determined by this call as 'obsolete', will be seen > as obsolete by any existing/future reader, i.e. can be physically deleted. > This is also necessary for multi-statement transactions where relying on the > state of Lock Manager is not sufficient. For example > Suppose txn 17 starts at t1 and sees txnid 13 with writeID 13 open. > 13 commits (via it's parent txn) at t2 > t1. (17 is still running). > Compaction runs at t3 >t2 to produce base_14 (or delta_10_14 for example) on > Table1/Part1 (17 is still running) > Now delta_13 may be cleaned since it can be seen as obsolete and there may be > no locks on it, i.e. no one is reading it. > Now at t4 > t3 17 may (multi stmt txn) needs to read Table1/Part1. It cannot > use base_14 is that may have absorbed delete events from delete_delta_14. > Using MIN_HISTORY_LEVEL solves this. > See description of HIVE-18747 for more details on MIN_HISTORY_LEVEL -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20306) Implement projection spec for fetching only requested fields from partitions
[
https://issues.apache.org/jira/browse/HIVE-20306?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16603810#comment-16603810
]
Hive QA commented on HIVE-20306:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
48s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
28s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
20s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
26s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 2m
31s{color} | {color:blue} standalone-metastore/metastore-common in master has 9
extant Findbugs warnings. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
14s{color} | {color:red} metastore-server in master failed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
28s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
10s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
22s{color} | {color:red} hcatalog-unit in the patch failed. {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
27s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 53 line(s) that end in whitespace. Use
git apply --whitespace=fix <>. Refer
https://git-scm.com/docs/git-apply {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
1s{color} | {color:red} The patch 1 line(s) with tabs. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
15s{color} | {color:red} metastore-server in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
23s{color} | {color:red} standalone-metastore_metastore-server generated 1 new
+ 61 unchanged - 0 fixed = 62 total (was 61) {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 23m 17s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-13591/dev-support/hive-personality.sh
|
| git revision | master / 33fa62f |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-13591/yetus/branch-findbugs-standalone-metastore_metastore-server.txt
|
| mvninstall |
http://104.198.109.242/logs//PreCommit-HIVE-Build-13591/yetus/patch-mvninstall-itests_hcatalog-unit.txt
|
| whitespace |
http://104.198.109.242/logs//PreCommit-HIVE-Build-13591/yetus/whitespace-eol.txt
|
| whitespace |
http://104.198.109.242/logs//PreCommit-HIVE-Build-13591/yetus/whitespace-tabs.txt
|
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-13591/yetus/patch-findbugs-standalone-metastore_metastore-server.txt
|
| javadoc |
http://104.198.109.242/logs//PreCommit-HIVE-Build-13591/yetus/diff-javadoc-javadoc-standalone-metastore_metastore-server.txt
|
| modules | C: standalone-metastore/metastore-common itests/hcatalog-unit
standalone-metastore/metastore-server U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-13591/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Implement projection spec for fetching only requested fields from partitions
>
[jira] [Updated] (HIVE-20499) GetTablesOperation pull all the tables meta irrespective of auth.
[
https://issues.apache.org/jira/browse/HIVE-20499?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rajkumar Singh updated HIVE-20499:
--
Attachment: HIVE-20499.patch
Status: Patch Available (was: In Progress)
> GetTablesOperation pull all the tables meta irrespective of auth.
> -
>
> Key: HIVE-20499
> URL: https://issues.apache.org/jira/browse/HIVE-20499
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 3.1.0
> Environment: hive-3,java-8,sqlstdauth/ranger auth enabled.
>Reporter: Rajkumar Singh
>Assignee: Rajkumar Singh
>Priority: Major
> Attachments: HIVE-20499.patch
>
>
> GetTablesOperation pull all the tables meta irrespective of auth.
> dbvisualizer and other ui based jdbc client pull tableemta similar to
> following operation:
> {code}
> ResultSet res = con.getMetaData().getTables("", "", "%", new String[] {
> "TABLE", "VIEW" });
> {code}
> https://github.com/rajkrrsingh/HiveServer2JDBCSample/blob/master/src/main/java/TestConnection.java#L20
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20423) Set NULLS LAST as the default null ordering
[
https://issues.apache.org/jira/browse/HIVE-20423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16603780#comment-16603780
]
Hive QA commented on HIVE-20423:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12937714/HIVE-20423.6.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 5 failed/errored test(s), 14924 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniDruidCliDriver.testCliDriver[druidmini_dynamic_partition]
(batchId=193)
org.apache.hadoop.hive.cli.TestMiniDruidCliDriver.testCliDriver[druidmini_expressions]
(batchId=193)
org.apache.hadoop.hive.cli.TestMiniDruidCliDriver.testCliDriver[druidmini_test1]
(batchId=193)
org.apache.hadoop.hive.cli.TestMiniDruidCliDriver.testCliDriver[druidmini_test_alter]
(batchId=193)
org.apache.hadoop.hive.cli.TestMiniDruidCliDriver.testCliDriver[druidmini_test_insert]
(batchId=193)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/13590/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/13590/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-13590/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 5 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12937714 - PreCommit-HIVE-Build
> Set NULLS LAST as the default null ordering
> ---
>
> Key: HIVE-20423
> URL: https://issues.apache.org/jira/browse/HIVE-20423
> Project: Hive
> Issue Type: Bug
>Reporter: Teddy Choi
>Assignee: Teddy Choi
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0, 3.2.0
>
> Attachments: HIVE-20423.1.patch, HIVE-20423.2.patch,
> HIVE-20423.3.patch, HIVE-20423.4.patch, HIVE-20423.4.patch,
> HIVE-20423.5.patch, HIVE-20423.6.patch
>
>
> HIVE-20150 TopNKeyOperator pushdown can be more efficient if NULLS LAST
> becomes the default null ordering.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20423) Set NULLS LAST as the default null ordering
[
https://issues.apache.org/jira/browse/HIVE-20423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16603745#comment-16603745
]
Hive QA commented on HIVE-20423:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
28s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
7s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
37s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
55s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
57s{color} | {color:blue} ql in master has 2310 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
18s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
9s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
35s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
43s{color} | {color:red} ql: The patch generated 4 new + 206 unchanged - 20
fixed = 210 total (was 226) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
33s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 28m 22s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-13590/dev-support/hive-personality.sh
|
| git revision | master / 33fa62f |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-13590/yetus/diff-checkstyle-ql.txt
|
| modules | C: itests/hive-blobstore ql U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-13590/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Set NULLS LAST as the default null ordering
> ---
>
> Key: HIVE-20423
> URL: https://issues.apache.org/jira/browse/HIVE-20423
> Project: Hive
> Issue Type: Bug
>Reporter: Teddy Choi
>Assignee: Teddy Choi
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0, 3.2.0
>
> Attachments: HIVE-20423.1.patch, HIVE-20423.2.patch,
> HIVE-20423.3.patch, HIVE-20423.4.patch, HIVE-20423.4.patch,
> HIVE-20423.5.patch, HIVE-20423.6.patch
>
>
> HIVE-20150 TopNKeyOperator pushdown can be more efficient if NULLS LAST
> becomes the default null ordering.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (HIVE-20501) Vectorization: Closed range fast-path for Fast Long hashset
[
https://issues.apache.org/jira/browse/HIVE-20501?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gopal V reassigned HIVE-20501:
--
Assignee: Gopal V
> Vectorization: Closed range fast-path for Fast Long hashset
>
>
> Key: HIVE-20501
> URL: https://issues.apache.org/jira/browse/HIVE-20501
> Project: Hive
> Issue Type: Improvement
>Reporter: Gopal V
>Assignee: Gopal V
>Priority: Major
>
> In scenarios where the surrogate keys are entirely contiguous, the cache can
> offer a fast-path for [min,max], without a further lookup in the hashtable.
> {code}
> hive> select min(c_customer_sk), max(c_customer_sk), max(c_customer_sk) -
> min(c_customer_sk), count(1) from customer;
> 1 650064996500
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20437) Handle schema evolution from float, double and decimal
[ https://issues.apache.org/jira/browse/HIVE-20437?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Janaki Lahorani updated HIVE-20437: --- Attachment: HIVE-20437.2.patch > Handle schema evolution from float, double and decimal > -- > > Key: HIVE-20437 > URL: https://issues.apache.org/jira/browse/HIVE-20437 > Project: Hive > Issue Type: Bug >Reporter: Janaki Lahorani >Assignee: Janaki Lahorani >Priority: Major > Attachments: HIVE-20437.1.patch, HIVE-20437.2.patch > > > When data created as float, double or decimal in parquet format is read back > using some other type, errors are seen. Parquet should behave just like any > other format. If the value is valid for the new type, data is retuned > otherwise null has to be returned. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20432) Rewrite BETWEEN to IN for integer types for stats estimation
[
https://issues.apache.org/jira/browse/HIVE-20432?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16603672#comment-16603672
]
Hive QA commented on HIVE-20432:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12937698/HIVE-20432.3.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 14924 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/13589/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/13589/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-13589/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12937698 - PreCommit-HIVE-Build
> Rewrite BETWEEN to IN for integer types for stats estimation
>
>
> Key: HIVE-20432
> URL: https://issues.apache.org/jira/browse/HIVE-20432
> Project: Hive
> Issue Type: Improvement
>Affects Versions: 4.0.0
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
> Attachments: HIVE-20432.1.patch, HIVE-20432.2.patch,
> HIVE-20432.3.patch
>
>
> IN clause provides better statistics estimation since predicates are equality
> based and it keeps into account selectivity of all values. BETWEEN is
> rewritten into {{>= AND <=}} i.e. non-equality predicates which cause worse
> estimation. For integer types BETWEEN could be rewritten into IN e.g. BETWEEN
> 4 and 7 is equivalent to IN (4,5,6,7).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20395) Parallelize files move in the ql.metadata.Hive#replaceFiles
[ https://issues.apache.org/jira/browse/HIVE-20395?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16603644#comment-16603644 ] Gopal V commented on HIVE-20395: Thanks [~medb], Pushing [^HIVE-20395.10.patch] to master. Will wait for branch-2 runs for the backport. > Parallelize files move in the ql.metadata.Hive#replaceFiles > --- > > Key: HIVE-20395 > URL: https://issues.apache.org/jira/browse/HIVE-20395 > Project: Hive > Issue Type: Improvement > Components: Hive >Affects Versions: 3.1.0 >Reporter: Igor Dvorzhak >Assignee: Igor Dvorzhak >Priority: Major > Labels: pull-request-available > Attachments: HIVE-20395.10-branch-2.patch, HIVE-20395.10.patch > > > Files move is not parallelized in `replaceFiles` method and could be very > slow on Cloud Object Stores. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20432) Rewrite BETWEEN to IN for integer types for stats estimation
[
https://issues.apache.org/jira/browse/HIVE-20432?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16603637#comment-16603637
]
Hive QA commented on HIVE-20432:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
54s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
12s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
41s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
5s{color} | {color:blue} ql in master has 2310 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
58s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
7s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
40s{color} | {color:red} ql: The patch generated 4 new + 72 unchanged - 0 fixed
= 76 total (was 72) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
57s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 25m 0s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-13589/dev-support/hive-personality.sh
|
| git revision | master / df1a02b |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-13589/yetus/diff-checkstyle-ql.txt
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-13589/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Rewrite BETWEEN to IN for integer types for stats estimation
>
>
> Key: HIVE-20432
> URL: https://issues.apache.org/jira/browse/HIVE-20432
> Project: Hive
> Issue Type: Improvement
>Affects Versions: 4.0.0
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
> Attachments: HIVE-20432.1.patch, HIVE-20432.2.patch,
> HIVE-20432.3.patch
>
>
> IN clause provides better statistics estimation since predicates are equality
> based and it keeps into account selectivity of all values. BETWEEN is
> rewritten into {{>= AND <=}} i.e. non-equality predicates which cause worse
> estimation. For integer types BETWEEN could be rewritten into IN e.g. BETWEEN
> 4 and 7 is equivalent to IN (4,5,6,7).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20499) GetTablesOperation pull all the tables meta irrespective of auth.
[
https://issues.apache.org/jira/browse/HIVE-20499?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rajkumar Singh updated HIVE-20499:
--
Environment: hive-3,java-8,sqlstdauth/ranger auth enabled. (was:
hive-3,java-8,sqlstdaut/ranger auth enabled.)
> GetTablesOperation pull all the tables meta irrespective of auth.
> -
>
> Key: HIVE-20499
> URL: https://issues.apache.org/jira/browse/HIVE-20499
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 3.1.0
> Environment: hive-3,java-8,sqlstdauth/ranger auth enabled.
>Reporter: Rajkumar Singh
>Assignee: Rajkumar Singh
>Priority: Major
>
> GetTablesOperation pull all the tables meta irrespective of auth.
> dbvisualizer and other ui based jdbc client pull tableemta similar to
> following operation:
> {code}
> ResultSet res = con.getMetaData().getTables("", "", "%", new String[] {
> "TABLE", "VIEW" });
> {code}
> https://github.com/rajkrrsingh/HiveServer2JDBCSample/blob/master/src/main/java/TestConnection.java#L20
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18873) Skipping predicate pushdown for MR silently at HiveInputFormat can cause storage handlers to produce erroneous result
[
https://issues.apache.org/jira/browse/HIVE-18873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16603603#comment-16603603
]
Vineet Garg commented on HIVE-18873:
Uploaded new patch which disable the test for now. Also opened HIVE-20500 to
investigate that failure
> Skipping predicate pushdown for MR silently at HiveInputFormat can cause
> storage handlers to produce erroneous result
> -
>
> Key: HIVE-18873
> URL: https://issues.apache.org/jira/browse/HIVE-18873
> Project: Hive
> Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Ankit Singhal
>Assignee: Ankit Singhal
>Priority: Major
> Attachments: HIVE-18873.2.patch, HIVE-18873.2_reattach.patch,
> HIVE-18873.3.patch, HIVE-18873.patch
>
>
> {code:java}
> // disable filter pushdown for mapreduce when there are more than one table
> aliases,
> // since we don't clone jobConf per alias
> if (mrwork != null && mrwork.getAliases() != null &&
> mrwork.getAliases().size() > 1 &&
> jobConf.get(ConfVars.HIVE_EXECUTION_ENGINE.varname).equals("mr")) {
> return;
> }
> {code}
> I believe this needs to be handled at OpProcFactory so that hive doesn't
> believe that predicate is handled by storage handler.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-15932) add "explain ast"
[
https://issues.apache.org/jira/browse/HIVE-15932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16603539#comment-16603539
]
Hive QA commented on HIVE-15932:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12938276/HIVE-15932.01.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 14923 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/13587/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/13587/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-13587/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12938276 - PreCommit-HIVE-Build
> add "explain ast"
> -
>
> Key: HIVE-15932
> URL: https://issues.apache.org/jira/browse/HIVE-15932
> Project: Hive
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Igor Kryvenko
>Priority: Major
> Attachments: HIVE-15932.01.patch
>
>
> AST was removed in explain extended in HIVE-13533; that makes sense from
> users perspective for the common case, but it would be useful for Hive
> developers and advanced users to see the AST to diagnose issues. "Explain
> ast" command can be added to dump the AST.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18873) Skipping predicate pushdown for MR silently at HiveInputFormat can cause storage handlers to produce erroneous result
[
https://issues.apache.org/jira/browse/HIVE-18873?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vineet Garg updated HIVE-18873:
---
Attachment: HIVE-18873.3.patch
> Skipping predicate pushdown for MR silently at HiveInputFormat can cause
> storage handlers to produce erroneous result
> -
>
> Key: HIVE-18873
> URL: https://issues.apache.org/jira/browse/HIVE-18873
> Project: Hive
> Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Ankit Singhal
>Assignee: Ankit Singhal
>Priority: Major
> Attachments: HIVE-18873.2.patch, HIVE-18873.2_reattach.patch,
> HIVE-18873.3.patch, HIVE-18873.patch
>
>
> {code:java}
> // disable filter pushdown for mapreduce when there are more than one table
> aliases,
> // since we don't clone jobConf per alias
> if (mrwork != null && mrwork.getAliases() != null &&
> mrwork.getAliases().size() > 1 &&
> jobConf.get(ConfVars.HIVE_EXECUTION_ENGINE.varname).equals("mr")) {
> return;
> }
> {code}
> I believe this needs to be handled at OpProcFactory so that hive doesn't
> believe that predicate is handled by storage handler.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18873) Skipping predicate pushdown for MR silently at HiveInputFormat can cause storage handlers to produce erroneous result
[
https://issues.apache.org/jira/browse/HIVE-18873?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vineet Garg updated HIVE-18873:
---
Status: Patch Available (was: Open)
> Skipping predicate pushdown for MR silently at HiveInputFormat can cause
> storage handlers to produce erroneous result
> -
>
> Key: HIVE-18873
> URL: https://issues.apache.org/jira/browse/HIVE-18873
> Project: Hive
> Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Ankit Singhal
>Assignee: Ankit Singhal
>Priority: Major
> Attachments: HIVE-18873.2.patch, HIVE-18873.2_reattach.patch,
> HIVE-18873.3.patch, HIVE-18873.patch
>
>
> {code:java}
> // disable filter pushdown for mapreduce when there are more than one table
> aliases,
> // since we don't clone jobConf per alias
> if (mrwork != null && mrwork.getAliases() != null &&
> mrwork.getAliases().size() > 1 &&
> jobConf.get(ConfVars.HIVE_EXECUTION_ENGINE.varname).equals("mr")) {
> return;
> }
> {code}
> I believe this needs to be handled at OpProcFactory so that hive doesn't
> believe that predicate is handled by storage handler.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18873) Skipping predicate pushdown for MR silently at HiveInputFormat can cause storage handlers to produce erroneous result
[
https://issues.apache.org/jira/browse/HIVE-18873?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vineet Garg updated HIVE-18873:
---
Attachment: (was: HIVE-18873.3.patch)
> Skipping predicate pushdown for MR silently at HiveInputFormat can cause
> storage handlers to produce erroneous result
> -
>
> Key: HIVE-18873
> URL: https://issues.apache.org/jira/browse/HIVE-18873
> Project: Hive
> Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Ankit Singhal
>Assignee: Ankit Singhal
>Priority: Major
> Attachments: HIVE-18873.2.patch, HIVE-18873.2_reattach.patch,
> HIVE-18873.patch
>
>
> {code:java}
> // disable filter pushdown for mapreduce when there are more than one table
> aliases,
> // since we don't clone jobConf per alias
> if (mrwork != null && mrwork.getAliases() != null &&
> mrwork.getAliases().size() > 1 &&
> jobConf.get(ConfVars.HIVE_EXECUTION_ENGINE.varname).equals("mr")) {
> return;
> }
> {code}
> I believe this needs to be handled at OpProcFactory so that hive doesn't
> believe that predicate is handled by storage handler.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18873) Skipping predicate pushdown for MR silently at HiveInputFormat can cause storage handlers to produce erroneous result
[
https://issues.apache.org/jira/browse/HIVE-18873?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vineet Garg updated HIVE-18873:
---
Attachment: HIVE-18873.3.patch
> Skipping predicate pushdown for MR silently at HiveInputFormat can cause
> storage handlers to produce erroneous result
> -
>
> Key: HIVE-18873
> URL: https://issues.apache.org/jira/browse/HIVE-18873
> Project: Hive
> Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Ankit Singhal
>Assignee: Ankit Singhal
>Priority: Major
> Attachments: HIVE-18873.2.patch, HIVE-18873.2_reattach.patch,
> HIVE-18873.3.patch, HIVE-18873.patch
>
>
> {code:java}
> // disable filter pushdown for mapreduce when there are more than one table
> aliases,
> // since we don't clone jobConf per alias
> if (mrwork != null && mrwork.getAliases() != null &&
> mrwork.getAliases().size() > 1 &&
> jobConf.get(ConfVars.HIVE_EXECUTION_ENGINE.varname).equals("mr")) {
> return;
> }
> {code}
> I believe this needs to be handled at OpProcFactory so that hive doesn't
> believe that predicate is handled by storage handler.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18873) Skipping predicate pushdown for MR silently at HiveInputFormat can cause storage handlers to produce erroneous result
[
https://issues.apache.org/jira/browse/HIVE-18873?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vineet Garg updated HIVE-18873:
---
Status: Open (was: Patch Available)
> Skipping predicate pushdown for MR silently at HiveInputFormat can cause
> storage handlers to produce erroneous result
> -
>
> Key: HIVE-18873
> URL: https://issues.apache.org/jira/browse/HIVE-18873
> Project: Hive
> Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Ankit Singhal
>Assignee: Ankit Singhal
>Priority: Major
> Attachments: HIVE-18873.2.patch, HIVE-18873.2_reattach.patch,
> HIVE-18873.3.patch, HIVE-18873.patch
>
>
> {code:java}
> // disable filter pushdown for mapreduce when there are more than one table
> aliases,
> // since we don't clone jobConf per alias
> if (mrwork != null && mrwork.getAliases() != null &&
> mrwork.getAliases().size() > 1 &&
> jobConf.get(ConfVars.HIVE_EXECUTION_ENGINE.varname).equals("mr")) {
> return;
> }
> {code}
> I believe this needs to be handled at OpProcFactory so that hive doesn't
> believe that predicate is handled by storage handler.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-17684) HoS memory issues with MapJoinMemoryExhaustionHandler
[
https://issues.apache.org/jira/browse/HIVE-17684?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16603530#comment-16603530
]
Sahil Takiar commented on HIVE-17684:
-
[~mi...@cloudera.com] sorry for the delay on this. I figured out why a bunch of
the {{TestSparkCliDriver}} tests were failing and attached an updated patch
with a fix.
As for the issues with {{auto_join25.q.out}} - it looks like there is a config
called {{hive.mapjoin.localtask.max.memory.usage}} /
{{hive.mapjoin.followby.gby.localtask.max.memory.usage}} which defines how much
memory the small table can consume before the memory exhaustion handler throws
an error. These tests define a very low value for these configs and thus expect
the tests to trigger the memory exhaustion handler.
We should probably do something similar. Introduce a new config that makes
{{CRITICAl_GC_TIME_PERCENTAGE_PROD}} configurable. We can set it to a lower
value in our tests in order to confirm that everything is working correctly.
Let me know if you need more help getting this done.
> HoS memory issues with MapJoinMemoryExhaustionHandler
> -
>
> Key: HIVE-17684
> URL: https://issues.apache.org/jira/browse/HIVE-17684
> Project: Hive
> Issue Type: Bug
> Components: Spark
>Reporter: Sahil Takiar
>Assignee: Misha Dmitriev
>Priority: Major
> Attachments: HIVE-17684.01.patch, HIVE-17684.02.patch,
> HIVE-17684.03.patch, HIVE-17684.04.patch, HIVE-17684.05.patch
>
>
> We have seen a number of memory issues due the {{HashSinkOperator}} use of
> the {{MapJoinMemoryExhaustionHandler}}. This handler is meant to detect
> scenarios where the small table is taking too much space in memory, in which
> case a {{MapJoinMemoryExhaustionError}} is thrown.
> The configs to control this logic are:
> {{hive.mapjoin.localtask.max.memory.usage}} (default 0.90)
> {{hive.mapjoin.followby.gby.localtask.max.memory.usage}} (default 0.55)
> The handler works by using the {{MemoryMXBean}} and uses the following logic
> to estimate how much memory the {{HashMap}} is consuming:
> {{MemoryMXBean#getHeapMemoryUsage().getUsed() /
> MemoryMXBean#getHeapMemoryUsage().getMax()}}
> The issue is that {{MemoryMXBean#getHeapMemoryUsage().getUsed()}} can be
> inaccurate. The value returned by this method returns all reachable and
> unreachable memory on the heap, so there may be a bunch of garbage data, and
> the JVM just hasn't taken the time to reclaim it all. This can lead to
> intermittent failures of this check even though a simple GC would have
> reclaimed enough space for the process to continue working.
> We should re-think the usage of {{MapJoinMemoryExhaustionHandler}} for HoS.
> In Hive-on-MR this probably made sense to use because every Hive task was run
> in a dedicated container, so a Hive Task could assume it created most of the
> data on the heap. However, in Hive-on-Spark there can be multiple Hive Tasks
> running in a single executor, each doing different things.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20459) add ThriftHiveMetastore.get_open_txns(long txnid)
[
https://issues.apache.org/jira/browse/HIVE-20459?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16603522#comment-16603522
]
Igor Kryvenko commented on HIVE-20459:
--
[~ekoifman] Hi Eugene. Could, you review my patch, please?
Thanks, Ihor.
> add ThriftHiveMetastore.get_open_txns(long txnid)
> -
>
> Key: HIVE-20459
> URL: https://issues.apache.org/jira/browse/HIVE-20459
> Project: Hive
> Issue Type: Improvement
> Components: Metastore, Transactions
>Reporter: Eugene Koifman
>Assignee: Igor Kryvenko
>Priority: Minor
> Attachments: HIVE-20459.01.patch, HIVE-20459.02.patch,
> HIVE-20459.03.patch
>
>
> we currently have {{ThriftHiveMetastore.get_open_txns()}} which maps to
> {{TxnHandler.getOpenTxns()}}. The usual usage is
> {{TxnUtils.createValidReadTxnList(GetOpenTxnsResponse txns, long
> currentTxn)}} where the complete list transactions is obtained from Metastore
> and then anything above currentTxn is thrown away.
> Would be useful to add {{ThriftHiveMetastore.get_open_txns(long txnid)}} and
> {{TxnHandler.getOpenTxns(long)}} to not retrieve things that will be thrown
> away. Especially when there are a lot of running transactions.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18453) ACID: Add "CREATE TRANSACTIONAL TABLE" syntax to unify ACID ORC & Parquet support
[
https://issues.apache.org/jira/browse/HIVE-18453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16603503#comment-16603503
]
Eugene Koifman commented on HIVE-18453:
---
[~ikryvenko], no. "create transactional..." should examine the create table
statement and create a full CRUD table if possible (i.e. if it's a managed
table stored as ORC, not sorted, etc.). If this table cannot be made full
CRUD, it should check if it can be made insert-only transactional. If that is
not possible (e.g. the table is external), it should raise an error.
This is exactly what happens when a "create table ..." (w/o any
transactional=true property in the create table stmt) is processed and both
{{HiveConf.ConfVars.HIVE_CREATE_TABLES_AS_INSERT_ONLY}} and
{{MetastoreConf.ConfVars.CREATE_TABLES_AS_ACID}} are set to true. That's why
I'm suggesting that you look at
{{SemanticAnalyzer.validateAndAddDefaultProperties()}} - this is where the
logic to examine the new table is. I think you can just make your "create
transactional table..." used this method.
> ACID: Add "CREATE TRANSACTIONAL TABLE" syntax to unify ACID ORC & Parquet
> support
> -
>
> Key: HIVE-18453
> URL: https://issues.apache.org/jira/browse/HIVE-18453
> Project: Hive
> Issue Type: Improvement
> Components: Transactions
>Reporter: Gopal V
>Assignee: Igor Kryvenko
>Priority: Major
> Attachments: HIVE-18453.01.patch, HIVE-18453.02.patch,
> HIVE-18453.03.patch, HIVE-18453.04.patch, HIVE-18453.05.patch,
> HIVE-18453.06.patch, HIVE-18453.07.patch
>
>
> The ACID table markers are currently done with TBLPROPERTIES which is
> inherently fragile.
> The "create transactional table" offers a way to standardize the syntax and
> allows for future compatibility changes to support Parquet ACIDv2 tables
> along with ORC tables.
> The ACIDv2 design is format independent, with the ability to add new
> vectorized input formats with no changes to the design.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-15932) add "explain ast"
[
https://issues.apache.org/jira/browse/HIVE-15932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16603498#comment-16603498
]
Hive QA commented on HIVE-15932:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
36s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
3s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
40s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
4s{color} | {color:blue} ql in master has 2310 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
0s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 24m 56s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-13587/dev-support/hive-personality.sh
|
| git revision | master / 8b73fbb |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-13587/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> add "explain ast"
> -
>
> Key: HIVE-15932
> URL: https://issues.apache.org/jira/browse/HIVE-15932
> Project: Hive
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Igor Kryvenko
>Priority: Major
> Attachments: HIVE-15932.01.patch
>
>
> AST was removed in explain extended in HIVE-13533; that makes sense from
> users perspective for the common case, but it would be useful for Hive
> developers and advanced users to see the AST to diagnose issues. "Explain
> ast" command can be added to dump the AST.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18873) Skipping predicate pushdown for MR silently at HiveInputFormat can cause storage handlers to produce erroneous result
[
https://issues.apache.org/jira/browse/HIVE-18873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16603497#comment-16603497
]
Prasanth Jayachandran commented on HIVE-18873:
--
{quote}Under the assumption that Hive is ensuring the results are accurate
anyways? (e.g. we'd want to check that all storage handlers work this way)
{quote}
Orc PPD works this way, if orc cannot handle the filter it will drop the filter
and continue with rest of the filters. The filters will still remain in hive
operator tree which will get reapplied anyways.
If the test failure is unrelated to the patch and is already happening in the
master, I think we can go ahead and commit the latest patch and track the test
failure in a separate ticket.
+1, comment failing test, track it separately.
> Skipping predicate pushdown for MR silently at HiveInputFormat can cause
> storage handlers to produce erroneous result
> -
>
> Key: HIVE-18873
> URL: https://issues.apache.org/jira/browse/HIVE-18873
> Project: Hive
> Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Ankit Singhal
>Assignee: Ankit Singhal
>Priority: Major
> Attachments: HIVE-18873.2.patch, HIVE-18873.2_reattach.patch,
> HIVE-18873.patch
>
>
> {code:java}
> // disable filter pushdown for mapreduce when there are more than one table
> aliases,
> // since we don't clone jobConf per alias
> if (mrwork != null && mrwork.getAliases() != null &&
> mrwork.getAliases().size() > 1 &&
> jobConf.get(ConfVars.HIVE_EXECUTION_ENGINE.varname).equals("mr")) {
> return;
> }
> {code}
> I believe this needs to be handled at OpProcFactory so that hive doesn't
> believe that predicate is handled by storage handler.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20372) WRTIE_SET typo in TxnHandler
[ https://issues.apache.org/jira/browse/HIVE-20372?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergio Peña updated HIVE-20372: --- Resolution: Fixed Status: Resolved (was: Patch Available) I committed this to master. Thanks [~karthik.manamcheri] for your contribution. > WRTIE_SET typo in TxnHandler > > > Key: HIVE-20372 > URL: https://issues.apache.org/jira/browse/HIVE-20372 > Project: Hive > Issue Type: Bug > Components: Standalone Metastore, Transactions >Affects Versions: 3.1.0 >Reporter: Laszlo Bodor >Assignee: Karthik Manamcheri >Priority: Trivial > Labels: Newbie, newbie, newbie++, newbiee > Fix For: 4.0.0 > > Attachments: HIVE-20372.2.patch, HIVE-20372.3.patch, HIVE-20372.patch > > > [https://github.com/prongs/apache-hive/blob/deabe59371e98a21f4c3a58a9d8da51e4632fca5/metastore/src/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java#L765] > minor typo -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20377) Hive Kafka Storage Handler
[
https://issues.apache.org/jira/browse/HIVE-20377?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
slim bouguerra updated HIVE-20377:
--
Attachment: HIVE-20377.19.patch
> Hive Kafka Storage Handler
> --
>
> Key: HIVE-20377
> URL: https://issues.apache.org/jira/browse/HIVE-20377
> Project: Hive
> Issue Type: New Feature
>Affects Versions: 4.0.0
>Reporter: slim bouguerra
>Assignee: slim bouguerra
>Priority: Major
> Attachments: HIVE-20377.10.patch, HIVE-20377.11.patch,
> HIVE-20377.12.patch, HIVE-20377.15.patch, HIVE-20377.18.patch,
> HIVE-20377.18.patch, HIVE-20377.19.patch, HIVE-20377.19.patch,
> HIVE-20377.19.patch, HIVE-20377.4.patch, HIVE-20377.5.patch,
> HIVE-20377.6.patch, HIVE-20377.8.patch, HIVE-20377.8.patch, HIVE-20377.patch
>
>
> h1. Goal
> * Read streaming data form Kafka queue as an external table.
> * Allow streaming navigation by pushing down filters on Kafka record
> partition id, offset and timestamp.
> * Insert streaming data form Kafka to an actual Hive internal table, using
> CTAS statement.
> h1. Example
> h2. Create the external table
> {code}
> CREATE EXTERNAL TABLE kafka_table (`timestamp` timestamp, page string, `user`
> string, language string, added int, deleted int, flags string,comment string,
> namespace string)
> STORED BY 'org.apache.hadoop.hive.kafka.KafkaStorageHandler'
> TBLPROPERTIES
> ("kafka.topic" = "wikipedia",
> "kafka.bootstrap.servers"="brokeraddress:9092",
> "kafka.serde.class"="org.apache.hadoop.hive.serde2.JsonSerDe");
> {code}
> h2. Kafka Metadata
> In order to keep track of Kafka records the storage handler will add
> automatically the Kafka row metadata eg partition id, record offset and
> record timestamp.
> {code}
> DESCRIBE EXTENDED kafka_table
> timestamp timestamp from deserializer
> page string from deserializer
> user string from deserializer
> language string from deserializer
> country string from deserializer
> continent string from deserializer
> namespace string from deserializer
> newpage boolean from deserializer
> unpatrolled boolean from deserializer
> anonymous boolean from deserializer
> robot boolean from deserializer
> added int from deserializer
> deleted int from deserializer
> delta bigint from deserializer
> __partition int from deserializer
> __offset bigint from deserializer
> __timestamp bigint from deserializer
> {code}
> h2. Filter push down.
> Newer Kafka consumers 0.11.0 and higher allow seeking on the stream based on
> a given offset. The proposed storage handler will be able to leverage such
> API by pushing down filters over metadata columns, namely __partition (int),
> __offset(long) and __timestamp(long)
> For instance Query like
> {code}
> select `__offset` from kafka_table where (`__offset` < 10 and `__offset`>3
> and `__partition` = 0) or (`__partition` = 0 and `__offset` < 105 and
> `__offset` > 99) or (`__offset` = 109);
> {code}
> Will result on a scan of partition 0 only then read only records between
> offset 4 and 109.
> h2. With timestamp seeks
> The seeking based on the internal timestamps allows the handler to run on
> recently arrived data, by doing
> {code}
> select count(*) from kafka_table where `__timestamp` > 1000 *
> to_unix_timestamp(CURRENT_TIMESTAMP - interval '20' hours) ;
> {code}
> This allows for implicit relationships between event timestamps and kafka
> timestamps to be expressed in queries (i.e event_timestamp is always < than
> kafka __timestamp and kafka __timestamp is never > 15 minutes from event etc).
> h2. More examples with Avro
> {code}
> CREATE EXTERNAL TABLE wiki_kafka_avro_table
> STORED BY 'org.apache.hadoop.hive.kafka.KafkaStorageHandler'
> TBLPROPERTIES
> ("kafka.topic" = "wiki_kafka_avro_table",
> "kafka.bootstrap.servers"="localhost:9092",
> "kafka.serde.class"="org.apache.hadoop.hive.serde2.avro.AvroSerDe",
> 'avro.schema.literal'='{
> "type" : "record",
> "name" : "Wikipedia",
> "namespace" : "org.apache.hive.kafka",
> "version": "1",
> "fields" : [ {
> "name" : "isrobot",
> "type" : "boolean"
> }, {
> "name" : "channel",
> "type" : "string"
> }, {
> "name" : "timestamp",
> "type" : "string"
> }, {
> "name" : "flags",
> "type" : "string"
> },
[jira] [Commented] (HIVE-20372) WRTIE_SET typo in TxnHandler
[ https://issues.apache.org/jira/browse/HIVE-20372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16603473#comment-16603473 ] Sergio Peña commented on HIVE-20372: The patch looks good. +1 > WRTIE_SET typo in TxnHandler > > > Key: HIVE-20372 > URL: https://issues.apache.org/jira/browse/HIVE-20372 > Project: Hive > Issue Type: Bug > Components: Standalone Metastore, Transactions >Affects Versions: 3.1.0 >Reporter: Laszlo Bodor >Assignee: Karthik Manamcheri >Priority: Trivial > Labels: Newbie, newbie, newbie++, newbiee > Fix For: 4.0.0 > > Attachments: HIVE-20372.2.patch, HIVE-20372.3.patch, HIVE-20372.patch > > > [https://github.com/prongs/apache-hive/blob/deabe59371e98a21f4c3a58a9d8da51e4632fca5/metastore/src/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java#L765] > minor typo -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-17084) Turn on hive.stats.fetch.column.stats configuration flag
[
https://issues.apache.org/jira/browse/HIVE-17084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16603467#comment-16603467
]
Hive QA commented on HIVE-17084:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12938262/HIVE-17084.11.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 14922 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[skewjoin_mapjoin10]
(batchId=35)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_dynamic_partition_pruning_recursive_mapjoin]
(batchId=188)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/13586/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/13586/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-13586/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12938262 - PreCommit-HIVE-Build
> Turn on hive.stats.fetch.column.stats configuration flag
>
>
> Key: HIVE-17084
> URL: https://issues.apache.org/jira/browse/HIVE-17084
> Project: Hive
> Issue Type: Task
> Components: Statistics
>Reporter: Vineet Garg
>Assignee: Zoltan Haindrich
>Priority: Major
> Fix For: 3.2.0
>
> Attachments: HIVE-17084.08.patch, HIVE-17084.09.patch,
> HIVE-17084.1.patch, HIVE-17084.10.patch, HIVE-17084.11.patch,
> HIVE-170884.4.patch, HIVE-170884.5.patch, HIVE-170884.7.patch
>
>
> This flag is off by default and could result in bad plans due to missing
> column statistics.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20306) Implement projection spec for fetching only requested fields from partitions
[ https://issues.apache.org/jira/browse/HIVE-20306?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexander Kolbasov updated HIVE-20306: -- Attachment: HIVE-20306.12.patch > Implement projection spec for fetching only requested fields from partitions > > > Key: HIVE-20306 > URL: https://issues.apache.org/jira/browse/HIVE-20306 > Project: Hive > Issue Type: Sub-task >Reporter: Vihang Karajgaonkar >Assignee: Alexander Kolbasov >Priority: Major > Attachments: HIVE-20306.02.patch, HIVE-20306.03.patch, > HIVE-20306.04.patch, HIVE-20306.05.patch, HIVE-20306.06.patch, > HIVE-20306.07.patch, HIVE-20306.08.patch, HIVE-20306.09.patch, > HIVE-20306.10.patch, HIVE-20306.11.patch, HIVE-20306.12.patch, > HIVE-20306.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-17084) Turn on hive.stats.fetch.column.stats configuration flag
[
https://issues.apache.org/jira/browse/HIVE-17084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16603458#comment-16603458
]
Hive QA commented on HIVE-17084:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
8s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
57s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
53s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
10s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
33s{color} | {color:blue} common in master has 64 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
31s{color} | {color:blue} accumulo-handler in master has 21 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
24s{color} | {color:blue} contrib in master has 13 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
33s{color} | {color:blue} hbase-handler in master has 15 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
1s{color} | {color:blue} ql in master has 2310 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
11s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
11s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
1s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 6m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
14s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 56m 41s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-13586/dev-support/hive-personality.sh
|
| git revision | master / 8b73fbb |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: common accumulo-handler contrib hbase-handler
itests/hive-blobstore ql U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-13586/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Turn on hive.stats.fetch.column.stats configuration flag
>
>
> Key: HIVE-17084
> URL: https://issues.apache.org/jira/browse/HIVE-17084
> Project: Hive
> Issue Type: Task
> Components: Statistics
>Reporter: Vineet Garg
>Assignee: Zoltan Haindrich
>Priority: Major
> Fix For: 3.2.0
>
> Attachments: HIVE-17084.08.patch, HIVE-17084.09.patch,
> HIVE-17084.1.patch, HIVE-17084.10.patch, HIVE-17084.11.patch,
> HIVE-170884.4.patch, HIVE-170884.5.patch, HIVE-170884.7.patch
>
>
> This flag is off by default and could result in
[jira] [Assigned] (HIVE-20381) Vectorization: Reduce dedup of GroupBy + PTF turns off vectorization
[
https://issues.apache.org/jira/browse/HIVE-20381?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt McCline reassigned HIVE-20381:
---
Assignee: (was: Matt McCline)
> Vectorization: Reduce dedup of GroupBy + PTF turns off vectorization
>
>
> Key: HIVE-20381
> URL: https://issues.apache.org/jira/browse/HIVE-20381
> Project: Hive
> Issue Type: Bug
> Components: Vectorization
>Affects Versions: 3.1.0, 4.0.0
>Reporter: Gopal V
>Priority: Major
>
> One of the PTF Reducers in Query51 is not vectorized because there's a reduce
> deduplication which combines a group-by and a windowing shuffle.
> {code}
> | Reducer 8 |
> | Execution mode: llap |
> | Reduce Vectorization: |
> | enabled: true |
> | enableConditionsMet:
> hive.vectorized.execution.reduce.enabled IS true, hive.execution.engine tez
> IN [tez, spark] IS true |
> | notVectorizedReason: PTF operator: Only PTF directly under
> reduce-shuffle is supported |
> | vectorized: false
> {code}
> It vectorizes all PTF vertices (after HIVE-20367), with {{ set
> hive.optimize.reducededuplication=false;}}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20423) Set NULLS LAST as the default null ordering
[ https://issues.apache.org/jira/browse/HIVE-20423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16603435#comment-16603435 ] Jesus Camacho Rodriguez commented on HIVE-20423: [~teddy.choi], I left a couple of comments in the PR. Thanks > Set NULLS LAST as the default null ordering > --- > > Key: HIVE-20423 > URL: https://issues.apache.org/jira/browse/HIVE-20423 > Project: Hive > Issue Type: Bug >Reporter: Teddy Choi >Assignee: Teddy Choi >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0, 3.2.0 > > Attachments: HIVE-20423.1.patch, HIVE-20423.2.patch, > HIVE-20423.3.patch, HIVE-20423.4.patch, HIVE-20423.4.patch, > HIVE-20423.5.patch, HIVE-20423.6.patch > > > HIVE-20150 TopNKeyOperator pushdown can be more efficient if NULLS LAST > becomes the default null ordering. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20499) GetTablesOperation pull all the tables meta irrespective of auth.
[
https://issues.apache.org/jira/browse/HIVE-20499?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rajkumar Singh updated HIVE-20499:
--
Description:
GetTablesOperation pull all the tables meta irrespective of auth.
dbvisualizer and other ui based jdbc client pull tableemta similar to following
operation:
{code}
ResultSet res = con.getMetaData().getTables("", "", "%", new String[] {
"TABLE", "VIEW" });
{code}
https://github.com/rajkrrsingh/HiveServer2JDBCSample/blob/master/src/main/java/TestConnection.java#L20
was:
GetTablesOperation pull all the tables meta irrespective of auth.
Steps to reproduce:
{code}
ResultSet res = con.getMetaData().getTables("", "", "%", new String[] {
"TABLE", "VIEW" });
{code}
https://github.com/rajkrrsingh/HiveServer2JDBCSample/blob/master/src/main/java/TestConnection.java#L20
> GetTablesOperation pull all the tables meta irrespective of auth.
> -
>
> Key: HIVE-20499
> URL: https://issues.apache.org/jira/browse/HIVE-20499
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 3.1.0
> Environment: hive-3,java-8,sqlstdaut/ranger auth enabled.
>Reporter: Rajkumar Singh
>Assignee: Rajkumar Singh
>Priority: Major
>
> GetTablesOperation pull all the tables meta irrespective of auth.
> dbvisualizer and other ui based jdbc client pull tableemta similar to
> following operation:
> {code}
> ResultSet res = con.getMetaData().getTables("", "", "%", new String[] {
> "TABLE", "VIEW" });
> {code}
> https://github.com/rajkrrsingh/HiveServer2JDBCSample/blob/master/src/main/java/TestConnection.java#L20
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work started] (HIVE-20499) GetTablesOperation pull all the tables meta irrespective of auth.
[
https://issues.apache.org/jira/browse/HIVE-20499?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on HIVE-20499 started by Rajkumar Singh.
-
> GetTablesOperation pull all the tables meta irrespective of auth.
> -
>
> Key: HIVE-20499
> URL: https://issues.apache.org/jira/browse/HIVE-20499
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 3.1.0
> Environment: hive-3,java-8,sqlstdaut/ranger auth enabled.
>Reporter: Rajkumar Singh
>Assignee: Rajkumar Singh
>Priority: Major
>
> GetTablesOperation pull all the tables meta irrespective of auth.
> Steps to reproduce:
> {code}
> ResultSet res = con.getMetaData().getTables("", "", "%", new String[] {
> "TABLE", "VIEW" });
> {code}
> https://github.com/rajkrrsingh/HiveServer2JDBCSample/blob/master/src/main/java/TestConnection.java#L20
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (HIVE-20499) GetTablesOperation pull all the tables meta irrespective of auth.
[
https://issues.apache.org/jira/browse/HIVE-20499?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rajkumar Singh reassigned HIVE-20499:
-
> GetTablesOperation pull all the tables meta irrespective of auth.
> -
>
> Key: HIVE-20499
> URL: https://issues.apache.org/jira/browse/HIVE-20499
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 3.1.0
> Environment: hive-3,java-8,sqlstdaut/ranger auth enabled.
>Reporter: Rajkumar Singh
>Assignee: Rajkumar Singh
>Priority: Major
>
> GetTablesOperation pull all the tables meta irrespective of auth.
> Steps to reproduce:
> {code}
> ResultSet res = con.getMetaData().getTables("", "", "%", new String[] {
> "TABLE", "VIEW" });
> {code}
> https://github.com/rajkrrsingh/HiveServer2JDBCSample/blob/master/src/main/java/TestConnection.java#L20
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18767) Some alterPartitions invocations throw 'NumberFormatException: null'
[
https://issues.apache.org/jira/browse/HIVE-18767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16603392#comment-16603392
]
Hive QA commented on HIVE-18767:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12938261/HIVE-18767.2-branch-2.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 15 failed/errored test(s), 10697 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestAccumuloCliDriver.testCliDriver[accumulo_queries]
(batchId=227)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[avro_tableproperty_optimize]
(batchId=22)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[explaindenpendencydiffengs]
(batchId=38)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[llap_smb]
(batchId=142)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[orc_ppd_basic]
(batchId=139)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[table_nonprintable]
(batchId=140)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[join_acid_non_acid]
(batchId=158)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[union_fast_stats]
(batchId=153)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vectorized_parquet_types]
(batchId=155)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[merge_negative_5]
(batchId=88)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[explaindenpendencydiffengs]
(batchId=115)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[vectorization_input_format_excludes]
(batchId=117)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[vectorized_ptf]
(batchId=125)
org.apache.hive.hcatalog.api.TestHCatClient.testTransportFailure (batchId=176)
org.apache.hive.jdbc.TestJdbcDriver2.testSelectExecAsync2 (batchId=222)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/13585/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/13585/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-13585/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 15 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12938261 - PreCommit-HIVE-Build
> Some alterPartitions invocations throw 'NumberFormatException: null'
>
>
> Key: HIVE-18767
> URL: https://issues.apache.org/jira/browse/HIVE-18767
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 2.3.3, 3.1.0, 4.0.0, 3.2.0
>Reporter: Yuming Wang
>Assignee: Mass Dosage
>Priority: Major
> Fix For: 2.3.3, 4.0.0
>
> Attachments: HIVE-18767-branch-2.3.patch, HIVE-18767-branch-2.patch,
> HIVE-18767-branch-3.1.patch, HIVE-18767-branch-3.patch, HIVE-18767.1.patch,
> HIVE-18767.2-branch-2.3.patch, HIVE-18767.2-branch-2.patch,
> HIVE-18767.2.patch, HIVE-18767.3.patch, HIVE-18767.4.patch,
> HIVE-18767.5.patch, HIVE-18767.6.patch
>
>
> Error messages:
> {noformat}
> [info] Cause: java.lang.NumberFormatException: null
> [info] at java.lang.Long.parseLong(Long.java:552)
> [info] at java.lang.Long.parseLong(Long.java:631)
> [info] at
> org.apache.hadoop.hive.metastore.MetaStoreUtils.isFastStatsSame(MetaStoreUtils.java:315)
> [info] at
> org.apache.hadoop.hive.metastore.HiveAlterHandler.alterPartitions(HiveAlterHandler.java:605)
> [info] at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.alter_partitions_with_environment_context(HiveMetaStore.java:3837)
> [info] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> [info] at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> [info] at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> [info] at java.lang.reflect.Method.invoke(Method.java:498)
> [info] at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:148)
> [info] at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:107)
> [info] at
> com.sun.proxy.$Proxy23.alter_partitions_with_environment_context(Unknown
> Source)
> [info] at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.alter_partitions(HiveMetaStoreClient.java:1527)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20432) Rewrite BETWEEN to IN for integer types for stats estimation
[
https://issues.apache.org/jira/browse/HIVE-20432?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vineet Garg updated HIVE-20432:
---
Status: Open (was: Patch Available)
> Rewrite BETWEEN to IN for integer types for stats estimation
>
>
> Key: HIVE-20432
> URL: https://issues.apache.org/jira/browse/HIVE-20432
> Project: Hive
> Issue Type: Improvement
>Affects Versions: 4.0.0
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
> Attachments: HIVE-20432.1.patch, HIVE-20432.2.patch,
> HIVE-20432.3.patch
>
>
> IN clause provides better statistics estimation since predicates are equality
> based and it keeps into account selectivity of all values. BETWEEN is
> rewritten into {{>= AND <=}} i.e. non-equality predicates which cause worse
> estimation. For integer types BETWEEN could be rewritten into IN e.g. BETWEEN
> 4 and 7 is equivalent to IN (4,5,6,7).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20432) Rewrite BETWEEN to IN for integer types for stats estimation
[
https://issues.apache.org/jira/browse/HIVE-20432?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vineet Garg updated HIVE-20432:
---
Status: Patch Available (was: Open)
> Rewrite BETWEEN to IN for integer types for stats estimation
>
>
> Key: HIVE-20432
> URL: https://issues.apache.org/jira/browse/HIVE-20432
> Project: Hive
> Issue Type: Improvement
>Affects Versions: 4.0.0
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
> Attachments: HIVE-20432.1.patch, HIVE-20432.2.patch,
> HIVE-20432.3.patch
>
>
> IN clause provides better statistics estimation since predicates are equality
> based and it keeps into account selectivity of all values. BETWEEN is
> rewritten into {{>= AND <=}} i.e. non-equality predicates which cause worse
> estimation. For integer types BETWEEN could be rewritten into IN e.g. BETWEEN
> 4 and 7 is equivalent to IN (4,5,6,7).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20296) Improve HivePointLookupOptimizerRule to be able to extract from more sophisticated contexts
[
https://issues.apache.org/jira/browse/HIVE-20296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16603376#comment-16603376
]
Zoltan Haindrich commented on HIVE-20296:
-
[~ashutoshc] Could you please take a look?
> Improve HivePointLookupOptimizerRule to be able to extract from more
> sophisticated contexts
> ---
>
> Key: HIVE-20296
> URL: https://issues.apache.org/jira/browse/HIVE-20296
> Project: Hive
> Issue Type: Improvement
> Components: Logical Optimizer
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Attachments: HIVE-20296.01.patch, HIVE-20296.01wip01.patch,
> HIVE-20296.01wip02.patch, HIVE-20296.01wip03.patch, HIVE-20296.02.patch,
> HIVE-20296.02.patch
>
>
> Currently it could handle conditions in the form:
> {code}
> ( ( a=1 && b=1 && ... ) || (a=2 && b=2 && ...) || ... )
> {code}
> If there is some noise somewhere in the or like:
> {code}
> ( x=18 || ( a=1 && b=1 && ... ) || (a=2 && b=2 && ...) || ... )
> {code}
> it stops processing.
> https://github.com/apache/hive/blob/2cabb8da150b8fb980223fbd6c2c93b842ca3ee5/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HivePointLookupOptimizerRule.java#L238
> HIVE-19097 right now unwinds some ors which can't be packed back; like query15
> https://github.com/apache/hive/blob/2cabb8da150b8fb980223fbd6c2c93b842ca3ee5/ql/src/test/queries/clientpositive/perf/query15.q#L14
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20225) SerDe to support Teradata Binary Format
[
https://issues.apache.org/jira/browse/HIVE-20225?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lu Li updated HIVE-20225:
-
Status: Patch Available (was: In Progress)
> SerDe to support Teradata Binary Format
> ---
>
> Key: HIVE-20225
> URL: https://issues.apache.org/jira/browse/HIVE-20225
> Project: Hive
> Issue Type: New Feature
> Components: Serializers/Deserializers
>Reporter: Lu Li
>Assignee: Lu Li
>Priority: Major
> Attachments: HIVE-20225.1.patch, HIVE-20225.10.patch,
> HIVE-20225.11.patch, HIVE-20225.12.patch, HIVE-20225.13.patch,
> HIVE-20225.14-branch-2.patch, HIVE-20225.15.patch,
> HIVE-20225.16-branch-2.patch, HIVE-20225.17-branch-3.patch,
> HIVE-20225.18-branch-3.patch, HIVE-20225.2.patch, HIVE-20225.3.patch,
> HIVE-20225.4.patch, HIVE-20225.5-branch-2.patch, HIVE-20225.6.patch,
> HIVE-20225.7.patch, HIVE-20225.8.patch, HIVE-20225.9.patch
>
>
> When using TPT/BTEQ to export/import Data from Teradata, Teradata will
> generate/require binary files based on the schema.
> A Customized SerDe is needed in order to directly read these files from Hive
> or write these files in order to load back to TD.
> {code:java}
> CREATE EXTERNAL TABLE `TABLE1`(
> ...)
> PARTITIONED BY (
> ...)
> ROW FORMAT SERDE
> 'org.apache.hadoop.hive.contrib.serde2.TeradataBinarySerde'
> STORED AS INPUTFORMAT
>
> 'org.apache.hadoop.hive.contrib.fileformat.teradata.TeradataBinaryFileInputFormat'
> OUTPUTFORMAT
>
> 'org.apache.hadoop.hive.contrib.fileformat.teradata.TeradataBinaryFileOutputFormat'
> LOCATION ...;
> SELECT * FROM `TABLE1`;{code}
> Problem Statement:
> Right now the fast way to export/import data from Teradata is using TPT.
> However, the Hive could not directly utilize/generate these binary format
> because it doesn't have a SerDe for these files.
> Result:
> Provided with the SerDe, Hive can operate upon/generate the exported Teradata
> Binary Format file transparently
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20225) SerDe to support Teradata Binary Format
[
https://issues.apache.org/jira/browse/HIVE-20225?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lu Li updated HIVE-20225:
-
Attachment: HIVE-20225.18-branch-3.patch
> SerDe to support Teradata Binary Format
> ---
>
> Key: HIVE-20225
> URL: https://issues.apache.org/jira/browse/HIVE-20225
> Project: Hive
> Issue Type: New Feature
> Components: Serializers/Deserializers
>Reporter: Lu Li
>Assignee: Lu Li
>Priority: Major
> Attachments: HIVE-20225.1.patch, HIVE-20225.10.patch,
> HIVE-20225.11.patch, HIVE-20225.12.patch, HIVE-20225.13.patch,
> HIVE-20225.14-branch-2.patch, HIVE-20225.15.patch,
> HIVE-20225.16-branch-2.patch, HIVE-20225.17-branch-3.patch,
> HIVE-20225.18-branch-3.patch, HIVE-20225.2.patch, HIVE-20225.3.patch,
> HIVE-20225.4.patch, HIVE-20225.5-branch-2.patch, HIVE-20225.6.patch,
> HIVE-20225.7.patch, HIVE-20225.8.patch, HIVE-20225.9.patch
>
>
> When using TPT/BTEQ to export/import Data from Teradata, Teradata will
> generate/require binary files based on the schema.
> A Customized SerDe is needed in order to directly read these files from Hive
> or write these files in order to load back to TD.
> {code:java}
> CREATE EXTERNAL TABLE `TABLE1`(
> ...)
> PARTITIONED BY (
> ...)
> ROW FORMAT SERDE
> 'org.apache.hadoop.hive.contrib.serde2.TeradataBinarySerde'
> STORED AS INPUTFORMAT
>
> 'org.apache.hadoop.hive.contrib.fileformat.teradata.TeradataBinaryFileInputFormat'
> OUTPUTFORMAT
>
> 'org.apache.hadoop.hive.contrib.fileformat.teradata.TeradataBinaryFileOutputFormat'
> LOCATION ...;
> SELECT * FROM `TABLE1`;{code}
> Problem Statement:
> Right now the fast way to export/import data from Teradata is using TPT.
> However, the Hive could not directly utilize/generate these binary format
> because it doesn't have a SerDe for these files.
> Result:
> Provided with the SerDe, Hive can operate upon/generate the exported Teradata
> Binary Format file transparently
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20225) SerDe to support Teradata Binary Format
[
https://issues.apache.org/jira/browse/HIVE-20225?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lu Li updated HIVE-20225:
-
Status: In Progress (was: Patch Available)
> SerDe to support Teradata Binary Format
> ---
>
> Key: HIVE-20225
> URL: https://issues.apache.org/jira/browse/HIVE-20225
> Project: Hive
> Issue Type: New Feature
> Components: Serializers/Deserializers
>Reporter: Lu Li
>Assignee: Lu Li
>Priority: Major
> Attachments: HIVE-20225.1.patch, HIVE-20225.10.patch,
> HIVE-20225.11.patch, HIVE-20225.12.patch, HIVE-20225.13.patch,
> HIVE-20225.14-branch-2.patch, HIVE-20225.15.patch,
> HIVE-20225.16-branch-2.patch, HIVE-20225.17-branch-3.patch,
> HIVE-20225.18-branch-3.patch, HIVE-20225.2.patch, HIVE-20225.3.patch,
> HIVE-20225.4.patch, HIVE-20225.5-branch-2.patch, HIVE-20225.6.patch,
> HIVE-20225.7.patch, HIVE-20225.8.patch, HIVE-20225.9.patch
>
>
> When using TPT/BTEQ to export/import Data from Teradata, Teradata will
> generate/require binary files based on the schema.
> A Customized SerDe is needed in order to directly read these files from Hive
> or write these files in order to load back to TD.
> {code:java}
> CREATE EXTERNAL TABLE `TABLE1`(
> ...)
> PARTITIONED BY (
> ...)
> ROW FORMAT SERDE
> 'org.apache.hadoop.hive.contrib.serde2.TeradataBinarySerde'
> STORED AS INPUTFORMAT
>
> 'org.apache.hadoop.hive.contrib.fileformat.teradata.TeradataBinaryFileInputFormat'
> OUTPUTFORMAT
>
> 'org.apache.hadoop.hive.contrib.fileformat.teradata.TeradataBinaryFileOutputFormat'
> LOCATION ...;
> SELECT * FROM `TABLE1`;{code}
> Problem Statement:
> Right now the fast way to export/import data from Teradata is using TPT.
> However, the Hive could not directly utilize/generate these binary format
> because it doesn't have a SerDe for these files.
> Result:
> Provided with the SerDe, Hive can operate upon/generate the exported Teradata
> Binary Format file transparently
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20395) Parallelize files move in the ql.metadata.Hive#replaceFiles
[
https://issues.apache.org/jira/browse/HIVE-20395?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16603320#comment-16603320
]
Hive QA commented on HIVE-20395:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12938256/HIVE-20395.10.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 14922 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/13584/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/13584/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-13584/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12938256 - PreCommit-HIVE-Build
> Parallelize files move in the ql.metadata.Hive#replaceFiles
> ---
>
> Key: HIVE-20395
> URL: https://issues.apache.org/jira/browse/HIVE-20395
> Project: Hive
> Issue Type: Improvement
> Components: Hive
>Affects Versions: 3.1.0
>Reporter: Igor Dvorzhak
>Assignee: Igor Dvorzhak
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-20395.10.patch
>
>
> Files move is not parallelized in `replaceFiles` method and could be very
> slow on Cloud Object Stores.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-15932) add "explain ast"
[ https://issues.apache.org/jira/browse/HIVE-15932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16603296#comment-16603296 ] Zoltan Haindrich commented on HIVE-15932: - +1 pending tests > add "explain ast" > - > > Key: HIVE-15932 > URL: https://issues.apache.org/jira/browse/HIVE-15932 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Igor Kryvenko >Priority: Major > Attachments: HIVE-15932.01.patch > > > AST was removed in explain extended in HIVE-13533; that makes sense from > users perspective for the common case, but it would be useful for Hive > developers and advanced users to see the AST to diagnose issues. "Explain > ast" command can be added to dump the AST. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (HIVE-20491) Fix mapjoin size estimations for Fast implementation
[ https://issues.apache.org/jira/browse/HIVE-20491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16603259#comment-16603259 ] Zoltan Haindrich edited comment on HIVE-20491 at 9/4/18 4:13 PM: - [~ashutoshc] yes; I was planning to do that - right now this patch only fixes the estimations; and sets the most conservative estimate(fast3) by default. I'm still working on deciding and forwarding that decision...right now I think that will not be in this patch was (Author: kgyrtkirk): yes; I was planning to do that - right now this patch only fixes the estimations; and sets the most conservative estimate(fast3) by default. I'm still working on deciding and forwarding that decision...right now I think that will not be in this patch > Fix mapjoin size estimations for Fast implementation > > > Key: HIVE-20491 > URL: https://issues.apache.org/jira/browse/HIVE-20491 > Project: Hive > Issue Type: Improvement > Components: Statistics >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Major > Attachments: HIVE-20491.01.patch, HIVE-20491.01wip02.patch, > HIVE-20491.02.patch > > > HIVE-19824 have fixed the estimations; but it calculated for the "optimized" > impl; the "fast" one has a little bit bigger footprint. > It also seems like fast is a bit overestimated at runtime...that should be > also taken care of. > | numkeys | implementation | compiler estimation | runtime estimation | > runtime measurement | ce / rm | re / rm | > | 25M | FAST | 1168435456 | 2189433712 | 1513584984 | .77 | 1.44 | > | 25M | OPTIMIZED | 1168435456 | 1191203764 | 1168439664 | 100% | 1.01 | -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20491) Fix mapjoin size estimations for Fast implementation
[ https://issues.apache.org/jira/browse/HIVE-20491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16603259#comment-16603259 ] Zoltan Haindrich commented on HIVE-20491: - yes; I was planning to do that - right now this patch only fixes the estimations; and sets the most conservative estimate(fast3) by default. I'm still working on deciding and forwarding that decision...right now I think that will not be in this patch > Fix mapjoin size estimations for Fast implementation > > > Key: HIVE-20491 > URL: https://issues.apache.org/jira/browse/HIVE-20491 > Project: Hive > Issue Type: Improvement > Components: Statistics >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Major > Attachments: HIVE-20491.01.patch, HIVE-20491.01wip02.patch, > HIVE-20491.02.patch > > > HIVE-19824 have fixed the estimations; but it calculated for the "optimized" > impl; the "fast" one has a little bit bigger footprint. > It also seems like fast is a bit overestimated at runtime...that should be > also taken care of. > | numkeys | implementation | compiler estimation | runtime estimation | > runtime measurement | ce / rm | re / rm | > | 25M | FAST | 1168435456 | 2189433712 | 1513584984 | .77 | 1.44 | > | 25M | OPTIMIZED | 1168435456 | 1191203764 | 1168439664 | 100% | 1.01 | -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20395) Parallelize files move in the ql.metadata.Hive#replaceFiles
[
https://issues.apache.org/jira/browse/HIVE-20395?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16603258#comment-16603258
]
Hive QA commented on HIVE-20395:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
29s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
8s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
42s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
3s{color} | {color:blue} ql in master has 2310 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
0s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 24m 47s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-13584/dev-support/hive-personality.sh
|
| git revision | master / 8b73fbb |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-13584/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Parallelize files move in the ql.metadata.Hive#replaceFiles
> ---
>
> Key: HIVE-20395
> URL: https://issues.apache.org/jira/browse/HIVE-20395
> Project: Hive
> Issue Type: Improvement
> Components: Hive
>Affects Versions: 3.1.0
>Reporter: Igor Dvorzhak
>Assignee: Igor Dvorzhak
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-20395.10.patch
>
>
> Files move is not parallelized in `replaceFiles` method and could be very
> slow on Cloud Object Stores.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20491) Fix mapjoin size estimations for Fast implementation
[ https://issues.apache.org/jira/browse/HIVE-20491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16603246#comment-16603246 ] Ashutosh Chauhan commented on HIVE-20491: - [~kgyrtkirk] Selection of kind of hashtable is done by Vectorizer which runs *after* ConvertJoinMapJoin which does algo selection. I see you have updated size computation assuming fast hashtable but wont it better that we first do memory computation using optimized version and then using fast. If fast qualifies set that in Join so that vectorizer can pick correct hashtable type? Though, since fast hashtables are bigger current approach also works though its more conservative than needed. > Fix mapjoin size estimations for Fast implementation > > > Key: HIVE-20491 > URL: https://issues.apache.org/jira/browse/HIVE-20491 > Project: Hive > Issue Type: Improvement > Components: Statistics >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Major > Attachments: HIVE-20491.01.patch, HIVE-20491.01wip02.patch, > HIVE-20491.02.patch > > > HIVE-19824 have fixed the estimations; but it calculated for the "optimized" > impl; the "fast" one has a little bit bigger footprint. > It also seems like fast is a bit overestimated at runtime...that should be > also taken care of. > | numkeys | implementation | compiler estimation | runtime estimation | > runtime measurement | ce / rm | re / rm | > | 25M | FAST | 1168435456 | 2189433712 | 1513584984 | .77 | 1.44 | > | 25M | OPTIMIZED | 1168435456 | 1191203764 | 1168439664 | 100% | 1.01 | -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-15932) add "explain ast"
[ https://issues.apache.org/jira/browse/HIVE-15932?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Igor Kryvenko updated HIVE-15932: - Status: Patch Available (was: Open) > add "explain ast" > - > > Key: HIVE-15932 > URL: https://issues.apache.org/jira/browse/HIVE-15932 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Igor Kryvenko >Priority: Major > Attachments: HIVE-15932.01.patch > > > AST was removed in explain extended in HIVE-13533; that makes sense from > users perspective for the common case, but it would be useful for Hive > developers and advanced users to see the AST to diagnose issues. "Explain > ast" command can be added to dump the AST. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-15932) add "explain ast"
[ https://issues.apache.org/jira/browse/HIVE-15932?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Igor Kryvenko updated HIVE-15932: - Attachment: HIVE-15932.01.patch > add "explain ast" > - > > Key: HIVE-15932 > URL: https://issues.apache.org/jira/browse/HIVE-15932 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Igor Kryvenko >Priority: Major > Attachments: HIVE-15932.01.patch > > > AST was removed in explain extended in HIVE-13533; that makes sense from > users perspective for the common case, but it would be useful for Hive > developers and advanced users to see the AST to diagnose issues. "Explain > ast" command can be added to dump the AST. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-15932) add "explain ast"
[ https://issues.apache.org/jira/browse/HIVE-15932?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Igor Kryvenko reassigned HIVE-15932: Assignee: Igor Kryvenko > add "explain ast" > - > > Key: HIVE-15932 > URL: https://issues.apache.org/jira/browse/HIVE-15932 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Igor Kryvenko >Priority: Major > > AST was removed in explain extended in HIVE-13533; that makes sense from > users perspective for the common case, but it would be useful for Hive > developers and advanced users to see the AST to diagnose issues. "Explain > ast" command can be added to dump the AST. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20491) Fix mapjoin size estimations for Fast implementation
[
https://issues.apache.org/jira/browse/HIVE-20491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16603216#comment-16603216
]
Hive QA commented on HIVE-20491:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12938251/HIVE-20491.02.patch
{color:green}SUCCESS:{color} +1 due to 3 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 14923 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/13583/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/13583/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-13583/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12938251 - PreCommit-HIVE-Build
> Fix mapjoin size estimations for Fast implementation
>
>
> Key: HIVE-20491
> URL: https://issues.apache.org/jira/browse/HIVE-20491
> Project: Hive
> Issue Type: Improvement
> Components: Statistics
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Attachments: HIVE-20491.01.patch, HIVE-20491.01wip02.patch,
> HIVE-20491.02.patch
>
>
> HIVE-19824 have fixed the estimations; but it calculated for the "optimized"
> impl; the "fast" one has a little bit bigger footprint.
> It also seems like fast is a bit overestimated at runtime...that should be
> also taken care of.
> | numkeys | implementation | compiler estimation | runtime estimation |
> runtime measurement | ce / rm | re / rm |
> | 25M | FAST | 1168435456 | 2189433712 | 1513584984 | .77 | 1.44 |
> | 25M | OPTIMIZED | 1168435456 | 1191203764 | 1168439664 | 100% | 1.01 |
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20349) Implement Retry Logic in HiveDruidSplit for Scan Queries
[ https://issues.apache.org/jira/browse/HIVE-20349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16603218#comment-16603218 ] Jesus Camacho Rodriguez commented on HIVE-20349: [~nishantbangarwa], is failure related? Should we rebase + push? Thanks > Implement Retry Logic in HiveDruidSplit for Scan Queries > > > Key: HIVE-20349 > URL: https://issues.apache.org/jira/browse/HIVE-20349 > Project: Hive > Issue Type: Bug > Components: Druid integration >Reporter: Nishant Bangarwa >Assignee: Nishant Bangarwa >Priority: Major > Attachments: HIVE-20349.1.patch, HIVE-20349.2.patch, HIVE-20349.patch > > > while distributing druid scan query we check where the segments are loaded > and then each HiveDruidSplit directly queries the historical node. > There are few cases when we need to retry and refetch the segments. > # The segment is loaded on multiple historical nodes and one of them went > down. in this case when we do not get response from one segment, we query the > next replica. > # The segment was loaded onto a realtime task and was handed over, when we > query the realtime task has already finished. In this case there is no > replica. The Split needs to query the broker again for the location of the > segment and then send the query to correct historical node. > This is also the root cause of failure of druidkafkamini_basic.q test, where > the segment handover happens before the scan query is executed. > Note: This is not a problem when we are directly querying Druid brokers as > the broker handles the retry logic. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20013) Add an Implicit cast to date type for to_date function
[
https://issues.apache.org/jira/browse/HIVE-20013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16603210#comment-16603210
]
Jesus Camacho Rodriguez commented on HIVE-20013:
Should this be pushed to branch-3 too? Thanks
> Add an Implicit cast to date type for to_date function
> --
>
> Key: HIVE-20013
> URL: https://issues.apache.org/jira/browse/HIVE-20013
> Project: Hive
> Issue Type: Bug
>Reporter: Nishant Bangarwa
>Assignee: Nishant Bangarwa
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-20013.02.patch, HIVE-20013.patch, HIVE-20013.patch
>
>
> Issue -
> SELECT TO_DATE(date1), TO_DATE(datetime1) FROM druid_table_n1;
> Running this query on Druid returns null values when date1 and datetime1 are
> of type String.
> {code}
> INFO : Executing
> command(queryId=hive_20180627144822_d4395567-e3cb-4b20-b53b-4e5eba2d7dac):
> EXPLAIN SELECT TO_DATE(datetime0) ,TO_DATE(date0) FROM calcs
> INFO : Starting task [Stage-1:EXPLAIN] in serial mode
> INFO : Completed executing
> command(queryId=hive_20180627144822_d4395567-e3cb-4b20-b53b-4e5eba2d7dac);
> Time taken: 0.003 seconds
> INFO : OK
> ++
> | Explain |
> ++
> | Plan optimized by CBO. |
> ||
> | Stage-0|
> | Fetch Operator |
> | limit:-1 |
> | Select Operator [SEL_1]|
> | Output:["_col0","_col1"] |
> | TableScan [TS_0] |
> |
> Output:["vc","vc0"],properties:{"druid.fieldNames":"vc,vc0","druid.fieldTypes":"date,date","druid.query.json":"{\"queryType\":\"scan\",\"dataSource\":\"druid_tableau.calcs\",\"intervals\":[\"1900-01-01T00:00:00.000Z/3000-01-01T00:00:00.000Z\"],\"virtualColumns\":[{\"type\":\"expression\",\"name\":\"vc\",\"expression\":\"timestamp_floor(\\\"datetime0\\\",'P1D','','UTC')\",\"outputType\":\"LONG\"},{\"type\":\"expression\",\"name\":\"vc0\",\"expression\":\"timestamp_floor(\\\"date0\\\",'P1D','','UTC')\",\"outputType\":\"LONG\"}],\"columns\":[\"vc\",\"vc0\"],\"resultFormat\":\"compactedList\"}","druid.query.type":"scan"}
> |
> ||
> ++
> 10 rows selected (0.606 seconds)
> {code}
> Reported by [~dileep529]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19552) Enable TestMiniDruidKafkaCliDriver#druidkafkamini_basic.q
[ https://issues.apache.org/jira/browse/HIVE-19552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16603196#comment-16603196 ] Jesus Camacho Rodriguez commented on HIVE-19552: +1 [~nishantbangarwa], can you rebase the patch and update the description of the issue to know what was going on + proposed solution? Thanks > Enable TestMiniDruidKafkaCliDriver#druidkafkamini_basic.q > - > > Key: HIVE-19552 > URL: https://issues.apache.org/jira/browse/HIVE-19552 > Project: Hive > Issue Type: Test > Components: Test >Affects Versions: 3.1.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Nishant Bangarwa >Priority: Critical > Attachments: HIVE-19552.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-17921) Aggregation with struct in LLAP produces wrong result
[
https://issues.apache.org/jira/browse/HIVE-17921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16603191#comment-16603191
]
Saurabh Seth commented on HIVE-17921:
-
This result is for the query:
{noformat}
select ROW__ID, count(*) from over10k_orc_bucketed group by ROW__ID having
count(*) > 1;
{noformat}
This table is an ACID table and doesn't actually have any null ROW__IDs. This
issue was logged for this specific incorrect output being present - the
description has more details as well.
> Aggregation with struct in LLAP produces wrong result
> -
>
> Key: HIVE-17921
> URL: https://issues.apache.org/jira/browse/HIVE-17921
> Project: Hive
> Issue Type: Sub-task
> Components: llap, Transactions
>Affects Versions: 3.0.0
>Reporter: Eugene Koifman
>Assignee: Saurabh Seth
>Priority: Blocker
> Attachments: HIVE-17921.2.patch, HIVE-17921.patch
>
>
> Consider
> {noformat}
> select ROW__ID, count(*) from over10k_orc_bucketed group by ROW__ID having
> count(*) > 1;
> {noformat}
> in acid_vectorization_original.q (available since HIVE-17458)
> when run using TestMiniLlapCliDriver produces "NULL, N" where N varies from
> run to run.
> The right answer is empty results set as can be seen by running
> {noformat}
> select ROW__ID, * from over10k_orc_bucketed where ROW__ID is null
> {noformat}
> in the same test.
> This is with
> {noformat}
> set hive.vectorized.execution.enabled=true;
> set hive.vectorized.row.identifier.enabled=true;
> {noformat}
> It fails with TestMiniLlapCliDriver but not TestMiniTezCliDriver. See
> acid_vectorization_original_tez.q which has identical query.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-17921) Aggregation with struct in LLAP produces wrong result
[
https://issues.apache.org/jira/browse/HIVE-17921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16603173#comment-16603173
]
Eugene Koifman commented on HIVE-17921:
---
acid_vectorization_original.q.out used to have 1 row of output from a query and
now it seems to have nothing. Could you explain why is this correct?
> Aggregation with struct in LLAP produces wrong result
> -
>
> Key: HIVE-17921
> URL: https://issues.apache.org/jira/browse/HIVE-17921
> Project: Hive
> Issue Type: Sub-task
> Components: llap, Transactions
>Affects Versions: 3.0.0
>Reporter: Eugene Koifman
>Assignee: Saurabh Seth
>Priority: Blocker
> Attachments: HIVE-17921.2.patch, HIVE-17921.patch
>
>
> Consider
> {noformat}
> select ROW__ID, count(*) from over10k_orc_bucketed group by ROW__ID having
> count(*) > 1;
> {noformat}
> in acid_vectorization_original.q (available since HIVE-17458)
> when run using TestMiniLlapCliDriver produces "NULL, N" where N varies from
> run to run.
> The right answer is empty results set as can be seen by running
> {noformat}
> select ROW__ID, * from over10k_orc_bucketed where ROW__ID is null
> {noformat}
> in the same test.
> This is with
> {noformat}
> set hive.vectorized.execution.enabled=true;
> set hive.vectorized.row.identifier.enabled=true;
> {noformat}
> It fails with TestMiniLlapCliDriver but not TestMiniTezCliDriver. See
> acid_vectorization_original_tez.q which has identical query.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (HIVE-20498) Support date type for column stats autogather
[
https://issues.apache.org/jira/browse/HIVE-20498?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Voros reassigned HIVE-20498:
---
Assignee: Daniel Voros
> Support date type for column stats autogather
> -
>
> Key: HIVE-20498
> URL: https://issues.apache.org/jira/browse/HIVE-20498
> Project: Hive
> Issue Type: Sub-task
> Components: Statistics
>Reporter: Zoltan Haindrich
>Assignee: Daniel Voros
>Priority: Major
>
> {code}
> set hive.stats.column.autogather=true;
> create table dx2(a int,b int,d date);
> explain insert into dx2 values(1,1,'2011-11-11');
> -- no compute_stats calls
> insert into dx2 values(1,1,'2011-11-11');
> insert into dx2 values(1,1,'2001-11-11');
> explain analyze table dx2 compute statistics for columns;
> -- as expected; has compute_stats calls
> analyze table dx2 compute statistics for columns;
> -- runs ok
> desc formatted dx2 d;
> -- looks good
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20491) Fix mapjoin size estimations for Fast implementation
[
https://issues.apache.org/jira/browse/HIVE-20491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16603168#comment-16603168
]
Hive QA commented on HIVE-20491:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
52s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
7s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
43s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
8s{color} | {color:blue} ql in master has 2310 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
1s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
6s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
41s{color} | {color:red} ql: The patch generated 3 new + 35 unchanged - 3 fixed
= 38 total (was 38) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
57s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 25m 1s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-13583/dev-support/hive-personality.sh
|
| git revision | master / 3287a09 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-13583/yetus/diff-checkstyle-ql.txt
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-13583/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Fix mapjoin size estimations for Fast implementation
>
>
> Key: HIVE-20491
> URL: https://issues.apache.org/jira/browse/HIVE-20491
> Project: Hive
> Issue Type: Improvement
> Components: Statistics
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Attachments: HIVE-20491.01.patch, HIVE-20491.01wip02.patch,
> HIVE-20491.02.patch
>
>
> HIVE-19824 have fixed the estimations; but it calculated for the "optimized"
> impl; the "fast" one has a little bit bigger footprint.
> It also seems like fast is a bit overestimated at runtime...that should be
> also taken care of.
> | numkeys | implementation | compiler estimation | runtime estimation |
> runtime measurement | ce / rm | re / rm |
> | 25M | FAST | 1168435456 | 2189433712 | 1513584984 | .77 | 1.44 |
> | 25M | OPTIMIZED | 1168435456 | 1191203764 | 1168439664 | 100% | 1.01 |
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19993) Using a table alias which also appears as a column name is not possible
[
https://issues.apache.org/jira/browse/HIVE-19993?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zoltan Haindrich updated HIVE-19993:
Resolution: Fixed
Fix Version/s: 4.0.0
Status: Resolved (was: Patch Available)
pushed to master. Thank you [~jmarhuen] for fixing this!
> Using a table alias which also appears as a column name is not possible
> ---
>
> Key: HIVE-19993
> URL: https://issues.apache.org/jira/browse/HIVE-19993
> Project: Hive
> Issue Type: Bug
>Reporter: Zoltan Haindrich
>Assignee: Jaume M
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-19993.1.patch, HIVE-19993.2.patch,
> HIVE-19993.3.patch
>
>
> {code}
> drop table if exists tableA;
> drop table if exists tableB;
> create table tableA (a integer,z integer);
> create table tableB (a integer,b integer,z integer);
> select a.z, b.b
> from tableB as b JOIN
> tableA as a
> on a.a=b.b;
> {code}
> {code}
> Error: Error while compiling statement: FAILED: SemanticException Column a
> Found in more than One Tables/Subqueries (state=42000,code=4)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-17921) Aggregation with struct in LLAP produces wrong result
[
https://issues.apache.org/jira/browse/HIVE-17921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16603135#comment-16603135
]
Saurabh Seth commented on HIVE-17921:
-
The test failures are unrelated to this patch. This patch fixes an existing
test result so I haven't added any more tests.
> Aggregation with struct in LLAP produces wrong result
> -
>
> Key: HIVE-17921
> URL: https://issues.apache.org/jira/browse/HIVE-17921
> Project: Hive
> Issue Type: Sub-task
> Components: llap, Transactions
>Affects Versions: 3.0.0
>Reporter: Eugene Koifman
>Assignee: Saurabh Seth
>Priority: Blocker
> Attachments: HIVE-17921.2.patch, HIVE-17921.patch
>
>
> Consider
> {noformat}
> select ROW__ID, count(*) from over10k_orc_bucketed group by ROW__ID having
> count(*) > 1;
> {noformat}
> in acid_vectorization_original.q (available since HIVE-17458)
> when run using TestMiniLlapCliDriver produces "NULL, N" where N varies from
> run to run.
> The right answer is empty results set as can be seen by running
> {noformat}
> select ROW__ID, * from over10k_orc_bucketed where ROW__ID is null
> {noformat}
> in the same test.
> This is with
> {noformat}
> set hive.vectorized.execution.enabled=true;
> set hive.vectorized.row.identifier.enabled=true;
> {noformat}
> It fails with TestMiniLlapCliDriver but not TestMiniTezCliDriver. See
> acid_vectorization_original_tez.q which has identical query.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-17084) Turn on hive.stats.fetch.column.stats configuration flag
[ https://issues.apache.org/jira/browse/HIVE-17084?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Haindrich updated HIVE-17084: Attachment: HIVE-17084.11.patch > Turn on hive.stats.fetch.column.stats configuration flag > > > Key: HIVE-17084 > URL: https://issues.apache.org/jira/browse/HIVE-17084 > Project: Hive > Issue Type: Task > Components: Statistics >Reporter: Vineet Garg >Assignee: Zoltan Haindrich >Priority: Major > Fix For: 3.2.0 > > Attachments: HIVE-17084.08.patch, HIVE-17084.09.patch, > HIVE-17084.1.patch, HIVE-17084.10.patch, HIVE-17084.11.patch, > HIVE-170884.4.patch, HIVE-170884.5.patch, HIVE-170884.7.patch > > > This flag is off by default and could result in bad plans due to missing > column statistics. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18767) Some alterPartitions invocations throw 'NumberFormatException: null'
[
https://issues.apache.org/jira/browse/HIVE-18767?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mass Dosage updated HIVE-18767:
---
Target Version/s: 3.1.0, 2.3.3, 4.0.0, 3.2.0 (was: 2.3.3, 3.1.0, 4.0.0,
3.2.0)
Status: Patch Available (was: In Progress)
> Some alterPartitions invocations throw 'NumberFormatException: null'
>
>
> Key: HIVE-18767
> URL: https://issues.apache.org/jira/browse/HIVE-18767
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 3.1.0, 2.3.3, 4.0.0, 3.2.0
>Reporter: Yuming Wang
>Assignee: Mass Dosage
>Priority: Major
> Fix For: 2.3.3, 4.0.0
>
> Attachments: HIVE-18767-branch-2.3.patch, HIVE-18767-branch-2.patch,
> HIVE-18767-branch-3.1.patch, HIVE-18767-branch-3.patch, HIVE-18767.1.patch,
> HIVE-18767.2-branch-2.3.patch, HIVE-18767.2-branch-2.patch,
> HIVE-18767.2.patch, HIVE-18767.3.patch, HIVE-18767.4.patch,
> HIVE-18767.5.patch, HIVE-18767.6.patch
>
>
> Error messages:
> {noformat}
> [info] Cause: java.lang.NumberFormatException: null
> [info] at java.lang.Long.parseLong(Long.java:552)
> [info] at java.lang.Long.parseLong(Long.java:631)
> [info] at
> org.apache.hadoop.hive.metastore.MetaStoreUtils.isFastStatsSame(MetaStoreUtils.java:315)
> [info] at
> org.apache.hadoop.hive.metastore.HiveAlterHandler.alterPartitions(HiveAlterHandler.java:605)
> [info] at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.alter_partitions_with_environment_context(HiveMetaStore.java:3837)
> [info] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> [info] at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> [info] at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> [info] at java.lang.reflect.Method.invoke(Method.java:498)
> [info] at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:148)
> [info] at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:107)
> [info] at
> com.sun.proxy.$Proxy23.alter_partitions_with_environment_context(Unknown
> Source)
> [info] at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.alter_partitions(HiveMetaStoreClient.java:1527)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18767) Some alterPartitions invocations throw 'NumberFormatException: null'
[
https://issues.apache.org/jira/browse/HIVE-18767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16603124#comment-16603124
]
Mass Dosage commented on HIVE-18767:
Added HIVE-18767.2-branch-2.patch to apply changes to "branch-2" for a
potential Hive 2.4.x release.
> Some alterPartitions invocations throw 'NumberFormatException: null'
>
>
> Key: HIVE-18767
> URL: https://issues.apache.org/jira/browse/HIVE-18767
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 2.3.3, 3.1.0, 4.0.0, 3.2.0
>Reporter: Yuming Wang
>Assignee: Mass Dosage
>Priority: Major
> Fix For: 2.3.3, 4.0.0
>
> Attachments: HIVE-18767-branch-2.3.patch, HIVE-18767-branch-2.patch,
> HIVE-18767-branch-3.1.patch, HIVE-18767-branch-3.patch, HIVE-18767.1.patch,
> HIVE-18767.2-branch-2.3.patch, HIVE-18767.2-branch-2.patch,
> HIVE-18767.2.patch, HIVE-18767.3.patch, HIVE-18767.4.patch,
> HIVE-18767.5.patch, HIVE-18767.6.patch
>
>
> Error messages:
> {noformat}
> [info] Cause: java.lang.NumberFormatException: null
> [info] at java.lang.Long.parseLong(Long.java:552)
> [info] at java.lang.Long.parseLong(Long.java:631)
> [info] at
> org.apache.hadoop.hive.metastore.MetaStoreUtils.isFastStatsSame(MetaStoreUtils.java:315)
> [info] at
> org.apache.hadoop.hive.metastore.HiveAlterHandler.alterPartitions(HiveAlterHandler.java:605)
> [info] at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.alter_partitions_with_environment_context(HiveMetaStore.java:3837)
> [info] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> [info] at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> [info] at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> [info] at java.lang.reflect.Method.invoke(Method.java:498)
> [info] at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:148)
> [info] at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:107)
> [info] at
> com.sun.proxy.$Proxy23.alter_partitions_with_environment_context(Unknown
> Source)
> [info] at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.alter_partitions(HiveMetaStoreClient.java:1527)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18767) Some alterPartitions invocations throw 'NumberFormatException: null'
[
https://issues.apache.org/jira/browse/HIVE-18767?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mass Dosage updated HIVE-18767:
---
Attachment: HIVE-18767.2-branch-2.patch
> Some alterPartitions invocations throw 'NumberFormatException: null'
>
>
> Key: HIVE-18767
> URL: https://issues.apache.org/jira/browse/HIVE-18767
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 2.3.3, 3.1.0, 4.0.0, 3.2.0
>Reporter: Yuming Wang
>Assignee: Mass Dosage
>Priority: Major
> Fix For: 2.3.3, 4.0.0
>
> Attachments: HIVE-18767-branch-2.3.patch, HIVE-18767-branch-2.patch,
> HIVE-18767-branch-3.1.patch, HIVE-18767-branch-3.patch, HIVE-18767.1.patch,
> HIVE-18767.2-branch-2.3.patch, HIVE-18767.2-branch-2.patch,
> HIVE-18767.2.patch, HIVE-18767.3.patch, HIVE-18767.4.patch,
> HIVE-18767.5.patch, HIVE-18767.6.patch
>
>
> Error messages:
> {noformat}
> [info] Cause: java.lang.NumberFormatException: null
> [info] at java.lang.Long.parseLong(Long.java:552)
> [info] at java.lang.Long.parseLong(Long.java:631)
> [info] at
> org.apache.hadoop.hive.metastore.MetaStoreUtils.isFastStatsSame(MetaStoreUtils.java:315)
> [info] at
> org.apache.hadoop.hive.metastore.HiveAlterHandler.alterPartitions(HiveAlterHandler.java:605)
> [info] at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.alter_partitions_with_environment_context(HiveMetaStore.java:3837)
> [info] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> [info] at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> [info] at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> [info] at java.lang.reflect.Method.invoke(Method.java:498)
> [info] at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:148)
> [info] at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:107)
> [info] at
> com.sun.proxy.$Proxy23.alter_partitions_with_environment_context(Unknown
> Source)
> [info] at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.alter_partitions(HiveMetaStoreClient.java:1527)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18767) Some alterPartitions invocations throw 'NumberFormatException: null'
[
https://issues.apache.org/jira/browse/HIVE-18767?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mass Dosage updated HIVE-18767:
---
Target Version/s: 3.1.0, 2.3.3, 4.0.0, 3.2.0 (was: 2.3.3, 3.1.0, 4.0.0,
3.2.0)
Status: In Progress (was: Patch Available)
> Some alterPartitions invocations throw 'NumberFormatException: null'
>
>
> Key: HIVE-18767
> URL: https://issues.apache.org/jira/browse/HIVE-18767
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 3.1.0, 2.3.3, 4.0.0, 3.2.0
>Reporter: Yuming Wang
>Assignee: Mass Dosage
>Priority: Major
> Fix For: 2.3.3, 4.0.0
>
> Attachments: HIVE-18767-branch-2.3.patch, HIVE-18767-branch-2.patch,
> HIVE-18767-branch-3.1.patch, HIVE-18767-branch-3.patch, HIVE-18767.1.patch,
> HIVE-18767.2-branch-2.3.patch, HIVE-18767.2-branch-2.patch,
> HIVE-18767.2.patch, HIVE-18767.3.patch, HIVE-18767.4.patch,
> HIVE-18767.5.patch, HIVE-18767.6.patch
>
>
> Error messages:
> {noformat}
> [info] Cause: java.lang.NumberFormatException: null
> [info] at java.lang.Long.parseLong(Long.java:552)
> [info] at java.lang.Long.parseLong(Long.java:631)
> [info] at
> org.apache.hadoop.hive.metastore.MetaStoreUtils.isFastStatsSame(MetaStoreUtils.java:315)
> [info] at
> org.apache.hadoop.hive.metastore.HiveAlterHandler.alterPartitions(HiveAlterHandler.java:605)
> [info] at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.alter_partitions_with_environment_context(HiveMetaStore.java:3837)
> [info] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> [info] at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> [info] at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> [info] at java.lang.reflect.Method.invoke(Method.java:498)
> [info] at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:148)
> [info] at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:107)
> [info] at
> com.sun.proxy.$Proxy23.alter_partitions_with_environment_context(Unknown
> Source)
> [info] at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.alter_partitions(HiveMetaStoreClient.java:1527)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18767) Some alterPartitions invocations throw 'NumberFormatException: null'
[
https://issues.apache.org/jira/browse/HIVE-18767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16603117#comment-16603117
]
Mass Dosage commented on HIVE-18767:
OK, so it makes sense that the above patch failed since the code in the patch
is already merged into "branch-2.3" (it just missed the 2.3.3 release for some
reason). I'll move on to the next one.
> Some alterPartitions invocations throw 'NumberFormatException: null'
>
>
> Key: HIVE-18767
> URL: https://issues.apache.org/jira/browse/HIVE-18767
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 2.3.3, 3.1.0, 4.0.0, 3.2.0
>Reporter: Yuming Wang
>Assignee: Mass Dosage
>Priority: Major
> Fix For: 2.3.3, 4.0.0
>
> Attachments: HIVE-18767-branch-2.3.patch, HIVE-18767-branch-2.patch,
> HIVE-18767-branch-3.1.patch, HIVE-18767-branch-3.patch, HIVE-18767.1.patch,
> HIVE-18767.2-branch-2.3.patch, HIVE-18767.2.patch, HIVE-18767.3.patch,
> HIVE-18767.4.patch, HIVE-18767.5.patch, HIVE-18767.6.patch
>
>
> Error messages:
> {noformat}
> [info] Cause: java.lang.NumberFormatException: null
> [info] at java.lang.Long.parseLong(Long.java:552)
> [info] at java.lang.Long.parseLong(Long.java:631)
> [info] at
> org.apache.hadoop.hive.metastore.MetaStoreUtils.isFastStatsSame(MetaStoreUtils.java:315)
> [info] at
> org.apache.hadoop.hive.metastore.HiveAlterHandler.alterPartitions(HiveAlterHandler.java:605)
> [info] at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.alter_partitions_with_environment_context(HiveMetaStore.java:3837)
> [info] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> [info] at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> [info] at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> [info] at java.lang.reflect.Method.invoke(Method.java:498)
> [info] at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:148)
> [info] at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:107)
> [info] at
> com.sun.proxy.$Proxy23.alter_partitions_with_environment_context(Unknown
> Source)
> [info] at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.alter_partitions(HiveMetaStoreClient.java:1527)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-17084) Turn on hive.stats.fetch.column.stats configuration flag
[
https://issues.apache.org/jira/browse/HIVE-17084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16603112#comment-16603112
]
Hive QA commented on HIVE-17084:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12938249/HIVE-17084.10.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 26 failed/errored test(s), 14921 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[fold_eq_with_case_when]
(batchId=86)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[list_bucket_query_oneskew_2]
(batchId=3)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[load_dyn_part4]
(batchId=67)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[masking_1] (batchId=90)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[query_result_fileformat]
(batchId=21)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[semijoin4] (batchId=91)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[skewjoin_mapjoin10]
(batchId=35)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_case_when_1]
(batchId=90)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vectorized_casts]
(batchId=87)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_dynamic_partition_pruning]
(batchId=187)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_dynamic_partition_pruning_recursive_mapjoin]
(batchId=188)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_vectorized_dynamic_partition_pruning]
(batchId=187)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[bucket_map_join_tez2]
(batchId=114)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[join32_lessSize]
(batchId=112)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[subquery_in]
(batchId=138)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[subquery_notin]
(batchId=141)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[subquery_scalar]
(batchId=127)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[vectorized_ptf]
(batchId=138)
org.apache.hadoop.hive.cli.TestSparkPerfCliDriver.testCliDriver[query23]
(batchId=266)
org.apache.hadoop.hive.cli.TestSparkPerfCliDriver.testCliDriver[query44]
(batchId=266)
org.apache.hadoop.hive.cli.TestSparkPerfCliDriver.testCliDriver[query47]
(batchId=266)
org.apache.hadoop.hive.cli.TestSparkPerfCliDriver.testCliDriver[query57]
(batchId=266)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query44]
(batchId=264)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query47]
(batchId=264)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query51]
(batchId=264)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query57]
(batchId=264)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/13582/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/13582/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-13582/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 26 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12938249 - PreCommit-HIVE-Build
> Turn on hive.stats.fetch.column.stats configuration flag
>
>
> Key: HIVE-17084
> URL: https://issues.apache.org/jira/browse/HIVE-17084
> Project: Hive
> Issue Type: Task
> Components: Statistics
>Reporter: Vineet Garg
>Assignee: Zoltan Haindrich
>Priority: Major
> Fix For: 3.2.0
>
> Attachments: HIVE-17084.08.patch, HIVE-17084.09.patch,
> HIVE-17084.1.patch, HIVE-17084.10.patch, HIVE-170884.4.patch,
> HIVE-170884.5.patch, HIVE-170884.7.patch
>
>
> This flag is off by default and could result in bad plans due to missing
> column statistics.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-17084) Turn on hive.stats.fetch.column.stats configuration flag
[
https://issues.apache.org/jira/browse/HIVE-17084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16603109#comment-16603109
]
Hive QA commented on HIVE-17084:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
25s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
0s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
52s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
7s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
33s{color} | {color:blue} common in master has 64 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
32s{color} | {color:blue} accumulo-handler in master has 21 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
26s{color} | {color:blue} contrib in master has 13 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
31s{color} | {color:blue} hbase-handler in master has 15 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
12s{color} | {color:blue} ql in master has 2310 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
12s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
10s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
1s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 6m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
14s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 57m 7s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-13582/dev-support/hive-personality.sh
|
| git revision | master / 3287a09 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: common accumulo-handler contrib hbase-handler
itests/hive-blobstore ql U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-13582/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Turn on hive.stats.fetch.column.stats configuration flag
>
>
> Key: HIVE-17084
> URL: https://issues.apache.org/jira/browse/HIVE-17084
> Project: Hive
> Issue Type: Task
> Components: Statistics
>Reporter: Vineet Garg
>Assignee: Zoltan Haindrich
>Priority: Major
> Fix For: 3.2.0
>
> Attachments: HIVE-17084.08.patch, HIVE-17084.09.patch,
> HIVE-17084.1.patch, HIVE-17084.10.patch, HIVE-170884.4.patch,
> HIVE-170884.5.patch, HIVE-170884.7.patch
>
>
> This flag is off by default and could result in bad plans due to
[jira] [Updated] (HIVE-20395) Parallelize files move in the ql.metadata.Hive#replaceFiles
[ https://issues.apache.org/jira/browse/HIVE-20395?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Igor Dvorzhak updated HIVE-20395: - Attachment: (was: HIVE-20395.patch) > Parallelize files move in the ql.metadata.Hive#replaceFiles > --- > > Key: HIVE-20395 > URL: https://issues.apache.org/jira/browse/HIVE-20395 > Project: Hive > Issue Type: Improvement > Components: Hive >Affects Versions: 3.1.0 >Reporter: Igor Dvorzhak >Assignee: Igor Dvorzhak >Priority: Major > Labels: pull-request-available > Attachments: HIVE-20395.10.patch > > > Files move is not parallelized in `replaceFiles` method and could be very > slow on Cloud Object Stores. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20395) Parallelize files move in the ql.metadata.Hive#replaceFiles
[ https://issues.apache.org/jira/browse/HIVE-20395?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Igor Dvorzhak updated HIVE-20395: - Attachment: HIVE-20395.10.patch > Parallelize files move in the ql.metadata.Hive#replaceFiles > --- > > Key: HIVE-20395 > URL: https://issues.apache.org/jira/browse/HIVE-20395 > Project: Hive > Issue Type: Improvement > Components: Hive >Affects Versions: 3.1.0 >Reporter: Igor Dvorzhak >Assignee: Igor Dvorzhak >Priority: Major > Labels: pull-request-available > Attachments: HIVE-20395.10.patch > > > Files move is not parallelized in `replaceFiles` method and could be very > slow on Cloud Object Stores. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20498) Support date type for column stats autogather
[
https://issues.apache.org/jira/browse/HIVE-20498?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zoltan Haindrich updated HIVE-20498:
Description:
{code}
set hive.stats.column.autogather=true;
create table dx2(a int,b int,d date);
explain insert into dx2 values(1,1,'2011-11-11');
-- no compute_stats calls
insert into dx2 values(1,1,'2011-11-11');
insert into dx2 values(1,1,'2001-11-11');
explain analyze table dx2 compute statistics for columns;
-- as expected; has compute_stats calls
analyze table dx2 compute statistics for columns;
-- runs ok
desc formatted dx2 d;
-- looks good
{code}
> Support date type for column stats autogather
> -
>
> Key: HIVE-20498
> URL: https://issues.apache.org/jira/browse/HIVE-20498
> Project: Hive
> Issue Type: Sub-task
> Components: Statistics
>Reporter: Zoltan Haindrich
>Priority: Major
>
> {code}
> set hive.stats.column.autogather=true;
> create table dx2(a int,b int,d date);
> explain insert into dx2 values(1,1,'2011-11-11');
> -- no compute_stats calls
> insert into dx2 values(1,1,'2011-11-11');
> insert into dx2 values(1,1,'2001-11-11');
> explain analyze table dx2 compute statistics for columns;
> -- as expected; has compute_stats calls
> analyze table dx2 compute statistics for columns;
> -- runs ok
> desc formatted dx2 d;
> -- looks good
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20498) Support date type for column stats autogather
[ https://issues.apache.org/jira/browse/HIVE-20498?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16603093#comment-16603093 ] Zoltan Haindrich commented on HIVE-20498: - most probably this only needs to enable support for it in - because during analyze it already seems to work fine: ColumnStatsAutoGatherContext.canRunAutogatherStats > Support date type for column stats autogather > - > > Key: HIVE-20498 > URL: https://issues.apache.org/jira/browse/HIVE-20498 > Project: Hive > Issue Type: Sub-task > Components: Statistics >Reporter: Zoltan Haindrich >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20491) Fix mapjoin size estimations for Fast implementation
[ https://issues.apache.org/jira/browse/HIVE-20491?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Haindrich updated HIVE-20491: Attachment: HIVE-20491.02.patch > Fix mapjoin size estimations for Fast implementation > > > Key: HIVE-20491 > URL: https://issues.apache.org/jira/browse/HIVE-20491 > Project: Hive > Issue Type: Improvement > Components: Statistics >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Major > Attachments: HIVE-20491.01.patch, HIVE-20491.01wip02.patch, > HIVE-20491.02.patch > > > HIVE-19824 have fixed the estimations; but it calculated for the "optimized" > impl; the "fast" one has a little bit bigger footprint. > It also seems like fast is a bit overestimated at runtime...that should be > also taken care of. > | numkeys | implementation | compiler estimation | runtime estimation | > runtime measurement | ce / rm | re / rm | > | 25M | FAST | 1168435456 | 2189433712 | 1513584984 | .77 | 1.44 | > | 25M | OPTIMIZED | 1168435456 | 1191203764 | 1168439664 | 100% | 1.01 | -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-17084) Turn on hive.stats.fetch.column.stats configuration flag
[ https://issues.apache.org/jira/browse/HIVE-17084?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Haindrich updated HIVE-17084: Attachment: HIVE-17084.10.patch > Turn on hive.stats.fetch.column.stats configuration flag > > > Key: HIVE-17084 > URL: https://issues.apache.org/jira/browse/HIVE-17084 > Project: Hive > Issue Type: Task > Components: Statistics >Reporter: Vineet Garg >Assignee: Zoltan Haindrich >Priority: Major > Fix For: 3.2.0 > > Attachments: HIVE-17084.08.patch, HIVE-17084.09.patch, > HIVE-17084.1.patch, HIVE-17084.10.patch, HIVE-170884.4.patch, > HIVE-170884.5.patch, HIVE-170884.7.patch > > > This flag is off by default and could result in bad plans due to missing > column statistics. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-20497) ParseException, failed to recognize quoted identifier when re-parsing the re-written query
[ https://issues.apache.org/jira/browse/HIVE-20497?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhuwei reassigned HIVE-20497: - > ParseException, failed to recognize quoted identifier when re-parsing the > re-written query > -- > > Key: HIVE-20497 > URL: https://issues.apache.org/jira/browse/HIVE-20497 > Project: Hive > Issue Type: Bug > Components: Parser > Environment: hive 2.1.1 >Reporter: zhuwei >Assignee: zhuwei >Priority: Major > > select `user` from team; > If we have a table `team`, and one of its column has been masked out with > `` with column level authorization. The above query will fail with error > "SemanticException org.apache.hadoop.hive.ql.parse.ParseException: line 1:9 > Failed to recognize predicate 'user'. Failed rule: 'identifier' in expression > specification" > The root cause is that after re-written the ast, the back quote has been lost. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-17921) Aggregation with struct in LLAP produces wrong result
[
https://issues.apache.org/jira/browse/HIVE-17921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16602901#comment-16602901
]
Hive QA commented on HIVE-17921:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12938220/HIVE-17921.2.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 14920 tests
executed
*Failed tests:*
{noformat}
org.apache.hive.jdbc.TestJdbcWithMiniLlapArrow.testComplexQuery (batchId=251)
org.apache.hive.jdbc.miniHS2.TestHs2ConnectionMetricsBinary.testOpenConnectionMetrics
(batchId=255)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/13581/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/13581/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-13581/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12938220 - PreCommit-HIVE-Build
> Aggregation with struct in LLAP produces wrong result
> -
>
> Key: HIVE-17921
> URL: https://issues.apache.org/jira/browse/HIVE-17921
> Project: Hive
> Issue Type: Sub-task
> Components: llap, Transactions
>Affects Versions: 3.0.0
>Reporter: Eugene Koifman
>Assignee: Saurabh Seth
>Priority: Blocker
> Attachments: HIVE-17921.2.patch, HIVE-17921.patch
>
>
> Consider
> {noformat}
> select ROW__ID, count(*) from over10k_orc_bucketed group by ROW__ID having
> count(*) > 1;
> {noformat}
> in acid_vectorization_original.q (available since HIVE-17458)
> when run using TestMiniLlapCliDriver produces "NULL, N" where N varies from
> run to run.
> The right answer is empty results set as can be seen by running
> {noformat}
> select ROW__ID, * from over10k_orc_bucketed where ROW__ID is null
> {noformat}
> in the same test.
> This is with
> {noformat}
> set hive.vectorized.execution.enabled=true;
> set hive.vectorized.row.identifier.enabled=true;
> {noformat}
> It fails with TestMiniLlapCliDriver but not TestMiniTezCliDriver. See
> acid_vectorization_original_tez.q which has identical query.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20293) Support Replication of ACID table truncate operation
[ https://issues.apache.org/jira/browse/HIVE-20293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16602874#comment-16602874 ] Sankar Hariappan commented on HIVE-20293: - 02.patch committed to master! Thanks [~maheshk114] for the contribution! > Support Replication of ACID table truncate operation > > > Key: HIVE-20293 > URL: https://issues.apache.org/jira/browse/HIVE-20293 > Project: Hive > Issue Type: Task > Components: repl, Transactions >Affects Versions: 3.1.0, 4.0.0 >Reporter: mahesh kumar behera >Assignee: mahesh kumar behera >Priority: Major > Labels: DR, pull-request-available, replication > Fix For: 4.0.0 > > Attachments: HIVE-20293.01.patch, HIVE-20293.02.patch > > > Support truncate acid table replication. > 1. Write id allocation needs to be removed -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20293) Support Replication of ACID table truncate operation
[ https://issues.apache.org/jira/browse/HIVE-20293?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sankar Hariappan updated HIVE-20293: Fix Version/s: (was: 3.2.0) > Support Replication of ACID table truncate operation > > > Key: HIVE-20293 > URL: https://issues.apache.org/jira/browse/HIVE-20293 > Project: Hive > Issue Type: Task > Components: repl, Transactions >Affects Versions: 3.1.0, 4.0.0 >Reporter: mahesh kumar behera >Assignee: mahesh kumar behera >Priority: Major > Labels: DR, pull-request-available, replication > Fix For: 4.0.0 > > Attachments: HIVE-20293.01.patch, HIVE-20293.02.patch > > > Support truncate acid table replication. > 1. Write id allocation needs to be removed -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20476) CopyUtils used by REPL LOAD and EXPORT/IMPORT operations ignore distcp error.
[ https://issues.apache.org/jira/browse/HIVE-20476?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sankar Hariappan updated HIVE-20476: Resolution: Fixed Status: Resolved (was: Patch Available) > CopyUtils used by REPL LOAD and EXPORT/IMPORT operations ignore distcp error. > - > > Key: HIVE-20476 > URL: https://issues.apache.org/jira/browse/HIVE-20476 > Project: Hive > Issue Type: Bug > Components: HiveServer2, repl >Affects Versions: 3.1.0 >Reporter: Sankar Hariappan >Assignee: Sankar Hariappan >Priority: Major > Labels: DR, Export, Import, Replication, pull-request-available > Fix For: 4.0.0, 3.2.0 > > Attachments: HIVE-20476.01.patch > > > CopyUtils uses FileUtils.distCp to copy files but doesn't check the return > value. It returns false if the copy fails. > Now, REPL LOAD and EXPORT/IMPORT commands internally uses CopyUtils to copy > data files across clusters and here it may return success even if file copy > fails and may cause data loss. > Need to throw error and retry. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-17921) Aggregation with struct in LLAP produces wrong result
[
https://issues.apache.org/jira/browse/HIVE-17921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16602871#comment-16602871
]
Hive QA commented on HIVE-17921:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
38s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
8s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
38s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
10s{color} | {color:blue} ql in master has 2310 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
59s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 24m 51s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-13581/dev-support/hive-personality.sh
|
| git revision | master / a4dd84b |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-13581/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Aggregation with struct in LLAP produces wrong result
> -
>
> Key: HIVE-17921
> URL: https://issues.apache.org/jira/browse/HIVE-17921
> Project: Hive
> Issue Type: Sub-task
> Components: llap, Transactions
>Affects Versions: 3.0.0
>Reporter: Eugene Koifman
>Assignee: Saurabh Seth
>Priority: Blocker
> Attachments: HIVE-17921.2.patch, HIVE-17921.patch
>
>
> Consider
> {noformat}
> select ROW__ID, count(*) from over10k_orc_bucketed group by ROW__ID having
> count(*) > 1;
> {noformat}
> in acid_vectorization_original.q (available since HIVE-17458)
> when run using TestMiniLlapCliDriver produces "NULL, N" where N varies from
> run to run.
> The right answer is empty results set as can be seen by running
> {noformat}
> select ROW__ID, * from over10k_orc_bucketed where ROW__ID is null
> {noformat}
> in the same test.
> This is with
> {noformat}
> set hive.vectorized.execution.enabled=true;
> set hive.vectorized.row.identifier.enabled=true;
> {noformat}
> It fails with TestMiniLlapCliDriver but not TestMiniTezCliDriver. See
> acid_vectorization_original_tez.q which has identical query.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20476) CopyUtils used by REPL LOAD and EXPORT/IMPORT operations ignore distcp error.
[ https://issues.apache.org/jira/browse/HIVE-20476?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sankar Hariappan updated HIVE-20476: Fix Version/s: 3.2.0 > CopyUtils used by REPL LOAD and EXPORT/IMPORT operations ignore distcp error. > - > > Key: HIVE-20476 > URL: https://issues.apache.org/jira/browse/HIVE-20476 > Project: Hive > Issue Type: Bug > Components: HiveServer2, repl >Affects Versions: 3.1.0 >Reporter: Sankar Hariappan >Assignee: Sankar Hariappan >Priority: Major > Labels: DR, Export, Import, Replication, pull-request-available > Fix For: 4.0.0, 3.2.0 > > Attachments: HIVE-20476.01.patch > > > CopyUtils uses FileUtils.distCp to copy files but doesn't check the return > value. It returns false if the copy fails. > Now, REPL LOAD and EXPORT/IMPORT commands internally uses CopyUtils to copy > data files across clusters and here it may return success even if file copy > fails and may cause data loss. > Need to throw error and retry. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20476) CopyUtils used by REPL LOAD and EXPORT/IMPORT operations ignore distcp error.
[ https://issues.apache.org/jira/browse/HIVE-20476?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sankar Hariappan updated HIVE-20476: Fix Version/s: 4.0.0 > CopyUtils used by REPL LOAD and EXPORT/IMPORT operations ignore distcp error. > - > > Key: HIVE-20476 > URL: https://issues.apache.org/jira/browse/HIVE-20476 > Project: Hive > Issue Type: Bug > Components: HiveServer2, repl >Affects Versions: 3.1.0 >Reporter: Sankar Hariappan >Assignee: Sankar Hariappan >Priority: Major > Labels: DR, Export, Import, Replication, pull-request-available > Fix For: 4.0.0 > > Attachments: HIVE-20476.01.patch > > > CopyUtils uses FileUtils.distCp to copy files but doesn't check the return > value. It returns false if the copy fails. > Now, REPL LOAD and EXPORT/IMPORT commands internally uses CopyUtils to copy > data files across clusters and here it may return success even if file copy > fails and may cause data loss. > Need to throw error and retry. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20293) Support Replication of ACID table truncate operation
[ https://issues.apache.org/jira/browse/HIVE-20293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16602861#comment-16602861 ] Sankar Hariappan commented on HIVE-20293: - +1 > Support Replication of ACID table truncate operation > > > Key: HIVE-20293 > URL: https://issues.apache.org/jira/browse/HIVE-20293 > Project: Hive > Issue Type: Task > Components: repl, Transactions >Affects Versions: 3.1.0, 4.0.0 >Reporter: mahesh kumar behera >Assignee: mahesh kumar behera >Priority: Major > Labels: DR, pull-request-available, replication > Fix For: 4.0.0, 3.2.0 > > Attachments: HIVE-20293.01.patch, HIVE-20293.02.patch > > > Support truncate acid table replication. > 1. Write id allocation needs to be removed -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20293) Support Replication of ACID table truncate operation
[
https://issues.apache.org/jira/browse/HIVE-20293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16602836#comment-16602836
]
Hive QA commented on HIVE-20293:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12938215/HIVE-20293.02.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 14920 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/13580/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/13580/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-13580/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12938215 - PreCommit-HIVE-Build
> Support Replication of ACID table truncate operation
>
>
> Key: HIVE-20293
> URL: https://issues.apache.org/jira/browse/HIVE-20293
> Project: Hive
> Issue Type: Task
> Components: repl, Transactions
>Affects Versions: 3.1.0, 4.0.0
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: DR, pull-request-available, replication
> Fix For: 4.0.0, 3.2.0
>
> Attachments: HIVE-20293.01.patch, HIVE-20293.02.patch
>
>
> Support truncate acid table replication.
> 1. Write id allocation needs to be removed
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20293) Support Replication of ACID table truncate operation
[
https://issues.apache.org/jira/browse/HIVE-20293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16602828#comment-16602828
]
Hive QA commented on HIVE-20293:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
36s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
49s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m
28s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
43s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 2m
23s{color} | {color:blue} standalone-metastore/metastore-common in master has 9
extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
28s{color} | {color:blue} hcatalog/server-extensions in master has 3 extant
Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
29s{color} | {color:blue} hcatalog/webhcat/java-client in master has 3 extant
Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
41s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
3s{color} | {color:blue} ql in master has 2310 extant Findbugs warnings.
{color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
14s{color} | {color:red} metastore-server in master failed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m
4s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
8s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
18s{color} | {color:red} server-extensions in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
20s{color} | {color:red} java-client in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
50s{color} | {color:red} ql in the patch failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m
21s{color} | {color:red} java-client in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 21s{color}
| {color:red} java-client in the patch failed. {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
7s{color} | {color:green} The patch metastore-common passed checkstyle {color}
|
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
12s{color} | {color:red} hcatalog/server-extensions: The patch generated 3 new
+ 76 unchanged - 4 fixed = 79 total (was 80) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
12s{color} | {color:green} hcatalog/webhcat/java-client: The patch generated 0
new + 56 unchanged - 2 fixed = 56 total (was 58) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
17s{color} | {color:red} itests/hive-unit: The patch generated 2 new + 81
unchanged - 0 fixed = 83 total (was 81) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
45s{color} | {color:red} ql: The patch generated 2 new + 626 unchanged - 0
fixed = 628 total (was 626) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
7s{color} | {color:green} The patch metastore-server passed checkstyle {color}
|
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 3 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
19s{color} | {color:red} java-client in the patch failed. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 4m
19s{color} | {color:red} ql generated 1 new + 2309 unchanged - 1 fixed = 2310
total (was 2310) {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
14s{color} | {color:red}
[jira] [Commented] (HIVE-20489) Explain plan of query hangs
[
https://issues.apache.org/jira/browse/HIVE-20489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16602769#comment-16602769
]
Hive QA commented on HIVE-20489:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12938211/HIVE-20489.3.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 14920 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/13579/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/13579/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-13579/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12938211 - PreCommit-HIVE-Build
> Explain plan of query hangs
> ---
>
> Key: HIVE-20489
> URL: https://issues.apache.org/jira/browse/HIVE-20489
> Project: Hive
> Issue Type: Bug
>Reporter: Janaki Lahorani
>Assignee: Janaki Lahorani
>Priority: Major
> Attachments: HIVE-20489.1.patch, HIVE-20489.2.patch,
> HIVE-20489.3.patch
>
>
> Explain on a query that joins 47 views, in effect around 94 joins after view
> expansion seems to take forever.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
