[jira] [Commented] (HIVE-18517) Vectorization: Fix VectorMapOperator to accept VRBs and check vectorized flag correctly to support LLAP Caching
[
https://issues.apache.org/jira/browse/HIVE-18517?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16337082#comment-16337082
]
Hive QA commented on HIVE-18517:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12907421/HIVE-18517.02.patch
{color:green}SUCCESS:{color} +1 due to 29 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 142 failed/errored test(s), 11681 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[llap_acid] (batchId=81)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[llap_acid_fast]
(batchId=40)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[mapjoin_hook]
(batchId=13)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[mergejoin] (batchId=60)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[parquet_complex_types_vectorization]
(batchId=72)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[parquet_map_type_vectorization]
(batchId=84)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[parquet_no_row_serde]
(batchId=70)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[parquet_vectorization_0]
(batchId=16)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[parquet_vectorization_10]
(batchId=23)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[parquet_vectorization_11]
(batchId=37)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[parquet_vectorization_12]
(batchId=23)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[parquet_vectorization_13]
(batchId=52)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[parquet_vectorization_14]
(batchId=39)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[parquet_vectorization_15]
(batchId=87)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[parquet_vectorization_16]
(batchId=82)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[parquet_vectorization_17]
(batchId=29)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[parquet_vectorization_1]
(batchId=11)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[parquet_vectorization_2]
(batchId=3)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[parquet_vectorization_3]
(batchId=77)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[parquet_vectorization_4]
(batchId=44)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[parquet_vectorization_5]
(batchId=70)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[parquet_vectorization_6]
(batchId=41)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[parquet_vectorization_7]
(batchId=85)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[parquet_vectorization_8]
(batchId=14)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[parquet_vectorization_9]
(batchId=30)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[parquet_vectorization_decimal_date]
(batchId=30)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[parquet_vectorization_div0]
(batchId=77)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[parquet_vectorization_limit]
(batchId=24)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[parquet_vectorization_offset_limit]
(batchId=33)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[parquet_vectorization_part_project]
(batchId=35)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[parquet_vectorization_pushdown]
(batchId=34)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ppd_join5] (batchId=36)
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver[encryption_move_tbl]
(batchId=175)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[parquet_complex_types_vectorization]
(batchId=151)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[parquet_map_type_vectorization]
(batchId=152)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucket_map_join_tez1]
(batchId=172)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
(batchId=167)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid]
(batchId=171)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid_fast]
(batchId=161)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[mergejoin]
(batchId=166)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[schema_evol_text_vec_part_all_complex_llap_io]
(batchId=153)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[schema_evol_text_vec_part_all_primitive_llap_io]
(batchId=170)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[schema_evol_text_vec_part_llap_io]
(batchId=172)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[schema_evol_text_vec_table_llap_io]
(batchId=154)
[jira] [Commented] (HIVE-18517) Vectorization: Fix VectorMapOperator to accept VRBs and check vectorized flag correctly to support LLAP Caching
[
https://issues.apache.org/jira/browse/HIVE-18517?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16337028#comment-16337028
]
Hive QA commented on HIVE-18517:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
18s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
25s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
15s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
55s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
4s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
19s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
19s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
42s{color} | {color:red} ql: The patch generated 4 new + 777 unchanged - 1
fixed = 781 total (was 778) {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 24 line(s) that end in whitespace. Use
git apply --whitespace=fix <>. Refer
https://git-scm.com/docs/git-apply {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
4s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
11s{color} | {color:red} The patch generated 6 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 16m 32s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /data/hiveptest/working/yetus/dev-support/hive-personality.sh |
| git revision | master / 09ce6cd |
| Default Java | 1.8.0_111 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8805/yetus/diff-checkstyle-ql.txt
|
| whitespace |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8805/yetus/whitespace-eol.txt
|
| asflicense |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8805/yetus/patch-asflicense-problems.txt
|
| modules | C: ql llap-server itests U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8805/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Vectorization: Fix VectorMapOperator to accept VRBs and check vectorized flag
> correctly to support LLAP Caching
> ---
>
> Key: HIVE-18517
> URL: https://issues.apache.org/jira/browse/HIVE-18517
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 3.0.0
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
> Attachments: HIVE-18517.01.patch, HIVE-18517.02.patch
>
>
> LLAP is able to deserialize and cache data from an input format (e.g.
> TextInputFormat) and will deliver that cached data to VectorMapOperator as
> VRBs.
> Symptoms:
> BinaryComparable -> VectorizedRowBatch ClassCastException
> Text -> VectorizedRowBatch ClassCastException
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18509) Create tests for table manipulation related methods (create, alter, drop)
[
https://issues.apache.org/jira/browse/HIVE-18509?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16337004#comment-16337004
]
Hive QA commented on HIVE-18509:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12907310/HIVE-18509.4.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 19 failed/errored test(s), 11370 tests
executed
*Failed tests:*
{noformat}
TestNegativeCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=94)
[jira] [Comment Edited] (HIVE-18301) Investigate to enable MapInput cache in Hive on Spark
[
https://issues.apache.org/jira/browse/HIVE-18301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16336946#comment-16336946
]
liyunzhang edited comment on HIVE-18301 at 1/24/18 5:47 AM:
In HIVE-18301.patch, it provides one solution to transfer the
{{IOContext::inputPath}}
{code:java}
inputRDD1inputRDD2
|CopyFunction| CopyFunction
CopyRDD1CopyRDD2
| |
MT_11 MT_12
| |
RT_1 RT_2
\ /
Union
{code}
MT_11 will call following stack to initialize IOContext::inputPath
{code:java}
CombineHiveRecordReader#init
->HiveContextAwareRecordReader.initIOContext
->IOContext.setInputPath
{code}
inputRDD1 and inputRDD2 are same table's rdd, so CopyRDD1 and CopyRDD2 are same
rdd if rdd cache is enabled. When MT_12 will not call
CombineHiveRecordReader#init to initialize {{IOContext::inputPath}} but
{{MapOperator#process(Writable value)}} still need this value. IOContext is
bound to single thread, so the value is different in different thread.
{{inputRDD1-CopyRDD1-MT_11-RT_1}} and {{inputRDD2-CopyRDD2-MT_12-RT_2}} is
called in different thread. So IOContext can not be shared between these two
threads.
For this issue, I gave following solution:
We save the inputPath in CopyRDD1 when {{inputRDD1-CopyRDD1-MT_11-RT_1}} is
executed. CopyRDD2 get the cached value and inputPath from CopyRDD1 which is
stored in spark cache manager. We reinitialized the {{IOContext::inputPath}} in
{{MapOperator#process(Writable value)}} in MT_12.
*where to setInputPath?*
MapInput#CopyFunction#call, save inputPath in the first element of returned
tuple
{code:java}
public MapInput(SparkPlan sparkPlan, JavaPairRDD
hadoopRDD) {
this(sparkPlan, hadoopRDD, false);
@@ -79,10 +83,19 @@ public void setToCache(boolean toCache) {
call(Tuple2 tuple) throws Exception {
if (conf == null) {
conf = new Configuration();
+conf.set("hive.execution.engine","spark");
}
-
- return new Tuple2(tuple._1(),
- WritableUtils.clone(tuple._2(), conf));
+ //CopyFunction MapFunction
+ // HADOOPRDD-> RDD1-> RDD2.
+ // these transformation are in one stage and will be executed by 1 spark
task(thread),
+ // IOContext.get(conf).getInputPath will not be null.
+ String inputPath = IOContextMap.get(conf).getInputPath().toString();
+ Text inputPathText = new Text(inputPath);
+ // save inputPath in the first element of returned tuple
+ // before we need not use tuple._1() in SparkMapRecordHandler#processRow
+ // so replace inputPathText with tuple._1().
+ return new Tuple2(inputPathText,
+WritableUtils.clone(tuple._2(), conf));
}
}
{code}
*where to getInputPath?*
{code:java}
SparkMapRecordHandler#getInputPath
public void processRow(Object key, Object value) throws IOException {
+if (HiveConf.getBoolVar(jc,
HiveConf.ConfVars.HIVE_SPARK_SHARED_WORK_OPTIMIZATION)) {
+ Path inputPath = IOContextMap.get(jc).getInputPath();
+ // when inputPath is null, it means the record is cached
+ if (inputPath == null) {
+Text pathText = (Text) key;
+IOContextMap.get(jc).setInputPath(new Path(pathText.toString()));
+ }
+}
{code}
[~lirui], [~xuefuz], [~stakiar],[~csun], please give me your suggesions, thanks!
was (Author: kellyzly):
In HIVE-18301.patch, it provides one solution to transfer the
{{IOContext::inputPath}}
{code}
inputRDD1inputRDD2
|CopyFunction| CopyFunction
CopyRDD1CopyRDD2
| |
MT_11 MT_12
| |
RT_1 RT_2
\ /
Union
{code}
MT_11 will call following stack to initialize IOContext::inputPath
{code}
CombineHiveRecordReader#init
->HiveContextAwareRecordReader.initIOContext
->IOContext.setInputPath
{code}
inputRDD1 and inputRDD2 are same table's rdd, so CopyRDD1 and CopyRDD2 are same
rdd if rdd cache is enabled. When MT_12 will not call
CombineHiveRecordReader#init to initialize {{IOContext::inputPath}} but
{{MapOperator#process(Writable value)}} still need this value. IOContext is
bound to single thread, so the value is different in different thread.
{{inputRDD1-CopyRDD1-MT_11-RT_1}} and {{inputRDD2-CopyRDD2-MT_12-RT_2}} is
called in different thread. So IOContext can not be shared between these two
threads.
For this issue, I gave following solution:
We save the inputPath in CopyRDD1
[jira] [Comment Edited] (HIVE-18301) Investigate to enable MapInput cache in Hive on Spark
[
https://issues.apache.org/jira/browse/HIVE-18301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16336946#comment-16336946
]
liyunzhang edited comment on HIVE-18301 at 1/24/18 5:39 AM:
In HIVE-18301.patch, it provides one solution to transfer the
{{IOContext::inputPath}}
{code}
inputRDD1inputRDD2
|CopyFunction| CopyFunction
CopyRDD1CopyRDD2
| |
MT_11 MT_12
| |
RT_1 RT_2
\ /
Union
{code}
MT_11 will call following stack to initialize IOContext::inputPath
{code}
CombineHiveRecordReader#init
->HiveContextAwareRecordReader.initIOContext
->IOContext.setInputPath
{code}
inputRDD1 and inputRDD2 are same table's rdd, so CopyRDD1 and CopyRDD2 are same
rdd if rdd cache is enabled. When MT_12 will not call
CombineHiveRecordReader#init to initialize {{IOContext::inputPath}} but
{{MapOperator#process(Writable value)}} still need this value. IOContext is
bound to single thread, so the value is different in different thread.
{{inputRDD1-CopyRDD1-MT_11-RT_1}} and {{inputRDD2-CopyRDD2-MT_12-RT_2}} is
called in different thread. So IOContext can not be shared between these two
threads.
For this issue, I gave following solution:
We save the inputPath in CopyRDD1 when {{inputRDD1-CopyRDD1-MT_11-RT_1}} is
executed. CopyRDD2 get the cached value and inputPath from CopyRDD1 which is
stored in spark cache manager. We reinitialized the {{IOContext::inputPath}}
in {{MapOperator#process(Writable value)}} in MT_12.
*where to setInputPath?*
MapInput#CopyFunction#call, save inputPath in the first element of returned
tuple
{code}
public MapInput(SparkPlan sparkPlan, JavaPairRDD
hadoopRDD) {
this(sparkPlan, hadoopRDD, false);
@@ -79,10 +83,19 @@ public void setToCache(boolean toCache) {
call(Tuple2 tuple) throws Exception {
if (conf == null) {
conf = new Configuration();
+conf.set("hive.execution.engine","spark");
}
-
- return new Tuple2(tuple._1(),
- WritableUtils.clone(tuple._2(), conf));
+ //CopyFunction MapFunction
+ // HADOOPRDD-> RDD1-> RDD2.
+ // these transformation are in one stage and will be executed by 1 spark
task(thread),
+ // IOContext.get(conf).getInputPath will not be null.
+ String inputPath = IOContextMap.get(conf).getInputPath().toString();
+ Text inputPathText = new Text(inputPath);
+ // save inputPath in the first element of returned tuple
+ return new Tuple2(inputPathText,
+WritableUtils.clone(tuple._2(), conf));
}
}
{code}
*where to getInputPath?*
{code}
SparkMapRecordHandler#getInputPath
public void processRow(Object key, Object value) throws IOException {
+if (HiveConf.getBoolVar(jc,
HiveConf.ConfVars.HIVE_SPARK_SHARED_WORK_OPTIMIZATION)) {
+ Path inputPath = IOContextMap.get(jc).getInputPath();
+ // when inputPath is null, it means the record is cached
+ if (inputPath == null) {
+Text pathText = (Text) key;
+IOContextMap.get(jc).setInputPath(new Path(pathText.toString()));
+ }
+}
{code}
[~lirui], [~xuefuz], [~stakiar],[~csun], please give me your suggesions, thanks!
was (Author: kellyzly):
In HIVE-18301.patch, it provides one solution to transfer the
{{IOContext::inputPath}}
{code}
inputRDD1inputRDD2
|CopyFunction| CopyFunction
CopyRDD1CopyRDD2
||
MT_11 MT_12
| |
RT_1 RT_2
\ /
Union
{code}
MT_11 will call following stack to initialize IOContext::inputPath
{code}
CombineHiveRecordReader#init
->HiveContextAwareRecordReader.initIOContext
->IOContext.setInputPath
{code}
inputRDD1 and inputRDD2 are same table's rdd, so CopyRDD1 and CopyRDD2 are same
rdd if rdd cache is enabled. When MT_12 will not call
CombineHiveRecordReader#init to initialize {{IOContext::inputPath}} but
{{MapOperator#process(Writable value)}} still need this value. IOContext is
bound to single thread, so the value is different in different thread.
{{inputRDD1-CopyRDD1-MT_11-RT_1}} and {{inputRDD2-CopyRDD2-MT_12-RT_2}} is
called in different thread. So IOContext can not be shared between these two
threads.
For this issue, I gave following solution:
We save the inputPath in CopyRDD1 when {{inputRDD1-CopyRDD1-MT_11-RT_1}} is
executed. CopyRDD2 get the cached value and inputPath from CopyRDD1 which is
[jira] [Commented] (HIVE-18301) Investigate to enable MapInput cache in Hive on Spark
[
https://issues.apache.org/jira/browse/HIVE-18301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16336946#comment-16336946
]
liyunzhang commented on HIVE-18301:
---
In HIVE-18301.patch, it provides one solution to transfer the
{{IOContext::inputPath}}
{code}
inputRDD1inputRDD2
|CopyFunction| CopyFunction
CopyRDD1CopyRDD2
||
MT_11 MT_12
| |
RT_1 RT_2
\ /
Union
{code}
MT_11 will call following stack to initialize IOContext::inputPath
{code}
CombineHiveRecordReader#init
->HiveContextAwareRecordReader.initIOContext
->IOContext.setInputPath
{code}
inputRDD1 and inputRDD2 are same table's rdd, so CopyRDD1 and CopyRDD2 are same
rdd if rdd cache is enabled. When MT_12 will not call
CombineHiveRecordReader#init to initialize {{IOContext::inputPath}} but
{{MapOperator#process(Writable value)}} still need this value. IOContext is
bound to single thread, so the value is different in different thread.
{{inputRDD1-CopyRDD1-MT_11-RT_1}} and {{inputRDD2-CopyRDD2-MT_12-RT_2}} is
called in different thread. So IOContext can not be shared between these two
threads.
For this issue, I gave following solution:
We save the inputPath in CopyRDD1 when {{inputRDD1-CopyRDD1-MT_11-RT_1}} is
executed. CopyRDD2 get the cached value and inputPath from CopyRDD1 which is
stored in spark cache manager. We reinitialized the {{IOContext::inputPath}}
in {{MapOperator#process(Writable value)}} in MT_12.
*where to setInputPath?*
MapInput#CopyFunction#call, save inputPath in the first element of returned
tuple
{code}
public MapInput(SparkPlan sparkPlan, JavaPairRDD
hadoopRDD) {
this(sparkPlan, hadoopRDD, false);
@@ -79,10 +83,19 @@ public void setToCache(boolean toCache) {
call(Tuple2 tuple) throws Exception {
if (conf == null) {
conf = new Configuration();
+conf.set("hive.execution.engine","spark");
}
-
- return new Tuple2(tuple._1(),
- WritableUtils.clone(tuple._2(), conf));
+ //CopyFunction MapFunction
+ // HADOOPRDD-> RDD1-> RDD2.
+ // these transformation are in one stage and will be executed by 1 spark
task(thread),
+ // IOContext.get(conf).getInputPath will not be null.
+ String inputPath = IOContextMap.get(conf).getInputPath().toString();
+ Text inputPathText = new Text(inputPath);
+ // save inputPath in the first element of returned tuple
+ return new Tuple2(inputPathText,
+WritableUtils.clone(tuple._2(), conf));
}
}
{code}
*where to getInputPath?*
{code}
SparkMapRecordHandler#getInputPath
public void processRow(Object key, Object value) throws IOException {
+if (HiveConf.getBoolVar(jc,
HiveConf.ConfVars.HIVE_SPARK_SHARED_WORK_OPTIMIZATION)) {
+ Path inputPath = IOContextMap.get(jc).getInputPath();
+ // when inputPath is null, it means the record is cached
+ if (inputPath == null) {
+Text pathText = (Text) key;
+IOContextMap.get(jc).setInputPath(new Path(pathText.toString()));
+ }
+}
{code}
[~lirui], [~xuefuz], [~stakiar],[~csun], please give me your suggesions, thanks!
> Investigate to enable MapInput cache in Hive on Spark
> -
>
> Key: HIVE-18301
> URL: https://issues.apache.org/jira/browse/HIVE-18301
> Project: Hive
> Issue Type: Bug
>Reporter: liyunzhang
>Assignee: liyunzhang
>Priority: Major
> Attachments: HIVE-18301.patch
>
>
> Before IOContext problem is found in MapTran when spark rdd cache is enabled
> in HIVE-8920.
> so we disabled rdd cache in MapTran at
> [SparkPlanGenerator|https://github.com/kellyzly/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkPlanGenerator.java#L202].
> The problem is IOContext seems not initialized correctly in the spark yarn
> client/cluster mode and caused the exception like
> {code}
> Job aborted due to stage failure: Task 93 in stage 0.0 failed 4 times, most
> recent failure: Lost task 93.3 in stage 0.0 (TID 616, bdpe48):
> java.lang.RuntimeException: Error processing row:
> java.lang.NullPointerException
> at
> org.apache.hadoop.hive.ql.exec.spark.SparkMapRecordHandler.processRow(SparkMapRecordHandler.java:165)
> at
> org.apache.hadoop.hive.ql.exec.spark.HiveMapFunctionResultList.processNextRecord(HiveMapFunctionResultList.java:48)
> at
>
[jira] [Updated] (HIVE-18301) Investigate to enable MapInput cache in Hive on Spark
[
https://issues.apache.org/jira/browse/HIVE-18301?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
liyunzhang updated HIVE-18301:
--
Attachment: HIVE-18301.patch
> Investigate to enable MapInput cache in Hive on Spark
> -
>
> Key: HIVE-18301
> URL: https://issues.apache.org/jira/browse/HIVE-18301
> Project: Hive
> Issue Type: Bug
>Reporter: liyunzhang
>Assignee: liyunzhang
>Priority: Major
> Attachments: HIVE-18301.patch
>
>
> Before IOContext problem is found in MapTran when spark rdd cache is enabled
> in HIVE-8920.
> so we disabled rdd cache in MapTran at
> [SparkPlanGenerator|https://github.com/kellyzly/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkPlanGenerator.java#L202].
> The problem is IOContext seems not initialized correctly in the spark yarn
> client/cluster mode and caused the exception like
> {code}
> Job aborted due to stage failure: Task 93 in stage 0.0 failed 4 times, most
> recent failure: Lost task 93.3 in stage 0.0 (TID 616, bdpe48):
> java.lang.RuntimeException: Error processing row:
> java.lang.NullPointerException
> at
> org.apache.hadoop.hive.ql.exec.spark.SparkMapRecordHandler.processRow(SparkMapRecordHandler.java:165)
> at
> org.apache.hadoop.hive.ql.exec.spark.HiveMapFunctionResultList.processNextRecord(HiveMapFunctionResultList.java:48)
> at
> org.apache.hadoop.hive.ql.exec.spark.HiveMapFunctionResultList.processNextRecord(HiveMapFunctionResultList.java:27)
> at
> org.apache.hadoop.hive.ql.exec.spark.HiveBaseFunctionResultList.hasNext(HiveBaseFunctionResultList.java:85)
> at
> scala.collection.convert.Wrappers$JIteratorWrapper.hasNext(Wrappers.scala:42)
> at
> org.apache.spark.shuffle.sort.BypassMergeSortShuffleWriter.write(BypassMergeSortShuffleWriter.java:125)
> at
> org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:79)
> at
> org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:47)
> at org.apache.spark.scheduler.Task.run(Task.scala:85)
> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:274)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.NullPointerException
> at
> org.apache.hadoop.hive.ql.exec.AbstractMapOperator.getNominalPath(AbstractMapOperator.java:101)
> at
> org.apache.hadoop.hive.ql.exec.MapOperator.cleanUpInputFileChangedOp(MapOperator.java:516)
> at
> org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1187)
> at
> org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:546)
> at
> org.apache.hadoop.hive.ql.exec.spark.SparkMapRecordHandler.processRow(SparkMapRecordHandler.java:152)
> ... 12 more
> Driver stacktrace:
> {code}
> in yarn client/cluster mode, sometimes
> [ExecMapperContext#currentInputPath|https://github.com/kellyzly/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/exec/mr/ExecMapperContext.java#L109]
> is null when rdd cach is enabled.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18301) Investigate to enable MapInput cache in Hive on Spark
[
https://issues.apache.org/jira/browse/HIVE-18301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16336941#comment-16336941
]
liyunzhang commented on HIVE-18301:
---
[~lirui]: {quote}
And we won't be able to tell the file boundaries because they're cached.
{quote}
yes. when IOContext.getInputPath() = null only means the record is cached.
Can not know file boundaries.
So IOContext.getInputPath is necessary.
> Investigate to enable MapInput cache in Hive on Spark
> -
>
> Key: HIVE-18301
> URL: https://issues.apache.org/jira/browse/HIVE-18301
> Project: Hive
> Issue Type: Bug
>Reporter: liyunzhang
>Assignee: liyunzhang
>Priority: Major
>
> Before IOContext problem is found in MapTran when spark rdd cache is enabled
> in HIVE-8920.
> so we disabled rdd cache in MapTran at
> [SparkPlanGenerator|https://github.com/kellyzly/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkPlanGenerator.java#L202].
> The problem is IOContext seems not initialized correctly in the spark yarn
> client/cluster mode and caused the exception like
> {code}
> Job aborted due to stage failure: Task 93 in stage 0.0 failed 4 times, most
> recent failure: Lost task 93.3 in stage 0.0 (TID 616, bdpe48):
> java.lang.RuntimeException: Error processing row:
> java.lang.NullPointerException
> at
> org.apache.hadoop.hive.ql.exec.spark.SparkMapRecordHandler.processRow(SparkMapRecordHandler.java:165)
> at
> org.apache.hadoop.hive.ql.exec.spark.HiveMapFunctionResultList.processNextRecord(HiveMapFunctionResultList.java:48)
> at
> org.apache.hadoop.hive.ql.exec.spark.HiveMapFunctionResultList.processNextRecord(HiveMapFunctionResultList.java:27)
> at
> org.apache.hadoop.hive.ql.exec.spark.HiveBaseFunctionResultList.hasNext(HiveBaseFunctionResultList.java:85)
> at
> scala.collection.convert.Wrappers$JIteratorWrapper.hasNext(Wrappers.scala:42)
> at
> org.apache.spark.shuffle.sort.BypassMergeSortShuffleWriter.write(BypassMergeSortShuffleWriter.java:125)
> at
> org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:79)
> at
> org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:47)
> at org.apache.spark.scheduler.Task.run(Task.scala:85)
> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:274)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.NullPointerException
> at
> org.apache.hadoop.hive.ql.exec.AbstractMapOperator.getNominalPath(AbstractMapOperator.java:101)
> at
> org.apache.hadoop.hive.ql.exec.MapOperator.cleanUpInputFileChangedOp(MapOperator.java:516)
> at
> org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1187)
> at
> org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:546)
> at
> org.apache.hadoop.hive.ql.exec.spark.SparkMapRecordHandler.processRow(SparkMapRecordHandler.java:152)
> ... 12 more
> Driver stacktrace:
> {code}
> in yarn client/cluster mode, sometimes
> [ExecMapperContext#currentInputPath|https://github.com/kellyzly/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/exec/mr/ExecMapperContext.java#L109]
> is null when rdd cach is enabled.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18509) Create tests for table manipulation related methods (create, alter, drop)
[
https://issues.apache.org/jira/browse/HIVE-18509?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16336940#comment-16336940
]
Hive QA commented on HIVE-18509:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
47s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
36s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
14s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
45s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
34s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
13s{color} | {color:red} standalone-metastore: The patch generated 8 new + 2
unchanged - 0 fixed = 10 total (was 2) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
48s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
12s{color} | {color:red} The patch generated 6 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 10m 59s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /data/hiveptest/working/yetus/dev-support/hive-personality.sh |
| git revision | master / 09ce6cd |
| Default Java | 1.8.0_111 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8804/yetus/diff-checkstyle-standalone-metastore.txt
|
| asflicense |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8804/yetus/patch-asflicense-problems.txt
|
| modules | C: standalone-metastore U: standalone-metastore |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8804/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Create tests for table manipulation related methods (create, alter, drop)
> -
>
> Key: HIVE-18509
> URL: https://issues.apache.org/jira/browse/HIVE-18509
> Project: Hive
> Issue Type: Sub-task
>Reporter: Peter Vary
>Assignee: Peter Vary
>Priority: Major
> Attachments: HIVE-18509.2.patch, HIVE-18509.3.patch,
> HIVE-18509.4.patch, HIVE-18509.patch
>
>
> Create API tests for table metadata manipulations
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18480) Create tests for function related methods
[
https://issues.apache.org/jira/browse/HIVE-18480?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16336917#comment-16336917
]
Hive QA commented on HIVE-18480:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12907308/HIVE-18480.3.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 29 failed/errored test(s), 11740 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_sortmerge_join_2]
(batchId=48)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[mapjoin_hook]
(batchId=12)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ppd_join5] (batchId=35)
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver[encryption_move_tbl]
(batchId=173)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[llap_smb]
(batchId=151)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucket_map_join_tez1]
(batchId=170)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
(batchId=165)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid]
(batchId=169)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid_fast]
(batchId=160)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb]
(batchId=160)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vectorization_input_format_excludes]
(batchId=162)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[bucketizedhiveinputformat]
(batchId=178)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[authorization_part]
(batchId=94)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[stats_aggregator_error_1]
(batchId=94)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[ppd_join5]
(batchId=121)
org.apache.hadoop.hive.ql.io.TestDruidRecordWriter.testWrite (batchId=254)
org.apache.hive.beeline.cli.TestHiveCli.testNoErrorDB (batchId=186)
org.apache.hive.hcatalog.streaming.mutate.TestMutations.testMulti (batchId=200)
org.apache.hive.hcatalog.streaming.mutate.TestMutations.testTransactionBatchAbort
(batchId=200)
org.apache.hive.hcatalog.streaming.mutate.TestMutations.testTransactionBatchCommitPartitioned
(batchId=200)
org.apache.hive.hcatalog.streaming.mutate.TestMutations.testTransactionBatchCommitUnpartitioned
(batchId=200)
org.apache.hive.hcatalog.streaming.mutate.TestMutations.testTransactionBatchEmptyAbortPartitioned
(batchId=200)
org.apache.hive.hcatalog.streaming.mutate.TestMutations.testTransactionBatchEmptyAbortUnartitioned
(batchId=200)
org.apache.hive.hcatalog.streaming.mutate.TestMutations.testTransactionBatchEmptyCommitPartitioned
(batchId=200)
org.apache.hive.hcatalog.streaming.mutate.TestMutations.testTransactionBatchEmptyCommitUnpartitioned
(batchId=200)
org.apache.hive.hcatalog.streaming.mutate.TestMutations.testUpdatesAndDeletes
(batchId=200)
org.apache.hive.jdbc.TestSSL.testConnectionMismatch (batchId=232)
org.apache.hive.jdbc.TestSSL.testConnectionWrongCertCN (batchId=232)
org.apache.hive.jdbc.TestSSL.testMetastoreConnectionWrongCertCN (batchId=232)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/8803/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/8803/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-8803/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 29 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12907308 - PreCommit-HIVE-Build
> Create tests for function related methods
> -
>
> Key: HIVE-18480
> URL: https://issues.apache.org/jira/browse/HIVE-18480
> Project: Hive
> Issue Type: Sub-task
>Reporter: Peter Vary
>Assignee: Peter Vary
>Priority: Major
> Attachments: HIVE-18480.2.patch, HIVE-18480.3.patch, HIVE-18480.patch
>
>
> Create IMetaStoreClient tests to cover the function related methods
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18516) load data should rename files consistent with insert statements for ACID Tables
[ https://issues.apache.org/jira/browse/HIVE-18516?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Deepak Jaiswal updated HIVE-18516: -- Attachment: HIVE-18516.2.patch > load data should rename files consistent with insert statements for ACID > Tables > --- > > Key: HIVE-18516 > URL: https://issues.apache.org/jira/browse/HIVE-18516 > Project: Hive > Issue Type: Bug >Reporter: Deepak Jaiswal >Assignee: Deepak Jaiswal >Priority: Major > Attachments: HIVE-18516.1.patch, HIVE-18516.2.patch > > > h1. load data should rename files consistent with insert statements for ACID > Tables. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18301) Investigate to enable MapInput cache in Hive on Spark
[
https://issues.apache.org/jira/browse/HIVE-18301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16336898#comment-16336898
]
Rui Li commented on HIVE-18301:
---
{quote}We need not to call MapOperator#cleanUpInputFileChanged because
MapOperator#cleanUpInputFileChanged is only designed for one Mapper scanning
multiple files
{quote}
When RDD is cached, mapper reads records from the cache. But I think those
records may come from multiple underlying files right? And we won't be able to
tell the file boundaries because they're cached.
> Investigate to enable MapInput cache in Hive on Spark
> -
>
> Key: HIVE-18301
> URL: https://issues.apache.org/jira/browse/HIVE-18301
> Project: Hive
> Issue Type: Bug
>Reporter: liyunzhang
>Assignee: liyunzhang
>Priority: Major
>
> Before IOContext problem is found in MapTran when spark rdd cache is enabled
> in HIVE-8920.
> so we disabled rdd cache in MapTran at
> [SparkPlanGenerator|https://github.com/kellyzly/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkPlanGenerator.java#L202].
> The problem is IOContext seems not initialized correctly in the spark yarn
> client/cluster mode and caused the exception like
> {code}
> Job aborted due to stage failure: Task 93 in stage 0.0 failed 4 times, most
> recent failure: Lost task 93.3 in stage 0.0 (TID 616, bdpe48):
> java.lang.RuntimeException: Error processing row:
> java.lang.NullPointerException
> at
> org.apache.hadoop.hive.ql.exec.spark.SparkMapRecordHandler.processRow(SparkMapRecordHandler.java:165)
> at
> org.apache.hadoop.hive.ql.exec.spark.HiveMapFunctionResultList.processNextRecord(HiveMapFunctionResultList.java:48)
> at
> org.apache.hadoop.hive.ql.exec.spark.HiveMapFunctionResultList.processNextRecord(HiveMapFunctionResultList.java:27)
> at
> org.apache.hadoop.hive.ql.exec.spark.HiveBaseFunctionResultList.hasNext(HiveBaseFunctionResultList.java:85)
> at
> scala.collection.convert.Wrappers$JIteratorWrapper.hasNext(Wrappers.scala:42)
> at
> org.apache.spark.shuffle.sort.BypassMergeSortShuffleWriter.write(BypassMergeSortShuffleWriter.java:125)
> at
> org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:79)
> at
> org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:47)
> at org.apache.spark.scheduler.Task.run(Task.scala:85)
> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:274)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.NullPointerException
> at
> org.apache.hadoop.hive.ql.exec.AbstractMapOperator.getNominalPath(AbstractMapOperator.java:101)
> at
> org.apache.hadoop.hive.ql.exec.MapOperator.cleanUpInputFileChangedOp(MapOperator.java:516)
> at
> org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1187)
> at
> org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:546)
> at
> org.apache.hadoop.hive.ql.exec.spark.SparkMapRecordHandler.processRow(SparkMapRecordHandler.java:152)
> ... 12 more
> Driver stacktrace:
> {code}
> in yarn client/cluster mode, sometimes
> [ExecMapperContext#currentInputPath|https://github.com/kellyzly/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/exec/mr/ExecMapperContext.java#L109]
> is null when rdd cach is enabled.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18516) load data should rename files consistent with insert statements for ACID Tables
[ https://issues.apache.org/jira/browse/HIVE-18516?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Deepak Jaiswal updated HIVE-18516: -- Status: Patch Available (was: Open) > load data should rename files consistent with insert statements for ACID > Tables > --- > > Key: HIVE-18516 > URL: https://issues.apache.org/jira/browse/HIVE-18516 > Project: Hive > Issue Type: Bug >Reporter: Deepak Jaiswal >Assignee: Deepak Jaiswal >Priority: Major > Attachments: HIVE-18516.1.patch > > > h1. load data should rename files consistent with insert statements for ACID > Tables. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18516) load data should rename files consistent with insert statements for ACID Tables
[ https://issues.apache.org/jira/browse/HIVE-18516?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Deepak Jaiswal updated HIVE-18516: -- Attachment: HIVE-18516.1.patch > load data should rename files consistent with insert statements for ACID > Tables > --- > > Key: HIVE-18516 > URL: https://issues.apache.org/jira/browse/HIVE-18516 > Project: Hive > Issue Type: Bug >Reporter: Deepak Jaiswal >Assignee: Deepak Jaiswal >Priority: Major > Attachments: HIVE-18516.1.patch > > > h1. load data should rename files consistent with insert statements for ACID > Tables. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18516) load data should rename files consistent with insert statements for ACID Tables
[ https://issues.apache.org/jira/browse/HIVE-18516?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Deepak Jaiswal updated HIVE-18516: -- Status: Open (was: Patch Available) > load data should rename files consistent with insert statements for ACID > Tables > --- > > Key: HIVE-18516 > URL: https://issues.apache.org/jira/browse/HIVE-18516 > Project: Hive > Issue Type: Bug >Reporter: Deepak Jaiswal >Assignee: Deepak Jaiswal >Priority: Major > > h1. load data should rename files consistent with insert statements for ACID > Tables. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18516) load data should rename files consistent with insert statements for ACID Tables
[ https://issues.apache.org/jira/browse/HIVE-18516?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Deepak Jaiswal updated HIVE-18516: -- Attachment: (was: HIVE-18516.1.patch) > load data should rename files consistent with insert statements for ACID > Tables > --- > > Key: HIVE-18516 > URL: https://issues.apache.org/jira/browse/HIVE-18516 > Project: Hive > Issue Type: Bug >Reporter: Deepak Jaiswal >Assignee: Deepak Jaiswal >Priority: Major > > h1. load data should rename files consistent with insert statements for ACID > Tables. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18521) Vectorization: query failing in reducer VectorUDAFAvgDecimalPartial2 java.lang.ClassCastException StructTypeInfo --> DecimalTypeInfo
[ https://issues.apache.org/jira/browse/HIVE-18521?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-18521: Status: Patch Available (was: Open) > Vectorization: query failing in reducer VectorUDAFAvgDecimalPartial2 > java.lang.ClassCastException StructTypeInfo --> DecimalTypeInfo > > > Key: HIVE-18521 > URL: https://issues.apache.org/jira/browse/HIVE-18521 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 3.0.0 >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-18521.01.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18521) Vectorization: query failing in reducer VectorUDAFAvgDecimalPartial2 java.lang.ClassCastException StructTypeInfo --> DecimalTypeInfo
[ https://issues.apache.org/jira/browse/HIVE-18521?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-18521: Attachment: HIVE-18521.01.patch > Vectorization: query failing in reducer VectorUDAFAvgDecimalPartial2 > java.lang.ClassCastException StructTypeInfo --> DecimalTypeInfo > > > Key: HIVE-18521 > URL: https://issues.apache.org/jira/browse/HIVE-18521 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 3.0.0 >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-18521.01.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-18521) Vectorization: query failing in reducer VectorUDAFAvgDecimalPartial2 java.lang.ClassCastException StructTypeInfo --> DecimalTypeInfo
[ https://issues.apache.org/jira/browse/HIVE-18521?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline reassigned HIVE-18521: --- > Vectorization: query failing in reducer VectorUDAFAvgDecimalPartial2 > java.lang.ClassCastException StructTypeInfo --> DecimalTypeInfo > > > Key: HIVE-18521 > URL: https://issues.apache.org/jira/browse/HIVE-18521 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 3.0.0 >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18480) Create tests for function related methods
[
https://issues.apache.org/jira/browse/HIVE-18480?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16336871#comment-16336871
]
Hive QA commented on HIVE-18480:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
1s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
48s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
35s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
13s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
48s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
31s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
15s{color} | {color:red} standalone-metastore: The patch generated 18 new + 0
unchanged - 0 fixed = 18 total (was 0) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
50s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
11s{color} | {color:red} The patch generated 6 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 11m 1s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /data/hiveptest/working/yetus/dev-support/hive-personality.sh |
| git revision | master / 09ce6cd |
| Default Java | 1.8.0_111 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8803/yetus/diff-checkstyle-standalone-metastore.txt
|
| asflicense |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8803/yetus/patch-asflicense-problems.txt
|
| modules | C: standalone-metastore U: standalone-metastore |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8803/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Create tests for function related methods
> -
>
> Key: HIVE-18480
> URL: https://issues.apache.org/jira/browse/HIVE-18480
> Project: Hive
> Issue Type: Sub-task
>Reporter: Peter Vary
>Assignee: Peter Vary
>Priority: Major
> Attachments: HIVE-18480.2.patch, HIVE-18480.3.patch, HIVE-18480.patch
>
>
> Create IMetaStoreClient tests to cover the function related methods
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18517) Vectorization: Fix VectorMapOperator to accept VRBs and check vectorized flag correctly to support LLAP Caching
[ https://issues.apache.org/jira/browse/HIVE-18517?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-18517: Attachment: HIVE-18517.02.patch > Vectorization: Fix VectorMapOperator to accept VRBs and check vectorized flag > correctly to support LLAP Caching > --- > > Key: HIVE-18517 > URL: https://issues.apache.org/jira/browse/HIVE-18517 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 3.0.0 >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-18517.01.patch, HIVE-18517.02.patch > > > LLAP is able to deserialize and cache data from an input format (e.g. > TextInputFormat) and will deliver that cached data to VectorMapOperator as > VRBs. > Symptoms: > BinaryComparable -> VectorizedRowBatch ClassCastException > Text -> VectorizedRowBatch ClassCastException -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18517) Vectorization: Fix VectorMapOperator to accept VRBs and check vectorized flag correctly to support LLAP Caching
[ https://issues.apache.org/jira/browse/HIVE-18517?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-18517: Attachment: (was: HIVE-18517.02.patch) > Vectorization: Fix VectorMapOperator to accept VRBs and check vectorized flag > correctly to support LLAP Caching > --- > > Key: HIVE-18517 > URL: https://issues.apache.org/jira/browse/HIVE-18517 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 3.0.0 >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-18517.01.patch > > > LLAP is able to deserialize and cache data from an input format (e.g. > TextInputFormat) and will deliver that cached data to VectorMapOperator as > VRBs. > Symptoms: > BinaryComparable -> VectorizedRowBatch ClassCastException > Text -> VectorizedRowBatch ClassCastException -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18518) Upgrade druid version to 0.11.0
[
https://issues.apache.org/jira/browse/HIVE-18518?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16336853#comment-16336853
]
Hive QA commented on HIVE-18518:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12907320/HIVE-18518.1.patch
{color:green}SUCCESS:{color} +1 due to 3 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 26 failed/errored test(s), 11647 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_sortmerge_join_2]
(batchId=48)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ppd_join5] (batchId=35)
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver[encryption_move_tbl]
(batchId=173)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[llap_smb]
(batchId=151)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucket_map_join_tez1]
(batchId=170)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
(batchId=165)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid]
(batchId=169)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid_fast]
(batchId=160)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb]
(batchId=160)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vectorization_input_format_excludes]
(batchId=162)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[archive_partspec2]
(batchId=94)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[authorization_alter_table_exchange_partition_fail]
(batchId=94)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[authorization_part]
(batchId=94)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[ctas_noemptyfolder]
(batchId=94)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[materialized_view_authorization_create_no_grant]
(batchId=94)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[materialized_view_authorization_drop_other]
(batchId=94)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[materialized_view_no_transactional_rewrite_2]
(batchId=94)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[stats_aggregator_error_1]
(batchId=94)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[stats_publisher_error_1]
(batchId=94)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[subquery_notin_implicit_gby]
(batchId=94)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[truncate_bucketed_column]
(batchId=94)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[ppd_join5]
(batchId=121)
org.apache.hadoop.hive.ql.io.TestDruidRecordWriter.testWrite (batchId=254)
org.apache.hive.jdbc.TestSSL.testConnectionMismatch (batchId=232)
org.apache.hive.jdbc.TestSSL.testConnectionWrongCertCN (batchId=232)
org.apache.hive.jdbc.TestSSL.testMetastoreConnectionWrongCertCN (batchId=232)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/8802/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/8802/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-8802/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 26 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12907320 - PreCommit-HIVE-Build
> Upgrade druid version to 0.11.0
> ---
>
> Key: HIVE-18518
> URL: https://issues.apache.org/jira/browse/HIVE-18518
> Project: Hive
> Issue Type: Bug
>Reporter: Nishant Bangarwa
>Assignee: Nishant Bangarwa
>Priority: Major
> Attachments: HIVE-18518.1.patch, HIVE-18518.patch
>
>
> this task is to upgrade to druid version 0.11.0
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18519) do not create materialized CTEs with ACID/MM
[ https://issues.apache.org/jira/browse/HIVE-18519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16336844#comment-16336844 ] Eugene Koifman commented on HIVE-18519: --- +1 pending tests > do not create materialized CTEs with ACID/MM > > > Key: HIVE-18519 > URL: https://issues.apache.org/jira/browse/HIVE-18519 > Project: Hive > Issue Type: Bug > Components: Transactions >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-18519.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18519) do not create materialized CTEs with ACID/MM
[ https://issues.apache.org/jira/browse/HIVE-18519?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman updated HIVE-18519: -- Component/s: Transactions > do not create materialized CTEs with ACID/MM > > > Key: HIVE-18519 > URL: https://issues.apache.org/jira/browse/HIVE-18519 > Project: Hive > Issue Type: Bug > Components: Transactions >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-18519.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18493) Add display escape for CR/LF to Hive CLI and Beeline
[ https://issues.apache.org/jira/browse/HIVE-18493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16336839#comment-16336839 ] Matt McCline commented on HIVE-18493: - [~djaiswal] Thank you very much for your review! > Add display escape for CR/LF to Hive CLI and Beeline > > > Key: HIVE-18493 > URL: https://issues.apache.org/jira/browse/HIVE-18493 > Project: Hive > Issue Type: Bug > Components: Beeline, Hive >Affects Versions: 3.0.0 >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-18493.01.patch, HIVE-18493.02.patch, > HIVE-18493.03.patch, HIVE-18493.04.patch, HIVE-18493.05.patch, > HIVE-18493.06.patch > > > Add optional display escaping of carriage return and line feed so row output > remains one line. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18518) Upgrade druid version to 0.11.0
[
https://issues.apache.org/jira/browse/HIVE-18518?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16336838#comment-16336838
]
Hive QA commented on HIVE-18518:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
36s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
54s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m
31s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
13s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 5m
58s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
39s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 5m
24s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
12s{color} | {color:red} druid-handler: The patch generated 6 new + 156
unchanged - 4 fixed = 162 total (was 160) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m
53s{color} | {color:red} root: The patch generated 6 new + 156 unchanged - 4
fixed = 162 total (was 160) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
3s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 6m
20s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
11s{color} | {color:red} The patch generated 6 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 43m 28s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc xml compile findbugs
checkstyle |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /data/hiveptest/working/yetus/dev-support/hive-personality.sh |
| git revision | master / c31c9bf |
| Default Java | 1.8.0_111 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8802/yetus/diff-checkstyle-druid-handler.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8802/yetus/diff-checkstyle-root.txt
|
| asflicense |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8802/yetus/patch-asflicense-problems.txt
|
| modules | C: druid-handler . itests/qtest-druid U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8802/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Upgrade druid version to 0.11.0
> ---
>
> Key: HIVE-18518
> URL: https://issues.apache.org/jira/browse/HIVE-18518
> Project: Hive
> Issue Type: Bug
>Reporter: Nishant Bangarwa
>Assignee: Nishant Bangarwa
>Priority: Major
> Attachments: HIVE-18518.1.patch, HIVE-18518.patch
>
>
> this task is to upgrade to druid version 0.11.0
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18517) Vectorization: Fix VectorMapOperator to accept VRBs and check vectorized flag correctly to support LLAP Caching
[ https://issues.apache.org/jira/browse/HIVE-18517?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-18517: Attachment: HIVE-18517.02.patch > Vectorization: Fix VectorMapOperator to accept VRBs and check vectorized flag > correctly to support LLAP Caching > --- > > Key: HIVE-18517 > URL: https://issues.apache.org/jira/browse/HIVE-18517 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 3.0.0 >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-18517.01.patch, HIVE-18517.02.patch > > > LLAP is able to deserialize and cache data from an input format (e.g. > TextInputFormat) and will deliver that cached data to VectorMapOperator as > VRBs. > Symptoms: > BinaryComparable -> VectorizedRowBatch ClassCastException > Text -> VectorizedRowBatch ClassCastException -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18517) Vectorization: Fix VectorMapOperator to accept VRBs and check vectorized flag correctly to support LLAP Caching
[ https://issues.apache.org/jira/browse/HIVE-18517?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-18517: Attachment: (was: HIVE-18517.02.patch) > Vectorization: Fix VectorMapOperator to accept VRBs and check vectorized flag > correctly to support LLAP Caching > --- > > Key: HIVE-18517 > URL: https://issues.apache.org/jira/browse/HIVE-18517 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 3.0.0 >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-18517.01.patch, HIVE-18517.02.patch > > > LLAP is able to deserialize and cache data from an input format (e.g. > TextInputFormat) and will deliver that cached data to VectorMapOperator as > VRBs. > Symptoms: > BinaryComparable -> VectorizedRowBatch ClassCastException > Text -> VectorizedRowBatch ClassCastException -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18493) Add display escape for CR/LF to Hive CLI and Beeline
[ https://issues.apache.org/jira/browse/HIVE-18493?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-18493: Resolution: Fixed Status: Resolved (was: Patch Available) > Add display escape for CR/LF to Hive CLI and Beeline > > > Key: HIVE-18493 > URL: https://issues.apache.org/jira/browse/HIVE-18493 > Project: Hive > Issue Type: Bug > Components: Beeline, Hive >Affects Versions: 3.0.0 >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-18493.01.patch, HIVE-18493.02.patch, > HIVE-18493.03.patch, HIVE-18493.04.patch, HIVE-18493.05.patch, > HIVE-18493.06.patch > > > Add optional display escaping of carriage return and line feed so row output > remains one line. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18493) Add display escape for CR/LF to Hive CLI and Beeline
[ https://issues.apache.org/jira/browse/HIVE-18493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16336827#comment-16336827 ] Matt McCline commented on HIVE-18493: - Committed to Apache master. > Add display escape for CR/LF to Hive CLI and Beeline > > > Key: HIVE-18493 > URL: https://issues.apache.org/jira/browse/HIVE-18493 > Project: Hive > Issue Type: Bug > Components: Beeline, Hive >Affects Versions: 3.0.0 >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-18493.01.patch, HIVE-18493.02.patch, > HIVE-18493.03.patch, HIVE-18493.04.patch, HIVE-18493.05.patch, > HIVE-18493.06.patch > > > Add optional display escaping of carriage return and line feed so row output > remains one line. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18493) Add display escape for CR/LF to Hive CLI and Beeline
[ https://issues.apache.org/jira/browse/HIVE-18493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16336823#comment-16336823 ] Matt McCline commented on HIVE-18493: - Test failures are unrelated. > Add display escape for CR/LF to Hive CLI and Beeline > > > Key: HIVE-18493 > URL: https://issues.apache.org/jira/browse/HIVE-18493 > Project: Hive > Issue Type: Bug > Components: Beeline, Hive >Affects Versions: 3.0.0 >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-18493.01.patch, HIVE-18493.02.patch, > HIVE-18493.03.patch, HIVE-18493.04.patch, HIVE-18493.05.patch, > HIVE-18493.06.patch > > > Add optional display escaping of carriage return and line feed so row output > remains one line. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18503) MM/ACID tables: make tests that will never be compatible with acid use non-txn tables explicitly
[ https://issues.apache.org/jira/browse/HIVE-18503?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-18503: Attachment: (was: HIVE-18503.WIP.patch) > MM/ACID tables: make tests that will never be compatible with acid use > non-txn tables explicitly > > > Key: HIVE-18503 > URL: https://issues.apache.org/jira/browse/HIVE-18503 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-18503.WIP.patch > > > Some tests do stuff that will simply never work with ACID tables, e.g. delete > table files. > They should be changed to use external table, or explicitly set > transactional=false > > List of things: > * Deleting table files via dfs commands. > * Computing query using stats - not supported for either ACID tables or > external tables, so with MM by default not supported in any scenario. > HIVE-18395 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18503) MM/ACID tables: make tests that will never be compatible with acid use non-txn tables explicitly
[ https://issues.apache.org/jira/browse/HIVE-18503?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-18503: Description: Some tests do stuff that will simply never work with ACID tables, e.g. delete table files. They should be changed to use external table, or explicitly set transactional=false List of things: * Deleting table files via dfs commands. * Computing query using stats - not supported for either ACID tables or external tables, so with MM by default not supported in any scenario. HIVE-18395 was: Some tests do stuff that will simply never work with ACID tables, e.g. delete table files. They should be changed to use external table, or explicitly set transactional=false > MM/ACID tables: make tests that will never be compatible with acid use > non-txn tables explicitly > > > Key: HIVE-18503 > URL: https://issues.apache.org/jira/browse/HIVE-18503 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-18503.WIP.patch > > > Some tests do stuff that will simply never work with ACID tables, e.g. delete > table files. > They should be changed to use external table, or explicitly set > transactional=false > > List of things: > * Deleting table files via dfs commands. > * Computing query using stats - not supported for either ACID tables or > external tables, so with MM by default not supported in any scenario. > HIVE-18395 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18503) MM/ACID tables: make tests that will never be compatible with acid use non-txn tables explicitly
[ https://issues.apache.org/jira/browse/HIVE-18503?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-18503: Attachment: HIVE-18503.WIP.patch > MM/ACID tables: make tests that will never be compatible with acid use > non-txn tables explicitly > > > Key: HIVE-18503 > URL: https://issues.apache.org/jira/browse/HIVE-18503 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-18503.WIP.patch, HIVE-18503.WIP.patch > > > Some tests do stuff that will simply never work with ACID tables, e.g. delete > table files. > They should be changed to use external table, or explicitly set > transactional=false -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18493) Add display escape for CR/LF to Hive CLI and Beeline
[
https://issues.apache.org/jira/browse/HIVE-18493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16336790#comment-16336790
]
Hive QA commented on HIVE-18493:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12907299/HIVE-18493.06.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 20 failed/errored test(s), 11642 tests
executed
*Failed tests:*
{noformat}
TestJdbcWithMiniKdc - did not produce a TEST-*.xml file (likely timed out)
(batchId=247)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_sortmerge_join_2]
(batchId=48)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ppd_join5] (batchId=35)
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver[encryption_move_tbl]
(batchId=173)
org.apache.hadoop.hive.cli.TestHBaseNegativeCliDriver.testCliDriver[generatehfiles_require_family_path]
(batchId=244)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[llap_smb]
(batchId=151)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucket_map_join_tez1]
(batchId=170)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
(batchId=165)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid]
(batchId=169)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid_fast]
(batchId=160)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb]
(batchId=160)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vectorization_input_format_excludes]
(batchId=162)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[bucketizedhiveinputformat]
(batchId=178)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[authorization_part]
(batchId=94)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[ppd_join5]
(batchId=121)
org.apache.hadoop.hive.ql.io.TestDruidRecordWriter.testWrite (batchId=254)
org.apache.hive.beeline.cli.TestHiveCli.testNoErrorDB (batchId=186)
org.apache.hive.jdbc.TestSSL.testConnectionMismatch (batchId=232)
org.apache.hive.jdbc.TestSSL.testConnectionWrongCertCN (batchId=232)
org.apache.hive.jdbc.TestSSL.testMetastoreConnectionWrongCertCN (batchId=232)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/8801/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/8801/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-8801/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 20 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12907299 - PreCommit-HIVE-Build
> Add display escape for CR/LF to Hive CLI and Beeline
>
>
> Key: HIVE-18493
> URL: https://issues.apache.org/jira/browse/HIVE-18493
> Project: Hive
> Issue Type: Bug
> Components: Beeline, Hive
>Affects Versions: 3.0.0
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
> Attachments: HIVE-18493.01.patch, HIVE-18493.02.patch,
> HIVE-18493.03.patch, HIVE-18493.04.patch, HIVE-18493.05.patch,
> HIVE-18493.06.patch
>
>
> Add optional display escaping of carriage return and line feed so row output
> remains one line.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18493) Add display escape for CR/LF to Hive CLI and Beeline
[
https://issues.apache.org/jira/browse/HIVE-18493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16336775#comment-16336775
]
Hive QA commented on HIVE-18493:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
54s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
51s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
32s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 3m
22s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 7m
21s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
20s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 7m
34s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
14s{color} | {color:red} common: The patch generated 3 new + 424 unchanged - 0
fixed = 427 total (was 424) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
11s{color} | {color:red} beeline: The patch generated 1 new + 68 unchanged - 0
fixed = 69 total (was 68) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m
44s{color} | {color:red} root: The patch generated 23 new + 710 unchanged - 0
fixed = 733 total (was 710) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
16s{color} | {color:red} itests/hive-unit: The patch generated 17 new + 156
unchanged - 0 fixed = 173 total (was 156) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
12s{color} | {color:red} itests/util: The patch generated 2 new + 30 unchanged
- 0 fixed = 32 total (was 30) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 7m
14s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
13s{color} | {color:red} The patch generated 6 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 53m 48s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /data/hiveptest/working/yetus/dev-support/hive-personality.sh |
| git revision | master / c31c9bf |
| Default Java | 1.8.0_111 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8801/yetus/diff-checkstyle-common.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8801/yetus/diff-checkstyle-beeline.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8801/yetus/diff-checkstyle-root.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8801/yetus/diff-checkstyle-itests_hive-unit.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8801/yetus/diff-checkstyle-itests_util.txt
|
| asflicense |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8801/yetus/patch-asflicense-problems.txt
|
| modules | C: common ql beeline cli . itests/hive-unit itests/util U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8801/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Add display escape for CR/LF to Hive CLI and Beeline
>
[jira] [Updated] (HIVE-18517) Vectorization: Fix VectorMapOperator to accept VRBs and check vectorized flag correctly to support LLAP Caching
[ https://issues.apache.org/jira/browse/HIVE-18517?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-18517: Attachment: HIVE-18517.02.patch > Vectorization: Fix VectorMapOperator to accept VRBs and check vectorized flag > correctly to support LLAP Caching > --- > > Key: HIVE-18517 > URL: https://issues.apache.org/jira/browse/HIVE-18517 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 3.0.0 >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-18517.01.patch, HIVE-18517.02.patch > > > LLAP is able to deserialize and cache data from an input format (e.g. > TextInputFormat) and will deliver that cached data to VectorMapOperator as > VRBs. > Symptoms: > BinaryComparable -> VectorizedRowBatch ClassCastException > Text -> VectorizedRowBatch ClassCastException -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18422) Vectorized input format should not be used when vectorized input format is excluded and row.serde is enabled
[
https://issues.apache.org/jira/browse/HIVE-18422?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vihang Karajgaonkar updated HIVE-18422:
---
Resolution: Fixed
Fix Version/s: 2.4.0
3.0.0
Status: Resolved (was: Patch Available)
Patch merged in master and branch-2. Thanks for the review [~mmccline]
> Vectorized input format should not be used when vectorized input format is
> excluded and row.serde is enabled
>
>
> Key: HIVE-18422
> URL: https://issues.apache.org/jira/browse/HIVE-18422
> Project: Hive
> Issue Type: Bug
> Components: Vectorization
>Affects Versions: 3.0.0, 2.4.0
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
>Priority: Minor
> Fix For: 3.0.0, 2.4.0
>
> Attachments: HIVE-18422.01.patch, HIVE-18422.02.patch
>
>
> HIVE-17534 introduced a config which gives a capability to exclude certain
> inputformat from vectorized execution without affecting other input formats.
> If an input format is excluded and row.serde is enabled at the same time,
> vectorizer still sets the {{useVectorizedInputFormat}} to true which causes
> Vectorized readers to be used in row.serde mode.
> In order to reproduce:
> {noformat}
> set hive.fetch.task.conversion=none;
> set hive.vectorized.use.row.serde.deserialize=true;
> set hive.vectorized.use.vector.serde.deserialize=true;
> set hive.vectorized.execution.enabled=true;
> set hive.vectorized.execution.reduce.enabled=true;
> set hive.vectorized.row.serde.inputformat.excludes=;
> -- SORT_QUERY_RESULTS
> -- exclude MapredParquetInputFormat from vectorization, this should cause
> mapwork vectorization to be disabled
> set
> hive.vectorized.input.format.excludes=org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat,org.apache.hadoop.hive.ql.io.orc.OrcInputFormat;
> set hive.vectorized.use.vectorized.input.format=true;
> create table orcTbl (t1 tinyint, t2 tinyint)
> stored as orc;
> insert into orcTbl values (54, 9), (-104, 25), (-112, 24);
> explain vectorization select t1, t2, (t1+t2) from orcTbl where (t1+t2) > 10;
> select t1, t2, (t1+t2) from orcTbl where (t1+t2) > 10;
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18517) Vectorization: Fix VectorMapOperator to accept VRBs and check vectorized flag correctly to support LLAP Caching
[ https://issues.apache.org/jira/browse/HIVE-18517?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-18517: Summary: Vectorization: Fix VectorMapOperator to accept VRBs and check vectorized flag correctly to support LLAP Caching (was: Vectorization: Fix VectorMapOperator to accept VRBs to support LLAP Caching) > Vectorization: Fix VectorMapOperator to accept VRBs and check vectorized flag > correctly to support LLAP Caching > --- > > Key: HIVE-18517 > URL: https://issues.apache.org/jira/browse/HIVE-18517 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 3.0.0 >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-18517.01.patch > > > LLAP is able to deserialize and cache data from an input format (e.g. > TextInputFormat) and will deliver that cached data to VectorMapOperator as > VRBs. > Symptoms: > BinaryComparable -> VectorizedRowBatch ClassCastException > Text -> VectorizedRowBatch ClassCastException -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18426) Memory leak in RoutingAppender for every hive operation
[ https://issues.apache.org/jira/browse/HIVE-18426?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16336701#comment-16336701 ] Aihua Xu commented on HIVE-18426: - Thanks [~kkalyan] The change looks good. +1. > Memory leak in RoutingAppender for every hive operation > --- > > Key: HIVE-18426 > URL: https://issues.apache.org/jira/browse/HIVE-18426 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Affects Versions: 3.0.0 >Reporter: kalyan kumar kalvagadda >Assignee: kalyan kumar kalvagadda >Priority: Major > Fix For: 3.0.0 > > Attachments: HIVE-18426.001.patch, HIVE-18426.002.patch, > HIVE-18426.003.patch, HIVE-18426.004.patch, Screen Shot 2018-01-10 at 9.17.19 > AM.png > > > Each new operation creates new entry in the ConcurrentMap in RoutingAppender > but when the operation ends, AppenderControl stored in the map is retrieved > and stopped but the entry in ConcurrentMap is never cleaned up. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18517) Vectorization: Fix VectorMapOperator to accept VRBs to support LLAP Caching
[ https://issues.apache.org/jira/browse/HIVE-18517?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-18517: Description: LLAP is able to deserialize and cache data from an input format (e.g. TextInputFormat) and will deliver that cached data to VectorMapOperator as VRBs. Symptoms: BinaryComparable -> VectorizedRowBatch ClassCastException Text -> VectorizedRowBatch ClassCastException was:LLAP is able to deserialize and cache data from an input format (e.g. TextInputFormat) and will deliver that cached data to VectorMapOperator as VRBs. > Vectorization: Fix VectorMapOperator to accept VRBs to support LLAP Caching > --- > > Key: HIVE-18517 > URL: https://issues.apache.org/jira/browse/HIVE-18517 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 3.0.0 >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-18517.01.patch > > > LLAP is able to deserialize and cache data from an input format (e.g. > TextInputFormat) and will deliver that cached data to VectorMapOperator as > VRBs. > Symptoms: > BinaryComparable -> VectorizedRowBatch ClassCastException > Text -> VectorizedRowBatch ClassCastException -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18048) Support Struct type with vectorization for Parquet file
[ https://issues.apache.org/jira/browse/HIVE-18048?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16336690#comment-16336690 ] Colin Ma commented on HIVE-18048: - [~Ferd], the patch is updated with the new solution. There will be a new UDF to process the field of Struct. Can you help to review the patch, thanks for your help. > Support Struct type with vectorization for Parquet file > --- > > Key: HIVE-18048 > URL: https://issues.apache.org/jira/browse/HIVE-18048 > Project: Hive > Issue Type: Sub-task >Reporter: Colin Ma >Assignee: Colin Ma >Priority: Major > Attachments: HIVE-18048.001.patch, HIVE-18048.002.patch, > HIVE-18048.003.patch > > > Struct type is not supported in MapWork with vectorization, it should be > supported to improve the performance. > The following improvements will be implemented: > * Add fields of struct type to VectorizedRowBatchCtx. > * Improve the VectorizedParquetRecordReader to support the struct type for > parquet file. > Note: > * Orc file won't be supported. > * Filter operator won't be supported. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-17751) Separate HMS Client and HMS server into separate sub-modules
[ https://issues.apache.org/jira/browse/HIVE-17751?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexander Kolbasov updated HIVE-17751: -- Attachment: HIVE-17751.05-standalone-metastore.patch > Separate HMS Client and HMS server into separate sub-modules > > > Key: HIVE-17751 > URL: https://issues.apache.org/jira/browse/HIVE-17751 > Project: Hive > Issue Type: Sub-task > Components: Standalone Metastore >Reporter: Vihang Karajgaonkar >Assignee: Alexander Kolbasov >Priority: Major > Attachments: HIVE-17751.03-standalone-metastore.patch, > HIVE-17751.05-standalone-metastore.patch > > > external applications which are interfacing with HMS should ideally only > include HMSClient library instead of one big library containing server as > well. We should ideally have a thin client library so that cross version > support for external applications is easier. We should sub-divide the > standalone module into possibly 3 modules (one for common classes, one for > client classes and one for server) or 2 sub-modules (one for client and one > for server) so that we can generate separate jars for HMS client and server. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-17751) Separate HMS Client and HMS server into separate sub-modules
[ https://issues.apache.org/jira/browse/HIVE-17751?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexander Kolbasov updated HIVE-17751: -- Status: Patch Available (was: Open) > Separate HMS Client and HMS server into separate sub-modules > > > Key: HIVE-17751 > URL: https://issues.apache.org/jira/browse/HIVE-17751 > Project: Hive > Issue Type: Sub-task > Components: Standalone Metastore >Reporter: Vihang Karajgaonkar >Assignee: Alexander Kolbasov >Priority: Major > Attachments: HIVE-17751.03-standalone-metastore.patch, > HIVE-17751.05-standalone-metastore.patch > > > external applications which are interfacing with HMS should ideally only > include HMSClient library instead of one big library containing server as > well. We should ideally have a thin client library so that cross version > support for external applications is easier. We should sub-divide the > standalone module into possibly 3 modules (one for common classes, one for > client classes and one for server) or 2 sub-modules (one for client and one > for server) so that we can generate separate jars for HMS client and server. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18513) Query results caching
[ https://issues.apache.org/jira/browse/HIVE-18513?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16336669#comment-16336669 ] Jason Dere commented on HIVE-18513: --- [~vgarg] it compares a modified version of the query string, which is generated based on the AST and fully qualifies table references. > Query results caching > - > > Key: HIVE-18513 > URL: https://issues.apache.org/jira/browse/HIVE-18513 > Project: Hive > Issue Type: Bug > Components: Query Planning >Reporter: Jason Dere >Assignee: Jason Dere >Priority: Major > Attachments: HIVE-18513.1.patch > > > Add a query results cache that can save the results of an executed Hive query > for reuse on subsequent queries. This may be useful in cases where the same > query is issued many times, since Hive can return back the results of a > cached query rather than having to execute the full query on the cluster. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-17751) Separate HMS Client and HMS server into separate sub-modules
[ https://issues.apache.org/jira/browse/HIVE-17751?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexander Kolbasov updated HIVE-17751: -- Attachment: (was: HIVE-17751.02-standalone-metastore.patch) > Separate HMS Client and HMS server into separate sub-modules > > > Key: HIVE-17751 > URL: https://issues.apache.org/jira/browse/HIVE-17751 > Project: Hive > Issue Type: Sub-task > Components: Standalone Metastore >Reporter: Vihang Karajgaonkar >Assignee: Alexander Kolbasov >Priority: Major > Attachments: HIVE-17751.03-standalone-metastore.patch > > > external applications which are interfacing with HMS should ideally only > include HMSClient library instead of one big library containing server as > well. We should ideally have a thin client library so that cross version > support for external applications is easier. We should sub-divide the > standalone module into possibly 3 modules (one for common classes, one for > client classes and one for server) or 2 sub-modules (one for client and one > for server) so that we can generate separate jars for HMS client and server. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-17751) Separate HMS Client and HMS server into separate sub-modules
[ https://issues.apache.org/jira/browse/HIVE-17751?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexander Kolbasov updated HIVE-17751: -- Attachment: (was: HIVE-17751.01-standalone-metastore.patch) > Separate HMS Client and HMS server into separate sub-modules > > > Key: HIVE-17751 > URL: https://issues.apache.org/jira/browse/HIVE-17751 > Project: Hive > Issue Type: Sub-task > Components: Standalone Metastore >Reporter: Vihang Karajgaonkar >Assignee: Alexander Kolbasov >Priority: Major > Attachments: HIVE-17751.03-standalone-metastore.patch > > > external applications which are interfacing with HMS should ideally only > include HMSClient library instead of one big library containing server as > well. We should ideally have a thin client library so that cross version > support for external applications is easier. We should sub-divide the > standalone module into possibly 3 modules (one for common classes, one for > client classes and one for server) or 2 sub-modules (one for client and one > for server) so that we can generate separate jars for HMS client and server. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-17751) Separate HMS Client and HMS server into separate sub-modules
[ https://issues.apache.org/jira/browse/HIVE-17751?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexander Kolbasov updated HIVE-17751: -- Status: Open (was: Patch Available) > Separate HMS Client and HMS server into separate sub-modules > > > Key: HIVE-17751 > URL: https://issues.apache.org/jira/browse/HIVE-17751 > Project: Hive > Issue Type: Sub-task > Components: Standalone Metastore >Reporter: Vihang Karajgaonkar >Assignee: Alexander Kolbasov >Priority: Major > Attachments: HIVE-17751.03-standalone-metastore.patch > > > external applications which are interfacing with HMS should ideally only > include HMSClient library instead of one big library containing server as > well. We should ideally have a thin client library so that cross version > support for external applications is easier. We should sub-divide the > standalone module into possibly 3 modules (one for common classes, one for > client classes and one for server) or 2 sub-modules (one for client and one > for server) so that we can generate separate jars for HMS client and server. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-17848) Bucket Map Join : Implement an efficient way to minimize loading hash table
[ https://issues.apache.org/jira/browse/HIVE-17848?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16336651#comment-16336651 ] Deepak Jaiswal commented on HIVE-17848: --- [~gopalv] [~jdere] a gentle reminder. > Bucket Map Join : Implement an efficient way to minimize loading hash table > --- > > Key: HIVE-17848 > URL: https://issues.apache.org/jira/browse/HIVE-17848 > Project: Hive > Issue Type: Bug >Reporter: Deepak Jaiswal >Assignee: Deepak Jaiswal >Priority: Major > Attachments: HIVE-17848.2.patch, HIVE-17848.4.patch, > HIVE-17848.5.patch > > > In bucket mapjoin, each task loads its own copy of hash table which is > inefficient as load is IO heavy and due to multiple copies of same hash > table, the tables may get GCed on a busy system. > Implement a subcache with softreference to each hash table corresponding to > its bucketID such that it can be reused by a task. > This needs changes from Tez side to push bucket id to TezProcessor. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18513) Query results caching
[ https://issues.apache.org/jira/browse/HIVE-18513?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16336637#comment-16336637 ] Vineet Garg commented on HIVE-18513: [~jdere] I haven't yet looked at the patch but had one question : How do you figure out if two queries are same? Do you compare ASTs? > Query results caching > - > > Key: HIVE-18513 > URL: https://issues.apache.org/jira/browse/HIVE-18513 > Project: Hive > Issue Type: Bug > Components: Query Planning >Reporter: Jason Dere >Assignee: Jason Dere >Priority: Major > Attachments: HIVE-18513.1.patch > > > Add a query results cache that can save the results of an executed Hive query > for reuse on subsequent queries. This may be useful in cases where the same > query is issued many times, since Hive can return back the results of a > cached query rather than having to execute the full query on the cluster. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18513) Query results caching
[ https://issues.apache.org/jira/browse/HIVE-18513?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Dere updated HIVE-18513: -- Status: Patch Available (was: Open) Initial patch, let's see how the unit tests look. > Query results caching > - > > Key: HIVE-18513 > URL: https://issues.apache.org/jira/browse/HIVE-18513 > Project: Hive > Issue Type: Bug > Components: Query Planning >Reporter: Jason Dere >Assignee: Jason Dere >Priority: Major > Attachments: HIVE-18513.1.patch > > > Add a query results cache that can save the results of an executed Hive query > for reuse on subsequent queries. This may be useful in cases where the same > query is issued many times, since Hive can return back the results of a > cached query rather than having to execute the full query on the cluster. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18513) Query results caching
[ https://issues.apache.org/jira/browse/HIVE-18513?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Dere updated HIVE-18513: -- Attachment: HIVE-18513.1.patch > Query results caching > - > > Key: HIVE-18513 > URL: https://issues.apache.org/jira/browse/HIVE-18513 > Project: Hive > Issue Type: Bug > Components: Query Planning >Reporter: Jason Dere >Assignee: Jason Dere >Priority: Major > Attachments: HIVE-18513.1.patch > > > Add a query results cache that can save the results of an executed Hive query > for reuse on subsequent queries. This may be useful in cases where the same > query is issued many times, since Hive can return back the results of a > cached query rather than having to execute the full query on the cluster. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-18520) add current txnid to ValidTxnList
[ https://issues.apache.org/jira/browse/HIVE-18520?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman reassigned HIVE-18520: - > add current txnid to ValidTxnList > - > > Key: HIVE-18520 > URL: https://issues.apache.org/jira/browse/HIVE-18520 > Project: Hive > Issue Type: Improvement > Components: Transactions >Affects Versions: 1.0.0 >Reporter: Eugene Koifman >Assignee: Eugene Koifman >Priority: Major > > add the Id of the transaction that obtained this ValidTxnList > if nothing else, convenient for debugging > in particular include it in ErrorMsg.ACID_NOT_ENOUGH_HISTORY -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18514) add service output for ranger to WM DDL operations
[
https://issues.apache.org/jira/browse/HIVE-18514?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16336587#comment-16336587
]
Hive QA commented on HIVE-18514:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12907198/HIVE-18514.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 18 failed/errored test(s), 11250 tests
executed
*Failed tests:*
{noformat}
TestNegativeCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=94)
[jira] [Commented] (HIVE-18519) do not create materialized CTEs with ACID/MM
[ https://issues.apache.org/jira/browse/HIVE-18519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16336579#comment-16336579 ] Sergey Shelukhin commented on HIVE-18519: - A simple patch. This is an obscure feature that creates a query-scoped temporary table for a CTE (with (...) expression). No reason for it to use MM or ACID. [~ekoifman] can you take a look? > do not create materialized CTEs with ACID/MM > > > Key: HIVE-18519 > URL: https://issues.apache.org/jira/browse/HIVE-18519 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-18519.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18519) do not create materialized CTEs with ACID/MM
[ https://issues.apache.org/jira/browse/HIVE-18519?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-18519: Status: Patch Available (was: Open) > do not create materialized CTEs with ACID/MM > > > Key: HIVE-18519 > URL: https://issues.apache.org/jira/browse/HIVE-18519 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-18519.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18519) do not create materialized CTEs with ACID/MM
[ https://issues.apache.org/jira/browse/HIVE-18519?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-18519: Attachment: HIVE-18519.patch > do not create materialized CTEs with ACID/MM > > > Key: HIVE-18519 > URL: https://issues.apache.org/jira/browse/HIVE-18519 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-18519.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-18519) do not create materialized CTEs with ACID/MM
[ https://issues.apache.org/jira/browse/HIVE-18519?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin reassigned HIVE-18519: --- > do not create materialized CTEs with ACID/MM > > > Key: HIVE-18519 > URL: https://issues.apache.org/jira/browse/HIVE-18519 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18393) Error returned when some other type is read as string from parquet tables
[
https://issues.apache.org/jira/browse/HIVE-18393?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vihang Karajgaonkar updated HIVE-18393:
---
Resolution: Fixed
Target Version/s: 3.0.0
Status: Resolved (was: Patch Available)
Patch merged to master branch. Thanks for your contribution [~janulatha]
> Error returned when some other type is read as string from parquet tables
> -
>
> Key: HIVE-18393
> URL: https://issues.apache.org/jira/browse/HIVE-18393
> Project: Hive
> Issue Type: Bug
>Reporter: Janaki Lahorani
>Assignee: Janaki Lahorani
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HIVE-18393.1.patch, HIVE-18393.2.patch,
> HIVE-18393.3.patch, HIVE-18393.4.patch, HIVE-18393.5.patch
>
>
> TimeStamp, Decimal, Double, Float, BigInt, Int, SmallInt, Tinyint and Boolean
> when read as String, Varchar or Char should return the correct data. Now
> this results in error for parquet tables.
> Test Case:
> {code}
> drop table if exists testAltCol;
> create table testAltCol
> (cId TINYINT,
> cTimeStamp TIMESTAMP,
> cDecimal DECIMAL(38,18),
> cDoubleDOUBLE,
> cFloat FLOAT,
> cBigIntBIGINT,
> cInt INT,
> cSmallInt SMALLINT,
> cTinyint TINYINT,
> cBoolean BOOLEAN);
> insert into testAltCol values
> (1,
> '2017-11-07 09:02:49.9',
> 12345678901234567890.123456789012345678,
> 1.79e308,
> 3.4e38,
> 1234567890123456789,
> 1234567890,
> 12345,
> 123,
> TRUE);
> insert into testAltCol values
> (2,
> '1400-01-01 01:01:01.1',
> 1.1,
> 2.2,
> 3.3,
> 1,
> 2,
> 3,
> 4,
> FALSE);
> insert into testAltCol values
> (3,
> '1400-01-01 01:01:01.1',
> 10.1,
> 20.2,
> 30.3,
> 1234567890123456789,
> 1234567890,
> 12345,
> 123,
> TRUE);
> select cId, cTimeStamp from testAltCol order by cId;
> select cId, cDecimal, cDouble, cFloat from testAltCol order by cId;
> select cId, cBigInt, cInt, cSmallInt, cTinyint from testAltCol order by cId;
> select cId, cBoolean from testAltCol order by cId;
> drop table if exists testAltColP;
> create table testAltColP stored as parquet as select * from testAltCol;
> select cId, cTimeStamp from testAltColP order by cId;
> select cId, cDecimal, cDouble, cFloat from testAltColP order by cId;
> select cId, cBigInt, cInt, cSmallInt, cTinyint from testAltColP order by cId;
> select cId, cBoolean from testAltColP order by cId;
> alter table testAltColP replace columns
> (cId TINYINT,
> cTimeStamp STRING,
> cDecimal STRING,
> cDoubleSTRING,
> cFloat STRING,
> cBigIntSTRING,
> cInt STRING,
> cSmallInt STRING,
> cTinyint STRING,
> cBoolean STRING);
> select cId, cTimeStamp from testAltColP order by cId;
> select cId, cDecimal, cDouble, cFloat from testAltColP order by cId;
> select cId, cBigInt, cInt, cSmallInt, cTinyint from testAltColP order by cId;
> select cId, cBoolean from testAltColP order by cId;
> alter table testAltColP replace columns
> (cId TINYINT,
> cTimeStamp VARCHAR(100),
> cDecimal VARCHAR(100),
> cDoubleVARCHAR(100),
> cFloat VARCHAR(100),
> cBigIntVARCHAR(100),
> cInt VARCHAR(100),
> cSmallInt VARCHAR(100),
> cTinyint VARCHAR(100),
> cBoolean VARCHAR(100));
> select cId, cTimeStamp from testAltColP order by cId;
> select cId, cDecimal, cDouble, cFloat from testAltColP order by cId;
> select cId, cBigInt, cInt, cSmallInt, cTinyint from testAltColP order by cId;
> select cId, cBoolean from testAltColP order by cId;
> alter table testAltColP replace columns
> (cId TINYINT,
> cTimeStamp CHAR(100),
> cDecimal CHAR(100),
> cDoubleCHAR(100),
> cFloat CHAR(100),
> cBigIntCHAR(100),
> cInt CHAR(100),
> cSmallInt CHAR(100),
> cTinyint CHAR(100),
> cBoolean CHAR(100));
> select cId, cTimeStamp from testAltColP order by cId;
> select cId, cDecimal, cDouble, cFloat from testAltColP order by cId;
> select cId, cBigInt, cInt, cSmallInt, cTinyint from testAltColP order by cId;
> select cId, cBoolean from testAltColP order by cId;
> drop table if exists testAltColP;
> {code}
> {code}
> Error:
> FAILED: Execution Error, return code 2 from
> org.apache.hadoop.hive.ql.exec.mr.MapRedTask
> Excerpt for log:
> 2018-01-05T15:54:05,756 ERROR [LocalJobRunner Map Task Executor #0]
> mr.ExecMapper: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime
> Error while processing row [Error getting row data with exception
> java.lang.UnsupportedOperationException: Cannot inspect
> org.apache.hadoop.hive.serde2.io.TimestampWritable
> at
> org.apache.hadoop.hive.ql.io.parquet.serde.primitive.ParquetStringInspector.getPrimitiveJavaObject(ParquetStringInspector.java:77)
> {code}
--
[jira] [Commented] (HIVE-18472) Beeline gives log4j warnings
[
https://issues.apache.org/jira/browse/HIVE-18472?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16336515#comment-16336515
]
Vihang Karajgaonkar commented on HIVE-18472:
Hi [~janulatha] Can you please check if the patch works with {{hiveserver2}}
script? I was doing some final checks before merging the patch and I found that
below script doesn't work with the patch. Before the patch it used to work for
me.
{{hiveserver2 --hiveconf hive.log.dir=/tmp --hiveconf hive.log.logger=INFO
--hiveconf hive.log.file=hive.log}}
I don't think the unit tests test the shell scripts so we don't catch these
errors by precommit job.
> Beeline gives log4j warnings
>
>
> Key: HIVE-18472
> URL: https://issues.apache.org/jira/browse/HIVE-18472
> Project: Hive
> Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Janaki Lahorani
>Assignee: Janaki Lahorani
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HIVE-18472.1.patch, HIVE-18472.2.patch
>
>
> Starting Beeline gives the following warnings multiple times:
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in
> [jar:file:/opt/cloudera/parcels/CDH-6.x-1.cdh6.x.p0.215261/jars/log4j-slf4j-impl-2.8.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in
> [jar:file:/opt/cloudera/parcels/CDH-6.x-1.cdh6.x.p0.215261/jars/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: See [http://www.slf4j.org/codes.html#multiple_bindings] for an
> explanation.
> SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
> ERROR StatusLogger No log4j2 configuration file found. Using default
> configuration: logging only errors to the console. Set system property
> 'org.apache.logging.log4j.simplelog.StatusLogger.level' to TRACE to show
> Log4j2 internal initialization logging.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (HIVE-18472) Beeline gives log4j warnings
[
https://issues.apache.org/jira/browse/HIVE-18472?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16336515#comment-16336515
]
Vihang Karajgaonkar edited comment on HIVE-18472 at 1/23/18 11:06 PM:
--
Hi [~janulatha] Can you please check if the patch works with {{hiveserver2}}
script? I was doing some final checks before merging the patch and I found that
below script doesn't work with the patch. Before the patch it used to work for
me.
{{hiveserver2 --hiveconf hive.log.dir=/tmp --hiveconf hive.log.logger=INFO
--hiveconf hive.log.file=hive.log}}
I received following error
{code}
Exception in thread "main" java.lang.NoClassDefFoundError:
org/apache/logging/log4j/spi/LoggerContextFactory
at org.apache.hive.service.server.HiveServer2.main(HiveServer2.java:807)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.util.RunJar.run(RunJar.java:239)
at org.apache.hadoop.util.RunJar.main(RunJar.java:153)
Caused by: java.lang.ClassNotFoundException:
org.apache.logging.log4j.spi.LoggerContextFactory
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 7 more
{code}
I don't think the unit tests test the shell scripts so we don't catch these
errors by precommit job.
was (Author: vihangk1):
Hi [~janulatha] Can you please check if the patch works with {{hiveserver2}}
script? I was doing some final checks before merging the patch and I found that
below script doesn't work with the patch. Before the patch it used to work for
me.
{{hiveserver2 --hiveconf hive.log.dir=/tmp --hiveconf hive.log.logger=INFO
--hiveconf hive.log.file=hive.log}}
I don't think the unit tests test the shell scripts so we don't catch these
errors by precommit job.
> Beeline gives log4j warnings
>
>
> Key: HIVE-18472
> URL: https://issues.apache.org/jira/browse/HIVE-18472
> Project: Hive
> Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Janaki Lahorani
>Assignee: Janaki Lahorani
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HIVE-18472.1.patch, HIVE-18472.2.patch
>
>
> Starting Beeline gives the following warnings multiple times:
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in
> [jar:file:/opt/cloudera/parcels/CDH-6.x-1.cdh6.x.p0.215261/jars/log4j-slf4j-impl-2.8.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in
> [jar:file:/opt/cloudera/parcels/CDH-6.x-1.cdh6.x.p0.215261/jars/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: See [http://www.slf4j.org/codes.html#multiple_bindings] for an
> explanation.
> SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
> ERROR StatusLogger No log4j2 configuration file found. Using default
> configuration: logging only errors to the console. Set system property
> 'org.apache.logging.log4j.simplelog.StatusLogger.level' to TRACE to show
> Log4j2 internal initialization logging.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18514) add service output for ranger to WM DDL operations
[
https://issues.apache.org/jira/browse/HIVE-18514?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16336510#comment-16336510
]
Hive QA commented on HIVE-18514:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
9s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
2s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
38s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
51s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
37s{color} | {color:green} ql: The patch generated 0 new + 244 unchanged - 1
fixed = 244 total (was 245) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
51s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
13s{color} | {color:red} The patch generated 6 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 13m 56s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /data/hiveptest/working/yetus/dev-support/hive-personality.sh |
| git revision | master / fff86f3 |
| Default Java | 1.8.0_111 |
| asflicense |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8784/yetus/patch-asflicense-problems.txt
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8784/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> add service output for ranger to WM DDL operations
> --
>
> Key: HIVE-18514
> URL: https://issues.apache.org/jira/browse/HIVE-18514
> Project: Hive
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Major
> Attachments: HIVE-18514.patch
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18387) Minimize time that REBUILD locks the materialized view
[ https://issues.apache.org/jira/browse/HIVE-18387?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16336496#comment-16336496 ] Jesus Camacho Rodriguez commented on HIVE-18387: Rebasing patch as it was not applying cleanly anymore. > Minimize time that REBUILD locks the materialized view > -- > > Key: HIVE-18387 > URL: https://issues.apache.org/jira/browse/HIVE-18387 > Project: Hive > Issue Type: Improvement > Components: Materialized views >Affects Versions: 3.0.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Major > Attachments: HIVE-18387.01.patch, HIVE-18387.02.patch, > HIVE-18387.patch > > > Currently, REBUILD will block the materialized view while the final move task > is being executed. The idea for this improvement is to create the new > materialization in a new folder (new version) and then just flip the pointer > to the folder in the MV definition in the metastore. REBUILD operations for a > given MV should get an exclusive lock though, i.e., they cannot be executed > concurrently. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18387) Minimize time that REBUILD locks the materialized view
[ https://issues.apache.org/jira/browse/HIVE-18387?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-18387: --- Attachment: HIVE-18387.02.patch > Minimize time that REBUILD locks the materialized view > -- > > Key: HIVE-18387 > URL: https://issues.apache.org/jira/browse/HIVE-18387 > Project: Hive > Issue Type: Improvement > Components: Materialized views >Affects Versions: 3.0.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Major > Attachments: HIVE-18387.01.patch, HIVE-18387.02.patch, > HIVE-18387.patch > > > Currently, REBUILD will block the materialized view while the final move task > is being executed. The idea for this improvement is to create the new > materialization in a new folder (new version) and then just flip the pointer > to the folder in the MV definition in the metastore. REBUILD operations for a > given MV should get an exclusive lock though, i.e., they cannot be executed > concurrently. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18387) Minimize time that REBUILD locks the materialized view
[
https://issues.apache.org/jira/browse/HIVE-18387?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16336491#comment-16336491
]
Hive QA commented on HIVE-18387:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12907190/HIVE-18387.01.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/8783/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/8783/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-8783/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2018-01-23 22:41:17.242
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-8783/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2018-01-23 22:41:17.246
+ cd apache-github-source-source
+ git fetch origin
+ git reset --hard HEAD
HEAD is now at fff86f3 HIVE-17833: Publish split generation counters (Prasanth
Jayachandran reviewed by Sergey Shelukhin)
+ git clean -f -d
+ git checkout master
Already on 'master'
Your branch is up-to-date with 'origin/master'.
+ git reset --hard origin/master
HEAD is now at fff86f3 HIVE-17833: Publish split generation counters (Prasanth
Jayachandran reviewed by Sergey Shelukhin)
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2018-01-23 22:41:19.499
+ rm -rf ../yetus
+ mkdir ../yetus
+ git gc
+ cp -R . ../yetus
+ mkdir /data/hiveptest/logs/PreCommit-HIVE-Build-8783/yetus
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh
/data/hiveptest/working/scratch/build.patch
error: a/ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java: does not
exist in index
error: a/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java:
does not exist in index
error: a/ql/src/java/org/apache/hadoop/hive/ql/plan/CreateViewDesc.java: does
not exist in index
error: a/ql/src/java/org/apache/hadoop/hive/ql/plan/ImportTableDesc.java: does
not exist in index
error: a/ql/src/test/queries/clientpositive/druidmini_mv.q: does not exist in
index
error: a/ql/src/test/results/clientpositive/druid/druidmini_mv.q.out: does not
exist in index
error:
a/ql/src/test/results/clientpositive/materialized_view_create_rewrite_3.q.out:
does not exist in index
error:
a/ql/src/test/results/clientpositive/materialized_view_create_rewrite_4.q.out:
does not exist in index
error: patch failed:
ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java:5135
error: repository lacks the necessary blob to fall back on 3-way merge.
error: ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java: patch does not
apply
error: patch failed:
ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java:12550
error: repository lacks the necessary blob to fall back on 3-way merge.
error: ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java: patch
does not apply
error: patch failed:
ql/src/test/results/clientpositive/materialized_view_create_rewrite_3.q.out:542
error: repository lacks the necessary blob to fall back on 3-way merge.
error:
ql/src/test/results/clientpositive/materialized_view_create_rewrite_3.q.out:
patch does not apply
error: patch failed:
ql/src/test/results/clientpositive/materialized_view_create_rewrite_4.q.out:629
error: repository lacks the necessary blob to fall back on 3-way merge.
error:
ql/src/test/results/clientpositive/materialized_view_create_rewrite_4.q.out:
patch does not apply
error: src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java: does not exist in
index
error: src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java: does not
exist in index
error:
[jira] [Commented] (HIVE-17751) Separate HMS Client and HMS server into separate sub-modules
[
https://issues.apache.org/jira/browse/HIVE-17751?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16336489#comment-16336489
]
Hive QA commented on HIVE-17751:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12907378/HIVE-17751.03-standalone-metastore.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/8782/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/8782/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-8782/
Messages:
{noformat}
This message was trimmed, see log for full details
location: class org.apache.hadoop.hive.druid.DruidStorageHandler
[ERROR]
/data/hiveptest/working/apache-github-standalone-metastore-source/druid-handler/src/java/org/apache/hadoop/hive/druid/DruidStorageHandler.java:[452,31]
cannot find symbol
symbol: class Table
location: class org.apache.hadoop.hive.druid.DruidStorageHandler
[ERROR]
/data/hiveptest/working/apache-github-standalone-metastore-source/druid-handler/src/java/org/apache/hadoop/hive/druid/DruidStorageHandler.java:[452,71]
cannot find symbol
symbol: class MetaException
location: class org.apache.hadoop.hive.druid.DruidStorageHandler
[ERROR]
/data/hiveptest/working/apache-github-standalone-metastore-source/druid-handler/src/java/org/apache/hadoop/hive/druid/DruidStorageHandler.java:[484,33]
cannot find symbol
symbol: class Table
location: class org.apache.hadoop.hive.druid.DruidStorageHandler
[ERROR]
/data/hiveptest/working/apache-github-standalone-metastore-source/druid-handler/src/java/org/apache/hadoop/hive/druid/DruidStorageHandler.java:[484,72]
cannot find symbol
symbol: class MetaException
location: class org.apache.hadoop.hive.druid.DruidStorageHandler
[ERROR]
/data/hiveptest/working/apache-github-standalone-metastore-source/druid-handler/src/java/org/apache/hadoop/hive/druid/DruidStorageHandler.java:[494,30]
cannot find symbol
symbol: class Table
location: class org.apache.hadoop.hive.druid.DruidStorageHandler
[ERROR]
/data/hiveptest/working/apache-github-standalone-metastore-source/druid-handler/src/java/org/apache/hadoop/hive/druid/DruidStorageHandler.java:[494,69]
cannot find symbol
symbol: class MetaException
location: class org.apache.hadoop.hive.druid.DruidStorageHandler
[ERROR]
/data/hiveptest/working/apache-github-standalone-metastore-source/druid-handler/src/java/org/apache/hadoop/hive/druid/DruidStorageHandler.java:[499,35]
cannot find symbol
symbol: class Table
location: class org.apache.hadoop.hive.druid.DruidStorageHandler
[ERROR]
/data/hiveptest/working/apache-github-standalone-metastore-source/druid-handler/src/java/org/apache/hadoop/hive/druid/DruidStorageHandler.java:[499,74]
cannot find symbol
symbol: class MetaException
location: class org.apache.hadoop.hive.druid.DruidStorageHandler
[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-compiler-plugin:3.6.1:compile (default-compile)
on project hive-jdbc-handler: Compilation failure: Compilation failure:
[ERROR]
/data/hiveptest/working/apache-github-standalone-metastore-source/jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/JdbcStorageHandler.java:[18,40]
cannot find symbol
[ERROR] symbol: class HiveMetaHook
[ERROR] location: package org.apache.hadoop.hive.metastore
[ERROR]
/data/hiveptest/working/apache-github-standalone-metastore-source/jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/JdbcStorageHandler.java:[75,10]
cannot find symbol
[ERROR] symbol: class HiveMetaHook
[ERROR] location: class org.apache.hive.storage.jdbc.JdbcStorageHandler
[ERROR] -> [Help 1]
[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-compiler-plugin:3.6.1:compile (default-compile)
on project hive-hbase-handler: Compilation failure: Compilation failure:
[ERROR]
/data/hiveptest/working/apache-github-standalone-metastore-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseStorageHandler.java:[42,40]
cannot find symbol
[ERROR] symbol: class HiveMetaHook
[ERROR] location: package org.apache.hadoop.hive.metastore
[ERROR]
/data/hiveptest/working/apache-github-standalone-metastore-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseStorageHandler.java:[43,44]
package org.apache.hadoop.hive.metastore.api does not exist
[ERROR]
/data/hiveptest/working/apache-github-standalone-metastore-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseStorageHandler.java:[44,44]
package org.apache.hadoop.hive.metastore.api does not exist
[ERROR]
/data/hiveptest/working/apache-github-standalone-metastore-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseStorageHandler.java:[152,10]
cannot find symbol
[ERROR] symbol: class HiveMetaHook
[ERROR] location: class
[jira] [Commented] (HIVE-16821) Vectorization: support Explain Analyze in vectorized mode
[
https://issues.apache.org/jira/browse/HIVE-16821?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16336476#comment-16336476
]
Hive QA commented on HIVE-16821:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12907178/HIVE-16821.10.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 18 failed/errored test(s), 11646 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_sortmerge_join_2]
(batchId=48)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[mapjoin_hook]
(batchId=12)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ppd_join5] (batchId=35)
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver[encryption_move_tbl]
(batchId=173)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[llap_smb]
(batchId=151)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucket_map_join_tez1]
(batchId=170)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
(batchId=165)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid]
(batchId=169)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid_fast]
(batchId=160)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[mergejoin]
(batchId=165)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb]
(batchId=160)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[authorization_part]
(batchId=94)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[stats_aggregator_error_1]
(batchId=94)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[ppd_join5]
(batchId=121)
org.apache.hadoop.hive.ql.io.TestDruidRecordWriter.testWrite (batchId=254)
org.apache.hive.jdbc.TestSSL.testConnectionMismatch (batchId=232)
org.apache.hive.jdbc.TestSSL.testConnectionWrongCertCN (batchId=232)
org.apache.hive.jdbc.TestSSL.testMetastoreConnectionWrongCertCN (batchId=232)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/8781/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/8781/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-8781/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 18 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12907178 - PreCommit-HIVE-Build
> Vectorization: support Explain Analyze in vectorized mode
> -
>
> Key: HIVE-16821
> URL: https://issues.apache.org/jira/browse/HIVE-16821
> Project: Hive
> Issue Type: Bug
> Components: Diagnosability, Vectorization
>Affects Versions: 2.1.1, 3.0.0
>Reporter: Gopal V
>Assignee: Gopal V
>Priority: Minor
> Attachments: HIVE-16821.1.patch, HIVE-16821.10.patch,
> HIVE-16821.2.patch, HIVE-16821.2.patch, HIVE-16821.3.patch,
> HIVE-16821.7.patch, HIVE-16821.8.patch, HIVE-16821.9.patch
>
>
> Currently, to avoid a branch in the operator inner loop - the runtime stats
> are only available in non-vector mode.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Issue Comment Deleted] (HIVE-17751) Separate HMS Client and HMS server into separate sub-modules
[
https://issues.apache.org/jira/browse/HIVE-17751?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alexander Kolbasov updated HIVE-17751:
--
Comment: was deleted
(was: So far it looks like the following things should be moved from standalone
metastore to common:
{code}
/Volumes/Dev/apache/hive/hive-metastore-common/src/main/java/org/apache/hadoop/hive/metastore/security/DelegationTokenIdentifier.java
/Volumes/Dev/apache/hive/hive-metastore-common/src/main/java/org/apache/hadoop/hive/metastore/security/DelegationTokenSecretManager.java
/Volumes/Dev/apache/hive/hive-metastore-common/src/main/java/org/apache/hadoop/hive/metastore/security/DelegationTokenStore.java
/Volumes/Dev/apache/hive/hive-metastore-common/src/main/java/org/apache/hadoop/hive/metastore/security/HadoopThriftAuthBridge.java
/Volumes/Dev/apache/hive/hive-metastore-common/src/main/java/org/apache/hadoop/hive/metastore/security/HadoopThriftAuthBridge23.java
/Volumes/Dev/apache/hive/hive-metastore-common/src/main/java/org/apache/hadoop/hive/metastore/security/TFilterTransport.java
/Volumes/Dev/apache/hive/hive-metastore-common/src/main/java/org/apache/hadoop/hive/metastore/security/TokenStoreDelegationTokenSecretManager.java
/Volumes/Dev/apache/hive/hive-metastore-common/src/main/java/org/apache/hadoop/hive/metastore/security/TUGIAssumingTransport.java
/Volumes/Dev/apache/hive/hive-metastore-common/src/main/java/org/apache/hadoop/hive/metastore/utils
/Volumes/Dev/apache/hive/hive-metastore-common/src/main/java/org/apache/hadoop/hive/metastore/DefaultMetaStoreFilterHookImpl.java
/Volumes/Dev/apache/hive/hive-metastore-common/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaHook.java
/Volumes/Dev/apache/hive/hive-metastore-common/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaHookLoader.java
/Volumes/Dev/apache/hive/hive-metastore-common/src/main/java/org/apache/hadoop/hive/metastore/IMetaStoreClient.java
/Volumes/Dev/apache/hive/hive-metastore-common/src/main/java/org/apache/hadoop/hive/metastore/MetaStoreFilterHook.java
/Volumes/Dev/apache/hive/hive-metastore-common/src/main/java/org/apache/hadoop/hive/metastore/PartitionDropOptions.java
/Volumes/Dev/apache/hive/hive-metastore-common/src/main/java/org/apache/hadoop/hive/metastore/TableType.java
/Volumes/Dev/apache/hive/hive-metastore-common/src/main/java/org/apache/hadoop/security/token/delegation
/Volumes/Dev/apache/hive/hive-metastore-common/src/main/java/org/apache/hadoop/security/token/delegation/MetastoreDelegationTokenSupport.java
{code})
> Separate HMS Client and HMS server into separate sub-modules
>
>
> Key: HIVE-17751
> URL: https://issues.apache.org/jira/browse/HIVE-17751
> Project: Hive
> Issue Type: Sub-task
> Components: Standalone Metastore
>Reporter: Vihang Karajgaonkar
>Assignee: Alexander Kolbasov
>Priority: Major
> Attachments: HIVE-17751.01-standalone-metastore.patch,
> HIVE-17751.02-standalone-metastore.patch,
> HIVE-17751.03-standalone-metastore.patch
>
>
> external applications which are interfacing with HMS should ideally only
> include HMSClient library instead of one big library containing server as
> well. We should ideally have a thin client library so that cross version
> support for external applications is easier. We should sub-divide the
> standalone module into possibly 3 modules (one for common classes, one for
> client classes and one for server) or 2 sub-modules (one for client and one
> for server) so that we can generate separate jars for HMS client and server.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-17751) Separate HMS Client and HMS server into separate sub-modules
[ https://issues.apache.org/jira/browse/HIVE-17751?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexander Kolbasov updated HIVE-17751: -- Status: Patch Available (was: Open) > Separate HMS Client and HMS server into separate sub-modules > > > Key: HIVE-17751 > URL: https://issues.apache.org/jira/browse/HIVE-17751 > Project: Hive > Issue Type: Sub-task > Components: Standalone Metastore >Reporter: Vihang Karajgaonkar >Assignee: Alexander Kolbasov >Priority: Major > Attachments: HIVE-17751.01-standalone-metastore.patch, > HIVE-17751.02-standalone-metastore.patch, > HIVE-17751.03-standalone-metastore.patch > > > external applications which are interfacing with HMS should ideally only > include HMSClient library instead of one big library containing server as > well. We should ideally have a thin client library so that cross version > support for external applications is easier. We should sub-divide the > standalone module into possibly 3 modules (one for common classes, one for > client classes and one for server) or 2 sub-modules (one for client and one > for server) so that we can generate separate jars for HMS client and server. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-17751) Separate HMS Client and HMS server into separate sub-modules
[ https://issues.apache.org/jira/browse/HIVE-17751?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexander Kolbasov updated HIVE-17751: -- Status: Open (was: Patch Available) > Separate HMS Client and HMS server into separate sub-modules > > > Key: HIVE-17751 > URL: https://issues.apache.org/jira/browse/HIVE-17751 > Project: Hive > Issue Type: Sub-task > Components: Standalone Metastore >Reporter: Vihang Karajgaonkar >Assignee: Alexander Kolbasov >Priority: Major > Attachments: HIVE-17751.01-standalone-metastore.patch, > HIVE-17751.02-standalone-metastore.patch, > HIVE-17751.03-standalone-metastore.patch > > > external applications which are interfacing with HMS should ideally only > include HMSClient library instead of one big library containing server as > well. We should ideally have a thin client library so that cross version > support for external applications is easier. We should sub-divide the > standalone module into possibly 3 modules (one for common classes, one for > client classes and one for server) or 2 sub-modules (one for client and one > for server) so that we can generate separate jars for HMS client and server. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-17751) Separate HMS Client and HMS server into separate sub-modules
[ https://issues.apache.org/jira/browse/HIVE-17751?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexander Kolbasov updated HIVE-17751: -- Attachment: HIVE-17751.03-standalone-metastore.patch > Separate HMS Client and HMS server into separate sub-modules > > > Key: HIVE-17751 > URL: https://issues.apache.org/jira/browse/HIVE-17751 > Project: Hive > Issue Type: Sub-task > Components: Standalone Metastore >Reporter: Vihang Karajgaonkar >Assignee: Alexander Kolbasov >Priority: Major > Attachments: HIVE-17751.01-standalone-metastore.patch, > HIVE-17751.02-standalone-metastore.patch, > HIVE-17751.03-standalone-metastore.patch > > > external applications which are interfacing with HMS should ideally only > include HMSClient library instead of one big library containing server as > well. We should ideally have a thin client library so that cross version > support for external applications is easier. We should sub-divide the > standalone module into possibly 3 modules (one for common classes, one for > client classes and one for server) or 2 sub-modules (one for client and one > for server) so that we can generate separate jars for HMS client and server. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-17331) Path must be used as key type of the pathToAlises
[
https://issues.apache.org/jira/browse/HIVE-17331?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Oleg Danilov updated HIVE-17331:
Attachment: HIVE-17331.4.patch
> Path must be used as key type of the pathToAlises
> -
>
> Key: HIVE-17331
> URL: https://issues.apache.org/jira/browse/HIVE-17331
> Project: Hive
> Issue Type: Bug
>Reporter: Oleg Danilov
>Assignee: Oleg Danilov
>Priority: Minor
> Labels: pull-request-available
> Attachments: HIVE-17331.2.patch, HIVE-17331.3.patch,
> HIVE-17331.4.patch, HIVE-17331.patch
>
>
> This code uses String instead of Path as key type of the pathToAliases map,
> so seems like get(String) always null.
> +*GenMapRedUtils.java*+
> {code:java}
> for (int pos = 0; pos < size; pos++) {
> String taskTmpDir = taskTmpDirLst.get(pos);
> TableDesc tt_desc = tt_descLst.get(pos);
> MapWork mWork = plan.getMapWork();
> if (mWork.getPathToAliases().get(taskTmpDir) == null) {
> taskTmpDir = taskTmpDir.intern();
> Path taskTmpDirPath =
> StringInternUtils.internUriStringsInPath(new Path(taskTmpDir));
> mWork.removePathToAlias(taskTmpDirPath);
> mWork.addPathToAlias(taskTmpDirPath, taskTmpDir);
> mWork.addPathToPartitionInfo(taskTmpDirPath, new
> PartitionDesc(tt_desc, null));
> mWork.getAliasToWork().put(taskTmpDir, topOperators.get(pos));
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18499) Amend point lookup tests to check for data
[ https://issues.apache.org/jira/browse/HIVE-18499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16336438#comment-16336438 ] Janaki Lahorani commented on HIVE-18499: Yetus failures are not related to this patch. > Amend point lookup tests to check for data > -- > > Key: HIVE-18499 > URL: https://issues.apache.org/jira/browse/HIVE-18499 > Project: Hive > Issue Type: Bug >Reporter: Janaki Lahorani >Assignee: Janaki Lahorani >Priority: Major > Attachments: HIVE-18499.1.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-8436) Modify SparkWork to split works with multiple child works [Spark Branch]
[
https://issues.apache.org/jira/browse/HIVE-8436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16336431#comment-16336431
]
liyunzhang commented on HIVE-8436:
--
[~csun]: thanks for reply,
{quote}
without the copying function, the RDD cache will cache *references*
{quote}
I have not found this in spark
[document|https://spark.apache.org/docs/2.2.0/rdd-programming-guide.html#rdd-persistence].
Spark RDD cache reference not value. If have time, can you provide a link
which explain it?
{code:java}
private static class CopyFunction implements
PairFunction,
WritableComparable, Writable> {
private transient Configuration conf;
@Override
public Tuple2
call(Tuple2 tuple) throws Exception {
if (conf == null) {
conf = new Configuration();
}
return new Tuple2(tuple._1(),
WritableUtils.clone(tuple._2(), conf));
}
}{code}
{{WritableUtils.clone(tuple._2(), conf))}} used to clone tuple._2() to a new
variable. This means tuple._2() is instance of Class which can be cloned. For
Text type, it is ok. For orc/parquet format, it is not ok because
[HIVE-18289|https://issues.apache.org/jira/browse/HIVE-18289]. The reason is
OrcStruct doesn't have an empty constructor for orc when
ReflectionUtils.newInstance is called and a similar reason for parquet format.
Is there anyway to solve it?
> Modify SparkWork to split works with multiple child works [Spark Branch]
>
>
> Key: HIVE-8436
> URL: https://issues.apache.org/jira/browse/HIVE-8436
> Project: Hive
> Issue Type: Sub-task
> Components: Spark
>Reporter: Xuefu Zhang
>Assignee: Chao Sun
>Priority: Major
> Fix For: 1.1.0
>
> Attachments: HIVE-8436.1-spark.patch, HIVE-8436.10-spark.patch,
> HIVE-8436.11-spark.patch, HIVE-8436.2-spark.patch, HIVE-8436.3-spark.patch,
> HIVE-8436.4-spark.patch, HIVE-8436.5-spark.patch, HIVE-8436.6-spark.patch,
> HIVE-8436.7-spark.patch, HIVE-8436.8-spark.patch, HIVE-8436.9-spark.patch
>
>
> Based on the design doc, we need to split the operator tree of a work in
> SparkWork if the work is connected to multiple child works. The way splitting
> the operator tree is performed by cloning the original work and removing
> unwanted branches in the operator tree. Please refer to the design doc for
> details.
> This process should be done right before we generate SparkPlan. We should
> have a utility method that takes the orignal SparkWork and return a modified
> SparkWork.
> This process should also keep the information about the original work and its
> clones. Such information will be needed during SparkPlan generation
> (HIVE-8437).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18289) Support Parquet/Orc format when enable rdd cache in Hive on Spark
[
https://issues.apache.org/jira/browse/HIVE-18289?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
liyunzhang updated HIVE-18289:
--
Summary: Support Parquet/Orc format when enable rdd cache in Hive on Spark
(was: Fix jar dependency when enable rdd cache in Hive on Spark)
> Support Parquet/Orc format when enable rdd cache in Hive on Spark
> -
>
> Key: HIVE-18289
> URL: https://issues.apache.org/jira/browse/HIVE-18289
> Project: Hive
> Issue Type: Bug
>Reporter: liyunzhang
>Assignee: liyunzhang
>Priority: Major
>
> running DS/query28 when enabling HIVE-17486's 4th patch
> on tpcds_bin_partitioned_orc_10 whether on spark local or yarn mode
> command
> {code}
> set spark.local=yarn-client;
> echo 'use tpcds_bin_partitioned_orc_10;source query28.sql;'|hive --hiveconf
> spark.app.name=query28.sql --hiveconf hive.spark.optimize.shared.work=true
> -i testbench.settings -i query28.sql.setting
> {code}
> the exception
> {code}
> ava.lang.RuntimeException: java.lang.NoSuchMethodException:
> org.apache.hadoop.hive.ql.io.orc.OrcStruct.()
> 748678 at
> org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:134)
> ~[hadoop-common-2.7.3.jar:?]
> 748679 at
> org.apache.hadoop.io.WritableUtils.clone(WritableUtils.java:217)
> ~[hadoop-common-2.7.3.jar:?]
> 748680 at
> org.apache.hadoop.hive.ql.exec.spark.MapInput$CopyFunction.call(MapInput.java:85)
> ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0. 0-SNAPSHOT]
> 748681 at
> org.apache.hadoop.hive.ql.exec.spark.MapInput$CopyFunction.call(MapInput.java:72)
> ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0. 0-SNAPSHOT]
> 748682 at
> org.apache.spark.api.java.JavaPairRDD$$anonfun$pairFunToScalaFun$1.apply(JavaPairRDD.scala:1031)
> ~[spark-core_2.11-2. 0.0.jar:2.0.0]
> 748683 at
> org.apache.spark.api.java.JavaPairRDD$$anonfun$pairFunToScalaFun$1.apply(JavaPairRDD.scala:1031)
> ~[spark-core_2.11-2. 0.0.jar:2.0.0]
> 748684 at scala.collection.Iterator$$anon$11.next(Iterator.scala:409)
> ~[scala-library-2.11.8.jar:?]
> 748685 at
> org.apache.spark.storage.memory.MemoryStore.putIteratorAsValues(MemoryStore.scala:214)
> ~[spark-core_2.11-2.0.0.jar:2. 0.0]
> 748686 at
> org.apache.spark.storage.BlockManager$$anonfun$doPutIterator$1.apply(BlockManager.scala:919)
> ~[spark-core_2.11-2.0.0. jar:2.0.0]
> 748687 at
> org.apache.spark.storage.BlockManager$$anonfun$doPutIterator$1.apply(BlockManager.scala:910)
> ~[spark-core_2.11-2.0.0. jar:2.0.0]
> 748688 at
> org.apache.spark.storage.BlockManager.doPut(BlockManager.scala:866)
> ~[spark-core_2.11-2.0.0.jar:2.0.0]
> 748689 at
> org.apache.spark.storage.BlockManager.doPutIterator(BlockManager.scala:910)
> ~[spark-core_2.11-2.0.0.jar:2.0.0]
> 748690 at
> org.apache.spark.storage.BlockManager.getOrElseUpdate(BlockManager.scala:668)
> ~[spark-core_2.11-2.0.0.jar:2.0.0]
> 748691 at org.apache.spark.rdd.RDD.getOrCompute(RDD.scala:330)
> ~[spark-core_2.11-2.0.0.jar:2.0.0]
> 748692 at org.apache.spark.rdd.RDD.iterator(RDD.scala:281)
> ~[spark-core_2.11-2.0.0.jar:2.0.0]
> 748693 at
> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
> ~[spark-core_2.11-2.0.0.jar:2.0.0]
> 748694 at
> org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319)
> ~[spark-core_2.11-2.0.0.jar:2.0.0]
> 748695 at org.apache.spark.rdd.RDD.iterator(RDD.scala:283)
> ~[spark-core_2.11-2.0.0.jar:2.0.0]
> 748696 at
> org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:79)
> ~[spark-core_2.11-2
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-16821) Vectorization: support Explain Analyze in vectorized mode
[
https://issues.apache.org/jira/browse/HIVE-16821?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16336410#comment-16336410
]
Hive QA commented on HIVE-16821:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
1s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
9s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
1s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
39s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
51s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
38s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 2 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
52s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
13s{color} | {color:red} The patch generated 6 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 14m 1s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /data/hiveptest/working/yetus/dev-support/hive-personality.sh |
| git revision | master / fff86f3 |
| Default Java | 1.8.0_111 |
| whitespace |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8781/yetus/whitespace-eol.txt
|
| asflicense |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8781/yetus/patch-asflicense-problems.txt
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8781/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Vectorization: support Explain Analyze in vectorized mode
> -
>
> Key: HIVE-16821
> URL: https://issues.apache.org/jira/browse/HIVE-16821
> Project: Hive
> Issue Type: Bug
> Components: Diagnosability, Vectorization
>Affects Versions: 2.1.1, 3.0.0
>Reporter: Gopal V
>Assignee: Gopal V
>Priority: Minor
> Attachments: HIVE-16821.1.patch, HIVE-16821.10.patch,
> HIVE-16821.2.patch, HIVE-16821.2.patch, HIVE-16821.3.patch,
> HIVE-16821.7.patch, HIVE-16821.8.patch, HIVE-16821.9.patch
>
>
> Currently, to avoid a branch in the operator inner loop - the runtime stats
> are only available in non-vector mode.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-15658) hive.ql.session.SessionState start() is not atomic, SessionState thread local variable can get into inconsistent state
[
https://issues.apache.org/jira/browse/HIVE-15658?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eugene Koifman updated HIVE-15658:
--
Component/s: Transactions
> hive.ql.session.SessionState start() is not atomic, SessionState thread local
> variable can get into inconsistent state
> --
>
> Key: HIVE-15658
> URL: https://issues.apache.org/jira/browse/HIVE-15658
> Project: Hive
> Issue Type: Bug
> Components: API, HCatalog, Transactions
>Affects Versions: 1.1.0, 1.2.1, 2.0.0, 2.0.1
> Environment: CDH5.8.0, Flume 1.6.0, Hive 1.1.0
>Reporter: Michal Klempa
>Priority: Major
> Attachments: HIVE-15658_branch-1.2_1.patch,
> HIVE-15658_branch-2.1_1.patch
>
>
> Method start() in hive.ql.session.SessionState is supposed to setup needed
> preconditions, like HDFS scratch directories for session.
> This happens to be not an atomic operation with setting thread local
> variable, which can later be obtained by calling SessionState.get().
> Therefore, even is the start() method itself fails, the SessionState.get()
> does not return null and further re-use of the thread which previously
> invoked start() may lead to obtaining SessionState object in inconsistent
> state.
> I have observed this using Flume Hive Sink, which uses Hive Streaming
> interface. When the directory /tmp/hive is not writable by session user, the
> start() method fails (throwing RuntimeException). If the thread is re-used
> (like it is in Flume), further executions work with wrongly initialized
> SessionState object (HDFS dirs are non-existent). In Flume, this happens to
> me when Flume should create partition if not exists (but the code doing this
> is in Hive Streaming).
> Steps to reproduce:
> 0. create test spooldir and allow flume to write to it, in my case
> /home/ubuntu/flume_test, 775, ubuntu:flume
> 1. create Flume config (see attachment)
> 2. create Hive table
> {code}
> create table default.flume_test (column1 string, column2 string) partitioned
> by (dt string) clustered by (column1) INTO 2 BUCKETS STORED AS ORC;
> {code}
> 3. start flume agent:
> {code}
> bin/flume-ng agent -n a1 -c conf -f conf/flume-config.txt
> {code}
> 4. hdfs dfs -chmod 600 /tmp/hive
> 5. put this file into spooldir:
> {code}
> echo value1,value2 > file1
> {code}
> Expected behavior:
> Exception regarding scratch dir permissions to be thrown repeatedly.
> example (note that the line numbers are wrong as Cloudera is cloning the
> source codes here https://github.com/cloudera/flume-ng/ and here
> https://github.com/cloudera/hive):
> {code}
> 2017-01-18 12:39:38,926 WARN org.apache.flume.sink.hive.HiveSink: sink_hive_1
> : Failed connecting to EndPoint {metaStoreUri='thrift://n02.cdh.ideata:9083',
> database='default', table='flume_test', partitionVals=[20170118] }
> org.apache.flume.sink.hive.HiveWriter$ConnectException: Failed connecting to
> EndPoint {metaStoreUri='thrift://n02.cdh.ideata:9083', database='default',
> table='flume_test', partitionVals=[20170118] }
> at org.apache.flume.sink.hive.HiveWriter.(HiveWriter.java:99)
> at
> org.apache.flume.sink.hive.HiveSink.getOrCreateWriter(HiveSink.java:344)
> at
> org.apache.flume.sink.hive.HiveSink.drainOneBatch(HiveSink.java:296)
> at org.apache.flume.sink.hive.HiveSink.process(HiveSink.java:254)
> at
> org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
> at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: org.apache.flume.sink.hive.HiveWriter$ConnectException: Failed
> connecting to EndPoint {metaStoreUri='thrift://n02.cdh.ideata:9083',
> database='default', table='flume_test', partitionVals=[20170118] }
> at
> org.apache.flume.sink.hive.HiveWriter.newConnection(HiveWriter.java:380)
> at org.apache.flume.sink.hive.HiveWriter.(HiveWriter.java:86)
> ... 6 more
> Caused by: java.lang.RuntimeException: java.lang.RuntimeException: The root
> scratch dir: /tmp/hive on HDFS should be writable. Current permissions are:
> rw---
> at
> org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:540)
> at
> org.apache.hive.hcatalog.streaming.HiveEndPoint$ConnectionImpl.createPartitionIfNotExists(HiveEndPoint.java:358)
> at
> org.apache.hive.hcatalog.streaming.HiveEndPoint$ConnectionImpl.(HiveEndPoint.java:276)
> at
> org.apache.hive.hcatalog.streaming.HiveEndPoint$ConnectionImpl.(HiveEndPoint.java:243)
> at
> org.apache.hive.hcatalog.streaming.HiveEndPoint.newConnectionImpl(HiveEndPoint.java:180)
> at
>
[jira] [Commented] (HIVE-18192) Introduce WriteID per table rather than using global transaction ID
[
https://issues.apache.org/jira/browse/HIVE-18192?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16336384#comment-16336384
]
Hive QA commented on HIVE-18192:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12907175/HIVE-18192.05.patch
{color:green}SUCCESS:{color} +1 due to 20 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 107 failed/errored test(s), 11632 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[materialized_view_create_rewrite]
(batchId=246)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_join] (batchId=16)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_mapjoin]
(batchId=10)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_table_stats]
(batchId=53)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[autoColumnStats_4]
(batchId=12)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[delete_tmp_table]
(batchId=51)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[insert_values_tmp_table]
(batchId=4)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[llap_acid] (batchId=80)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[mapjoin_hook]
(batchId=12)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[materialized_view_create_rewrite]
(batchId=3)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[materialized_view_create_rewrite_2]
(batchId=89)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[materialized_view_create_rewrite_3]
(batchId=48)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[materialized_view_create_rewrite_4]
(batchId=16)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[materialized_view_create_rewrite_multi_db]
(batchId=67)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[materialized_view_rewrite_ssb]
(batchId=37)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[materialized_view_rewrite_ssb_2]
(batchId=54)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[mm_all] (batchId=67)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[mm_buckets] (batchId=60)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[mm_conversions]
(batchId=75)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[mm_default] (batchId=82)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[mm_exchangepartition]
(batchId=71)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[mm_loaddata] (batchId=46)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ppd_join5] (batchId=35)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[row__id] (batchId=78)
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver[encryption_insert_partition_dynamic]
(batchId=175)
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver[encryption_insert_partition_static]
(batchId=172)
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver[encryption_move_tbl]
(batchId=173)
org.apache.hadoop.hive.cli.TestMiniDruidCliDriver.testCliDriver[druidmini_mv]
(batchId=246)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[acid_bucket_pruning]
(batchId=148)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[llap_smb]
(batchId=151)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[mm_all]
(batchId=150)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[acid_vectorization_original]
(batchId=169)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucket_map_join_tez1]
(batchId=170)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[delete_tmp_table]
(batchId=163)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
(batchId=165)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_tmp_table]
(batchId=152)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid]
(batchId=169)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid_fast]
(batchId=160)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[materialized_view_create_rewrite]
(batchId=152)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[materialized_view_create_rewrite_2]
(batchId=171)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[materialized_view_create_rewrite_3]
(batchId=162)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[materialized_view_create_rewrite_dummy]
(batchId=163)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[materialized_view_create_rewrite_multi_db]
(batchId=166)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[materialized_view_create_rewrite_rebuild_dummy]
(batchId=157)
[jira] [Work started] (HIVE-18259) Automatic cleanup of invalidation cache for materialized views
[ https://issues.apache.org/jira/browse/HIVE-18259?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HIVE-18259 started by Jesus Camacho Rodriguez. -- > Automatic cleanup of invalidation cache for materialized views > -- > > Key: HIVE-18259 > URL: https://issues.apache.org/jira/browse/HIVE-18259 > Project: Hive > Issue Type: Improvement > Components: Materialized views >Affects Versions: 3.0.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Major > Attachments: HIVE-18259.patch > > > HIVE-14498 introduces the invalidation cache for materialized views, which > keeps track of the transactions executed on a given table to infer whether > materialized view contents are outdated or not. > Currently, the cache keeps information of transactions in memory to guarantee > quick response time, i.e., quick resolution about the view freshness, at > query rewriting time. This information can grow large, thus we would like to > run a thread that cleans useless transactions from the cache, i.e., > transactions that do invalidate any materialized view in the system, at an > interval defined by a property. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18259) Automatic cleanup of invalidation cache for materialized views
[ https://issues.apache.org/jira/browse/HIVE-18259?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-18259: --- Attachment: HIVE-18259.patch > Automatic cleanup of invalidation cache for materialized views > -- > > Key: HIVE-18259 > URL: https://issues.apache.org/jira/browse/HIVE-18259 > Project: Hive > Issue Type: Improvement > Components: Materialized views >Affects Versions: 3.0.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Major > Attachments: HIVE-18259.patch > > > HIVE-14498 introduces the invalidation cache for materialized views, which > keeps track of the transactions executed on a given table to infer whether > materialized view contents are outdated or not. > Currently, the cache keeps information of transactions in memory to guarantee > quick response time, i.e., quick resolution about the view freshness, at > query rewriting time. This information can grow large, thus we would like to > run a thread that cleans useless transactions from the cache, i.e., > transactions that do invalidate any materialized view in the system, at an > interval defined by a property. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18259) Automatic cleanup of invalidation cache for materialized views
[ https://issues.apache.org/jira/browse/HIVE-18259?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-18259: --- Status: Patch Available (was: In Progress) > Automatic cleanup of invalidation cache for materialized views > -- > > Key: HIVE-18259 > URL: https://issues.apache.org/jira/browse/HIVE-18259 > Project: Hive > Issue Type: Improvement > Components: Materialized views >Affects Versions: 3.0.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Major > Attachments: HIVE-18259.patch > > > HIVE-14498 introduces the invalidation cache for materialized views, which > keeps track of the transactions executed on a given table to infer whether > materialized view contents are outdated or not. > Currently, the cache keeps information of transactions in memory to guarantee > quick response time, i.e., quick resolution about the view freshness, at > query rewriting time. This information can grow large, thus we would like to > run a thread that cleans useless transactions from the cache, i.e., > transactions that do invalidate any materialized view in the system, at an > interval defined by a property. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18492) Wrong argument in the WorkloadManager.resetAndQueryKill()
[
https://issues.apache.org/jira/browse/HIVE-18492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16336362#comment-16336362
]
Sergey Shelukhin commented on HIVE-18492:
-
Actually given the nature of the objects equals could be replaced with ==
> Wrong argument in the WorkloadManager.resetAndQueryKill()
> -
>
> Key: HIVE-18492
> URL: https://issues.apache.org/jira/browse/HIVE-18492
> Project: Hive
> Issue Type: Bug
>Reporter: Oleg Danilov
>Assignee: Oleg Danilov
>Priority: Trivial
> Attachments: HIVE-18492.03.patch, HIVE-18492.2.patch, HIVE-18492.patch
>
>
> Caused by HIVE-18088, [~prasanth_j], could you please check this?
> {code:java}
> private void resetAndQueueKill(Map
> toKillQuery,
> KillQueryContext killQueryContext, Map toReuse) {
> WmTezSession toKill = killQueryContext.session;
> ...
> if (poolState != null) {
> poolState.getSessions().remove(toKill);
> poolState.getInitializingSessions().remove(toKill);
> ...
> {code}
> getInitializingSessions() returns List of SessionInitContext, so toKill
> definitely can't be in this list and therefore no needs to remove it.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18492) Wrong argument in the WorkloadManager.resetAndQueryKill()
[
https://issues.apache.org/jira/browse/HIVE-18492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16336359#comment-16336359
]
Sergey Shelukhin commented on HIVE-18492:
-
Updated the patch for tests. Looks like something (line endings?) is making git
not like the original patches.
+1 for the fix; is it easy to add a regression unit test for this?
TestWorkloadManager has various tests.
> Wrong argument in the WorkloadManager.resetAndQueryKill()
> -
>
> Key: HIVE-18492
> URL: https://issues.apache.org/jira/browse/HIVE-18492
> Project: Hive
> Issue Type: Bug
>Reporter: Oleg Danilov
>Assignee: Oleg Danilov
>Priority: Trivial
> Attachments: HIVE-18492.03.patch, HIVE-18492.2.patch, HIVE-18492.patch
>
>
> Caused by HIVE-18088, [~prasanth_j], could you please check this?
> {code:java}
> private void resetAndQueueKill(Map
> toKillQuery,
> KillQueryContext killQueryContext, Map toReuse) {
> WmTezSession toKill = killQueryContext.session;
> ...
> if (poolState != null) {
> poolState.getSessions().remove(toKill);
> poolState.getInitializingSessions().remove(toKill);
> ...
> {code}
> getInitializingSessions() returns List of SessionInitContext, so toKill
> definitely can't be in this list and therefore no needs to remove it.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (HIVE-18492) Wrong argument in the WorkloadManager.resetAndQueryKill()
[
https://issues.apache.org/jira/browse/HIVE-18492?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergey Shelukhin reassigned HIVE-18492:
---
Assignee: Sergey Shelukhin (was: Oleg Danilov)
> Wrong argument in the WorkloadManager.resetAndQueryKill()
> -
>
> Key: HIVE-18492
> URL: https://issues.apache.org/jira/browse/HIVE-18492
> Project: Hive
> Issue Type: Bug
>Reporter: Oleg Danilov
>Assignee: Sergey Shelukhin
>Priority: Trivial
> Attachments: HIVE-18492.03.patch, HIVE-18492.2.patch, HIVE-18492.patch
>
>
> Caused by HIVE-18088, [~prasanth_j], could you please check this?
> {code:java}
> private void resetAndQueueKill(Map
> toKillQuery,
> KillQueryContext killQueryContext, Map toReuse) {
> WmTezSession toKill = killQueryContext.session;
> ...
> if (poolState != null) {
> poolState.getSessions().remove(toKill);
> poolState.getInitializingSessions().remove(toKill);
> ...
> {code}
> getInitializingSessions() returns List of SessionInitContext, so toKill
> definitely can't be in this list and therefore no needs to remove it.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18492) Wrong argument in the WorkloadManager.resetAndQueryKill()
[
https://issues.apache.org/jira/browse/HIVE-18492?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergey Shelukhin updated HIVE-18492:
Attachment: HIVE-18492.03.patch
> Wrong argument in the WorkloadManager.resetAndQueryKill()
> -
>
> Key: HIVE-18492
> URL: https://issues.apache.org/jira/browse/HIVE-18492
> Project: Hive
> Issue Type: Bug
>Reporter: Oleg Danilov
>Assignee: Sergey Shelukhin
>Priority: Trivial
> Attachments: HIVE-18492.03.patch, HIVE-18492.2.patch, HIVE-18492.patch
>
>
> Caused by HIVE-18088, [~prasanth_j], could you please check this?
> {code:java}
> private void resetAndQueueKill(Map
> toKillQuery,
> KillQueryContext killQueryContext, Map toReuse) {
> WmTezSession toKill = killQueryContext.session;
> ...
> if (poolState != null) {
> poolState.getSessions().remove(toKill);
> poolState.getInitializingSessions().remove(toKill);
> ...
> {code}
> getInitializingSessions() returns List of SessionInitContext, so toKill
> definitely can't be in this list and therefore no needs to remove it.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (HIVE-18492) Wrong argument in the WorkloadManager.resetAndQueryKill()
[
https://issues.apache.org/jira/browse/HIVE-18492?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergey Shelukhin reassigned HIVE-18492:
---
Assignee: Oleg Danilov (was: Sergey Shelukhin)
> Wrong argument in the WorkloadManager.resetAndQueryKill()
> -
>
> Key: HIVE-18492
> URL: https://issues.apache.org/jira/browse/HIVE-18492
> Project: Hive
> Issue Type: Bug
>Reporter: Oleg Danilov
>Assignee: Oleg Danilov
>Priority: Trivial
> Attachments: HIVE-18492.03.patch, HIVE-18492.2.patch, HIVE-18492.patch
>
>
> Caused by HIVE-18088, [~prasanth_j], could you please check this?
> {code:java}
> private void resetAndQueueKill(Map
> toKillQuery,
> KillQueryContext killQueryContext, Map toReuse) {
> WmTezSession toKill = killQueryContext.session;
> ...
> if (poolState != null) {
> poolState.getSessions().remove(toKill);
> poolState.getInitializingSessions().remove(toKill);
> ...
> {code}
> getInitializingSessions() returns List of SessionInitContext, so toKill
> definitely can't be in this list and therefore no needs to remove it.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-17627) Use druid scan query instead of the select query.
[ https://issues.apache.org/jira/browse/HIVE-17627?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nishant Bangarwa updated HIVE-17627: Attachment: HIVE-17627.1.patch > Use druid scan query instead of the select query. > - > > Key: HIVE-17627 > URL: https://issues.apache.org/jira/browse/HIVE-17627 > Project: Hive > Issue Type: Bug > Components: Druid integration >Reporter: slim bouguerra >Assignee: Nishant Bangarwa >Priority: Major > Attachments: HIVE-17627.1.patch, HIVE-17627.patch > > > The biggest difference between select query and scan query is that, scan > query doesn't retain all rows in memory before rows can be returned to client. > It will cause memory pressure if too many rows required by select query. > Scan query doesn't have this issue. > Scan query can return all rows without issuing another pagination query, > which is extremely useful when query against historical or realtime node > directly. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18506) LlapBaseInputFormat - negative array index
[
https://issues.apache.org/jira/browse/HIVE-18506?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergey Shelukhin updated HIVE-18506:
Attachment: HIVE-18506.01.patch
> LlapBaseInputFormat - negative array index
> --
>
> Key: HIVE-18506
> URL: https://issues.apache.org/jira/browse/HIVE-18506
> Project: Hive
> Issue Type: Bug
>Reporter: Oleg Danilov
>Assignee: Sergey Shelukhin
>Priority: Minor
> Attachments: HIVE-18506.01.patch, HIVE-18506.patch
>
>
>
> "nextInt % value" may generate negative values:
> {code:java}
> int randIdx = rand.nextInt() % allInstances.size();
> serviceInstance = allInstances.toArray(serviceInstanceArray)[randIdx];
> {code}
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (HIVE-18506) LlapBaseInputFormat - negative array index
[
https://issues.apache.org/jira/browse/HIVE-18506?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergey Shelukhin reassigned HIVE-18506:
---
Assignee: Sergey Shelukhin (was: Oleg Danilov)
> LlapBaseInputFormat - negative array index
> --
>
> Key: HIVE-18506
> URL: https://issues.apache.org/jira/browse/HIVE-18506
> Project: Hive
> Issue Type: Bug
>Reporter: Oleg Danilov
>Assignee: Sergey Shelukhin
>Priority: Minor
> Attachments: HIVE-18506.01.patch, HIVE-18506.patch
>
>
>
> "nextInt % value" may generate negative values:
> {code:java}
> int randIdx = rand.nextInt() % allInstances.size();
> serviceInstance = allInstances.toArray(serviceInstanceArray)[randIdx];
> {code}
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18192) Introduce WriteID per table rather than using global transaction ID
[
https://issues.apache.org/jira/browse/HIVE-18192?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16336341#comment-16336341
]
Hive QA commented on HIVE-18192:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
1s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
21s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
41s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m
2s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
30s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
44s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
21s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m
0s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 9s{color}
| {color:red} storage-api generated 2 new + 0 unchanged - 2 fixed = 2 total
(was 2) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
10s{color} | {color:red} storage-api: The patch generated 29 new + 22 unchanged
- 6 fixed = 51 total (was 28) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
28s{color} | {color:red} standalone-metastore: The patch generated 18 new +
1329 unchanged - 13 fixed = 1347 total (was 1342) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m
1s{color} | {color:red} ql: The patch generated 67 new + 2632 unchanged - 43
fixed = 2699 total (was 2675) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
15s{color} | {color:red} hcatalog/streaming: The patch generated 26 new + 463
unchanged - 20 fixed = 489 total (was 483) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
15s{color} | {color:red} itests/hive-unit: The patch generated 9 new + 73
unchanged - 9 fixed = 82 total (was 82) {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 84 line(s) that end in whitespace. Use
git apply --whitespace=fix <>. Refer
https://git-scm.com/docs/git-apply {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
50s{color} | {color:red} standalone-metastore generated 3 new + 62 unchanged -
0 fixed = 65 total (was 62) {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
13s{color} | {color:red} The patch generated 6 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 29m 32s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /data/hiveptest/working/yetus/dev-support/hive-personality.sh |
| git revision | master / fff86f3 |
| Default Java | 1.8.0_111 |
| javac |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8780/yetus/diff-compile-javac-storage-api.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8780/yetus/diff-checkstyle-storage-api.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8780/yetus/diff-checkstyle-standalone-metastore.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8780/yetus/diff-checkstyle-ql.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8780/yetus/diff-checkstyle-hcatalog_streaming.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8780/yetus/diff-checkstyle-itests_hive-unit.txt
|
| whitespace |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8780/yetus/whitespace-eol.txt
|
| javadoc |
[jira] [Updated] (HIVE-18426) Memory leak in RoutingAppender for every hive operation
[ https://issues.apache.org/jira/browse/HIVE-18426?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] kalyan kumar kalvagadda updated HIVE-18426: --- Attachment: HIVE-18426.004.patch > Memory leak in RoutingAppender for every hive operation > --- > > Key: HIVE-18426 > URL: https://issues.apache.org/jira/browse/HIVE-18426 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Affects Versions: 3.0.0 >Reporter: kalyan kumar kalvagadda >Assignee: kalyan kumar kalvagadda >Priority: Major > Fix For: 3.0.0 > > Attachments: HIVE-18426.001.patch, HIVE-18426.002.patch, > HIVE-18426.003.patch, HIVE-18426.004.patch, Screen Shot 2018-01-10 at 9.17.19 > AM.png > > > Each new operation creates new entry in the ConcurrentMap in RoutingAppender > but when the operation ends, AppenderControl stored in the map is retrieved > and stopped but the entry in ConcurrentMap is never cleaned up. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-17833) Publish split generation counters
[
https://issues.apache.org/jira/browse/HIVE-17833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16336304#comment-16336304
]
Hive QA commented on HIVE-17833:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12907168/HIVE-17833.11.patch
{color:green}SUCCESS:{color} +1 due to 4 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 16 failed/errored test(s), 11250 tests
executed
*Failed tests:*
{noformat}
TestNegativeCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=94)
[jira] [Commented] (HIVE-18359) Extend grouping set limits from int to long
[ https://issues.apache.org/jira/browse/HIVE-18359?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16336285#comment-16336285 ] Prasanth Jayachandran commented on HIVE-18359: -- Rebased the patch. To handle mapperIsAbsent, we probably can rely on some path in filesystem that will be atomically cleaned up by GBY mapper. Alternate approach is for optimizer to detect such cases, and introduce an empty split (or something similar to NullScanOptimizer) to make sure atleast one mapper is launched. [~kgyrtkirk] any chance you had created a jira for it? If not I can create a separate ticket. > Extend grouping set limits from int to long > --- > > Key: HIVE-18359 > URL: https://issues.apache.org/jira/browse/HIVE-18359 > Project: Hive > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > Attachments: HIVE-18359.1.patch, HIVE-18359.2.patch, > HIVE-18359.3.patch, HIVE-18359.4.patch, HIVE-18359.5.patch, HIVE-18359.6.patch > > > Grouping sets is broken for >32 columns because of usage of Int for bitmap > (also GROUPING__ID virtual column). This assumption breaks grouping > sets/rollups/cube when number of participating aggregation columns is >32. > The easier fix would be extend it to Long for now. The correct fix would be > to use BitSets everywhere but that would require GROUPING__ID column type to > binary which will make predicates on GROUPING__ID difficult to deal with. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18359) Extend grouping set limits from int to long
[ https://issues.apache.org/jira/browse/HIVE-18359?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-18359: - Attachment: HIVE-18359.6.patch > Extend grouping set limits from int to long > --- > > Key: HIVE-18359 > URL: https://issues.apache.org/jira/browse/HIVE-18359 > Project: Hive > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > Attachments: HIVE-18359.1.patch, HIVE-18359.2.patch, > HIVE-18359.3.patch, HIVE-18359.4.patch, HIVE-18359.5.patch, HIVE-18359.6.patch > > > Grouping sets is broken for >32 columns because of usage of Int for bitmap > (also GROUPING__ID virtual column). This assumption breaks grouping > sets/rollups/cube when number of participating aggregation columns is >32. > The easier fix would be extend it to Long for now. The correct fix would be > to use BitSets everywhere but that would require GROUPING__ID column type to > binary which will make predicates on GROUPING__ID difficult to deal with. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-17627) Use druid scan query instead of the select query.
[ https://issues.apache.org/jira/browse/HIVE-17627?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nishant Bangarwa updated HIVE-17627: Attachment: (was: HIVE-17627.patch) > Use druid scan query instead of the select query. > - > > Key: HIVE-17627 > URL: https://issues.apache.org/jira/browse/HIVE-17627 > Project: Hive > Issue Type: Bug > Components: Druid integration >Reporter: slim bouguerra >Assignee: Nishant Bangarwa >Priority: Major > Attachments: HIVE-17627.patch > > > The biggest difference between select query and scan query is that, scan > query doesn't retain all rows in memory before rows can be returned to client. > It will cause memory pressure if too many rows required by select query. > Scan query doesn't have this issue. > Scan query can return all rows without issuing another pagination query, > which is extremely useful when query against historical or realtime node > directly. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-17627) Use druid scan query instead of the select query.
[ https://issues.apache.org/jira/browse/HIVE-17627?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nishant Bangarwa updated HIVE-17627: Attachment: HIVE-17627.patch > Use druid scan query instead of the select query. > - > > Key: HIVE-17627 > URL: https://issues.apache.org/jira/browse/HIVE-17627 > Project: Hive > Issue Type: Bug > Components: Druid integration >Reporter: slim bouguerra >Assignee: Nishant Bangarwa >Priority: Major > Attachments: HIVE-17627.patch > > > The biggest difference between select query and scan query is that, scan > query doesn't retain all rows in memory before rows can be returned to client. > It will cause memory pressure if too many rows required by select query. > Scan query doesn't have this issue. > Scan query can return all rows without issuing another pagination query, > which is extremely useful when query against historical or realtime node > directly. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-17833) Publish split generation counters
[
https://issues.apache.org/jira/browse/HIVE-17833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16336255#comment-16336255
]
Hive QA commented on HIVE-17833:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
1s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
38s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
50s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
34s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
53s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
13s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
21s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
36s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
37s{color} | {color:red} ql: The patch generated 18 new + 228 unchanged - 3
fixed = 246 total (was 231) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
14s{color} | {color:red} itests/hive-unit: The patch generated 7 new + 28
unchanged - 0 fixed = 35 total (was 28) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
14s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
13s{color} | {color:red} The patch generated 6 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 17m 35s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /data/hiveptest/working/yetus/dev-support/hive-personality.sh |
| git revision | master / 525f8ab |
| Default Java | 1.8.0_111 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8779/yetus/diff-checkstyle-ql.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8779/yetus/diff-checkstyle-itests_hive-unit.txt
|
| asflicense |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8779/yetus/patch-asflicense-problems.txt
|
| modules | C: ql itests itests/hive-unit U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8779/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Publish split generation counters
> -
>
> Key: HIVE-17833
> URL: https://issues.apache.org/jira/browse/HIVE-17833
> Project: Hive
> Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HIVE-17833.1.patch, HIVE-17833.10.patch,
> HIVE-17833.11.patch, HIVE-17833.2.patch, HIVE-17833.3.patch,
> HIVE-17833.4.patch, HIVE-17833.5.patch, HIVE-17833.6.patch,
> HIVE-17833.7.patch, HIVE-17833.8.patch, HIVE-17833.9.patch
>
>
> With TEZ-3856, tez counters are exposed via input initializers which can be
> used to publish split generation counters.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18413) Grouping of an empty result set may only contain null values
[
https://issues.apache.org/jira/browse/HIVE-18413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16336239#comment-16336239
]
Prasanth Jayachandran commented on HIVE-18413:
--
[~kgyrtkirk] Does this also fix the empty grouping sets issue that was
discussed in HIVE-18359?
> Grouping of an empty result set may only contain null values
>
>
> Key: HIVE-18413
> URL: https://issues.apache.org/jira/browse/HIVE-18413
> Project: Hive
> Issue Type: Bug
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HIVE-18413.01.patch, HIVE-18413.02.patch,
> HIVE-18413.03.patch
>
>
> exposed by: HIVE-18359
> in case of vectorization, the summary row object was left as is (presumed
> null earlier); which may cause it to be inconsistent isNull conditions in
> .VectorHashKeyWrapperBatch
> issue happens only if:
> * vectorizable groupby
> * groupping set contains empty
> * non-trivial empty; mapper is run
> * groupping key is select ; with a type which is backed by a bytea; ex:string
> {code}
> set hive.vectorized.execution.enabled=true;
> create table tx2 (a integer,b integer,c integer,d double,u string,bi binary)
> stored as orc;
> insert into tx2 values
> (1,2,3,1.1,'x','b'),
> (3,2,3,1.1,'y','b');
> select sum(a),
> u,
> bi,
> 'asd',
> grouping(bi),
> 'NULL,1' as expected
> fromtx2
> where a=2
> group by a,u,bi grouping sets ( u, (), bi);
> {code}
> causes:
> {code}
> Caused by: java.lang.NullPointerException
> at java.lang.System.arraycopy(Native Method)
> at
> org.apache.hadoop.hive.ql.exec.vector.BytesColumnVector.setVal(BytesColumnVector.java:173)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorHashKeyWrapperBatch.assignRowColumn(VectorHashKeyWrapperBatch.java:1065)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorGroupByOperator.writeSingleRow(VectorGroupByOperator.java:1134)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorGroupByOperator.access$800(VectorGroupByOperator.java:74)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorGroupByOperator$ProcessingModeReduceMergePartial.close(VectorGroupByOperator.java:862)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorGroupByOperator.closeOp(VectorGroupByOperator.java:1176)
> at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:705)
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.close(ReduceRecordProcessor.java:383)
> ... 16 more
> ]], Vertex did not succeed due to OWN_TASK_FAILURE, failedTasks:1
> killedTasks:0, Vertex vertex_1515531021543_0001_12_01 [Reducer 2]
> killed/failed due to:OWN_TASK_FAILURE]DAG did not succeed due to
> VERTEX_FAILURE. failedVertices:1 killedVertices:0
> 2018-01-09T12:50:30,611 DEBUG [01fdcefd-40b0-45a6-8e5b-b1cd14241088 main]
> ql.Driver: Shutting down query
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-17833) Publish split generation counters
[ https://issues.apache.org/jira/browse/HIVE-17833?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-17833: - Resolution: Fixed Fix Version/s: 3.0.0 Status: Resolved (was: Patch Available) Test failures are unrelated to this patch and are happening in master already. Committed patch to master. Thanks for the reviews! > Publish split generation counters > - > > Key: HIVE-17833 > URL: https://issues.apache.org/jira/browse/HIVE-17833 > Project: Hive > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > Fix For: 3.0.0 > > Attachments: HIVE-17833.1.patch, HIVE-17833.10.patch, > HIVE-17833.11.patch, HIVE-17833.2.patch, HIVE-17833.3.patch, > HIVE-17833.4.patch, HIVE-17833.5.patch, HIVE-17833.6.patch, > HIVE-17833.7.patch, HIVE-17833.8.patch, HIVE-17833.9.patch > > > With TEZ-3856, tez counters are exposed via input initializers which can be > used to publish split generation counters. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
