[jira] [Created] (HIVE-15745) TestMiniLlapLocalCliDriver. vector_varchar_simple,vector_char_simple

2017-01-26 Thread Thejas M Nair (JIRA) 
Thejas M Nair created HIVE-15745:


 Summary: TestMiniLlapLocalCliDriver. 
vector_varchar_simple,vector_char_simple
 Key: HIVE-15745
 URL: https://issues.apache.org/jira/browse/HIVE-15745
 Project: Hive
  Issue Type: Sub-task
Affects Versions: 2.2.0
Reporter: Thejas M Nair


TestMiniLlapLocalCliDriver. vector_varchar_simple,vector_char_simple are 
failing occasionally

vector_varchar_simple failed in 
https://builds.apache.org/job/PreCommit-HIVE-Build/3204/testReport/
vector_char_simple failed in 
https://builds.apache.org/job/PreCommit-HIVE-Build/3205/testReport/org.apache.hadoop.hive.cli/TestMiniLlapLocalCliDriver/testCliDriver_vector_char_simple_/





--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-15744) Flaky test: TestPerfCliDriver.query23, query14

2017-01-26 Thread Thejas M Nair (JIRA) 
Thejas M Nair created HIVE-15744:


 Summary: Flaky test: TestPerfCliDriver.query23, query14
 Key: HIVE-15744
 URL: https://issues.apache.org/jira/browse/HIVE-15744
 Project: Hive
  Issue Type: Sub-task
Affects Versions: 2.2.0
Reporter: Thejas M Nair


There is some flakiness in these tests -
https://builds.apache.org/job/PreCommit-HIVE-Build/3206/testReport/org.apache.hadoop.hive.cli/TestPerfCliDriver/testCliDriver_query23_/
https://builds.apache.org/job/PreCommit-HIVE-Build/3204/testReport/org.apache.hadoop.hive.cli/TestPerfCliDriver/testCliDriver_query14_/

The diff looks like this - 
{code}
Running: diff -a 
/home/hiveptest/130.211.230.155-hiveptest-1/apache-github-source-source/itests/qtest/target/qfile-results/clientpositive/query14.q.out
 
/home/hiveptest/130.211.230.155-hiveptest-1/apache-github-source-source/ql/src/test/results/clientpositive/perf/query14.q.out
0a1,2
> Warning: Shuffle Join MERGEJOIN[916][tables = [$hdt$_1, $hdt$_2]] in Stage 
> 'Reducer 114' is a cross product
> Warning: Shuffle Join MERGEJOIN[917][tables = [$hdt$_1, $hdt$_2, $hdt$_0]] in 
> Stage 'Reducer 115' is a cross product
5,6d6
< Warning: Shuffle Join MERGEJOIN[916][tables = [$hdt$_1, $hdt$_2]] in Stage 
'Reducer 114' is a cross product
< Warning: Shuffle Join MERGEJOIN[917][tables = [$hdt$_1, $hdt$_2, $hdt$_0]] in 
Stage 'Reducer 115' is a cross product

{code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-15743) vectorized text parsing: speed up double parse

2017-01-26 Thread Sergey Shelukhin (JIRA) 
Sergey Shelukhin created HIVE-15743:
---

 Summary: vectorized text parsing: speed up double parse
 Key: HIVE-15743
 URL: https://issues.apache.org/jira/browse/HIVE-15743
 Project: Hive
  Issue Type: Bug
Reporter: Sergey Shelukhin


{noformat}
Double.parseDouble(
new String(bytes, fieldStart, fieldLength, 
StandardCharsets.UTF_8));{noformat}
This takes ~25% of the query time in some cases.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 55994: HIVE-15736: Add unit tests to Utilities.getInputSummary() method for multi-threading cases

2017-01-26 Thread Sahil Takiar 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/55994/#review163222
---




ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java (line 2211)


Yeah feels like this was suppose to use `jobConf` instead of `myJobConf`



ql/src/test/org/apache/hadoop/hive/ql/exec/InputEstimatorTestClass.java (line 
40)


Is this necessary?



ql/src/test/org/apache/hadoop/hive/ql/exec/TestUtilities.java (line 31)


nit: remove the `.*`

same with ones below.



ql/src/test/org/apache/hadoop/hive/ql/exec/TestUtilities.java (line 363)


Do these need to be so high? My understanding is that this will create 100 
folders? Can it just be something smaller like 5?


- Sahil Takiar


On Jan. 26, 2017, 8:09 p.m., Sergio Pena wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/55994/
> ---
> 
> (Updated Jan. 26, 2017, 8:09 p.m.)
> 
> 
> Review request for hive, Mohit Sabharwal, Sahil Takiar, and Vihang 
> Karajgaonkar.
> 
> 
> Bugs: HIVE-15736
> https://issues.apache.org/jira/browse/HIVE-15736
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Added unit tests on TestUtilities to validate 
> - Single and multiple threads 
> - InputEstimator usage
> - ContentSummaryInputFormat usage.
> 
> It also fixed an issue with the InputEstimator scenario where the values 
> returned by the InputEstimator where overriden later by the correct 
> filesystem calls.
> 
> An interesting thing (code commented while it is on review) is that when 
> executing the InputEstimator code path the line commented seems are not 
> needed. It might
> be that the idea was to set some configurations to the jobConf, but the 
> jobConf was never passed as parameter to the estimate method. Please help me 
> verify this.
> 
> 
> Diffs
> -
> 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 
> 68dd5e7247415dec1e353010ea34481c4f2fc6cd 
>   ql/src/test/org/apache/hadoop/hive/ql/exec/InputEstimatorTestClass.java 
> PRE-CREATION 
>   ql/src/test/org/apache/hadoop/hive/ql/exec/TestUtilities.java 
> e444946e990d9adb90ce24837cfe4edcf5126d3a 
> 
> Diff: https://reviews.apache.org/r/55994/diff/
> 
> 
> Testing
> ---
> 
> Waiting for tests HiveQA
> 
> 
> Thanks,
> 
> Sergio Pena
> 
>

[jira] [Created] (HIVE-15742) Column stats should be preserved when it is renamed

2017-01-26 Thread Chaoyu Tang (JIRA) 
Chaoyu Tang created HIVE-15742:
--

 Summary: Column stats should be preserved when it is renamed
 Key: HIVE-15742
 URL: https://issues.apache.org/jira/browse/HIVE-15742
 Project: Hive
  Issue Type: Improvement
  Components: Statistics
Reporter: Chaoyu Tang
Assignee: Chaoyu Tang


Currently, when a column is renamed, its stats is delete. Recreating it could 
be expensive and we need preserve it if possible.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 55977: HIVE-15672 LLAP text cache: improve first query perf II

2017-01-26 Thread Sergey Shelukhin 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/55977/
---

(Updated Jan. 27, 2017, 3:17 a.m.)


Review request for hive, Gopal V and Prasanth_J.


Repository: hive-git


Description
---

see jira


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 291ca7d 
  
llap-server/src/java/org/apache/hadoop/hive/llap/cache/SerDeLowLevelCacheImpl.java
 85fae9a 
  
llap-server/src/java/org/apache/hadoop/hive/llap/io/decode/OrcEncodedDataConsumer.java
 d0e70d1 
  
llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/SerDeEncodedDataReader.java
 9ab26e6 
  
llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/VertorDeserializeOrcWriter.java
 98fc9df 
  orc/src/java/org/apache/orc/impl/TreeReaderFactory.java 8b097ba 
  orc/src/java/org/apache/orc/impl/WriterImpl.java b1f3cfb 
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcInputFormat.java 6281edd 
  
ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/EncodedTreeReaderFactory.java
 0dba1a0 
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/Reader.java 1c5f0e6 
  
storage-api/src/java/org/apache/hadoop/hive/common/io/encoded/EncodedColumnBatch.java
 b894c11e 
  
storage-api/src/java/org/apache/hadoop/hive/ql/exec/vector/BytesColumnVector.java
 552982c 
  storage-api/src/java/org/apache/hadoop/hive/ql/exec/vector/ColumnVector.java 
6f090a1 
  
storage-api/src/java/org/apache/hadoop/hive/ql/exec/vector/DecimalColumnVector.java
 e4f8d82 
  
storage-api/src/java/org/apache/hadoop/hive/ql/exec/vector/DoubleColumnVector.java
 bd421f4 
  
storage-api/src/java/org/apache/hadoop/hive/ql/exec/vector/IntervalDayTimeColumnVector.java
 c4a6c0f 
  
storage-api/src/java/org/apache/hadoop/hive/ql/exec/vector/LongColumnVector.java
 80d4731 
  
storage-api/src/java/org/apache/hadoop/hive/ql/exec/vector/MultiValuedColumnVector.java
 1aeff83 
  
storage-api/src/java/org/apache/hadoop/hive/ql/exec/vector/StructColumnVector.java
 cf07bca 
  
storage-api/src/java/org/apache/hadoop/hive/ql/exec/vector/TimestampColumnVector.java
 28997a0 
  
storage-api/src/java/org/apache/hadoop/hive/ql/exec/vector/UnionColumnVector.java
 0c61243 

Diff: https://reviews.apache.org/r/55977/diff/


Testing
---


Thanks,

Sergey Shelukhin

[jira] [Created] (HIVE-15741) Faster unsafe byte array comparisons

2017-01-26 Thread Teddy Choi (JIRA) 
Teddy Choi created HIVE-15741:
-

 Summary: Faster unsafe byte array comparisons
 Key: HIVE-15741
 URL: https://issues.apache.org/jira/browse/HIVE-15741
 Project: Hive
  Issue Type: Improvement
Reporter: Teddy Choi
Assignee: Teddy Choi
Priority: Minor


Byte array comparison is heavily used in joins and string conditions. Pure Java 
implementation is simple but not performant. An implementation with 
Unsafe#getLong is much faster. It's already implemented in 
org.apache.hadoop.io.WritableComparator#compare. The WritableComparator class 
handles exceptional cases, including a different endian and no access to 
Unsafe, and it was used for many years in production.

This patch will replace pure Java byte array comparisons with safe and faster 
unsafe ones to get more performance.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 55898: HIVE-15698 Vectorization support for min/max/bloomfilter runtime filtering

2017-01-26 Thread Jason Dere 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/55898/
---

(Updated Jan. 26, 2017, 11:47 p.m.)


Review request for hive, Deepak Jaiswal and Matt McCline.


Changes
---

Incorporating review feedback.


Bugs: HIVE-15698
https://issues.apache.org/jira/browse/HIVE-15698


Repository: hive-git


Description
---

Adds vectorized support for ExprNodeDynamicValue, BETWEEN() with DynamicValue, 
bloom_filter() aggregation function, and in_bloom_filter()


Diffs (updated)
-

  ant/src/org/apache/hadoop/hive/ant/GenVectorCode.java e9fe8fa 
  itests/src/test/resources/testconfiguration.properties e966959 
  
ql/src/gen/vectorization/ExpressionTemplates/FilterColumnBetweenDynamicValue.txt
 PRE-CREATION 
  ql/src/gen/vectorization/ExpressionTemplates/FilterDecimalColumnBetween.txt 
d68edfa 
  ql/src/gen/vectorization/ExpressionTemplates/FilterStringColumnBetween.txt 
e8049da 
  ql/src/gen/vectorization/ExpressionTemplates/FilterTimestampColumnBetween.txt 
4298d79 
  
ql/src/gen/vectorization/ExpressionTemplates/FilterTruncStringColumnBetween.txt 
94a174d 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/DynamicValueRegistryTez.java 
7bbedf6 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorExpressionDescriptor.java
 217af3f 
  ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorFilterOperator.java 
261246b 
  ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorSelectOperator.java 
f7fec8f 
  ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorizationContext.java 
c887757 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/DynamicValueVectorExpression.java
 PRE-CREATION 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/VectorExpression.java
 8fca8a1 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/VectorInBloomFilterColDynamicValue.java
 PRE-CREATION 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/aggregates/VectorUDAFBloomFilter.java
 PRE-CREATION 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/aggregates/VectorUDAFBloomFilterMerge.java
 PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/Vectorizer.java 
e3d9d7f 
  ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFBloomFilter.java 
fb9a140 
  
ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFInBloomFilter.java 
1b7de6c 
  ql/src/test/org/apache/hadoop/hive/ql/optimizer/physical/TestVectorizer.java 
59cb31e 
  ql/src/test/queries/clientpositive/vectorized_dynamic_semijoin_reduction.q 
PRE-CREATION 
  ql/src/test/results/clientpositive/llap/mergejoin.q.out 4ec2a71 
  ql/src/test/results/clientpositive/llap/orc_llap.q.out 90055a5 
  ql/src/test/results/clientpositive/llap/vector_binary_join_groupby.q.out 
9fbce7d 
  
ql/src/test/results/clientpositive/llap/vectorized_dynamic_partition_pruning.q.out
 3d087b3 
  
ql/src/test/results/clientpositive/llap/vectorized_dynamic_semijoin_reduction.q.out
 PRE-CREATION 
  ql/src/test/results/clientpositive/vector_binary_join_groupby.q.out 850278e 
  storage-api/src/java/org/apache/hive/common/util/BloomFilter.java d44bba8 
  storage-api/src/test/org/apache/hive/common/util/TestBloomFilter.java 63c7050 

Diff: https://reviews.apache.org/r/55898/diff/


Testing
---

qtests


Thanks,

Jason Dere

Re: Review Request 55898: HIVE-15698 Vectorization support for min/max/bloomfilter runtime filtering

2017-01-26 Thread Jason Dere 


> On Jan. 26, 2017, 10:32 a.m., Matt McCline wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/DynamicValueVectorExpression.java,
> >  line 76
> > 
> >
> > One pattern we have added is also setting isNull to false on the value 
> > path.

Ok, will add this to all of the evaluateLong/Double/etc methods


> On Jan. 26, 2017, 10:32 a.m., Matt McCline wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/DynamicValueVectorExpression.java,
> >  line 146
> > 
> >
> > So, boolean is not applicable for dynamic values?

Missed that - will add boolean.


> On Jan. 26, 2017, 10:32 a.m., Matt McCline wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/VectorInBloomFilterColDynamicValue.java,
> >  line 270
> > 
> >
> > With the FastDecimal you can get the String from the 
> > HiveDecimalWritable directly now.  I.e. the DecimalColumnVector.vector.  
> > So, you do not have to call getHiveDecimal and get better performance.
> > 
> > An additional performance option is available, too.  You can call a 
> > variation of toString that passes a scratch byte[] that makes to String 
> > conversion even faster...

Thanks for the tip. Will try using.


- Jason


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/55898/#review163114
---


On Jan. 25, 2017, 12:03 a.m., Jason Dere wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/55898/
> ---
> 
> (Updated Jan. 25, 2017, 12:03 a.m.)
> 
> 
> Review request for hive, Deepak Jaiswal and Matt McCline.
> 
> 
> Bugs: HIVE-15698
> https://issues.apache.org/jira/browse/HIVE-15698
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Adds vectorized support for ExprNodeDynamicValue, BETWEEN() with 
> DynamicValue, bloom_filter() aggregation function, and in_bloom_filter()
> 
> 
> Diffs
> -
> 
>   ant/src/org/apache/hadoop/hive/ant/GenVectorCode.java e9fe8fa 
>   
> ql/src/gen/vectorization/ExpressionTemplates/FilterColumnBetweenDynamicValue.txt
>  PRE-CREATION 
>   ql/src/gen/vectorization/ExpressionTemplates/FilterDecimalColumnBetween.txt 
> d68edfa 
>   ql/src/gen/vectorization/ExpressionTemplates/FilterStringColumnBetween.txt 
> e8049da 
>   
> ql/src/gen/vectorization/ExpressionTemplates/FilterTimestampColumnBetween.txt 
> 4298d79 
>   
> ql/src/gen/vectorization/ExpressionTemplates/FilterTruncStringColumnBetween.txt
>  94a174d 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/DynamicValueRegistryTez.java 
> 7bbedf6 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorExpressionDescriptor.java
>  217af3f 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorFilterOperator.java 
> 261246b 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorSelectOperator.java 
> f7fec8f 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorizationContext.java 
> c887757 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/DynamicValueVectorExpression.java
>  PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/VectorExpression.java
>  8fca8a1 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/VectorInBloomFilterColDynamicValue.java
>  PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/aggregates/VectorUDAFBloomFilter.java
>  PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/aggregates/VectorUDAFBloomFilterMerge.java
>  PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/Vectorizer.java 
> e3d9d7f 
>   
> ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFInBloomFilter.java
>  1b7de6c 
>   
> ql/src/test/org/apache/hadoop/hive/ql/optimizer/physical/TestVectorizer.java 
> 59cb31e 
>   ql/src/test/queries/clientpositive/vectorized_dynamic_semijoin_reduction.q 
> PRE-CREATION 
>   
> ql/src/test/results/clientpositive/llap/vectorized_dynamic_partition_pruning.q.out
>  3d087b3 
>   
> ql/src/test/results/clientpositive/llap/vectorized_dynamic_semijoin_reduction.q.out
>  PRE-CREATION 
>   storage-api/src/java/org/apache/hive/common/util/BloomFilter.java d44bba8 
>   storage-api/src/test/org/apache/hive/common/util/TestBloomFilter.java 
> 63c7050 
> 
> Diff: https://reviews.apache.org/r/55898/diff/
> 
> 
> Testing
> ---
> 
> qtests
> 
> 
> Thanks,
> 
> Jason Dere
> 
>

[jira] [Created] (HIVE-15740) Include hive-hcatalog-core.jar and hive-hcatalog-server-extensions.jar in binary distribution

2017-01-26 Thread Daniel Dai (JIRA) 
Daniel Dai created HIVE-15740:
-

 Summary: Include hive-hcatalog-core.jar and 
hive-hcatalog-server-extensions.jar in binary distribution
 Key: HIVE-15740
 URL: https://issues.apache.org/jira/browse/HIVE-15740
 Project: Hive
  Issue Type: Bug
  Components: distribution
Reporter: Daniel Dai
Assignee: Daniel Dai


Currently both jars are in hcatalog/share/hcatalog and not in classpath. 
Metastore using DbNotificationListener will fail with CNF exception.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 55816: HIVE-15680: Incorrect results when hive.optimize.index.filter=true and same ORC table is referenced twice in query

2017-01-26 Thread Anthony Hsu 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/55816/
---

(Updated Jan. 26, 2017, 10:45 p.m.)


Review request for hive.


Changes
---

Fix NPEs in LLAP tests.


Bugs: HIVE-15680
https://issues.apache.org/jira/browse/HIVE-15680


Repository: hive-git


Description
---

HIVE-15680: Incorrect results when hive.optimize.index.filter=true and same ORC 
table is referenced twice in query


Diffs (updated)
-

  ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 
1cf24b41c047b9bc43e42a2940ff54a3e331190c 
  ql/src/java/org/apache/hadoop/hive/ql/io/HiveInputFormat.java 
3ee8fdc24aa115710d2c42f5c44c7f28e0544589 
  ql/src/test/queries/clientpositive/orc_ppd_same_table_multiple_aliases.q 
PRE-CREATION 
  ql/src/test/results/clientpositive/orc_ppd_same_table_multiple_aliases.q.out 
PRE-CREATION 

Diff: https://reviews.apache.org/r/55816/diff/


Testing
---

Added qtest.


Thanks,

Anthony Hsu

[jira] [Created] (HIVE-15739) Incorrect exception message in PartExprEvalUtils

2017-01-26 Thread Mark Wagner (JIRA) 
Mark Wagner created HIVE-15739:
--

 Summary: Incorrect exception message in PartExprEvalUtils
 Key: HIVE-15739
 URL: https://issues.apache.org/jira/browse/HIVE-15739
 Project: Hive
  Issue Type: Bug
Reporter: Mark Wagner
Assignee: Mark Wagner
Priority: Minor


The check is on partSpec, not partProps:
{noformat}
if (partSpec.size() != partKeyTypes.length) {
throw new HiveException("Internal error : Partition Spec size, " + 
partProps.size() +
" doesn't match partition key definition size, " + 
partKeyTypes.length);
}
{noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-15738) Codahale metrics logs constant errors from CLI

2017-01-26 Thread Sergey Shelukhin (JIRA) 
Sergey Shelukhin created HIVE-15738:
---

 Summary: Codahale metrics logs constant errors from CLI
 Key: HIVE-15738
 URL: https://issues.apache.org/jira/browse/HIVE-15738
 Project: Hive
  Issue Type: Bug
Reporter: Sergey Shelukhin


Seen on master
{noformat}
2017-01-26T16:16:43,948  WARN [Timer-0] metrics2.CodahaleMetrics: Error writing 
JSON Metrics to file
java.io.FileNotFoundException: /var/log/hive/hivemetastore-report.json.tmp 
(Permission denied)
at java.io.FileOutputStream.open0(Native Method) ~[?:1.8.0_77]
at java.io.FileOutputStream.open(FileOutputStream.java:270) 
~[?:1.8.0_77]
at java.io.FileOutputStream.(FileOutputStream.java:213) 
~[?:1.8.0_77]
at 
org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.(RawLocalFileSystem.java:222)
 ~[hadoop-common-2.7.1.2.3.0.0-2557.jar:?]
at 
org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.(RawLocalFileSystem.java:209)
 ~[hadoop-common-2.7.1.2.3.0.0-2557.jar:?]
at 
org.apache.hadoop.fs.RawLocalFileSystem.createOutputStreamWithMode(RawLocalFileSystem.java:305)
 ~[hadoop-common-2.7.1.2.3.0.0-2557.jar:?]
at 
org.apache.hadoop.fs.RawLocalFileSystem.create(RawLocalFileSystem.java:293) 
~[hadoop-common-2.7.1.2.3.0.0-2557.jar:?]
at 
org.apache.hadoop.fs.RawLocalFileSystem.create(RawLocalFileSystem.java:326) 
~[hadoop-common-2.7.1.2.3.0.0-2557.jar:?]
at 
org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSOutputSummer.(ChecksumFileSystem.java:393)
 ~[hadoop-common-2.7.1.2.3.0.0-2557.jar:?]
at 
org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:456) 
~[hadoop-common-2.7.1.2.3.0.0-2557.jar:?]
at 
org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:435) 
~[hadoop-common-2.7.1.2.3.0.0-2557.jar:?]
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:909) 
~[hadoop-common-2.7.1.2.3.0.0-2557.jar:?]
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:890) 
~[hadoop-common-2.7.1.2.3.0.0-2557.jar:?]
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:787) 
~[hadoop-common-2.7.1.2.3.0.0-2557.jar:?]
at 
org.apache.hadoop.hive.common.metrics.metrics2.CodahaleMetrics$JsonFileReporter$1.run(CodahaleMetrics.java:433)
 ~[hive-common-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
{noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-15737) Get rid of HiveSubQRemoveRelBuilder and replace it's usage with RelBuilder

2017-01-26 Thread Vineet Garg (JIRA) 
Vineet Garg created HIVE-15737:
--

 Summary: Get rid of HiveSubQRemoveRelBuilder and replace it's 
usage with RelBuilder
 Key: HIVE-15737
 URL: https://issues.apache.org/jira/browse/HIVE-15737
 Project: Hive
  Issue Type: Task
  Components: Query Planning
Reporter: Vineet Garg
Assignee: Vineet Garg


{{HiveSubQRemoveRelBuilder}} was replicated on Hive side to get fix for 
CALCITE-1493. Since now this is fixed and released in Calcite 1.11 
{{HiveSubQRemoveRelBuilder}} could be removed.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 55994: HIVE-15736: Add unit tests to Utilities.getInputSummary() method for multi-threading cases

2017-01-26 Thread Sergio Pena 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/55994/
---

(Updated Jan. 26, 2017, 8:09 p.m.)


Review request for hive, Mohit Sabharwal, Sahil Takiar, and Vihang Karajgaonkar.


Bugs: HIVE-15736
https://issues.apache.org/jira/browse/HIVE-15736


Repository: hive-git


Description (updated)
---

Added unit tests on TestUtilities to validate 
- Single and multiple threads 
- InputEstimator usage
- ContentSummaryInputFormat usage.

It also fixed an issue with the InputEstimator scenario where the values 
returned by the InputEstimator where overriden later by the correct filesystem 
calls.

An interesting thing (code commented while it is on review) is that when 
executing the InputEstimator code path the line commented seems are not needed. 
It might
be that the idea was to set some configurations to the jobConf, but the jobConf 
was never passed as parameter to the estimate method. Please help me verify 
this.


Diffs
-

  ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 
68dd5e7247415dec1e353010ea34481c4f2fc6cd 
  ql/src/test/org/apache/hadoop/hive/ql/exec/InputEstimatorTestClass.java 
PRE-CREATION 
  ql/src/test/org/apache/hadoop/hive/ql/exec/TestUtilities.java 
e444946e990d9adb90ce24837cfe4edcf5126d3a 

Diff: https://reviews.apache.org/r/55994/diff/


Testing
---

Waiting for tests HiveQA


Thanks,

Sergio Pena

Review Request 55994: HIVE-15736: Add unit tests to Utilities.getInputSummary() method for multi-threading cases

2017-01-26 Thread Sergio Pena 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/55994/
---

Review request for hive, Mohit Sabharwal, Sahil Takiar, and Vihang Karajgaonkar.


Bugs: HIVE-15736
https://issues.apache.org/jira/browse/HIVE-15736


Repository: hive-git


Description
---

Added unit tests on TestUtilities to validate 
- Single and multiple threads 
- InputEstimator usage
- ContentSummaryInputFormat usage.

It also fixed an issue with the InputEstimator scenario where the values 
returned by the InputEstimator where overriden later by the correct filesystem 
calls.


Diffs
-

  ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 
68dd5e7247415dec1e353010ea34481c4f2fc6cd 
  ql/src/test/org/apache/hadoop/hive/ql/exec/InputEstimatorTestClass.java 
PRE-CREATION 
  ql/src/test/org/apache/hadoop/hive/ql/exec/TestUtilities.java 
e444946e990d9adb90ce24837cfe4edcf5126d3a 

Diff: https://reviews.apache.org/r/55994/diff/


Testing
---

Waiting for tests HiveQA


Thanks,

Sergio Pena

[jira] [Created] (HIVE-15736) Add unit tests to Utilities.getInputSummary() method for multi-threading cases

2017-01-26 Thread JIRA 
Sergio Peña created HIVE-15736:
--

 Summary: Add unit tests to Utilities.getInputSummary() method for 
multi-threading cases
 Key: HIVE-15736
 URL: https://issues.apache.org/jira/browse/HIVE-15736
 Project: Hive
  Issue Type: Test
  Components: Query Planning
Reporter: Sergio Peña
Assignee: Sergio Peña
Priority: Minor


The {{Utilities.getInputSummary}} method has a configuration to use multiple 
threads to get the content summary of tables and partitions. This configuration 
variable, {{mapred.dfsclient.parallelism.max}}, is disabled by default and 
there are no tests that validate the quality of using multi threads.

This JIRA is used to add tests to such method with multiple threads and fix any 
issue found.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-15735) In some cases, view objects inside a view do not have parents

2017-01-26 Thread Yongzhi Chen (JIRA) 
Yongzhi Chen created HIVE-15735:
---

 Summary: In some cases, view objects inside a view do not have 
parents
 Key: HIVE-15735
 URL: https://issues.apache.org/jira/browse/HIVE-15735
 Project: Hive
  Issue Type: Bug
Reporter: Yongzhi Chen
Assignee: Yongzhi Chen


This cause Sentry throws "No valid privileges" error:
Error: Error while compiling statement: FAILED: SemanticException No valid 
privileges.
To reproduce:
Enable sentry:
create table t1( i int);
create view v1 as select * from t1;
create view v2 as select * from v1 union all select * from v1;
If the user does not have read permission on t1 and v1, the query
select * from v2;  
This will fail with:
Error: Error while compiling statement: FAILED: SemanticException No valid 
privileges
 User foo does not have privileges for QUERY
 The required privileges: 
Server=server1->Db=database2->Table=v1->action=select; (state=42000,code=4)
Sentry should not check v1's permission, for v1 has at least one parent(v2).




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 55816: HIVE-15680: Incorrect results when hive.optimize.index.filter=true and same ORC table is referenced twice in query

2017-01-26 Thread Anthony Hsu 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/55816/
---

(Updated Jan. 26, 2017, 5:42 p.m.)


Review request for hive.


Changes
---

Added some missing null checks.


Bugs: HIVE-15680
https://issues.apache.org/jira/browse/HIVE-15680


Repository: hive-git


Description
---

HIVE-15680: Incorrect results when hive.optimize.index.filter=true and same ORC 
table is referenced twice in query


Diffs (updated)
-

  ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 
1cf24b41c047b9bc43e42a2940ff54a3e331190c 
  ql/src/java/org/apache/hadoop/hive/ql/io/HiveInputFormat.java 
3ee8fdc24aa115710d2c42f5c44c7f28e0544589 
  ql/src/test/queries/clientpositive/orc_ppd_same_table_multiple_aliases.q 
PRE-CREATION 
  ql/src/test/results/clientpositive/orc_ppd_same_table_multiple_aliases.q.out 
PRE-CREATION 

Diff: https://reviews.apache.org/r/55816/diff/


Testing
---

Added qtest.


Thanks,

Anthony Hsu

Re: Review Request 55846: HIVE-15604: Improve contents of qfile test output

2017-01-26 Thread Barna Zsombor Klara 


> On Jan. 24, 2017, 10:36 a.m., Peter Vary wrote:
> > itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java, lines 
> > 1819-1821
> > 
> >
> > Why do we change the error aggregation here?

sortFiles will now throw an exception if the result of the operation was 
non-zero. This way we can fail faster and won't perform additional operations 
(like the diff) which will probably fail anyway.


- Barna Zsombor


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/55846/#review162795
---


On Jan. 25, 2017, 1:12 p.m., Barna Zsombor Klara wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/55846/
> ---
> 
> (Updated Jan. 25, 2017, 1:12 p.m.)
> 
> 
> Review request for hive.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-15604: Improve contents of qfile test output
> 
> 
> Diffs
> -
> 
>   
> itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/TestLocationQueries.java
>  06888467ec70eb1780e11969dafe2bff2c432d9d 
>   
> itests/util/src/main/java/org/apache/hadoop/hive/cli/control/AbstractCoreBlobstoreCliDriver.java
>  9c97c318ee89f598515d7b2e14f7f8184dc1119d 
>   
> itests/util/src/main/java/org/apache/hadoop/hive/cli/control/CoreAccumuloCliDriver.java
>  3e4b3732cf49c1ac14baa58b65731f407f2b1dcd 
>   
> itests/util/src/main/java/org/apache/hadoop/hive/cli/control/CoreCliDriver.java
>  a735346c062684eb6069565e7354b875cde44bd7 
>   
> itests/util/src/main/java/org/apache/hadoop/hive/cli/control/CoreCompareCliDriver.java
>  71a02bca0b416c264beb00e95d2c3ab8da30b5c1 
>   
> itests/util/src/main/java/org/apache/hadoop/hive/cli/control/CoreHBaseCliDriver.java
>  956a42d2711296061aa09f79596885af475ce6e6 
>   
> itests/util/src/main/java/org/apache/hadoop/hive/cli/control/CoreHBaseNegativeCliDriver.java
>  6225180a0735e41f5eee2c32e5b05c674e2edbdd 
>   
> itests/util/src/main/java/org/apache/hadoop/hive/cli/control/CoreNegativeCliDriver.java
>  65b2ce71d3797823c6c14002384857cb2a61749a 
>   
> itests/util/src/main/java/org/apache/hadoop/hive/cli/control/CorePerfCliDriver.java
>  8620cde37894aa5f15a4f7850c6f62c9ac8bf6d1 
>   
> itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestProcessExecResult.java
>  PRE-CREATION 
>   itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java 
> 1eaf7af5381a164f442a1094fb2543a08dc6c253 
>   
> itests/util/src/main/java/org/apache/hadoop/hive/ql/parse/CoreParseNegative.java
>  8dba0bb9d5288240414a59f9468d3adae4dd8ef1 
> 
> Diff: https://reviews.apache.org/r/55846/diff/
> 
> 
> Testing
> ---
> 
> Ran qtests manually and checked the error messages.
> Checked that for diff failures we now have the output captured from the diff 
> process in the error message of the failed test.
> Checkout how multiple tests, all with failures, look on the console to make 
> sure that the results are still readeable/easy to follow.
> 
> 
> Thanks,
> 
> Barna Zsombor Klara
> 
>

[jira] [Created] (HIVE-15734) LazySimpleDeserializeRead.readField needs to catch IllegalArgumentException

2017-01-26 Thread Matt McCline (JIRA) 
Matt McCline created HIVE-15734:
---

 Summary: LazySimpleDeserializeRead.readField needs to catch 
IllegalArgumentException
 Key: HIVE-15734
 URL: https://issues.apache.org/jira/browse/HIVE-15734
 Project: Hive
  Issue Type: Bug
  Components: Hive
Reporter: Matt McCline
Assignee: Matt McCline
Priority: Critical
 Fix For: 2.2.0


E.g. java.sql.Date.valueOf can throw that exception if it encounters a parse 
error for a date.

With changes to CHAR padding, I think this may be why 
schema_evol_text_vec_part.q is failing.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 55898: HIVE-15698 Vectorization support for min/max/bloomfilter runtime filtering

2017-01-26 Thread Matt McCline 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/55898/#review163114
---




ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorizationContext.java 
(line 2706)


I'm glad we are porting HIVE-13713.



ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/DynamicValueVectorExpression.java
 (line 76)


One pattern we have added is also setting isNull to false on the value path.



ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/DynamicValueVectorExpression.java
 (line 146)


So, boolean is not applicable for dynamic values?



ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/VectorInBloomFilterColDynamicValue.java
 (line 270)


With the FastDecimal you can get the String from the HiveDecimalWritable 
directly now.  I.e. the DecimalColumnVector.vector.  So, you do not have to 
call getHiveDecimal and get better performance.

An additional performance option is available, too.  You can call a 
variation of toString that passes a scratch byte[] that makes to String 
conversion even faster...



ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/aggregates/VectorUDAFBloomFilter.java
 (line 85)


Nit: whitespace.



ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/aggregates/VectorUDAFBloomFilter.java
 (line 147)


So implicitly !noNulls implies all NULL.



ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/aggregates/VectorUDAFBloomFilter.java
 (line 367)


As Sergey would say: is it?



ql/src/test/results/clientpositive/llap/vectorized_dynamic_semijoin_reduction.q.out
 (line 53)


I'm glad it shows up so nicely in EXPLAIN.


- Matt McCline


On Jan. 25, 2017, 12:03 a.m., Jason Dere wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/55898/
> ---
> 
> (Updated Jan. 25, 2017, 12:03 a.m.)
> 
> 
> Review request for hive, Deepak Jaiswal and Matt McCline.
> 
> 
> Bugs: HIVE-15698
> https://issues.apache.org/jira/browse/HIVE-15698
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Adds vectorized support for ExprNodeDynamicValue, BETWEEN() with 
> DynamicValue, bloom_filter() aggregation function, and in_bloom_filter()
> 
> 
> Diffs
> -
> 
>   ant/src/org/apache/hadoop/hive/ant/GenVectorCode.java e9fe8fa 
>   
> ql/src/gen/vectorization/ExpressionTemplates/FilterColumnBetweenDynamicValue.txt
>  PRE-CREATION 
>   ql/src/gen/vectorization/ExpressionTemplates/FilterDecimalColumnBetween.txt 
> d68edfa 
>   ql/src/gen/vectorization/ExpressionTemplates/FilterStringColumnBetween.txt 
> e8049da 
>   
> ql/src/gen/vectorization/ExpressionTemplates/FilterTimestampColumnBetween.txt 
> 4298d79 
>   
> ql/src/gen/vectorization/ExpressionTemplates/FilterTruncStringColumnBetween.txt
>  94a174d 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/DynamicValueRegistryTez.java 
> 7bbedf6 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorExpressionDescriptor.java
>  217af3f 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorFilterOperator.java 
> 261246b 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorSelectOperator.java 
> f7fec8f 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorizationContext.java 
> c887757 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/DynamicValueVectorExpression.java
>  PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/VectorExpression.java
>  8fca8a1 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/VectorInBloomFilterColDynamicValue.java
>  PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/aggregates/VectorUDAFBloomFilter.java
>  PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/aggregates/VectorUDAFBloomFilterMerge.java
>  PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/Vectorizer.java 
> e3d9d7f 
>   
> ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFInBloomFilter.java
>  1b7de6c 
>   
> ql/src/test/org/apache/hadoop/hive/ql/optimizer/physical/TestVectorizer.java 
> 59cb31e 
>   ql/src/test/queries/clientpositive/vectorized_dynamic_semijoin_reduction.q 
> PRE-CREATION 
>   
>

[jira] [Created] (HIVE-15733) LLAP AMHeartbeat threads should be restricted per app

2017-01-26 Thread Siddharth Seth (JIRA) 
Siddharth Seth created HIVE-15733:
-

 Summary: LLAP AMHeartbeat threads should be restricted per app
 Key: HIVE-15733
 URL: https://issues.apache.org/jira/browse/HIVE-15733
 Project: Hive
  Issue Type: Bug
Reporter: Siddharth Seth


Don't allow all threads to get stuck trying to heartbeat to a bad AM. That 
would adversely affect every other query.

cc [~rajesh.balamohan], [~prasanth_j]



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)