date:20171207

Re: Review Request 64326: HIVE-18208

2017-12-07 Thread Deepak Jaiswal



> On Dec. 8, 2017, 12:59 a.m., Jason Dere wrote:
> > ql/src/test/queries/clientpositive/auto_sortmerge_join_1.q
> > Lines 29 (patched)
> > 
> >
> > Please add comment stating that the point of this setting is to disable 
> > hashjoins from being chosen in the query plan. As well as in the other 
> > qfiles.

Adding comment on all the tests. Thanks


> On Dec. 8, 2017, 12:59 a.m., Jason Dere wrote:
> > ql/src/test/queries/clientpositive/bucketsortoptimize_insert_2.q
> > Lines 36 (patched)
> > 
> >
> > Is this config needed for all of the bucketsort tests?

Thanks for the catch. I added it there for testing some other stuff. Will take 
it out.


> On Dec. 8, 2017, 12:59 a.m., Jason Dere wrote:
> > ql/src/test/results/clientpositive/llap/auto_sortmerge_join_11.q.out
> > Line 113 (original), 113 (patched)
> > 
> >
> > Looks like MapJoin is replaced by a reduce-side mergejoin, is this 
> > correct?

Fix coming in updated patch. Applies to all such queries in the test file.


> On Dec. 8, 2017, 12:59 a.m., Jason Dere wrote:
> > ql/src/test/results/clientpositive/llap/auto_sortmerge_join_16.q.out
> > Lines 263 (patched)
> > 
> >
> > Is this one supposed to be a smb join? The mergejoin is in the Reducer.

Found the issue earlier today, updated the results.


> On Dec. 8, 2017, 12:59 a.m., Jason Dere wrote:
> > ql/src/test/results/clientpositive/llap/auto_sortmerge_join_9.q.out
> > Line 1204 (original), 1154 (patched)
> > 
> >
> > Looks like MapJoin is replaced by a reduce-side mergejoin, is this 
> > correct?

Yes. Since the noconditionaltasksize virtually disables hash joins, the chance 
of getting mapside join other than SMB is low.
For this query, SMB does not apply, thus it falls back to reduce side join.


> On Dec. 8, 2017, 12:59 a.m., Jason Dere wrote:
> > ql/src/test/results/clientpositive/llap/smb_cache.q.out
> > Line 206 (original), 212 (patched)
> > 
> >
> > Are these supposed to be smb joins? Looks like reducer-side mergejoin 
> > here.

Thanks for pointing out. I must have missed it. Looking into it.


- Deepak


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64326/#review192957
---


On Dec. 5, 2017, 4:43 a.m., Deepak Jaiswal wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/64326/
> ---
> 
> (Updated Dec. 5, 2017, 4:43 a.m.)
> 
> 
> Review request for hive and Jason Dere.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> SMB Join : Fix the unit tests to run SMB Joins.
> Updated tests and result files.
> 
> 
> Diffs
> -
> 
>   ql/src/test/queries/clientpositive/auto_sortmerge_join_1.q a1d5249448 
>   ql/src/test/queries/clientpositive/auto_sortmerge_join_10.q e65344dd6d 
>   ql/src/test/queries/clientpositive/auto_sortmerge_join_11.q 11499f8eab 
>   ql/src/test/queries/clientpositive/auto_sortmerge_join_12.q b512cc5c74 
>   ql/src/test/queries/clientpositive/auto_sortmerge_join_13.q 1c868dcd15 
>   ql/src/test/queries/clientpositive/auto_sortmerge_join_14.q dd59c74fc0 
>   ql/src/test/queries/clientpositive/auto_sortmerge_join_15.q 1480b15488 
>   ql/src/test/queries/clientpositive/auto_sortmerge_join_16.q 12ab1fa1d1 
>   ql/src/test/queries/clientpositive/auto_sortmerge_join_2.q e77d937991 
>   ql/src/test/queries/clientpositive/auto_sortmerge_join_3.q 183f03335a 
>   ql/src/test/queries/clientpositive/auto_sortmerge_join_4.q 21f273a17b 
>   ql/src/test/queries/clientpositive/auto_sortmerge_join_7.q cf12331e13 
>   ql/src/test/queries/clientpositive/auto_sortmerge_join_8.q 5ec4e26d4b 
>   ql/src/test/queries/clientpositive/auto_sortmerge_join_9.q f95631f429 
>   ql/src/test/queries/clientpositive/bucketsortoptimize_insert_2.q 4a14587857 
>   ql/src/test/queries/clientpositive/bucketsortoptimize_insert_6.q ec0c2dc254 
>   ql/src/test/queries/clientpositive/bucketsortoptimize_insert_7.q 45635c1209 
>   ql/src/test/queries/clientpositive/quotedid_smb.q 25d1f0eee7 
>   ql/src/test/queries/clientpositive/smb_cache.q e415e51053 
>   ql/src/test/results/clientpositive/auto_sortmerge_join_10.q.out 22ac2a201a 
>   ql/src/test/results/clientpositive/auto_sortmerge_join_11.q.out 243a49b45f 
>   ql/src/test/results/clientpositive/auto_sortmerge_join_12.q.out 3d0559a47c 
>

[jira] [Created] (HIVE-18252) Limit the size of the object inspector caches

2017-12-07 Thread Jason Dere (JIRA)

Jason Dere created HIVE-18252:
-

 Summary: Limit the size of the object inspector caches
 Key: HIVE-18252
 URL: https://issues.apache.org/jira/browse/HIVE-18252
 Project: Hive
  Issue Type: Bug
  Components: Types
Reporter: Jason Dere
Assignee: Jason Dere


Was running some tests that had a lot of queries with constant values, and 
noticed that ObjectInspectorFactory.cachedStandardStructObjectInspector started 
using up a lot of memory.

It appears that StructObjectInspector caching does not work properly with 
constant values. Constant ObjectInspectors are not cached, so each constant 
expression creates a new constant ObjectInspector. And since object inspectors 
do not override equals(), object inspector comparison relies on object instance 
comparison. So even if the values are exactly the same as what is already in 
the cache, the StructObjectInspector cache lookup would fail, and Hive would 
create a new object inspector and add it to the cache, creating another entry 
that would never be used. Plus, there is no max cache size - it's just a map 
that is allowed to grow as long as values keep getting added to it.

Some possible solutions I can think of:
1. Limit the size of the object inspector caches, rather than growing without 
bound.
2. Try to fix the caching to work with constant values. This would require 
implementing equals() on the constant object inspectors (which could be slow in 
nested cases), or else we would have to start caching constant object 
inspectors, which could be expensive in terms of memory usage. Could be used in 
combination with (1). By itself this is not a great solution because this still 
has the unbounded cache growth issue.
3. Disable caching in the case of constant object inspectors since this 
scenario currently doesn't work. This could be used in combination with (1).



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: checkstyle changes

2017-12-07 Thread Rui Li

I also believe 140 is a little too long.

BTW do we use 2 or 4 chars for continuation indent? I personally prefer 4,
but I do find both cases in out code.

On Fri, Dec 8, 2017 at 6:20 AM, Alexander Kolbasov 
wrote:

> Problem with 140-wide code isn't just laptops - in many cases we need to do
> side-by-side diffs (e.g. for code reviews) and this doubles the required
> size.
>
> - Alex.
>
> On Thu, Dec 7, 2017 at 1:38 PM, Sergey Shelukhin 
> wrote:
>
> > I think the 140-character change will make the code hard to use on a
> > laptop without a monitor.
> >
> >
> > On 17/12/7, 02:43, "Peter Vary"  wrote:
> >
> > >Disclaimer: I did not have time to test it out, but according to
> > >http://checkstyle.sourceforge.net/config_misc.html#Indentation
> > >
> > >Maybe the indentation could be solved by:
> > >lineWrappingIndentation=2 (default 4)
> > >forceStrictCondition=false (default false)
> > >
> > >http://checkstyle.sourceforge.net/config_misc.html#TrailingComment
> > >
> > >might help with the comments
> > >
> > >Sorry for not being more helpful. Maybe sometime later I will have time
> > >to check these out.
> > >
> > >Thanks,
> > >Peter
> > >
> > >> On Dec 7, 2017, at 10:26 AM, Zoltan Haindrich
> > >> wrote:
> > >>
> > >> Hello Eugene!
> > >>
> > >> I've looked into doing something with these; but I was not able to
> > >>relieve the warnings you've mentioned:
> > >>
> > >> * the ;// is seems to be not configurable
> > >>   It seems like its handled by the whitespaceafter module; I'm not
> sure
> > >>how to allow / after ;
> > >> * I think that indentation of 4 for many method arguments makes it
> more
> > >>readable; so I think it would be the best to just drop this check...but
> > >>I've not seen any way to do this(w/o disabling the whole indentation
> > >>module...)
> > >>
> > >> maybe someone else should take a look at itI find it pretty hard
> to
> > >>get docs about specific chechkstyle configurations; since the search
> > >>keywords mostly contain keywords like: semicolon, whitespace,
> > >>comment...which tends to pull in all kind of garbage results :)
> > >>
> > >> cheers,
> > >> Zoltan
> > >>
> > >> On 6 Dec 2017 8:53 p.m., Eugene Koifman 
> > >>wrote:
> > >> It currently complains about no space between ; and // as in
> “…);//foo”
> > >>
> > >> And also about indentation when a single method call is split into
> > >>multiple lines.
> > >> It insists on 4 chars in this case, though we use 2 in (all?) other
> > >>cases.
> > >>
> > >> Could this be dialed down as well?
> > >>
> > >>
> > >> On 12/5/17, 7:26 AM, "Peter Vary"  wrote:
> > >>
> > >>+1 for the changes
> > >>
> > >>> On Dec 5, 2017, at 1:02 PM, Zoltan Haindrich  wrote:
> > >>>
> > >>> Hello,
> > >>>
> > >>> I've filed a ticket to make the checkstyle warnings less noisy
> > >>>(https://issues.apache.org/jira/browse/HIVE-18222)
> > >>>
> > >>> * set maxlinelength to 140
> > >>>   I think everyone is working with big-enough displays to handle this
> > >>>:)
> > >>>   There are many methods which have complicated names / arguments /
> > >>>etc ; breaking the lines more frequently hurts readability...
> > >>> * disabled some restrictions like: declaration via get/set
> > >>>methods for protected/package fields are not mandatory
> > >>>
> > >>> If you don't feel comfortable with these changes, please share your
> > >>>point of view.
> > >>>
> > >>> cheers,
> > >>> Zoltan
> > >>>
> > >>>
> > >>
> > >>
> > >>
> > >>
> > >
> >
> >
>



-- 
Best regards!
Rui Li

[jira] [Created] (HIVE-18251) Loosen restriction for some checks

2017-12-07 Thread Ashutosh Chauhan (JIRA)

Ashutosh Chauhan created HIVE-18251:
---

 Summary: Loosen restriction for some checks
 Key: HIVE-18251
 URL: https://issues.apache.org/jira/browse/HIVE-18251
 Project: Hive
  Issue Type: Improvement
  Components: Query Planning
Reporter: Ashutosh Chauhan
Assignee: Ashutosh Chauhan






--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: Review Request 64326: HIVE-18208

2017-12-07 Thread Jason Dere


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64326/#review192957
---




ql/src/test/queries/clientpositive/auto_sortmerge_join_1.q
Lines 29 (patched)


Please add comment stating that the point of this setting is to disable 
hashjoins from being chosen in the query plan. As well as in the other qfiles.



ql/src/test/queries/clientpositive/bucketsortoptimize_insert_2.q
Lines 36 (patched)


Is this config needed for all of the bucketsort tests?



ql/src/test/results/clientpositive/llap/auto_sortmerge_join_11.q.out
Line 113 (original), 113 (patched)


Looks like MapJoin is replaced by a reduce-side mergejoin, is this correct?



ql/src/test/results/clientpositive/llap/auto_sortmerge_join_11.q.out
Line 403 (original), 412 (patched)


Looks like MapJoin is replaced by a reduce-side mergejoin, is this correct?



ql/src/test/results/clientpositive/llap/auto_sortmerge_join_11.q.out
Line 693 (original), 711 (patched)


Looks like MapJoin is replaced by a reduce-side mergejoin, is this correct?



ql/src/test/results/clientpositive/llap/auto_sortmerge_join_11.q.out
Line 975 (original), 1002 (patched)


Looks like MapJoin is replaced by a reduce-side mergejoin, is this correct?



ql/src/test/results/clientpositive/llap/auto_sortmerge_join_16.q.out
Lines 263 (patched)


Is this one supposed to be a smb join? The mergejoin is in the Reducer.



ql/src/test/results/clientpositive/llap/auto_sortmerge_join_9.q.out
Line 1204 (original), 1154 (patched)


Looks like MapJoin is replaced by a reduce-side mergejoin, is this correct?



ql/src/test/results/clientpositive/llap/smb_cache.q.out
Line 206 (original), 212 (patched)


Are these supposed to be smb joins? Looks like reducer-side mergejoin here.


- Jason Dere


On Dec. 5, 2017, 4:43 a.m., Deepak Jaiswal wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/64326/
> ---
> 
> (Updated Dec. 5, 2017, 4:43 a.m.)
> 
> 
> Review request for hive and Jason Dere.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> SMB Join : Fix the unit tests to run SMB Joins.
> Updated tests and result files.
> 
> 
> Diffs
> -
> 
>   ql/src/test/queries/clientpositive/auto_sortmerge_join_1.q a1d5249448 
>   ql/src/test/queries/clientpositive/auto_sortmerge_join_10.q e65344dd6d 
>   ql/src/test/queries/clientpositive/auto_sortmerge_join_11.q 11499f8eab 
>   ql/src/test/queries/clientpositive/auto_sortmerge_join_12.q b512cc5c74 
>   ql/src/test/queries/clientpositive/auto_sortmerge_join_13.q 1c868dcd15 
>   ql/src/test/queries/clientpositive/auto_sortmerge_join_14.q dd59c74fc0 
>   ql/src/test/queries/clientpositive/auto_sortmerge_join_15.q 1480b15488 
>   ql/src/test/queries/clientpositive/auto_sortmerge_join_16.q 12ab1fa1d1 
>   ql/src/test/queries/clientpositive/auto_sortmerge_join_2.q e77d937991 
>   ql/src/test/queries/clientpositive/auto_sortmerge_join_3.q 183f03335a 
>   ql/src/test/queries/clientpositive/auto_sortmerge_join_4.q 21f273a17b 
>   ql/src/test/queries/clientpositive/auto_sortmerge_join_7.q cf12331e13 
>   ql/src/test/queries/clientpositive/auto_sortmerge_join_8.q 5ec4e26d4b 
>   ql/src/test/queries/clientpositive/auto_sortmerge_join_9.q f95631f429 
>   ql/src/test/queries/clientpositive/bucketsortoptimize_insert_2.q 4a14587857 
>   ql/src/test/queries/clientpositive/bucketsortoptimize_insert_6.q ec0c2dc254 
>   ql/src/test/queries/clientpositive/bucketsortoptimize_insert_7.q 45635c1209 
>   ql/src/test/queries/clientpositive/quotedid_smb.q 25d1f0eee7 
>   ql/src/test/queries/clientpositive/smb_cache.q e415e51053 
>   ql/src/test/results/clientpositive/auto_sortmerge_join_10.q.out 22ac2a201a 
>   ql/src/test/results/clientpositive/auto_sortmerge_join_11.q.out 243a49b45f 
>   ql/src/test/results/clientpositive/auto_sortmerge_join_12.q.out 3d0559a47c 
>   ql/src/test/results/clientpositive/llap/auto_sortmerge_join_1.q.out 
> 36bfac3f4c 
>   ql/src/test/results/clientpositive/llap/auto_sortmerge_join_10.q.out 
> b8f10fec67 
>   ql/src/test/results/clientpositive/llap/auto_sortmerge_join_11.q.out 
> 37d97d2252 
>   ql/src/test/results/clientpositive/llap/auto_sortmerge_join_12.q.out 
> 655573650b 
>

Review Request 64433: HIVE-18203 change the way WM is enabled and allow dropping the last resource plan

2017-12-07 Thread Sergey Shelukhin


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64433/
---

Review request for hive, Harish Jaiprakash and Prasanth_J.


Repository: hive-git


Description
---

see jira


Diffs
-

  
itests/hcatalog-unit/src/test/java/org/apache/hive/hcatalog/listener/DummyRawStoreFailEvent.java
 62c9172ef5 
  ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 55ef8de9a5 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionPoolManager.java 
8417ebb7d5 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/WorkloadManager.java 
dbdbbf25db 
  ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java 50bdce89a4 
  ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java 
a09b7961c2 
  ql/src/java/org/apache/hadoop/hive/ql/parse/HiveLexer.g 1dcfe9dd28 
  ql/src/java/org/apache/hadoop/hive/ql/parse/ResourcePlanParser.g 95c87259c5 
  ql/src/java/org/apache/hadoop/hive/ql/plan/AlterResourcePlanDesc.java 
d70d52054d 
  ql/src/test/org/apache/hadoop/hive/ql/exec/tez/TestWorkloadManager.java 
c58e4507f2 
  ql/src/test/queries/clientpositive/resourceplan.q 002b21c1b9 
  ql/src/test/results/clientpositive/llap/resourceplan.q.out 093e5d58b6 
  service/src/java/org/apache/hive/service/server/HiveServer2.java 223be6aa32 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
 6e0da5781e 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java
 fc254c6f53 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/IMetaStoreClient.java
 573ac0173d 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
 2e80c9d3b1 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/RawStore.java
 75fbfa23d2 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java
 da518ab6e3 
  standalone-metastore/src/main/thrift/hive_metastore.thrift 1085ce566a 
  
standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/DummyRawStoreControlledCommit.java
 24c59f2f1b 
  
standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/DummyRawStoreForJdoConnection.java
 1e4fe5d973 


Diff: https://reviews.apache.org/r/64433/diff/1/


Testing
---


Thanks,

Sergey Shelukhin

Re: Review Request 62228: HIVE-17495: CachedStore: prewarm improvements, refactoring and caching some aggregate stats

2017-12-07 Thread Vaibhav Gumashta


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62228/
---

(Updated Dec. 8, 2017, 12:06 a.m.)


Review request for hive, Ashutosh Chauhan, Daniel Dai, and Thejas Nair.


Changes
---

Rebased on master


Bugs: HIVE-17495
https://issues.apache.org/jira/browse/HIVE-17495


Repository: hive-git


Description
---

https://issues.apache.org/jira/browse/HIVE-17495


Diffs (updated)
-

  
itests/hcatalog-unit/src/test/java/org/apache/hive/hcatalog/listener/DummyRawStoreFailEvent.java
 62c9172ef5 
  
itests/hive-unit/src/test/java/org/apache/hadoop/hive/metastore/TestHiveMetaStore.java
 f344c47443 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java
 14653b4043 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
 2e80c9d3b1 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/RawStore.java
 75fbfa23d2 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/cache/CacheUtils.java
 ab6b90fb6b 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java
 da518ab6e3 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/cache/SharedCache.java
 b606779709 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/columnstats/aggr/BinaryColumnStatsAggregator.java
 45d5d8c984 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/columnstats/aggr/BooleanColumnStatsAggregator.java
 8aac0fe33d 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/columnstats/aggr/ColumnStatsAggregator.java
 cd0392d6c0 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/columnstats/aggr/DateColumnStatsAggregator.java
 7f2956152c 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/columnstats/aggr/DecimalColumnStatsAggregator.java
 05c0280262 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/columnstats/aggr/DoubleColumnStatsAggregator.java
 faf22dcd7c 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/columnstats/aggr/LongColumnStatsAggregator.java
 d12cdc08ea 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/columnstats/aggr/StringColumnStatsAggregator.java
 4539e6b026 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/utils/MetaStoreUtils.java
 cde34bcf42 
  
standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/DummyRawStoreControlledCommit.java
 24c59f2f1b 
  
standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/DummyRawStoreForJdoConnection.java
 1e4fe5d973 


Diff: https://reviews.apache.org/r/62228/diff/5/

Changes: https://reviews.apache.org/r/62228/diff/4-5/


Testing
---


Thanks,

Vaibhav Gumashta

[jira] [Created] (HIVE-18249) Remove Thrift dependency on fb303

2017-12-07 Thread Alexander Kolbasov (JIRA)

Alexander Kolbasov created HIVE-18249:
-

 Summary: Remove Thrift dependency on fb303
 Key: HIVE-18249
 URL: https://issues.apache.org/jira/browse/HIVE-18249
 Project: Hive
  Issue Type: Bug
  Components: Hive, Metastore
Affects Versions: 3.0.0
Reporter: Alexander Kolbasov
Assignee: Alexander Kolbasov


Looks like we are not really using fb303 and can remove fb303 dependency.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: Review Request 64402: HIVE-18240 support getClientInfo/setClientInfo in JDBC

2017-12-07 Thread Vaibhav Gumashta


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64402/#review193193
---


Ship it!




Ship It!

- Vaibhav Gumashta


On Dec. 7, 2017, 3:24 a.m., Sergey Shelukhin wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/64402/
> ---
> 
> (Updated Dec. 7, 2017, 3:24 a.m.)
> 
> 
> Review request for hive and Thejas Nair.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> see jira
> 
> 
> Diffs
> -
> 
>   itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestJdbcDriver2.java 
> 87595ee415 
>   jdbc/src/java/org/apache/hive/jdbc/HiveConnection.java edf93859fe 
>   jdbc/src/java/org/apache/hive/jdbc/HiveDatabaseMetaData.java 7f21bd3842 
>   service-rpc/if/TCLIService.thrift a1f293bdc2 
>   service/src/java/org/apache/hive/service/cli/session/HiveSessionImpl.java 
> 7fbcd13b63 
>   service/src/java/org/apache/hive/service/cli/thrift/ThriftCLIService.java 
> fc9e6b2a91 
> 
> 
> Diff: https://reviews.apache.org/r/64402/diff/1/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Sergey Shelukhin
> 
>

[jira] [Created] (HIVE-18250) CBO gets turned off with duplicates in RR error

2017-12-07 Thread Ashutosh Chauhan (JIRA)

Ashutosh Chauhan created HIVE-18250:
---

 Summary: CBO gets turned off with duplicates in RR error
 Key: HIVE-18250
 URL: https://issues.apache.org/jira/browse/HIVE-18250
 Project: Hive
  Issue Type: Bug
  Components: CBO, Query Planning
Affects Versions: 2.3.0, 2.2.0, 2.1.0, 2.0.0
Reporter: Ashutosh Chauhan


{code}
 create table t1 (a int);
explain select t1.a as a1, min(t1.a) as a from t1 group by t1.a;
{code}

CBO gets turned off with:
{code}
WARN [2e80e34e-dc46-49cf-88bf-2c24c0262d41 main] parse.RowResolver: Found 
duplicate column alias in RR: null.a => {null, a1, _col0: int} adding null.a => 
{null, null, _col1: int}
2017-12-07T15:27:47,651 ERROR [2e80e34e-dc46-49cf-88bf-2c24c0262d41 main] 
parse.CalcitePlanner: CBO failed, skipping CBO.
org.apache.hadoop.hive.ql.optimizer.calcite.CalciteSemanticException: Cannot 
add column to RR: null.a => _col1: int due to duplication, see previous warnings
at 
org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.genSelectLogicalPlan(CalcitePlanner.java:3985)
 ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
at 
org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.genLogicalPlan(CalcitePlanner.java:4313)
 ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
at 
org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.apply(CalcitePlanner.java:1392)
 ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
at 
org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.apply(CalcitePlanner.java:1322)
 ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
{code}
After that non-CBO path completes the query.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: [DISCUSS] Do-it-yourself docs

2017-12-07 Thread Lefty Leverenz

We have three goals with just a few tools of persuasion.

Goals:

   1. Identify JIRA issues that need documentation.
   2. Add doc notes or release notes to JIRA issues.
   3. Document Hive code and procedures in the wiki.  *(main goal)*

I've been helping with #1 and #2, adding ~17 TODOC labels & doc notes a
month, but soon I'll stop that and focus on #3.

Tools:

   - TODOC labels
   - Yetus checks by Hive QA (?)
   - JIRA subtasks (?)
   - Education:  mailing list, wiki (How to Contribute
   , How
   to Commit
   

   )
   - Nudging:  JIRA comments

-- Lefty


On Mon, Dec 4, 2017 at 10:59 AM, Eugene Koifman 
wrote:

> Perhaps this should be a 2 stage process.  One to approve the code and one
> to approve the doc.
> It seems odd to update the Wiki (which isn’t tracked using the same Git
> repo as the code) before
> the code changes have been agreed to.  Both approvals would be required to
> commit.
>
> Eugene
>
>
> On 12/3/17, 2:49 PM, "Prasanth Jayachandran" 
> wrote:
>
> +1 for Yetus integration to -1 patches without docs.
>
>
> Thanks and Regards,
> Prasanth Jayachandran
>
>
> On Sat, Dec 2, 2017 at 3:04 AM, Klára Barna Zsombor <
> zsomb...@gmail.com>
> wrote:
>
> > Could this be somehow integrated into the Yetus checks? I'm thinking
> that
> > if the Jira being tested does not have one of the "Doc-Performed",
> > "To-Doc", "Doc-Not-Needed" labels then it would get a -1 from Yetus.
> > Peter what do you think? Is Yetus extendable in this way?
> >
> > On Thu, Nov 30, 2017 at 2:58 AM, Lefty Leverenz <
> leftylever...@gmail.com>
> > wrote:
> >
> > > Hive contributors are responsible for documenting their own
> commits,
> > > although many seem to be unaware of this or too busy with other
> tasks.
> > How
> > > can we boost the number of jiras that get documented?
> > >
> > >
> > > Our current process is to put a TODOC** label on each
> committed
> > > issue that needs wiki documentation, then remove it when the doc
> is done.
> > > But nobody tallies the TODOC labels at release time or pressures
> > > contributors to do their documentation, so we have a large backlog
> of
> > > unfinished doc tasks.
> > >
> > >
> > > For several years I've monitored the dev@hive mailing list for
> issues
> > that
> > > should be documented in the wiki.  Whenever a committed patch
> needs doc
> > and
> > > the contributor hasn't taken care of it, I add a TODOC label and
> write a
> > > doc note naming new configuration parameters, reserved words, or
> HiveQL
> > > syntax.  (This is convenient for searches.)  I also give links to
> places
> > in
> > > the wiki where the docs belong.
> > >
> > >
> > > Soon, I'll stop monitoring the Hive mailing lists and writing doc
> notes.
> > > My time can be better spent doing documentation, instead of just
> pointing
> > > out that it needs to be done.  But I can't tackle the whole
> backlog, and
> > > many future commits won't even get a TODOC label.
> > >
> > >
> > > What can we do to improve the Hive doc process?
> > >
> > > -- Lefty
> > >
> >
>
>
>

[jira] [Created] (HIVE-18248) Clean up parameters

2017-12-07 Thread Janaki Lahorani (JIRA)

Janaki Lahorani created HIVE-18248:
--

 Summary: Clean up parameters
 Key: HIVE-18248
 URL: https://issues.apache.org/jira/browse/HIVE-18248
 Project: Hive
  Issue Type: Bug
Reporter: Janaki Lahorani
Assignee: Janaki Lahorani


Clean up of parameters that need not change at run time.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (HIVE-18247) Use DB auto-increment for indexes

2017-12-07 Thread Alexander Kolbasov (JIRA)

Alexander Kolbasov created HIVE-18247:
-

 Summary: Use DB auto-increment for indexes
 Key: HIVE-18247
 URL: https://issues.apache.org/jira/browse/HIVE-18247
 Project: Hive
  Issue Type: Bug
  Components: Hive, Metastore
Affects Versions: 3.0.0
Reporter: Alexander Kolbasov


I initially noticed this problem in Apache Sentry - see SENTRY-1960. Hive has 
the same issue. DataNucleus uses SEQUENCE table to allocate IDs which requires 
raw locks on multiple tables during transactions and this creates scalability 
problems. 

Instead DN should rely on DB auto-increment mechanisms which are much more 
scalable.

See SENTRY-1960 for extra details.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (HIVE-18246) Replace toString with getExprString in AbstractOperatorDesc::getColumnExprMapForExplain

2017-12-07 Thread Vineet Garg (JIRA)

Vineet Garg created HIVE-18246:
--

 Summary: Replace toString with getExprString in 
AbstractOperatorDesc::getColumnExprMapForExplain
 Key: HIVE-18246
 URL: https://issues.apache.org/jira/browse/HIVE-18246
 Project: Hive
  Issue Type: Improvement
Reporter: Vineet Garg
Assignee: Vineet Garg


AbstractOperatorDesc::getColumnExprMapForExplain uses toString on ExprNode to 
get the string representation of an expr. getExprString is better suited here 
since each ExprNode class has suitable implementation for this.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (HIVE-18245) clean up

2017-12-07 Thread Eugene Koifman (JIRA)

Eugene Koifman created HIVE-18245:
-

 Summary: clean up 
 Key: HIVE-18245
 URL: https://issues.apache.org/jira/browse/HIVE-18245
 Project: Hive
  Issue Type: Sub-task
  Components: Transactions
Reporter: Eugene Koifman
Assignee: Eugene Koifman


now that HIVE-17923 is fixed, 
acid_vectorization_original_tez.q/acid_vectorization_original.q can be cleaned 
up



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: checkstyle changes

2017-12-07 Thread Alexander Kolbasov

Problem with 140-wide code isn't just laptops - in many cases we need to do
side-by-side diffs (e.g. for code reviews) and this doubles the required
size.

- Alex.

On Thu, Dec 7, 2017 at 1:38 PM, Sergey Shelukhin 
wrote:

> I think the 140-character change will make the code hard to use on a
> laptop without a monitor.
>
>
> On 17/12/7, 02:43, "Peter Vary"  wrote:
>
> >Disclaimer: I did not have time to test it out, but according to
> >http://checkstyle.sourceforge.net/config_misc.html#Indentation
> >
> >Maybe the indentation could be solved by:
> >lineWrappingIndentation=2 (default 4)
> >forceStrictCondition=false (default false)
> >
> >http://checkstyle.sourceforge.net/config_misc.html#TrailingComment
> >
> >might help with the comments
> >
> >Sorry for not being more helpful. Maybe sometime later I will have time
> >to check these out.
> >
> >Thanks,
> >Peter
> >
> >> On Dec 7, 2017, at 10:26 AM, Zoltan Haindrich
> >> wrote:
> >>
> >> Hello Eugene!
> >>
> >> I've looked into doing something with these; but I was not able to
> >>relieve the warnings you've mentioned:
> >>
> >> * the ;// is seems to be not configurable
> >>   It seems like its handled by the whitespaceafter module; I'm not sure
> >>how to allow / after ;
> >> * I think that indentation of 4 for many method arguments makes it more
> >>readable; so I think it would be the best to just drop this check...but
> >>I've not seen any way to do this(w/o disabling the whole indentation
> >>module...)
> >>
> >> maybe someone else should take a look at itI find it pretty hard to
> >>get docs about specific chechkstyle configurations; since the search
> >>keywords mostly contain keywords like: semicolon, whitespace,
> >>comment...which tends to pull in all kind of garbage results :)
> >>
> >> cheers,
> >> Zoltan
> >>
> >> On 6 Dec 2017 8:53 p.m., Eugene Koifman 
> >>wrote:
> >> It currently complains about no space between ; and // as in “…);//foo”
> >>
> >> And also about indentation when a single method call is split into
> >>multiple lines.
> >> It insists on 4 chars in this case, though we use 2 in (all?) other
> >>cases.
> >>
> >> Could this be dialed down as well?
> >>
> >>
> >> On 12/5/17, 7:26 AM, "Peter Vary"  wrote:
> >>
> >>+1 for the changes
> >>
> >>> On Dec 5, 2017, at 1:02 PM, Zoltan Haindrich  wrote:
> >>>
> >>> Hello,
> >>>
> >>> I've filed a ticket to make the checkstyle warnings less noisy
> >>>(https://issues.apache.org/jira/browse/HIVE-18222)
> >>>
> >>> * set maxlinelength to 140
> >>>   I think everyone is working with big-enough displays to handle this
> >>>:)
> >>>   There are many methods which have complicated names / arguments /
> >>>etc ; breaking the lines more frequently hurts readability...
> >>> * disabled some restrictions like: declaration via get/set
> >>>methods for protected/package fields are not mandatory
> >>>
> >>> If you don't feel comfortable with these changes, please share your
> >>>point of view.
> >>>
> >>> cheers,
> >>> Zoltan
> >>>
> >>>
> >>
> >>
> >>
> >>
> >
>
>

Re: checkstyle changes

2017-12-07 Thread Sergey Shelukhin

I think the 140-character change will make the code hard to use on a
laptop without a monitor.


On 17/12/7, 02:43, "Peter Vary"  wrote:

>Disclaimer: I did not have time to test it out, but according to
>http://checkstyle.sourceforge.net/config_misc.html#Indentation
>
>Maybe the indentation could be solved by:
>lineWrappingIndentation=2 (default 4)
>forceStrictCondition=false (default false)
>
>http://checkstyle.sourceforge.net/config_misc.html#TrailingComment
>
>might help with the comments
>
>Sorry for not being more helpful. Maybe sometime later I will have time
>to check these out.
>
>Thanks,
>Peter
>
>> On Dec 7, 2017, at 10:26 AM, Zoltan Haindrich
>> wrote:
>> 
>> Hello Eugene!
>> 
>> I've looked into doing something with these; but I was not able to
>>relieve the warnings you've mentioned:
>> 
>> * the ;// is seems to be not configurable
>>   It seems like its handled by the whitespaceafter module; I'm not sure
>>how to allow / after ;
>> * I think that indentation of 4 for many method arguments makes it more
>>readable; so I think it would be the best to just drop this check...but
>>I've not seen any way to do this(w/o disabling the whole indentation
>>module...)
>> 
>> maybe someone else should take a look at itI find it pretty hard to
>>get docs about specific chechkstyle configurations; since the search
>>keywords mostly contain keywords like: semicolon, whitespace,
>>comment...which tends to pull in all kind of garbage results :)
>> 
>> cheers,
>> Zoltan
>> 
>> On 6 Dec 2017 8:53 p.m., Eugene Koifman 
>>wrote:
>> It currently complains about no space between ; and // as in “…);//foo”
>> 
>> And also about indentation when a single method call is split into
>>multiple lines.
>> It insists on 4 chars in this case, though we use 2 in (all?) other
>>cases.
>> 
>> Could this be dialed down as well?
>> 
>> 
>> On 12/5/17, 7:26 AM, "Peter Vary"  wrote:
>> 
>>+1 for the changes
>> 
>>> On Dec 5, 2017, at 1:02 PM, Zoltan Haindrich  wrote:
>>> 
>>> Hello,
>>> 
>>> I've filed a ticket to make the checkstyle warnings less noisy
>>>(https://issues.apache.org/jira/browse/HIVE-18222)
>>> 
>>> * set maxlinelength to 140
>>>   I think everyone is working with big-enough displays to handle this
>>>:)
>>>   There are many methods which have complicated names / arguments /
>>>etc ; breaking the lines more frequently hurts readability...
>>> * disabled some restrictions like: declaration via get/set
>>>methods for protected/package fields are not mandatory
>>> 
>>> If you don't feel comfortable with these changes, please share your
>>>point of view.
>>> 
>>> cheers,
>>> Zoltan
>>> 
>>> 
>> 
>> 
>> 
>> 
>

Re: Review Request 64193: HIVE-18054: Make Lineage work with concurrent queries on a Session

2017-12-07 Thread Sahil Takiar


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64193/#review193155
---




ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMRProcContext.java
Line 201 (original), 208 (patched)


Can we use `parseContext.getQueryState()`?



ql/src/java/org/apache/hadoop/hive/ql/parse/spark/GenSparkProcContext.java
Lines 164 (patched)


Think u can just use `parseContext.getQueryState()`



ql/src/java/org/apache/hadoop/hive/ql/parse/spark/SparkCompiler.java
Lines 351 (patched)


I think `pCtx#getQueryState` could be used for all the changes to the 
`TaskCompiler`s?


- Sahil Takiar


On Dec. 7, 2017, 7:12 p.m., Andrew Sherman wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/64193/
> ---
> 
> (Updated Dec. 7, 2017, 7:12 p.m.)
> 
> 
> Review request for hive.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> A Hive Session can contain multiple concurrent sql Operations.
> Lineage is currently tracked in SessionState and is cleared when a query
> completes. This results in Lineage for other running queries being lost.
> 
> To fix this, move LineageState from SessionState to QueryState.
> In MoveTask/MoveWork use the LineageState from the MoveTask's QueryState
> rather than trying to use it from MoveWork.
> Add a test which runs multiple jdbc queries in a thread pool
> against the same connection and show that Vertices are not lost from Lineage.
> As part of this test, add ReadableHook, an ExecuteWithHookContext that stores
> HookContexts in memory and makes them available for reading.
> Make LineageLogger methods static so they can be used elsewhere.
> 
> Sometimes a running query (originating in a Driver) will instantiate
> another Driver to run or compile another query. Because these Drivers
> shared a Session, the child Driver would accumulate Lineage information
> along with that of the parent Driver. For consistency a LineageState is
> passed to these child Drivers and stored in the new Driver's QueryState.
> 
> 
> Diffs
> -
> 
>   itests/hive-unit/src/test/java/org/apache/hive/jdbc/ReadableHook.java 
> PRE-CREATION 
>   
> itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestJdbcWithMiniHS2.java 
> 70bd29c5178456c683652cf2377206059b735514 
>   ql/src/java/org/apache/hadoop/hive/ql/Driver.java 
> d3df015288fe1963d2b548e32db53cfc2310af21 
>   ql/src/java/org/apache/hadoop/hive/ql/QueryState.java 
> f3a46dbcaf151706521c735654f377a2f2f76a81 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 
> 55ef8de9a5c7144931d0a6ff13224765ee737fea 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/MoveTask.java 
> f5a5e713bb0e081591a53a30caf56f97750c3f8e 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/Task.java 
> 1f0487f4f72ab18bcf876f45ad5758d83a7f001b 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/repl/bootstrap/load/table/LoadPartitions.java
>  262225fc202d4627652acfd77350e44b0284b3da 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/repl/bootstrap/load/table/LoadTable.java
>  545b7a8b7e9f1370b767fc777cb10fa59bd81917 
>   ql/src/java/org/apache/hadoop/hive/ql/hooks/HookContext.java 
> 7b617309f6b0d8a7ce0dea80ab1f790c2651b147 
>   ql/src/java/org/apache/hadoop/hive/ql/hooks/LineageLogger.java 
> 2f764f8a29a9d41a7db013a949ffe3a8a9417d32 
>   ql/src/java/org/apache/hadoop/hive/ql/index/AggregateIndexHandler.java 
> 68709b4d3baf15d78e60e948ccdef3df84f28cec 
>   ql/src/java/org/apache/hadoop/hive/ql/index/HiveIndexHandler.java 
> 1e577da82343a1b7361467fb662661f9c6642ec0 
>   ql/src/java/org/apache/hadoop/hive/ql/index/TableBasedIndexHandler.java 
> 29886ae7f97f8dae7116f4fc9a2417ab8f9dac0a 
>   ql/src/java/org/apache/hadoop/hive/ql/index/bitmap/BitmapIndexHandler.java 
> 7b067a0d45e33bc3347c43b050af933c296a9227 
>   
> ql/src/java/org/apache/hadoop/hive/ql/index/compact/CompactIndexHandler.java 
> 504b0623142a6fa6cdb45a26b49f146e12ec2d7a 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMRFileSink1.java 
> d7a83f775abca39b219f71aff88173a14ffaee9f 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMRProcContext.java 
> 4387c4297fee48d4c03e95d5a2fcb822ab480eeb 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java 
> bdaf105697fd2c2074885fa3a35548043167c7e7 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/IndexUtils.java 
> 338c1856672f09bb7da35d2336ebb5b6f3fdc5a6 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/lineage/Generator.java 
> e6c07713b24df719315d804f006151106eea9aed 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java 
>

Re: Review Request 64193: HIVE-18054: Make Lineage work with concurrent queries on a Session

2017-12-07 Thread Andrew Sherman via Review Board


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64193/
---

(Updated Dec. 7, 2017, 7:12 p.m.)


Review request for hive.


Repository: hive-git


Description
---

A Hive Session can contain multiple concurrent sql Operations.
Lineage is currently tracked in SessionState and is cleared when a query
completes. This results in Lineage for other running queries being lost.

To fix this, move LineageState from SessionState to QueryState.
In MoveTask/MoveWork use the LineageState from the MoveTask's QueryState
rather than trying to use it from MoveWork.
Add a test which runs multiple jdbc queries in a thread pool
against the same connection and show that Vertices are not lost from Lineage.
As part of this test, add ReadableHook, an ExecuteWithHookContext that stores
HookContexts in memory and makes them available for reading.
Make LineageLogger methods static so they can be used elsewhere.

Sometimes a running query (originating in a Driver) will instantiate
another Driver to run or compile another query. Because these Drivers
shared a Session, the child Driver would accumulate Lineage information
along with that of the parent Driver. For consistency a LineageState is
passed to these child Drivers and stored in the new Driver's QueryState.


Diffs (updated)
-

  itests/hive-unit/src/test/java/org/apache/hive/jdbc/ReadableHook.java 
PRE-CREATION 
  itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestJdbcWithMiniHS2.java 
70bd29c5178456c683652cf2377206059b735514 
  ql/src/java/org/apache/hadoop/hive/ql/Driver.java 
d3df015288fe1963d2b548e32db53cfc2310af21 
  ql/src/java/org/apache/hadoop/hive/ql/QueryState.java 
f3a46dbcaf151706521c735654f377a2f2f76a81 
  ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 
55ef8de9a5c7144931d0a6ff13224765ee737fea 
  ql/src/java/org/apache/hadoop/hive/ql/exec/MoveTask.java 
f5a5e713bb0e081591a53a30caf56f97750c3f8e 
  ql/src/java/org/apache/hadoop/hive/ql/exec/Task.java 
1f0487f4f72ab18bcf876f45ad5758d83a7f001b 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/repl/bootstrap/load/table/LoadPartitions.java
 262225fc202d4627652acfd77350e44b0284b3da 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/repl/bootstrap/load/table/LoadTable.java
 545b7a8b7e9f1370b767fc777cb10fa59bd81917 
  ql/src/java/org/apache/hadoop/hive/ql/hooks/HookContext.java 
7b617309f6b0d8a7ce0dea80ab1f790c2651b147 
  ql/src/java/org/apache/hadoop/hive/ql/hooks/LineageLogger.java 
2f764f8a29a9d41a7db013a949ffe3a8a9417d32 
  ql/src/java/org/apache/hadoop/hive/ql/index/AggregateIndexHandler.java 
68709b4d3baf15d78e60e948ccdef3df84f28cec 
  ql/src/java/org/apache/hadoop/hive/ql/index/HiveIndexHandler.java 
1e577da82343a1b7361467fb662661f9c6642ec0 
  ql/src/java/org/apache/hadoop/hive/ql/index/TableBasedIndexHandler.java 
29886ae7f97f8dae7116f4fc9a2417ab8f9dac0a 
  ql/src/java/org/apache/hadoop/hive/ql/index/bitmap/BitmapIndexHandler.java 
7b067a0d45e33bc3347c43b050af933c296a9227 
  ql/src/java/org/apache/hadoop/hive/ql/index/compact/CompactIndexHandler.java 
504b0623142a6fa6cdb45a26b49f146e12ec2d7a 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMRFileSink1.java 
d7a83f775abca39b219f71aff88173a14ffaee9f 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMRProcContext.java 
4387c4297fee48d4c03e95d5a2fcb822ab480eeb 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java 
bdaf105697fd2c2074885fa3a35548043167c7e7 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/IndexUtils.java 
338c1856672f09bb7da35d2336ebb5b6f3fdc5a6 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/lineage/Generator.java 
e6c07713b24df719315d804f006151106eea9aed 
  ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java 
a09b7961c2dbc26b4d2fa912d0be7037885f63e4 
  ql/src/java/org/apache/hadoop/hive/ql/parse/ExplainSemanticAnalyzer.java 
065c7e50986872cd35386feee712f3452597d643 
  ql/src/java/org/apache/hadoop/hive/ql/parse/GenTezProcContext.java 
0c160acf46eb1eb07c5f04091099c1024e166638 
  ql/src/java/org/apache/hadoop/hive/ql/parse/GenTezUtils.java 
b6f1139fe1a78283277bf4d0c5224ab1d718c634 
  ql/src/java/org/apache/hadoop/hive/ql/parse/ImportSemanticAnalyzer.java 
83d53bc157f35b4b57fc37bb24b6c400ac58d8ca 
  ql/src/java/org/apache/hadoop/hive/ql/parse/IndexUpdater.java 
f31775ed942160da73344c4dca707da7b8c658a6 
  ql/src/java/org/apache/hadoop/hive/ql/parse/LoadSemanticAnalyzer.java 
cc956da57567114aa29ee0552566ca62c68f6be7 
  ql/src/java/org/apache/hadoop/hive/ql/parse/MapReduceCompiler.java 
d7a56e5846d5754dec5070d8c3543a3695e4 
  ql/src/java/org/apache/hadoop/hive/ql/parse/ReplicationSemanticAnalyzer.java 
498b6741c3f40b92ce3fb218e91e7809a17383f0 
  ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 
28e3621d3264f4f704da0d775b396f7b7764fdb6 
  ql/src/java/org/apache/hadoop/hive/ql/parse/TaskCompiler.java

Dataworks Summit EU call for abstracts

2017-12-07 Thread Alan Gates

Dataworks Summit EU 2018 is in Berlin, April 16-19.   The call for
abstracts is open through December 15th.  One of the tracks is
Datawarehousing and Operational Data Store, which is a great fit for Hive
talks. https://dataworkssummit.com/abstracts/


Alan.

[jira] [Created] (HIVE-18244) CachedStore: Fix UT when CachedStore is enabled

2017-12-07 Thread Vaibhav Gumashta (JIRA)

Vaibhav Gumashta created HIVE-18244:
---

 Summary: CachedStore: Fix UT when CachedStore is enabled
 Key: HIVE-18244
 URL: https://issues.apache.org/jira/browse/HIVE-18244
 Project: Hive
  Issue Type: Bug
Reporter: Vaibhav Gumashta
Assignee: Vaibhav Gumashta






--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] hive pull request #274: HIVE-17981 Create a set of builders for Thrift class...

2017-12-07 Thread alanfgates

GitHub user alanfgates opened a pull request:

https://github.com/apache/hive/pull/274

HIVE-17981 Create a set of builders for Thrift classes



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/alanfgates/hive hive17981

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/hive/pull/274.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #274


commit 5360d684f12444bbaa846c80bdf972ca1748e875
Author: Alan Gates 
Date:   2017-09-12T20:18:09Z

HIVE-17981 Create a set of builders for Thrift classes




---

[jira] [Created] (HIVE-18243) Cartesian error for joins defined in where clause

2017-12-07 Thread Zoltan Haindrich (JIRA)

Zoltan Haindrich created HIVE-18243:
---

 Summary: Cartesian error for joins defined in where clause
 Key: HIVE-18243
 URL: https://issues.apache.org/jira/browse/HIVE-18243
 Project: Hive
  Issue Type: Bug
Reporter: Zoltan Haindrich


this issue was hidden; because of HIVE-18238

{code}
create table agg_01 (amount int, dim_shops_id int);
create table dim_shops (id int);

EXPLAIN SELECT agg.amount
FROM agg_01 agg,
dim_shops d1
WHERE agg.dim_shops_id = d1.id
and agg.dim_shops_id = 1;
{code}

emits a cartesian product error
{code}
2017-12-07T04:48:20,612 ERROR [c7a4797b-2635-4e28-9e0b-af2e4d26f2bc main] 
ql.Driver: FAILED: SemanticException Cartesian products are disabled for safety 
reasons. If you know what you are doing, please 
sethive.strict.checks.cartesian.product to false and that hive.mapred.mode is 
not set to 'strict' to proceed. Note that if you may get errors or incorrect 
results if you make a mistake while using some of the unsafe features.
{code}


from the plan: 
https://github.com/apache/hive/blob/7ddd915bf82a68c8ab73b0c4ca409f1a6d43d227/ql/src/test/results/clientpositive/llap/dynamic_partition_pruning_2.q.out#L591

It doesn't seem to be that a cartesian join being happening...possibly the 
check is overreacting...



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: checkstyle changes

2017-12-07 Thread Peter Vary

Disclaimer: I did not have time to test it out, but according to 
http://checkstyle.sourceforge.net/config_misc.html#Indentation 

Maybe the indentation could be solved by:
lineWrappingIndentation=2 (default 4)
forceStrictCondition=false (default false)

http://checkstyle.sourceforge.net/config_misc.html#TrailingComment 
 might help 
with the comments

Sorry for not being more helpful. Maybe sometime later I will have time to 
check these out.

Thanks,
Peter

> On Dec 7, 2017, at 10:26 AM, Zoltan Haindrich  
> wrote:
> 
> Hello Eugene!
> 
> I've looked into doing something with these; but I was not able to relieve 
> the warnings you've mentioned:
> 
> * the ;// is seems to be not configurable
>   It seems like its handled by the whitespaceafter module; I'm not sure how 
> to allow / after ;
> * I think that indentation of 4 for many method arguments makes it more 
> readable; so I think it would be the best to just drop this check...but I've 
> not seen any way to do this(w/o disabling the whole indentation module...)
> 
> maybe someone else should take a look at itI find it pretty hard to get 
> docs about specific chechkstyle configurations; since the search keywords 
> mostly contain keywords like: semicolon, whitespace, comment...which tends to 
> pull in all kind of garbage results :)
> 
> cheers,
> Zoltan
> 
> On 6 Dec 2017 8:53 p.m., Eugene Koifman  wrote:
> It currently complains about no space between ; and // as in “…);//foo”
> 
> And also about indentation when a single method call is split into multiple 
> lines.
> It insists on 4 chars in this case, though we use 2 in (all?) other cases.
> 
> Could this be dialed down as well?
> 
> 
> On 12/5/17, 7:26 AM, "Peter Vary"  wrote:
> 
>+1 for the changes
> 
>> On Dec 5, 2017, at 1:02 PM, Zoltan Haindrich  wrote:
>> 
>> Hello,
>> 
>> I've filed a ticket to make the checkstyle warnings less noisy 
>> (https://issues.apache.org/jira/browse/HIVE-18222)
>> 
>> * set maxlinelength to 140
>>   I think everyone is working with big-enough displays to handle this :)
>>   There are many methods which have complicated names / arguments / etc ; 
>> breaking the lines more frequently hurts readability...
>> * disabled some restrictions like: declaration via get/set methods 
>> for protected/package fields are not mandatory
>> 
>> If you don't feel comfortable with these changes, please share your point of 
>> view.
>> 
>> cheers,
>> Zoltan
>> 
>> 
> 
> 
> 
>

Re: checkstyle changes

2017-12-07 Thread Zoltan Haindrich

Hello Eugene!

I've looked into doing something with these; but I was not able to relieve the 
warnings you've mentioned:

* the ;// is seems to be not configurable
   It seems like its handled by the whitespaceafter module; I'm not sure how to 
allow / after ;
* I think that indentation of 4 for many method arguments makes it more 
readable; so I think it would be the best to just drop this check...but I've 
not seen any way to do this(w/o disabling the whole indentation module...)

maybe someone else should take a look at itI find it pretty hard to get 
docs about specific chechkstyle configurations; since the search keywords 
mostly contain keywords like: semicolon, whitespace, comment...which tends to 
pull in all kind of garbage results :)

cheers,
Zoltan

On 6 Dec 2017 8:53 p.m., Eugene Koifman  wrote:
It currently complains about no space between ; and // as in “…);//foo”

And also about indentation when a single method call is split into multiple 
lines.
It insists on 4 chars in this case, though we use 2 in (all?) other cases.

Could this be dialed down as well?


On 12/5/17, 7:26 AM, "Peter Vary"  wrote:

+1 for the changes

> On Dec 5, 2017, at 1:02 PM, Zoltan Haindrich  wrote:
>
> Hello,
>
> I've filed a ticket to make the checkstyle warnings less noisy 
(https://issues.apache.org/jira/browse/HIVE-18222)
>
> * set maxlinelength to 140
>I think everyone is working with big-enough displays to handle this :)
>There are many methods which have complicated names / arguments / etc 
; breaking the lines more frequently hurts readability...
> * disabled some restrictions like: declaration via get/set methods 
for protected/package fields are not mandatory
>
> If you don't feel comfortable with these changes, please share your point 
of view.
>
> cheers,
> Zoltan
>
>

ptests are stuck

2017-12-07 Thread Zoltan Haindrich


Hello,

It's stuck again; but the current state is very interesting:

* current build is testing HIVE-18237
  https://issues.apache.org/jira/browse/HIVE-18237
* it's currently running for >5 hours
* hiveqa have posted ptest results for HIVE-18237 around 4 hours ago!
* from the jenkins build console it's currently executing batch #134 
(134-TestSparkCliDriver)

https://builds.apache.org/job/PreCommit-HIVE-Build/8139/consoleFull
* but at the corresponding ptest site:
http://104.198.109.242/logs/PreCommit-HIVE-Build-8139/succeeded/
  not that batch 134 have been finished; but all of them! (299?)
* I've searched for batch "177-TestMiniSparkOnYarnCliD" in the console 
output

https://builds.apache.org/job/PreCommit-HIVE-Build/8139/consoleFull
  and it only appears in the "generation" phase.
* taking a closer look at:
http://104.198.109.242/logs/PreCommit-HIVE-Build-8139/
  * according to the directory creation dates:
    all these directories are seem to be newer than the creation date 
of http://104.198.109.242/logs/PreCommit-HIVE-Build-8139/ + 2 hours;
    the dir http://104.198.109.242/logs/PreCommit-HIVE-Build-8139/ is 
lesser than 2 hours later created than 
http://104.198.109.242/logs/PreCommit-HIVE-Build-8138/

  * the patch matches with the one submitted in HIVE-18237
* closer look
  * 122-TestSpark
    the "completed executing" appears in 
http://104.198.109.242/logs/PreCommit-HIVE-Build-8139/execution.txt; but 
not in the actual build output console

  * 134-TestSpark
    the completed executing appears in both logs; with exactly the same 
timestamp


From the above: I suspect that somehow there are 2 "builds" executing 
the same set of tests is happening somehow

but anyway...could someone please take a closer look?

cheers,
Zoltan

Re: Review Request 64358: HIVE-18003 add explicit jdbc connection string args for mappings

2017-12-07 Thread j . prasanth . j


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64358/#review193098
---




ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezTask.java
Lines 167 (patched)


nit: use local variable userName?



ql/src/java/org/apache/hadoop/hive/ql/exec/tez/UserPoolMapping.java
Lines 133 (patched)


nit: can be combined in if condition



ql/src/java/org/apache/hadoop/hive/ql/exec/tez/UserPoolMapping.java
Lines 150 (patched)


nit: non-static


- Prasanth_J


On Dec. 6, 2017, 8:03 p.m., Sergey Shelukhin wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/64358/
> ---
> 
> (Updated Dec. 6, 2017, 8:03 p.m.)
> 
> 
> Review request for hive and Prasanth_J.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> see jira
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 75b77072c6 
>   jdbc/src/java/org/apache/hive/jdbc/HiveConnection.java edf93859fe 
>   jdbc/src/java/org/apache/hive/jdbc/Utils.java 855de881e9 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezTask.java 8795cfcee1 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/UserPoolMapping.java 
> 33ee8f791f 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/WorkloadManager.java 
> dbdbbf25db 
>   ql/src/test/org/apache/hadoop/hive/ql/exec/tez/TestWorkloadManager.java 
> c58e4507f2 
> 
> 
> Diff: https://reviews.apache.org/r/64358/diff/3/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Sergey Shelukhin
> 
>

Re: Review Request 64326: HIVE-18208

[jira] [Created] (HIVE-18252) Limit the size of the object inspector caches

Re: checkstyle changes

[jira] [Created] (HIVE-18251) Loosen restriction for some checks

Re: Review Request 64326: HIVE-18208

Review Request 64433: HIVE-18203 change the way WM is enabled and allow dropping the last resource plan

Re: Review Request 62228: HIVE-17495: CachedStore: prewarm improvements, refactoring and caching some aggregate stats

[jira] [Created] (HIVE-18249) Remove Thrift dependency on fb303

Re: Review Request 64402: HIVE-18240 support getClientInfo/setClientInfo in JDBC

[jira] [Created] (HIVE-18250) CBO gets turned off with duplicates in RR error

Re: [DISCUSS] Do-it-yourself docs

[jira] [Created] (HIVE-18248) Clean up parameters

[jira] [Created] (HIVE-18247) Use DB auto-increment for indexes

[jira] [Created] (HIVE-18246) Replace toString with getExprString in AbstractOperatorDesc::getColumnExprMapForExplain

[jira] [Created] (HIVE-18245) clean up

Re: checkstyle changes

Re: checkstyle changes

Re: Review Request 64193: HIVE-18054: Make Lineage work with concurrent queries on a Session

Re: Review Request 64193: HIVE-18054: Make Lineage work with concurrent queries on a Session

Dataworks Summit EU call for abstracts

[jira] [Created] (HIVE-18244) CachedStore: Fix UT when CachedStore is enabled

[GitHub] hive pull request #274: HIVE-17981 Create a set of builders for Thrift class...

[jira] [Created] (HIVE-18243) Cartesian error for joins defined in where clause

Re: checkstyle changes

Re: checkstyle changes

ptests are stuck

Re: Review Request 64358: HIVE-18003 add explicit jdbc connection string args for mappings

27 matches

Site Navigation

Mail list logo

Footer information