Re: Review Request 67966: HIVE-20202: Add profiler endpoint to httpserver

[Gopal V]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67966/#review206233
---


Ship it!





common/src/java/org/apache/hive/http/ProfileServlet.java
Lines 211 (patched)


final boolean variables for "thread", "simple" and "reverse"


- Gopal V


On July 19, 2018, 1:08 a.m., Prasanth_J wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/67966/
> ---
> 
> (Updated July 19, 2018, 1:08 a.m.)
> 
> 
> Review request for hive and Gopal V.
> 
> 
> Bugs: HIVE-20202
> https://issues.apache.org/jira/browse/HIVE-20202
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-20202: Add profiler endpoint to httpserver
> 
> 
> Diffs
> -
> 
>   bin/hive 1ade51e 
>   common/src/java/org/apache/hive/common/util/ProcessUtils.java PRE-CREATION 
>   common/src/java/org/apache/hive/http/HttpServer.java 3cb7a33 
>   common/src/java/org/apache/hive/http/ProfileServlet.java PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/67966/diff/3/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Prasanth_J
> 
>

[jira] [Created] (HIVE-20208) order by position alias do not work

[xulongfetion (JIRA)]

xulongfetion created HIVE-20208:
---

 Summary: order by position alias do not work
 Key: HIVE-20208
 URL: https://issues.apache.org/jira/browse/HIVE-20208
 Project: Hive
  Issue Type: Bug
  Components: Hive
Reporter: xulongfetion
 Attachments: order_by_position_alias.patch

the order by position alias do not work if there's more than one position 
alias. For query like

"select a, b, c from table1 order by 1, 2;", hive will report column not found 
error.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[VOTE] Apache Hive 3.1.0 Release Candidate 0

[Vineet Garg]

Apache Hive 3.1.0 Release Candidate 0 is available here:

http://people.apache.org/~vgarg/apache-hive-3.1.0-rc-0

Maven artifacts are available here:

https://repository.apache.org/content/repositories/orgapachehive-1089/

Source tag: https://github.com/apache/hive/tree/release-3.1.0-rc0

Voting will conclude in 72 hours.

Hive PMC Members: Please test and vote.

Thanks.

Re: Review Request 67966: HIVE-20202: Add profiler endpoint to httpserver

[j . prasanth . j]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67966/
---

(Updated July 19, 2018, 1:08 a.m.)


Review request for hive and Gopal V.


Changes
---

addressed review comments.


Bugs: HIVE-20202
https://issues.apache.org/jira/browse/HIVE-20202


Repository: hive-git


Description
---

HIVE-20202: Add profiler endpoint to httpserver


Diffs (updated)
-

  bin/hive 1ade51e 
  common/src/java/org/apache/hive/common/util/ProcessUtils.java PRE-CREATION 
  common/src/java/org/apache/hive/http/HttpServer.java 3cb7a33 
  common/src/java/org/apache/hive/http/ProfileServlet.java PRE-CREATION 


Diff: https://reviews.apache.org/r/67966/diff/3/

Changes: https://reviews.apache.org/r/67966/diff/2-3/


Testing
---


Thanks,

Prasanth_J

Review Request 67970: HIVE-20204 Type conversion during IN () comparisons is using different rules from other comparison operations

[Jason Dere]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67970/
---

Review request for hive, Ashutosh Chauhan and Jesús Camacho Rodríguez.


Bugs: HIVE-20204
https://issues.apache.org/jira/browse/HIVE-20204


Repository: hive-git


Description
---

Change GenericUDFIn to use FunctionRegistry.getCommonClassForComparison() to 
match type conversion done during other comparison operations.


Diffs
-

  ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java 0800a10541 
  ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFIn.java 
cf26fce00f 
  ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFUtils.java 
c91865b173 
  ql/src/test/queries/clientpositive/orc_ppd_decimal.q 2134a9f207 
  ql/src/test/queries/clientpositive/parquet_ppd_decimal.q e8e118d541 
  ql/src/test/queries/clientpositive/vectorization_parquet_ppd_decimal.q 
0b0811b055 
  ql/src/test/results/clientpositive/llap/orc_ppd_decimal.q.out 4b535d4480 
  ql/src/test/results/clientpositive/parquet_ppd_decimal.q.out c9a4338dbf 
  ql/src/test/results/clientpositive/vectorization_parquet_ppd_decimal.q.out 
49d7354b60 


Diff: https://reviews.apache.org/r/67970/diff/1/


Testing
---


Thanks,

Jason Dere

Re: Review Request 67954: HIVE-20194: HiveMetastoreClient should use reflection to instantiate embedded HMS instance

[Vihang Karajgaonkar via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67954/#review206230
---




standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java
Lines 111-116 (patched)


Do you think we can get these values from a config instead of hard coding?


- Vihang Karajgaonkar


On July 18, 2018, 12:10 a.m., Alexander Kolbasov wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/67954/
> ---
> 
> (Updated July 18, 2018, 12:10 a.m.)
> 
> 
> Review request for hive, Alan Gates, Peter Vary, Sahil Takiar, and Vihang 
> Karajgaonkar.
> 
> 
> Bugs: HIVE-20194
> https://issues.apache.org/jira/browse/HIVE-20194
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-20194: HiveMetastoreClient should use reflection to instantiate embedded 
> HMS instance
> 
> 
> Diffs
> -
> 
>   
> standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
>  47f819b60a6c7eb2acac9217724e67a976341093 
>   
> standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java
>  92e280570f44cd56fe87c3123001b9263d28186f 
>   
> standalone-metastore/metastore-common/src/test/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClientPreCatalog.java
>  d91f737450efa9de6db3806401072645c96bbb7d 
> 
> 
> Diff: https://reviews.apache.org/r/67954/diff/1/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Alexander Kolbasov
> 
>

[jira] [Created] (HIVE-20207) Vectorization: Fix NULL / Wrong Results issues in Filter / Compare

[Matt McCline (JIRA)]

Matt McCline created HIVE-20207:
---

 Summary: Vectorization: Fix NULL / Wrong Results issues in Filter 
/ Compare
 Key: HIVE-20207
 URL: https://issues.apache.org/jira/browse/HIVE-20207
 Project: Hive
  Issue Type: Bug
  Components: Hive
Reporter: Matt McCline
Assignee: Matt McCline


Write new UT tests that use random data and intentional isRepeating batches to 
checks for NULL and Wrong Results for vectorized filter and compare.

BUGS:

1) LongColLessLongColumn SIMD optimization do not work for very large integers:
 -7272907770454997143 < 8976171455044006767
 outputVector[i] = (vector1[i] - vector2[i]) >>> 63;
 Produces 0 instead of 1...



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Review Request 67969: HIVE-20115 Acid tables should not use footer scan for analyze

[Sergey Shelukhin]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67969/
---

Review request for hive and Eugene Koifman.


Repository: hive-git


Description
---

see jira


Diffs
-

  ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMRTableScan1.java 
64f9c70f05 
  ql/src/java/org/apache/hadoop/hive/ql/parse/ProcessAnalyzeTable.java 
03cceace40 
  ql/src/java/org/apache/hadoop/hive/ql/parse/TaskCompiler.java 49709e596e 
  
ql/src/java/org/apache/hadoop/hive/ql/parse/spark/SparkProcessAnalyzeTable.java 
28d4de7f7b 
  ql/src/test/queries/clientpositive/acid_no_buckets.q bcf9e0634b 
  ql/src/test/results/clientpositive/llap/acid_no_buckets.q.out 36a6a5d5d1 


Diff: https://reviews.apache.org/r/67969/diff/1/


Testing
---


Thanks,

Sergey Shelukhin

Re: Review Request 67966: HIVE-20202: Add profiler endpoint to httpserver

[Gopal V]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67966/#review206218
---




common/src/java/org/apache/hive/http/HttpServer.java
Lines 561 (patched)


Enable this only if the "async.profiler.home" property is setup - so that 
this is not enabled accidentally for anyone.


- Gopal V


On July 18, 2018, 6:42 p.m., Prasanth_J wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/67966/
> ---
> 
> (Updated July 18, 2018, 6:42 p.m.)
> 
> 
> Review request for hive and Gopal V.
> 
> 
> Bugs: HIVE-20202
> https://issues.apache.org/jira/browse/HIVE-20202
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-20202: Add profiler endpoint to httpserver
> 
> 
> Diffs
> -
> 
>   bin/hive 1ade51e 
>   common/src/java/org/apache/hive/common/util/ProcessUtils.java PRE-CREATION 
>   common/src/java/org/apache/hive/http/HttpServer.java 3cb7a33 
>   common/src/java/org/apache/hive/http/ProfileServlet.java PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/67966/diff/2/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Prasanth_J
> 
>

Re: Review Request 67966: HIVE-20202: Add profiler endpoint to httpserver

[Gopal V]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67966/#review206217
---




common/src/java/org/apache/hive/http/ProfileServlet.java
Lines 172 (patched)


Input validation (most of these are typed, as far as I know)



common/src/java/org/apache/hive/http/ProfileServlet.java
Lines 184 (patched)


Change to a tryLock + timeout - fail with an error if the GET request can't 
get a lock after a while.


- Gopal V


On July 18, 2018, 6:42 p.m., Prasanth_J wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/67966/
> ---
> 
> (Updated July 18, 2018, 6:42 p.m.)
> 
> 
> Review request for hive and Gopal V.
> 
> 
> Bugs: HIVE-20202
> https://issues.apache.org/jira/browse/HIVE-20202
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-20202: Add profiler endpoint to httpserver
> 
> 
> Diffs
> -
> 
>   bin/hive 1ade51e 
>   common/src/java/org/apache/hive/common/util/ProcessUtils.java PRE-CREATION 
>   common/src/java/org/apache/hive/http/HttpServer.java 3cb7a33 
>   common/src/java/org/apache/hive/http/ProfileServlet.java PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/67966/diff/2/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Prasanth_J
> 
>

[jira] [Created] (HIVE-20206) Hive Union all query with two views on the same hbase external table producing incorrect results

[Venkata (JIRA)]

Venkata created HIVE-20206:
--

 Summary: Hive Union all query with two views on the same hbase 
external table producing incorrect results
 Key: HIVE-20206
 URL: https://issues.apache.org/jira/browse/HIVE-20206
 Project: Hive
  Issue Type: Bug
  Components: Beeline
Affects Versions: 1.1.0
Reporter: Venkata
 Attachments: hive_view_union_all.txt

We are running this in Cloudera cdh5.13.3 version.

Hive version - 1.1.0-cdh5.13.3

Hbase version - 1.2.0-cdh5.13.3

When running hive union all query with two views created on the same hive 
*external* hbase table, the query is returning incorrect results.

The query is returning correct results if *set hive.optimize.ppd=false;* 

We don't want to turn off the ppd as it will greatly impact performance. 

Note:  The union all is working fine for the views created on the same Hive 
*Managed* table.

 

 

The below are the queries ( i have attached the same queries as an attachment):

*HBASE:*

create '*test*','default'

put 'test','111','default:name','john1'
put 'test','111','default:dept','hr1'
put 'test','111','default:type','a'

put 'test','112','default:name','rambo1'
put 'test','112','default:dept','eng1'
put 'test','112','default:type','a'


put 'test','113','default:name','alex1'
put 'test','113','default:dept','dev1'
put 'test','113','default:type','a'


put 'test','211','default:name','john2'
put 'test','211','default:dept','hr2'
put 'test','211','default:type','b'

put 'test','212','default:name','rambo2'
put 'test','212','default:dept','eng2'
put 'test','212','default:type','b'


put 'test','213','default:name','alex2'
put 'test','213','default:dept','dev2'
put 'test','213','default:type','b'

put 'test','311','default:name','john3'
put 'test','311','default:dept','hr3'
put 'test','311','default:type','c'

put 'test','312','default:name','rambo3'
put 'test','312','default:dept','eng3'
put 'test','312','default:type','c'


put 'test','313','default:name','alex3'
put 'test','313','default:dept','dev3'
put 'test','313','default:type','c'



*BEELINE*

CREATE external TABLE *hbase_ext_table*(id string, name string,dept string,type 
string) STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler' WITH 
SERDEPROPERTIES ("hbase.columns.mapping" = 
":key,default:name,default:dept,default:type") TBLPROPERTIES 
("hbase.table.name" = "*test*");

create view *view1* as select id ,name,dept,type from *hbase_ext_table* where 
id>='1' and id<'2' and type is not null;
create view *view2* as select id,name,dept,type from *hbase_ext_table* where 
id>='2' and id<'3' and type is not null;

> select * from view1;
+-+---+---+---+--+
| view1.id | view1.name | view1.dept | view1.type |
+-+---+---+---+--+
| 111 | john1 | hr1 | a |
| 112 | rambo1 | eng1 | a |
| 113 | alex1 | dev1 | a |
+-+---+---+---+--+

> select * from view2;
+-+---+---+---+--+
| view2.id | view2.name | view2.dept | view2.type |
+-+---+---+---+--+
| 211 | john2 | hr2 | b |
| 212 | rambo2 | eng2 | b |
| 213 | alex2 | dev2 | b |
+-+---+---+---+--+


> select id,name,dept,type from *view1* union all select id,name,dept,type from 
> *view2*;

+-+---+---+---+--+
| _u1.id | _u1.name | _u1.dept | _u1.type |
+-+---+---+---+--+
| 111 | john1 | hr1 | a |
| 111 | john1 | hr1 | a |
| 112 | rambo1 | eng1 | a |
| 112 | rambo1 | eng1 | a |
| 113 | alex1 | dev1 | a |
| 113 | alex1 | dev1 | a |
| 211 | john2 | hr2 | b |
| 211 | john2 | hr2 | b |
| 212 | rambo2 | eng2 | b |
| 212 | rambo2 | eng2 | b |
| 213 | alex2 | dev2 | b |
| 213 | alex2 | dev2 | b |
| {color:#FF}*311 | john3 | hr3 | c |*{color}
{color:#FF}*| 311 | john3 | hr3 | c |*{color}
{color:#FF}*| 312 | rambo3 | eng3 | c |*{color}
{color:#FF}*| 312 | rambo3 | eng3 | c |*{color}
{color:#FF}*| 313 | alex3 | dev3 | c |*{color}
{color:#FF}*| 313 | alex3 | dev3 | c |*{color}
+-+---+---+---+--+



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [VOTE] Should we release storage-api 2.7.0 rc1?

[Deepak Jaiswal]

Thanks for testing out the RC and your vote.
With 3 +1s, the vote passes. I will work on the release now.

Regards,
Deepak

On 7/18/18, 9:14 AM, "Jesus Camacho Rodriguez"  wrote:

+1

Built from sources and ran tests.

-Jesús



On 7/16/18, 10:31 AM, "Ashutosh Chauhan"  wrote:

+1
Built from sources.
Ran unit tests.
Checksums and sigs matched up.

On Mon, Jul 16, 2018 at 8:58 AM Owen O'Malley 
wrote:

> +1
>
> built & ran tests
> checked checksums & signature
> tested with ORC
>
> On Thu, Jul 12, 2018 at 4:37 PM, Deepak Jaiswal 

> wrote:
>
> > Hi,
> >
> > I have prepared the rc1 off of branch-3.1.
> > Artifacts:
> > Tag : https://github.com/apache/hive/releases/tag/storage-
> > release-2.7.0-rc1
> > Tar Ball : http://home.apache.org/~djaiswal/hive-storage-2.7.0/
> >
> > Regards,
> > Deepak
> >
> > On 7/10/18, 10:16 AM, "Deepak Jaiswal" 
> wrote:
> >
> > Thanks Owen for finding this out. I will work on the next RC 
once
> this
> > blocker is resolved.
> >
> > Regards,
> > Deepak
> >
> > On 7/10/18, 9:40 AM, "Owen O'Malley"  
wrote:
> >
> > Ok, Jesus and I tracked it down and I've filed
> > https://issues.apache.org/jira/browse/HIVE-20135 that is a
> > blocker on
> > storage-api 2.7.0.
> >
> > The impact was that orc 1.5 and master failed with the RC. 
orc
> 1.4
> > and
> > older were fine.
> >
> > .. Owen
> >
> > On Tue, Jul 10, 2018 at 8:17 AM, Owen O'Malley <
> > owen.omal...@gmail.com>
> > wrote:
> >
> > > I wanted to give an update on this. For now, I'm -1 
because the
> > ORC
> > > (branch-1.5) tests fail with this RC. I'll dig into what 
is
> > wrong, but it
> > > looks like something in the timezone changes broke 
backwards
> > compatibility.
> > >
> > > .. Owen
> > >
> > > On Mon, Jul 9, 2018 at 11:12 AM, Deepak Jaiswal <
> > djais...@hortonworks.com>
> > > wrote:
> > >
> > >> Thanks Alan.
> > >>
> > >> On 7/9/18, 10:17 AM, "Alan Gates" 
> wrote:
> > >>
> > >> +1.  Did a build with a clean maven repo, checked the
> > signature and
> > >> sha
> > >> hash, ran RAT.
> > >>
> > >> Alan.
> > >>
> > >> On Fri, Jul 6, 2018 at 2:21 PM Deepak Jaiswal <
> > >> djais...@hortonworks.com>
> > >> wrote:
> > >>
> > >> > Hi,
> > >> >
> > >> > I would like to make a new release of the 
storage-api.
> It
> > contains
> > >> changes
> > >> > required for Hive 3.1 release.
> > >> >
> > >> > Artifcats:
> > >> > Tag :
> > >> > 
https://github.com/apache/hive/releases/tag/storage-
> > release-
> > >> 2.7.0-rc0
> > >> > Tar Ball : http://home.apache.org/~
> > djaiswal/hive-storage-2.7.0/
> > >> >
> > >> > Regards,
> > >> > Deepak
> > >> >
> > >>
> > >>
> > >>
> > >
> >
> >
> >
> >
> >
>

[jira] [Created] (HIVE-20205) Upgrade HBase dependencies off alpha4 release

[Naveen Gangam (JIRA)]

Naveen Gangam created HIVE-20205:


 Summary: Upgrade HBase dependencies off alpha4 release
 Key: HIVE-20205
 URL: https://issues.apache.org/jira/browse/HIVE-20205
 Project: Hive
  Issue Type: Bug
  Components: Hive
Affects Versions: 3.0.0
Reporter: Naveen Gangam
Assignee: Naveen Gangam


Appears Hive has dependencies on hbase 2.0.0-alpha4 releases. HBase 2.0.0 and 
2.0.1 have been released. HBase team recommends 2.0.1 and says there shouldnt 
be any API surprises. (but we never know)



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-20204) Type conversion during IN () comparisons is using different rules from other comparison operations

[Jason Dere (JIRA)]

Jason Dere created HIVE-20204:
-

 Summary: Type conversion during IN () comparisons is using 
different rules from other comparison operations
 Key: HIVE-20204
 URL: https://issues.apache.org/jira/browse/HIVE-20204
 Project: Hive
  Issue Type: Bug
  Components: Types
Reporter: Jason Dere
Assignee: Jason Dere


Noticed this while looking at HIVE-20082.
The type conversion done during GenericUDFIn (via 
ReturnObjectInspectorResolver) uses FunctionRegistry.getCommonClass(), whereas 
the other comparison operators (=, <, >, <=, >=) use 
FunctionRegistry.getCommonClassForComparison(). As a result, dec_column IN 
('1.1', '2.2') compares the values as strings, whereas dec_column = '1.1' would 
compare the values as doubles. This makes a difference for HIVE-20082 since it 
is related to changing the 0-padding during decimal-to-string conversions.

cc [~ashutoshc]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-20203) Arrow SerDe leaks a DirectByteBuffer

[Eric Wohlstadter (JIRA)]

Eric Wohlstadter created HIVE-20203:
---

 Summary: Arrow SerDe leaks a DirectByteBuffer
 Key: HIVE-20203
 URL: https://issues.apache.org/jira/browse/HIVE-20203
 Project: Hive
  Issue Type: Bug
Reporter: Eric Wohlstadter
Assignee: Eric Wohlstadter


ArrowColumnarBatchSerDe allocates an arrow NullableMapVector for each task that 
uses the serde.

The vector is a DirectByteBuffer allocated from Arrow's off-heap buffer pool.

This buffer is never closed and leaks about 1K of physical memory for each task.

This patch does three things:
 # Ensure the buffer is closed when the RecordWriter for the task is closed. 
 # Adds per-task memory accounting by assigning a ChildAllocator to each task 
from the RootAllocator.
 # Enforces that the ChildAllocator for a task has released all memory assigned 
to it, when the task is completed. 

The patch assumes that close() is always called on the RecordWriter when a task 
is finished (even if their is a failure during task execution). 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 67966: HIVE-20202: Add profiler endpoint to httpserver

[j . prasanth . j]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67966/
---

(Updated July 18, 2018, 6:42 p.m.)


Review request for hive and Gopal V.


Changes
---

minor comment fix


Bugs: HIVE-20202
https://issues.apache.org/jira/browse/HIVE-20202


Repository: hive-git


Description
---

HIVE-20202: Add profiler endpoint to httpserver


Diffs (updated)
-

  bin/hive 1ade51e 
  common/src/java/org/apache/hive/common/util/ProcessUtils.java PRE-CREATION 
  common/src/java/org/apache/hive/http/HttpServer.java 3cb7a33 
  common/src/java/org/apache/hive/http/ProfileServlet.java PRE-CREATION 


Diff: https://reviews.apache.org/r/67966/diff/2/

Changes: https://reviews.apache.org/r/67966/diff/1-2/


Testing
---


Thanks,

Prasanth_J

Review Request 67966: HIVE-20202: Add profiler endpoint to httpserver

[j . prasanth . j]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67966/
---

Review request for hive and Gopal V.


Bugs: HIVE-20202
https://issues.apache.org/jira/browse/HIVE-20202


Repository: hive-git


Description
---

HIVE-20202: Add profiler endpoint to httpserver


Diffs
-

  bin/hive 1ade51eebd6eac18c8a2295dd995263988d03e97 
  common/src/java/org/apache/hive/common/util/ProcessUtils.java PRE-CREATION 
  common/src/java/org/apache/hive/http/HttpServer.java 
3cb7a33c4e753bced603d8ef1a5917748b7a2c0a 
  common/src/java/org/apache/hive/http/ProfileServlet.java PRE-CREATION 


Diff: https://reviews.apache.org/r/67966/diff/1/


Testing
---


Thanks,

Prasanth_J

[jira] [Created] (HIVE-20202) Add profiler endpoint to httpserver

[Prasanth Jayachandran (JIRA)]

Prasanth Jayachandran created HIVE-20202:


 Summary: Add profiler endpoint to httpserver
 Key: HIVE-20202
 URL: https://issues.apache.org/jira/browse/HIVE-20202
 Project: Hive
  Issue Type: New Feature
Affects Versions: 4.0.0, 3.2.0
Reporter: Prasanth Jayachandran
Assignee: Prasanth Jayachandran


Add a web endpoint for profiling based on async-profiler. This servlet should 
be added to httpserver so that HS2 and LLAP daemons can output flamegraphs when 
their /prof endpoint is hit. Since this will be based on 
[https://github.com/jvm-profiling-tools/async-profiler] heap allocation, lock 
contentions, HW counters etc. will also be supported in addition to cpu 
profiling. In most cases the profiling overhead is pretty low and is safe to 
run on production. More analysis on CPU and memory overhead here 
[https://github.com/jvm-profiling-tools/async-profiler/issues/14] and 
[https://github.com/jvm-profiling-tools/async-profiler/issues/131] 

 

For the impatient, here is the usage doc and the sample output 
[https://github.com/prasanthj/nightswatch/blob/master/README.md] 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] hive pull request #403: HIVE-19927: Last Repl ID set by bootstrap dump is in...

[sankarh]

GitHub user sankarh opened a pull request:

https://github.com/apache/hive/pull/403

HIVE-19927: Last Repl ID set by bootstrap dump is incorrect and may cause 
data loss  if have ACID/MM tables.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/sankarh/hive HIVE-19927

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/hive/pull/403.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #403


commit 434aa1fe7060d1aab393d693e471e67880d599d8
Author: Sankar Hariappan 
Date:   2018-07-08T10:43:47Z

HIVE-19927: Last Repl ID set by bootstrap dump is incorrect and may cause 
data loss  if have ACID/MM tables.




---

[jira] [Created] (HIVE-20201) Hive shouldn't use HBase's Base64 implementation

[Mike Drob (JIRA)]

Mike Drob created HIVE-20201:


 Summary: Hive shouldn't use HBase's Base64 implementation
 Key: HIVE-20201
 URL: https://issues.apache.org/jira/browse/HIVE-20201
 Project: Hive
  Issue Type: Task
  Components: HBase Handler
Reporter: Mike Drob
Assignee: Mike Drob


HBase is removing their Base64 implementation because it never should have been 
public, so Hive should switch to a different provider. Hive already uses 
Commons-Codec Base64 in other places, so that would be a natural replacement.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [VOTE] Should we release storage-api 2.7.0 rc1?

[Jesus Camacho Rodriguez]

+1

Built from sources and ran tests.

-Jesús



On 7/16/18, 10:31 AM, "Ashutosh Chauhan"  wrote:

+1
Built from sources.
Ran unit tests.
Checksums and sigs matched up.

On Mon, Jul 16, 2018 at 8:58 AM Owen O'Malley 
wrote:

> +1
>
> built & ran tests
> checked checksums & signature
> tested with ORC
>
> On Thu, Jul 12, 2018 at 4:37 PM, Deepak Jaiswal 
> wrote:
>
> > Hi,
> >
> > I have prepared the rc1 off of branch-3.1.
> > Artifacts:
> > Tag : https://github.com/apache/hive/releases/tag/storage-
> > release-2.7.0-rc1
> > Tar Ball : http://home.apache.org/~djaiswal/hive-storage-2.7.0/
> >
> > Regards,
> > Deepak
> >
> > On 7/10/18, 10:16 AM, "Deepak Jaiswal" 
> wrote:
> >
> > Thanks Owen for finding this out. I will work on the next RC once
> this
> > blocker is resolved.
> >
> > Regards,
> > Deepak
> >
> > On 7/10/18, 9:40 AM, "Owen O'Malley"  wrote:
> >
> > Ok, Jesus and I tracked it down and I've filed
> > https://issues.apache.org/jira/browse/HIVE-20135 that is a
> > blocker on
> > storage-api 2.7.0.
> >
> > The impact was that orc 1.5 and master failed with the RC. orc
> 1.4
> > and
> > older were fine.
> >
> > .. Owen
> >
> > On Tue, Jul 10, 2018 at 8:17 AM, Owen O'Malley <
> > owen.omal...@gmail.com>
> > wrote:
> >
> > > I wanted to give an update on this. For now, I'm -1 because 
the
> > ORC
> > > (branch-1.5) tests fail with this RC. I'll dig into what is
> > wrong, but it
> > > looks like something in the timezone changes broke backwards
> > compatibility.
> > >
> > > .. Owen
> > >
> > > On Mon, Jul 9, 2018 at 11:12 AM, Deepak Jaiswal <
> > djais...@hortonworks.com>
> > > wrote:
> > >
> > >> Thanks Alan.
> > >>
> > >> On 7/9/18, 10:17 AM, "Alan Gates" 
> wrote:
> > >>
> > >> +1.  Did a build with a clean maven repo, checked the
> > signature and
> > >> sha
> > >> hash, ran RAT.
> > >>
> > >> Alan.
> > >>
> > >> On Fri, Jul 6, 2018 at 2:21 PM Deepak Jaiswal <
> > >> djais...@hortonworks.com>
> > >> wrote:
> > >>
> > >> > Hi,
> > >> >
> > >> > I would like to make a new release of the storage-api.
> It
> > contains
> > >> changes
> > >> > required for Hive 3.1 release.
> > >> >
> > >> > Artifcats:
> > >> > Tag :
> > >> > https://github.com/apache/hive/releases/tag/storage-
> > release-
> > >> 2.7.0-rc0
> > >> > Tar Ball : http://home.apache.org/~
> > djaiswal/hive-storage-2.7.0/
> > >> >
> > >> > Regards,
> > >> > Deepak
> > >> >
> > >>
> > >>
> > >>
> > >
> >
> >
> >
> >
> >
>

[jira] [Created] (HIVE-20200) Huge performance gap when processing ORC files created by Spark

[Vinoth Sathappan (JIRA)]

Vinoth Sathappan created HIVE-20200:
---

 Summary: Huge performance gap when processing ORC files created by 
Spark
 Key: HIVE-20200
 URL: https://issues.apache.org/jira/browse/HIVE-20200
 Project: Hive
  Issue Type: Bug
  Components: Hive, ORC
Affects Versions: 2.1.0
Reporter: Vinoth Sathappan


Seeing a huge performance difference while running a simple filter query on ORC 
files created by Spark. I'm seeing better performance if the files are written 
by Hive i.e. after doing a "create table x as select * from y". 

 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] hive pull request #402: HIVE-20192: HS2 with embedded metastore is leaking J...

[sankarh]

GitHub user sankarh opened a pull request:

https://github.com/apache/hive/pull/402

HIVE-20192: HS2 with embedded metastore is leaking JDOPersistenceManager 
objects.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/sankarh/hive HIVE-20192

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/hive/pull/402.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #402


commit fdd94019ca5c2616d796905fd7748c9aa77c9a12
Author: Sankar Hariappan 
Date:   2018-07-18T14:14:55Z

HIVE-20192: HS2 with embedded metastore is leaking JDOPersistenceManager 
objects.




---

[jira] [Created] (HIVE-20199) Improved filtering performance for a large number of partitions in a single table.

[Biao Wu (JIRA)]

Biao Wu created HIVE-20199:
--

 Summary: Improved filtering performance for a large number of 
partitions in a single table.
 Key: HIVE-20199
 URL: https://issues.apache.org/jira/browse/HIVE-20199
 Project: Hive
  Issue Type: Improvement
  Components: Metastore
Reporter: Biao Wu
Assignee: Biao Wu


eg:
{code:sql}
select * from test where dt = '20180606'
{code}
The filter 'dt=20180606' will be pushed down to mysql for execution, but the 
test table contains a large number of partitions, this is a poor performance.




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Change patch naming convention

[Dániel Vörös]

Hi,

Thanks Zoltan for starting this discussion. I ran into this a couple times
lately.

I've just opened https://issues.apache.org/jira/browse/HIVE-20191 to
mitigate the
effect of a wrong submission. Since I was backporting from master, I did't
even
notice that the patches were applied on master.

I'd be in favor of changing the pattern, or allowing both '.' and '-'.

@Vihang, from the Hive QE comments it seems that your first two patches were
also applied on master instead of branch-3.

Regards,
Daniel

On Tue, Jul 17, 2018 at 6:29 PM Vihang Karajgaonkar
 wrote:

> It doesn't matter if you use a dot v/s a dash to separate the patch version
> with the profile name (same as branch name but its technically different).
> See https://issues.apache.org/jira/browse/HIVE-18885 as an example where I
> had submitted both the naming styles and precommit picked it up. Having
> said that I find this naming convention confusing too. Does anyone know how
> do the other projects handle branch specific patch names?
>
> On Tue, Jul 17, 2018 at 2:26 AM, Zoltan Haindrich  wrote:
>
> > Hello!
> >
> > I've seen a few people running into some problems because the current
> > patch naming convention is a little bit odd:
> >
> > HIVE-[.][-].patch
> >
> > ...so if you would like to submit patch #02 to branch-2:
> >
> > A file named: HIVE-111.02-branch-2 patch should be uploaded (note that
> > there is a mandatory dash after the patch number)
> > I think this looks a bit odd also because most of the branch-es has a "-"
> > in there name (branch-2/branch-3).
> >
> > Because most people are naturally uploading files with the following
> > format, I would like to recommend to change the convention to:
> > HIVE-[.][.].patch
> >
> > Examples:
> > HIVE-111.feature-branch.patch
> > HIVE-111.123.feature-branch.patch
> > HIVE-111.123.feature-branch.patch
> >
> > cheers,
> > Zoltan
> >
>

Re: Review Request 67731: HIVE-19902: Provide Metastore micro-benchmarks

[Alexander Kolbasov]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67731/
---

(Updated July 18, 2018, 6:42 a.m.)


Review request for hive, Andrew Sherman, Alan Gates, Janaki Lahorani, Marta 
Kuczora, Peter Vary, Sahil Takiar, and Vihang Karajgaonkar.


Changes
---

Moved everything under standalone-metastore


Summary (updated)
-

HIVE-19902: Provide Metastore micro-benchmarks


Bugs: HIVE-19902
https://issues.apache.org/jira/browse/HIVE-19902


Repository: hive-git


Description (updated)
---

HIVE-19902: Provide Metastore micro-benchmarks


Diffs (updated)
-

  standalone-metastore/metastore-tools/metastore-benchmarks/README.md 
PRE-CREATION 
  standalone-metastore/metastore-tools/metastore-benchmarks/pom.xml 
PRE-CREATION 
  
standalone-metastore/metastore-tools/metastore-benchmarks/src/main/java/org/apache/hadoop/hive/metastore/tools/BenchData.java
 PRE-CREATION 
  
standalone-metastore/metastore-tools/metastore-benchmarks/src/main/java/org/apache/hadoop/hive/metastore/tools/BenchmarkTool.java
 PRE-CREATION 
  
standalone-metastore/metastore-tools/metastore-benchmarks/src/main/java/org/apache/hadoop/hive/metastore/tools/HMSBenchmarks.java
 PRE-CREATION 
  
standalone-metastore/metastore-tools/metastore-benchmarks/src/main/resources/log4j.properties
 PRE-CREATION 
  
standalone-metastore/metastore-tools/metastore-benchmarks/src/main/resources/log4j2.xml
 PRE-CREATION 
  standalone-metastore/metastore-tools/pom.xml PRE-CREATION 
  standalone-metastore/metastore-tools/tools-common/pom.xml PRE-CREATION 
  
standalone-metastore/metastore-tools/tools-common/src/main/java/org/apache/hadoop/hive/metastore/tools/BenchmarkSuite.java
 PRE-CREATION 
  
standalone-metastore/metastore-tools/tools-common/src/main/java/org/apache/hadoop/hive/metastore/tools/Constants.java
 PRE-CREATION 
  
standalone-metastore/metastore-tools/tools-common/src/main/java/org/apache/hadoop/hive/metastore/tools/HMSClient.java
 PRE-CREATION 
  
standalone-metastore/metastore-tools/tools-common/src/main/java/org/apache/hadoop/hive/metastore/tools/MicroBenchmark.java
 PRE-CREATION 
  
standalone-metastore/metastore-tools/tools-common/src/main/java/org/apache/hadoop/hive/metastore/tools/Util.java
 PRE-CREATION 
  
standalone-metastore/metastore-tools/tools-common/src/test/java/org/apache/hadoop/hive/metastore/tools/HMSClientTest.java
 PRE-CREATION 
  
standalone-metastore/metastore-tools/tools-common/src/test/java/org/apache/hadoop/hive/metastore/tools/UtilTest.java
 PRE-CREATION 
  standalone-metastore/pom.xml 65bb86ee82cdeca63f9b6daf9bb262f65c6c61a0 


Diff: https://reviews.apache.org/r/67731/diff/4/

Changes: https://reviews.apache.org/r/67731/diff/3-4/


Testing
---


Thanks,

Alexander Kolbasov