[jira] [Created] (HIVE-20763) Add google cloud storage (gs) to the exim uri schema whitelist

[Siddharth Seth (JIRA)]

Siddharth Seth created HIVE-20763:
-

 Summary: Add google cloud storage (gs) to the exim uri schema 
whitelist
 Key: HIVE-20763
 URL: https://issues.apache.org/jira/browse/HIVE-20763
 Project: Hive
  Issue Type: Task
Reporter: Siddharth Seth
Assignee: Siddharth Seth
 Attachments: HIVE-20763.01.patch

import/export is enabled for s3a by default. Ideally this list should include 
other cloud storage options. This Jira adds Google Storage to the list.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [DISCUSS] Unsustainable situation with ptests

[Siddharth Seth]

Very nice. There was an effort to get fast and green builds back in 2016.
There wasn't any strict "must be a green build" before commit at the time
though. Instead jiras were filed and the expectation was that they'd be
cited / new ones created pre commit(looking at the jiras now - this was
likely followed for a while, many fixes, and eventually got annoying?).
Think the enforcement step is absolutely required to get to, and maintain a
green build. May want to consider performance characteristics of tests as
well - must complete with X seconds.
Jiras for reference (including test infra improvements which were not done
at the time): HIVE-13503, HIVE-15058, HIVE-14547

This will be painful initially, but eventually it'll be great to be able to
commit without having to scan through a bunch of 'known failures', analyze,
document etc.

On Tue, May 15, 2018 at 5:30 PM, Prasanth Jayachandran <
pjayachand...@hortonworks.com> wrote:

> Wow! Awesome. This is the 3rd time I remember seeing green run in >4yrs. :)
>
> Thanks
> Prasanth
>
> > On May 15, 2018, at 5:28 PM, Jesus Camacho Rodriguez <
> jcama...@apache.org> wrote:
> >
> > We have just had the first clean run in a while:
> > https://builds.apache.org/job/PreCommit-HIVE-Build/10971/testReport/
> >
> > I will continue monitoring follow-up runs.
> >
> > Thanks,
> > -Jesús
> >
> >
> > On 5/14/18, 11:28 PM, "Prasanth Jayachandran" <
> pjayachand...@hortonworks.com> wrote:
> >
> >Wondering if we can add a state transition from “Patch Available” to
> “Ready To Commit” which can only be triggered by ptest bot on green test
> run.
> >
> >Thanks
> >Prasanth
> >
> >
> >
> >On Mon, May 14, 2018 at 10:44 PM -0700, "Jesus Camacho Rodriguez" <
> jcama...@apache.org> wrote:
> >
> >
> >I have been working on fixing this situation while commits were still
> coming in.
> >
> >All the tests that have been disabled are in:
> >https://issues.apache.org/jira/browse/HIVE-19509
> >I have created new issues to reenable each of them, they are linked
> to that issue.
> >Maybe I was slightly aggressive disabling some of the tests, however
> that seemed to be the only way to bring the tests failures with age count >
> 1 to zero.
> >
> >Instead of starting a vote to freeze the commits in another thread, I
> will start a vote to be stricter wrt committing to master, i.e., only
> commit if we get a clean QA run.
> >
> >We can discuss more about this issue over there.
> >
> >Thanks,
> >Jesús
> >
> >
> >
> >On 5/14/18, 4:11 PM, "Sergey Shelukhin"  wrote:
> >
> >Can we please make this freeze conditional, i.e. we unfreeze
> automatically
> >after ptest is clean (as evidenced by the clean HiveQA run on a
> given
> >JIRA).
> >
> >On 18/5/14, 15:16, "Alan Gates"  wrote:
> >
> >> We should do it in a separate thread so that people can see it with the
> >> [VOTE] subject.  Some people use that as a filter in their email to know
> >> when to pay attention to things.
> >>
> >> Alan.
> >>
> >> On Mon, May 14, 2018 at 2:36 PM, Prasanth Jayachandran <
> >> pjayachand...@hortonworks.com> wrote:
> >>
> >>> Will there be a separate voting thread? Or the voting on this thread is
> >>> sufficient for lock down?
> >>>
> >>> Thanks
> >>> Prasanth
> >>>
>  On May 14, 2018, at 2:34 PM, Alan Gates  wrote:
> 
>  ​I see there's support for this, but people are still pouring in
> >>> commits.
>  I proposed we have a quick vote on this to lock down the commits
> >>> until we
>  get to green.  That way everyone knows we have drawn the line at a
> >>> specific
>  point.  Any commits after that point would be reverted.  There isn't a
>  category in the bylaws that fits this kind of vote but I suggest lazy
>  majority as the most appropriate one (at least 3 votes, more +1s than
>  -1s).
> 
>  Alan.​
> 
>  On Mon, May 14, 2018 at 10:34 AM, Vihang Karajgaonkar <
> >>> vih...@cloudera.com>
>  wrote:
> 
> > I worked on a few quick-fix optimizations in Ptest infrastructure
> >>> over
> >>> the
> > weekend which reduced the execution run from ~90 min to ~70 min per
> >>> run. I
> > had to restart Ptest multiple times. I was resubmitting the patches
> >>> which
> > were in the queue manually, but I may have missed a few. In case you
> >>> have a
> > patch which is pending pre-commit and you don't see it in the queue,
> >>> please
> > submit it manually or let me know if you don't have access to the
> >>> jenkins
> > job. I will continue to work on the sub-tasks in HIVE-19425 and will
> >>> do
> > some maintenance next weekend as well.
> >
> > On Mon, May 14, 2018 at 7:42 AM, Jesus Camacho Rodriguez <
> > jcama...@apache.org> wrote:
> >
> >> Vineet has already been working on disabling those tests that were
> >>> timing
> >> out. I am working on disabling those that are generating different q
> > 

Re: [VOTE] Stricter commit guidelines

[Siddharth Seth]

+1

On Mon, May 14, 2018 at 10:44 PM, Jesus Camacho Rodriguez <
jcama...@apache.org> wrote:

> After work has been done to ignore most of the tests that were failing
> consistently/intermittently [1], I wanted to start this vote to gather
> support from the community to be stricter wrt committing patches to Hive.
> The committers guide [2] already specifies that a +1 should be obtained
> before committing, but there is another clause that allows committing under
> the presence of flaky tests (clause 4). Flaky tests are as good as having
> no tests, hence I propose to remove clause 4 and enforce the +1 from
> testing infra before committing.
>
>
>
> As I see it, by enforcing that we always get a +1 from the testing infra
> before committing, 1) we will have a more stable project, and 2) we will
> have another incentive as a community to create a more robust testing
> infra, e.g., replacing flaky tests for similar unit tests that are not
> flaky, trying to decrease running time for tests, etc.
>
>
>
> Please, share your thoughts about this.
>
>
>
> Here is my +1.
>
>
>
> Thanks,
>
> Jesús
>
>
>
> [1] http://mail-archives.apache.org/mod_mbox/hive-dev/201805.
> mbox/%3C63023673-AEE5-41A9-BA52-5A5DFB2078B6%40apache.org%3E
>
> [2] https://cwiki.apache.org/confluence/display/Hive/
> HowToCommit#HowToCommit-PreCommitruns,andcommittingpatches
>
>
>
>

[jira] [Created] (HIVE-17093) LLAP ssl configs need to be localized to talk to a wire encrypted hdfs

[Siddharth Seth (JIRA)]

Siddharth Seth created HIVE-17093:
-

 Summary: LLAP ssl configs need to be localized to talk to a wire 
encrypted hdfs
 Key: HIVE-17093
 URL: https://issues.apache.org/jira/browse/HIVE-17093
 Project: Hive
  Issue Type: Bug
Reporter: Siddharth Seth
Assignee: Siddharth Seth






--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: [Announce] New PMC members

[Siddharth Seth]

Congratulations!

On Tue, May 30, 2017 at 8:55 AM, Sergio Pena 
wrote:

> Thanks all for your comments, and congratulations to all new PMC members
> too !!!
>
> - Sergio
>
> On Sun, May 28, 2017 at 8:43 PM, Rajesh Balamohan 
> wrote:
>
> > Congratulations to new PMC members!
> >
> > ~Rajesh.B
> >
> > On Sat, May 27, 2017 at 12:14 PM, Zoltan Haindrich <
> > zhaindr...@hortonworks.com> wrote:
> >
> > > Congratulations!
> > >
> > > On 27 May 2017 06:26, Lefty Leverenz  wrote:
> > > Congratulations to the new PMC members, and also to Hive for having
> such
> > a
> > > strong and talented community!
> > >
> > > -- Lefty
> > >
> > >
> > > On Fri, May 26, 2017 at 1:12 PM, Eugene Koifman <
> > ekoif...@hortonworks.com>
> > > wrote:
> > >
> > > > Congratualtions!
> > > >
> > > > On 5/26/17, 10:10 AM, "Prasanth Jayachandran" <
> > > > pjayachand...@hortonworks.com> wrote:
> > > >
> > > > Congratulations to all of you!
> > > >
> > > > Thanks
> > > > Prasanth
> > > > > On May 26, 2017, at 10:09 AM, Pengcheng Xiong <
> pxi...@apache.org
> > >
> > > > wrote:
> > > > >
> > > > > Yongzhi, Daniel, Vaibhav, Sergio, Aihua, and Chaoyu!
> > > > >
> > > > > Congratulations to all of you!
> > > > >
> > > > > Best
> > > > > Pengcheng
> > > > >
> > > > > On Thu, May 25, 2017 at 10:26 PM, Peter Vary <
> pv...@cloudera.com
> > >
> > > > wrote:
> > > > >
> > > > >> Wow!
> > > > >> That's a spring shower of PMCs. :)
> > > > >> Well deserved Yongzhi, Daniel, Vaibhav, Sergio, Aihua, Chaoyu!
> > > > >>
> > > > >> Congratulations to all of you!
> > > > >>
> > > > >> Peter
> > > > >>
> > > > >> 2017. máj. 26. 6:42 ezt írta ("Ashutosh Chauhan" <
> > > > hashut...@apache.org>):
> > > > >>
> > > > >> The Project Management Committee (PMC) for Apache Hive has
> > invited
> > > > Yongzhi
> > > > >> Chen to become a PMC member and we are pleased to announce
> that
> > he
> > > > has
> > > > >> accepted.
> > > > >>
> > > > >> Please join me in congratulating Yongzhi!
> > > > >>
> > > > >> Thanks,
> > > > >> Ashutosh on behalf of Hive PMC
> > > > >>
> > > >
> > > >
> > > >
> > > >
> > >
> > >
> >
>

[jira] [Created] (HIVE-16781) Improve UGI handling for AM restarts, external services

[Siddharth Seth (JIRA)]

Siddharth Seth created HIVE-16781:
-

 Summary: Improve UGI handling for AM restarts, external services
 Key: HIVE-16781
 URL: https://issues.apache.org/jira/browse/HIVE-16781
 Project: Hive
  Issue Type: Task
Reporter: Siddharth Seth


Currently UGI instances are cached. HIVE-16777 changes this to not use the 
cache for external submissions. There's potentially better approaches to handle 
this. Also deal with AM restarts (recovery)



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (HIVE-16777) LLAP: Use separate tokens and UGI instances when an external client is used

[Siddharth Seth (JIRA)]

Siddharth Seth created HIVE-16777:
-

 Summary: LLAP: Use separate tokens and UGI instances when an 
external client is used
 Key: HIVE-16777
 URL: https://issues.apache.org/jira/browse/HIVE-16777
 Project: Hive
  Issue Type: Bug
Reporter: Siddharth Seth
Assignee: Siddharth Seth


Otherwise leads to errors since the token is shared, and there's different 
nodes running Umbilical.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Re: Review Request 59325: Cleanup of structures required when LLAP access from external clients completes

[Siddharth Seth]

> On May 17, 2017, 2:02 a.m., Sergey Shelukhin wrote:
> > llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/QueryTracker.java
> > Lines 464 (patched)
> > <https://reviews.apache.org/r/59325/diff/1/?file=1722071#file1722071line483>
> >
> > will there be a callable created after every single fragment that 
> > finishes for the query with no other fragments running? Perhaps one 
> > callable should wait for the entire query and the fragments coming it 
> > should just push back the timestamp at which the query would time out and 
> > be cleaned?
> > 
> > also what if readLock blocks all cleanup attempts?
> 
> Siddharth Seth wrote:
> In terms of the callable - was trying to keep it simple. Can try making 
> use of a single callable.
> 
> Why would a readLock block all cleanup attempts? If I'm not mistaken, a 
> readLock means something else is running - which would cause another cleanup 
> to be scheduled.

Haven't made the Callable change here. Made small improvements to when cleanup 
runs (it will not create a new lock).
Will file a follow up for the single Callable. Think it adds a bit of code 
since we can't have a thread wait, so it'll really be a one callable optionally 
schedules the next callable etc.


- Siddharth


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59325/#review175184
---


On May 17, 2017, 9:56 p.m., Siddharth Seth wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/59325/
> ---
> 
> (Updated May 17, 2017, 9:56 p.m.)
> 
> 
> Review request for hive, Jason Dere and Sergey Shelukhin.
> 
> 
> Bugs: HIVE-14052
> https://issues.apache.org/jira/browse/HIVE-14052
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Cleanup of structures required when LLAP access from external clients 
> completes
> 
> 
> Diffs
> -
> 
>   
> llap-common/src/gen/protobuf/gen-java/org/apache/hadoop/hive/llap/daemon/rpc/LlapDaemonProtocolProtos.java
>  ece31ed 
>   llap-common/src/protobuf/LlapDaemonProtocol.proto 3a3a2b8 
>   llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/QueryInfo.java 
> ce2f457 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/QueryTracker.java
>  daeb555 
>   
> llap-server/src/test/org/apache/hadoop/hive/llap/daemon/impl/TaskExecutorTestHelpers.java
>  27c426c 
>   ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDTFGetSplits.java 
> 868eec7 
> 
> 
> Diff: https://reviews.apache.org/r/59325/diff/2/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Siddharth Seth
> 
>

Re: Review Request 59325: Cleanup of structures required when LLAP access from external clients completes

[Siddharth Seth]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59325/
---

(Updated May 17, 2017, 9:56 p.m.)


Review request for hive, Jason Dere and Sergey Shelukhin.


Changes
---

Fixes review comments.

Adds one additional change to use a consistent "queryId". Earlier, RecordReader 
would pick up a different queryId from the rest of the system, which would 
geneate 2 log files. Only one of the two log files would get rotated.


Bugs: HIVE-14052
https://issues.apache.org/jira/browse/HIVE-14052


Repository: hive-git


Description
---

Cleanup of structures required when LLAP access from external clients completes


Diffs (updated)
-

  
llap-common/src/gen/protobuf/gen-java/org/apache/hadoop/hive/llap/daemon/rpc/LlapDaemonProtocolProtos.java
 ece31ed 
  llap-common/src/protobuf/LlapDaemonProtocol.proto 3a3a2b8 
  llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/QueryInfo.java 
ce2f457 
  
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/QueryTracker.java 
daeb555 
  
llap-server/src/test/org/apache/hadoop/hive/llap/daemon/impl/TaskExecutorTestHelpers.java
 27c426c 
  ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDTFGetSplits.java 
868eec7 


Diff: https://reviews.apache.org/r/59325/diff/2/

Changes: https://reviews.apache.org/r/59325/diff/1-2/


Testing
---


Thanks,

Siddharth Seth

Re: Review Request 59325: Cleanup of structures required when LLAP access from external clients completes

[Siddharth Seth]

> On May 17, 2017, 2:46 a.m., Jason Dere wrote:
> > llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/QueryTracker.java
> > Lines 490 (patched)
> > <https://reviews.apache.org/r/59325/diff/1/?file=1722071#file1722071line509>
> >
> > Does the size() check need to happen within the lock - what if size() 
> > == 0, but a new fragment is added while we try to get the dagLock?

I don't think it makes a difference. Will move the check into the writeLock to 
be safe.


- Siddharth


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59325/#review175192
---


On May 17, 2017, 12:30 a.m., Siddharth Seth wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/59325/
> ---
> 
> (Updated May 17, 2017, 12:30 a.m.)
> 
> 
> Review request for hive, Jason Dere and Sergey Shelukhin.
> 
> 
> Bugs: HIVE-14052
> https://issues.apache.org/jira/browse/HIVE-14052
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Cleanup of structures required when LLAP access from external clients 
> completes
> 
> 
> Diffs
> -
> 
>   
> llap-common/src/gen/protobuf/gen-java/org/apache/hadoop/hive/llap/daemon/rpc/LlapDaemonProtocolProtos.java
>  ece31ed 
>   llap-common/src/protobuf/LlapDaemonProtocol.proto 3a3a2b8 
>   llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/QueryInfo.java 
> ce2f457 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/QueryTracker.java
>  daeb555 
>   
> llap-server/src/test/org/apache/hadoop/hive/llap/daemon/impl/TaskExecutorTestHelpers.java
>  27c426c 
>   ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDTFGetSplits.java 
> 868eec7 
> 
> 
> Diff: https://reviews.apache.org/r/59325/diff/1/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Siddharth Seth
> 
>

Re: Review Request 59325: Cleanup of structures required when LLAP access from external clients completes

[Siddharth Seth]

> On May 17, 2017, 2:02 a.m., Sergey Shelukhin wrote:
> > llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/QueryTracker.java
> > Lines 192 (patched)
> > <https://reviews.apache.org/r/59325/diff/1/?file=1722071#file1722071line192>
> >
> > external tasks cannot use shuffle? just checking

No. They have their own data transfer mechanism, which should be complete by 
the time a task completes.


> On May 17, 2017, 2:02 a.m., Sergey Shelukhin wrote:
> > llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/QueryTracker.java
> > Lines 464 (patched)
> > <https://reviews.apache.org/r/59325/diff/1/?file=1722071#file1722071line483>
> >
> > will there be a callable created after every single fragment that 
> > finishes for the query with no other fragments running? Perhaps one 
> > callable should wait for the entire query and the fragments coming it 
> > should just push back the timestamp at which the query would time out and 
> > be cleaned?
> > 
> > also what if readLock blocks all cleanup attempts?

In terms of the callable - was trying to keep it simple. Can try making use of 
a single callable.

Why would a readLock block all cleanup attempts? If I'm not mistaken, a 
readLock means something else is running - which would cause another cleanup to 
be scheduled.


- Siddharth


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59325/#review175184
---


On May 17, 2017, 12:30 a.m., Siddharth Seth wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/59325/
> ---
> 
> (Updated May 17, 2017, 12:30 a.m.)
> 
> 
> Review request for hive, Jason Dere and Sergey Shelukhin.
> 
> 
> Bugs: HIVE-14052
> https://issues.apache.org/jira/browse/HIVE-14052
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Cleanup of structures required when LLAP access from external clients 
> completes
> 
> 
> Diffs
> -
> 
>   
> llap-common/src/gen/protobuf/gen-java/org/apache/hadoop/hive/llap/daemon/rpc/LlapDaemonProtocolProtos.java
>  ece31ed 
>   llap-common/src/protobuf/LlapDaemonProtocol.proto 3a3a2b8 
>   llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/QueryInfo.java 
> ce2f457 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/QueryTracker.java
>  daeb555 
>   
> llap-server/src/test/org/apache/hadoop/hive/llap/daemon/impl/TaskExecutorTestHelpers.java
>  27c426c 
>   ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDTFGetSplits.java 
> 868eec7 
> 
> 
> Diff: https://reviews.apache.org/r/59325/diff/1/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Siddharth Seth
> 
>

Review Request 59325: Cleanup of structures required when LLAP access from external clients completes

[Siddharth Seth]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59325/
---

Review request for hive, Jason Dere and Sergey Shelukhin.


Bugs: HIVE-14052
https://issues.apache.org/jira/browse/HIVE-14052


Repository: hive-git


Description
---

Cleanup of structures required when LLAP access from external clients completes


Diffs
-

  
llap-common/src/gen/protobuf/gen-java/org/apache/hadoop/hive/llap/daemon/rpc/LlapDaemonProtocolProtos.java
 ece31ed 
  llap-common/src/protobuf/LlapDaemonProtocol.proto 3a3a2b8 
  llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/QueryInfo.java 
ce2f457 
  
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/QueryTracker.java 
daeb555 
  
llap-server/src/test/org/apache/hadoop/hive/llap/daemon/impl/TaskExecutorTestHelpers.java
 27c426c 
  ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDTFGetSplits.java 
868eec7 


Diff: https://reviews.apache.org/r/59325/diff/1/


Testing
---


Thanks,

Siddharth Seth

[jira] [Created] (HIVE-16655) LLAP: Avoid preempting fragments before they enter the running state

[Siddharth Seth (JIRA)]

Siddharth Seth created HIVE-16655:
-

 Summary: LLAP: Avoid preempting fragments before they enter the 
running state
 Key: HIVE-16655
 URL: https://issues.apache.org/jira/browse/HIVE-16655
 Project: Hive
  Issue Type: Bug
Reporter: Siddharth Seth
Assignee: Siddharth Seth


Currently in the AM, fragments may be preempted as soon as they are allocated, 
without knowing whether they will move into the RUNNING state or not. Leads to 
a lot of unnecessary kills.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (HIVE-16634) LLAP Use a pool of connections to a single AM from a daemon

[Siddharth Seth (JIRA)]

Siddharth Seth created HIVE-16634:
-

 Summary: LLAP Use a pool of connections to a single AM from a 
daemon
 Key: HIVE-16634
 URL: https://issues.apache.org/jira/browse/HIVE-16634
 Project: Hive
  Issue Type: Bug
  Components: llap
Reporter: Siddharth Seth
Assignee: Siddharth Seth






--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Re: [Announce] New committer: Vineet Garg

[Siddharth Seth]

Congratulations Vineet.

On Tue, May 9, 2017 at 1:24 PM, Ashutosh Chauhan 
wrote:

> The Project Management Committee (PMC) for Apache Hive has invited Vineet
> Garg to become a committer and we are pleased to announce that he has
> accepted.
>
> Welcome, Vineet!
>
> Thanks,
> Ashutosh
>

[jira] [Created] (HIVE-16460) In the console output, show vertex list in topological order instead of an alphabetical sort

[Siddharth Seth (JIRA)]

Siddharth Seth created HIVE-16460:
-

 Summary: In the console output, show vertex list in topological 
order instead of an alphabetical sort
 Key: HIVE-16460
 URL: https://issues.apache.org/jira/browse/HIVE-16460
 Project: Hive
  Issue Type: Improvement
Reporter: Siddharth Seth


cc [~prasanth_j]



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (HIVE-16375) HiveConf type validation should handle variables as the value

[Siddharth Seth (JIRA)]

Siddharth Seth created HIVE-16375:
-

 Summary: HiveConf type validation should handle variables as the 
value
 Key: HIVE-16375
 URL: https://issues.apache.org/jira/browse/HIVE-16375
 Project: Hive
  Issue Type: Bug
Reporter: Siddharth Seth


e.g

Set hive.llap.io.threadpool.size to ${hive.llap.daemon.vcpus.per.instance}

ends up displaying a WARN message that the expected type is an INTEGER



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (HIVE-16325) Tez session refresh based on a time interval fails

[Siddharth Seth (JIRA)]

Siddharth Seth created HIVE-16325:
-

 Summary: Tez session refresh based on a time interval fails
 Key: HIVE-16325
 URL: https://issues.apache.org/jira/browse/HIVE-16325
 Project: Hive
  Issue Type: Bug
Reporter: Siddharth Seth
Assignee: Siddharth Seth
Priority: Critical


TezSessionSTate setup requires access to SessionState (which is a little 
absurd).
The expire / renew threads are not setup with any SessionState.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (HIVE-16319) LLAP: Better handling of an empty wait queue, should try scheduling checks

[Siddharth Seth (JIRA)]

Siddharth Seth created HIVE-16319:
-

 Summary: LLAP: Better handling of an empty wait queue, should try 
scheduling checks
 Key: HIVE-16319
 URL: https://issues.apache.org/jira/browse/HIVE-16319
 Project: Hive
  Issue Type: Bug
Reporter: Siddharth Seth
Assignee: Siddharth Seth






--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (HIVE-16233) llap: Query failed with AllocatorOutOfMemoryException

[Siddharth Seth (JIRA)]

Siddharth Seth created HIVE-16233:
-

 Summary: llap: Query failed with AllocatorOutOfMemoryException
 Key: HIVE-16233
 URL: https://issues.apache.org/jira/browse/HIVE-16233
 Project: Hive
  Issue Type: Bug
  Components: llap
Reporter: Siddharth Seth


{code}
TaskAttempt 5 failed, info=[Error: Error while running task ( failure ) : 
attempt_1488231257387_2288_25_05_56_5:java.lang.RuntimeException: 
org.apache.hadoop.hive.ql.metadata.HiveException: java.io.IOException: 
java.io.IOException: 
org.apache.hadoop.hive.common.io.Allocator$AllocatorOutOfMemoryException: 
Failed to allocate 262144; at 0 out of 1
at 
org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:211)
at 
org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:168)
at 
org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:370)
at 
org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:73)
at 
org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:61)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1724)
at 
org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:61)
at 
org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:37)
at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
at 
org.apache.hadoop.hive.llap.daemon.impl.StatsRecordingThreadPool$WrappedCallable.call(StatsRecordingThreadPool.java:110)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: 
java.io.IOException: java.io.IOException: 
org.apache.hadoop.hive.common.io.Allocator$AllocatorOutOfMemoryException: 
Failed to allocate 262144; at 0 out of 1
at 
org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.pushRecord(MapRecordSource.java:74)
at 
org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.run(MapRecordProcessor.java:419)
at 
org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:185)
... 15 more
Caused by: java.io.IOException: java.io.IOException: 
org.apache.hadoop.hive.common.io.Allocator$AllocatorOutOfMemoryException: 
Failed to allocate 262144; at 0 out of 1
at 
org.apache.hadoop.hive.io.HiveIOExceptionHandlerChain.handleRecordReaderNextException(HiveIOExceptionHandlerChain.java:121)
at 
org.apache.hadoop.hive.io.HiveIOExceptionHandlerUtil.handleRecordReaderNextException(HiveIOExceptionHandlerUtil.java:77)
at 
org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.doNext(HiveContextAwareRecordReader.java:365)
at 
org.apache.hadoop.hive.ql.io.HiveRecordReader.doNext(HiveRecordReader.java:79)
at 
org.apache.hadoop.hive.ql.io.HiveRecordReader.doNext(HiveRecordReader.java:33)
at 
org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.next(HiveContextAwareRecordReader.java:116)
at 
org.apache.hadoop.mapred.split.TezGroupedSplitsInputFormat$TezGroupedSplitsRecordReader.next(TezGroupedSplitsInputFormat.java:151)
at 
org.apache.tez.mapreduce.lib.MRReaderMapred.next(MRReaderMapred.java:116)
at 
org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.pushRecord(MapRecordSource.java:62)
... 17 more
Caused by: java.io.IOException: 
org.apache.hadoop.hive.common.io.Allocator$AllocatorOutOfMemoryException: 
Failed to allocate 262144; at 0 out of 1
at 
org.apache.hadoop.hive.ql.io.orc.encoded.EncodedReaderImpl.readEncodedColumns(EncodedReaderImpl.java:425)
at 
org.apache.hadoop.hive.llap.io.encoded.OrcEncodedDataReader.performDataRead(OrcEncodedDataReader.java:413)
at 
org.apache.hadoop.hive.llap.io.encoded.OrcEncodedDataReader$4.run(OrcEncodedDataReader.java:235)
at 
org.apache.hadoop.hive.llap.io.encoded.OrcEncodedDataReader$4.run(OrcEncodedDataReader.java:232)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1724)
at 
org.apache.hadoop.hive.llap.io.encoded.OrcEncodedDataReader.callInternal(OrcEncodedDataReader.java:232)
at 
org.apache.hadoop.hive.llap.io.encoded.OrcEncodedDataReader.callInternal(OrcEncodedDataReader.java:94

[jira] [Created] (HIVE-16217) Move llap properties which are not user configurable out of HiveConf

[Siddharth Seth (JIRA)]

Siddharth Seth created HIVE-16217:
-

 Summary: Move llap properties which are not user configurable out 
of HiveConf
 Key: HIVE-16217
 URL: https://issues.apache.org/jira/browse/HIVE-16217
 Project: Hive
  Issue Type: Task
Reporter: Siddharth Seth


LLAP_DAEMON_NM_ADDRESS, LLAP_DAEMON_CONTAINER_ID are two specific ones.




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Re: Green unit test results

[Siddharth Seth]

There's far too many flaky tests to enforce a clean test run prior to
commit.
However, there also needs to be a very conscious effort to fix flaky tests
- otherwise, we're not going to reach the state where this can be enforced.
I'm also not sure whether this is just flaky tests, or whether there are
real problems. Maybe a few volunteers could drive the test fixes, and as an
artifact of this, come up with guidelines on writing tests (avoid stats
comparisons in qfiles, skip lines which may be environment specific,
introduce a new compare mechanism for lines which may be in a different
order based on internal structures, etc).

On Thu, Mar 9, 2017 at 2:39 PM, Ashutosh Chauhan 
wrote:

> I went through all the builds of last 24 hours and though there are couple
> of green runs, we still have quite a bit of flakiness in our tests. Most of
> that is captured via jiras on :
> https://issues.apache.org/jira/browse/HIVE-15058 Till we have that
> flakiness I am not sure if we can enforce no commit on test failures
> policy. Once flakiness is straightened out then this discussion will become
> moot anyways.
> Towards that I think this flaky detector job is a step in right direction.
>
>
> On Thu, Mar 9, 2017 at 11:38 AM, Sergio Pena 
> wrote:
>
> > - Probably avoiding committing a patch if a flaky test is shown on the
> test
> > results?
> > - Should we add a jenkins job that checks for flaky tests like the hbase
> > project did?
> > https://builds.apache.org/view/H-L/view/HBase/job/HBase-
> Find-Flaky-Tests/
> >
> > On Thu, Mar 9, 2017 at 10:21 AM, Ashutosh Chauhan 
> > wrote:
> >
> > > Great news! Thanks to everyone who contributed in getting our tests and
> > > test infra sorted out.
> > > We would definitely want to keep the status either green or blue
> > definitely
> > > not red :) All our previous efforts in keeping builds green didn't bear
> > > fruit.
> > > So, I think we need to make some changes here.
> > >
> > > Any ideas what we can do to ensure green builds going forward?
> > >
> > > Thanks,
> > > Ashutosh
> > >
> > > On Thu, Mar 9, 2017 at 8:07 AM, Sergio Pena 
> > > wrote:
> > >
> > > > It's actually blue Peter :).
> > > >
> > > > But good job, I see that the console output is:
> > > >
> > > > {color:red}ERROR:{color} -1 due to no test(s) being added or
> modified.
> > > >
> > > > {color:green}SUCCESS:{color} +1 due to 10336 tests passed
> > > >
> > > >
> > > > On Thu, Mar 9, 2017 at 8:12 AM, Peter Vary 
> wrote:
> > > >
> > > > > Hi,
> > > > >
> > > > > Congratulations for everyone who have helped taking care of the
> unit
> > > test
> > > > > failures!
> > > > > I have got my first green run! :)
> > > > >
> > > > > If any of you interested in:
> > > > > https://builds.apache.org/job/PreCommit-HIVE-Build/4049/
> testReport/
> > <
> > > > > https://builds.apache.org/job/PreCommit-HIVE-Build/4049/
> testReport/>
> > > :)
> > > > >
> > > > > Great day, and again thanks everyone!
> > > > >
> > > > > Peter
> > > >
> > >
> >
>

[jira] [Created] (HIVE-16168) llap log links should use the NM nodeId port instead of web port

[Siddharth Seth (JIRA)]

Siddharth Seth created HIVE-16168:
-

 Summary: llap log links should use the NM nodeId port instead of 
web port
 Key: HIVE-16168
 URL: https://issues.apache.org/jira/browse/HIVE-16168
 Project: Hive
  Issue Type: Bug
  Components: llap
Reporter: Siddharth Seth
Assignee: Siddharth Seth






--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (HIVE-16165) use database broken on master

[Siddharth Seth (JIRA)]

Siddharth Seth created HIVE-16165:
-

 Summary: use database broken on master
 Key: HIVE-16165
 URL: https://issues.apache.org/jira/browse/HIVE-16165
 Project: Hive
  Issue Type: Bug
Affects Versions: 2.2.0
Reporter: Siddharth Seth
Priority: Blocker


{code}
2017-03-09T19:37:20,765  INFO [9171ecb2-4f38-4254-8138-f78a73c24181 main] 
parse.CalcitePlanner: Starting Semantic Analysis
2017-03-09T19:37:20,766  INFO [9171ecb2-4f38-4254-8138-f78a73c24181 main] 
parse.CalcitePlanner: Completed phase 1 of Semantic Analysis
2017-03-09T19:37:20,766  INFO [9171ecb2-4f38-4254-8138-f78a73c24181 main] 
parse.CalcitePlanner: Get metadata for source tables
2017-03-09T19:37:20,766  INFO [9171ecb2-4f38-4254-8138-f78a73c24181 main] 
parse.CalcitePlanner: Get metadata for subqueries
2017-03-09T19:37:20,766  INFO [9171ecb2-4f38-4254-8138-f78a73c24181 main] 
parse.CalcitePlanner: Get metadata for destination tables
2017-03-09T19:37:20,766  INFO [9171ecb2-4f38-4254-8138-f78a73c24181 main] 
parse.CalcitePlanner: Completed getting MetaData in Semantic Analysis
2017-03-09T19:37:20,766  INFO [9171ecb2-4f38-4254-8138-f78a73c24181 main] 
parse.BaseSemanticAnalyzer: Not invoking CBO because the statement doesn't have 
QUERY or EXPLAIN as root and not a CTAS; is not a query with at least one 
source table  or there is a subquery without a source table, or CTAS, or insert
2017-03-09T19:37:20,810  INFO [9171ecb2-4f38-4254-8138-f78a73c24181 main] 
ql.Context: New scratch dir is 
hdfs://cn105-10.l42scl.hortonworks.com:8020/tmp/hive/sseth/9171ecb2-4f38-4254-8138-f78a73c24181/hive_2017-03-09_19-37-20_763_6998351573308778636-1
2017-03-09T19:37:20,894  INFO [9171ecb2-4f38-4254-8138-f78a73c24181 main] 
ppd.OpProcFactory: Processing for TS(0)
2017-03-09T19:37:20,900 ERROR [9171ecb2-4f38-4254-8138-f78a73c24181 main] 
ql.Driver: FAILED: NullPointerException null
java.lang.NullPointerException
at 
org.apache.hadoop.hive.ql.stats.StatsUtils.estimateRowSizeFromSchema(StatsUtils.java:543)
at 
org.apache.hadoop.hive.ql.stats.StatsUtils.getNumRows(StatsUtils.java:180)
at 
org.apache.hadoop.hive.ql.stats.StatsUtils.collectStatistics(StatsUtils.java:204)
at 
org.apache.hadoop.hive.ql.stats.StatsUtils.collectStatistics(StatsUtils.java:154)
at 
org.apache.hadoop.hive.ql.stats.StatsUtils.collectStatistics(StatsUtils.java:142)
at 
org.apache.hadoop.hive.ql.optimizer.stats.annotation.StatsRulesProcFactory$TableScanStatsRule.process(StatsRulesProcFactory.java:130)
at 
org.apache.hadoop.hive.ql.lib.DefaultRuleDispatcher.dispatch(DefaultRuleDispatcher.java:90)
at 
org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatchAndReturn(DefaultGraphWalker.java:105)
at 
org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatch(DefaultGraphWalker.java:89)
at 
org.apache.hadoop.hive.ql.lib.LevelOrderWalker.walk(LevelOrderWalker.java:143)
at 
org.apache.hadoop.hive.ql.lib.LevelOrderWalker.startWalking(LevelOrderWalker.java:122)
at 
org.apache.hadoop.hive.ql.optimizer.stats.annotation.AnnotateWithStatistics.transform(AnnotateWithStatistics.java:78)
at 
org.apache.hadoop.hive.ql.parse.TezCompiler.runStatsAnnotation(TezCompiler.java:302)
at 
org.apache.hadoop.hive.ql.parse.TezCompiler.optimizeOperatorPlan(TezCompiler.java:96)
at 
org.apache.hadoop.hive.ql.parse.TaskCompiler.compile(TaskCompiler.java:140)
at 
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:11174)
at 
org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:285)
at 
org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:258)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:511)
at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1316)
at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1456)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1236)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1226)
at 
org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:233)
at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:184)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:403)
at 
org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:821)
at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:759)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:686)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.jav

[jira] [Created] (HIVE-16134) LLAP: Differentiate between preemption of a running fragment vs a queued fragment

[Siddharth Seth (JIRA)]

Siddharth Seth created HIVE-16134:
-

 Summary: LLAP: Differentiate between preemption of a running 
fragment vs a queued fragment
 Key: HIVE-16134
 URL: https://issues.apache.org/jira/browse/HIVE-16134
 Project: Hive
  Issue Type: Improvement
Reporter: Siddharth Seth






--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (HIVE-16097) minor fixes to metrics and logs in LlapTaskScheduler

[Siddharth Seth (JIRA)]

Siddharth Seth created HIVE-16097:
-

 Summary: minor fixes to metrics and logs in LlapTaskScheduler
 Key: HIVE-16097
 URL: https://issues.apache.org/jira/browse/HIVE-16097
 Project: Hive
  Issue Type: Bug
  Components: llap
Reporter: Siddharth Seth
Assignee: Siddharth Seth






--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (HIVE-16094) queued containers may timeout if they don't get to run for a long time

[Siddharth Seth (JIRA)]

Siddharth Seth created HIVE-16094:
-

 Summary: queued containers may timeout if they don't get to run 
for a long time
 Key: HIVE-16094
 URL: https://issues.apache.org/jira/browse/HIVE-16094
 Project: Hive
  Issue Type: Bug
Affects Versions: 2.2.0
Reporter: Siddharth Seth
Assignee: Siddharth Seth
Priority: Critical


I believe this happened after HIVE-15958 - since we end up keeping amNodeInfo 
in knownAppMaters, and that can result in the callable not being scheduled on 
new task registration.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (HIVE-16067) LLAP: send out container complete messages after a fragment completes

[Siddharth Seth (JIRA)]

Siddharth Seth created HIVE-16067:
-

 Summary: LLAP: send out container complete messages after a 
fragment completes
 Key: HIVE-16067
 URL: https://issues.apache.org/jira/browse/HIVE-16067
 Project: Hive
  Issue Type: Bug
Reporter: Siddharth Seth
Assignee: Siddharth Seth






--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (HIVE-16062) Share ATS domain with Tez

[Siddharth Seth (JIRA)]

Siddharth Seth created HIVE-16062:
-

 Summary: Share ATS domain with Tez
 Key: HIVE-16062
 URL: https://issues.apache.org/jira/browse/HIVE-16062
 Project: Hive
  Issue Type: Improvement
  Components: llap
Reporter: Siddharth Seth






--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Re: [VOTE] Drop support for Java7 in master branch

[Siddharth Seth]

+1

On Mon, Feb 27, 2017 at 8:54 PM, Thejas Nair  wrote:

> There was a [DISCUSS] thread on the topic of moving to jdk8 for unit tests
> [1], and many people also expressed the opinion that we should drop JDK 7
> support in Hive. Public updates by Oracle was stopped on Apr 2015 [2].
>
> This vote thread proposes to dropping JDK 7 support in the next Apache Hive
> 2.x release (ie master branch), so that we can start leveraging new
> features in Java 8 and also libraries that require java8.
>
> [1] https://s.apache.org/hive-jdk8-test
> [2] http://www.oracle.com/technetwork/java/eol-135779.html
>
> Here is my +1.
>
> Vote ends in 72 hours.
> Thanks,
> Thejas
>
> PS: I think this would fall under "Code change" under Hive-bylaws, so it
> doesn't seem to really require a formal vote thread. But I think this does
> merit wider attention than a jira ticket.
>

[jira] [Created] (HIVE-16053) Remove newRatio from llap JAVA_OPTS_BASE

[Siddharth Seth (JIRA)]

Siddharth Seth created HIVE-16053:
-

 Summary: Remove newRatio from llap JAVA_OPTS_BASE
 Key: HIVE-16053
 URL: https://issues.apache.org/jira/browse/HIVE-16053
 Project: Hive
  Issue Type: Bug
  Components: llap
Reporter: Siddharth Seth
Assignee: Siddharth Seth
 Attachments: HIVE-16053.01.patch

The G1GC is supposed to be able to resize regions as required. Setting the 
newRatio or other parameters which size the new gen disables this capability.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (HIVE-16036) Setup umbilical ugi once per app

[Siddharth Seth (JIRA)]

Siddharth Seth created HIVE-16036:
-

 Summary: Setup umbilical ugi once per app
 Key: HIVE-16036
 URL: https://issues.apache.org/jira/browse/HIVE-16036
 Project: Hive
  Issue Type: Improvement
Reporter: Siddharth Seth


HIVE-16020 moved this to once per query. Ideal is once per app.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (HIVE-16021) Avoid trying to inform AMs about fragment kills on query completion

[Siddharth Seth (JIRA)]

Siddharth Seth created HIVE-16021:
-

 Summary: Avoid trying to inform AMs about fragment kills on query 
completion
 Key: HIVE-16021
 URL: https://issues.apache.org/jira/browse/HIVE-16021
 Project: Hive
  Issue Type: Bug
  Components: llap
Reporter: Siddharth Seth


Query completion comes from the AM. There's no need to inform it about the 
fragments that have been killed on a specific daemon.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (HIVE-16005) miscellaneous small fixes to help with debuggability

[Siddharth Seth (JIRA)]

Siddharth Seth created HIVE-16005:
-

 Summary: miscellaneous small fixes to help with debuggability
 Key: HIVE-16005
 URL: https://issues.apache.org/jira/browse/HIVE-16005
 Project: Hive
  Issue Type: Task
Reporter: Siddharth Seth


- Include proc_ in cli, beeline, metastore, hs2 process args
- LLAP history logger - log QueryId instead of dagName (dag name is free 
flowing text)
- LLAP JXM ExecutorStatus - Log QueryId instead of dagName. Sort by running / 
queued
- Include thread name in TaskRunnerCallable so that it shows up in stack traces 
(will cause extra output in logs)



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (HIVE-15921) Re-order the slider stop command to avoid a force if possible

[Siddharth Seth (JIRA)]

Siddharth Seth created HIVE-15921:
-

 Summary: Re-order the slider stop command to avoid a force if 
possible
 Key: HIVE-15921
 URL: https://issues.apache.org/jira/browse/HIVE-15921
 Project: Hive
  Issue Type: Bug
Reporter: Siddharth Seth
Assignee: Siddharth Seth


A graceful stop is required for slider --service llapstatus to work properly



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (HIVE-15918) Add some debug messages to identify an issue getting runtimeInfo from tez

[Siddharth Seth (JIRA)]

Siddharth Seth created HIVE-15918:
-

 Summary: Add some debug messages to identify an issue getting 
runtimeInfo from tez
 Key: HIVE-15918
 URL: https://issues.apache.org/jira/browse/HIVE-15918
 Project: Hive
  Issue Type: Task
  Components: llap
Reporter: Siddharth Seth
Assignee: Siddharth Seth






--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (HIVE-15865) Improvements to hive --service llapstatus

[Siddharth Seth (JIRA)]

Siddharth Seth created HIVE-15865:
-

 Summary: Improvements to hive --service llapstatus 
 Key: HIVE-15865
 URL: https://issues.apache.org/jira/browse/HIVE-15865
 Project: Hive
  Issue Type: Improvement
Reporter: Siddharth Seth


Integrate with additional slider changes
- Sort container list by completed time
- Handle additional reasons like OS launch failed
- Information about pending requests, last allocation time

Maybe add the YARN URL for the app (HA is where this is problematic)



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (HIVE-15830) Allow additional view ACLs for tez jobs

[Siddharth Seth (JIRA)]

Siddharth Seth created HIVE-15830:
-

 Summary: Allow additional view ACLs for tez jobs
 Key: HIVE-15830
 URL: https://issues.apache.org/jira/browse/HIVE-15830
 Project: Hive
  Issue Type: Improvement
Reporter: Siddharth Seth
Assignee: Siddharth Seth


Allow users to grant view access to additional users when running tez jobs.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (HIVE-15801) Some logging improvements in LlapTaskScheduler

[Siddharth Seth (JIRA)]

Siddharth Seth created HIVE-15801:
-

 Summary: Some logging improvements in LlapTaskScheduler
 Key: HIVE-15801
 URL: https://issues.apache.org/jira/browse/HIVE-15801
 Project: Hive
  Issue Type: Task
Reporter: Siddharth Seth
Assignee: Siddharth Seth






--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (HIVE-15786) Provide additional information from the llapstatus command

[Siddharth Seth (JIRA)]

Siddharth Seth created HIVE-15786:
-

 Summary: Provide additional information from the llapstatus command
 Key: HIVE-15786
 URL: https://issues.apache.org/jira/browse/HIVE-15786
 Project: Hive
  Issue Type: Improvement
Reporter: Siddharth Seth
Assignee: Siddharth Seth


Slider is making enhancements to provide additional information like completed 
containers, pending containers etc.

Integrate with this to provide additional details in llapstatus.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (HIVE-15770) Test jira

[Siddharth Seth (JIRA)]

Siddharth Seth created HIVE-15770:
-

 Summary: Test jira
 Key: HIVE-15770
 URL: https://issues.apache.org/jira/browse/HIVE-15770
 Project: Hive
  Issue Type: Bug
Reporter: Siddharth Seth
Assignee: Sergey Shelukhin






--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (HIVE-15733) LLAP AMHeartbeat threads should be restricted per app

[Siddharth Seth (JIRA)]

Siddharth Seth created HIVE-15733:
-

 Summary: LLAP AMHeartbeat threads should be restricted per app
 Key: HIVE-15733
 URL: https://issues.apache.org/jira/browse/HIVE-15733
 Project: Hive
  Issue Type: Bug
Reporter: Siddharth Seth


Don't allow all threads to get stuck trying to heartbeat to a bad AM. That 
would adversely affect every other query.

cc [~rajesh.balamohan], [~prasanth_j]



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-15731) sessions are not returned to the sessionPool in case of an interrupt

[Siddharth Seth (JIRA)]

Siddharth Seth created HIVE-15731:
-

 Summary: sessions are not returned to the sessionPool in case of 
an interrupt
 Key: HIVE-15731
 URL: https://issues.apache.org/jira/browse/HIVE-15731
 Project: Hive
  Issue Type: Bug
Reporter: Siddharth Seth
Assignee: Siddharth Seth
Priority: Critical


While returning a session to the pool, the interrupt status on the thread seems 
to be set, which causes the pool return to fail.
The session slot is useless at this point. A HS2 instance configured for a 
single session will stop running queries.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-15722) LLAP: Avoid marking a query as complete if the AMReporter runs into an error

[Siddharth Seth (JIRA)]

Siddharth Seth created HIVE-15722:
-

 Summary: LLAP: Avoid marking a query as complete if the AMReporter 
runs into an error
 Key: HIVE-15722
 URL: https://issues.apache.org/jira/browse/HIVE-15722
 Project: Hive
  Issue Type: Bug
Reporter: Siddharth Seth
Assignee: Siddharth Seth


When the AMReporter runs into an error (typically intermittent), we end up 
killing all fragments on the daemon. This is done by marking the query as 
complete.
The AM would continue to try scheduling on this node - which would lead to task 
failures if the daemon structures are updated.

Instead of clearing the structures, it's better to kill the fragments, and let 
a queryComplete call come in from the AM.

Later, we could make enhancements in the AM to avoid such nodes. That's not 
simple though, since the AM will not find out what happened due to the 
communication failure from the daemon.

Leads to 
{code}
org.apache.hadoop.ipc.RemoteException(java.lang.RuntimeException): Dag query16 
already complete. Rejecting fragment [Map 7, 29, 0]
at 
org.apache.hadoop.hive.llap.daemon.impl.QueryTracker.registerFragment(QueryTracker.java:149)
at 
org.apache.hadoop.hive.llap.daemon.impl.ContainerRunnerImpl.submitWork(ContainerRunnerImpl.java:226)
at 
org.apache.hadoop.hive.llap.daemon.impl.LlapDaemon.submitWork(LlapDaemon.java:487)
at 
org.apache.hadoop.hive.llap.daemon.impl.LlapProtocolServerImpl.submitWork(LlapProtocolServerImpl.java:101)
at 
org.apache.hadoop.hive.llap.daemon.rpc.LlapDaemonProtocolProtos$LlapDaemonProtocol$2.callBlockingMethod(LlapDaemonProtocolProtos.java:16728)
at 
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:640)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:982)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2313)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2309)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1724)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2307)
{code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-15657) publish information to the vertex description field

[Siddharth Seth (JIRA)]

Siddharth Seth created HIVE-15657:
-

 Summary: publish information to the vertex description field
 Key: HIVE-15657
 URL: https://issues.apache.org/jira/browse/HIVE-15657
 Project: Hive
  Issue Type: Improvement
Reporter: Siddharth Seth


This can include information like the tables being processed, specific 
operations, etc. Maybe a sub-part of the explain plan?




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-15620) Enable a MiniLLAP cluster via the CLI

[Siddharth Seth (JIRA)]

Siddharth Seth created HIVE-15620:
-

 Summary: Enable a MiniLLAP cluster via the CLI
 Key: HIVE-15620
 URL: https://issues.apache.org/jira/browse/HIVE-15620
 Project: Hive
  Issue Type: Task
Reporter: Siddharth Seth
Assignee: Siddharth Seth


Starting LLAP on a node can be a challenge. Start YARN,HDFS, ZK, etc - or a 
manual single node setup.
The MiniLLAP cluster, which starts ZK, HDFS, YARN in proc can be used instead 
for simple testing.




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-15562) LLAP TaskExecutorService race can lead to some fragments being permanently lost

[Siddharth Seth (JIRA)]

Siddharth Seth created HIVE-15562:
-

 Summary: LLAP TaskExecutorService race can lead to some fragments 
being permanently lost
 Key: HIVE-15562
 URL: https://issues.apache.org/jira/browse/HIVE-15562
 Project: Hive
  Issue Type: Bug
Reporter: Siddharth Seth
Assignee: Siddharth Seth
Priority: Critical


A fragment can be evicted before it's submission completed. A race can cause a 
situation As a result, we end up trying to unregister the fragment for 
stateChangeNotifications before it has been registered - leading to an 
exception, which skips the actual fragment kill.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Invitation for Hive committers to become ORC committers

[Siddharth Seth]

Hi Owen,
I'd be interested as well, if not too late.
Thanks,
Sid

On Wed, Jan 4, 2017 at 10:34 AM, Owen O'Malley  wrote:

> Ferd, I've added you.
>
> Suneel, I'm sorry, but the offer is limited to current Hive committers.
> http://people.apache.org/phonebook.html?unix=hive
>
> .. Owen
>
> On Mon, Jan 2, 2017 at 6:39 PM, Suneel Jakka 
> wrote:
>
> > Hi Owen,
> >
> > Am also interested.
> >
> > Regards,
> > Suneel Jakka
> >
> >
> > On Mon, Jan 2, 2017 at 8:19 PM, Xu, Cheng A 
> wrote:
> >
> > > Hi Owen,
> > > Sorry for my late response. I'm also interested.
> > >
> > > Thanks,
> > > Ferd
> > >
> > > -Original Message-
> > > From: Owen O'Malley [mailto:omal...@apache.org]
> > > Sent: Friday, December 23, 2016 11:55 AM
> > > To: dev@hive.apache.org
> > > Subject: Re: Invitation for Hive committers to become ORC committers
> > >
> > > Ok, I believe that I have got everyone. If you don't have karma as
> shown
> > > here: http://people.apache.org/phonebook.html?unix=orc
> > >
> > > Please, let me know. I believe I have also updated the ORC website with
> > > everyone.
> > >
> > > Thanks,
> > >Owen
> > >
> > > On Sat, Dec 17, 2016 at 5:16 AM, Lars Francke 
> > > wrote:
> > >
> > > > Hi Owen,
> > > >
> > > > I'm also interested.
> > > >
> > > > Thanks,
> > > > Lars
> > > >
> > > > On Fri, Dec 16, 2016 at 10:20 PM, Sergio Pena
> > > > 
> > > > wrote:
> > > >
> > > > > Hi Ownen,
> > > > >
> > > > > I'm also interested.
> > > > > - Sergio
> > > > >
> > > > > On Fri, Dec 16, 2016 at 11:39 AM, Daniel Dai <
> da...@hortonworks.com>
> > > > > wrote:
> > > > >
> > > > > > I am interested.
> > > > > >
> > > > > > Thanks,
> > > > > > Daniel
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > On 12/15/16, 1:12 PM, "Owen O'Malley" 
> wrote:
> > > > > >
> > > > > > >All,
> > > > > > >   As you are aware, we are in the last stages of removing the
> > > > > > >forked
> > > > > ORC
> > > > > > >code out of Hive. The goal of moving ORC out of Hive was to
> > > > > > >increase
> > > > its
> > > > > > >community and we want to be very deliberately inclusive of the
> > > > > > >Hive development community. Towards that end, the ORC PMC wants
> > > > > > >to welcome anyone who is already a Hive committer to become a
> > > committer on ORC.
> > > > > > >
> > > > > > >  Please respond on this thread to let us know if you are
> > > interested.
> > > > > > >
> > > > > > >Thanks,
> > > > > > >   Owen on behalf of the ORC PMC
> > > > > >
> > > > >
> > > >
> > >
> >
>

[jira] [Created] (HIVE-15242) LLAP: Act on Node update notifications from registry, fix isAlive checks

[Siddharth Seth (JIRA)]

Siddharth Seth created HIVE-15242:
-

 Summary: LLAP: Act on Node update notifications from registry, fix 
isAlive checks
 Key: HIVE-15242
 URL: https://issues.apache.org/jira/browse/HIVE-15242
 Project: Hive
  Issue Type: Task
Reporter: Siddharth Seth
Assignee: Siddharth Seth


isAlive checks are currently completely broken, since the ZK registry does not 
update existing ServiceIsntances. Instead it creates new instances each time.
This causes non-existant nodes to be used for scheduling in case of node 
failures.

Also, act on the notifications sent by the registry about nodes going down, or 
new nodes being added. (new nodes already handled)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-15219) LLAP: Allow additional slider global parameters to be set while creating the LLAP package

[Siddharth Seth (JIRA)]

Siddharth Seth created HIVE-15219:
-

 Summary: LLAP: Allow additional slider global parameters to be set 
while creating the LLAP package
 Key: HIVE-15219
 URL: https://issues.apache.org/jira/browse/HIVE-15219
 Project: Hive
  Issue Type: Task
  Components: llap
Reporter: Siddharth Seth
Assignee: Siddharth Seth






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Number of retained PreCommit runs

[Siddharth Seth]

Looks like we retain the last 10 runs at max. With tests running in less
than an hour, very often, we end up with situations where test reports are
not available?
I suspect the 10 run limit is due to the size of the logs generated by each
run - close to 6GB (on the ptest server).
I think it will be useful to increase the number of runs - something like
30 or 50. ptest now has an option which allows log collection for failed
tests only. For a test with ~10 failures - the retained log size ended up
being 250MB. Does it make sense to turn on this option, and increase the
number of retained results? (Anyone know how to change the number of
retained runs?)
Thoughts?

Thanks,
Sid

Re: [DISCUSS] Yetus pre-commit tests

[Siddharth Seth]

Peter
In terms of the modules - ignoring the time taken - I would vote for
asflicense, author, findbugs, javac, maybe javadoc, wrhitespace. Not sure
what checkstyle does, and some form of test4tests is already covered in
ptest. This will at least help preventing new issues. Fixing the existing
set would be quite an exercise.

The numbers that you have posted - I think they are on your local system?
I'd expect these to be higher on the build machines. Not too keen on having
the runtime go up by 10+ minutes though. Would this run before ptest is
actually started? Is it possible to start this from within ptest as a
parallel phase? The ptest server doesn't do much while tests are running.
Running the regular ptest flow, and this set of checks could be
parallelized there.

Thank you for taking this up.

Sid

On Thu, Nov 10, 2016 at 7:57 AM, Peter Vary  wrote:

> Hi there,
>
> Previously we discussed that it would be good to integrate some automated
> checks to the pre-commit flow.
> Alan Gates suggested Apache Yetus and I checked what it can do for us
> (Yetus 0.3.0).
>
> The good things that I have found:
>
>- Several existing tests (asflicense, author, checkstyle, findbugs,
>javac, javadoc, test4tests, unitveto, whitespace, xml, junit)
>- It shows changes in errors/failures so we do not have to clean up
>the original code, but new code will be checked.
>- Used by multiple ASF projects already - so we will be Apache conform
>using it.
>- Extensible, so if we decide to add the ptest framework to these test
>this could be done
>- It is possible to run the test only on the modules which contain
>changed files
>
> The bad thing is it could take long time to run the tests even with
> patches touching a single module.
>
> I think we should decide on which test to include into our pre-commit flow
> based on our needs and the resource requirements. For reference I have run
> the test for a fairly small patch on my macbook pro 2 times:
>
>1. Adding 3 new files to the beeline module (1 java, 1 xml, 1 q.out) -
>took ~4 mins - see the result in the attached beeline.out file
>2. Adding 3 new files (same as before) to the ql module (1 java, 1
>xml, 1 q.out) - took ~12 mins - see the result in the attached ql.out file
>
> In nutshell, the out of the box tests which are available in Yetus are
> (the numbers are the time in seconds required to run the test in beeline/ql
> plugin):
>
>- asflicense (24/23) - apache-rat:check - currently this runs for the
>full path
>- author (0/0) - Checks for @author tags
>- checkstyle (31/66) - checkstyle:checksyle
>- findbugs (73/353) - findbugs:findbugs
>- javac (53/147) - install compilation warnings (the runtime presented
>in the tables are not valid)
>- javadoc (34/92) - javadoc warnings
>- test4tests (0/0) - checks if there is any test changed
>- unitveto (0/0) - checks for files in patch
>- whitespace (1/2) - tabs, whitespaces at the end of the line
>- xml (1/1) - xml basic validation
>- junit - running junit tests - we will use ptest anyway, so not
>played with this
>
>
> I would like to know your opinion on which test should we enable, and
> which test should we leave out in our pre-commit workflow.
>
> Thanks,
> Peter
>
>
>
>

[jira] [Created] (HIVE-15165) Flaky test: TestSparkNegativeCliDriver

[Siddharth Seth (JIRA)]

Siddharth Seth created HIVE-15165:
-

 Summary: Flaky test: TestSparkNegativeCliDriver
 Key: HIVE-15165
 URL: https://issues.apache.org/jira/browse/HIVE-15165
 Project: Hive
  Issue Type: Sub-task
Reporter: Siddharth Seth






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-15164) Change default RPC port for llap to be a dynamic port

[Siddharth Seth (JIRA)]

Siddharth Seth created HIVE-15164:
-

 Summary: Change default RPC port for llap to be a dynamic port
 Key: HIVE-15164
 URL: https://issues.apache.org/jira/browse/HIVE-15164
 Project: Hive
  Issue Type: Task
Reporter: Siddharth Seth
Assignee: Siddharth Seth






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-15163) LLAP Finishable state updates may not go to daemons restarted on the same node

[Siddharth Seth (JIRA)]

Siddharth Seth created HIVE-15163:
-

 Summary: LLAP Finishable state updates may not go to daemons 
restarted on the same node
 Key: HIVE-15163
 URL: https://issues.apache.org/jira/browse/HIVE-15163
 Project: Hive
  Issue Type: Bug
Reporter: Siddharth Seth


Finishable state updates only go out to nodes when the source state changes. 
When a node goes down, and comes back up on the same port - updates which had 
been sent to the old node will not be sent to the new node. This could lead to 
a fragment being in a permanent preemptable state, and lead tp starvation on 
busy clusters.




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-15159) LLAP ContainerRunner should not reduce the available heap while distributing to individual executors

[Siddharth Seth (JIRA)]

Siddharth Seth created HIVE-15159:
-

 Summary: LLAP ContainerRunner should not reduce the available heap 
while distributing to individual executors
 Key: HIVE-15159
 URL: https://issues.apache.org/jira/browse/HIVE-15159
 Project: Hive
  Issue Type: Task
  Components: llap
Reporter: Siddharth Seth
Assignee: Siddharth Seth
 Attachments: HIVE-15159.01.patch

hive.llap.daemon.memory.per.instance.mb is set to the heapsize of the process 
when llap service driver is used.
ContainerRunner currently cuts this down to 80% of the available. This 
reduction is not necessary if the heapsize is used, instead of a container size.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-15135) Add an llap mode which fails if queries cannot run in llap

[Siddharth Seth (JIRA)]

Siddharth Seth created HIVE-15135:
-

 Summary: Add an llap mode which fails if queries cannot run in llap
 Key: HIVE-15135
 URL: https://issues.apache.org/jira/browse/HIVE-15135
 Project: Hive
  Issue Type: Task
Reporter: Siddharth Seth
Assignee: Siddharth Seth


ALL currently ends up launching new containers for queries which cannot run in 
llap.
There should be a mode where these queries don't run.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-15122) Hive: Upcasting types should not obscure stats (min/max/ndv)

[Siddharth Seth (JIRA)]

Siddharth Seth created HIVE-15122:
-

 Summary: Hive: Upcasting types should not obscure stats 
(min/max/ndv)
 Key: HIVE-15122
 URL: https://issues.apache.org/jira/browse/HIVE-15122
 Project: Hive
  Issue Type: Bug
Reporter: Siddharth Seth


A UDFToLong breaks PK/FK inferences and triggers mis-estimation of joins in 
LLAP.

Snippet from the bad plan.
{code}
| STAGE PLANS:  

   |
|   Stage: Stage-1  

   |
| Tez   

   |
|   DagId: hive_20161031222730_a700058f-78eb-40d6-a67d-43add60a50e2:6   

   |
|   Edges:  

   |
| Map 2 <- Map 1 (BROADCAST_EDGE)   

   |
| Map 3 <- Map 2 (BROADCAST_EDGE)   

   |
| Reducer 4 <- Map 3 (CUSTOM_SIMPLE_EDGE), Map 7 (CUSTOM_SIMPLE_EDGE), 
Map 8 (BROADCAST_EDGE), Map 9 (BROADCAST_EDGE)  
|
| Reducer 5 <- Reducer 4 (SIMPLE_EDGE)  

   |
| Reducer 6 <- Reducer 5 (SIMPLE_EDGE)  

   |
|   DagName:

   |
|   Vertices:   

   |
| Map 1 

   |
| Map Operator Tree:

   |
| TableScan 

   |
|   alias: supplier 

   |
|   filterExpr: (s_suppkey is not null and s_nationkey is not 
null) (type: boolean)   
 |
|   Statistics: Num rows: 1000 Data size: 16000 Basic 
stats: COMPLETE Column stats: COMPLETE  
 |
|   Filter Operator 

   |
| predicate: (s_suppkey is not null and s_nationkey is not 
null) (type: boolean)   
|
| Statistics: Num rows: 1000 Data size: 16000 Basic 
stats: COMPLETE Column stats: COMPLETE  
   |
| Select Operator   

   |
|   expressions: s_suppkey (type: bigint), s_nationkey 
(type: bigint)  
|
|   outputColumnNames: _col0, _col1 

   |
|   Statistics: Num rows: 1000 Data size: 16000 
Basic stats: COMPLETE Column stats: COMPLETE
   |
|   Reduce 

Re: build bot on master

[Siddharth Seth]

Restarted the ptest server. That normally fixes this error.

On Fri, Oct 28, 2016 at 10:33 AM, Eugene Koifman 
wrote:

> Hi,
> all test runs are failing with the error below.   Does anyone know how to
> fix it?
>
> Thanks,
> Eugene
>
> from https://builds.apache.org/view/H-L/view/Hive/job/
> PreCommit-HIVE-Build/1866/console
>
>
>
>
> Build/hive/build/hive/testutils/ptest2/target/lib/
> guice-assistedinject-3.0.jar
> [INFO] Copying velocity-1.7.jar to /home/jenkins/jenkins-slave/
> workspace/PreCommit-HIVE-Build/hive/build/hive/
> testutils/ptest2/target/lib/velocity-1.7.jar
> [INFO] 
> 
> [INFO] BUILD SUCCESS
> [INFO] 
> 
> [INFO] Total time: 10.833 s
> [INFO] Finished at: 2016-10-28T15:57:26+00:00
> [INFO] Final Memory: 29M/723M
> [INFO] 
> 
> + local 'PTEST_CLASSPATH=/home/jenkins/jenkins-slave/
> workspace/PreCommit-HIVE-Build/hive/build/hive/
> testutils/ptest2/target/hive-ptest-1.0-classes.jar:/home/
> jenkins/jenkins-slave/workspace/PreCommit-HIVE-Build/hive/build/hive/
> testutils/ptest2/target/lib/*'
> + java -cp '/home/jenkins/jenkins-slave/workspace/PreCommit-HIVE-
> Build/hive/build/hive/testutils/ptest2/target/hive-
> ptest-1.0-classes.jar:/home/jenkins/jenkins-slave/
> workspace/PreCommit-HIVE-Build/hive/build/hive/testutils/ptest2/target/lib/*'
> org.apache.hive.ptest.api.client.PTestClient --command testStart
> --outputDir /home/jenkins/jenkins-slave/workspace/PreCommit-HIVE-
> Build/hive/build/hive/testutils/ptest2/target --password ''
> --testHandle PreCommit-HIVE-Build-1866 --endpoint
> http://104.198.109.242:8080/hive-ptest-1.0 --logsEndpoint
> http://104.198.109.242/logs/ --profile master-mr2 --patch
> https://issues.apache.org/jira/secure/attachment/
> 12835837/HIVE-15093.1.patch --jira HIVE-15093
>
>
> Logs are located: http://104.198.109.242/logs/PreCommit-HIVE-Build-1866
>
>
> Exception in thread "main" java.lang.RuntimeException: 404 Not Found
> at org.apache.hive.ptest.api.client.PTestClient.
> downloadTestResults(PTestClient.java:189)
> at org.apache.hive.ptest.api.client.PTestClient.testStart(
> PTestClient.java:137)
> at org.apache.hive.ptest.api.client.PTestClient.main(
> PTestClient.java:320)
> + ret=1
> + unpack_test_results
> + '[' -z /home/jenkins/jenkins-slave/workspace/PreCommit-HIVE-Build/hive/build
> ']'
> + cd /home/jenkins/jenkins-slave/workspace/PreCommit-HIVE-
> Build/hive/build/hive/testutils/ptest2/target
> + [[ -f test-results.tar.gz ]]
> + exit 1
> + rm -f /tmp/tmp.r59JTiDlN1
> Build step 'Execute shell' marked build as failure
> Recording test results
> ERROR: Step ?Publish JUnit test result report? failed: Test reports were
> found but none of them are new. Did tests run?
> For example, /home/jenkins/jenkins-slave/workspace/PreCommit-HIVE-
> Build/hive/build/test-results/TEST-1-TestCliDriver-udf_
> upper.q-ctas_date.q-schema_evol_orc_acidvec_table_update.
> q-and-27-more-TEST-org.apache.hadoop.hive.cli.TestCliDriver.xml is 2 days
> 19 hr old
>
> [description-setter] Description set ci.org/search?query=[description-setter]%20Description%20set>:
> HIVE-15093  /   master-mr2
> Finished: FAILURE
>

[jira] [Created] (HIVE-15085) Reduce the memory used by unit tests, MiniCliDriver, MiniLlapLocal, MiniSpark

[Siddharth Seth (JIRA)]

Siddharth Seth created HIVE-15085:
-

 Summary: Reduce the memory used by unit tests, MiniCliDriver, 
MiniLlapLocal, MiniSpark
 Key: HIVE-15085
 URL: https://issues.apache.org/jira/browse/HIVE-15085
 Project: Hive
  Issue Type: Sub-task
Reporter: Siddharth Seth
Assignee: Siddharth Seth






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-15084) Flaky test: TestMiniTezCliDriver:explainanalyze_2

[Siddharth Seth (JIRA)]

Siddharth Seth created HIVE-15084:
-

 Summary: Flaky test: TestMiniTezCliDriver:explainanalyze_2
 Key: HIVE-15084
 URL: https://issues.apache.org/jira/browse/HIVE-15084
 Project: Hive
  Issue Type: Sub-task
Reporter: Siddharth Seth






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [DISCUSS] Pre-commit tests before commits

[Siddharth Seth]

Added the following.
https://cwiki.apache.org/confluence/display/Hive/HowToCommit#HowToCommit-PreCommitruns,andcommittingpatches

On Tue, Oct 25, 2016 at 12:03 AM, Lefty Leverenz 
wrote:

> >
> > @Prasanth - separate page on the wiki for 'Adding / modifying tests'.
> That can
> > be discussed on the list, or modified directly.
>
>
> Here's what the wiki has so far:
>
>- Tips for Adding New Tests
><https://cwiki.apache.org/confluence/display/Hive/TipsForAddingNewTests
> >
> (written
>in 2011)
>- Developer Guide:  Adding new unit tests
><https://cwiki.apache.org/confluence/display/Hive/
> DeveloperGuide#DeveloperGuide-Addingnewunittests>
> (outdated
>instructions)
>
> -- Lefty
>
>
> On Mon, Oct 24, 2016 at 4:49 PM, Siddharth Seth  wrote:
>
> > There seems to be general consensus to follow this approach for commits,
> > and waiting for pre-commit tests. I'm going to add this to the Hive Wiki.
> >
> > @Alan, On using Yetus - I don't really know much about it, so cannot
> > comment. However, Hive does use a parallel test framework which generates
> > reports, compiles hive via custom scripts, etc - I suspect that will
> > complicate integration with a third party tool. Any one want to volunteer
> > for the findbugs, javac error integration?
> >
> > @Prasanth - separate page on the wiki for 'Adding / modifying tests'.
> That
> > can be discussed on the list, or modified directly.
> >
> > Thanks,
> > Sid
> >
> > On Mon, Oct 17, 2016 at 12:07 PM, Sergey Shelukhin <
> ser...@hortonworks.com
> > >
> > wrote:
> >
> > > +1
> > >
> > > On 16/10/13, 23:10, "Siddharth Seth"  wrote:
> > >
> > > >There's been a lot of work to make the test runs faster, as well as
> more
> > > >reliable via HIVE-14547, HIVE-13503, and several other jiras. Test
> > > >runtimes
> > > >are around the 1 hour mark, and going down. There were a few green
> > > >pre-commit runs (after years?). At the same time, there's still some
> > flaky
> > > >tests.
> > > >
> > > >We really should try to keep the test runtimes down, as well as the
> > number
> > > >of failures - so that the pre-commit runs can provide useful
> > information.
> > > >
> > > >I'm not sure what the current approach w.r.t precommit runs before a
> > > >commit. What I've seen in other projects is that the pre-commit needs
> to
> > > >run, and come back clean (mostly) before a commit goes in. Between
> what
> > > >used to be 5 day wait times, and inconsistent runs - I don't think
> this
> > is
> > > >always followed in Hive.
> > > >
> > > >It'll be useful to start relying on pre-commit test results again.
> Given
> > > >the flaky tests, I'd suggest the following
> > > >1. Pre-commit must be run on a patch before committing (with very few
> > > >exceptions)
> > > >2. A green test run is ideal
> > > >3. In case there are failures - keep track of these as sub-jiras
> under a
> > > >flaky test umbrella jira (Some under HIVE-14547 already) - to be
> > > >eventually
> > > >fixed.
> > > >4. Before committing - cite relevant jiras for a flaky test (create
> and
> > > >cite if it doesn't already exist).
> > > >
> > > >This should help us build up a list of flaky tests over various runs,
> > > >which
> > > >will hopefully get fixed at some point.
> > > >
> > > >Thoughts?
> > > >
> > > >Thanks,
> > > >Sid
> > >
> > >
> >
>

[jira] [Created] (HIVE-15058) [Umbrella] Current Flaky Tests

[Siddharth Seth (JIRA)]

Siddharth Seth created HIVE-15058:
-

 Summary: [Umbrella] Current Flaky Tests
 Key: HIVE-15058
 URL: https://issues.apache.org/jira/browse/HIVE-15058
 Project: Hive
  Issue Type: Task
Reporter: Siddharth Seth


Jira to track known flaky tests, to make it easier to cite jiras before 
committing.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [DISCUSS] Pre-commit tests before commits

[Siddharth Seth]

There seems to be general consensus to follow this approach for commits,
and waiting for pre-commit tests. I'm going to add this to the Hive Wiki.

@Alan, On using Yetus - I don't really know much about it, so cannot
comment. However, Hive does use a parallel test framework which generates
reports, compiles hive via custom scripts, etc - I suspect that will
complicate integration with a third party tool. Any one want to volunteer
for the findbugs, javac error integration?

@Prasanth - separate page on the wiki for 'Adding / modifying tests'. That
can be discussed on the list, or modified directly.

Thanks,
Sid

On Mon, Oct 17, 2016 at 12:07 PM, Sergey Shelukhin 
wrote:

> +1
>
> On 16/10/13, 23:10, "Siddharth Seth"  wrote:
>
> >There's been a lot of work to make the test runs faster, as well as more
> >reliable via HIVE-14547, HIVE-13503, and several other jiras. Test
> >runtimes
> >are around the 1 hour mark, and going down. There were a few green
> >pre-commit runs (after years?). At the same time, there's still some flaky
> >tests.
> >
> >We really should try to keep the test runtimes down, as well as the number
> >of failures - so that the pre-commit runs can provide useful information.
> >
> >I'm not sure what the current approach w.r.t precommit runs before a
> >commit. What I've seen in other projects is that the pre-commit needs to
> >run, and come back clean (mostly) before a commit goes in. Between what
> >used to be 5 day wait times, and inconsistent runs - I don't think this is
> >always followed in Hive.
> >
> >It'll be useful to start relying on pre-commit test results again. Given
> >the flaky tests, I'd suggest the following
> >1. Pre-commit must be run on a patch before committing (with very few
> >exceptions)
> >2. A green test run is ideal
> >3. In case there are failures - keep track of these as sub-jiras under a
> >flaky test umbrella jira (Some under HIVE-14547 already) - to be
> >eventually
> >fixed.
> >4. Before committing - cite relevant jiras for a flaky test (create and
> >cite if it doesn't already exist).
> >
> >This should help us build up a list of flaky tests over various runs,
> >which
> >will hopefully get fixed at some point.
> >
> >Thoughts?
> >
> >Thanks,
> >Sid
>
>

[jira] [Created] (HIVE-15013) Config dir generated for tests should not be under the test tmp directory

[Siddharth Seth (JIRA)]

Siddharth Seth created HIVE-15013:
-

 Summary: Config dir generated for tests should not be under the 
test tmp directory
 Key: HIVE-15013
 URL: https://issues.apache.org/jira/browse/HIVE-15013
 Project: Hive
  Issue Type: Sub-task
Reporter: Siddharth Seth
Assignee: Siddharth Seth


mvn is used to clean up tmp directories created for tests, and to setup the 
config directory. The current structure is 
target/tmp
target/tmp/config

All of this is setup when mvn test is executed.

Tests generate data under tmp - warehouse, metastore, etc. Having the conf dir 
there (generated by mvn) makes it complicate to add per test cleanup - since 
the entire tmp directory cannot be removed.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-15009) Additional cleanup from previous test runs when running a batch in ptest

[Siddharth Seth (JIRA)]

Siddharth Seth created HIVE-15009:
-

 Summary: Additional cleanup from previous test runs when running a 
batch in ptest
 Key: HIVE-15009
 URL: https://issues.apache.org/jira/browse/HIVE-15009
 Project: Hive
  Issue Type: Sub-task
Reporter: Siddharth Seth
Assignee: Siddharth Seth






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-15008) Cleanup before/after test execution - TestJdbcWithSQLAuthorization, TestJdbcWithSQLAuthUDFBlacklist

[Siddharth Seth (JIRA)]

Siddharth Seth created HIVE-15008:
-

 Summary: Cleanup before/after test execution - 
TestJdbcWithSQLAuthorization, TestJdbcWithSQLAuthUDFBlacklist
 Key: HIVE-15008
 URL: https://issues.apache.org/jira/browse/HIVE-15008
 Project: Hive
  Issue Type: Sub-task
Reporter: Siddharth Seth






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-15006) Flaky test: TestBeelineWithHS2ConnectionFile

[Siddharth Seth (JIRA)]

Siddharth Seth created HIVE-15006:
-

 Summary: Flaky test: TestBeelineWithHS2ConnectionFile
 Key: HIVE-15006
 URL: https://issues.apache.org/jira/browse/HIVE-15006
 Project: Hive
  Issue Type: Sub-task
Reporter: Siddharth Seth


Seems to time out fairly often.

https://issues.apache.org/jira/browse/HIVE-14391, 
https://builds.apache.org/job/PreCommit-HIVE-Build/1621/testReport

https://issues.apache.org/jira/browse/HIVE-14887, 
https://builds.apache.org/job/PreCommit-HIVE-Build/1606/testReport



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [DISCUSS] Pre-commit tests before commits

[Siddharth Seth]

hout initialization time) under a minute.
> >>
> >> 6) Avoid adding more read-only tables to init script as this will
> increase
> >> the initialization time.
> >>
> >> 7) If the test case does not require explain plan then avoid it as most
> >> failures are explain diffs.
> >>
> >> 8) If the test case requires explain and if it does not depend on table
> or
> >> partition stats explicitly set stats for the table or partition.
> >> Explicitly setting stats will avoid expensive stats computation time and
> >> avoids flakiness due to stats diff.
> >>
> >> 9) Prefer jUnit over qtest.
> >>
> >> 10) Add explicitly timeout for jUnit test to avoid indefinite hanging of
> >> tests (surefire timeouts after 40 mins)
> >>
> >> Thoughts?
> >>
> >> Thanks
> >> Prasanth
> >>
> >> On Oct 13, 2016, at 11:10 PM, Siddharth Seth  >> ss...@apache.org>> wrote:
> >>
> >> There's been a lot of work to make the test runs faster, as well as more
> >> reliable via HIVE-14547, HIVE-13503, and several other jiras. Test
> runtimes
> >> are around the 1 hour mark, and going down. There were a few green
> >> pre-commit runs (after years?). At the same time, there's still some
> flaky
> >> tests.
> >>
> >> We really should try to keep the test runtimes down, as well as the
> number
> >> of failures - so that the pre-commit runs can provide useful
> information.
> >>
> >> I'm not sure what the current approach w.r.t precommit runs before a
> >> commit. What I've seen in other projects is that the pre-commit needs to
> >> run, and come back clean (mostly) before a commit goes in. Between what
> >> used to be 5 day wait times, and inconsistent runs - I don't think this
> is
> >> always followed in Hive.
> >>
> >> It'll be useful to start relying on pre-commit test results again. Given
> >> the flaky tests, I'd suggest the following
> >> 1. Pre-commit must be run on a patch before committing (with very few
> >> exceptions)
> >> 2. A green test run is ideal
> >> 3. In case there are failures - keep track of these as sub-jiras under a
> >> flaky test umbrella jira (Some under HIVE-14547 already) - to be
> eventually
> >> fixed.
> >> 4. Before committing - cite relevant jiras for a flaky test (create and
> >> cite if it doesn't already exist).
> >>
> >> This should help us build up a list of flaky tests over various runs,
> which
> >> will hopefully get fixed at some point.
> >>
> >> Thoughts?
> >>
> >> Thanks,
> >> Sid
> >>
> >>
>
>

[jira] [Created] (HIVE-14958) Improve the 'TestClass' did not produce a TEST-*.xml file message to include list of all qfiles in a batch, batch id

[Siddharth Seth (JIRA)]

Siddharth Seth created HIVE-14958:
-

 Summary: Improve the 'TestClass' did not produce a TEST-*.xml file 
message to include list of all qfiles in a batch, batch id
 Key: HIVE-14958
 URL: https://issues.apache.org/jira/browse/HIVE-14958
 Project: Hive
  Issue Type: Sub-task
Reporter: Siddharth Seth
Assignee: Siddharth Seth


Should make it easier to hunt down the logs.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[DISCUSS] Pre-commit tests before commits

[Siddharth Seth]

There's been a lot of work to make the test runs faster, as well as more
reliable via HIVE-14547, HIVE-13503, and several other jiras. Test runtimes
are around the 1 hour mark, and going down. There were a few green
pre-commit runs (after years?). At the same time, there's still some flaky
tests.

We really should try to keep the test runtimes down, as well as the number
of failures - so that the pre-commit runs can provide useful information.

I'm not sure what the current approach w.r.t precommit runs before a
commit. What I've seen in other projects is that the pre-commit needs to
run, and come back clean (mostly) before a commit goes in. Between what
used to be 5 day wait times, and inconsistent runs - I don't think this is
always followed in Hive.

It'll be useful to start relying on pre-commit test results again. Given
the flaky tests, I'd suggest the following
1. Pre-commit must be run on a patch before committing (with very few
exceptions)
2. A green test run is ideal
3. In case there are failures - keep track of these as sub-jiras under a
flaky test umbrella jira (Some under HIVE-14547 already) - to be eventually
fixed.
4. Before committing - cite relevant jiras for a flaky test (create and
cite if it doesn't already exist).

This should help us build up a list of flaky tests over various runs, which
will hopefully get fixed at some point.

Thoughts?

Thanks,
Sid

[jira] [Created] (HIVE-14938) Add deployed ptest properties file to repo, update to remove isolated tests

[Siddharth Seth (JIRA)]

Siddharth Seth created HIVE-14938:
-

 Summary: Add deployed ptest properties file to repo, update to 
remove isolated tests
 Key: HIVE-14938
 URL: https://issues.apache.org/jira/browse/HIVE-14938
 Project: Hive
  Issue Type: Task
Reporter: Siddharth Seth
Assignee: Siddharth Seth


The intent is to checkin the original file, and then modify it to remove 
isolated tests (and move relevant ones to the skipBatching list), which 
normally lead to stragglers, and sub-optimal resource utilization.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-14937) Flaky test: TestHBaseCliDriver.testCliDriver[hbase_bulk]

[Siddharth Seth (JIRA)]

Siddharth Seth created HIVE-14937:
-

 Summary: Flaky test: TestHBaseCliDriver.testCliDriver[hbase_bulk]
 Key: HIVE-14937
 URL: https://issues.apache.org/jira/browse/HIVE-14937
 Project: Hive
  Issue Type: Sub-task
Reporter: Siddharth Seth


https://builds.apache.org/job/PreCommit-HIVE-Build/1499/testReport/junit/org.apache.hadoop.hive.cli/TestHBaseCliDriver/testCliDriver_hbase_bulk_/

Has failed on several runs. This test was merged with hbaseCliDriver tests. 
Need to see why it has turned flaky, or whether it was always flaky.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-14936) Flaky test: TestMiniLlapCliDriver.testCliDriver[orc_ppd_schema_evol_3a]

[Siddharth Seth (JIRA)]

Siddharth Seth created HIVE-14936:
-

 Summary: Flaky test: 
TestMiniLlapCliDriver.testCliDriver[orc_ppd_schema_evol_3a]
 Key: HIVE-14936
 URL: https://issues.apache.org/jira/browse/HIVE-14936
 Project: Hive
  Issue Type: Sub-task
Reporter: Siddharth Seth


https://builds.apache.org/job/PreCommit-HIVE-Build/1489/testReport/org.apache.hadoop.hive.cli/TestMiniLlapCliDriver/testCliDriver_orc_ppd_schema_evol_3a_/
{code}
224c224
<HDFS_BYTES_READ: 16384
---
>HDFS_BYTES_READ: 17046
226c226
<HDFS_READ_OPS: 4
---
>HDFS_READ_OPS: 6
{code}

Have seen this diff fairly often.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-14934) Flaky test: TestHCatMultiOutputFormat

[Siddharth Seth (JIRA)]

Siddharth Seth created HIVE-14934:
-

 Summary: Flaky test: TestHCatMultiOutputFormat
 Key: HIVE-14934
 URL: https://issues.apache.org/jira/browse/HIVE-14934
 Project: Hive
  Issue Type: Sub-task
Reporter: Siddharth Seth


https://builds.apache.org/job/PreCommit-HIVE-Build/lastCompletedBuild/testReport/

{code}
org.apache.hadoop.yarn.exceptions.YarnRuntimeException: Failed to intialize 
existing directories
org.apache.hadoop.yarn.exceptions.YarnRuntimeException: Failed to intialize 
existing directories
at 
org.apache.hadoop.mapreduce.v2.hs.JobHistory.serviceInit(JobHistory.java:98) 
~[hadoop-mapreduce-client-hs-2.7.2.jar:?]
at 
org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) 
[hadoop-common-2.7.2.jar:?]
at 
org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:107)
 [hadoop-common-2.7.2.jar:?]
at 
org.apache.hadoop.mapreduce.v2.hs.JobHistoryServer.serviceInit(JobHistoryServer.java:143)
 [hadoop-mapreduce-client-hs-2.7.2.jar:?]
at 
org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) 
[hadoop-common-2.7.2.jar:?]
at 
org.apache.hadoop.mapreduce.v2.MiniMRYarnCluster$JobHistoryServerWrapper.serviceStart(MiniMRYarnCluster.java:211)
 [hadoop-mapreduce-client-jobclient-2.7.2-tests.jar:?]
at 
org.apache.hadoop.service.AbstractService.start(AbstractService.java:193) 
[hadoop-common-2.7.2.jar:?]
at 
org.apache.hadoop.service.CompositeService.serviceStart(CompositeService.java:120)
 [hadoop-common-2.7.2.jar:?]
at 
org.apache.hadoop.service.AbstractService.start(AbstractService.java:193) 
[hadoop-common-2.7.2.jar:?]
at 
org.apache.hadoop.mapred.MiniMRClientClusterFactory.create(MiniMRClientClusterFactory.java:80)
 [hadoop-mapreduce-client-jobclient-2.7.2-tests.jar:?]
at 
org.apache.hadoop.mapred.MiniMRCluster.(MiniMRCluster.java:187) 
[hadoop-mapreduce-client-jobclient-2.7.2-tests.jar:?]
at 
org.apache.hadoop.mapred.MiniMRCluster.(MiniMRCluster.java:175) 
[hadoop-mapreduce-client-jobclient-2.7.2-tests.jar:?]
at 
org.apache.hadoop.mapred.MiniMRCluster.(MiniMRCluster.java:167) 
[hadoop-mapreduce-client-jobclient-2.7.2-tests.jar:?]
at 
org.apache.hadoop.mapred.MiniMRCluster.(MiniMRCluster.java:128) 
[hadoop-mapreduce-client-jobclient-2.7.2-tests.jar:?]
at 
org.apache.hive.hcatalog.mapreduce.TestHCatMultiOutputFormat.setup(TestHCatMultiOutputFormat.java:173)
 [test-classes/:?]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 
~[?:1.8.0_25]
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) 
~[?:1.8.0_25]
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 ~[?:1.8.0_25]
at java.lang.reflect.Method.invoke(Method.java:483) ~[?:1.8.0_25]
at 
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
 [junit-4.11.jar:?]
at 
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
 [junit-4.11.jar:?]
at 
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
 [junit-4.11.jar:?]
at 
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:24) 
[junit-4.11.jar:?]
at 
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) 
[junit-4.11.jar:?]
at org.junit.runners.ParentRunner.run(ParentRunner.java:309) 
[junit-4.11.jar:?]
at 
org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:367)
 [surefire-junit4-2.19.1.jar:2.19.1]
at 
org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:274)
 [surefire-junit4-2.19.1.jar:2.19.1]
at 
org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
 [surefire-junit4-2.19.1.jar:2.19.1]
at 
org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:161) 
[surefire-junit4-2.19.1.jar:2.19.1]
at 
org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:290)
 [surefire-booter-2.19.1.jar:2.19.1]
at 
org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:242)
 [surefire-booter-2.19.1.jar:2.19.1]
at 
org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:121) 
[surefire-booter-2.19.1.jar:2.19.1]
Caused by: java.io.FileNotFoundException: File 
file:/tmp/hadoop-yarn/staging/history/done does not exist
at 
org.apache.hadoop.fs.RawLocalFileSystem.listStatus(RawLocalFileSystem.java:429) 
~[hadoop-common-2.7.2.jar:?]
at 
org.apache.hadoop.fs.DelegateToFileSystem.listStatus(DelegateToFileSystem.java:168)
 ~[hadoop-common-2.7.2.jar:?]
at org.apache.hadoop.fs.ChecksumFs.listStatus

[jira] [Created] (HIVE-14915) Add an option to skip log collection for successful tests

[Siddharth Seth (JIRA)]

Siddharth Seth created HIVE-14915:
-

 Summary: Add an option to skip log collection for successful tests
 Key: HIVE-14915
 URL: https://issues.apache.org/jira/browse/HIVE-14915
 Project: Hive
  Issue Type: Sub-task
  Components: Testing Infrastructure
Reporter: Siddharth Seth
Assignee: Siddharth Seth


We generate multiple gigs of tests at the moment. An option to skip log 
collection for successful tests could be useful.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-14914) Improve the 'TestClass' did not produce a TEST-*.xml file message

[Siddharth Seth (JIRA)]

Siddharth Seth created HIVE-14914:
-

 Summary: Improve the 'TestClass' did not produce a TEST-*.xml file 
message
 Key: HIVE-14914
 URL: https://issues.apache.org/jira/browse/HIVE-14914
 Project: Hive
  Issue Type: Sub-task
Reporter: Siddharth Seth
Assignee: Siddharth Seth


For timed out unit test batches - this report may not be generated correctly.
Also, there's no differentiation between 0 tests in a batch vs an actual 
missing report.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-14910) Flaky test: TestSparkClient.testJobSubmission

[Siddharth Seth (JIRA)]

Siddharth Seth created HIVE-14910:
-

 Summary: Flaky test: TestSparkClient.testJobSubmission
 Key: HIVE-14910
 URL: https://issues.apache.org/jira/browse/HIVE-14910
 Project: Hive
  Issue Type: Sub-task
Reporter: Siddharth Seth


Have seen this fail in multiple runs (not consistently)

e.g. https://builds.apache.org/job/PreCommit-HIVE-Build/1426/testReport/



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-14887) Reduce the memory requirements for tests

[Siddharth Seth (JIRA)]

Siddharth Seth created HIVE-14887:
-

 Summary: Reduce the memory requirements for tests
 Key: HIVE-14887
 URL: https://issues.apache.org/jira/browse/HIVE-14887
 Project: Hive
  Issue Type: Sub-task
Reporter: Siddharth Seth
Assignee: Siddharth Seth


The clusters that we spin up end up requiring 16GB at times. Also the maven 
arguments seem a little heavy weight.
Reducing this will allow for additional ptest drones per box, which should 
bring down the runtime.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-14860) Flaky test: TestMiniLlapCliDriver acid_bucket_pruning.q

[Siddharth Seth (JIRA)]

Siddharth Seth created HIVE-14860:
-

 Summary: Flaky test: TestMiniLlapCliDriver acid_bucket_pruning.q
 Key: HIVE-14860
 URL: https://issues.apache.org/jira/browse/HIVE-14860
 Project: Hive
  Issue Type: Sub-task
Reporter: Siddharth Seth


Succeeds when run independently.

Fails when run in the following batch

-Dqfile=acid_globallimit.q,acid_bucket_pruning.q,mapreduce2.q,import_exported_table.q,tez_union_dynamic_partition.q,file_with_header_footer.q,insert_into1.q,schemeAuthority.q,dynamic_partition_pruning_2.q,orc_merge2.q,insert_into2.q,script_env_var1.q,load_fs2.q,llapdecider.q,remote_script.q



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-14854) Add a core cluster type to QTestUtil

[Siddharth Seth (JIRA)]

Siddharth Seth created HIVE-14854:
-

 Summary: Add a core cluster type to QTestUtil
 Key: HIVE-14854
 URL: https://issues.apache.org/jira/browse/HIVE-14854
 Project: Hive
  Issue Type: Sub-task
Reporter: Siddharth Seth
Assignee: Siddharth Seth


Follow up to HIVE-14824. There's tez, tez_local, llap, llap_local - all of 
which are of a single type, similaryl spark, sparkOnYarn, and none,mr. 
Introducing a core cluster type to make a bunch of conditional checks simpler.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-14852) Change qtest logging to not redirect all logs to console

[Siddharth Seth (JIRA)]

Siddharth Seth created HIVE-14852:
-

 Summary: Change qtest logging to not redirect all logs to console
 Key: HIVE-14852
 URL: https://issues.apache.org/jira/browse/HIVE-14852
 Project: Hive
  Issue Type: Sub-task
Reporter: Siddharth Seth
 Attachments: HIVE-14852.01.patch

A change was made recently to redirect all logs to console, to make IDE 
debugging of regular tests easier. That unfortunately makes qtest debugging 
tougher - since there's a lot of noise along with the diffs in the output file.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-14834) Reduce the retry attempts for HiveServer startup

[Siddharth Seth (JIRA)]

Siddharth Seth created HIVE-14834:
-

 Summary: Reduce the retry attempts for HiveServer startup
 Key: HIVE-14834
 URL: https://issues.apache.org/jira/browse/HIVE-14834
 Project: Hive
  Issue Type: Task
Reporter: Siddharth Seth


Currently, 30 attempts with a 1 minute sleep in between - 30 minutes.

That seems a little too much. Early feedback (and failure) seems like a better 
approach.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-14830) Move a majority of the MiniLlapCliDriver tests to use an inline AM

[Siddharth Seth (JIRA)]

Siddharth Seth created HIVE-14830:
-

 Summary: Move a majority of the MiniLlapCliDriver tests to use an 
inline AM
 Key: HIVE-14830
 URL: https://issues.apache.org/jira/browse/HIVE-14830
 Project: Hive
  Issue Type: Sub-task
Reporter: Siddharth Seth
Assignee: Siddharth Seth






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-14824) Separate fstype from cluster type in QTestUtil

[Siddharth Seth (JIRA)]

Siddharth Seth created HIVE-14824:
-

 Summary: Separate fstype from cluster type in QTestUtil
 Key: HIVE-14824
 URL: https://issues.apache.org/jira/browse/HIVE-14824
 Project: Hive
  Issue Type: Sub-task
Reporter: Siddharth Seth
Assignee: Siddharth Seth


QTestUtil cluster type encodes the file system. e.g. MiniClusterType.encrypted 
means mr + encrypted hdfs, spark means file://, mr means hdfs etc.
These can be separated out. e.g. To add tests for tez against encrypted, and 
llap against encrypted - I'd need to introduce 2 new cluster types.

Instead it's better to separate the storage into it's own types.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-14818) Reduce number of retries while starting HiveServer for tests

[Siddharth Seth (JIRA)]

Siddharth Seth created HIVE-14818:
-

 Summary: Reduce number of retries while starting HiveServer for 
tests
 Key: HIVE-14818
 URL: https://issues.apache.org/jira/browse/HIVE-14818
 Project: Hive
  Issue Type: Sub-task
Reporter: Siddharth Seth
Assignee: Siddharth Seth


Current is 30 retries, with a 1minute sleep between each one.

The settings are likely bad for a production cluster as well. For tests, this 
should be a lot lower.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-14817) Shutdown the SessionManager timeoutChecker thread properly upon shutdown

[Siddharth Seth (JIRA)]

Siddharth Seth created HIVE-14817:
-

 Summary: Shutdown the SessionManager timeoutChecker thread 
properly upon shutdown
 Key: HIVE-14817
 URL: https://issues.apache.org/jira/browse/HIVE-14817
 Project: Hive
  Issue Type: Task
Reporter: Siddharth Seth
Assignee: Siddharth Seth


Shutdown for SessionManager waits 10seconds for all threads on the 
threadpoolExecutor to shutdown correctly.
The cleaner thread - with default settings - will take 6 hours to shutdown, so 
essentially any shutdown of HS2 is always delayed by 10s.

The cleaner thread should be shutdown properly.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-14812) Failing test: TestCliDriver acid_mapjoin

[Siddharth Seth (JIRA)]

Siddharth Seth created HIVE-14812:
-

 Summary: Failing test: TestCliDriver acid_mapjoin
 Key: HIVE-14812
 URL: https://issues.apache.org/jira/browse/HIVE-14812
 Project: Hive
  Issue Type: Sub-task
Reporter: Siddharth Seth






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-14811) Failing test: TestCliDriver ctas

[Siddharth Seth (JIRA)]

Siddharth Seth created HIVE-14811:
-

 Summary: Failing test: TestCliDriver ctas
 Key: HIVE-14811
 URL: https://issues.apache.org/jira/browse/HIVE-14811
 Project: Hive
  Issue Type: Sub-task
Reporter: Siddharth Seth






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-14810) Failing test: TestMetaStoreMetrics.testMetaDataCounts

[Siddharth Seth (JIRA)]

Siddharth Seth created HIVE-14810:
-

 Summary: Failing test: TestMetaStoreMetrics.testMetaDataCounts
 Key: HIVE-14810
 URL: https://issues.apache.org/jira/browse/HIVE-14810
 Project: Hive
  Issue Type: Sub-task
Reporter: Siddharth Seth






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-14800) Handle off by 3 in ORC split generation based on split strategy used

[Siddharth Seth (JIRA)]

Siddharth Seth created HIVE-14800:
-

 Summary: Handle off by 3 in ORC split generation based on split 
strategy used
 Key: HIVE-14800
 URL: https://issues.apache.org/jira/browse/HIVE-14800
 Project: Hive
  Issue Type: Bug
Reporter: Siddharth Seth


BI will apparently generate splits starting at offset 0.
ETL will skip the ORC header and generate a split starting at offset 3.

There's a workaround in the HiveSplitGenreator to handle this for consistent 
splits. Ideally, Orc split generation should take care of this.


cc [~prasanth_j], [~gopalv]



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-14793) Allow ptest branch to be specified, PROFILE override

[Siddharth Seth (JIRA)]

Siddharth Seth created HIVE-14793:
-

 Summary: Allow ptest branch to be specified, PROFILE override
 Key: HIVE-14793
 URL: https://issues.apache.org/jira/browse/HIVE-14793
 Project: Hive
  Issue Type: Sub-task
Reporter: Siddharth Seth


Post HIVE-14734 - the profile is automatically determined. Add an option to 
override this via Jenkins. Also add an option to specify the branch from which 
ptest is built (This is hardcoded to github.com/apache/hive)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-14781) ptest killall command does not work

[Siddharth Seth (JIRA)]

Siddharth Seth created HIVE-14781:
-

 Summary: ptest killall command does not work
 Key: HIVE-14781
 URL: https://issues.apache.org/jira/browse/HIVE-14781
 Project: Hive
  Issue Type: Sub-task
Reporter: Siddharth Seth
Assignee: Siddharth Seth


killall -f is not a valid flag.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

ptest result structure change (batched unit tests)

[Siddharth Seth]

HIVE-14540 changes ptest to batch unit tests together (instead of invoking
mvn test on individual test classes).
As a result, output for multiple tests is a common directory - similar to
what happens for batched q tests.

To identify the relevant directory, lookup the consoleOuput for the run.

e.g.
To find TestDummy
Look for UTBatch.*TestDummy in consoleOutput.

example output.
[name=829_UTBatch_itests__qtest_8_tests, id=829, moduleName=itests/qtest,
isParallel=true, testList=[TestContribNegativeCliDriver,
TestHBaseNegativeCliDriver, TestCompareCliDriver,
TestEncryptedHDFSCliDriver, TestPerfCliDriver, TestContribCliDriver,
TestParseNegativeDriver, TestDummy]]

829_UTBatch would be the directory where the output for this test exists.

[jira] [Created] (HIVE-14780) Determine unit tests to batch together based on previous run info

[Siddharth Seth (JIRA)]

Siddharth Seth created HIVE-14780:
-

 Summary: Determine unit tests to batch together based on previous 
run info
 Key: HIVE-14780
 URL: https://issues.apache.org/jira/browse/HIVE-14780
 Project: Hive
  Issue Type: Sub-task
Reporter: Siddharth Seth
Assignee: Siddharth Seth


Post HIVE-14540 - batch unit tests together with a time target, to avoid skew.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-14771) Add a warning to the jenkins report for slow tests

[Siddharth Seth (JIRA)]

Siddharth Seth created HIVE-14771:
-

 Summary: Add a warning to the jenkins report for slow tests
 Key: HIVE-14771
 URL: https://issues.apache.org/jira/browse/HIVE-14771
 Project: Hive
  Issue Type: Sub-task
Reporter: Siddharth Seth


Initially for new tests being slow, or existing tests slowing down a lot.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-14740) LLAP DynamicServiceInstasnce.isAlive does not seem to work as it should

[Siddharth Seth (JIRA)]

Siddharth Seth created HIVE-14740:
-

 Summary: LLAP DynamicServiceInstasnce.isAlive does not seem to 
work as it should
 Key: HIVE-14740
 URL: https://issues.apache.org/jira/browse/HIVE-14740
 Project: Hive
  Issue Type: Bug
Reporter: Siddharth Seth
Priority: Critical


Follow up from HIVE-14608. Is used in multiple places in LlapTaskScheduler. 
Critical to validate the broken functionality does not cause problems in the 
scheduler.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-14691) Investigate removal of TestHBaseMinimrCliDriver

[Siddharth Seth (JIRA)]

Siddharth Seth created HIVE-14691:
-

 Summary: Investigate removal of TestHBaseMinimrCliDriver
 Key: HIVE-14691
 URL: https://issues.apache.org/jira/browse/HIVE-14691
 Project: Hive
  Issue Type: Sub-task
Reporter: Siddharth Seth


Follow up from discussion on HIVE-14576. Look at whether this test can be 
completely removed given the MiniHBaseCliDriver tests.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-14689) Failing test: TestCliDriver explainuser_3

[Siddharth Seth (JIRA)]

Siddharth Seth created HIVE-14689:
-

 Summary: Failing test: TestCliDriver explainuser_3
 Key: HIVE-14689
 URL: https://issues.apache.org/jira/browse/HIVE-14689
 Project: Hive
  Issue Type: Sub-task
Reporter: Siddharth Seth


Consistent failures for quite a while.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 51312: HIVE-14589 add consistent node replacement to LLAP for splits

[Siddharth Seth]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51312/#review147610
---


Ship it!




Think there's several issues which still need resolving. Follow up jiras.
+1, assuming it's been tested locally for regresions and the new functionality.

- Siddharth Seth


On Sept. 1, 2016, 6:35 p.m., Sergey Shelukhin wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/51312/
> ---
> 
> (Updated Sept. 1, 2016, 6:35 p.m.)
> 
> 
> Review request for hive and Prasanth_J.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> see jira
> 
> 
> Diffs
> -
> 
>   llap-client/pom.xml 0243340 
>   
> llap-client/src/java/org/apache/hadoop/hive/llap/registry/ServiceInstanceSet.java
>  99ead9b 
>   
> llap-client/src/java/org/apache/hadoop/hive/llap/registry/impl/LlapFixedRegistryImpl.java
>  e9456f2 
>   
> llap-client/src/java/org/apache/hadoop/hive/llap/registry/impl/LlapZookeeperRegistryImpl.java
>  64d2617 
>   
> llap-client/src/java/org/apache/hadoop/hive/llap/registry/impl/SlotZnode.java 
> PRE-CREATION 
>   
> llap-client/src/java/org/apache/hadoop/hive/llap/security/LlapTokenClient.java
>  921e050 
>   
> llap-client/src/test/org/apache/hadoop/hive/llap/registry/impl/TestSlotZnode.java
>  PRE-CREATION 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/cli/LlapStatusServiceDriver.java
>  17ce69b 
>   
> llap-tez/src/java/org/apache/hadoop/hive/llap/tezplugins/LlapTaskSchedulerService.java
>  efd774d 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/Utils.java 8a4fc08 
>   
> ql/src/test/org/apache/hadoop/hive/ql/exec/tez/TestHostAffinitySplitLocationProvider.java
>  54f7363 
> 
> Diff: https://reviews.apache.org/r/51312/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Sergey Shelukhin
> 
>