Access to jira

[Kamesh]

Hi All,
 Can I get rights to assign/modify JIRA issues to myself

-- 
Kamesh.

Re: [VOTE] Release Apache Drill 1.3.0 (rc1)

[Aman Sinha]

+1.
Downloaded the binary tar ball and installed on Mac.  Ran Drill in embedded
mode.  Tested a few join, aggregation and limit queries against parquet
data with and without metadata cache.  Tested cancellation for couple of
queries.  Looked at query profiles on the WebUI.   Looks good.

On Sat, Nov 7, 2015 at 11:53 PM, Abhijit Pol  wrote:

> +1
>
> On Sat, Nov 7, 2015 at 9:41 PM, andrew  wrote:
>
> > +1 (non-binding, non-comitter)
> >
> > > On Nov 6, 2015, at 10:15 PM, Jacques Nadeau 
> wrote:
> > >
> > > Hey Everybody,
> > >
> > > I'd like to propose a new release candidate of Apache Drill, version
> > > 1.3.0.  This is the second release candidate (rc1).  This addresses
> some
> > > issues identified in the first release candidate including some test
> > > threading issues, Parquet dictionary read issue with binary values,
> Joda
> > > upgrade to avoid a concurrency bug and a couple other small things.
> > >
> > > The tarball artifacts are hosted at [2] and the maven artifacts are
> > hosted
> > > at [3]. This release candidate is based on commit
> > > f17ebd2fbf2bf5ba201a635f5b6ec3615afe2305 located at [4].
> > >
> > > The vote will be open for 72 hours ending at 10PM Pacific, November 9,
> > 2015.
> > >
> > > [ ] +1
> > > [ ] +0
> > > [ ] -1
> > >
> > > thanks,
> > > Jacques
> > >
> > > [1]
> > >
> >
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12313820=12332946[2]
> > > http://people.apache.org/~jacques/apache-drill-1.3.0.rc1/
> > > [3]
> > https://repository.apache.org/content/repositories/orgapachedrill-1014/
> > > [4] https://github.com/jacques-n/drill/tree/drill-1.3.0-rc1
> > >
> > > --
> > > Jacques Nadeau
> > > CTO and Co-Founder, Dremio
> >
> >
>

Re: [VOTE] Release Apache Drill 1.3.0 (rc1)

[Hsuan Yi Chu]

+1
mvn clean install on mac and linux

tried a few queries on distributed mode.

Things looked fine

On Sun, Nov 8, 2015 at 8:16 AM, Aman Sinha  wrote:

> +1.
> Downloaded the binary tar ball and installed on Mac.  Ran Drill in embedded
> mode.  Tested a few join, aggregation and limit queries against parquet
> data with and without metadata cache.  Tested cancellation for couple of
> queries.  Looked at query profiles on the WebUI.   Looks good.
>
> On Sat, Nov 7, 2015 at 11:53 PM, Abhijit Pol  wrote:
>
> > +1
> >
> > On Sat, Nov 7, 2015 at 9:41 PM, andrew  wrote:
> >
> > > +1 (non-binding, non-comitter)
> > >
> > > > On Nov 6, 2015, at 10:15 PM, Jacques Nadeau 
> > wrote:
> > > >
> > > > Hey Everybody,
> > > >
> > > > I'd like to propose a new release candidate of Apache Drill, version
> > > > 1.3.0.  This is the second release candidate (rc1).  This addresses
> > some
> > > > issues identified in the first release candidate including some test
> > > > threading issues, Parquet dictionary read issue with binary values,
> > Joda
> > > > upgrade to avoid a concurrency bug and a couple other small things.
> > > >
> > > > The tarball artifacts are hosted at [2] and the maven artifacts are
> > > hosted
> > > > at [3]. This release candidate is based on commit
> > > > f17ebd2fbf2bf5ba201a635f5b6ec3615afe2305 located at [4].
> > > >
> > > > The vote will be open for 72 hours ending at 10PM Pacific, November
> 9,
> > > 2015.
> > > >
> > > > [ ] +1
> > > > [ ] +0
> > > > [ ] -1
> > > >
> > > > thanks,
> > > > Jacques
> > > >
> > > > [1]
> > > >
> > >
> >
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12313820=12332946[2]
> > > > http://people.apache.org/~jacques/apache-drill-1.3.0.rc1/
> > > > [3]
> > >
> https://repository.apache.org/content/repositories/orgapachedrill-1014/
> > > > [4] https://github.com/jacques-n/drill/tree/drill-1.3.0-rc1
> > > >
> > > > --
> > > > Jacques Nadeau
> > > > CTO and Co-Founder, Dremio
> > >
> > >
> >
>

Re: Access to jira

[Jacques Nadeau]

Updated. Let me know if it doesn't work.

--
Jacques Nadeau
CTO and Co-Founder, Dremio

On Sun, Nov 8, 2015 at 6:50 AM, Kamesh  wrote:

> Hi All,
>  Can I get rights to assign/modify JIRA issues to myself
>
> --
> Kamesh.
>

[GitHub] drill pull request: DRILL-3751: Reduce zookeeper's retry time to 1...

[jacques-n]

Github user jacques-n commented on the pull request:

https://github.com/apache/drill/pull/248#issuecomment-154840757
  
Can you start a design discussion about zookeeper connection management on 
the list? We need to talk through the behavior of initial connect failure and 
long term failure. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: Zookeeper down before query starts/after query finishes

[Jacques Nadeau]

I think we need to talk through a couple of different scenarios and decide
on Drill behavior in each.

Client Based
1) Initial connection to ZK from client fails
2) Client loses ZK Connection
  a) Reconnects within session timeout
  b) Cannot reconnect within session timeout (loses session)
3) ZK Connection is gets reconnected with new session (2b)

Drillbit Based
4) Drillbit initial connection fails to complete
5) Drillbit loses connection
  a) reconnects within session timeout
  b) cannot reconnect within session timeout (loses session)
6) Drillbit reestablishes connection after timeout (5b)

It seems like your initial proposal is entirely focused on item (5b) in the
list above. However, the code change affects all items 1-6. I think it
would be worthwhile to come up with clear definition of desired behavior
for all items 1-6. I also think the behavior in 2b should probably be very
different than in 5b.

Note, I'm not suggesting that this initial fix needs to resolve all items
to the desired behavior. However, it is hard to review the patch without
measuring against what are target is across the items. My hope out of this
is a clear framework to review the patch as well as a number of jiras to
resolve issues across each of these issues where there are gaps.

thanks!
jacques



--
Jacques Nadeau
CTO and Co-Founder, Dremio

On Sun, Nov 8, 2015 at 9:36 AM, Hsuan Yi Chu  wrote:

> I just submitted a pull request to address DRILL-3751, which focuses on the
> scenario where query already finishes and zookeeper dies. So Foreman cannot
> delete the profiles of running queries in zookeeper.
>
> I think in this case, after a few retries, Foreman can assume Zookeeper is
> down. And, this query is assumed to fail since client might not be able to
> receive the result (see the behavior in DRILL-3751
> ).
>
> Does this make sense?
>
>
> On Fri, Nov 6, 2015 at 10:43 AM, Hsuan Yi Chu  wrote:
>
> > My understanding is :
> > Before query starts/After query finishes, Foreman will put/delete running
> > query profiles in zookeeper.
> >
> > However, if zookeeper is down before the put/delete is successful, Drill
> > would be blocked at the put/delete operation.
> >
> > See https://issues.apache.org/jira/browse/DRILL-3751
> >
> > I think it is not quite right to let Drill just wait for Zookeeper to
> > respond. Does it make sense to use "time-out" here?
> >
> >
> >
>

Re: Zookeeper down before query starts/after query finishes

[Hsuan Yi Chu]

I just submitted a pull request to address DRILL-3751, which focuses on the
scenario where query already finishes and zookeeper dies. So Foreman cannot
delete the profiles of running queries in zookeeper.

I think in this case, after a few retries, Foreman can assume Zookeeper is
down. And, this query is assumed to fail since client might not be able to
receive the result (see the behavior in DRILL-3751
).

Does this make sense?


On Fri, Nov 6, 2015 at 10:43 AM, Hsuan Yi Chu  wrote:

> My understanding is :
> Before query starts/After query finishes, Foreman will put/delete running
> query profiles in zookeeper.
>
> However, if zookeeper is down before the put/delete is successful, Drill
> would be blocked at the put/delete operation.
>
> See https://issues.apache.org/jira/browse/DRILL-3751
>
> I think it is not quite right to let Drill just wait for Zookeeper to
> respond. Does it make sense to use "time-out" here?
>
>
>

Re: Zookeeper down before query starts/after query finishes

[yuliya Feldman]

In the reality if you can not connect to ZK (and ConnectionLoss is a client 
side error) it either means issues with network on client node itself or issues 
with ZK quorum.  In those situations unless you receive (eventually) "Session 
Expiration" or "Connection reestablished" again you don't know what is going 
on. What probably would be prudent to do is to timeout if after ConnectionLoss 
you do not have anything back from ZK server for time > ZK client timeout (30 
sec. by default I think).
And again it will need to depend on the client - in your example it is a good 
idea to fail in some other cases it may be a good idea to wait (e.g if you deal 
with non-idempotent operations)
  From: Hsuan Yi Chu 
 To: dev@drill.apache.org 
 Sent: Sunday, November 8, 2015 9:36 AM
 Subject: Re: Zookeeper down before query starts/after query finishes
   
I just submitted a pull request to address DRILL-3751, which focuses on the
scenario where query already finishes and zookeeper dies. So Foreman cannot
delete the profiles of running queries in zookeeper.

I think in this case, after a few retries, Foreman can assume Zookeeper is
down. And, this query is assumed to fail since client might not be able to
receive the result (see the behavior in DRILL-3751
).

Does this make sense?




On Fri, Nov 6, 2015 at 10:43 AM, Hsuan Yi Chu  wrote:

> My understanding is :
> Before query starts/After query finishes, Foreman will put/delete running
> query profiles in zookeeper.
>
> However, if zookeeper is down before the put/delete is successful, Drill
> would be blocked at the put/delete operation.
>
> See https://issues.apache.org/jira/browse/DRILL-3751
>
> I think it is not quite right to let Drill just wait for Zookeeper to
> respond. Does it make sense to use "time-out" here?
>
>
>


  

[jira] [Created] (DRILL-4050) Add zip archives to the list of artifacts in verify_release.sh

[Aditya Kishore (JIRA)]

Aditya Kishore created DRILL-4050:
-

 Summary: Add zip archives to the list of artifacts in 
verify_release.sh
 Key: DRILL-4050
 URL: https://issues.apache.org/jira/browse/DRILL-4050
 Project: Apache Drill
  Issue Type: Task
Reporter: Aditya Kishore
Assignee: Aditya Kishore
Priority: Minor






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [VOTE] Release Apache Drill 1.3.0 (rc1)

[Abhijit Pol]

resending the vote with details

+1, non-binding

ran maven tests, installed on Mac, ran drill in embedded mode, verified Web
UI for CSV header config changes, ran few queries on CSV files with and
without header, ran few queries against large CSV files with header in HDFS

On Sun, Nov 8, 2015 at 9:07 AM, Hsuan Yi Chu  wrote:

> +1
> mvn clean install on mac and linux
>
> tried a few queries on distributed mode.
>
> Things looked fine
>
> On Sun, Nov 8, 2015 at 8:16 AM, Aman Sinha  wrote:
>
> > +1.
> > Downloaded the binary tar ball and installed on Mac.  Ran Drill in
> embedded
> > mode.  Tested a few join, aggregation and limit queries against parquet
> > data with and without metadata cache.  Tested cancellation for couple of
> > queries.  Looked at query profiles on the WebUI.   Looks good.
> >
> > On Sat, Nov 7, 2015 at 11:53 PM, Abhijit Pol  wrote:
> >
> > > +1
> > >
> > > On Sat, Nov 7, 2015 at 9:41 PM, andrew  wrote:
> > >
> > > > +1 (non-binding, non-comitter)
> > > >
> > > > > On Nov 6, 2015, at 10:15 PM, Jacques Nadeau 
> > > wrote:
> > > > >
> > > > > Hey Everybody,
> > > > >
> > > > > I'd like to propose a new release candidate of Apache Drill,
> version
> > > > > 1.3.0.  This is the second release candidate (rc1).  This addresses
> > > some
> > > > > issues identified in the first release candidate including some
> test
> > > > > threading issues, Parquet dictionary read issue with binary values,
> > > Joda
> > > > > upgrade to avoid a concurrency bug and a couple other small things.
> > > > >
> > > > > The tarball artifacts are hosted at [2] and the maven artifacts are
> > > > hosted
> > > > > at [3]. This release candidate is based on commit
> > > > > f17ebd2fbf2bf5ba201a635f5b6ec3615afe2305 located at [4].
> > > > >
> > > > > The vote will be open for 72 hours ending at 10PM Pacific, November
> > 9,
> > > > 2015.
> > > > >
> > > > > [ ] +1
> > > > > [ ] +0
> > > > > [ ] -1
> > > > >
> > > > > thanks,
> > > > > Jacques
> > > > >
> > > > > [1]
> > > > >
> > > >
> > >
> >
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12313820=12332946[2]
> > > > > http://people.apache.org/~jacques/apache-drill-1.3.0.rc1/
> > > > > [3]
> > > >
> > https://repository.apache.org/content/repositories/orgapachedrill-1014/
> > > > > [4] https://github.com/jacques-n/drill/tree/drill-1.3.0-rc1
> > > > >
> > > > > --
> > > > > Jacques Nadeau
> > > > > CTO and Co-Founder, Dremio
> > > >
> > > >
> > >
> >
>

[GitHub] drill pull request: DRILL-4050: Add zip archives to the list of ar...

[adityakishore]

GitHub user adityakishore opened a pull request:

https://github.com/apache/drill/pull/249

DRILL-4050: Add zip archives to the list of artifacts in verify_relea…

…se.sh

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/adityakishore/drill DRILL-4050

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/drill/pull/249.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #249


commit dac1acb66bdc0c24794df0cacef834c7abdb3944
Author: Aditya Kishore 
Date:   2015-11-08T22:08:15Z

DRILL-4050: Add zip archives to the list of artifacts in verify_release.sh




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: Zookeeper down before query starts/after query finishes

[yuliya Feldman]

Did not notice your reply :)
Yes - I agree with Jacques - we should consider variety of the scenarios here.
Thanks,Yuliya
  From: Jacques Nadeau 
 To: dev  
 Sent: Sunday, November 8, 2015 11:56 AM
 Subject: Re: Zookeeper down before query starts/after query finishes
   
I think we need to talk through a couple of different scenarios and decide
on Drill behavior in each.

Client Based
1) Initial connection to ZK from client fails
2) Client loses ZK Connection
  a) Reconnects within session timeout
  b) Cannot reconnect within session timeout (loses session)
3) ZK Connection is gets reconnected with new session (2b)

Drillbit Based
4) Drillbit initial connection fails to complete
5) Drillbit loses connection
  a) reconnects within session timeout
  b) cannot reconnect within session timeout (loses session)
6) Drillbit reestablishes connection after timeout (5b)

It seems like your initial proposal is entirely focused on item (5b) in the
list above. However, the code change affects all items 1-6. I think it
would be worthwhile to come up with clear definition of desired behavior
for all items 1-6. I also think the behavior in 2b should probably be very
different than in 5b.

Note, I'm not suggesting that this initial fix needs to resolve all items
to the desired behavior. However, it is hard to review the patch without
measuring against what are target is across the items. My hope out of this
is a clear framework to review the patch as well as a number of jiras to
resolve issues across each of these issues where there are gaps.

thanks!
jacques



--
Jacques Nadeau
CTO and Co-Founder, Dremio



On Sun, Nov 8, 2015 at 9:36 AM, Hsuan Yi Chu  wrote:

> I just submitted a pull request to address DRILL-3751, which focuses on the
> scenario where query already finishes and zookeeper dies. So Foreman cannot
> delete the profiles of running queries in zookeeper.
>
> I think in this case, after a few retries, Foreman can assume Zookeeper is
> down. And, this query is assumed to fail since client might not be able to
> receive the result (see the behavior in DRILL-3751
> ).
>
> Does this make sense?
>
>
> On Fri, Nov 6, 2015 at 10:43 AM, Hsuan Yi Chu  wrote:
>
> > My understanding is :
> > Before query starts/After query finishes, Foreman will put/delete running
> > query profiles in zookeeper.
> >
> > However, if zookeeper is down before the put/delete is successful, Drill
> > would be blocked at the put/delete operation.
> >
> > See https://issues.apache.org/jira/browse/DRILL-3751
> >
> > I think it is not quite right to let Drill just wait for Zookeeper to
> > respond. Does it make sense to use "time-out" here?
> >
> >
> >
>


  

Re: Announcing new committer: Sudheesh Katkam

[Abhishek Girish]

Congrats Sudheesh!

On Sunday, November 8, 2015, Khurram Faraaz  wrote:

> Congrats Sudheesh!
>
> On Sun, Nov 8, 2015 at 3:13 PM, Jacques Nadeau  > wrote:
>
> > The Apache Drill PMC is very proud to announce Sudheesh Katkam as a
> > new committer.
> >
> > Sudheesh has done a great deal of work on Drill and it is an honor to now
> > have him as a committer. We're lucky to have him as part of our
> community.
> >
> > Welcome Sudheesh!
> >
>

[GitHub] drill pull request: DRILL-3987: Componentize Drill, extracting vec...

[jacques-n]

GitHub user jacques-n opened a pull request:

https://github.com/apache/drill/pull/250

DRILL-3987: Componentize Drill, extracting vectors as separate library



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/jacques-n/drill DRILL-3987

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/drill/pull/250.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #250


commit 6bf5d2988e780f2bbec8614ba9b8384bb70df577
Author: Jacques Nadeau 
Date:   2015-11-07T23:54:35Z

Extract key vector, field reader, complex/field writer classes.

commit e123e9fcc945863586e64a11628a2a2480703e82
Author: Jacques Nadeau 
Date:   2015-11-08T02:16:05Z

Common and Vector modules building.

commit 5a351517bce28cfae4049ee415970a8f76ea7798
Author: Jacques Nadeau 
Date:   2015-11-08T02:41:03Z

Working TPCH unit tests

commit dcc309527abe9602b5a7e737abc8df81b162c3da
Author: Jacques Nadeau 
Date:   2015-11-08T03:28:54Z

Extract RPC, memory-base and memory-impl as separate modules.

commit 7724c236f476cf986fc909f748d90c656b10f941
Author: Jacques Nadeau 
Date:   2015-11-08T03:44:58Z

Extract BoundsChecking check from AssertionUtil. Remove unused file.

commit fd91a1771f17125a33e7df860c39bb7ef11f5446
Author: Jacques Nadeau 
Date:   2015-11-08T16:20:19Z

Delete unused files

commit ca6b5ef14486279e4a822646daee0628c2f95fd7
Author: Jacques Nadeau 
Date:   2015-11-08T17:09:05Z

Remove any parent Drill dependencies for drill-memory-base.

commit 88031387993db1816311498a49ea882a96794694
Author: Jacques Nadeau 
Date:   2015-11-08T17:35:45Z

Move logical expressions and operators out of common. Move to new 
drill-logical model.

commit dc9d48bc87d7a1d1af55facfe26ef81af3012fc2
Author: Jacques Nadeau 
Date:   2015-11-08T18:37:05Z

DRILL-3987: Final cleanups to get complete working build/distribution

- small cleanups
- move Hook to drill-adbc
- update distribution assembly to include new modules




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: [DISCUSS] Proposal to turn ValueVectors into separate reusable library & project

[Jacques Nadeau]

FYI, the patch also just successfully completed the extended regression
suite.

--
Jacques Nadeau
CTO and Co-Founder, Dremio

On Sun, Nov 8, 2015 at 5:09 PM, Jacques Nadeau  wrote:

> Ok guys,
>
> I took the quiet time directly after the release candidate went out to do
> the first phase of componentization. You can see my work at [1].
>
> This set of commits has little functional impact. I've also done my best
> to avoid package or file renaming, rather keeping things in their same
> packages but in different modules (so that other patches are more easily
> applied). There are nine commits in the branch. They break down into three
> categories: MOVE, REFACTOR & CLEANUP.
>
> I've separated the changes out so that it should be reasonably
> straightforward to review. The MOVE patches are constrained primarily to
> moving files from module to another.
>
> DRILL-3987: (MOVE) Extract key vector, field reader, complex/field wr… …
> 21cbd84
> DRILL-3987: (REFACTOR) Common and Vector modules building. … e390db9
> DRILL-3987: (REFACTOR) Working TPCH unit tests … 2cc1d30
> DRILL-3987: (MOVE) Extract RPC, memory-base and memory-impl as separa… …
> d5f3211
> DRILL-3987: (REFACTOR) Extract BoundsChecking check from AssertionUti… …
> 83c53d8
> DRILL-3987: (CLEANUP) Delete unused files 5d596d5
> DRILL-3987: (REFACTOR) Remove any parent Drill dependencies for drill… …
> 76f578c
> DRILL-3987: (MOVE) Move logical expressions and operators out of comm… …
> f908b8b
> DRILL-3987: (CLEANUP) Final cleanups to get complete working build/di… …
> d09aa3b
>
> The main goal was to extract a number of separate java-exec submodules.
> I've also outlined the modularization in a couple slides at [2]. In those
> slides you'll see that there are some orange dependencies that will need to
> be removed in a second phase of effort. We also need to decide which
> portions of the third slide at [2] would be appropriate as a separate
> project versus maintained inside of Drill.
>
> Some of the dependencies will need a finer grained hand to separate. The
> biggest remaining is cleaning up VectorDescriptor, MaterializedField,
> SerializedField, SchemaPath and FieldReference so that vector can stop
> depending on the new drill-logical module.
>
> My preference would be to merge this straight away as the functional
> impact is limited and it would be exceedingly difficult to maintain this
> patch. This patch set provides a complete set of changes for modularization
> and passes all unit tests. I'm running the extended regression suite now to
> confirm no impact on those issues. I don't expect any since the only bugs
> I've had to track down thus far are drill-module or pom dependency issues.
>
> Let me know your thoughts.
>
> [1] https://github.com/apache/drill/pull/250
> [2]
> https://docs.google.com/presentation/d/1HD-EzAgNe4EJvoP91ILFLFJdFjT2T5yfM9MEv79BaiM/edit
>
>
> --
> Jacques Nadeau
> CTO and Co-Founder, Dremio
>
> On Tue, Oct 27, 2015 at 5:59 PM, Jacques Nadeau 
> wrote:
>
>> Yes, I've started the umbrella @
>> https://issues.apache.org/jira/browse/DRILL-3986
>>
>> And the first sub task: extraction poc @
>> https://issues.apache.org/jira/browse/DRILL-3987
>>
>> I posted some existing materials. I'll start looking at how we can
>> extract. Would love others thoughts about how we might slice things. I'll
>> post some initial thoughts on the jiras in this regard.
>>
>> --
>> Jacques Nadeau
>> CTO and Co-Founder, Dremio
>>
>> On Tue, Oct 27, 2015 at 5:39 PM, Julian Hyde  wrote:
>>
>>> Jacques, Can you please log the JIRA case you mentioned, and also attach
>>> any documentation (e.g. javadoc) you already have.
>>>
>>>
>>
>

Re: Access to jira

[Kamesh]

Thanks Jacques. It is working.

On Sun, Nov 8, 2015 at 10:22 PM, Jacques Nadeau  wrote:

> Updated. Let me know if it doesn't work.
>
> --
> Jacques Nadeau
> CTO and Co-Founder, Dremio
>
> On Sun, Nov 8, 2015 at 6:50 AM, Kamesh  wrote:
>
> > Hi All,
> >  Can I get rights to assign/modify JIRA issues to myself
> >
> > --
> > Kamesh.
> >
>



-- 
Kamesh.

Re: Announcing new committer: Sudheesh Katkam

[Amit Hadke]

Congrats Sudheesh!

On Sun, Nov 8, 2015 at 6:27 PM, Abdel Hakim Deneche 
wrote:

> Congrats Sudheesh!
>
> On Sun, Nov 8, 2015 at 5:16 PM, Abhishek Girish  >
> wrote:
>
> > Congrats Sudheesh!
> >
> > On Sunday, November 8, 2015, Khurram Faraaz 
> wrote:
> >
> > > Congrats Sudheesh!
> > >
> > > On Sun, Nov 8, 2015 at 3:13 PM, Jacques Nadeau  > > > wrote:
> > >
> > > > The Apache Drill PMC is very proud to announce Sudheesh Katkam as a
> > > > new committer.
> > > >
> > > > Sudheesh has done a great deal of work on Drill and it is an honor to
> > now
> > > > have him as a committer. We're lucky to have him as part of our
> > > community.
> > > >
> > > > Welcome Sudheesh!
> > > >
> > >
> >
>
>
>
> --
>
> Abdelhakim Deneche
>
> Software Engineer
>
>   
>
>
> Now Available - Free Hadoop On-Demand Training
> <
> http://www.mapr.com/training?utm_source=Email_medium=Signature_campaign=Free%20available
> >
>

Re: [VOTE] Release Apache Drill 1.3.0 (rc1)

[Amit Hadke]

+1, non-binding
Downloaded tarball, mvn clean install
Ran join queries,, was able to run queries on json, parquet and sequence
files.

On Sun, Nov 8, 2015 at 10:01 AM, Abhijit Pol  wrote:

> resending the vote with details
>
> +1, non-binding
>
> ran maven tests, installed on Mac, ran drill in embedded mode, verified Web
> UI for CSV header config changes, ran few queries on CSV files with and
> without header, ran few queries against large CSV files with header in HDFS
>
> On Sun, Nov 8, 2015 at 9:07 AM, Hsuan Yi Chu  wrote:
>
> > +1
> > mvn clean install on mac and linux
> >
> > tried a few queries on distributed mode.
> >
> > Things looked fine
> >
> > On Sun, Nov 8, 2015 at 8:16 AM, Aman Sinha  wrote:
> >
> > > +1.
> > > Downloaded the binary tar ball and installed on Mac.  Ran Drill in
> > embedded
> > > mode.  Tested a few join, aggregation and limit queries against parquet
> > > data with and without metadata cache.  Tested cancellation for couple
> of
> > > queries.  Looked at query profiles on the WebUI.   Looks good.
> > >
> > > On Sat, Nov 7, 2015 at 11:53 PM, Abhijit Pol 
> wrote:
> > >
> > > > +1
> > > >
> > > > On Sat, Nov 7, 2015 at 9:41 PM, andrew  wrote:
> > > >
> > > > > +1 (non-binding, non-comitter)
> > > > >
> > > > > > On Nov 6, 2015, at 10:15 PM, Jacques Nadeau 
> > > > wrote:
> > > > > >
> > > > > > Hey Everybody,
> > > > > >
> > > > > > I'd like to propose a new release candidate of Apache Drill,
> > version
> > > > > > 1.3.0.  This is the second release candidate (rc1).  This
> addresses
> > > > some
> > > > > > issues identified in the first release candidate including some
> > test
> > > > > > threading issues, Parquet dictionary read issue with binary
> values,
> > > > Joda
> > > > > > upgrade to avoid a concurrency bug and a couple other small
> things.
> > > > > >
> > > > > > The tarball artifacts are hosted at [2] and the maven artifacts
> are
> > > > > hosted
> > > > > > at [3]. This release candidate is based on commit
> > > > > > f17ebd2fbf2bf5ba201a635f5b6ec3615afe2305 located at [4].
> > > > > >
> > > > > > The vote will be open for 72 hours ending at 10PM Pacific,
> November
> > > 9,
> > > > > 2015.
> > > > > >
> > > > > > [ ] +1
> > > > > > [ ] +0
> > > > > > [ ] -1
> > > > > >
> > > > > > thanks,
> > > > > > Jacques
> > > > > >
> > > > > > [1]
> > > > > >
> > > > >
> > > >
> > >
> >
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12313820=12332946[2]
> > > > > > http://people.apache.org/~jacques/apache-drill-1.3.0.rc1/
> > > > > > [3]
> > > > >
> > >
> https://repository.apache.org/content/repositories/orgapachedrill-1014/
> > > > > > [4] https://github.com/jacques-n/drill/tree/drill-1.3.0-rc1
> > > > > >
> > > > > > --
> > > > > > Jacques Nadeau
> > > > > > CTO and Co-Founder, Dremio
> > > > >
> > > > >
> > > >
> > >
> >
>

Re: Announcing new committer: Sudheesh Katkam

[Khurram Faraaz]

Congrats Sudheesh!

On Sun, Nov 8, 2015 at 3:13 PM, Jacques Nadeau  wrote:

> The Apache Drill PMC is very proud to announce Sudheesh Katkam as a
> new committer.
>
> Sudheesh has done a great deal of work on Drill and it is an honor to now
> have him as a committer. We're lucky to have him as part of our community.
>
> Welcome Sudheesh!
>

Announcing new committer: Sudheesh Katkam

[Jacques Nadeau]

The Apache Drill PMC is very proud to announce Sudheesh Katkam as a
new committer.

Sudheesh has done a great deal of work on Drill and it is an honor to now
have him as a committer. We're lucky to have him as part of our community.

Welcome Sudheesh!

Re: [DISCUSS] Proposal to turn ValueVectors into separate reusable library & project

[Jacques Nadeau]

Ok guys,

I took the quiet time directly after the release candidate went out to do
the first phase of componentization. You can see my work at [1].

This set of commits has little functional impact. I've also done my best to
avoid package or file renaming, rather keeping things in their same
packages but in different modules (so that other patches are more easily
applied). There are nine commits in the branch. They break down into three
categories: MOVE, REFACTOR & CLEANUP.

I've separated the changes out so that it should be reasonably
straightforward to review. The MOVE patches are constrained primarily to
moving files from module to another.

DRILL-3987: (MOVE) Extract key vector, field reader, complex/field wr… …
21cbd84
DRILL-3987: (REFACTOR) Common and Vector modules building. … e390db9
DRILL-3987: (REFACTOR) Working TPCH unit tests … 2cc1d30
DRILL-3987: (MOVE) Extract RPC, memory-base and memory-impl as separa… …
d5f3211
DRILL-3987: (REFACTOR) Extract BoundsChecking check from AssertionUti… …
83c53d8
DRILL-3987: (CLEANUP) Delete unused files 5d596d5
DRILL-3987: (REFACTOR) Remove any parent Drill dependencies for drill… …
76f578c
DRILL-3987: (MOVE) Move logical expressions and operators out of comm… …
f908b8b
DRILL-3987: (CLEANUP) Final cleanups to get complete working build/di… …
d09aa3b

The main goal was to extract a number of separate java-exec submodules.
I've also outlined the modularization in a couple slides at [2]. In those
slides you'll see that there are some orange dependencies that will need to
be removed in a second phase of effort. We also need to decide which
portions of the third slide at [2] would be appropriate as a separate
project versus maintained inside of Drill.

Some of the dependencies will need a finer grained hand to separate. The
biggest remaining is cleaning up VectorDescriptor, MaterializedField,
SerializedField, SchemaPath and FieldReference so that vector can stop
depending on the new drill-logical module.

My preference would be to merge this straight away as the functional impact
is limited and it would be exceedingly difficult to maintain this patch.
This patch set provides a complete set of changes for modularization and
passes all unit tests. I'm running the extended regression suite now to
confirm no impact on those issues. I don't expect any since the only bugs
I've had to track down thus far are drill-module or pom dependency issues.

Let me know your thoughts.

[1] https://github.com/apache/drill/pull/250
[2]
https://docs.google.com/presentation/d/1HD-EzAgNe4EJvoP91ILFLFJdFjT2T5yfM9MEv79BaiM/edit


--
Jacques Nadeau
CTO and Co-Founder, Dremio

On Tue, Oct 27, 2015 at 5:59 PM, Jacques Nadeau  wrote:

> Yes, I've started the umbrella @
> https://issues.apache.org/jira/browse/DRILL-3986
>
> And the first sub task: extraction poc @
> https://issues.apache.org/jira/browse/DRILL-3987
>
> I posted some existing materials. I'll start looking at how we can
> extract. Would love others thoughts about how we might slice things. I'll
> post some initial thoughts on the jiras in this regard.
>
> --
> Jacques Nadeau
> CTO and Co-Founder, Dremio
>
> On Tue, Oct 27, 2015 at 5:39 PM, Julian Hyde  wrote:
>
>> Jacques, Can you please log the JIRA case you mentioned, and also attach
>> any documentation (e.g. javadoc) you already have.
>>
>>
>

Re: Announcing new committer: Sudheesh Katkam

[Abdel Hakim Deneche]

Congrats Sudheesh!

On Sun, Nov 8, 2015 at 5:16 PM, Abhishek Girish 
wrote:

> Congrats Sudheesh!
>
> On Sunday, November 8, 2015, Khurram Faraaz  wrote:
>
> > Congrats Sudheesh!
> >
> > On Sun, Nov 8, 2015 at 3:13 PM, Jacques Nadeau  > > wrote:
> >
> > > The Apache Drill PMC is very proud to announce Sudheesh Katkam as a
> > > new committer.
> > >
> > > Sudheesh has done a great deal of work on Drill and it is an honor to
> now
> > > have him as a committer. We're lucky to have him as part of our
> > community.
> > >
> > > Welcome Sudheesh!
> > >
> >
>



-- 

Abdelhakim Deneche

Software Engineer

  


Now Available - Free Hadoop On-Demand Training

[jira] [Resolved] (DRILL-2352) MongoDB join queries can't select fields

[Bhallamudi Venkata Siva Kamesh (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-2352?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bhallamudi Venkata Siva Kamesh resolved DRILL-2352.
---
   Resolution: Fixed
Fix Version/s: (was: Future)

> MongoDB join queries can't select fields
> 
>
> Key: DRILL-2352
> URL: https://issues.apache.org/jira/browse/DRILL-2352
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Storage - MongoDB
>Affects Versions: 0.7.0
> Environment: Ubuntu 14.04 64 bits, OpenJDK 1.7.0_75, Mongo 2.6.8
>Reporter: Mariano Ruiz
> Attachments: drill-error-select-without-limit.log, 
> drill-select-with-limit.log
>
>
> When I perform this *JOIN* query between two MongoDB documents, there is no 
> problem:
> {code:sql}
> select * from zips as z join states as s on z.state = s.code;
> {code}
> Result:
> {code}
> +++
> | *  | *0 |
> +++
> | { "city" : "AGAWAM" , "loc" : [ -72.622739 , 42.070206] , "pop" : 15338 , 
> "state" : "MA"} | { "code" : "CA" , "name" : "California"} |
> | { "city" : "AGAWAM" , "loc" : [ -72.622739 , 42.070206] , "pop" : 15338 , 
> "state" : "MA"} | { "code" : "DC" , "name" : "Washington"} |
> | { "city" : "AGAWAM" , "loc" : [ -72.622739 , 42.070206] , "pop" : 15338 , 
> "state" : "MA"} | { "code" : "NY" , "name" : "New York"} |
> | { "city" : "CUSHMAN" , "loc" : [ -72.51565 , 42.377017] , "pop" : 36963 , 
> "state" : "MA"} | { "code" : "CA" , "name" : "California"} |
> | { "city" : "CUSHMAN" , "loc" : [ -72.51565 , 42.377017] , "pop" : 36963 , 
> "state" : "MA"} | { "code" : "DC" , "name" : "Washington"} |
> {code}
> But when I try to select which field I want to see I get this:
> {code}
> 0: jdbc:drill:zk=local> select z.city, s.name from zips as z join states as s 
> on z.state = s.code;
> Query failed: Query failed: Unexpected exception during fragment 
> initialization: null
> Error: exception while executing query: Failure while executing query. 
> (state=,code=0)
> {code}
> I realized after a few tests, that the problem is you can't specify a column 
> from the table left operator in the JOIN-ON clause.
> For example, this works:
> {code}
> jdbc:drill:zk=local> select z.city from states as s join zips as z on s.code 
> = z.state limit 5;
> ++
> |city|
> ++
> | LOS ANGELES |
> | TRUCKEE|
> | TRUCKEE|
> | SOUTH LAKE TAHOE |
> | TAHOE VISTA |
> {code}
> But this doesn't:
> {code}
> jdbc:drill:zk=local> select s.name from states as s join zips as z on s.code 
> = z.state limit 5;
> ++
> |name|
> ++
> ++
> {code}
> The same occurs in the WHERE clause, for example this works:
> {code:sql}
> select z.city from states as s join zips as z on s.code = z.state where 
> z.city = 'LOS ANGELES';
> {code}
> And this doesn't:
> {code}
> 0: jdbc:drill:zk=local> select z.city from states as s join zips as z on 
> s.code = z.state where s.code = 'CA' limit 5;
> Query failed: Query failed: Unexpected exception during fragment 
> initialization: null
> Error: exception while executing query: Failure while executing query. 
> (state=,code=0)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (DRILL-4051) I have used the latest drill-1.3,but the error occurred when starting:

[david_hudavy (JIRA)]

david_hudavy created DRILL-4051:
---

 Summary:  I have used the latest drill-1.3,but the error occurred 
when starting:
 Key: DRILL-4051
 URL: https://issues.apache.org/jira/browse/DRILL-4051
 Project: Apache Drill
  Issue Type: Bug
Affects Versions: 1.3.0
Reporter: david_hudavy


 I have used the latest drill-1.3-SNAPSHOT,but the error occurred when starting:

vm1-4:/opt/drill/bin # ./sqlline -u jdbc:drill:zk=local -n admin -p admin
[INFO] Unable to bind key for unsupported operation: backward-delete-word
[INFO] Unable to bind key for unsupported operation: backward-delete-word
[INFO] Unable to bind key for unsupported operation: down-history
[INFO] Unable to bind key for unsupported operation: up-history
[INFO] Unable to bind key for unsupported operation: up-history
[INFO] Unable to bind key for unsupported operation: down-history
[INFO] Unable to bind key for unsupported operation: up-history
[INFO] Unable to bind key for unsupported operation: down-history
[INFO] Unable to bind key for unsupported operation: up-history
[INFO] Unable to bind key for unsupported operation: down-history
[INFO] Unable to bind key for unsupported operation: up-history
[INFO] Unable to bind key for unsupported operation: down-history
log4j:WARN No appenders could be found for logger 
(com.mapr.baseutils.cldbutils.CLDBRpcCommonUtils).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more 
info.
2015-11-09 10:30:00,2722 ERROR Cidcache 
fs/client/fileclient/cc/cidcache.cc:1586 Thread: 5641 MoveToNextCldb: No CLDB 
entries, cannot run, sleeping 5 seconds!
2015-11-09 10:30:05,2725 ERROR Client fs/client/fileclient/cc/client.cc:831 
Thread: 5641 Failed to initialize client for cluster 127.0.0.1:7222, error 
Connection reset by peer(104)
Error: Failure in starting embedded Drillbit: java.io.IOException: Could not 
create FileClient (state=,code=0)
java.sql.SQLException: Failure in starting embedded Drillbit: 
java.io.IOException: Could not create FileClient
at 
org.apache.drill.jdbc.impl.DrillConnectionImpl.(DrillConnectionImpl.java:109)
at 
org.apache.drill.jdbc.impl.DrillJdbc41Factory.newDrillConnection(DrillJdbc41Factory.java:66)
at 
org.apache.drill.jdbc.impl.DrillFactory.newConnection(DrillFactory.java:69)
at 
net.hydromatic.avatica.UnregisteredDriver.connect(UnregisteredDriver.java:126)
at org.apache.drill.jdbc.Driver.connect(Driver.java:72)
at sqlline.DatabaseConnection.connect(DatabaseConnection.java:167)
at sqlline.DatabaseConnection.getConnection(DatabaseConnection.java:213)
at sqlline.Commands.connect(Commands.java:1083)
at sqlline.Commands.connect(Commands.java:1015)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at 
sqlline.ReflectiveCommandHandler.execute(ReflectiveCommandHandler.java:36)
at sqlline.SqlLine.dispatch(SqlLine.java:742)
at sqlline.SqlLine.initArgs(SqlLine.java:528)
at sqlline.SqlLine.begin(SqlLine.java:596)
at sqlline.SqlLine.start(SqlLine.java:375)
at sqlline.SqlLine.main(SqlLine.java:268)
Caused by: java.io.IOException: Could not create FileClient
at com.mapr.fs.MapRFileSystem.lookupClient(MapRFileSystem.java:579)
at com.mapr.fs.MapRFileSystem.lookupClient(MapRFileSystem.java:640)
at 
com.mapr.fs.MapRFileSystem.getMapRFileStatus(MapRFileSystem.java:1296)
at com.mapr.fs.MapRFileSystem.getFileStatus(MapRFileSystem.java:928)
at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1460)
at 
org.apache.drill.exec.store.dfs.DrillFileSystem.mkdirs(DrillFileSystem.java:460)
at 
org.apache.drill.exec.store.sys.local.FilePStore.getFileSystem(FilePStore.java:86)
at 
org.apache.drill.exec.store.sys.local.LocalPStoreProvider.(LocalPStoreProvider.java:50)
at org.apache.drill.exec.server.Drillbit.(Drillbit.java:197)
at org.apache.drill.exec.server.Drillbit.(Drillbit.java:178)
at 
org.apache.drill.jdbc.impl.DrillConnectionImpl.(DrillConnectionImpl.java:99)
... 18 more
apache drill 1.3.0-SNAPSHOT
"say hello to my little drill"
0: jdbc:drill:zk=local>
Command canceled.
0: jdbc:drill:zk=local>
Command canceled.
0: jdbc:drill:zk=local>
Command canceled.
0: jdbc:drill:zk=local> show databases;
No current connection




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Zookeeper down before query starts/after query finishes

[Hsuan Yi Chu]

Thanks Jacques and Yuliya's points.

The list summarized by Jacques should be a good guideline to deal with the
variety of zookeeper disconnection problems.

Regarding the initial target (5.b), I would like to propose that, after session
timeouts (instead of setting the time-out on the global level, we should
have another parameter in the drill-module.conf to decide this value), the
drill-bit is supposed to return to the state where drill-bit seeks for
initial connection to zookeeper. Meanwhile, fail the queries initiated by
this drill bit's foreman.

On Sun, Nov 8, 2015 at 12:39 PM, yuliya Feldman  wrote:

> Did not notice your reply :)
> Yes - I agree with Jacques - we should consider variety of the scenarios
> here.
> Thanks,Yuliya
>   From: Jacques Nadeau 
>  To: dev 
>  Sent: Sunday, November 8, 2015 11:56 AM
>  Subject: Re: Zookeeper down before query starts/after query finishes
>
> I think we need to talk through a couple of different scenarios and decide
> on Drill behavior in each.
>
> Client Based
> 1) Initial connection to ZK from client fails
> 2) Client loses ZK Connection
>   a) Reconnects within session timeout
>   b) Cannot reconnect within session timeout (loses session)
> 3) ZK Connection is gets reconnected with new session (2b)
>
> Drillbit Based
> 4) Drillbit initial connection fails to complete
> 5) Drillbit loses connection
>   a) reconnects within session timeout
>   b) cannot reconnect within session timeout (loses session)
> 6) Drillbit reestablishes connection after timeout (5b)
>
> It seems like your initial proposal is entirely focused on item (5b) in the
> list above. However, the code change affects all items 1-6. I think it
> would be worthwhile to come up with clear definition of desired behavior
> for all items 1-6. I also think the behavior in 2b should probably be very
> different than in 5b.
>
> Note, I'm not suggesting that this initial fix needs to resolve all items
> to the desired behavior. However, it is hard to review the patch without
> measuring against what are target is across the items. My hope out of this
> is a clear framework to review the patch as well as a number of jiras to
> resolve issues across each of these issues where there are gaps.
>
> thanks!
> jacques
>
>
>
> --
> Jacques Nadeau
> CTO and Co-Founder, Dremio
>
>
>
> On Sun, Nov 8, 2015 at 9:36 AM, Hsuan Yi Chu  wrote:
>
> > I just submitted a pull request to address DRILL-3751, which focuses on
> the
> > scenario where query already finishes and zookeeper dies. So Foreman
> cannot
> > delete the profiles of running queries in zookeeper.
> >
> > I think in this case, after a few retries, Foreman can assume Zookeeper
> is
> > down. And, this query is assumed to fail since client might not be able
> to
> > receive the result (see the behavior in DRILL-3751
> > ).
> >
> > Does this make sense?
> >
> >
> > On Fri, Nov 6, 2015 at 10:43 AM, Hsuan Yi Chu 
> wrote:
> >
> > > My understanding is :
> > > Before query starts/After query finishes, Foreman will put/delete
> running
> > > query profiles in zookeeper.
> > >
> > > However, if zookeeper is down before the put/delete is successful,
> Drill
> > > would be blocked at the put/delete operation.
> > >
> > > See https://issues.apache.org/jira/browse/DRILL-3751
> > >
> > > I think it is not quite right to let Drill just wait for Zookeeper to
> > > respond. Does it make sense to use "time-out" here?
> > >
> > >
> > >
> >
>
>
>
>

Re: Announcing new committer: Sudheesh Katkam

[AnilKumar B]

Congrats Sudheesh.

Thanks & Regards,
B Anil Kumar.

On Mon, Nov 9, 2015 at 10:33 AM, Hsuan Yi Chu  wrote:

> Congrats Sudheesh!
>
> On Sun, Nov 8, 2015 at 7:10 PM, Amit Hadke  wrote:
>
> > Congrats Sudheesh!
> >
> > On Sun, Nov 8, 2015 at 6:27 PM, Abdel Hakim Deneche <
> adene...@maprtech.com
> > >
> > wrote:
> >
> > > Congrats Sudheesh!
> > >
> > > On Sun, Nov 8, 2015 at 5:16 PM, Abhishek Girish <
> > abhishek.gir...@gmail.com
> > > >
> > > wrote:
> > >
> > > > Congrats Sudheesh!
> > > >
> > > > On Sunday, November 8, 2015, Khurram Faraaz 
> > > wrote:
> > > >
> > > > > Congrats Sudheesh!
> > > > >
> > > > > On Sun, Nov 8, 2015 at 3:13 PM, Jacques Nadeau  > > > > > wrote:
> > > > >
> > > > > > The Apache Drill PMC is very proud to announce Sudheesh Katkam
> as a
> > > > > > new committer.
> > > > > >
> > > > > > Sudheesh has done a great deal of work on Drill and it is an
> honor
> > to
> > > > now
> > > > > > have him as a committer. We're lucky to have him as part of our
> > > > > community.
> > > > > >
> > > > > > Welcome Sudheesh!
> > > > > >
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > >
> > > Abdelhakim Deneche
> > >
> > > Software Engineer
> > >
> > >   
> > >
> > >
> > > Now Available - Free Hadoop On-Demand Training
> > > <
> > >
> >
> http://www.mapr.com/training?utm_source=Email_medium=Signature_campaign=Free%20available
> > > >
> > >
> >
>