Re: Hangout Discussion Topics for 04-16-2019

[Padma Penumarthy]

You can look at RecordBatchMemoryManager.java and follow one of the
operator code (like flatten) to see how this was done.

Thanks
Padma


On Wed, Apr 24, 2019 at 12:00 PM Paul Rogers 
wrote:

> Hi Igor,
>
> Thanks for the recap. You asked about vector allocation. Here is where I
> think things stand. Others can fill in details that I may miss.
>
> We have several ways to size value vectors; but no single standard. As you
> note, the most common way is simply to accept the cost of letting the
> vector double in size multiple times.
>
> One way to pre-allocate vectors is to use the "sizer" along with its
> associated allocation helper. This was always meant to be a quick & dirty
> temporary solution, but has turned out, I believe, to be the primary vector
> size management solution in most operators.
>
> Another is the new row set framework: vector size (in terms of number of
> items and estimated item size) is expressed in metadata, then is used to
> allocate each new batch to the desired size.
>
> You can also just do the work yourself: pick a number, and, when
> allocating a vector, tell it to use that size. You then take on the task of
> estimating average width, picking a good target number of rows for your
> batch, working out the number of items in arrays, etc. (This is, in fact,
> what the other two methods mentioned above actually do.)
>
> The key problem with the ad-hoc techniques is that they can't limit
> maximum vector size to 16 MB (to avoid Netty fragmentation) nor limit
> overall batch size to some reasonable number. The ad-hoc techniques can
> also lead to internal fragmentation (excessive unused space within each
> vector.) Solving these problems is what the row set framework was designed
> to do.
>
> Thanks,
> - Paul
>
>
>
> On Wednesday, April 24, 2019, 10:48:44 AM PDT, Igor Guzenko <
> ihor.huzenko@gmail.com> wrote:
>
>  Hello Everyone,
>
> Sorry for the late reply, here is presentations about
>
> Map vector-
>
> https://docs.google.com/presentation/d/1FG4swOrkFIRL7qjiP7PSOPy8a1vnxs5Z9PM3ZfRPRYo/edit#slide=id.p
> Hive complex types  -
>
> https://docs.google.com/presentation/d/1nc0ID5aju-qj-7hjquFpH-TwGjeReWTYogsExuOe8ZA/edit?usp=sharing
> .
>
> Discussion results for Map new vector:
> - Need to eliminate possibility of key duplication;
> - Need to check Hive behavior when ORDER BY is performed for Map
> complex type column;
> - Need to describe design and all use cases for the vector in design
> document.
>
> Discussion results for Hive complex types:
> - Aman Sinha made few great suggestions. First is that creation of
> Hive writers may be done once for table scan and second is that at
> this moment
>   would be good to calculate size for vectors and allocate early. Need
> to provide few examples describing how will the allocation work for
> complex types.
> - Need to describe suggested approach in design document and proceed
> discussion there.
>
> Question from my side. Do we have already implemented somewhere
> predicted allocation of value vectors ? Any example would be useful,
> because
> now I can see that our existing vector writers usually use mutator's
> setSafe(...) methods inside which size of buffer may be increased when
> necessary.
>
> The future design document will be located at
>
> https://docs.google.com/document/d/1yEcaJi9dyksfMs4w5_GsZCQH_Pffe-HLeLVNNKsV7CA/edit?usp=sharing
> .
> Please feel free to leave your comments and suggestions in the
> document and presentations.
>
> Thanks,
> Igor Guzenko
>
>
> On Wed, Apr 17, 2019 at 3:04 AM Jyothsna Reddy 
> wrote:
> >
> > Hi All,
> > The hangout will start at 9:30 AM PST instead of 10 AM PST on 04-18-2019.
> >
> >
> > Thank you,
> > Jyothsna
> >
> >
> >
> >
> > On Tue, Apr 16, 2019 at 2:00 PM Jyothsna Reddy 
> > wrote:
> >
> > > Hi Charles,
> > > Yes, sure!! Probably we can start with your discussion first and Hive
> > > complex types later since there will be some discussion around the
> later
> > > topic.
> > >
> > > Thank you,
> > > Jyothsna
> > >
> > >
> > >
> > >
> > > On Tue, Apr 16, 2019 at 1:40 PM Charles Givre 
> wrote:
> > >
> > >> Hi Jyothsna,
> > >> Could I get a few minutes on the next Hangout to promote the Drill
> day at
> > >> ApacheCon?
> > >> Thanks
> > >>
> > >> > On Apr 16, 2019, at 16:38, Jyothsna Reddy 
> > >> wrote:
> > >> >
> > >> > Hi Everyone,
> > >> >
> > >> > Here are some key points of today's hangout discussion:
> > >> >
> > >> > Sorabh mentioned that there are some regressions in TPCDS queries
> and
> > >> its a
> > >> > blocker for 1.16 release.
> > >> >
> > >> > Bohdan presented tehir proposal for Hive Complex types support.
> Here are
> > >> > some of the important points
> > >> >
> > >> >  - Structure of MapVector : Keys are of primitive type where values
> can
> > >> >  be of either primitive or complex type.
> > >> >  - MapReader and MapWriter are used to read and write from the
> > >> MapVector
> > >> >  - MapWriter tracks the current row/length and is used to calculate

Re: [ANNOUNCE] New Committer: Karthikeyan Manivannan

[Padma Penumarthy]

Congrats Karthik.

Thanks
Padma


On Fri, Dec 7, 2018 at 1:33 PM Paul Rogers 
wrote:

> Congrats Karthik!
>
> - Paul
>
> Sent from my iPhone
>
> > On Dec 7, 2018, at 11:12 AM, Abhishek Girish  wrote:
> >
> > Congratulations Karthik!
> >
> >> On Fri, Dec 7, 2018 at 11:11 AM Arina Ielchiieva 
> wrote:
> >>
> >> The Project Management Committee (PMC) for Apache Drill has invited
> >> Karthikeyan
> >> Manivannan to become a committer, and we are pleased to announce that he
> >> has accepted.
> >>
> >> Karthik started contributing to the Drill project in 2016. He has
> >> implemented changes in various Drill areas, including batch sizing,
> >> security, code-gen, C++ part. One of his latest improvements is  ACL
> >> support for Drill ZK nodes.
> >>
> >> Welcome Karthik, and thank you for your contributions!
> >>
> >> - Arina
> >> (on behalf of Drill PMC)
> >>
>

Re: November Apache Drill board report

[Padma Penumarthy]

1.14 release notes should mention batch sizing also.

For the board report, can you please add the following :
Batch processing improvements to limit the amount of memory for Hash Join,
Union All, Project, Hash Aggregate and Nested Loop Join.

Just FYI for everyone. Here is the link to the document which has details
about batch sizing work:
https://docs.google.com/document/d/1Z-67Y_KNcbA2YYWCHEwf2PUEmXRPWSXsw-CHnXW_98Q/

Thanks
Padma





On Wed, Nov 7, 2018 at 5:56 AM Arina Ielchiieva  wrote:

> Hi Padma,
>
> I can include mention about batch sizing but I am not sure what I should
> mention, quick search over release notes shows a couple of changes related
> to batch sizing:
> https://drill.apache.org/docs/apache-drill-1-14-0-release-notes/
> Could you please propose what I should include?
>
> @PMCs and committers
> Only one PMC member has given +1 for the report. Could more folks please
> review the report?
>
> Kind regards,
> Arina
>
> On Fri, Nov 2, 2018 at 8:33 PM Padma Penumarthy <
> penumarthy.pa...@gmail.com>
> wrote:
>
> > Hi Arina,
> >
> > Can you add batch sizing (for bunch of operators and parquet reader)
> also ?
> >
> > Thanks
> > Padma
> >
> >
> > On Fri, Nov 2, 2018 at 2:55 AM Arina Ielchiieva 
> wrote:
> >
> > > Sure, let's mention.
> > > Updated the report.
> > >
> > > =
> > >
> > >  ## Description:
> > >  - Drill is a Schema-free SQL Query Engine for Hadoop, NoSQL and Cloud
> > > Storage.
> > >
> > > ## Issues:
> > >  - There are no issues requiring board attention at this time.
> > >
> > > ## Activity:
> > >  - Since the last board report, Drill has released version 1.14.0,
> > >including the following enhancements:
> > > - Drill in a Docker container
> > > - Image metadata format plugin
> > > - Upgrade to Calcite 1.16.0
> > > - Kafka plugin push down support
> > > - Phonetic and String functions
> > > - Enhanced decimal data support
> > > - Spill to disk for the Hash Join support
> > > - CGROUPs resource management support
> > > - Lateral / Unnest support (disabled by default)
> > >  - There were active discussions about schema provision in Drill.
> > >Based on these discussions two projects are currently evolving:
> > >Drill metastore and schema provision in the file and in a query.
> > >  - Apache Drill book has been written by two PMC members (Charles and
> > > Paul).
> > >  - Drill developer meet up will be held on November 14, 2018.
> > >
> > >The following areas are going to be discussed:
> > > - Storage plugins
> > > - Schema discovery & Evolution
> > > - Metadata Management
> > > - Resource management
> > > - Integration with Apache Arrow
> > >
> > > ## Health report:
> > >  - The project is healthy. Development activity
> > >as reflected in the pull requests and JIRAs is good.
> > >  - Activity on the dev and user mailing lists are stable.
> > >  - Three committers and three new PMC member were added in the last
> > period.
> > >
> > > ## PMC changes:
> > >
> > >  - Currently 23 PMC members.
> > >  - New PMC members:
> > > - Boaz Ben-Zvi was added to the PMC on Fri Aug 17 2018
> > > - Charles Givre was added to the PMC on Mon Sep 03 2018
> > > - Vova Vysotskyi was added to the PMC on Fri Aug 24 2018
> > >
> > > ## Committer base changes:
> > >
> > >  - Currently 48 committers.
> > >  - New commmitters:
> > > - Chunhui Shi was added as a committer on Thu Sep 27 2018
> > > - Gautam Parai was added as a committer on Mon Oct 22 2018
> > > - Weijie Tong was added as a committer on Fri Aug 31 2018
> > >
> > > ## Releases:
> > >
> > >  - 1.14.0 was released on Sat Aug 04 2018
> > >
> > > ## Mailing list activity:
> > >
> > >  - dev@drill.apache.org:
> > > - 427 subscribers (down -6 in the last 3 months):
> > > - 2827 emails sent to list (2126 in previous quarter)
> > >
> > >  - iss...@drill.apache.org:
> > > - 18 subscribers (down -1 in the last 3 months):
> > > - 3487 emails sent to list (4769 in previous quarter)
> > >
> > >  - u...@drill.apache.org:
> > > - 597 subscribers (down -6 in the last 3 months):
> > > - 332 emails sent to list (346 in previous quarter)
> > >
> > &

Re: November Apache Drill board report

[Padma Penumarthy]

Hi Arina,

Can you add batch sizing (for bunch of operators and parquet reader) also ?

Thanks
Padma


On Fri, Nov 2, 2018 at 2:55 AM Arina Ielchiieva  wrote:

> Sure, let's mention.
> Updated the report.
>
> =
>
>  ## Description:
>  - Drill is a Schema-free SQL Query Engine for Hadoop, NoSQL and Cloud
> Storage.
>
> ## Issues:
>  - There are no issues requiring board attention at this time.
>
> ## Activity:
>  - Since the last board report, Drill has released version 1.14.0,
>including the following enhancements:
> - Drill in a Docker container
> - Image metadata format plugin
> - Upgrade to Calcite 1.16.0
> - Kafka plugin push down support
> - Phonetic and String functions
> - Enhanced decimal data support
> - Spill to disk for the Hash Join support
> - CGROUPs resource management support
> - Lateral / Unnest support (disabled by default)
>  - There were active discussions about schema provision in Drill.
>Based on these discussions two projects are currently evolving:
>Drill metastore and schema provision in the file and in a query.
>  - Apache Drill book has been written by two PMC members (Charles and
> Paul).
>  - Drill developer meet up will be held on November 14, 2018.
>
>The following areas are going to be discussed:
> - Storage plugins
> - Schema discovery & Evolution
> - Metadata Management
> - Resource management
> - Integration with Apache Arrow
>
> ## Health report:
>  - The project is healthy. Development activity
>as reflected in the pull requests and JIRAs is good.
>  - Activity on the dev and user mailing lists are stable.
>  - Three committers and three new PMC member were added in the last period.
>
> ## PMC changes:
>
>  - Currently 23 PMC members.
>  - New PMC members:
> - Boaz Ben-Zvi was added to the PMC on Fri Aug 17 2018
> - Charles Givre was added to the PMC on Mon Sep 03 2018
> - Vova Vysotskyi was added to the PMC on Fri Aug 24 2018
>
> ## Committer base changes:
>
>  - Currently 48 committers.
>  - New commmitters:
> - Chunhui Shi was added as a committer on Thu Sep 27 2018
> - Gautam Parai was added as a committer on Mon Oct 22 2018
> - Weijie Tong was added as a committer on Fri Aug 31 2018
>
> ## Releases:
>
>  - 1.14.0 was released on Sat Aug 04 2018
>
> ## Mailing list activity:
>
>  - dev@drill.apache.org:
> - 427 subscribers (down -6 in the last 3 months):
> - 2827 emails sent to list (2126 in previous quarter)
>
>  - iss...@drill.apache.org:
> - 18 subscribers (down -1 in the last 3 months):
> - 3487 emails sent to list (4769 in previous quarter)
>
>  - u...@drill.apache.org:
> - 597 subscribers (down -6 in the last 3 months):
> - 332 emails sent to list (346 in previous quarter)
>
>
> ## JIRA activity:
>
>  - 164 JIRA tickets created in the last 3 months
>  - 128 JIRA tickets closed/resolved in the last 3 months
>
>
>
> On Fri, Nov 2, 2018 at 12:25 AM Sorabh Hamirwasia 
> wrote:
>
> > Hi Arina,
> > Lateral/Unnest feature was part of 1.14 though it was disabled by
> default.
> > Should we mention it as part of 1.14 enhancements in the report?
> >
> > Thanks,
> > Sorabh
> >
> > On Thu, Nov 1, 2018 at 9:29 AM Arina Yelchiyeva <
> > arina.yelchiy...@gmail.com>
> > wrote:
> >
> > > Thanks, Aman!  Updated the report.
> > > I went too far with 2019, luckily the meet up will be much earlier :)
> > >
> > > =
> > >
> > >  ## Description:
> > >  - Drill is a Schema-free SQL Query Engine for Hadoop, NoSQL and Cloud
> > > Storage.
> > >
> > > ## Issues:
> > >  - There are no issues requiring board attention at this time.
> > >
> > > ## Activity:
> > >  - Since the last board report, Drill has released version 1.14.0,
> > >including the following enhancements:
> > > - Drill in a Docker container
> > > - Image metadata format plugin
> > > - Upgrade to Calcite 1.16.0
> > > - Kafka plugin push down support
> > > - Phonetic and String functions
> > > - Enhanced decimal data support
> > > - Spill to disk for the Hash Join support
> > > - CGROUPs resource management support
> > >  - There were active discussions about schema provision in Drill.
> > >Based on these discussions two projects are currently evolving:
> > >Drill metastore and schema provision in the file and in a query.
> > >  - Apache Drill book has been written by two PMC members (Charles and
> > > Paul).
> > >  - Drill developer meet up will be held on November 14, 2018.
> > >The following areas are going to be discussed:
> > > - Storage plugins
> > > - Schema discovery & Evolution
> > > - Metadata Management
> > > - Resource management
> > > - Integration with Apache Arrow
> > >
> > > ## Health report:
> > >  - The project is healthy. Development activity
> > >as reflected in the pull requests and JIRAs is good.
> > >  - Activity on the dev and user mailing lists are stable.
> > >  - Three committers and three new PMC member were added in the last
> > period.
> > >
> > > ## PMC changes:
> > >
> > >  - Currently 23 PMC 

Re: [ANNOUNCE] New Committer: Hanumath Rao Maduri

[Padma Penumarthy]

Congratulations Hanu.

Thanks
Padma


On Thu, Nov 1, 2018 at 7:44 PM weijie tong  wrote:

> Congratulations, Hanu!
>
> On Fri, Nov 2, 2018 at 8:22 AM Robert Hou  wrote:
>
> > Congratulations, Hanu.  Thanks for contributing to Drill.
> >
> > --Robert
> >
> > On Thu, Nov 1, 2018 at 4:06 PM Jyothsna Reddy 
> > wrote:
> >
> > > Congrats Hanu!! Well deserved :D
> > >
> > > Thank you,
> > > Jyothsna
> > >
> > > On Thu, Nov 1, 2018 at 2:15 PM Sorabh Hamirwasia  >
> > > wrote:
> > >
> > > > Congratulations Hanu!
> > > >
> > > > Thanks,
> > > > Sorabh
> > > >
> > > > On Thu, Nov 1, 2018 at 1:35 PM Hanumath Rao Maduri <
> hanu@gmail.com
> > >
> > > > wrote:
> > > >
> > > > > Thank you all for the wishes!
> > > > >
> > > > > Thanks,
> > > > > -Hanu
> > > > >
> > > > > On Thu, Nov 1, 2018 at 1:28 PM Chunhui Shi  > > > > .invalid>
> > > > > wrote:
> > > > >
> > > > > > Congratulations Hanu!
> > > > > >
> --
> > > > > > From:Arina Ielchiieva 
> > > > > > Send Time:2018 Nov 1 (Thu) 06:05
> > > > > > To:dev ; user 
> > > > > > Subject:[ANNOUNCE] New Committer: Hanumath Rao Maduri
> > > > > >
> > > > > > The Project Management Committee (PMC) for Apache Drill has
> invited
> > > > > > Hanumath
> > > > > > Rao Maduri to become a committer, and we are pleased to announce
> > that
> > > > he
> > > > > > has accepted.
> > > > > >
> > > > > > Hanumath became a contributor in 2017, making changes mostly in
> the
> > > > Drill
> > > > > > planning side, including lateral / unnest support. He is also one
> > of
> > > > the
> > > > > > contributors of index based planning and execution support.
> > > > > >
> > > > > > Welcome Hanumath, and thank you for your contributions!
> > > > > >
> > > > > > - Arina
> > > > > > (on behalf of Drill PMC)
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: msgpack reading schema files checksum error

[Padma Penumarthy]

How are you modifying the file manually ?
Are you copying to local file system, make changes and copying back to HDFS
?

Thanks
Padma


On Tue, Oct 30, 2018 at 5:45 PM Jean-Claude Cote  wrote:

> I'm writing a msgpack reader which supports schema validation. The msgpack
> reader is able to discover the schema and store the result in a file
> named .schema.proto along side the data files. There is also an additional
> ..schema.proto.crc file created by the hadoop file system I believe.
>
> However even if the reader can discover the schema I would like to be able
> to edit the file manually. However when I do the checksum file does not
> match anymore and my reader fails to load the file.
>
> My question is how can I read a file ignoring the checksum file. Or how
> difficult is it to produce these checksum files.
>
> I save the file like so
> try (FSDataOutputStream out = fileSystem.create(schemaLocation, true))
> {
>
> This call fails if I modify the schema file manually.
>   try (FSDataInputStream in = fileSystem.open(schemaLocation)) {
>
> Thank you
> jc
>

Re: [ANNOUNCE] New Committer: Gautam Parai

[Padma Penumarthy]

Congratulations Gautam.

Thanks
Padma


On Mon, Oct 22, 2018 at 7:25 AM Arina Ielchiieva  wrote:

> The Project Management Committee (PMC) for Apache Drill has invited Gautam
> Parai to become a committer, and we are pleased to announce that he has
> accepted.
>
> Gautam has become a contributor since 2016, making changes in various Drill
> areas including planning side. He is also one of the contributors of the
> upcoming feature to support index based planning and execution.
>
> Welcome Gautam, and thank you for your contributions!
>
> - Arina
> (on behalf of Drill PMC)
>

Re: [ANNOUNCE] New Committer: Chunhui Shi

[Padma Penumarthy]

Congratulations Chunhui.

Thanks
Padma


On Fri, Sep 28, 2018 at 2:17 AM Arina Ielchiieva  wrote:

> The Project Management Committee (PMC) for Apache Drill has invited Chunhui
> Shi to become a committer, and we are pleased to announce that he has
> accepted.
>
> Chunhui Shi has become a contributor since 2016, making changes in various
> Drill areas. He has shown profound knowledge in Drill planning side during
> his work to support lateral join. He is also one of the contributors of the
> upcoming feature to support index based planning and execution.
>
> Welcome Chunhui, and thank you for your contributions!
>
> - Arina
> (on behalf of Drill PMC)
>

Re: [ANNOUNCE] New PMC member: Boaz Ben-Zvi

[Padma Penumarthy]

Congratulations Boaz.

Thanks
Padma


On Fri, Aug 17, 2018 at 2:33 PM, Robert Wu  wrote:

> Congratulations, Boaz!
>
> Best regards,
>
> Rob
>
> -Original Message-
> From: Abhishek Girish 
> Sent: Friday, August 17, 2018 2:17 PM
> To: dev 
> Subject: Re: [ANNOUNCE] New PMC member: Boaz Ben-Zvi
>
> Congratulations, Boaz!
>
> On Fri, Aug 17, 2018 at 2:15 PM Sorabh Hamirwasia 
> wrote:
>
> > Congratulations Boaz!
> >
> > On Fri, Aug 17, 2018 at 11:42 AM, Karthikeyan Manivannan <
> > kmanivan...@mapr.com> wrote:
> >
> > > Congrats! Well deserved!
> > >
> > > On Fri, Aug 17, 2018, 11:31 AM Timothy Farkas 
> wrote:
> > >
> > > > Congrats!
> > > >
> > > > On Fri, Aug 17, 2018 at 11:27 AM, Gautam Parai 
> > wrote:
> > > >
> > > > > Congratulations Boaz!!
> > > > >
> > > > > Gautam
> > > > >
> > > > > On Fri, Aug 17, 2018 at 11:04 AM, Khurram Faraaz
> > > > > 
> > > > wrote:
> > > > >
> > > > > > Congratulations Boaz.
> > > > > >
> > > > > > On Fri, Aug 17, 2018 at 10:47 AM, shi.chunhui <
> > > > > > shi.chun...@aliyun.com.invalid> wrote:
> > > > > >
> > > > > > > Congrats Boaz!
> > > > > > >
> > --
> > > > > > > Sender:Arina Ielchiieva  Sent at:2018 Aug
> > > > > > > 17 (Fri) 17:51 To:dev ; user
> > > > > > >  Subject:[ANNOUNCE] New PMC member:
> > > > > > > Boaz Ben-Zvi
> > > > > > >
> > > > > > > I am pleased to announce that Drill PMC invited Boaz Ben-Zvi
> > > > > > > to
> > the
> > > > PMC
> > > > > > and
> > > > > > > he has accepted the invitation.
> > > > > > >
> > > > > > > Congratulations Boaz and thanks for your contributions!
> > > > > > >
> > > > > > > - Arina
> > > > > > > (on behalf of Drill PMC)
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: Metadata management improvement

[Padma Penumarthy]

If your use case can be addressed by adding session option to skip the
checks, that
would be simpler to do and it can be done much faster.
Adding TTL support would be more complex.

I will let someone else comment on long term plans as I don't know the
details.

Thanks
Padma





On Thu, Aug 2, 2018 at 8:54 AM, Joel Pfaff  wrote:

> Hello,
>
> "I think the simplest thing that should be done first is to provide option
> to skip the check"
> I agree that whatever we do, we should not introduce any change in user
> experience by default.
> But since the default's behaviour is to not set any TTL in the meta-data, I
> have conflicted feelings about that.
> But in our usecase, we will mostly try to optimize a single application, so
> we could add this session option at querying time as well.
> Long story short: I do not have strong opinions about what the default
> should be.
>
> Concerning the overall change: the introduction of TTL, can we submit a
> design document, or would you prefer to invest on the longer term meta data
> repository?
>
> Regards, Joel
>
>
> On Thu, Aug 2, 2018 at 6:28 AM, Padma Penumarthy <
> penumarthy.pa...@gmail.com
> > wrote:
>
> > I think the simplest thing that should be done first is to provide option
> > to skip the check.
> > The default behavior for that option will be what we do today i.e. check
> > root directory
> > and all sub directories underneath.
> >
> > Thanks
> > Padma
> >
> >
> >
> > On Mon, Jul 30, 2018 at 3:01 AM, Joel Pfaff 
> wrote:
> >
> > > Hello,
> > >
> > > Thanks a lot for all these feedbacks, trying to respond to everything
> > > below:
> > >
> > > @Parth:
> > > "I don't think we would want to maintain a TTL for the metadata store
> so
> > > introducing one now would mean that we might break backward
> compatibility
> > > down the road."
> > > Yes, I am aware of this activity starting, and I agree that whatever
> the
> > > solution decided later on for the new metadata store, it most probably
> > > won't support a concept of TTL. This means that we would either have to
> > > break the support of the `WITH TTL` extension of the SQL command, or to
> > > ignore it down the road. None of these solutions seem particularly
> > > appealing to me.
> > >
> > > @Padma:
> > > "What issues we need to worry about if different directories in the
> > > hierarchy are checked last at different times ?"
> > > Knowing that the refresh is always recursive, we can only have two
> cases:
> > > the parent level cache is refreshed at the same time as the child level
> > > cache, or the parent level cache is older than the child level cache
> > > (because a query has run in a sub-directory, that triggered a refresh
> of
> > > the metadata in this sub-directory). In both cases, checking the
> > timestamp
> > > of the cache file at the root directory of the query is enough to know
> if
> > > the TTL criteria is respected.
> > > In the case the cache files are not refreshed at the same time between
> > > parent and children directories, and that the parent's cache is still
> > valid
> > > with regards to its TTL, Drill would trust the parent cache, and issue
> an
> > > execution plan with this set of files. The same query on a child folder
> > > would use the children cache, that would have been refreshed more
> > recently,
> > > and this would potentially result in issuing an execution plan with
> > another
> > > set of files.
> > > So this basically, this TTL feature could create discrepancies in the
> > > results, and these discrepancies could last up to the TTL value.
> > >
> > > "Do we have to worry about coordinating against multiple drillbits ?"
> > > That would be better indeed, as the problem already exists today (I
> have
> > > not found any locking mechanism on metadata file), I am not sure this
> > > change would make it worse.
> > > So the reply is yes, we should worry, but I think the fix for that
> would
> > be
> > > independent to this change.
> > >
> > > "Another option is if the difference between modification time of
> > directory
> > > and metadata cache file is within TTL limit, do not do anything. If we
> do
> > > that, do we get errors during execution (I think so) ?"
> > > We would get errors if there would be files removed between the time of
> > the
> > 

Re: Metadata management improvement

[Padma Penumarthy]

I think the simplest thing that should be done first is to provide option
to skip the check.
The default behavior for that option will be what we do today i.e. check
root directory
and all sub directories underneath.

Thanks
Padma



On Mon, Jul 30, 2018 at 3:01 AM, Joel Pfaff  wrote:

> Hello,
>
> Thanks a lot for all these feedbacks, trying to respond to everything
> below:
>
> @Parth:
> "I don't think we would want to maintain a TTL for the metadata store so
> introducing one now would mean that we might break backward compatibility
> down the road."
> Yes, I am aware of this activity starting, and I agree that whatever the
> solution decided later on for the new metadata store, it most probably
> won't support a concept of TTL. This means that we would either have to
> break the support of the `WITH TTL` extension of the SQL command, or to
> ignore it down the road. None of these solutions seem particularly
> appealing to me.
>
> @Padma:
> "What issues we need to worry about if different directories in the
> hierarchy are checked last at different times ?"
> Knowing that the refresh is always recursive, we can only have two cases:
> the parent level cache is refreshed at the same time as the child level
> cache, or the parent level cache is older than the child level cache
> (because a query has run in a sub-directory, that triggered a refresh of
> the metadata in this sub-directory). In both cases, checking the timestamp
> of the cache file at the root directory of the query is enough to know if
> the TTL criteria is respected.
> In the case the cache files are not refreshed at the same time between
> parent and children directories, and that the parent's cache is still valid
> with regards to its TTL, Drill would trust the parent cache, and issue an
> execution plan with this set of files. The same query on a child folder
> would use the children cache, that would have been refreshed more recently,
> and this would potentially result in issuing an execution plan with another
> set of files.
> So this basically, this TTL feature could create discrepancies in the
> results, and these discrepancies could last up to the TTL value.
>
> "Do we have to worry about coordinating against multiple drillbits ?"
> That would be better indeed, as the problem already exists today (I have
> not found any locking mechanism on metadata file), I am not sure this
> change would make it worse.
> So the reply is yes, we should worry, but I think the fix for that would be
> independent to this change.
>
> "Another option is if the difference between modification time of directory
> and metadata cache file is within TTL limit, do not do anything. If we do
> that, do we get errors during execution (I think so) ?"
> We would get errors if there would be files removed between the time of the
> last generation of meta-data, and the time of the execution. As in the case
> above, this can already happen, since there is currently no guarantee that
> the files at planning time will still be there at execution time. The
> timeframe would increase from a few milliseconds to several minutes, so the
> frequency of this kind of problem occurring would be much higher.
> I would recommend to quietly ignore missing files by considering them as
> empty files.
>
> "Also, how to reset that state and do metadata cache refresh eventually ?"
> We could reuse the REFRESH TABLE METADATA command to force the refresh.
> This would allow for collaborative ingestion jobs to force the refresh when
> the datasets have changed.
> Non-collaborative jobs would then rely on the TTL to get the new dataset
> available.
>
> "Instead of TTL, I think having a system/session option that will let us
> skip this check altogether would be a good thing to have. So, if we know we
> are not adding new data, we can set that option."
> I would see the need to set TTL per Table. Since different tables will have
> different update frequencies.
> I agree on a session option to bypass TTL check, so that this user will
> always see the last dataset.
> The question then becomes: what would be the default value for this option?
>
> Regards, Joel
>
>
> On Fri, Jul 13, 2018 at 9:06 AM, Padma Penumarthy <
> penumarthy.pa...@gmail.com> wrote:
>
> > Hi Joel,
> >
> > This is my understanding:
> > We have list of all directories (i.e. all subdirectories and their
> > subdirectories etc.) in the metadata
> > cache file of each directory. We go through that list of directories and
> > check
> > directory modification time against modification time of metadata cache
> > file in that directory.
> > If this does not match for

Re: [ANNOUNCE] New PMC Chair of Apache Drill

[Padma Penumarthy]

Arina, Congratulations and best wishes.

Thanks
Padma



On Wed, Jul 18, 2018 at 4:54 PM, Bridget Bevens  wrote:

> Congratulations, Arina!!!
>
> On Wed, Jul 18, 2018 at 3:20 PM, Parth Chandra  wrote:
>
> > Congratulations
> >
> > On Wed, Jul 18, 2018 at 3:14 PM, Kunal Khatua  wrote:
> >
> > > Congratulations, Arina !
> > > On 7/18/2018 2:26:05 PM, Volodymyr Vysotskyi 
> > wrote:
> > > Congratulations, Arina! Well deserved!
> > >
> > > Kind regards,
> > > Volodymyr Vysotskyi
> > >
> > >
> > > On Thu, Jul 19, 2018 at 12:24 AM Abhishek Girish wrote:
> > >
> > > > Congratulations, Arina!
> > > >
> > > > On Wed, Jul 18, 2018 at 2:19 PM Aman Sinha wrote:
> > > >
> > > > > Drill developers,
> > > > > Time flies and it is time for a new PMC chair ! Thank you all for
> > your
> > > > > support during the past year.
> > > > >
> > > > > I am very pleased to announce that the Drill PMC has voted to elect
> > > Arina
> > > > > Ielchiieva as the new PMC chair of Apache Drill. She has also been
> > > > > approved unanimously by the Apache Board in today's board meeting.
> > > > Please
> > > > > join me in congratulating Arina !
> > > > >
> > > > > Thanks,
> > > > > Aman
> > > > >
> > > >
> > >
> >
>

Re: Metadata management improvement

[Padma Penumarthy]

Hi Joel,

This is my understanding:
We have list of all directories (i.e. all subdirectories and their
subdirectories etc.) in the metadata
cache file of each directory. We go through that list of directories and
check
directory modification time against modification time of metadata cache
file in that directory.
If this does not match for any of the directories, we build the metadata
cache for the whole hierarchy.
The reason we have to do this adding new files will only update
modification time of immediate
parent directory and not the whole hierarchy.

Regarding your proposal, some random thoughts:
How will you get current time that can be compared against last
modification time set by the file system ?
I think you meant compare current system time of the running java process
i.e. drillbit
against last time we checked if metadata cache needs to be updated for that
directory.
What issues we need to worry about if different directories in the
hierarchy are checked last at different times ?
Do we have to worry about coordinating against multiple drillbits ?

Another option is if the difference between modification time of directory
and metadata cache file is within
TTL limit, do not do anything. If we do that, do we get errors during
execution (I think so) ?
Also, how to reset that state and do metadata cache refresh eventually ?
We are not saving time for modification time checks here.

Instead of TTL, I think having a system/session option that will let us
skip this check altogether would be a
good thing to have. So, if we know we are not adding new data, we can set
that option.

Instead of saving this TTL in metadata cache file for each table(directory),
is it better to have this TTL as global system or session option ?
In that case, we cannot have a different TTL for each table, but it makes
it much simpler.
Otherwise, there are some complications to think about.
We have a root metadata file per directory with each of the subdirectories
underneath having their own metadata file.
So, if we update the TTL of the root directory, do we update for all the
subdirectories or just the top level directory ?
What issues we need to think about if TTL of the root directory and
subdirectories are different ?


Thanks
Padma




On Thu, Jul 12, 2018 at 8:07 AM, Joel Pfaff  wrote:

> Hello,
>
> Thanks for the feedback.
>
> The logic I had in mind was to add the TTL, as a refresh_interval field in
> the root metadata file.
>
> At each query, the current time would be compared to the addition of the
> modification time of the root metadata file and the refresh_interval.
> If the current time is greater, it would mean the metadata may be invalid,
> so the regular process would apply: recursively going through the file to
> check for updates, and trig a full metadata cache refresh any change is
> detected, or just touch the metadata file to align its modification time
> with the current time if no change is detected.
> If the current time is smaller, the root metadata would be trusted (without
> additional checks) and the planning would continue.
>
> So in most of the cases, only the timestamp of the root metadata file would
> be checked. In the worst case (at most once per TTL), all the timestamps
> would be checked.
>
> Regards, Joel
>
> On Thu, Jul 12, 2018 at 4:47 PM, Vitalii Diravka <
> vitalii.dira...@gmail.com>
> wrote:
>
> > Hi Joel,
> >
> > Sounds reasonable.
> > But if Drill checks this TTL property from metadata cache file for every
> > query and for every file instead of file timestamp, it will not give the
> > benefit.
> > I suppose we can add this TTL property to only root metadata cache file
> and
> > check it only once per query.
> >
> > Could you clarify the details, what is the TTL time?
> > How TTL info could be used to determine whether refresh is needed for the
> > query?
> >
> > Kind regards
> > Vitalii
> >
> >
> > On Thu, Jul 12, 2018 at 4:40 PM Joel Pfaff  wrote:
> >
> > > Hello,
> > >
> > > Today, on a table for which we have created statistics (through the
> > REFRESH
> > > TABLE METADATA  command), Drill validates the timestamp
> of
> > > every files or directory involved in the scan.
> > >
> > > If the timestamps of the files are greater than the one of the metadata
> > > file, then a re-regeneration of the meta-data file is triggered.
> > > In the case the timestamp of the metadata file is the greatest, then
> the
> > > planning continues without regenerating the metadata.
> > >
> > > When the number of files to be queried increases, this operation can
> > take a
> > > significant amount of time.
> > > We have seen cases where this validation step alone is taking 3 to 5
> > > seconds (just checking the timestamps), meaning the planning time was
> > > taking way more time than the querying time.
> > > And this can be problematic in some usecases where the response time is
> > > favored compared to the `accuracy` of the data.
> > >
> > > What would you think about adding an option to the 

Re: [DISCUSS] 1.14.0 release

[Padma Penumarthy]

If possible, please include PR 1363 in this release so we can complete our
batch sizing work (except for exchange operators)
DRILL-6549: batch sizing for nested loop join.

Thanks
Padma



On Fri, Jul 6, 2018 at 2:51 PM, Pritesh Maker  wrote:

> Here is the release 1.14 dashboard (https://issues.apache.org/
> jira/secure/Dashboard.jspa?selectPageId=12332463 ) and agile board (
> https://issues.apache.org/jira/secure/RapidBoard.jspa?rapidView=185)
>
> I believe Volodymyr is targeting DRILL-6422 (Guava update) for 1.15
> release so it shouldn't be blocking the release. So overall, we have 2 open
> bugs, 2 in progress bugs (+2 doc issues), and 12 in review (+1 ready to
> commit).
>
> If the reviewable commits won't be ready soon, can the developers please
> remove the 1.14 fix version for these issues.
>
> Pritesh
>
>
>
>
> On 7/6/18, 11:54 AM, "Boaz Ben-Zvi"  b...@mapr.com> wrote:
>
>   Current status: There's a blocker, and some work in progress that
> will
> stretch into next week.
>
> Current detail:
>
> ==
>
> Open/blocker - DRILL-6453 + DRILL-6517: Two issues - Parquet Scanner
> not setting record num (to zero), and a hang following this failure.
>
> In Progress - DRILL-6104: Generic Logfile Format Plugin
>
> PR - DRILL-6422: Update Guava to 23.0 and shade it
>
> PR - DRILL-5999 (DRILL-6516): Support for EMIT outcome in Streaming
> Agg (I'm reviewing)
>
>
> Ready2Commit: DRILL-6519: Add String Distance and Phonetic Functions
> (Arina gave it a +1 ; is it "Ready-To-Commit" or waiting for more reviews ?)
>
>
> Committed: DRILL-6570: Mentioned as a blocker by Kunal (I just merge
> #1354; the Jira was (mistakenly ?) marked "Resolved" so it missed the batch
> commit).
>
> Committed: DRILL-5977: predicate pushdown support kafkaMsgOffset
>
> Committed: DRILL-6577: Change Hash-Join default to not fallback (into
> pre-1.14 unlimited memory)
>
> Committed: DRILL-6353: Upgrade Parquet MR dependencies
>
> Committed: DRILL-6310: limit batch size for hash aggregate
>
> ===
>
> Thanks,
>
>  Boaz
>
> On 7/2/18 9:51 PM, Khurram Faraaz wrote:
> > Do we plan to fix this one too, because this is a regression from
> Apache
> > Drill 1.13.0.
> > https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.
> apache.org_jira_browse_DRILL-2D6453=DwIBaQ=cskdkSMqhcnjZxdQVpwTXg=
> EqulKDxxEDCX6zbp1AZAa1-iAPQGgCioAqgDp7DE2BU=9HpIZKDh_
> DBcx9yXCI2TtUWum8vRhdQgmw_3ljJQi1M=w-1C-HnlUAGRHvDFUusZ78WZrHahDka2r-
> F-cNb-pZA=
> >
> > On Mon, Jul 2, 2018 at 9:33 PM, Kunal Khatua 
> wrote:
> >
> >> DRILL-6570 seems like a must-have (release blocker, IMHO).
> >> On 7/2/2018 8:02:00 PM, Boaz Ben-Zvi  wrote:
> >> Let's try to make progress on the 1.14 release, aiming for a Release
> >> Candidate towards the end of this week (a little ambitious, with the
> >> July 4th and people on vacations).
> >>
> >> Current Status of the previously requested Jiras:
> >>
> >> ==
> >>
> >> In Progress - DRILL-6104: Generic Logfile Format Plugin
> >>
> >> PR - DRILL-6422: Update Guava to 23.0 and shade it
> >>
> >> PR - DRILL-5999 (DRILL-6516): Support for EMIT outcome in Streaming
> Agg
> >>
> >> Ready2Commit: DRILL-5977: predicate pushdown support kafkaMsgOffset
> >>
> >> Ready2Commit: DRILL-6519: Add String Distance and Phonetic Functions
> >>
> >> Ready2Commit: DRILL-6577: Change Hash-Join default to not fallback
> (into
> >> pre-1.14 unlimited memory)
> >>
> >> Committed: DRILL-6353: Upgrade Parquet MR dependencies
> >>
> >> Committed: DRILL-6310: limit batch size for hash aggregate
> >>
> >> ===
> >>
> >> And there are few more open or in a PR state.
> >>
> >> Lets try and most of these ready by the end of the week.
> >>
> >> Boaz
> >>
> >>
> >>
>
>
>

[jira] [Created] (DRILL-6549) batch sizing for nested loop join

[Padma Penumarthy (JIRA)]

Padma Penumarthy created DRILL-6549:
---

 Summary: batch sizing for nested loop join
 Key: DRILL-6549
 URL: https://issues.apache.org/jira/browse/DRILL-6549
 Project: Apache Drill
  Issue Type: Improvement
  Components: Execution - Relational Operators
Affects Versions: 1.13.0
Reporter: Padma Penumarthy
Assignee: Padma Penumarthy
 Fix For: 1.14.0


limit output batch size for nested loop join based on memory.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Drill Hangout tomorrow 06/26

[Padma Penumarthy]

Here is the link to the document. Any feedback/comments welcome.

https://docs.google.com/document/d/1Z-67Y_KNcbA2YYWCHEwf2PUEmXRPWSXsw-CHnXW_98Q/edit?usp=sharing

Thanks
Padma


On Jun 26, 2018, at 12:12 PM, Aman Sinha 
mailto:amansi...@gmail.com>> wrote:

Hangout attendees on 06/26:
Padma, Hanumath, Boaz, Aman, Jyothsna, Sorabh, Arina, Bohdan, Vitalii,
Volodymyr, Abhishek, Robert

2 topics were discussed:
1.  Vitalii brought up the Travis timeout issue for which he has sent out
an email in this thread;  Actually Vitalli can you send it in a separate
email with explicit subject otherwise people may miss it.
2. Padma went over the batch sizing work and current status.  Padma, pls
add a link to your document.  Summarizing some of the discussion:

  - Does batch sizing affect output batches only or internal batches also
  ?  For certain operators such as HashAgg it does affect the internal
  batches held in the hash table since these batches are transferred as-is to
  the output container.
  - 16 MB limit on the batch size is a best effort but in some cases it
  could slightly exceed.  The number of rows per output batch is estimated as
  nearest lower power of 2.  For example, if based on input batch size, the
  number of output rows is 600, it will be rounded down to 512.
  - An optimization could be done in future to have upstream operator
  provide the batch size information in metadata instead of downstream
  operator computing it for each incoming.
  - There was discussion on estimating the size of complex type columns
  especially ones with nesting levels.  It would be good to add details in
  the document.


-Aman

On Tue, Jun 26, 2018 at 10:48 AM Vitalii Diravka 
mailto:vitalii.dira...@gmail.com>>
wrote:

Lately Drill Travis Build fails more often because of Travis job time
expires.
The right way is to accelerate Drill execution :)

Nevertheless I believe we should consider excluding some more tests from
Travis Build.
We can add all TPCH tests (
TestTpchLimit0, TestTpchExplain, TestTpchPlanning, TestTpchExplain) to the
SlowTest category.

Is there other solution for this issue? What are other tests are executed
very slowly?

Kind regards
Vitalii


On Tue, Jun 26, 2018 at 3:34 AM Aman Sinha 
mailto:amansi...@apache.org>> wrote:

We'll have the Drill hangout tomorrow Jun26th, 2018 at 10:00 PDT.

If you have any topics to discuss, send a reply to this post or just join
the hangout.

( Drill hangout link

 )

Re: [ANNOUNCE] New PMC member: Vitalii Diravka

[Padma Penumarthy]

Congrats Vitalii.

Thanks
Padma


> On Jun 26, 2018, at 6:14 PM, Vlad Rozov  wrote:
> 
> Congratulations Vitalii!
> 
> Thank you,
> 
> Vlad
> 
> On 6/26/18 17:11, Paul Rogers wrote:
>> Congratulations Vitalii!
>> - Paul
>> 
>>  
>> On Tuesday, June 26, 2018, 11:12:16 AM PDT, Aman Sinha 
>>  wrote:
>>I am pleased to announce that Drill PMC invited Vitalii Diravka to the PMC
>> and he has accepted the invitation.
>> 
>> Congratulations Vitalii and thanks for your contributions !
>> 
>> -Aman
>> (on behalf of Drill PMC)
>>   
> 

[jira] [Created] (DRILL-6539) Record count not set for this vector container error

[Padma Penumarthy (JIRA)]

Padma Penumarthy created DRILL-6539:
---

 Summary: Record count not set for this vector container error 
 Key: DRILL-6539
 URL: https://issues.apache.org/jira/browse/DRILL-6539
 Project: Apache Drill
  Issue Type: Bug
  Components: Execution - Relational Operators
Affects Versions: 1.13.0
Reporter: Padma Penumarthy
Assignee: Padma Penumarthy
 Fix For: 1.14.0


This error is randomly seen when executing queries.


[Error Id: 6a2a49e5-28d9-4587-ab8b-5262c07f8fdc on drill196:31010]

  (java.lang.IllegalStateException) Record count not set for this vector 
container
com.google.common.base.Preconditions.checkState():173
org.apache.drill.exec.record.VectorContainer.getRecordCount():394
org.apache.drill.exec.record.RecordBatchSizer.():681
org.apache.drill.exec.record.RecordBatchSizer.():665

org.apache.drill.exec.physical.impl.common.HashTableTemplate$BatchHolder.getActualSize():441

org.apache.drill.exec.physical.impl.common.HashTableTemplate.getActualSize():882

org.apache.drill.exec.physical.impl.common.HashTableTemplate.makeDebugString():891

org.apache.drill.exec.physical.impl.common.HashPartition.makeDebugString():578
org.apache.drill.exec.physical.impl.join.HashJoinBatch.makeDebugString():937

org.apache.drill.exec.physical.impl.join.HashJoinBatch.executeBuildPhase():754
org.apache.drill.exec.physical.impl.join.HashJoinBatch.innerNext():335
org.apache.drill.exec.record.AbstractRecordBatch.next():172
org.apache.drill.exec.record.AbstractRecordBatch.next():119
org.apache.drill.exec.record.AbstractRecordBatch.next():109
org.apache.drill.exec.record.AbstractUnaryRecordBatch.innerNext():63

org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext():137
org.apache.drill.exec.record.AbstractRecordBatch.next():172
org.apache.drill.exec.record.AbstractRecordBatch.next():119
org.apache.drill.exec.record.AbstractRecordBatch.next():109
org.apache.drill.exec.record.AbstractUnaryRecordBatch.innerNext():63

org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext():137
org.apache.drill.exec.record.AbstractRecordBatch.next():172
org.apache.drill.exec.record.AbstractRecordBatch.next():119
org.apache.drill.exec.test.generated.HashAggregatorGen89497.doWork():617
org.apache.drill.exec.physical.impl.aggregate.HashAggBatch.innerNext():176
org.apache.drill.exec.record.AbstractRecordBatch.next():172
org.apache.drill.exec.record.AbstractRecordBatch.next():119
org.apache.drill.exec.test.generated.HashAggregatorGen89497.doWork():617
org.apache.drill.exec.physical.impl.aggregate.HashAggBatch.innerNext():176
org.apache.drill.exec.record.AbstractRecordBatch.next():172
org.apache.drill.exec.record.AbstractRecordBatch.next():119
org.apache.drill.exec.record.AbstractRecordBatch.next():109

org.apache.drill.exec.physical.impl.xsort.managed.ExternalSortBatch.loadBatch():403

org.apache.drill.exec.physical.impl.xsort.managed.ExternalSortBatch.load():354

org.apache.drill.exec.physical.impl.xsort.managed.ExternalSortBatch.innerNext():299
org.apache.drill.exec.record.AbstractRecordBatch.next():172
org.apache.drill.exec.record.AbstractRecordBatch.next():119
org.apache.drill.exec.record.AbstractRecordBatch.next():109
org.apache.drill.exec.record.AbstractUnaryRecordBatch.innerNext():63
org.apache.drill.exec.record.AbstractRecordBatch.next():172
org.apache.drill.exec.record.AbstractRecordBatch.next():119
org.apache.drill.exec.record.AbstractRecordBatch.next():109
org.apache.drill.exec.record.AbstractUnaryRecordBatch.innerNext():63

org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext():137
org.apache.drill.exec.record.AbstractRecordBatch.next():172
org.apache.drill.exec.physical.impl.BaseRootExec.next():103
org.apache.drill.exec.physical.impl.ScreenCreator$ScreenRoot.innerNext():83
org.apache.drill.exec.physical.impl.BaseRootExec.next():93
org.apache.drill.exec.work.fragment.FragmentExecutor$1.run():294
org.apache.drill.exec.work.fragment.FragmentExecutor$1.run():281
java.security.AccessController.doPrivileged():-2
javax.security.auth.Subject.doAs():422
org.apache.hadoop.security.UserGroupInformation.doAs():1595
org.apache.drill.exec.work.fragment.FragmentExecutor.run():281
org.apache.drill.common.SelfCleaningRunnable.run():38
java.util.concurrent.ThreadPoolExecutor.runWorker():1149
java.util.concurrent.ThreadPoolExecutor$Worker.run():624
java.lang.Thread.run():748



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (DRILL-6537) Limit the batch size for buffering operators based on how much memory they get

[Padma Penumarthy (JIRA)]

Padma Penumarthy created DRILL-6537:
---

 Summary: Limit the batch size for buffering operators based on how 
much memory they get
 Key: DRILL-6537
 URL: https://issues.apache.org/jira/browse/DRILL-6537
 Project: Apache Drill
  Issue Type: Bug
  Components: Execution - Relational Operators
Affects Versions: 1.13.0
Reporter: Padma Penumarthy
Assignee: Padma Penumarthy
 Fix For: 1.14.0


we are using 16MB for output batch size for all operators by default. however, 
for buffering operators, depending upon how much memory they get allocated, it 
is possible that 16MB might be too much to use for output batch size, causing 
them to spill sometimes.

Have output batch size to be minimum of 16 MB or 20% of the allocated memory.

 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (DRILL-6427) outputBatchSize is missing from the DEBUG output for HashJoinBatch operator

[Padma Penumarthy (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-6427?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Padma Penumarthy resolved DRILL-6427.
-
Resolution: Fixed

> outputBatchSize is missing from the DEBUG output for HashJoinBatch operator 
> 
>
> Key: DRILL-6427
> URL: https://issues.apache.org/jira/browse/DRILL-6427
> Project: Apache Drill
>  Issue Type: Bug
>Reporter: Khurram Faraaz
>    Assignee: Padma Penumarthy
>Priority: Minor
>
> Drill 1.14.0-SNAPSHOT commit : f99d1f1323c0a5bac99842d6283d3025f3cb527f 
> outputBatchSize is missing from the DEBUG output for HashJoinBatch operator 
> Query used in test,
> {noformat}
> select count(*) from `twovarchar_asc_128MB.parquet` t1, 
> `twovarchar_asc_16MB.parquet` t2 WHERE t1.Varbinaryvalue = t2.Varbinaryvalue
> {noformat} 
>  
> Snippet from drillbit.log 
> {noformat}
> 2018-05-18 11:23:59,655 [2500e5c3-8f54-1f92-6eeb-7a81499a8abd:frag:0:0] DEBUG 
> o.a.d.e.p.impl.join.HashJoinBatch - left input: batch count : 1, avg batch 
> bytes : 90145920, avg row bytes : 8166, record count : 11040
> 2018-05-18 11:23:59,655 [2500e5c3-8f54-1f92-6eeb-7a81499a8abd:frag:0:0] DEBUG 
> o.a.d.e.p.impl.join.HashJoinBatch - right input: batch count : 1, avg batch 
> bytes : 10567808, avg row bytes : 7506, record count : 1408
> 2018-05-18 11:23:59,655 [2500e5c3-8f54-1f92-6eeb-7a81499a8abd:frag:0:0] DEBUG 
> o.a.d.e.p.impl.join.HashJoinBatch - output: batch count : 166, avg batch 
> bytes : 15951453, avg row bytes : 15672, record count : 168960
> {noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (DRILL-6512) Remove unnecessary processing overhead from RecordBatchSizer

[Padma Penumarthy (JIRA)]

Padma Penumarthy created DRILL-6512:
---

 Summary: Remove unnecessary processing overhead from 
RecordBatchSizer
 Key: DRILL-6512
 URL: https://issues.apache.org/jira/browse/DRILL-6512
 Project: Apache Drill
  Issue Type: Bug
  Components: Execution - Relational Operators
Affects Versions: 1.13.0
Reporter: Padma Penumarthy
Assignee: Padma Penumarthy
 Fix For: 1.14.0


record batch sizer collects lot of information about the record batch. Since it 
is used now in every operator, for every batch, it makes sense to make it as 
efficient as possible. Remove anything that is not needed and also, may be 
provide two options one light weight and another which is more comprehensive. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [ANNOUNCE] New Committer: Padma Penumarthy

[Padma Penumarthy]

Thanks to all for your wishes.
To become a committer means a lot to me and I hope to continue making
significant contributions to drill.

Thanks
Padma


> On Jun 15, 2018, at 1:15 PM, Boaz Ben-Zvi  wrote:
> 
> Congratulations Padma; welcome to our “club” 
> 
> On 6/15/18, 12:47 PM, "Jyothsna Reddy"  wrote:
> 
>Congratulations, Padma !!
> 
> 
> 
>On Fri, Jun 15, 2018 at 12:39 PM, AnilKumar B  
> wrote:
> 
>> Congratulations, Padma
>> 
>> On Fri, Jun 15, 2018 at 12:36 PM Kunal Khatua  wrote:
>> 
>>> Congratulations, Padma !
>>> 
>>> 
>>> On 6/15/2018 12:34:15 PM, Robert Wu  wrote:
>>> Congratulations, Padma!
>>> 
>>> Best regards,
>>> 
>>> Rob
>>> 
>>> -----Original Message-
>>> From: Hanumath Rao Maduri
>>> Sent: Friday, June 15, 2018 12:25 PM
>>> To: dev@drill.apache.org
>>> Subject: Re: [ANNOUNCE] New Committer: Padma Penumarthy
>>> 
>>> Congratulations Padma!
>>> 
>>> On Fri, Jun 15, 2018 at 12:04 PM, Gautam Parai wrote:
>>> 
>>>> Congratulations Padma!!
>>>> 
>>>> 
>>>> Gautam
>>>> 
>>>> 
>>>> From: Vlad Rozov
>>>> Sent: Friday, June 15, 2018 11:56:37 AM
>>>> To: dev@drill.apache.org
>>>> Subject: Re: [ANNOUNCE] New Committer: Padma Penumarthy
>>>> 
>>>> Congrats Padma!
>>>> 
>>>> Thank you,
>>>> 
>>>> Vlad
>>>> 
>>>> On 6/15/18 11:38, Charles Givre wrote:
>>>>> Congrats Padma!!
>>>>> 
>>>>>> On Jun 15, 2018, at 13:57, Bridget Bevens wrote:
>>>>>> 
>>>>>> Congratulations, Padma!!! 
>>>>>> 
>>>>>> 
>>>>>> From: Prasad Nagaraj Subramanya
>>>>>> Sent: Friday, June 15, 2018 10:32:04 AM
>>>>>> To: dev@drill.apache.org
>>>>>> Subject: Re: [ANNOUNCE] New Committer: Padma Penumarthy
>>>>>> 
>>>>>> Congratulations Padma!
>>>>>> 
>>>>>> Thanks,
>>>>>> Prasad
>>>>>> 
>>>>>> On Fri, Jun 15, 2018 at 9:59 AM Vitalii Diravka <>
>>>> vitalii.dira...@gmail.com>
>>>>>> wrote:
>>>>>> 
>>>>>>> Congrats Padma!
>>>>>>> 
>>>>>>> Kind regards
>>>>>>> Vitalii
>>>>>>> 
>>>>>>> 
>>>>>>> On Fri, Jun 15, 2018 at 7:40 PM Arina Ielchiieva
>>>>>>> 
>>>> wrote:
>>>>>>> 
>>>>>>>> Padma, congratulations and welcome!
>>>>>>>> 
>>>>>>>> Kind regards,
>>>>>>>> Arina
>>>>>>>> 
>>>>>>>> On Fri, Jun 15, 2018 at 7:36 PM Aman Sinha
>>>> wrote:
>>>>>>>> 
>>>>>>>>> The Project Management Committee (PMC) for Apache Drill has
>>>>>>>>> invited
>>>>>>> Padma
>>>>>>>>> Penumarthy to become a committer, and we are pleased to announce
>>>>>>>>> that
>>>>>>> she
>>>>>>>>> has
>>>>>>>>> accepted.
>>>>>>>>> 
>>>>>>>>> Padma has been contributing to Drill for about 1 1/2 years. She
>>>>>>>>> has
>>>>>>> made
>>>>>>>>> improvements for work-unit assignment in the parallelizer,
>>>> performance
>>>>>>> of
>>>>>>>>> filter operator for pattern matching and (more recently) on the
>>>>>>>>> batch sizing for several operators: Flatten, MergeJoin, HashJoin,
>>> UnionAll.
>>>>>>>>> 
>>>>>>>>> Welcome Padma, and thank you for your contributions. Keep up
>>>>>>>>> the
>>>> good
>>>>>>>> work
>>>>>>>>> !
>>>>>>>>> 
>>>>>>>>> -Aman
>>>>>>>>> (on behalf of Drill PMC)
>>>>>>>>> 
>>>> 
>>>> 
>>> 
>> --
>> Thanks & Regards,
>> B Anil Kumar.
>> 
> 
> 

[jira] [Created] (DRILL-6499) No need to calculate stdRowWidth for every batch by default

[Padma Penumarthy (JIRA)]

Padma Penumarthy created DRILL-6499:
---

 Summary: No need to calculate stdRowWidth for every batch by 
default
 Key: DRILL-6499
 URL: https://issues.apache.org/jira/browse/DRILL-6499
 Project: Apache Drill
  Issue Type: Bug
  Components: Execution - Relational Operators
Affects Versions: 1.13.0
Reporter: Padma Penumarthy
Assignee: Padma Penumarthy
 Fix For: 1.14.0


RecordBatchSizer calculates stdRowWidth by default. This is an expensive 
operation especially if the columns are deeply nested. Remove this from 
RecordBatchSizer and instead, provide a method to get the value if needed.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [DISCUSS] 1.14.0 release

[Padma Penumarthy]

I am planning to open couple of batch sizing PRs this week that I would like to 
get in.

Thanks
Padma


> On Jun 13, 2018, at 11:59 AM, Vlad Rozov  wrote:
> 
> DRILL-6422: Update guava to 23.0 and shade it (PR in review)
> DRILL-6353: Upgrade Parquet MR dependencies (ready-to-commit)
> 
> Thank you,
> 
> Vlad
> 
> On 6/12/18 17:16, Boaz Ben-Zvi wrote:
>> Hello Drillers,
>>   Nearly three months have passed since the 1.13.0 release, and it is time 
>> to start planning for the 1.14.0 ; I volunteer to manage the new release.
>>   If there is any ongoing work not yet committed into the Apache Drill 
>> master, that you strongly feel MUST be included in the 1.14 release, please 
>> reply to this thread. There are quite a few pending PRs, and we should 
>> prioritize and close the needed ones soon enough.
>> Thanks,
>>Boaz
>> 
> 

[jira] [Created] (DRILL-6478) enhance debug logs for batch sizing

[Padma Penumarthy (JIRA)]

Padma Penumarthy created DRILL-6478:
---

 Summary: enhance debug logs for batch sizing
 Key: DRILL-6478
 URL: https://issues.apache.org/jira/browse/DRILL-6478
 Project: Apache Drill
  Issue Type: Bug
Reporter: Padma Penumarthy
Assignee: Padma Penumarthy


Fix some issues with debug logs so QA  scripts work better. Also, added batch 
sizing logs for union all.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (DRILL-6274) MergeJoin Memory Manager is still using Fragmentation Factor

[Padma Penumarthy (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-6274?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Padma Penumarthy resolved DRILL-6274.
-
Resolution: Fixed

> MergeJoin Memory Manager is still using Fragmentation Factor
> 
>
> Key: DRILL-6274
> URL: https://issues.apache.org/jira/browse/DRILL-6274
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Execution - Flow
>Affects Versions: 1.13.0
>Reporter: Sorabh Hamirwasia
>    Assignee: Padma Penumarthy
>Priority: Major
> Fix For: 1.14.0
>
>
> MergeJoinMemoryManager is using 
> [WORST_CASE_FRAGMENTATION_FACTOR|https://github.com/apache/drill/blob/master/exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/join/MergeJoinBatch.java#L156]
>  for memory computation in outgoing batch. This needs to be updated to not 
> use it anymore.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (DRILL-6298) Add debug log for merge join batch sizing

[Padma Penumarthy (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-6298?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Padma Penumarthy resolved DRILL-6298.
-
Resolution: Fixed

> Add debug log for merge join batch sizing
> -
>
> Key: DRILL-6298
> URL: https://issues.apache.org/jira/browse/DRILL-6298
> Project: Apache Drill
>  Issue Type: Bug
>Affects Versions: 1.13.0
>    Reporter: Padma Penumarthy
>Assignee: Padma Penumarthy
>Priority: Major
> Fix For: 1.14.0
>
>
> Add debug log for merge join batch sizers so QA can verify the batch sizes.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [ANNOUNCE] New Committer: Timothy Farkas

[Padma Penumarthy]

Congrats Tim.

Thanks
Padma


> On May 25, 2018, at 12:15 PM, Vitalii Diravka  
> wrote:
> 
> Good news! Congratulations, Timothy!
> 
> Kind regards
> Vitalii
> 
> 
> On Fri, May 25, 2018 at 10:04 PM Arina Yelchiyeva <
> arina.yelchiy...@gmail.com> wrote:
> 
>> Congrats, Tim!
>> 
>> Kind regards,
>> Arina
>> 
>>> On May 25, 2018, at 9:59 PM, Kunal Khatua  wrote:
>>> 
>>> Congratulations, Timothy !
>>> 
>>> On 5/25/2018 11:58:31 AM, Aman Sinha  wrote:
>>> The Project Management Committee (PMC) for Apache Drill has invited
>> Timothy
>>> Farkas to become a committer, and we are pleased to announce that he
>>> has accepted.
>>> 
>>> Tim has become an active contributor to Drill in less than a year. During
>>> this time he has contributed to addressing flaky unit tests, fixing
>> memory
>>> leaks in certain operators, enhancing the system options framework to be
>>> more extensible and setting up the Travis CI tests. More recently, he
>>> worked on the memory sizing calculations for hash join.
>>> 
>>> Welcome Tim, and thank you for your contributions. Keep up the good work
>> !
>>> 
>>> -Aman
>>> (on behalf of Drill PMC)
>> 

[jira] [Resolved] (DRILL-6411) Make batch memory sizing logs uniform across all operators

[Padma Penumarthy (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-6411?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Padma Penumarthy resolved DRILL-6411.
-
Resolution: Fixed

> Make batch memory sizing logs uniform across all operators
> --
>
> Key: DRILL-6411
> URL: https://issues.apache.org/jira/browse/DRILL-6411
> Project: Apache Drill
>  Issue Type: Bug
>Affects Versions: 1.13.0
>    Reporter: Padma Penumarthy
>Assignee: Padma Penumarthy
>Priority: Major
>  Labels: ready-to-commit
> Fix For: 1.14.0
>
>
> Make batch memory sizing logs uniform across all operators so QA can parse 
> and verify easily. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (DRILL-6411) Make batch memory sizing logs uniform across all operators

[Padma Penumarthy (JIRA)]

Padma Penumarthy created DRILL-6411:
---

 Summary: Make batch memory sizing logs uniform across all operators
 Key: DRILL-6411
 URL: https://issues.apache.org/jira/browse/DRILL-6411
 Project: Apache Drill
  Issue Type: Bug
Affects Versions: 1.13.0
Reporter: Padma Penumarthy
Assignee: Padma Penumarthy
 Fix For: 1.14.0


Make batch memory sizing logs uniform across all operators so QA can parse and 
verify easily. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (DRILL-6402) Repeated Value Vectors copyFrom methods are not updating the value count and writer index correctly for values vector

[Padma Penumarthy (JIRA)]

Padma Penumarthy created DRILL-6402:
---

 Summary: Repeated Value Vectors copyFrom methods are not updating 
the value count and writer index correctly for values vector
 Key: DRILL-6402
 URL: https://issues.apache.org/jira/browse/DRILL-6402
 Project: Apache Drill
  Issue Type: Bug
Affects Versions: 1.13.0
Reporter: Padma Penumarthy
Assignee: Padma Penumarthy
 Fix For: 1.14.0


copyFrom and copyFromSafe methods of repeated value vectors do not update the 
value count after values are copied. We update before the copy starts. Offset 
vector value count is updated correctly. But, values vector value count is not 
updated. So, writer index for values vector will have wrong value and we get 
index out of bounds error when trying to read after copy. This problem is seen 
when we do split and transfer of repeated value vector. 

 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [ANNOUNCE] New Committer: Sorabh Hamirwasia

[Padma Penumarthy]

Congrats Sorabh.

Thanks
Padma


> On Apr 30, 2018, at 11:53 AM, Gautam Parai  wrote:
> 
> Congratulations Sorabh! Well deserved.
> 
> 
> Gautam
> 
> 
> From: rahul challapalli 
> Sent: Monday, April 30, 2018 11:09:24 AM
> To: dev
> Subject: Re: [ANNOUNCE] New Committer: Sorabh Hamirwasia
> 
> Congratulations Sorabh!
> 
> On Mon, Apr 30, 2018 at 11:07 AM, Khurram Faraaz  wrote:
> 
>> Congratulations Sorabh!
>> 
>> 
>> From: Andries Engelbrecht 
>> Sent: Monday, April 30, 2018 11:04:11 AM
>> To: dev@drill.apache.org
>> Subject: Re: [ANNOUNCE] New Committer: Sorabh Hamirwasia
>> 
>> Congrats Sorabh!!!
>> 
>> --Andries
>> 
>> On 4/30/18, 8:35 AM, "Aman Sinha"  wrote:
>> 
>>The Project Management Committee (PMC) for Apache Drill has invited
>> Sorabh
>>Hamirwasia  to become a committer, and we are pleased to announce that
>> he
>>has accepted.
>> 
>>Over the last 1 1/2 years Sorabh's contributions have been in a few
>>different areas. He took
>>the lead in designing and implementing network encryption support for
>>Drill. He has contributed
>>to the web server and UI side.  More recently, he is involved in
>> design and
>>implementation of the lateral join operator.
>> 
>>Welcome Sorabh, and thank you for your contributions.  Keep up the good
>>work !
>> 
>>-Aman
>>(on behalf of Drill PMC)
>> 
>> 
>> 

[jira] [Created] (DRILL-6356) batch sizing for union all

[Padma Penumarthy (JIRA)]

Padma Penumarthy created DRILL-6356:
---

 Summary: batch sizing for union all
 Key: DRILL-6356
 URL: https://issues.apache.org/jira/browse/DRILL-6356
 Project: Apache Drill
  Issue Type: Improvement
  Components: Execution - Relational Operators
Affects Versions: 1.13.0
Reporter: Padma Penumarthy
Assignee: Padma Penumarthy
 Fix For: 1.14.0


batch sizing changes for union all operator



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (DRILL-6343) bit vector copyFromSafe is not doing realloc

[Padma Penumarthy (JIRA)]

Padma Penumarthy created DRILL-6343:
---

 Summary: bit vector copyFromSafe is not doing realloc 
 Key: DRILL-6343
 URL: https://issues.apache.org/jira/browse/DRILL-6343
 Project: Apache Drill
  Issue Type: Bug
  Components: Execution - Flow
Affects Versions: 1.13.0
Reporter: Padma Penumarthy
Assignee: Padma Penumarthy
 Fix For: 1.14.0


bit vector copyFromSafe simply returns false when asked to copy to an index 
greater than it's capacity. It should do realloc if needed and copy correctly.

This bug is very subtle and shows up as random wrong result. allocateNew for 
bit vector allocates 4096 bytes initially and if we do copyFromSafe to any 
index greater than that, we basically do not do any copy, but just return 
false. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [DISCUSS] Regarding mutator interface

[Padma Penumarthy]

I guess you can add a setSafe method which recursively does setSafe for all 
children.

Thanks
Padma


> On Apr 11, 2018, at 1:19 PM, Gautam Parai <gpa...@mapr.com> wrote:
> 
> Hi Paul/Padma,
> 
> 
> Thank you so much for the responses. This function is supposed to return `any 
> value` from the batch of incoming rows. Hence, the need to handle maps/lists.
> 
> 
> This codegen is for the StreamingAggregator for Complex type(e.g. maps) in 
> the incoming batch. It is trying to assign the values in the ComplexHolder to 
> the outgoing MapVector.
> 
> 
> MapVector vv9; // Output VV of StreamingAgg
> 
> 
> 
> 
>public void outputRecordValues(int outIndex)
> 
>throws SchemaChangeException
> 
>{
> 
>{
> 
>ComplexHolder out8;
> 
>{
> 
>final ComplexHolder out = new ComplexHolder();
> 
>FieldReader fr = work0;
> 
>MapHolder value = work1;
> 
>BigIntHolder nonNullCount = work2;
> 
> 
> 
> AnyValueComplexFunctions$MapAnyValue_output: {
> 
>out.reader = fr;
> 
> }
> 
> 
> 
>work0 = fr;
> 
>work1 = value;
> 
>work2 = nonNullCount;
> 
>out8 = out;
> 
>}
> 
>vv9 .getMutator().setSafe((outIndex), out8); //Don't have setSafe 
> for MapVector
> 
>}
> 
>}
> 
> 
> Please let me know your thoughts.
> 
> 
> Gautam
> 
> 
> 
> 
> From: Paul Rogers <par0...@yahoo.com.INVALID>
> Sent: Wednesday, April 11, 2018 12:40:15 PM
> To: dev@drill.apache.org
> Subject: Re: [DISCUSS] Regarding mutator interface
> 
> Note that, for maps and lists, there is nothing to set. Maps are purely 
> containers for other vectors. Lists (you didn't mention whether "repeated" or 
> "non-repeated") are also containers. Non-repeated lists are containers for 
> unions, repeated-lists are containers for arrays.
> Any setting should be done on the contained vectors. For lists, only the 
> offset vector is updated.
> So, another question is: what is the generated code trying to set?
> 
> Thanks,
> - Paul
> 
> 
> 
>On Wednesday, April 11, 2018, 12:33:52 PM PDT, Padma Penumarthy 
> <ppenumar...@mapr.com> wrote:
> 
> Can you explain how aggregation on complex type works (or supposed to work).
> 
> Thanks
> Padma
> 
> 
>> On Apr 11, 2018, at 12:15 PM, Gautam Parai <gpa...@mapr.com> wrote:
>> 
>> Hi all,
>> 
>> 
>> I am implementing a new aggregate function which also handles Complex types 
>> (map and list). However, the codegen barfs with
>> 
>> 
>> CompileException: Line 104, Column 39: A method named "setSafe" is not 
>> declared in any enclosing class nor any supertype, nor through a static 
>> import
>> 
>> 
>> It looks like we do not have set()/ setSafe() methods for 
>> MapVector/ListVector mutators.
>> 
>> 
>> Should we add these methods to the Mutator interface to ensure all mutators 
>> implement them? Is these a reason we chose not to do so?
>> 
>> 
>> Please let me know your thoughts. Thanks!
>> 
>> 
>> Gautam
> 

Re: [DISCUSS] Regarding mutator interface

[Padma Penumarthy]

Can you explain how aggregation on complex type works (or supposed to work).

Thanks
Padma


> On Apr 11, 2018, at 12:15 PM, Gautam Parai  wrote:
> 
> Hi all,
> 
> 
> I am implementing a new aggregate function which also handles Complex types 
> (map and list). However, the codegen barfs with
> 
> 
> CompileException: Line 104, Column 39: A method named "setSafe" is not 
> declared in any enclosing class nor any supertype, nor through a static import
> 
> 
> It looks like we do not have set()/ setSafe() methods for 
> MapVector/ListVector mutators.
> 
> 
> Should we add these methods to the Mutator interface to ensure all mutators 
> implement them? Is these a reason we chose not to do so?
> 
> 
> Please let me know your thoughts. Thanks!
> 
> 
> Gautam

[jira] [Created] (DRILL-6310) limit batch size for hash aggregate

[Padma Penumarthy (JIRA)]

Padma Penumarthy created DRILL-6310:
---

 Summary: limit batch size for hash aggregate
 Key: DRILL-6310
 URL: https://issues.apache.org/jira/browse/DRILL-6310
 Project: Apache Drill
  Issue Type: Bug
  Components: Execution - Flow
Affects Versions: 1.13.0
Reporter: Padma Penumarthy
Assignee: Padma Penumarthy
 Fix For: 1.14.0


limit batch size for hash aggregate based on memory.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (DRILL-6307) Handle empty batches in record batch sizer correctly

[Padma Penumarthy (JIRA)]

Padma Penumarthy created DRILL-6307:
---

 Summary: Handle empty batches in record batch sizer correctly
 Key: DRILL-6307
 URL: https://issues.apache.org/jira/browse/DRILL-6307
 Project: Apache Drill
  Issue Type: Bug
  Components: Execution - Flow
Affects Versions: 1.13.0
Reporter: Padma Penumarthy
Assignee: Padma Penumarthy
 Fix For: 1.14.0


when we get empty batch, record batch sizer calculates row width as zero. In 
that case, we do not do accounting and memory allocation correctly for outgoing 
batches. 

For example, in merge join, for outer left join, if right side batch is empty, 
we still have to include the right side columns as null in outgoing batch. We 
have to allocate memory for those vectors correctly. We also have to include 
row width of those columns in the outgoing row width and number of rows 
calculation. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (DRILL-6298) Add debug log for merge join batch sizing

[Padma Penumarthy (JIRA)]

Padma Penumarthy created DRILL-6298:
---

 Summary: Add debug log for merge join batch sizing
 Key: DRILL-6298
 URL: https://issues.apache.org/jira/browse/DRILL-6298
 Project: Apache Drill
  Issue Type: Bug
Affects Versions: 1.13.0
Reporter: Padma Penumarthy
Assignee: Padma Penumarthy
 Fix For: 1.14.0


Add debug log for merge join batch sizers so QA can verify the batch sizes.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (DRILL-6296) Add operator metrics for batch sizing for merge join

[Padma Penumarthy (JIRA)]

Padma Penumarthy created DRILL-6296:
---

 Summary: Add operator metrics for batch sizing for merge join
 Key: DRILL-6296
 URL: https://issues.apache.org/jira/browse/DRILL-6296
 Project: Apache Drill
  Issue Type: Improvement
  Components: Execution - Relational Operators
Affects Versions: 1.13.0
Reporter: Padma Penumarthy
Assignee: Padma Penumarthy
 Fix For: 1.14.0


Add operator metrics for batch sizing stats for merge join.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (DRILL-6284) Add operator metrics for batch sizing for flatten

[Padma Penumarthy (JIRA)]

Padma Penumarthy created DRILL-6284:
---

 Summary: Add operator metrics for batch sizing for flatten
 Key: DRILL-6284
 URL: https://issues.apache.org/jira/browse/DRILL-6284
 Project: Apache Drill
  Issue Type: Improvement
  Components: Execution - Flow
Affects Versions: 1.13.0
Reporter: Padma Penumarthy
Assignee: Padma Penumarthy
 Fix For: 1.14.0


Add the following operator metrics for flatten.

NUM_INCOMING_BATCHES,
AVG_INPUT_BATCH_SIZE,
AVG_INPUT_ROW_WIDTH,
TOTAL_INPUT_RECORDS,
NUM_OUTGOING_BATCHES,
AVG_OUTPUT_BATCH_SIZE,
AVG_OUTPUT_ROW_WIDTH,
TOTAL_OUTPUT_RECORDS;

 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

batch sizing for operators

[Padma Penumarthy]

I have written a small design document explaining the batch sizing work we are
doing for operators (other than scan).

https://issues.apache.org/jira/browse/DRILL-6238

Some of this work has already been done in 1.13.
flatten, merge join and external sort are changed to adhere to batch size 
limits.
The work will continue in future releases.

I would like to get feedback on the approach being used.
All comments are welcome.
Please update the JIRA or respond to this email with your comments.

Thanks
Padma

[jira] [Created] (DRILL-6238) Batch sizing for operators

[Padma Penumarthy (JIRA)]

Padma Penumarthy created DRILL-6238:
---

 Summary: Batch sizing for operators
 Key: DRILL-6238
 URL: https://issues.apache.org/jira/browse/DRILL-6238
 Project: Apache Drill
  Issue Type: New Feature
Reporter: Padma Penumarthy
Assignee: Padma Penumarthy


*Batch Sizing For Operators*

This document describes the approach we are taking for limiting batch sizes for 
operators other than scan. 

*Motivation*

Main goals are
 # Improve concurrency 
 # Reduce query failures because of out of memory errors

To accomplish these goals, we need to make queries execute within a specified 
memory budget. To enforce per query memory limit, we need to be able to enforce 
per fragment and per operator memory limits. Controlling individual operators 
batch sizes is the first step towards all this. 

*Background*

In Drill, different operators have different limits w.r.to outgoing batches. 
Some use hard coded row counts, some use hard coded memory and some have none 
at all. Based on input data size and what the operator is doing, memory used by 
the outgoing batch can vary widely as there are no limits imposed. Queries fail 
because we are not able to allocate the memory needed. Some operators produce 
very large batches, causing blocking operators like sort, hash agg which have 
to work under tight memory constraints to fail. Size of batches should be a 
function of available memory rather than input data size and/or what the 
operator does. Please refer to table at the end of this document for details on 
what each operator does today.

*Design*

Goal is to have all operators behave the same way i.e. produce batches with 
size less than or equal to configured outgoing batch size with a minimum of 1 
row per batch and maximum of 64k rows per batch. A new system option 
‘drill.exec.memory.operator.output_batch_size’ is added which has default value 
of 16MB.

The basic idea is to limit size of outgoing batch by deciding how many rows we 
can have in the batch based on average entry size of each outgoing column, 
taking into account actual data size and metadata vector overhead we add on top 
for tracking variable length, mode(repeated, optional, required) etc. This 
calculation will be different for each operator, based on what the operator is 
doing, incoming data and what is being projected out. 

By taking this adaptive approach based on actual data sizes, for operators 
which were limiting batch size to something less than 64K before can possibly 
do lot more rows (upto 64K) in a batch if the memory stays within the budget. 
This should help improve performance.

Also, to improve performance and utilize memory more efficiently, we will
 # Allocate memory for value vectors upfront. Since we know the number of rows 
and sizing information for each column in the  outgoing batch, we will use that 
information to allocate memory for value vectors upfront. This will help 
improve performance by reducing the memory copies and zeroing the new half we 
do every time we double.
 # Make the number of rows in outgoing batch a power of two. Since memory is 
allocated in powers of two, this will help us pack the value vectors densely 
thereby reducing the amount of memory that gets wasted because of doubling 
effect.

So, to summarize, the benefits we will get are improved memory utilization, 
better performance, higher concurrency and less queries dying because of out of 
memory errors. 

So, what are the cons ? 
 * Since this is based on averages, strict enforcement is not possible. There 
could be pathological cases where because of uneven data distribution, we might 
exceed the configured output batch size potentially causing OOM errors and 
problems in downstream operators.

Other issues that will be addressed:
 * We are adding extra processing for each batch in each operator to figure out 
the sizing information. This overhead can be reduced by passing this 
information along with the batch between operators. 
 * For some operators, it will be complex to figure out average size of 
outgoing columns especially if we have to evaluate complex expression trees and 
UDFs to figure out the transformation on incoming batches. We will use 
approximations as appropriate.

Following table summarizes the limits we have today for each operator. 

flatten, merge join and external sort have already been changed to adhere to 
batch size limits as described in this document as of drill release 1.13.

 
|*Operator*|*Limit* 
*(Rows, Memory)*|*Notes*|
|Flatten|4K, 512MB|Flatten can produce very large batches based on average 
cardinality of the flatten column. |
|Merge Receiver|32K|No memory limit. |
|Hash Aggregate|64K|No memory limit.|
|Streaming Aggregate|32K|No memory limit.|
|Broadcast Sender|None|No limits. |
|Filter, Limit|None|No limits.|
|Hash Join|4K|No memory limit|
|Merge Join|4K|No memory limit|
|Nested

[jira] [Created] (DRILL-6236) batch sizing for hash join and hash aggregate

[Padma Penumarthy (JIRA)]

Padma Penumarthy created DRILL-6236:
---

 Summary: batch sizing for hash join and hash aggregate
 Key: DRILL-6236
 URL: https://issues.apache.org/jira/browse/DRILL-6236
 Project: Apache Drill
  Issue Type: Improvement
  Components: Execution - Flow
Affects Versions: 1.13.0
Reporter: Padma Penumarthy
Assignee: Padma Penumarthy
 Fix For: 1.14.0


limit output batch size for hash join and hash aggregate based on memory.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (DRILL-6232) Vector initializer used for memory allocation in external sort is subject to aliasing

[Padma Penumarthy (JIRA)]

Padma Penumarthy created DRILL-6232:
---

 Summary: Vector initializer used for memory allocation in external 
sort is subject to aliasing
 Key: DRILL-6232
 URL: https://issues.apache.org/jira/browse/DRILL-6232
 Project: Apache Drill
  Issue Type: Bug
  Components: Execution - Flow
Affects Versions: 1.13.0
Reporter: Padma Penumarthy
Assignee: Padma Penumarthy
 Fix For: 1.14.0


vector initializer class builds hints for individual columns and saves them in 
a map with lookup key of column name. For children of map columns, this name 
will be parent name + "." + child column name. This can cause problem when we 
have dots in column names i.e. {{a.b(c) and a(b.c) will have the same key and 
can cause overwrites of each other values. Change this to use allocation 
routines in column size which do not have this problem. }}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (DRILL-6231) Fix memory allocation for repeated list vector

[Padma Penumarthy (JIRA)]

Padma Penumarthy created DRILL-6231:
---

 Summary: Fix memory allocation for repeated list vector
 Key: DRILL-6231
 URL: https://issues.apache.org/jira/browse/DRILL-6231
 Project: Apache Drill
  Issue Type: Improvement
  Components: Execution - Flow
Affects Versions: 1.13.0
Reporter: Padma Penumarthy
Assignee: Padma Penumarthy
 Fix For: 1.14.0


Helper routines in record batch sizer and AllocationHelper can be enhanced to 
allocate memory for repeated list vector more accurately rather than using 
default functions. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (DRILL-6206) RowSetBuilder does not support repeated list

[Padma Penumarthy (JIRA)]

Padma Penumarthy created DRILL-6206:
---

 Summary: RowSetBuilder does not support repeated list
 Key: DRILL-6206
 URL: https://issues.apache.org/jira/browse/DRILL-6206
 Project: Apache Drill
  Issue Type: Bug
  Components: Execution - Flow
Affects Versions: 1.12.0
Reporter: Padma Penumarthy


Fails with the following exception.

java.lang.UnsupportedOperationException: Unable to get size for minor type 
[LIST] and mode [REPEATED]

at org.apache.drill.exec.expr.BasicTypeHelper.getSize(BasicTypeHelper.java:163)
 at 
org.apache.drill.exec.record.TupleSchema$PrimitiveColumnMetadata.(TupleSchema.java:178)
 at org.apache.drill.exec.record.TupleSchema.fromField(TupleSchema.java:341)
 at org.apache.drill.exec.record.TupleSchema.add(TupleSchema.java:390)
 at org.apache.drill.test.rowSet.SchemaBuilder.add(SchemaBuilder.java:163)
 at org.apache.drill.test.rowSet.SchemaBuilder.add(SchemaBuilder.java:183)
 at org.apache.drill.test.rowSet.SchemaBuilder.addArray(SchemaBuilder.java:211)
 at 
org.apache.drill.exec.record.TestRecordBatchSizer.testSizerRepeatedList(TestRecordBatchSizer.java:631)
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 at java.lang.reflect.Method.invoke(Method.java:497)
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 at java.lang.reflect.Method.invoke(Method.java:497)
 at java.util.concurrent.FutureTask.run(FutureTask.java:266)
 at java.lang.Thread.run(Thread.java:745)


Process finished with exit code 255



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (DRILL-6205) Reduce memory consumption of testFlattenUpperLimit test

[Padma Penumarthy (JIRA)]

Padma Penumarthy created DRILL-6205:
---

 Summary: Reduce memory consumption of testFlattenUpperLimit test
 Key: DRILL-6205
 URL: https://issues.apache.org/jira/browse/DRILL-6205
 Project: Apache Drill
  Issue Type: Bug
  Components: Execution - Flow
Affects Versions: 1.12.0
Reporter: Padma Penumarthy
Assignee: Padma Penumarthy


Reduce memory usage of testFlattenUpperLimit test - decrease total number of 
rows and remove columns. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [DISCUSS] 1.13.0 release

[Padma Penumarthy]

I can look at failure in testFlattenUpperLimit. I added this test recently.

Thanks
Padma


On Mar 2, 2018, at 8:19 AM, Volodymyr Tkach 
> wrote:

testFlattenUpperLimit

[jira] [Created] (DRILL-6203) Repeated Map Vector does not give correct payload bytecount

[Padma Penumarthy (JIRA)]

Padma Penumarthy created DRILL-6203:
---

 Summary: Repeated Map Vector does not give correct payload 
bytecount
 Key: DRILL-6203
 URL: https://issues.apache.org/jira/browse/DRILL-6203
 Project: Apache Drill
  Issue Type: Bug
  Components: Execution - Flow
Affects Versions: 1.12.0
Reporter: Padma Penumarthy
Assignee: Padma Penumarthy


Repeated Map Vector does not give correct payload byte count. It calls 
abstractMapVector method which gives payload byte count for a given value count 
for simple map (non repetitive) case. We need to overload this method for 
repeated map to get the right numbers. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [ANNOUNCE] New Committer: Kunal Khatua

[Padma Penumarthy]

Congratulations Kunal !

Thanks
Padma


> On Feb 27, 2018, at 8:42 AM, Aman Sinha  wrote:
> 
> The Project Management Committee (PMC) for Apache Drill has invited Kunal
> Khatua  to become a committer, and we are pleased to announce that he
> has accepted.
> 
> Over the last couple of years, Kunal has made substantial contributions to
> the process of creating and interpreting of query profiles, among other
> code contributions. He has led the efforts for Drill performance evaluation
> and benchmarking.  He is a prolific writer on the user mailing list,
> providing detailed responses.
> 
> Welcome Kunal, and thank you for your contributions.  Keep up the good
> work !
> 
> - Aman
> (on behalf of the Apache Drill PMC)

[jira] [Created] (DRILL-6184) Add batch sizing information to query profile

[Padma Penumarthy (JIRA)]

Padma Penumarthy created DRILL-6184:
---

 Summary: Add batch sizing information to query profile
 Key: DRILL-6184
 URL: https://issues.apache.org/jira/browse/DRILL-6184
 Project: Apache Drill
  Issue Type: Improvement
  Components: Execution - Flow
Affects Versions: 1.12.0
Reporter: Padma Penumarthy
Assignee: Padma Penumarthy
 Fix For: 1.13.0


for debugging, we need batch sizing information for each operator in query 
profile.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (DRILL-6180) Use System Option "output_batch_size" for External Sort

[Padma Penumarthy (JIRA)]

Padma Penumarthy created DRILL-6180:
---

 Summary: Use System Option "output_batch_size" for External Sort
 Key: DRILL-6180
 URL: https://issues.apache.org/jira/browse/DRILL-6180
 Project: Apache Drill
  Issue Type: Improvement
  Components: Execution - Flow
Affects Versions: 1.12.0
Reporter: Padma Penumarthy
Assignee: Padma Penumarthy
 Fix For: 1.13.0


External Sort has boot time configuration for output batch size 
"drill.exec.sort.external.spill.merge_batch_size" which is defaulted to 16M.

To make batch sizing configuration uniform across all operators, change this to 
use new system option that is added 
"drill.exec.memory.operator.output_batch_size". This option has default value 
of 32M.

So, what are the implications if default is changed to 32M for external sort ?

Instead, should we change the output batch size default to 16M for all 
operators ?

 

 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (DRILL-6177) Merge Join - Allocate memory for outgoing value vectors based on sizes of incoming batches.

[Padma Penumarthy (JIRA)]

Padma Penumarthy created DRILL-6177:
---

 Summary: Merge Join - Allocate memory for outgoing value vectors 
based on sizes of incoming batches.
 Key: DRILL-6177
 URL: https://issues.apache.org/jira/browse/DRILL-6177
 Project: Apache Drill
  Issue Type: Improvement
  Components: Execution - Flow
Affects Versions: 1.12.0
Reporter: Padma Penumarthy
Assignee: Padma Penumarthy
 Fix For: 1.13.0


Merge join currently assigns memory by default for 64k rows. Change this to 
assign memory for vectors based on number of rows in the outgoing batch and 
sizing information of each column. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (DRILL-6166) RecordBatchSizer does not handle hyper vectors

[Padma Penumarthy (JIRA)]

Padma Penumarthy created DRILL-6166:
---

 Summary: RecordBatchSizer does not handle hyper vectors
 Key: DRILL-6166
 URL: https://issues.apache.org/jira/browse/DRILL-6166
 Project: Apache Drill
  Issue Type: Bug
  Components: Execution - Flow
Affects Versions: 1.12.0
Reporter: Padma Penumarthy
Assignee: Padma Penumarthy
 Fix For: 1.13.0


RecordBatchSizer throws an exception when incoming batch has hyper vector.

(java.lang.UnsupportedOperationException) null
 org.apache.drill.exec.record.HyperVectorWrapper.getValueVector():61
 org.apache.drill.exec.record.RecordBatchSizer.():346
 org.apache.drill.exec.record.RecordBatchSizer.():311
 
org.apache.drill.exec.physical.impl.aggregate.StreamingAggBatch$StreamingAggregateMemoryManager.update():198
 org.apache.drill.exec.physical.impl.aggregate.StreamingAggBatch.innerNext():328
 org.apache.drill.exec.record.AbstractRecordBatch.next():164
 
org.apache.drill.exec.physical.impl.validate.IteratorValidatorBatchIterator.next():228
 org.apache.drill.exec.physical.impl.BaseRootExec.next():105
 
org.apache.drill.exec.physical.impl.partitionsender.PartitionSenderRootExec.innerNext():155
 org.apache.drill.exec.physical.impl.BaseRootExec.next():95
 org.apache.drill.exec.work.fragment.FragmentExecutor$1.run():233
 org.apache.drill.exec.work.fragment.FragmentExecutor$1.run():226
 java.security.AccessController.doPrivileged():-2
 javax.security.auth.Subject.doAs():422
 org.apache.hadoop.security.UserGroupInformation.doAs():1657
 org.apache.drill.exec.work.fragment.FragmentExecutor.run():226
 org.apache.drill.common.SelfCleaningRunnable.run():38
 java.util.concurrent.ThreadPoolExecutor.runWorker():1142
 java.util.concurrent.ThreadPoolExecutor$Worker.run():617
 java.lang.Thread.run():745



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (DRILL-6162) Enhance record batch sizer to retain nesting information for map columns.

[Padma Penumarthy (JIRA)]

Padma Penumarthy created DRILL-6162:
---

 Summary: Enhance record batch sizer to retain nesting information 
for map columns.
 Key: DRILL-6162
 URL: https://issues.apache.org/jira/browse/DRILL-6162
 Project: Apache Drill
  Issue Type: Improvement
  Components: Execution - Flow
Affects Versions: 1.12.0
Reporter: Padma Penumarthy
Assignee: Padma Penumarthy
 Fix For: 1.13.0


Enhance the record batch sizer to maintain the columnSizes in nested fashion 
for maps so given a column, we can get sizing information of all children 
underneath. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (DRILL-6161) Allocate memory for outgoing vectors based on sizing calculations

[Padma Penumarthy (JIRA)]

Padma Penumarthy created DRILL-6161:
---

 Summary: Allocate memory for outgoing vectors based on sizing 
calculations
 Key: DRILL-6161
 URL: https://issues.apache.org/jira/browse/DRILL-6161
 Project: Apache Drill
  Issue Type: Improvement
  Components: Execution - Flow
Affects Versions: 1.12.0
Reporter: Padma Penumarthy
Assignee: Padma Penumarthy
 Fix For: 1.13.0


Currently, in drill, we allocate memory for outgoing value vectors either for 
max value of 64k entries or start from 4096 and keep doubling as we need more 
memory. Every time we double, we allocate a new vector and do a copy. We also 
zero fill the new half. This has performance penalty. As part of batch sizing 
project, based on incoming batch(es) sizing information, we are limiting number 
of rows in outgoing batch based on memory. Since we know the number of rows and 
the average size of each column in the outgoing batch, we should use that 
information to preallocate memory for the outgoing vectors. This will be done 
as each operator is being changed to adhere to produce configured batch sizes.

Another improvement that can be done is packing the value vectors as dense as 
possible to improve the over all memory utilization. Since we allocate memory 
in powers of 2, once we figure out the number of rows to include in the 
outgoing batch, round it down to closest power of 2 and allocate memory for 
that many rows.

 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (DRILL-6160) Limit batch size for streaming aggregate based on memory

[Padma Penumarthy (JIRA)]

Padma Penumarthy created DRILL-6160:
---

 Summary: Limit batch size for streaming aggregate based on memory
 Key: DRILL-6160
 URL: https://issues.apache.org/jira/browse/DRILL-6160
 Project: Apache Drill
  Issue Type: Improvement
  Components: Execution - Flow
Affects Versions: 1.12.0
Reporter: Padma Penumarthy
Assignee: Padma Penumarthy
 Fix For: 1.13.0


Streaming aggregate has batch size of 32k rows. Change it to figure out number 
of rows in the outgoing batch based on memory configured with the option 
outputBatchSize and actual incoming batch size. Limit it to max of 64k and min 
of 1. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review for PR #1112: Metadata revisions

[Padma Penumarthy]

I will review. But, you still need to get a committer to do the final review 
and approve it.

Thanks
Padma


> On Feb 12, 2018, at 8:37 AM, Paul Rogers  wrote:
> 
> Hi All,
> Anyone available to review PR #1112? It is another step in committing the 
> final result set loader revisions.
> Recall that we more-or-less decided to commit the remaining work as a series 
> of small PRs rather than a single huge one. [1] This does require quite a bit 
> of work to constantly rebase on master, and to redevelop each partial PR so 
> it works by itself. This is done to ease reviews. But, the cost is that there 
> are more PRs, and the schedule is more sensitive to the "aging" delays before 
> someone starts reviewing a PR.
> We've also discussed Salim's intent to re-implement some of this work for 
> Parquet (at least the part that limits batch size). That intent may come, in 
> part, because it appears easier to just do new code rather than wait for the 
> existing code to be committed. Would be great if we could redirect some of 
> that new-code effort into reviews of the existing stuff so it can be used.
> Thanks,
> - Paul
> [1]  
> https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_paul-2Drogers_drill_wiki_BH-2DCode-2DIntro-23pull-2Drequest-2Dplan=DwIFaQ=cskdkSMqhcnjZxdQVpwTXg=XVqW14B9eGK9QR_fKKCb5H5LxKnqNMmq1U7RdNlhq1c=tdX_VJLpPDX8hmtQlncHPudDN9QANn5g_DIM9QSVChc=rU3uOmbYX3e-zztMTViuLl0-d6y2tIkCdE9qz6J7qdw=
> 

Re: Batch Sizing for Parquet Flat Reader

[Padma Penumarthy]

If our goal is to not to allocate more than 16MB for individual vectors to 
avoid external fragmentation, I guess
we can take that also into consideration in our calculations to figure out the 
outgoing number of rows.
The math might become more complex. But, the main point like you said is 
operators know what they are
getting and can figure out how to deal with that to honor the constraints 
imposed.

Thanks
Padma


On Feb 12, 2018, at 8:25 AM, Paul Rogers 
<par0...@yahoo.com.INVALID<mailto:par0...@yahoo.com.INVALID>> wrote:

Agreed that allocating vectors up front is another good improvement.
The average batch size approach gets us 80% of the way to the goal: it limits 
batch size and allows vector preallocation.
What it cannot do is limit individual vector sizes. Nor can it ensure that the 
resulting batch is optimally loaded with data. Getting the remaining 20% 
requires the level of detail provided by the result set loader.
We are driving to use the result set loader first in readers, since readers 
can't use the average batch size (they don't have an input batch to use to 
obtain sizes.)
To use the result set loader in non-leaf operators, we'd need to modify code 
generation. AFAIK, that is not something anyone is working on, so another 
advantage of the average batch size method is that it works with the code 
generation we already have.
Thanks,
- Paul



   On Sunday, February 11, 2018, 7:28:52 PM PST, Padma Penumarthy 
<ppenumar...@mapr.com<mailto:ppenumar...@mapr.com>> wrote:

With average row size method, since I know number of rows and the average size 
for each column,
I am planning to use that information to allocate required memory for each 
vector upfront.
This should help avoid copying every time we double and also improve memory 
utilization.

Thanks
Padma


On Feb 11, 2018, at 3:44 PM, Paul Rogers 
<par0...@yahoo.com.INVALID<mailto:par0...@yahoo.com.INVALID>> wrote:

One more thought:
3) Assuming that you go with the average batch size calculation approach,

The average batch size approach is a quick and dirty approach for non-leaf 
operators that can observe an incoming batch to estimate row width. Because 
Drill batches are large, the law of large numbers means that the average of a 
large input batch is likely to be a good estimator for the average size of a 
large output batch.
Note that this works only because non-leaf operators have an input batch to 
sample. Leaf operators (readers) do not have this luxury. Hence the result set 
loader uses the actual accumulated size for the current batch.
Also note that the average row method, while handy, is not optimal. It will, in 
general, result in greater internal fragmentation than the result set loader. 
Why? The result set loader packs vectors right up to the point where the 
largest would overflow. The average row method works at the aggregate level and 
will likely result in wasted space (internal fragmentation) in the largest 
vector. Said another way, with the average row size method, we can usually pack 
in a few more rows before the batch actually fills, and so we end up with 
batches with lower "density" than the optimal. This is important when the 
consuming operator is a buffering one such as sort.
The key reason Padma is using the quick & dirty average row size method is not 
that it is ideal (it is not), but rather that it is, in fact, quick.
We do want to move to the result set loader over time so we get improved memory 
utilization. And, it is the only way to control row size in readers such as CSV 
or JSON in which we have no size information until we read the data.
- Paul

Re: Batch Sizing for Parquet Flat Reader

[Padma Penumarthy]

With average row size method, since I know number of rows and the average size 
for each column, 
I am planning to use that information to allocate required memory for each 
vector upfront. 
This should help avoid copying every time we double and also improve memory 
utilization.

Thanks
Padma


> On Feb 11, 2018, at 3:44 PM, Paul Rogers  wrote:
> 
> One more thought:
>>> 3) Assuming that you go with the average batch size calculation approach,
> 
> The average batch size approach is a quick and dirty approach for non-leaf 
> operators that can observe an incoming batch to estimate row width. Because 
> Drill batches are large, the law of large numbers means that the average of a 
> large input batch is likely to be a good estimator for the average size of a 
> large output batch.
> Note that this works only because non-leaf operators have an input batch to 
> sample. Leaf operators (readers) do not have this luxury. Hence the result 
> set loader uses the actual accumulated size for the current batch.
> Also note that the average row method, while handy, is not optimal. It will, 
> in general, result in greater internal fragmentation than the result set 
> loader. Why? The result set loader packs vectors right up to the point where 
> the largest would overflow. The average row method works at the aggregate 
> level and will likely result in wasted space (internal fragmentation) in the 
> largest vector. Said another way, with the average row size method, we can 
> usually pack in a few more rows before the batch actually fills, and so we 
> end up with batches with lower "density" than the optimal. This is important 
> when the consuming operator is a buffering one such as sort.
> The key reason Padma is using the quick & dirty average row size method is 
> not that it is ideal (it is not), but rather that it is, in fact, quick.
> We do want to move to the result set loader over time so we get improved 
> memory utilization. And, it is the only way to control row size in readers 
> such as CSV or JSON in which we have no size information until we read the 
> data.
> - Paul   

[jira] [Created] (DRILL-6138) Move RecordBatchSizer to org.apache.drill.exec.record package

[Padma Penumarthy (JIRA)]

Padma Penumarthy created DRILL-6138:
---

 Summary: Move RecordBatchSizer to org.apache.drill.exec.record 
package
 Key: DRILL-6138
 URL: https://issues.apache.org/jira/browse/DRILL-6138
 Project: Apache Drill
  Issue Type: Task
  Components: Execution - Flow
Affects Versions: 1.12.0
Reporter: Padma Penumarthy
Assignee: Padma Penumarthy
 Fix For: 1.13.0


Move RecordBatchSizer from org.apache.drill.exec.physical.impl.spill package to 
org.apache.drill.exec.record package.

Minor refactoring - change columnSizes from list to map. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (DRILL-6133) RecordBatchSizer throws IndexOutOfBounds Exception for union vector

[Padma Penumarthy (JIRA)]

Padma Penumarthy created DRILL-6133:
---

 Summary: RecordBatchSizer throws IndexOutOfBounds Exception for 
union vector
 Key: DRILL-6133
 URL: https://issues.apache.org/jira/browse/DRILL-6133
 Project: Apache Drill
  Issue Type: Bug
  Components: Execution - Flow
Affects Versions: 1.12.0
Reporter: Padma Penumarthy
Assignee: Padma Penumarthy
 Fix For: 1.13.0


RecordBatchSizer throws IndexOutOfBoundsException when trying to get payload 
byte count of union vector. 

[Error Id: 430026a7-a963-40f1-bae2-1850649e8434 on 172.30.8.158:31013]
 at 
org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:633)
 ~[classes/:na]
 at 
org.apache.drill.exec.work.fragment.FragmentExecutor.sendFinalState(FragmentExecutor.java:300)
 [classes/:na]
 at 
org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:160)
 [classes/:na]
 at 
org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:266)
 [classes/:na]
 at 
org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38) 
[classes/:na]
 at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 
[na:1.8.0_45]
 at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
[na:1.8.0_45]
 at java.lang.Thread.run(Thread.java:745) [na:1.8.0_45]
Caused by: java.lang.IndexOutOfBoundsException: DrillBuf[2], udle: [1 0..0], 
index: 4, length: 4 (expected: range(0, 0))
DrillBuf[2], udle: [1 0..0]
 at 
org.apache.drill.exec.memory.BoundsChecking.checkIndex(BoundsChecking.java:80) 
~[classes/:na]
 at 
org.apache.drill.exec.memory.BoundsChecking.lengthCheck(BoundsChecking.java:86) 
~[classes/:na]
 at io.netty.buffer.DrillBuf.chk(DrillBuf.java:114) ~[classes/:4.0.48.Final]
 at io.netty.buffer.DrillBuf.getInt(DrillBuf.java:484) ~[classes/:4.0.48.Final]
 at org.apache.drill.exec.vector.UInt4Vector$Accessor.get(UInt4Vector.java:432) 
~[classes/:na]
 at 
org.apache.drill.exec.vector.VarCharVector.getPayloadByteCount(VarCharVector.java:308)
 ~[classes/:na]
 at 
org.apache.drill.exec.vector.NullableVarCharVector.getPayloadByteCount(NullableVarCharVector.java:256)
 ~[classes/:na]
 at 
org.apache.drill.exec.vector.complex.AbstractMapVector.getPayloadByteCount(AbstractMapVector.java:303)
 ~[classes/:na]
 at 
org.apache.drill.exec.vector.complex.UnionVector.getPayloadByteCount(UnionVector.java:574)
 ~[classes/:na]
 at 
org.apache.drill.exec.physical.impl.spill.RecordBatchSizer$ColumnSize.(RecordBatchSizer.java:147)
 ~[classes/:na]
 at 
org.apache.drill.exec.physical.impl.spill.RecordBatchSizer.measureColumn(RecordBatchSizer.java:403)
 ~[classes/:na]
 at 
org.apache.drill.exec.physical.impl.spill.RecordBatchSizer.(RecordBatchSizer.java:350)
 ~[classes/:na]
 at 
org.apache.drill.exec.physical.impl.spill.RecordBatchSizer.(RecordBatchSizer.java:320)
 ~[classes/:na]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (DRILL-6126) Allocate memory for value vectors upfront in flatten operator

[Padma Penumarthy (JIRA)]

Padma Penumarthy created DRILL-6126:
---

 Summary: Allocate memory for value vectors upfront in flatten 
operator
 Key: DRILL-6126
 URL: https://issues.apache.org/jira/browse/DRILL-6126
 Project: Apache Drill
  Issue Type: Improvement
Reporter: Padma Penumarthy
Assignee: Padma Penumarthy
 Fix For: 1.12.0


With recent changes to control batch size for flatten operator, we figure out 
row count in the output batch based on memory. Since we know how many rows we 
are going to include in the batch, we can also allocate the memory needed 
upfront instead of starting with initial value (4096) and doubling, copying 
every time we need more. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (DRILL-6123) Limit batch size for Merge Join based on memory

[Padma Penumarthy (JIRA)]

Padma Penumarthy created DRILL-6123:
---

 Summary: Limit batch size for Merge Join based on memory
 Key: DRILL-6123
 URL: https://issues.apache.org/jira/browse/DRILL-6123
 Project: Apache Drill
  Issue Type: Improvement
  Components: Execution - Flow
Affects Versions: 1.12.0
Reporter: Padma Penumarthy
Assignee: Padma Penumarthy
 Fix For: 1.13.0


Merge join limits output batch size to 32K rows irrespective of row size. This 
can create very large or very small batches (in terms of memory), depending 
upon average row width. Change this to figure out output row count based on 
memory specified with the new outputBatchSize option and average row width of 
incoming left and right batches. Output row count will be minimum of 1 and max 
of 64k. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [ANNOUNCE] New PMC member: Paul Rogers

[Padma Penumarthy]

Congratulations Paul. 

Thanks
Padma


> On Jan 30, 2018, at 1:55 PM, Gautam Parai  wrote:
> 
> Congratulations Paul!
> 
> 
> From: Timothy Farkas 
> Sent: Tuesday, January 30, 2018 1:54:43 PM
> To: dev@drill.apache.org
> Subject: Re: [ANNOUNCE] New PMC member: Paul Rogers
> 
> Congrats!
> 
> 
> From: Aman Sinha 
> Sent: Tuesday, January 30, 2018 1:50:07 PM
> To: dev@drill.apache.org
> Subject: [ANNOUNCE] New PMC member: Paul Rogers
> 
> I am pleased to announce that Drill PMC invited Paul Rogers to the PMC and
> he has accepted the invitation.
> 
> Congratulations Paul and thanks for your contributions !
> 
> -Aman
> (on behalf of Drill PMC)

[jira] [Created] (DRILL-6113) Limit batch size for Merge Receiver

[Padma Penumarthy (JIRA)]

Padma Penumarthy created DRILL-6113:
---

 Summary: Limit batch size for Merge Receiver
 Key: DRILL-6113
 URL: https://issues.apache.org/jira/browse/DRILL-6113
 Project: Apache Drill
  Issue Type: Bug
  Components: Execution - Flow
Affects Versions: 1.12.0
Reporter: Padma Penumarthy
Assignee: Padma Penumarthy
 Fix For: 1.13.0


Merge receiver has hard coded limit of 32K rows as batch. Since rows can be of 
varying width, it is difficult to predict the output batch size (in terms of 
memory) for this operator. Change this to derive row count based on actual 
memory available. We are introducing a new option called outputBatchSize to 
limit the batch size of each operator. Use the memory configured from that. 
Figure out the average row width of outgoing batch based on averages of batches 
coming from incoming streams. Limit the row count based on memory available to 
use and average row width.

 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (DRILL-6071) Limit batch size for flatten operator

[Padma Penumarthy (JIRA)]

Padma Penumarthy created DRILL-6071:
---

 Summary: Limit batch size for flatten operator
 Key: DRILL-6071
 URL: https://issues.apache.org/jira/browse/DRILL-6071
 Project: Apache Drill
  Issue Type: Bug
  Components: Execution - Flow
Affects Versions: 1.12.0
Reporter: Padma Penumarthy
Assignee: Padma Penumarthy
 Fix For: 1.13.0


flatten currently uses an adaptive algorithm to control the outgoing batch 
size. 
While processing the input batch, it adjusts the number of records in outgoing 
batch based on memory usage so far. Once memory usage exceeds the configured 
limit, the algorithm becomes more proactive and adjusts the limit half way 
through  and end of every batch. All this periodic checking of memory usage is 
unnecessary overhead and impacts performance. Also, we will know only after the 
fact. 

Instead, figure out how many rows should be there in the outgoing batch from 
incoming batch.
The way to do that would be to figure out average row size of the outgoing 
batch and based on that figure out how many rows can be there for a given 
amount of memory. value vectors provide us the necessary information to be able 
to figure this out.








--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (DRILL-5972) Slow performance for query on INFORMATION_SCHEMA.TABLE

[Padma Penumarthy (JIRA)]

Padma Penumarthy created DRILL-5972:
---

 Summary: Slow performance for query on INFORMATION_SCHEMA.TABLE
 Key: DRILL-5972
 URL: https://issues.apache.org/jira/browse/DRILL-5972
 Project: Apache Drill
  Issue Type: Bug
  Components: Storage - Information Schema
Affects Versions: 1.11.0
Reporter: Padma Penumarthy
Assignee: Padma Penumarthy
 Fix For: 1.13.0


A query like the following on INFORMATION_SCHEMA takes a long time to execute. 

select TABLE_CATALOG, TABLE_SCHEMA, TABLE_NAME, TABLE_TYPE from 
INFORMATION_SCHEMA.`TABLES` WHERE TABLE_NAME LIKE '%' AND ( TABLE_SCHEMA = 
'hive.default' ) ORDER BY TABLE_TYPE, TABLE_CATALOG, TABLE_SCHEMA, TABLE_NAME; 

Reason being we fetch table information for all schemas instead of just 
'hive.default' schema.

If we  change the predicate like this, it executes very fast.

select TABLE_CATALOG, TABLE_SCHEMA, TABLE_NAME, TABLE_TYPE from 
INFORMATION_SCHEMA.`TABLES` WHERE  ( TABLE_SCHEMA = 'hive.default' ) AND 
TABLE_NAME LIKE '%'  ORDER BY TABLE_TYPE, TABLE_CATALOG, TABLE_SCHEMA, 
TABLE_NAME; 

The difference is in the order in which we evaluate the expressions in the 
predicate.
In the first case,  we first evaluate TABLE_NAME LIKE '%' and decide that it is 
inconclusive (since we do not know the schema). So, we go get all tables for 
all the schemas.

In the second case, we first evaluate  TABLE_SCHEMA = 'hive.default' and decide 
that we need to fetch only tables for that schema.




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: Hangout Topics for 11/14/2017

[Padma Penumarthy]

Here are the topics so far:

Unit Testing - Tim
1.12 Release - Arina
Metadata Management - Padma

Thanks
Padma


On Nov 13, 2017, at 1:15 PM, Padma Penumarthy 
<ppenumar...@mapr.com<mailto:ppenumar...@mapr.com>> wrote:

Drill hangout tomorrow Nov 14th, at 10 AM PST.
Please send email or bring them up tomorrow, if you have topics to discuss.

Hangout link:
https://plus.google.com/hangouts/_/event/ci4rdiju8bv04a64efj5fedd0lc

Thanks
Padma

Hangout Topics for 11/14/2017

[Padma Penumarthy]

Drill hangout tomorrow Nov 14th, at 10 AM PST.
Please send email or bring them up tomorrow, if you have topics to discuss.

Hangout link:
https://plus.google.com/hangouts/_/event/ci4rdiju8bv04a64efj5fedd0lc

Thanks
Padma

[jira] [Created] (DRILL-5899) No need to do isAscii check for simple pattern matcher

[Padma Penumarthy (JIRA)]

Padma Penumarthy created DRILL-5899:
---

 Summary: No need to do isAscii check for simple pattern matcher
 Key: DRILL-5899
 URL: https://issues.apache.org/jira/browse/DRILL-5899
 Project: Apache Drill
  Issue Type: Improvement
  Components: Execution - Flow
Reporter: Padma Penumarthy
Assignee: Padma Penumarthy
Priority: Critical


For simple pattern matcher, we do not have to do isAscii check. 
UTF-8 encoding ensures that no UTF-8 character is a prefix of any other valid 
character. So, for the 4 simple patterns we have i.e. startsWith, endsWith, 
contains and constant, we can get rid of this check. This will help improve 
performance. 




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: Excessive review comments

[Padma Penumarthy]

To subscribe to PRs and JIRAs of interest, we need to know about them first.
Is it possible to get notified when a new PR or JIRA is created and not for
further updates unless you subscribe to them ?

Thanks
Padma


> On Oct 19, 2017, at 10:13 AM, Paul Rogers  wrote:
> 
> So, rather than copying all messages to dev, simply ask interested folks 
> subscribe to the PRs of interest. (Or, subscribe to the corresponding JIRA 
> tickets, since JIRA echos all comments in another spray of e-mail…)
> 
> This way, semi-active folks on the dev list can see substantive discussions 
> without being bombarded with day-to-day minutia.
> 
> Thanks,
> 
> - Paul
> 
>> On Oct 19, 2017, at 10:07 AM, Parth Chandra  wrote:
>> 
>> I generally keep track of dev activity only through the emails sent. I
>> don't mind getting duplicate emails either - it's not too hard to write a
>> filter to take out the stuff you're not interested in.
>> 
>> Interestingly, I've never noticed whether 'start a review' sends one email
>> or many; mostly because gmail does a nice job of grouping the emails
>> together. I have seen a single email with many comments being sent out so
>> the feature did work as advertised at one point.
>> 
>> That said, if there is a way I can stay updated via email (repo watching,
>> jira updates, etc.) then I am fine with turning the feature off.
>> 
>> 
>> 
>> On Thu, Oct 19, 2017 at 9:51 AM, Timothy Farkas  wrote:
>> 
>>> +1 for turning off the feature. If someone really needs to be emailed with
>>> comment updates they can become a watcher of the repo on Github.
>>> 
>>> 
>>> From: Paul Rogers 
>>> Sent: Thursday, October 19, 2017 9:43:26 AM
>>> To: dev@drill.apache.org
>>> Subject: Re: Excessive review comments
>>> 
>>> Can we simply turn off the feature? I never, ever read the e-mails coming
>>> from this source; I always follow the link back to the PR. Can we reduce it
>>> to “Hey, just wanted to let you know that a new comment was posted. Click
>>> _here_ to read it.”
>>> 
>>> The only other solution is to give few review comments; not sure if we
>>> want to go that route...
>>> 
>>> - Paul
>>> 
 On Oct 19, 2017, at 8:35 AM, Arina Yelchiyeva <
>>> arina.yelchiy...@gmail.com> wrote:
 
 Agree, I am not sure I saw this feature working as well.
 All it did it was sending all the emails at once, rather in the process
>>> of
 comments emergence.
 
 Kind regards
 Arina
 
 On Thu, Oct 19, 2017 at 6:27 PM, Julian Hyde  wrote:
 
> I don’t know whether anything is broken. I believed that the GitHub
>>> “start
> a review” feature would cause all review comments to be sent in a single
> email. But now I think of it, I’m not sure I ever saw it working. I
>>> wonder
> whether Github-ASF integration is at fault.
> 
> Whatever the reasons for it, 39 emails to dev list is quite a blast.
> People tend to unsubscribe from lists if the volume is too high.
> 
> Julian
> 
> 
>> On Oct 18, 2017, at 5:54 PM, Paul Rogers  wrote:
>> 
>> With all due respect, I did start a review. Is something broken?
>> 
>> - Paul
>> 
>>> On Oct 18, 2017, at 3:36 PM, julianhyde  wrote:
>>> 
>>> Github user julianhyde commented on a diff in the pull request:
>>> 
>>> https://github.com/apache/drill/pull/984#discussion_r145561518
>>> 
>>> --- Diff: exec/java-exec/src/test/java/org/apache/drill/test/
>>> ClusterFixture.java
> ---
>>> @@ -584,11 +492,14 @@ public static void defineWorkspace(Drillbit
> drillbit, String pluginName,
>>>   public static final String EXPLAIN_PLAN_TEXT = "text";
>>>   public static final String EXPLAIN_PLAN_JSON = "json";
>>> 
>>> -  public static FixtureBuilder builder() {
>>> -FixtureBuilder builder = new FixtureBuilder()
>>> +  public static FixtureBuilder builder(DirTestWatcher
> dirTestWatcher) {
>>> --- End diff --
>>> 
>>> Jeez Paul, please start a review rather than making single review
> comments. I just got 39 emails from you, and so did everyone else on
> dev@drill.
>>> 
>>> 
>>> ---
>> 
> 
> 
>>> 
>>> 
> 

Re: Parquet Metadata table on Rolling window

[Padma Penumarthy]

No, it does not. We rebuild metadata for all directories under mydata when you
query /mydata. 
If new files get added anywhere in the whole hierarchy, metadata gets 
regenerated
for all of them.

However, if you query only  /mydata/3 and nothing is changed under 3,
no metadata is generated.

Thanks
Padma

> On Oct 16, 2017, at 12:33 PM, François Méthot  wrote:
> 
> Thanks Padma,
> 
> Would we benefits at all from running metadata on directories that we know
> we will never modify?
> 
> We would end up with:
> /mydata/3/(Metadata generated...)
> /mydata/4/(Metadata generated...)
> /mydata/.../(Metadata generated...)
> /mydata/109/(Metadata generated...)
> /mydata/110/(Current partition : Metadata NOT generated yet...)
> 
> When users query /mydata, would drill take advantage of the metadata
> available in each subfolder?
> 
> Francois

[jira] [Created] (DRILL-5854) IllegalStateException when empty batch with valid schema is received.

[Padma Penumarthy (JIRA)]

Padma Penumarthy created DRILL-5854:
---

 Summary: IllegalStateException when empty batch with valid schema 
is received.
 Key: DRILL-5854
 URL: https://issues.apache.org/jira/browse/DRILL-5854
 Project: Apache Drill
  Issue Type: Bug
  Components: Execution - Flow
Affects Versions: 1.11.0
Reporter: Padma Penumarthy
Assignee: Padma Penumarthy
 Fix For: 1.12.0


When we get a batch with record count 0 and valid schema, the following 
exception is thrown sometimes.
Problem seem to be that downstream operators are not getting NONE after 
OK_NEW_SCHEMA with record count 0. 

Error: SYSTEM ERROR: IllegalStateException: Cleanup before finished. 0 out of 1 
streams have finished

Fragment 0:0

[Error Id: 0dec4617-eb06-48fd-80c9-75e9c1e74ce2 on sidrill1:31010]

  (java.lang.IllegalStateException) Cleanup before finished. 0 out of 1 streams 
have finished
org.apache.drill.exec.work.batch.BaseRawBatchBuffer.close():107
org.apache.drill.common.AutoCloseables.close():76
org.apache.drill.common.AutoCloseables.close():64
org.apache.drill.exec.work.batch.AbstractDataCollector.close():119
org.apache.drill.common.AutoCloseables.close():76
org.apache.drill.exec.work.batch.IncomingBuffers.close():140
org.apache.drill.exec.ops.FragmentContext.suppressingClose():436
org.apache.drill.exec.ops.FragmentContext.close():429
org.apache.drill.exec.work.fragment.FragmentExecutor.closeOutResources():320
org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup():155
org.apache.drill.exec.work.fragment.FragmentExecutor.run():264
org.apache.drill.common.SelfCleaningRunnable.run():38
java.util.concurrent.ThreadPoolExecutor.runWorker():1149
java.util.concurrent.ThreadPoolExecutor$Worker.run():624
java.lang.Thread.run():748 (state=,code=0)





--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: Parquet Metadata table on Rolling window

[Padma Penumarthy]

Unfortunately, we do not do incremental metadata updates. 
If new files are getting added constantly, refresh table metadata will not help.

Thanks
Padma


> On Oct 5, 2017, at 5:36 PM, François Méthot  wrote:
> 
> Hi,
> 
>  I have been using drill for more than year now, we are running 1.10.
> 
> My queries can spend from 5 to 10 minutes for planning because I am dealing
> with lots of file in HDFS. (then 5 min to 60 min for execution)
> 
> I maintain a rolling window of data  partitionned by the epoch seconds
> rounded to the hour.
> /mydata/3/   -> Next partition to be deleted (nightly check)
> /mydata/4/
> /mydata/.../
> /mydata/109/
> /mydata/110/ -> current hour, this is where new parquet files are added
> 
> I am  considering using REFRESH TABLE METADATA.
> Is it beneficial at all in a situation where new files are added
> constantly, (but only to the latest partition, older partition are set in
> stone)?
> Will drill detect that new files are added to the latest partition (110) ?
> -Will it trigger a refresh metadata on all the directory, on just on
> /mydata/110?
> 
> 
> Thanks for your help
> François

[jira] [Created] (DRILL-5839) Handle Empty Batches in Merge Receiver

[Padma Penumarthy (JIRA)]

Padma Penumarthy created DRILL-5839:
---

 Summary: Handle Empty Batches in Merge Receiver
 Key: DRILL-5839
 URL: https://issues.apache.org/jira/browse/DRILL-5839
 Project: Apache Drill
  Issue Type: Bug
  Components: Execution - Flow
Affects Versions: 1.11.0
Reporter: Padma Penumarthy
Assignee: Padma Penumarthy
 Fix For: 1.12.0


merge receiver throws an exception when it receives first batch as empty batch 
(no rows and no schema) from any of the senders. Problem is that the operator 
expects at least one batch with schema (0 rows is ok, 0 columns is not) from 
each of its senders. 

The way algorithm works is as follows:
Get the first batch from each of the senders.
Create hyper vector container with this first batch from each of the senders.
Add the batches from senders to the priority queue
Pop from priority queue, get the index for the current batch from that sender, 
and use that to copy from the hyper vector to the outgoing vector
When the end of batch from a sender is reached, load the next batch from the 
sender.
Stop when there are no more batches from any of the senders.

If any of the senders do not send first batch with schema and if we skip adding 
that batch to the hyper vector, hyper vector is not setup correctly and all the 
offsets from selection vector to individual batches from senders with in the 
hyper vector are messed up. 

Fix for this problem is when we receive empty batch from any of the senders, 
create dummy batch with schema  from one of the other senders and add it to the 
hyper vector. 

If all senders send empty first batches, we just return NONE to downstream 
operator.







--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: IntelliJ code format

[Padma Penumarthy]

@weijie, link is updated now with the new jar.
Please check it out and let us know if any other issues.

Thanks,
Padma


On Aug 21, 2017, at 8:59 AM, weijie tong 
<tongweijie...@gmail.com<mailto:tongweijie...@gmail.com>> wrote:

@padma ,what's the process?

On Wed, 9 Aug 2017 at 1:04 AM Padma Penumarthy 
<ppenumar...@mapr.com<mailto:ppenumar...@mapr.com>> wrote:

You are right. It is configured as 4. We should fix that.

Thanks,
Padma


On Aug 8, 2017, at 8:12 AM, weijie tong 
<tongweijie...@gmail.com<mailto:tongweijie...@gmail.com>> wrote:

the download  site url :
https://drill.apache.org/docs/apache-drill-contribution-guidelines/

On Tue, Aug 8, 2017 at 10:59 PM, weijie tong <tongweijie...@gmail.com>
wrote:

The IntelliJ code format downloaded from the Drill web site seems to
have
4 indent .The eclipse one is 2 indent. Wonder ti's my personal env
problem
or the provided Intellij code format error?

Re: IntelliJ code format

[Padma Penumarthy]

Sorry for the delay. I am working with our doc team to get this link updated.
Will let you know once done.

Thanks,
Padma


> On Aug 21, 2017, at 8:59 AM, weijie tong <tongweijie...@gmail.com> wrote:
> 
> @padma ,what's the process?
> 
> On Wed, 9 Aug 2017 at 1:04 AM Padma Penumarthy <ppenumar...@mapr.com> wrote:
> 
>> You are right. It is configured as 4. We should fix that.
>> 
>> Thanks,
>> Padma
>> 
>> 
>>> On Aug 8, 2017, at 8:12 AM, weijie tong <tongweijie...@gmail.com> wrote:
>>> 
>>> the download  site url :
>>> https://drill.apache.org/docs/apache-drill-contribution-guidelines/
>>> 
>>> On Tue, Aug 8, 2017 at 10:59 PM, weijie tong <tongweijie...@gmail.com>
>>> wrote:
>>> 
>>>> The IntelliJ code format downloaded from the Drill web site seems to
>> have
>>>> 4 indent .The eclipse one is 2 indent. Wonder ti's my personal env
>> problem
>>>> or the provided Intellij code format error?
>>>> 
>> 
>> 

Re: Discuss about Drill's schedule policy

[Padma Penumarthy]

If control RPC is down to a drillbit i.e if  a drillbit is not responding,
zookeeper should detect that and notify other drillbits to remove the dead 
drillbit
from their active list. Once that happens, the next query that comes in should
not even see that drillbit.
We need a way to differentiate drillbits based on their resource (i.e. CPU and 
memory)
availability/usage and consider that information for planning.
We do not have that capability. We treat all of them the same.

Thanks,
Padma


On Aug 20, 2017, at 5:39 AM, weijie tong 
> wrote:

HI all:

 Drill's current schedule policy seems a little simple. The
SimpleParallelizer assigns endpoints in round robin model which ignores the
system's load and other factors. To critical scenario, some drillbits are
suffering frequent full GCs which will let their control RPC blocked.
Current assignment will not exclude these drillbits from the next coming
queries's assignment. then the problem will get worse .
 I propose to add a zk path to hold bad drillbits. Forman will recognize
bad drillbits by waiting timeout (timeout of  control response from
intermediate fragments), then update the bad drillbits path. Next coming
queries will exclude these drillbits from the assignment list.
 How do you think about it or any suggests ? If sounds ok ,will file a
JIRA and give some contributes.

[jira] [Created] (DRILL-5731) regionsToScan is computed multiple times for MapR DB Json and Binary tables

[Padma Penumarthy (JIRA)]

Padma Penumarthy created DRILL-5731:
---

 Summary: regionsToScan is computed multiple times for MapR DB Json 
and Binary tables
 Key: DRILL-5731
 URL: https://issues.apache.org/jira/browse/DRILL-5731
 Project: Apache Drill
  Issue Type: Improvement
Reporter: Padma Penumarthy
Assignee: Padma Penumarthy


We are computing regionsToScan for both Json and binary tables in groupScan 
init, which gets called multiple times during planning. This is expensive and 
not needed. Instead, compute only when needed for parallelization assignment.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: Failing Tests In Master

[Padma Penumarthy]

These tests fail on my mac with JDK 1.8.

> On Aug 17, 2017, at 4:42 PM, Jinfeng Ni <j...@apache.org> wrote:
> 
> Running on CentOS (JDK1.7) with latest master against those unit testcase,
> and I did not see errors.  Any difference in terms of OS/JDK?
> 
> git log
> commit e9065b55ea560e7f737d6fcb4948f9e945b9b14f
> 
> mvn -DfailIfNoTests=false
> -Dtest=TestMetadataProvider,TestCustomUserAuthenticator,TestInfoSchema,TestViewSupport,TestParquetScan
> surefire:test
> 
> Running org.apache.drill.exec.work.metadata.TestMetadataProvider
> Running
> org.apache.drill.exec.work.metadata.TestMetadataProvider#columnsWithColumnNameFilter
> Running org.apache.drill.exec.sql.TestViewSupport
> ...
> 
> Results :
> 
> Tests run: 82, Failures: 0, Errors: 0, Skipped: 1
> 
> 
> On Thu, Aug 17, 2017 at 3:59 PM, Padma Penumarthy <ppenumar...@mapr.com>
> wrote:
> 
>> Yes, I do see these failures.
>> 
>> Thanks,
>> Padma
>> 
>>> On Aug 17, 2017, at 3:55 PM, Timothy Farkas <tfar...@mapr.com> wrote:
>>> 
>>> These tests are consistently failing for me with the latest master. Does
>> anyone else see this issue?
>>> 
>>> 
>>> Failed tests:
>>> 
>>> TestMetadataProvider.tables:153 expected: but was:
>>> 
>>> TestMetadataProvider.tablesWithTableNameFilter:212 expected: but
>> was:
>>> 
>>> TestMetadataProvider.tablesWithSystemTableFilter:187 expected: but
>> was:
>>> 
>>> TestMetadataProvider.tablesWithTableFilter:176 expected: but
>> was:
>>> 
>>> 
>>> Tests in error:
>>> 
>>> TestCustomUserAuthenticator.positiveUserAuth » UserRemote SYSTEM
>> ERROR: URISyn...
>>> 
>>> TestCustomUserAuthenticator.positiveUserAuthAfterNegativeUserAuth »
>> UserRemote
>>> 
>>> TestInfoSchema.selectFromAllTables » UserRemote SYSTEM ERROR:
>> URISyntaxExcepti...
>>> 
>>> TestViewSupport.infoSchemaWithView:350->BaseTestQuery.testRunAndReturn:344
>> » Rpc
>>> 
>>> TestInfoSchemaFilterPushDown.testFilterPushdown_NonEqual » UserRemote
>> SYSTEM E...
>>> 
>>> TestParquetScan.testSuccessFile:58->BaseTestQuery.testRunAndReturn:344
>> » Rpc o...
>>> 
>>> Thanks,
>>> Tim
>>> 
>> 
>> 

Re: Failing Tests In Master

[Padma Penumarthy]

Yes, I do see these failures.

Thanks,
Padma

> On Aug 17, 2017, at 3:55 PM, Timothy Farkas  wrote:
> 
> These tests are consistently failing for me with the latest master. Does 
> anyone else see this issue?
> 
> 
> Failed tests:
> 
>  TestMetadataProvider.tables:153 expected: but was:
> 
>  TestMetadataProvider.tablesWithTableNameFilter:212 expected: but 
> was:
> 
>  TestMetadataProvider.tablesWithSystemTableFilter:187 expected: but 
> was:
> 
>  TestMetadataProvider.tablesWithTableFilter:176 expected: but was:
> 
> 
> Tests in error:
> 
>  TestCustomUserAuthenticator.positiveUserAuth » UserRemote SYSTEM ERROR: 
> URISyn...
> 
>  TestCustomUserAuthenticator.positiveUserAuthAfterNegativeUserAuth » 
> UserRemote
> 
>  TestInfoSchema.selectFromAllTables » UserRemote SYSTEM ERROR: 
> URISyntaxExcepti...
> 
>  TestViewSupport.infoSchemaWithView:350->BaseTestQuery.testRunAndReturn:344 » 
> Rpc
> 
>  TestInfoSchemaFilterPushDown.testFilterPushdown_NonEqual » UserRemote SYSTEM 
> E...
> 
>  TestParquetScan.testSuccessFile:58->BaseTestQuery.testRunAndReturn:344 » Rpc 
> o...
> 
> Thanks,
> Tim
> 

filter performance with like

[Padma Penumarthy]

I was playing around trying to figure out how to improve performance for
simple pattern matching predicates with like.

Documented my findings here.
https://issues.apache.org/jira/browse/DRILL-5697

Want to find out if folks have any thoughts or feedback on this.
Plan is to go with simple character by character comparison.


Thanks,
Padma

Re: IntelliJ code format

[Padma Penumarthy]

You are right. It is configured as 4. We should fix that. 

Thanks,
Padma


> On Aug 8, 2017, at 8:12 AM, weijie tong  wrote:
> 
> the download  site url :
> https://drill.apache.org/docs/apache-drill-contribution-guidelines/
> 
> On Tue, Aug 8, 2017 at 10:59 PM, weijie tong 
> wrote:
> 
>> The IntelliJ code format downloaded from the Drill web site seems to have
>> 4 indent .The eclipse one is 2 indent. Wonder ti's my personal env problem
>> or the provided Intellij code format error?
>> 

[jira] [Created] (DRILL-5697) Improve performance of filter operator for pattern matching

[Padma Penumarthy (JIRA)]

Padma Penumarthy created DRILL-5697:
---

 Summary: Improve performance of filter operator for pattern 
matching
 Key: DRILL-5697
 URL: https://issues.apache.org/jira/browse/DRILL-5697
 Project: Apache Drill
  Issue Type: Improvement
  Components: Execution - Flow
Affects Versions: 1.11.0
Reporter: Padma Penumarthy
Assignee: Padma Penumarthy


Queries using filter with sql like operator use Java regex library for pattern 
matching. However, for cases like %abc (ends with abc), abc% (starts with abc), 
%abc% (contains abc), it is observed that implementing these cases with simple 
code instead of using regex library provides good performance boost (4-6x). 
Idea is to use special case code for simple, common cases and fall back to Java 
regex library for complicated ones. That will provide good performance benefit 
for most common cases.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: [VOTE] Release Apache Drill 1.11.0 - rc0

[Padma Penumarthy]

+1 (non-binding)

Tried in embedded mode on my mac. Ran some queries. 
Downloaded and built on CentOS VM. 
Installed the build on the 4 node cluster. Ran some queries on parquet files.

Thanks,
Padma
  

> On Jul 26, 2017, at 2:54 PM, Kunal Khatua  wrote:
> 
> +1 (non-binding)
> 
> Downloaded and built in CentOS. Tried a handful of queries and it worked 
> fine. 
> 
> Tried repro for DRILL-5420 (CPU churn) and no issues. Ran a few 100 query 
> cancellations and limit queries too, but didn't see any issues. (guessing 
> this resolves DRILL-5435 and DRILL-5609 as well).
> Tested the above with DRILL-5481 (in-memory profile persistence) and that 
> worked fine too.  
> 
> 
> 
> -Original Message-
> From: Parth Chandra [mailto:par...@apache.org] 
> Sent: Tuesday, July 25, 2017 4:26 PM
> To: dev 
> Subject: Re: [VOTE] Release Apache Drill 1.11.0 - rc0
> 
> +1
> 
> Downloaded src and checked all signatures.
> Built on MacOS. Built C++ client on MacOS.
> Verified fix for DRILL-5659
> Ran a couple of test queries in sqlline.
> 
> Looks good.
> 
> 
> 
> 
> 
> 
> On Tue, Jul 25, 2017 at 3:36 AM, Arina Yelchiyeva < 
> arina.yelchiy...@gmail.com> wrote:
> 
>> Hi all,
>> 
>> I'd like to propose the first release candidate (rc0) of Apache Drill, 
>> version 1.11.0.
>> 
>> The release candidate covers a total of 126 resolved JIRAs [1]. Thanks 
>> to everyone who contributed to this release.
>> 
>> The tarball artifacts are hosted at [2] and the maven artifacts are 
>> hosted at [3].
>> 
>> This release candidate is based on commit
>> 4220fb2fffbc81883df3e5fea575fa0a584852b3 located at [4].
>> 
>> The vote ends at 1:00 PM UTC (5:00 AM PT), July 28, 2017.
>> 
>> [ ] +1
>> [ ] +0
>> [ ] -1
>> 
>> Here's my vote: +1 (non-binding)
>> 
>> 
>> [1]
>> https://issues.apache.org/jira/secure/ReleaseNote.jspa?
>> projectId=12313820=12339943
>> [2] http://home.apache.org/~arina/drill/releases/1.11.0/rc0/
>> [3] https://repository.apache.org/content/repositories/
>> orgapachedrill-1042/
>> [4] https://github.com/arina-ielchiieva/drill/commits/drill-1.11.0
>> 
>> Kind regards
>> Arina
>> 

Re: [HANGOUT] Topics for 7/25/17

[Padma Penumarthy]

I have a topic to discuss. Lot of folks on the user mailing list raised
the issue of not being able to access all S3 regions using Drill.
We need hadoop version 2.8 or higher to be able to connect to
regions which support only Version 4 signature.
I tried with 2.8.1, which just got released and it works i.e. I am able to
connect to both old and new regions (by specifying the endpoint in the config).
There are some failures in unit tests, which can be fixed.

Fixing S3 connectivity issues is important.
However, the hadoop release notes for 2.8.1 (and 2.8.0 as well) say the 
following:
"Please note that 2.8.x release line continues to be not yet ready for 
production use”.

So, should we or not move to 2.8.1 ?

Thanks,
Padma


On Jul 24, 2017, at 9:46 AM, Arina Yelchiyeva 
> wrote:

Hi all,

We'll have the hangout tomorrow at the usual time [1]. Any topics to be
discussed?

[1] https://drill.apache.org/community-resources/

Kind regards
Arina

[jira] [Created] (DRILL-5587) Validate Parquet blockSize and pageSize configured with SYSTEM/SESSION option

[Padma Penumarthy (JIRA)]

Padma Penumarthy created DRILL-5587:
---

 Summary: Validate Parquet blockSize and pageSize configured with 
SYSTEM/SESSION option
 Key: DRILL-5587
 URL: https://issues.apache.org/jira/browse/DRILL-5587
 Project: Apache Drill
  Issue Type: Bug
  Components: Storage - Parquet
Affects Versions: 1.10.0
Reporter: Padma Penumarthy
Assignee: Padma Penumarthy
 Fix For: 1.11.0


We can set Parquet blockSize, pageSize and dictionary pageSize to any value. It 
uses LongValidator which is not exactly validating the value. Since all these 
sizes are used as int in the code, even though user is able to set them to any 
value (could be greater than MAXINT and/or negative), parsing the value later 
in the code as int can throw an error. Instead, restrict the value that can be 
set to MAXINT. 
There is a bug open for validating system/session options in general. 
https://issues.apache.org/jira/browse/DRILL-2478




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: [HANGOUT] Topics for 6/12/17

[Padma Penumarthy]

Thank you all for attending the hangout today.

Here are the meeting minutes:

Muhammed asked question about how the client chooses which drillbit to connect.
Client gets the information about available drillbits from zookeeper and
just does round robin to select the node for a query. That drillbit becomes the 
foreman for the query.

Paul discussed in detail the memory fragmentation problem we have in drill and 
how we
are planning to address that. It was very insightful. Thanks Paul.
You can refer to DRILL-5211<https://issues.apache.org/jira/browse/DRILL-5211> 
for more details. It has links to background information and design docs.
Please ask questions and provide comments in the JIRA.

For the next meeting, we can choose another topic like this and go in detail.

Thanks,
Padma


On Jun 12, 2017, at 10:21 AM, Padma Penumarthy 
<ppenumar...@mapr.com<mailto:ppenumar...@mapr.com>> wrote:


Drill hangout will be tomorrow, 10 AM PST.

In the last hangout, we talked about discussing one of the ongoing Drill 
projects in detail.
Please let me know who wants to volunteer to discuss the topic they are working 
on -
memory fragmentation, spill to disk for hash agg, external sort and schema 
change.

Also, please let me know if you have any topics you want to discuss by 
responding to this email.
We will also ask for topics at the beginning of the hangout.

Thanks,
Padma

[HANGOUT] Topics for 6/12/17

[Padma Penumarthy]

Drill hangout will be tomorrow, 10 AM PST.

In the last hangout, we talked about discussing one of the ongoing Drill 
projects in detail.
Please let me know who wants to volunteer to discuss the topic they are working 
on - 
 memory fragmentation, spill to disk for hash agg, external sort and schema 
change.

Also, please let me know if you have any topics you want to discuss by 
responding to this email. 
We will also ask for topics at the beginning of the hangout.

Thanks,
Padma

[jira] [Created] (DRILL-5560) Create configuration file for distribution specific configuration

[Padma Penumarthy (JIRA)]

Padma Penumarthy created DRILL-5560:
---

 Summary: Create configuration file for distribution specific 
configuration
 Key: DRILL-5560
 URL: https://issues.apache.org/jira/browse/DRILL-5560
 Project: Apache Drill
  Issue Type: Bug
Affects Versions: 1.10.0
Reporter: Padma Penumarthy
Assignee: Padma Penumarthy
 Fix For: 1.11.0


Create a configuration file for distribution specific settings 
"drill-distrib.conf". 
This will be used to add distribution specific configuration. 
The order in which configuration gets loaded and overriden is 
"drill-default.conf", per module configuration files "drill-module.conf", 
"drill-distrib.conf" and "drill-override.conf".




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Re: Parquet, Arrow, and Drill Roadmap

[Padma Penumarthy]

One thing I want to add is use_new_reader uses reader from parquet-mr library, 
where as
default one is drill’s native reader which is supposed to be better, 
performance wise.
But, it does not support complex types and we automatically switch to use 
reader from parquet library
when we have to read complex types.

Thanks,
Padma


On May 2, 2017, at 11:09 AM, Jinfeng Ni 
> wrote:


- What the two readers are (is one a special drill thing, is the other  a
standard reader from the parquet project?)
- What is the eventual goal here... to be able to use and switch between
both? To provide the option? To have code parity with another project?

Both readers were for reading parquet data into Drill's value vector.
The default one (when store.parquet.use_new_reader is false) was
faster (based on measurements done by people worked on the two
readers), but it could not support complex type like map/array.  The
new reader would be used by Drill either if you change the option to
true, or when the parquet data you are querying contain complex type
(even with the default option being false). Therefore, both readers
might be used by Drill code.

There was a Parquet hackathon some time ago, which aimed to make
people in different projects using parquet work together to
standardize a vectorized reader. I did not keep track of that effort.
People with better knowledge of that may share their inputs.


- Do either of the readers work with Arrow?

For now, neither works with Arrow, since Drill has not integrated with
Arrow yet. See DRILL-4455 for the latest discussion
(https://issues.apache.org/jira/browse/DRILL-4455).  I would expect
Drill's parquet reader will work with Arrow, once the integration is
done.

Re: [Drill 1.10.0] : Memory was leaked by query

[Padma Penumarthy]

Seems like you are running into  
DRILL-5435.
Try  turning off async parquet reader and see if that helps.
alter session set `store.parquet.reader.pagereader.async`=false;

Thanks,
Padma


On Apr 18, 2017, at 6:14 AM, Anup Tiwari 
> wrote:

Hi Team,

Please find following information :

*Cluster configuration :*
Number of Nodes : 5
Cores/Node : 8
RAM : 32

*Variable values :*
planner.width.max_per_node = 5
planner.width.max_per_query = 30
planner.memory.max_query_memory_per_node = 4294967296

I am getting following error on simple select statement which is coming 6
times out of 10 times, let me know if i am missing anything:

*Query :*
select udf_channel,uid from dfs.tmp.tt1 where (event = 'ajax' and ajaxurl
like '%/j_check%' and ajaxResponse like '%success%true%') limit 5;

*Error :*

ERROR o.a.d.e.w.fragment.FragmentExecutor - SYSTEM ERROR:
IllegalStateException: Memory was leaked by query. Memory leaked: (1048576)
Allocator(op:1:24:6:ParquetRowGroupScan)
100/1048576/27140096/100 (res/actual/peak/limit)


Fragment 1:24

[Error Id: a54cc1bf-794a-4143-bd82-0dd5fa3c8f52 on
prod-hadoop-101.bom-prod.aws.games24x7.com:31010]
org.apache.drill.common.exceptions.UserException: SYSTEM ERROR:
IllegalStateException: Memory was leaked by query. Memory leaked: (1048576)
Allocator(op:1:24:6:ParquetRowGroupScan)
100/1048576/27140096/100 (res/actual/peak/limit)


Fragment 1:24

[Error Id: a54cc1bf-794a-4143-bd82-0dd5fa3c8f52 on
prod-hadoop-101.bom-prod.aws.games24x7.com:31010]
   at
org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:544)
~[drill-common-1.10.0.jar:1.10.0]
   at
org.apache.drill.exec.work.fragment.FragmentExecutor.sendFinalState(FragmentExecutor.java:293)
[drill-java-exec-1.10.0.jar:1.10.0]
   at
org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:160)
[drill-java-exec-1.10.0.jar:1.10.0]
   at
org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:262)
[drill-java-exec-1.10.0.jar:1.10.0]
   at
org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38)
[drill-common-1.10.0.jar:1.10.0]
   at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[na:1.8.0_72]
   at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[na:1.8.0_72]
   at java.lang.Thread.run(Thread.java:745) [na:1.8.0_72]
Caused by: java.lang.IllegalStateException: Memory was leaked by query.
Memory leaked: (1048576)
Allocator(op:1:24:6:ParquetRowGroupScan)
100/1048576/27140096/100 (res/actual/peak/limit)

   at
org.apache.drill.exec.memory.BaseAllocator.close(BaseAllocator.java:502)
~[drill-memory-base-1.10.0.jar:1.10.0]
   at
org.apache.drill.exec.ops.OperatorContextImpl.close(OperatorContextImpl.java:149)
~[drill-java-exec-1.10.0.jar:1.10.0]
   at
org.apache.drill.exec.ops.FragmentContext.suppressingClose(FragmentContext.java:422)
~[drill-java-exec-1.10.0.jar:1.10.0]
   at
org.apache.drill.exec.ops.FragmentContext.close(FragmentContext.java:411)
~[drill-java-exec-1.10.0.jar:1.10.0]
   at
org.apache.drill.exec.work.fragment.FragmentExecutor.closeOutResources(FragmentExecutor.java:318)
[drill-java-exec-1.10.0.jar:1.10.0]
   at
org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:155)
[drill-java-exec-1.10.0.jar:1.10.0]
   ... 5 common frames omitted
2017-04-18 18:21:54,172 [BitServer-4] INFO
o.a.d.e.w.fragment.FragmentExecutor -
2709f415-c08a-13b9-9f05-fcf9008c484f:1:21: State change requested RUNNING
--> CANCELLATION_REQUESTED
2017-04-18 18:21:54,172 [BitServer-4] INFO
o.a.d.e.w.f.FragmentStatusReporter -
2709f415-c08a-13b9-9f05-fcf9008c484f:1:21: State to report:
CANCELLATION_REQUESTED
2017-04-18 18:21:54,173 [BitServer-4] WARN
o.a.d.e.w.b.ControlMessageHandler - Dropping request to cancel fragment.
2709f415-c08a-13b9-9f05-fcf9008c484f:1:24 does not exist.
2017-04-18 18:21:54,229 [2709f415-c08a-13b9-9f05-fcf9008c484f:frag:1:21]
INFO  o.a.d.e.w.fragment.FragmentExecutor -
2709f415-c08a-13b9-9f05-fcf9008c484f:1:21: State change requested
CANCELLATION_REQUESTED --> FAILED
2017-04-18 18:21:54,229 [2709f415-c08a-13b9-9f05-fcf9008c484f:frag:1:21]
INFO  o.a.d.e.w.fragment.FragmentExecutor -
2709f415-c08a-13b9-9f05-fcf9008c484f:1:21: State change requested FAILED
--> FAILED
2017-04-18 18:21:54,229 [2709f415-c08a-13b9-9f05-fcf9008c484f:frag:1:21]
INFO  o.a.d.e.w.fragment.FragmentExecutor -
2709f415-c08a-13b9-9f05-fcf9008c484f:1:21: State change requested FAILED
--> FINISHED
2017-04-18 18:21:54,230 [2709f415-c08a-13b9-9f05-fcf9008c484f:frag:1:21]
ERROR o.a.d.e.w.fragment.FragmentExecutor - SYSTEM ERROR:
IllegalStateException: Memory was leaked by query. Memory leaked: (1048576)
Allocator(op:1:21:6:ParquetRowGroupScan)

[jira] [Created] (DRILL-5429) Cache tableStats per query for MapR DB JSON Tables

[Padma Penumarthy (JIRA)]

Padma Penumarthy created DRILL-5429:
---

 Summary: Cache tableStats per query for MapR DB JSON Tables
 Key: DRILL-5429
 URL: https://issues.apache.org/jira/browse/DRILL-5429
 Project: Apache Drill
  Issue Type: Bug
  Components: Query Planning & Optimization, Storage - MapRDB
Affects Versions: 1.10.0
Reporter: Padma Penumarthy
 Fix For: 1.11.0


For MapR DB JSON Tables, cache (per query) and reuse tableStats. Getting 
tableStats is an expensive operation. Saving it and reusing it helps reduce 
query latency. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Re: Single Hdfs block per parquet file

[Padma Penumarthy]

Yes, seems like it is possible to create files with different block sizes.
We could potentially pass the configured store.parquet.block-size to the create 
call.
I will try it out and see. will let you know.

Thanks,
Padma 


> On Mar 22, 2017, at 4:16 PM, François Méthot <fmetho...@gmail.com> wrote:
> 
> Here are 2 links I could find:
> 
> http://archive.cloudera.com/cdh4/cdh/4/hadoop/api/org/apache/hadoop/fs/FileSystem.html#create(org.apache.hadoop.fs.Path,%20boolean,%20int,%20short,%20long)
> 
> http://archive.cloudera.com/cdh4/cdh/4/hadoop/api/org/apache/hadoop/fs/FileSystem.html#create(org.apache.hadoop.fs.Path,%20boolean,%20int,%20short,%20long)
> 
> Francois
> 
> On Wed, Mar 22, 2017 at 4:29 PM, Padma Penumarthy <ppenumar...@mapr.com>
> wrote:
> 
>> I think we create one file for each parquet block.
>> If underlying HDFS block size is 128 MB and parquet block size  is  >
>> 128MB,
>> it will create more blocks on HDFS.
>> Can you let me know what is the HDFS API that would allow you to
>> do otherwise ?
>> 
>> Thanks,
>> Padma
>> 
>> 
>>> On Mar 22, 2017, at 11:54 AM, François Méthot <fmetho...@gmail.com>
>> wrote:
>>> 
>>> Hi,
>>> 
>>> Is there a way to force Drill to store CTAS generated parquet file as a
>>> single block when using HDFS? Java HDFS API allows to do that, files
>> could
>>> be created with the Parquet block-size.
>>> 
>>> We are using Drill on hdfs configured with block size of 128MB. Changing
>>> this size is not an option at this point.
>>> 
>>> It would be ideal for us to have single parquet file per hdfs block,
>> setting
>>> store.parquet.block-size to 128MB would fix our issue but we end up with
>> a
>>> lot more files to deal with.
>>> 
>>> Thanks
>>> Francois
>> 
>> 

Re: Single Hdfs block per parquet file

[Padma Penumarthy]

I think we create one file for each parquet block.
If underlying HDFS block size is 128 MB and parquet block size  is  > 128MB, 
it will create more blocks on HDFS. 
Can you let me know what is the HDFS API that would allow you to
do otherwise ?

Thanks,
Padma


> On Mar 22, 2017, at 11:54 AM, François Méthot  wrote:
> 
> Hi,
> 
> Is there a way to force Drill to store CTAS generated parquet file as a
> single block when using HDFS? Java HDFS API allows to do that, files could
> be created with the Parquet block-size.
> 
> We are using Drill on hdfs configured with block size of 128MB. Changing
> this size is not an option at this point.
> 
> It would be ideal for us to have single parquet file per hdfs block, setting
> store.parquet.block-size to 128MB would fix our issue but we end up with a
> lot more files to deal with.
> 
> Thanks
> Francois

Re: Time for 1.10 release

[Padma Penumarthy]

Hi Jinfeng,

Please include DRILL-5287, DRILL-5290 and DRILL-5304.

Thanks,
Padma


> On Feb 22, 2017, at 11:16 PM, Jinfeng Ni  wrote:
> 
> Hi Drillers,
> 
> It has been almost 3 months since we release Drill 1.9. We have
> resolved plenty of fixes and improvements (closed around 88 JIRAs
> [1]). I propose that we start the 1.10 release process, and set
> Wednesday 3/1 as the cutoff day for code checkin. After 3/1, we should
> start build a release candidate.
> 
> Please reply in this email thread if you have something near complete
> and you would like to include in 1.10 release.
> 
> I volunteer as the release manager, unless someone else come forward.
> 
> Thanks,
> 
> Jinfeng
> 
> [1] https://issues.apache.org/jira/browse/DRILL/fixforversion/12338769

[jira] [Created] (DRILL-5290) Provide an option to build operator table once for built-in static functions and reuse it across queries.

[Padma Penumarthy (JIRA)]

Padma Penumarthy created DRILL-5290:
---

 Summary: Provide an option to build operator table once for 
built-in static functions and reuse it across queries.
 Key: DRILL-5290
 URL: https://issues.apache.org/jira/browse/DRILL-5290
 Project: Apache Drill
  Issue Type: Bug
Affects Versions: 1.9.0
Reporter: Padma Penumarthy
Assignee: Padma Penumarthy
 Fix For: 1.10


Currently, DrillOperatorTable which contains standard SQL operators and 
functions and Drill User Defined Functions (UDFs) (built-in and dynamic) gets 
built for each query as part of creating QueryContext. This is an expensive 
operation ( ~30 msec to build) and allocates  ~2M on heap for each query. For 
high throughput, concurrent low latency operational queries, we quickly run out 
of heap memory, causing JVM hangs. Build operator table once during startup for 
static built-in functions and save in DrillbitContext, so we can reuse it 
across queries.
Provide an system/session option to not use dynamic UDFs so we can use the 
operator table saved in DrillbitContext and avoid building each time.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (DRILL-5287) Provide option to skip updates of ephemeral state changes in Zookeeper

[Padma Penumarthy (JIRA)]

Padma Penumarthy created DRILL-5287:
---

 Summary: Provide option to skip updates of ephemeral state changes 
in Zookeeper
 Key: DRILL-5287
 URL: https://issues.apache.org/jira/browse/DRILL-5287
 Project: Apache Drill
  Issue Type: Bug
Affects Versions: 1.9.0
Reporter: Padma Penumarthy
Assignee: Padma Penumarthy
 Fix For: 1.10


We put transient profiles in zookeeper and update state as query progresses and 
changes states. It is observed that this adds latency of ~45msec for each 
update in the query execution path. This gets even worse when high number of 
concurrent queries are in progress. For concurrency=100, the average query 
response time even for short queries  is 8 sec vs 0.2 sec with these updates 
disabled. For short lived queries in a high-throughput scenario, it is of no 
value to update state changes in zookeeper. We need an option to disable these 
updates for short running operational queries.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)