Re: Create 1.4 Release branch soon?

[Venki Korukanti]

Remaining JIRAs:

DRILL-4109 (in review)
DRILL-4125
DRILL-4145 (in review)


On Tue, Dec 1, 2015 at 10:45 PM, Venki Korukanti 
wrote:

> Sure, will merge DRILL-4111.
>
> On Tue, Dec 1, 2015 at 10:43 PM, Sudheesh Katkam 
> wrote:
>
>> Venki, can you please commit DRILL-4111?
>>
>> Thank you,
>> Sudheesh
>>
>> > On Dec 1, 2015, at 6:22 PM, Jacques Nadeau  wrote:
>> >
>> > It seems like we should also try to include:
>> >
>> > DRILL-4109
>> > DRILL-4125
>> > DRILL-4111
>> >
>> > 4111 is ready to merge if someone wants to merge it. It looks like
>> Sudheesh
>> > just needs to commit.
>> >
>> > For 4125/4109, it seems like we should get Vicki's feedback if those
>> > changes are good for merge or if we should punt.
>> >
>> >
>> >
>> >
>> > --
>> > Jacques Nadeau
>> > CTO and Co-Founder, Dremio
>> >
>> > On Tue, Dec 1, 2015 at 3:59 PM, Venki Korukanti <
>> venki.koruka...@gmail.com>
>> > wrote:
>> >
>> >> I can manage the release.
>> >>
>> >> Here are the pending JIRAs so far:
>> >>
>> >> 1) DRILL-4053
>> >>
>> >> Let me know if you have any pending patches and want to get them into
>> 1.4.
>> >>
>> >> Thanks
>> >> Venki
>> >>
>> >> On Tue, Dec 1, 2015 at 10:43 AM, Parth Chandra 
>> wrote:
>> >>
>> >>> +1 on creating the release branch today.
>> >>> I'd like to get DRILL-4053 in. Patch is ready and doing one more
>> round of
>> >>> regression tests.
>> >>>
>> >>>
>> >>>
>> >>> On Tue, Dec 1, 2015 at 9:56 AM, Jacques Nadeau 
>> >> wrote:
>> >>>
>>  It is December (already!). Seems like we should create a 1.4 release
>> >>> branch
>>  soon so we can get a vote going. Does someone want to volunteer as
>> the
>>  release manager?
>> 
>>  thanks,
>>  Jacques
>> 
>>  --
>>  Jacques Nadeau
>>  CTO and Co-Founder, Dremio
>> 
>> >>>
>> >>
>>
>>
>

Re: Create 1.4 Release branch soon?

[Venki Korukanti]

I posted a comment on the commit DRILL-4126 (for some reason no emails are
sent to dev list, may be because of the multiple commits in the pull
request).

On Wed, Dec 2, 2015 at 7:53 AM, Jinfeng Ni  wrote:

> If the window is still open, I would like to have DRILL-4126,
> DRILL-4127 merged as well.
>
> Venki, could you please review the revised patch?
>
>
>
> On Wed, Dec 2, 2015 at 7:49 AM, Venki Korukanti
>  wrote:
> > Remaining JIRAs:
> >
> > DRILL-4109 (in review)
> > DRILL-4125
> > DRILL-4145 (in review)
> >
> >
> > On Tue, Dec 1, 2015 at 10:45 PM, Venki Korukanti <
> venki.koruka...@gmail.com>
> > wrote:
> >
> >> Sure, will merge DRILL-4111.
> >>
> >> On Tue, Dec 1, 2015 at 10:43 PM, Sudheesh Katkam 
> >> wrote:
> >>
> >>> Venki, can you please commit DRILL-4111?
> >>>
> >>> Thank you,
> >>> Sudheesh
> >>>
> >>> > On Dec 1, 2015, at 6:22 PM, Jacques Nadeau 
> wrote:
> >>> >
> >>> > It seems like we should also try to include:
> >>> >
> >>> > DRILL-4109
> >>> > DRILL-4125
> >>> > DRILL-4111
> >>> >
> >>> > 4111 is ready to merge if someone wants to merge it. It looks like
> >>> Sudheesh
> >>> > just needs to commit.
> >>> >
> >>> > For 4125/4109, it seems like we should get Vicki's feedback if those
> >>> > changes are good for merge or if we should punt.
> >>> >
> >>> >
> >>> >
> >>> >
> >>> > --
> >>> > Jacques Nadeau
> >>> > CTO and Co-Founder, Dremio
> >>> >
> >>> > On Tue, Dec 1, 2015 at 3:59 PM, Venki Korukanti <
> >>> venki.koruka...@gmail.com>
> >>> > wrote:
> >>> >
> >>> >> I can manage the release.
> >>> >>
> >>> >> Here are the pending JIRAs so far:
> >>> >>
> >>> >> 1) DRILL-4053
> >>> >>
> >>> >> Let me know if you have any pending patches and want to get them
> into
> >>> 1.4.
> >>> >>
> >>> >> Thanks
> >>> >> Venki
> >>> >>
> >>> >> On Tue, Dec 1, 2015 at 10:43 AM, Parth Chandra 
> >>> wrote:
> >>> >>
> >>> >>> +1 on creating the release branch today.
> >>> >>> I'd like to get DRILL-4053 in. Patch is ready and doing one more
> >>> round of
> >>> >>> regression tests.
> >>> >>>
> >>> >>>
> >>> >>>
> >>> >>> On Tue, Dec 1, 2015 at 9:56 AM, Jacques Nadeau  >
> >>> >> wrote:
> >>> >>>
> >>>  It is December (already!). Seems like we should create a 1.4
> release
> >>> >>> branch
> >>>  soon so we can get a vote going. Does someone want to volunteer as
> >>> the
> >>>  release manager?
> >>> 
> >>>  thanks,
> >>>  Jacques
> >>> 
> >>>  --
> >>>  Jacques Nadeau
> >>>  CTO and Co-Founder, Dremio
> >>> 
> >>> >>>
> >>> >>
> >>>
> >>>
> >>
>

Scanning HBase for all versions

[Ranjit Reddy]

In Drill with HBase storage plugin, is it possible to retrieve all versions of 
a column from HBase ?If it is, can I do a scan based on timestamp?
Thanks,
-Ranjit

Re: Create 1.4 Release branch soon?

[Jinfeng Ni]

If the window is still open, I would like to have DRILL-4126,
DRILL-4127 merged as well.

Venki, could you please review the revised patch?



On Wed, Dec 2, 2015 at 7:49 AM, Venki Korukanti
 wrote:
> Remaining JIRAs:
>
> DRILL-4109 (in review)
> DRILL-4125
> DRILL-4145 (in review)
>
>
> On Tue, Dec 1, 2015 at 10:45 PM, Venki Korukanti 
> wrote:
>
>> Sure, will merge DRILL-4111.
>>
>> On Tue, Dec 1, 2015 at 10:43 PM, Sudheesh Katkam 
>> wrote:
>>
>>> Venki, can you please commit DRILL-4111?
>>>
>>> Thank you,
>>> Sudheesh
>>>
>>> > On Dec 1, 2015, at 6:22 PM, Jacques Nadeau  wrote:
>>> >
>>> > It seems like we should also try to include:
>>> >
>>> > DRILL-4109
>>> > DRILL-4125
>>> > DRILL-4111
>>> >
>>> > 4111 is ready to merge if someone wants to merge it. It looks like
>>> Sudheesh
>>> > just needs to commit.
>>> >
>>> > For 4125/4109, it seems like we should get Vicki's feedback if those
>>> > changes are good for merge or if we should punt.
>>> >
>>> >
>>> >
>>> >
>>> > --
>>> > Jacques Nadeau
>>> > CTO and Co-Founder, Dremio
>>> >
>>> > On Tue, Dec 1, 2015 at 3:59 PM, Venki Korukanti <
>>> venki.koruka...@gmail.com>
>>> > wrote:
>>> >
>>> >> I can manage the release.
>>> >>
>>> >> Here are the pending JIRAs so far:
>>> >>
>>> >> 1) DRILL-4053
>>> >>
>>> >> Let me know if you have any pending patches and want to get them into
>>> 1.4.
>>> >>
>>> >> Thanks
>>> >> Venki
>>> >>
>>> >> On Tue, Dec 1, 2015 at 10:43 AM, Parth Chandra 
>>> wrote:
>>> >>
>>> >>> +1 on creating the release branch today.
>>> >>> I'd like to get DRILL-4053 in. Patch is ready and doing one more
>>> round of
>>> >>> regression tests.
>>> >>>
>>> >>>
>>> >>>
>>> >>> On Tue, Dec 1, 2015 at 9:56 AM, Jacques Nadeau 
>>> >> wrote:
>>> >>>
>>>  It is December (already!). Seems like we should create a 1.4 release
>>> >>> branch
>>>  soon so we can get a vote going. Does someone want to volunteer as
>>> the
>>>  release manager?
>>> 
>>>  thanks,
>>>  Jacques
>>> 
>>>  --
>>>  Jacques Nadeau
>>>  CTO and Co-Founder, Dremio
>>> 
>>> >>>
>>> >>
>>>
>>>
>>

[GitHub] drill pull request: Drill 4127: Reduce Hive metastore client API c...

[jinfengni]

Github user jinfengni commented on the pull request:

https://github.com/apache/drill/pull/286#issuecomment-161335544
  
@vkorukanti , the patches pass the unit test, pre-commit test, and 
impersonation test.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: Can we pass the #skipped records with RecordBatch?

[Hsuan Yi Chu]

+1 on having a framework.

But how about the #skipped records use case, and warning one (
https://github.com/abhipol/drill/commit/137059cd44ec28e8ba3bf2aa73d2c1cbcd55d604
)

Implementing the framework at this moment sounds a good timing because it
can benefit those two use cases in one shot.



On Tue, Dec 1, 2015 at 3:52 PM, Parth Chandra  wrote:

> +1 on having a framework.
> OTOH, as with the warnings implementation, we might want to go ahead with a
> simpler implementation while we get a more generic framework design in
> place.
>
> Jacques, do you have any preliminary thoughts on the framework?
>
> On Tue, Dec 1, 2015 at 2:08 PM, Julian Hyde  wrote:
>
> > +1 for a sideband mechanism.
> >
> > Sideband can also allow correlated restart of sub-queries.
> >
> > In sideband use cases you described, the messages ran in the opposite
> > direction to the data. Would the sideband also run in the same direction
> as
> > the data? If so it could carry warnings, rejected rows, progress
> > indications, and (for online aggregation[1]) notifications that a better
> > approximate query result is available.
> >
> > Julian
> >
> > [1] https://en.wikipedia.org/wiki/Online_aggregation
> >
> >
> >
> > > On Dec 1, 2015, at 1:51 PM, Jacques Nadeau  wrote:
> > >
> > > This seems like a form of sideband communication. I think we should
> have
> > a
> > > framework for this type of thing in general rather than a one-off for
> > this
> > > particular need. Other forms of sideband might be small table
> bloomfilter
> > > generation and pushdown into hbase, separate file
> assignment/partitioning
> > > providers balancing/generating scanner workloads, statistics generation
> > for
> > > adaptive execution, etc.
> > >
> > > --
> > > Jacques Nadeau
> > > CTO and Co-Founder, Dremio
> > >
> > > On Tue, Dec 1, 2015 at 11:35 AM, Hsuan Yi Chu 
> > wrote:
> > >
> > >> I am trying to deal with the following scenario:
> > >>
> > >> A bunch of minor fragments are doing things in parallel. Each of them
> > could
> > >> skip some records. Since the downstream minor fragment needs to know
> the
> > >> sum of skipped-record-counts (in order to just display or see if the
> > number
> > >> exceeds the threshold) in the upstreams, each upstream minor fragment
> > needs
> > >> to pass this scalar with RecordBatch.
> > >>
> > >> Since this seems impacting the protocol of RecordBatch, I am looking
> for
> > >> some advice here.
> > >>
> > >> Thanks.
> > >>
> >
> >
>

[jira] [Resolved] (DRILL-4047) Select with options

[Venki Korukanti (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-4047?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Venki Korukanti resolved DRILL-4047.

   Resolution: Fixed
Fix Version/s: 1.4.0

> Select with options
> ---
>
> Key: DRILL-4047
> URL: https://issues.apache.org/jira/browse/DRILL-4047
> Project: Apache Drill
>  Issue Type: Improvement
>  Components: Execution - Relational Operators
>Reporter: Julien Le Dem
>Assignee: Julien Le Dem
> Fix For: 1.4.0
>
>
> Add a mechanism to pass parameters down to the StoragePlugin when writing a 
> Select statement.
> Some discussion here:
> http://mail-archives.apache.org/mod_mbox/drill-dev/201510.mbox/%3CCAO%2Bvc4AcGK3%2B3QYvQV1-xPPdpG3Tc%2BfG%3D0xDGEUPrhd6ktHv5Q%40mail.gmail.com%3E
> http://mail-archives.apache.org/mod_mbox/drill-dev/201511.mbox/%3ccao+vc4clzylvjevisfjqtcyxb-zsmfy4bqrm-jhbidwzgqf...@mail.gmail.com%3E



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (DRILL-4152) Add additional logging and metrics to the Parquet reader

[Parth Chandra (JIRA)]

Parth Chandra created DRILL-4152:


 Summary: Add additional logging and metrics to the Parquet reader
 Key: DRILL-4152
 URL: https://issues.apache.org/jira/browse/DRILL-4152
 Project: Apache Drill
  Issue Type: Bug
  Components: Storage - Parquet
Reporter: Parth Chandra
Assignee: Parth Chandra


In some cases, we see the Parquet reader as the bottleneck in reading from the 
file system. RWSpeedTest is able to read 10x faster than the Parquet reader so 
reading from disk is not the issue. This issue is to add more instrumentation 
to the Parquet reader so speed bottlenecks can be better diagnosed.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] drill pull request: DRILL-4134: Allocator updates

[adeneche]

Github user adeneche commented on a diff in the pull request:

https://github.com/apache/drill/pull/283#discussion_r46452877
  
--- Diff: 
exec/memory/base/src/main/java/org/apache/drill/exec/memory/BaseAllocator.java 
---
@@ -0,0 +1,689 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.memory;
+
+import io.netty.buffer.ByteBufAllocator;
+import io.netty.buffer.DrillBuf;
+import io.netty.buffer.UnsafeDirectLittleEndian;
+
+import java.util.Arrays;
+import java.util.IdentityHashMap;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.concurrent.atomic.AtomicLong;
+
+import org.apache.drill.common.HistoricalLog;
+import org.apache.drill.exec.exception.OutOfMemoryException;
+import org.apache.drill.exec.memory.AllocatorManager.BufferLedger;
+import org.apache.drill.exec.ops.BufferManager;
+import org.apache.drill.exec.util.AssertionUtil;
+
+import com.google.common.base.Preconditions;
+
+public abstract class BaseAllocator extends Accountant implements 
BufferAllocator {
+  private static final org.slf4j.Logger logger = 
org.slf4j.LoggerFactory.getLogger(BaseAllocator.class);
+
+  public static final String DEBUG_ALLOCATOR = 
"drill.memory.debug.allocator";
+
+  private static final AtomicLong ID_GENERATOR = new AtomicLong(0);
+  private static final int CHUNK_SIZE = 
AllocatorManager.INNER_ALLOCATOR.getChunkSize();
+
+  public static final int DEBUG_LOG_LENGTH = 6;
+  public static final boolean DEBUG = AssertionUtil.isAssertionsEnabled()
+  || Boolean.parseBoolean(System.getProperty(DEBUG_ALLOCATOR, 
"false"));
+  private final Object DEBUG_LOCK = DEBUG ? new Object() : null;
+
+  private final BaseAllocator parentAllocator;
+  private final ByteBufAllocator thisAsByteBufAllocator;
+  private final IdentityHashMap childAllocators;
+  private final DrillBuf empty;
+
+  private volatile boolean isClosed = false; // the allocator has been 
closed
+
+  // Package exposed for sharing between AllocatorManger and BaseAllocator 
objects
+  final long id = ID_GENERATOR.incrementAndGet(); // unique ID assigned to 
each allocator
+  final String name;
+  final RootAllocator root;
+
+  // members used purely for debugging
+  private final IdentityHashMap childLedgers;
+  private final IdentityHashMap reservations;
+  private final HistoricalLog historicalLog;
+
+  protected BaseAllocator(
+  final BaseAllocator parentAllocator,
+  final String name,
+  final long initReservation,
+  final long maxAllocation) throws OutOfMemoryException {
+super(parentAllocator, initReservation, maxAllocation);
+
+if (parentAllocator != null) {
+  this.root = parentAllocator.root;
+  empty = parentAllocator.empty;
+} else if (this instanceof RootAllocator) {
+  this.root = (RootAllocator) this;
+  empty = createEmpty();
+} else {
+  throw new IllegalStateException("An parent allocator must either 
carry a root or be the root.");
+}
+
+this.parentAllocator = parentAllocator;
+this.name = name;
+
+// TODO: DRILL-4131
+// this.thisAsByteBufAllocator = new DrillByteBufAllocator(this);
+this.thisAsByteBufAllocator = 
AllocatorManager.INNER_ALLOCATOR.allocator;
+
+if (DEBUG) {
+  childAllocators = new IdentityHashMap<>();
+  reservations = new IdentityHashMap<>();
+  childLedgers = new IdentityHashMap<>();
+  historicalLog = new HistoricalLog(DEBUG_LOG_LENGTH, "allocator[%d]", 
id);
+  hist("created by \"%s\", owned = %d", name.toString(), 
this.getAllocatedMemory());
+} else {
+  childAllocators = null;
+  reservations = null;
+  historicalLog = null;
+  childLedgers = null;
+}
+
+  }
+
+  

[jira] [Resolved] (DRILL-4053) Reduce metadata cache file size

[Venki Korukanti (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-4053?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Venki Korukanti resolved DRILL-4053.

Resolution: Fixed

> Reduce metadata cache file size
> ---
>
> Key: DRILL-4053
> URL: https://issues.apache.org/jira/browse/DRILL-4053
> Project: Apache Drill
>  Issue Type: Improvement
>  Components: Metadata
>Affects Versions: 1.3.0
>Reporter: Parth Chandra
>Assignee: Parth Chandra
> Fix For: 1.4.0
>
>
> The parquet metadata cache file has fair amount of redundant metadata that 
> causes the size of the cache file to bloat. Two things that we can reduce are 
> :
> 1) Schema is repeated for every row group. We can keep a merged schema 
> (similar to what was discussed for insert into functionality) 2) The max and 
> min value in the stats are used for partition pruning when the values are the 
> same. We can keep the maxValue only and that too only if it is the same as 
> the minValue.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (DRILL-4108) Query on csv file w/ header fails with an exception when non existing column is requested

[Venki Korukanti (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-4108?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Venki Korukanti resolved DRILL-4108.

Resolution: Fixed
  Assignee: Abhijit Pol

> Query on csv file w/ header fails with an exception when non existing column 
> is requested
> -
>
> Key: DRILL-4108
> URL: https://issues.apache.org/jira/browse/DRILL-4108
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Storage - Text & CSV
>Affects Versions: 1.3.0
>Reporter: Abhi Pol
>Assignee: Abhijit Pol
> Fix For: 1.4.0
>
>
> Drill query on a csv file with header requesting column(s) that do not exists 
> in header fails with an exception.
> *Current behavior:* once extractHeader is enabled, query columns must be 
> columns from the header
> *Expected behavior:* non existing columns should appear with 'null' values 
> like default drill behavior
> {noformat}
> 0: jdbc:drill:zk=local> select Category from dfs.`/tmp/cars.csvh` limit 10;
> java.lang.ArrayIndexOutOfBoundsException: -1
>   at 
> org.apache.drill.exec.store.easy.text.compliant.FieldVarCharOutput.(FieldVarCharOutput.java:104)
>   at 
> org.apache.drill.exec.store.easy.text.compliant.CompliantTextRecordReader.setup(CompliantTextRecordReader.java:118)
>   at 
> org.apache.drill.exec.physical.impl.ScanBatch.(ScanBatch.java:108)
>   at 
> org.apache.drill.exec.store.dfs.easy.EasyFormatPlugin.getReaderBatch(EasyFormatPlugin.java:198)
>   at 
> org.apache.drill.exec.store.dfs.easy.EasyReaderBatchCreator.getBatch(EasyReaderBatchCreator.java:35)
>   at 
> org.apache.drill.exec.store.dfs.easy.EasyReaderBatchCreator.getBatch(EasyReaderBatchCreator.java:28)
>   at 
> org.apache.drill.exec.physical.impl.ImplCreator.getRecordBatch(ImplCreator.java:151)
>   at 
> org.apache.drill.exec.physical.impl.ImplCreator.getChildren(ImplCreator.java:174)
>   at 
> org.apache.drill.exec.physical.impl.ImplCreator.getRecordBatch(ImplCreator.java:131)
>   at 
> org.apache.drill.exec.physical.impl.ImplCreator.getChildren(ImplCreator.java:174)
>   at 
> org.apache.drill.exec.physical.impl.ImplCreator.getRecordBatch(ImplCreator.java:131)
>   at 
> org.apache.drill.exec.physical.impl.ImplCreator.getChildren(ImplCreator.java:174)
>   at 
> org.apache.drill.exec.physical.impl.ImplCreator.getRecordBatch(ImplCreator.java:131)
>   at 
> org.apache.drill.exec.physical.impl.ImplCreator.getChildren(ImplCreator.java:174)
>   at 
> org.apache.drill.exec.physical.impl.ImplCreator.getRootExec(ImplCreator.java:105)
>   at 
> org.apache.drill.exec.physical.impl.ImplCreator.getExec(ImplCreator.java:79)
>   at 
> org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:230)
>   at 
> org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>   at java.lang.Thread.run(Thread.java:745)
> Error: SYSTEM ERROR: ArrayIndexOutOfBoundsException: -1
> Fragment 0:0
> [Error Id: f272960e-fa2f-408e-918c-722190398cd3 on blackhole:31010] 
> (state=,code=0)
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (DRILL-4081) Handle schema changes in ExternalSort

[Venki Korukanti (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-4081?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Venki Korukanti resolved DRILL-4081.

   Resolution: Fixed
Fix Version/s: 1.4.0

> Handle schema changes in ExternalSort
> -
>
> Key: DRILL-4081
> URL: https://issues.apache.org/jira/browse/DRILL-4081
> Project: Apache Drill
>  Issue Type: Improvement
>Reporter: Steven Phillips
>Assignee: Steven Phillips
> Fix For: 1.4.0
>
>
> This improvement will make use of the Union vector to handle schema changes. 
> When a new schema appears, the schema will be "merged" with the previous 
> schema. The result will be a new schema that uses Union type to store the 
> columns where this is a type conflict. All of the batches (including the 
> batches that have already arrived) will be coerced into this new schema.
> A new comparison function will be included to handle the comparison of Union 
> type. Comparison of union type will work as follows:
> 1. All numeric types can be mutually compared, and will be compared using 
> Drill implicit cast rules.
> 2. All other types will not be compared against other types, but only among 
> values of the same type.
> 3. There will be an overall precedence of types with regards to ordering. 
> This precedence is not yet defined, but will be as part of the work on this 
> issue.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (DRILL-4151) CSV Support with multiline header

[Jaroslaw Sosnicki (JIRA)]

Jaroslaw Sosnicki created DRILL-4151:


 Summary: CSV Support with multiline header
 Key: DRILL-4151
 URL: https://issues.apache.org/jira/browse/DRILL-4151
 Project: Apache Drill
  Issue Type: Wish
  Components: Functions - Drill
Reporter: Jaroslaw Sosnicki


The modern data sources produce CSV files with two header lines:
first line contains field descriptions while second line contains filed types
Would be feasible to implement such a format in to DRILL as additional storage 
format type?

This example demonstrates an output CSV header from one of the data sources.


LDEV_COUNT,MONITORED_LDEV_COUNT,READ_IO_COUNT,READ_IO_RATE,READ_HIT_IO_COUNT,READ_HIT_RATE,WRITE_IO_COUNT,WRITE_IO_RATE,WRITE_HIT_IO_COUNT,WRITE_HIT_RATE,READ_MBYTES,READ_XFER_RATE,WRITE_MBYTES,WRITE_XFER_RATE,INTERVAL,INPUT_RECORD_TYPE,RECORD_TIME
ulong,ulong,double,double,double,double,double,double,double,double,double,double,double,double,ulong,string(8),time_t







--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (DRILL-4094) Respect -DskipTests=true for JDBC plugin tests

[Venki Korukanti (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-4094?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Venki Korukanti resolved DRILL-4094.

   Resolution: Fixed
Fix Version/s: 1.4.0

> Respect -DskipTests=true for JDBC plugin tests
> --
>
> Key: DRILL-4094
> URL: https://issues.apache.org/jira/browse/DRILL-4094
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Storage - Other
>Reporter: Andrew
>Assignee: Andrew
>Priority: Trivial
> Fix For: 1.4.0
>
>
> The maven config for the JDBC storage plugin does not respect the -DskipTests 
> option.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] drill pull request: DRILL-4134: Allocator updates

[adeneche]

Github user adeneche commented on a diff in the pull request:

https://github.com/apache/drill/pull/283#discussion_r46453393
  
--- Diff: 
exec/memory/base/src/main/java/org/apache/drill/exec/memory/BaseAllocator.java 
---
@@ -0,0 +1,689 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.memory;
+
+import io.netty.buffer.ByteBufAllocator;
+import io.netty.buffer.DrillBuf;
+import io.netty.buffer.UnsafeDirectLittleEndian;
+
+import java.util.Arrays;
+import java.util.IdentityHashMap;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.concurrent.atomic.AtomicLong;
+
+import org.apache.drill.common.HistoricalLog;
+import org.apache.drill.exec.exception.OutOfMemoryException;
+import org.apache.drill.exec.memory.AllocatorManager.BufferLedger;
+import org.apache.drill.exec.ops.BufferManager;
+import org.apache.drill.exec.util.AssertionUtil;
+
+import com.google.common.base.Preconditions;
+
+public abstract class BaseAllocator extends Accountant implements 
BufferAllocator {
+  private static final org.slf4j.Logger logger = 
org.slf4j.LoggerFactory.getLogger(BaseAllocator.class);
+
+  public static final String DEBUG_ALLOCATOR = 
"drill.memory.debug.allocator";
+
+  private static final AtomicLong ID_GENERATOR = new AtomicLong(0);
+  private static final int CHUNK_SIZE = 
AllocatorManager.INNER_ALLOCATOR.getChunkSize();
+
+  public static final int DEBUG_LOG_LENGTH = 6;
+  public static final boolean DEBUG = AssertionUtil.isAssertionsEnabled()
+  || Boolean.parseBoolean(System.getProperty(DEBUG_ALLOCATOR, 
"false"));
+  private final Object DEBUG_LOCK = DEBUG ? new Object() : null;
+
+  private final BaseAllocator parentAllocator;
+  private final ByteBufAllocator thisAsByteBufAllocator;
+  private final IdentityHashMap childAllocators;
+  private final DrillBuf empty;
+
+  private volatile boolean isClosed = false; // the allocator has been 
closed
+
+  // Package exposed for sharing between AllocatorManger and BaseAllocator 
objects
+  final long id = ID_GENERATOR.incrementAndGet(); // unique ID assigned to 
each allocator
+  final String name;
+  final RootAllocator root;
+
+  // members used purely for debugging
+  private final IdentityHashMap childLedgers;
+  private final IdentityHashMap reservations;
+  private final HistoricalLog historicalLog;
+
+  protected BaseAllocator(
+  final BaseAllocator parentAllocator,
+  final String name,
+  final long initReservation,
+  final long maxAllocation) throws OutOfMemoryException {
+super(parentAllocator, initReservation, maxAllocation);
+
+if (parentAllocator != null) {
+  this.root = parentAllocator.root;
+  empty = parentAllocator.empty;
+} else if (this instanceof RootAllocator) {
+  this.root = (RootAllocator) this;
+  empty = createEmpty();
+} else {
+  throw new IllegalStateException("An parent allocator must either 
carry a root or be the root.");
+}
+
+this.parentAllocator = parentAllocator;
+this.name = name;
+
+// TODO: DRILL-4131
+// this.thisAsByteBufAllocator = new DrillByteBufAllocator(this);
+this.thisAsByteBufAllocator = 
AllocatorManager.INNER_ALLOCATOR.allocator;
+
+if (DEBUG) {
+  childAllocators = new IdentityHashMap<>();
+  reservations = new IdentityHashMap<>();
+  childLedgers = new IdentityHashMap<>();
+  historicalLog = new HistoricalLog(DEBUG_LOG_LENGTH, "allocator[%d]", 
id);
+  hist("created by \"%s\", owned = %d", name.toString(), 
this.getAllocatedMemory());
+} else {
+  childAllocators = null;
+  reservations = null;
+  historicalLog = null;
+  childLedgers = null;
+}
+
+  }
+
+  

[GitHub] drill pull request: DRILL-4111: turn tests off in travis as they d...

[julienledem]

Github user julienledem commented on the pull request:

https://github.com/apache/drill/pull/267#issuecomment-161412734
  
Thank you!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: Create 1.4 Release branch soon?

[Julien Le Dem]

DRILL-4124 is ready to merge: https://github.com/apache/drill/pull/281
It is a small change.

On Wed, Dec 2, 2015 at 7:58 AM, Venki Korukanti 
wrote:

> I posted a comment on the commit DRILL-4126 (for some reason no emails are
> sent to dev list, may be because of the multiple commits in the pull
> request).
>
> On Wed, Dec 2, 2015 at 7:53 AM, Jinfeng Ni  wrote:
>
> > If the window is still open, I would like to have DRILL-4126,
> > DRILL-4127 merged as well.
> >
> > Venki, could you please review the revised patch?
> >
> >
> >
> > On Wed, Dec 2, 2015 at 7:49 AM, Venki Korukanti
> >  wrote:
> > > Remaining JIRAs:
> > >
> > > DRILL-4109 (in review)
> > > DRILL-4125
> > > DRILL-4145 (in review)
> > >
> > >
> > > On Tue, Dec 1, 2015 at 10:45 PM, Venki Korukanti <
> > venki.koruka...@gmail.com>
> > > wrote:
> > >
> > >> Sure, will merge DRILL-4111.
> > >>
> > >> On Tue, Dec 1, 2015 at 10:43 PM, Sudheesh Katkam <
> skat...@maprtech.com>
> > >> wrote:
> > >>
> > >>> Venki, can you please commit DRILL-4111?
> > >>>
> > >>> Thank you,
> > >>> Sudheesh
> > >>>
> > >>> > On Dec 1, 2015, at 6:22 PM, Jacques Nadeau 
> > wrote:
> > >>> >
> > >>> > It seems like we should also try to include:
> > >>> >
> > >>> > DRILL-4109
> > >>> > DRILL-4125
> > >>> > DRILL-4111
> > >>> >
> > >>> > 4111 is ready to merge if someone wants to merge it. It looks like
> > >>> Sudheesh
> > >>> > just needs to commit.
> > >>> >
> > >>> > For 4125/4109, it seems like we should get Vicki's feedback if
> those
> > >>> > changes are good for merge or if we should punt.
> > >>> >
> > >>> >
> > >>> >
> > >>> >
> > >>> > --
> > >>> > Jacques Nadeau
> > >>> > CTO and Co-Founder, Dremio
> > >>> >
> > >>> > On Tue, Dec 1, 2015 at 3:59 PM, Venki Korukanti <
> > >>> venki.koruka...@gmail.com>
> > >>> > wrote:
> > >>> >
> > >>> >> I can manage the release.
> > >>> >>
> > >>> >> Here are the pending JIRAs so far:
> > >>> >>
> > >>> >> 1) DRILL-4053
> > >>> >>
> > >>> >> Let me know if you have any pending patches and want to get them
> > into
> > >>> 1.4.
> > >>> >>
> > >>> >> Thanks
> > >>> >> Venki
> > >>> >>
> > >>> >> On Tue, Dec 1, 2015 at 10:43 AM, Parth Chandra  >
> > >>> wrote:
> > >>> >>
> > >>> >>> +1 on creating the release branch today.
> > >>> >>> I'd like to get DRILL-4053 in. Patch is ready and doing one more
> > >>> round of
> > >>> >>> regression tests.
> > >>> >>>
> > >>> >>>
> > >>> >>>
> > >>> >>> On Tue, Dec 1, 2015 at 9:56 AM, Jacques Nadeau <
> jacq...@dremio.com
> > >
> > >>> >> wrote:
> > >>> >>>
> > >>>  It is December (already!). Seems like we should create a 1.4
> > release
> > >>> >>> branch
> > >>>  soon so we can get a vote going. Does someone want to volunteer
> as
> > >>> the
> > >>>  release manager?
> > >>> 
> > >>>  thanks,
> > >>>  Jacques
> > >>> 
> > >>>  --
> > >>>  Jacques Nadeau
> > >>>  CTO and Co-Founder, Dremio
> > >>> 
> > >>> >>>
> > >>> >>
> > >>>
> > >>>
> > >>
> >
>



-- 
Julien

[GitHub] drill pull request: DRILL-4134: Allocator updates

[adeneche]

Github user adeneche commented on a diff in the pull request:

https://github.com/apache/drill/pull/283#discussion_r46466143
  
--- Diff: 
exec/memory/base/src/main/java/org/apache/drill/exec/memory/BaseAllocator.java 
---
@@ -0,0 +1,689 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.memory;
+
+import io.netty.buffer.ByteBufAllocator;
+import io.netty.buffer.DrillBuf;
+import io.netty.buffer.UnsafeDirectLittleEndian;
+
+import java.util.Arrays;
+import java.util.IdentityHashMap;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.concurrent.atomic.AtomicLong;
+
+import org.apache.drill.common.HistoricalLog;
+import org.apache.drill.exec.exception.OutOfMemoryException;
+import org.apache.drill.exec.memory.AllocatorManager.BufferLedger;
+import org.apache.drill.exec.ops.BufferManager;
+import org.apache.drill.exec.util.AssertionUtil;
+
+import com.google.common.base.Preconditions;
+
+public abstract class BaseAllocator extends Accountant implements 
BufferAllocator {
+  private static final org.slf4j.Logger logger = 
org.slf4j.LoggerFactory.getLogger(BaseAllocator.class);
+
+  public static final String DEBUG_ALLOCATOR = 
"drill.memory.debug.allocator";
+
+  private static final AtomicLong ID_GENERATOR = new AtomicLong(0);
+  private static final int CHUNK_SIZE = 
AllocatorManager.INNER_ALLOCATOR.getChunkSize();
+
+  public static final int DEBUG_LOG_LENGTH = 6;
+  public static final boolean DEBUG = AssertionUtil.isAssertionsEnabled()
+  || Boolean.parseBoolean(System.getProperty(DEBUG_ALLOCATOR, 
"false"));
+  private final Object DEBUG_LOCK = DEBUG ? new Object() : null;
+
+  private final BaseAllocator parentAllocator;
+  private final ByteBufAllocator thisAsByteBufAllocator;
+  private final IdentityHashMap childAllocators;
+  private final DrillBuf empty;
+
+  private volatile boolean isClosed = false; // the allocator has been 
closed
+
+  // Package exposed for sharing between AllocatorManger and BaseAllocator 
objects
+  final long id = ID_GENERATOR.incrementAndGet(); // unique ID assigned to 
each allocator
+  final String name;
+  final RootAllocator root;
+
+  // members used purely for debugging
+  private final IdentityHashMap childLedgers;
+  private final IdentityHashMap reservations;
+  private final HistoricalLog historicalLog;
+
+  protected BaseAllocator(
+  final BaseAllocator parentAllocator,
+  final String name,
+  final long initReservation,
+  final long maxAllocation) throws OutOfMemoryException {
+super(parentAllocator, initReservation, maxAllocation);
+
+if (parentAllocator != null) {
+  this.root = parentAllocator.root;
+  empty = parentAllocator.empty;
+} else if (this instanceof RootAllocator) {
+  this.root = (RootAllocator) this;
+  empty = createEmpty();
+} else {
+  throw new IllegalStateException("An parent allocator must either 
carry a root or be the root.");
+}
+
+this.parentAllocator = parentAllocator;
+this.name = name;
+
+// TODO: DRILL-4131
+// this.thisAsByteBufAllocator = new DrillByteBufAllocator(this);
+this.thisAsByteBufAllocator = 
AllocatorManager.INNER_ALLOCATOR.allocator;
+
+if (DEBUG) {
+  childAllocators = new IdentityHashMap<>();
+  reservations = new IdentityHashMap<>();
+  childLedgers = new IdentityHashMap<>();
+  historicalLog = new HistoricalLog(DEBUG_LOG_LENGTH, "allocator[%d]", 
id);
+  hist("created by \"%s\", owned = %d", name.toString(), 
this.getAllocatedMemory());
+} else {
+  childAllocators = null;
+  reservations = null;
+  historicalLog = null;
+  childLedgers = null;
+}
+
+  }
+
+  

[GitHub] drill pull request: DRILL-4134: Allocator updates

[adeneche]

Github user adeneche commented on a diff in the pull request:

https://github.com/apache/drill/pull/283#discussion_r46468356
  
--- Diff: 
exec/memory/base/src/main/java/org/apache/drill/exec/memory/BaseAllocator.java 
---
@@ -0,0 +1,689 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.memory;
+
+import io.netty.buffer.ByteBufAllocator;
+import io.netty.buffer.DrillBuf;
+import io.netty.buffer.UnsafeDirectLittleEndian;
+
+import java.util.Arrays;
+import java.util.IdentityHashMap;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.concurrent.atomic.AtomicLong;
+
+import org.apache.drill.common.HistoricalLog;
+import org.apache.drill.exec.exception.OutOfMemoryException;
+import org.apache.drill.exec.memory.AllocatorManager.BufferLedger;
+import org.apache.drill.exec.ops.BufferManager;
+import org.apache.drill.exec.util.AssertionUtil;
+
+import com.google.common.base.Preconditions;
+
+public abstract class BaseAllocator extends Accountant implements 
BufferAllocator {
+  private static final org.slf4j.Logger logger = 
org.slf4j.LoggerFactory.getLogger(BaseAllocator.class);
+
+  public static final String DEBUG_ALLOCATOR = 
"drill.memory.debug.allocator";
+
+  private static final AtomicLong ID_GENERATOR = new AtomicLong(0);
+  private static final int CHUNK_SIZE = 
AllocatorManager.INNER_ALLOCATOR.getChunkSize();
+
+  public static final int DEBUG_LOG_LENGTH = 6;
+  public static final boolean DEBUG = AssertionUtil.isAssertionsEnabled()
+  || Boolean.parseBoolean(System.getProperty(DEBUG_ALLOCATOR, 
"false"));
+  private final Object DEBUG_LOCK = DEBUG ? new Object() : null;
+
+  private final BaseAllocator parentAllocator;
+  private final ByteBufAllocator thisAsByteBufAllocator;
+  private final IdentityHashMap childAllocators;
+  private final DrillBuf empty;
+
+  private volatile boolean isClosed = false; // the allocator has been 
closed
+
+  // Package exposed for sharing between AllocatorManger and BaseAllocator 
objects
+  final long id = ID_GENERATOR.incrementAndGet(); // unique ID assigned to 
each allocator
+  final String name;
+  final RootAllocator root;
+
+  // members used purely for debugging
+  private final IdentityHashMap childLedgers;
+  private final IdentityHashMap reservations;
+  private final HistoricalLog historicalLog;
+
+  protected BaseAllocator(
+  final BaseAllocator parentAllocator,
+  final String name,
+  final long initReservation,
+  final long maxAllocation) throws OutOfMemoryException {
+super(parentAllocator, initReservation, maxAllocation);
+
+if (parentAllocator != null) {
+  this.root = parentAllocator.root;
+  empty = parentAllocator.empty;
+} else if (this instanceof RootAllocator) {
+  this.root = (RootAllocator) this;
+  empty = createEmpty();
+} else {
+  throw new IllegalStateException("An parent allocator must either 
carry a root or be the root.");
+}
+
+this.parentAllocator = parentAllocator;
+this.name = name;
+
+// TODO: DRILL-4131
+// this.thisAsByteBufAllocator = new DrillByteBufAllocator(this);
+this.thisAsByteBufAllocator = 
AllocatorManager.INNER_ALLOCATOR.allocator;
+
+if (DEBUG) {
+  childAllocators = new IdentityHashMap<>();
+  reservations = new IdentityHashMap<>();
+  childLedgers = new IdentityHashMap<>();
+  historicalLog = new HistoricalLog(DEBUG_LOG_LENGTH, "allocator[%d]", 
id);
+  hist("created by \"%s\", owned = %d", name.toString(), 
this.getAllocatedMemory());
+} else {
+  childAllocators = null;
+  reservations = null;
+  historicalLog = null;
+  childLedgers = null;
+}
+
+  }
+
+  

[jira] [Resolved] (DRILL-2419) UDF that returns string representation of expression type

[Mehant Baid (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-2419?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mehant Baid resolved DRILL-2419.

   Resolution: Fixed
Fix Version/s: (was: Future)
   1.3.0

Fixed in eb6325dc9b59291582cd7d3c3e5d02efd5d15906. 



> UDF that returns string representation of expression type
> -
>
> Key: DRILL-2419
> URL: https://issues.apache.org/jira/browse/DRILL-2419
> Project: Apache Drill
>  Issue Type: Improvement
>  Components: Functions - Drill
>Reporter: Victoria Markman
>Assignee: Steven Phillips
> Fix For: 1.3.0
>
>
> Suggested name: typeof (credit goes to Aman)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Create 1.4 Release branch soon?

[Venki Korukanti]

Sure, we can include DRILL-4124. I will merge it soon. Any idea when the
following JIRAs are going to be ready to commit?

DRILL-4109 (in review)
DRILL-4125
DRILL-4145 (in review)

Jinfeng and I discussed about including metastore caching and decided to
move it to 1.5.

Lets make 5pm as the cutoff time for 1.4 branch.

Thanks
Venki

On Wed, Dec 2, 2015 at 11:42 AM, Julien Le Dem  wrote:

> DRILL-4124 is ready to merge: https://github.com/apache/drill/pull/281
> It is a small change.
>
> On Wed, Dec 2, 2015 at 7:58 AM, Venki Korukanti  >
> wrote:
>
> > I posted a comment on the commit DRILL-4126 (for some reason no emails
> are
> > sent to dev list, may be because of the multiple commits in the pull
> > request).
> >
> > On Wed, Dec 2, 2015 at 7:53 AM, Jinfeng Ni 
> wrote:
> >
> > > If the window is still open, I would like to have DRILL-4126,
> > > DRILL-4127 merged as well.
> > >
> > > Venki, could you please review the revised patch?
> > >
> > >
> > >
> > > On Wed, Dec 2, 2015 at 7:49 AM, Venki Korukanti
> > >  wrote:
> > > > Remaining JIRAs:
> > > >
> > > > DRILL-4109 (in review)
> > > > DRILL-4125
> > > > DRILL-4145 (in review)
> > > >
> > > >
> > > > On Tue, Dec 1, 2015 at 10:45 PM, Venki Korukanti <
> > > venki.koruka...@gmail.com>
> > > > wrote:
> > > >
> > > >> Sure, will merge DRILL-4111.
> > > >>
> > > >> On Tue, Dec 1, 2015 at 10:43 PM, Sudheesh Katkam <
> > skat...@maprtech.com>
> > > >> wrote:
> > > >>
> > > >>> Venki, can you please commit DRILL-4111?
> > > >>>
> > > >>> Thank you,
> > > >>> Sudheesh
> > > >>>
> > > >>> > On Dec 1, 2015, at 6:22 PM, Jacques Nadeau 
> > > wrote:
> > > >>> >
> > > >>> > It seems like we should also try to include:
> > > >>> >
> > > >>> > DRILL-4109
> > > >>> > DRILL-4125
> > > >>> > DRILL-4111
> > > >>> >
> > > >>> > 4111 is ready to merge if someone wants to merge it. It looks
> like
> > > >>> Sudheesh
> > > >>> > just needs to commit.
> > > >>> >
> > > >>> > For 4125/4109, it seems like we should get Vicki's feedback if
> > those
> > > >>> > changes are good for merge or if we should punt.
> > > >>> >
> > > >>> >
> > > >>> >
> > > >>> >
> > > >>> > --
> > > >>> > Jacques Nadeau
> > > >>> > CTO and Co-Founder, Dremio
> > > >>> >
> > > >>> > On Tue, Dec 1, 2015 at 3:59 PM, Venki Korukanti <
> > > >>> venki.koruka...@gmail.com>
> > > >>> > wrote:
> > > >>> >
> > > >>> >> I can manage the release.
> > > >>> >>
> > > >>> >> Here are the pending JIRAs so far:
> > > >>> >>
> > > >>> >> 1) DRILL-4053
> > > >>> >>
> > > >>> >> Let me know if you have any pending patches and want to get them
> > > into
> > > >>> 1.4.
> > > >>> >>
> > > >>> >> Thanks
> > > >>> >> Venki
> > > >>> >>
> > > >>> >> On Tue, Dec 1, 2015 at 10:43 AM, Parth Chandra <
> par...@apache.org
> > >
> > > >>> wrote:
> > > >>> >>
> > > >>> >>> +1 on creating the release branch today.
> > > >>> >>> I'd like to get DRILL-4053 in. Patch is ready and doing one
> more
> > > >>> round of
> > > >>> >>> regression tests.
> > > >>> >>>
> > > >>> >>>
> > > >>> >>>
> > > >>> >>> On Tue, Dec 1, 2015 at 9:56 AM, Jacques Nadeau <
> > jacq...@dremio.com
> > > >
> > > >>> >> wrote:
> > > >>> >>>
> > > >>>  It is December (already!). Seems like we should create a 1.4
> > > release
> > > >>> >>> branch
> > > >>>  soon so we can get a vote going. Does someone want to
> volunteer
> > as
> > > >>> the
> > > >>>  release manager?
> > > >>> 
> > > >>>  thanks,
> > > >>>  Jacques
> > > >>> 
> > > >>>  --
> > > >>>  Jacques Nadeau
> > > >>>  CTO and Co-Founder, Dremio
> > > >>> 
> > > >>> >>>
> > > >>> >>
> > > >>>
> > > >>>
> > > >>
> > >
> >
>
>
>
> --
> Julien
>

[jira] [Created] (DRILL-4153) Query with "select columnName, *" fails with IOB

[Abhishek Girish (JIRA)]

Abhishek Girish created DRILL-4153:
--

 Summary: Query with "select columnName, *" fails with IOB
 Key: DRILL-4153
 URL: https://issues.apache.org/jira/browse/DRILL-4153
 Project: Apache Drill
  Issue Type: Bug
  Components: SQL Parser
Reporter: Abhishek Girish


Query with select columnName, * fails with IOB:

{code}
select c_customer_sk, * as c from dfs.tpcds_sf1_parquet.customer limit 1;

Query Failed: An Error Occurred
org.apache.drill.common.exceptions.UserRemoteException: SYSTEM ERROR: 
IndexOutOfBoundsException: index (-1) must not be negative [Error Id: 
05b29f77-7668-48e3-a423-a13f0fe9e79a on atsqa6c64.qa.lab:31010]
{code}

This issue isn't seen when * precedes columnName.

Log attached. 




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Can we pass the #skipped records with RecordBatch?

[Zelaine Fong]

I agree with Parth's earlier comment.  Getting a framework in place sounds
like the right long term solution.  But given that we have more immediate
needs for something sooner rather than later (warnings + skip records), can
we implement something simpler now until we have the more general framework
in place?

-- Zelaine

On Wed, Dec 2, 2015 at 10:43 AM, Hsuan Yi Chu  wrote:

> +1 on having a framework.
>
> But how about the #skipped records use case, and warning one (
>
> https://github.com/abhipol/drill/commit/137059cd44ec28e8ba3bf2aa73d2c1cbcd55d604
> )
>
> Implementing the framework at this moment sounds a good timing because it
> can benefit those two use cases in one shot.
>
>
>
> On Tue, Dec 1, 2015 at 3:52 PM, Parth Chandra  wrote:
>
> > +1 on having a framework.
> > OTOH, as with the warnings implementation, we might want to go ahead
> with a
> > simpler implementation while we get a more generic framework design in
> > place.
> >
> > Jacques, do you have any preliminary thoughts on the framework?
> >
> > On Tue, Dec 1, 2015 at 2:08 PM, Julian Hyde  wrote:
> >
> > > +1 for a sideband mechanism.
> > >
> > > Sideband can also allow correlated restart of sub-queries.
> > >
> > > In sideband use cases you described, the messages ran in the opposite
> > > direction to the data. Would the sideband also run in the same
> direction
> > as
> > > the data? If so it could carry warnings, rejected rows, progress
> > > indications, and (for online aggregation[1]) notifications that a
> better
> > > approximate query result is available.
> > >
> > > Julian
> > >
> > > [1] https://en.wikipedia.org/wiki/Online_aggregation
> > >
> > >
> > >
> > > > On Dec 1, 2015, at 1:51 PM, Jacques Nadeau 
> wrote:
> > > >
> > > > This seems like a form of sideband communication. I think we should
> > have
> > > a
> > > > framework for this type of thing in general rather than a one-off for
> > > this
> > > > particular need. Other forms of sideband might be small table
> > bloomfilter
> > > > generation and pushdown into hbase, separate file
> > assignment/partitioning
> > > > providers balancing/generating scanner workloads, statistics
> generation
> > > for
> > > > adaptive execution, etc.
> > > >
> > > > --
> > > > Jacques Nadeau
> > > > CTO and Co-Founder, Dremio
> > > >
> > > > On Tue, Dec 1, 2015 at 11:35 AM, Hsuan Yi Chu 
> > > wrote:
> > > >
> > > >> I am trying to deal with the following scenario:
> > > >>
> > > >> A bunch of minor fragments are doing things in parallel. Each of
> them
> > > could
> > > >> skip some records. Since the downstream minor fragment needs to know
> > the
> > > >> sum of skipped-record-counts (in order to just display or see if the
> > > number
> > > >> exceeds the threshold) in the upstreams, each upstream minor
> fragment
> > > needs
> > > >> to pass this scalar with RecordBatch.
> > > >>
> > > >> Since this seems impacting the protocol of RecordBatch, I am looking
> > for
> > > >> some advice here.
> > > >>
> > > >> Thanks.
> > > >>
> > >
> > >
> >
>

Codehale Metrics JMXReporter Disabled?

[Sudheesh Katkam]

Jacques,

Do you happen to remember why JMXReporter was disabled 
?

Thank you,
Sudheesh

[GitHub] drill pull request: DRILL-4134: Allocator updates

[adeneche]

Github user adeneche commented on a diff in the pull request:

https://github.com/apache/drill/pull/283#discussion_r46477028
  
--- Diff: 
exec/memory/base/src/main/java/org/apache/drill/exec/memory/AllocatorManager.java
 ---
@@ -0,0 +1,386 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.memory;
+
+import static org.apache.drill.exec.memory.BaseAllocator.indent;
+import io.netty.buffer.DrillBuf;
+import io.netty.buffer.PooledByteBufAllocatorL;
+import io.netty.buffer.UnsafeDirectLittleEndian;
+
+import java.util.IdentityHashMap;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.concurrent.locks.Lock;
+import java.util.concurrent.locks.ReadWriteLock;
+import java.util.concurrent.locks.ReentrantReadWriteLock;
+
+import org.apache.drill.common.HistoricalLog;
+import org.apache.drill.exec.memory.BaseAllocator.Verbosity;
+import org.apache.drill.exec.metrics.DrillMetrics;
+import org.apache.drill.exec.ops.BufferManager;
+
+import com.carrotsearch.hppc.LongObjectOpenHashMap;
+import com.google.common.base.Preconditions;
+
+/**
+ * Manages the relationship between one or more allocators and a 
particular UDLE. Ensures that one allocator owns the
+ * memory that multiple allocators may be referencing. Manages a 
BufferLedger between each of its associated allocators.
+ * This class is also responsible for managing when memory is allocated 
and returned to the Netty-based
+ * PooledByteBufAllocatorL.
+ *
+ * The only reason that this isn't package private is we're forced to put 
DrillBuf in Netty's package which need access
+ * to these objects or methods.
+ *
+ * Threading: AllocatorManager manages thread-safety internally. 
Operations within the context of a single BufferLedger
+ * are lockless in nature and can be leveraged by multiple threads. 
Operations that cross the context of two ledgers
+ * will acquire a lock on the AllocatorManager instance. Important note, 
there is one AllocatorManager per
+ * UnsafeDirectLittleEndian buffer allocation. As such, there will be 
thousands of these in a typical query. The
+ * contention of acquiring a lock on AllocatorManager should be very low.
+ *
+ */
+public class AllocatorManager {
+  // private static final org.slf4j.Logger logger = 
org.slf4j.LoggerFactory.getLogger(AllocatorManager.class);
+
+  private static final AtomicLong LEDGER_ID_GENERATOR = new AtomicLong(0);
+  static final PooledByteBufAllocatorL INNER_ALLOCATOR = new 
PooledByteBufAllocatorL(DrillMetrics.getInstance());
+
+  private final RootAllocator root;
+  private volatile BufferLedger owningLedger;
+  private final int size;
+  private final UnsafeDirectLittleEndian underlying;
+  private final ReadWriteLock lock = new ReentrantReadWriteLock();
+  private final LongObjectOpenHashMap map = new 
LongObjectOpenHashMap<>();
+  private final AutoCloseableLock readLock = new 
AutoCloseableLock(lock.readLock());
+  private final AutoCloseableLock writeLock = new 
AutoCloseableLock(lock.writeLock());
+  private final IdentityHashMap buffers =
+  BaseAllocator.DEBUG ? new IdentityHashMap() : null;
+
+  AllocatorManager(BaseAllocator accountingAllocator, int size) {
+Preconditions.checkNotNull(accountingAllocator);
+this.root = accountingAllocator.root;
+this.underlying = INNER_ALLOCATOR.allocate(size);
+this.owningLedger = associate(accountingAllocator);
+this.size = underlying.capacity();
+  }
+
+  /**
+   * Associate the existing underlying buffer with a new allocator.
+   *
+   * @param allocator
+   *  The target allocator to associate this buffer with.
+   * @return The Ledger (new or existing) that associates the underlying 
buffer to this new ledger.
+   */
+  public BufferLedger associate(BaseAllocator allocator) {
   

Re: Create 1.4 Release branch soon?

[Jacques Nadeau]

I believe 4109 is ready but if I understand correctly, it can't go in
without 4125. Amit said he needs another hour for that.

On 4145: Venki, can someone on your side running any extended/customer
tests you have against this to make sure that it doesn't cause any
regressions. We've run on our side without issue but I'd like to get as
broad of coverage as possible to ensure no issues.

--
Jacques Nadeau
CTO and Co-Founder, Dremio

On Wed, Dec 2, 2015 at 11:52 AM, Venki Korukanti 
wrote:

> Sure, we can include DRILL-4124. I will merge it soon. Any idea when the
> following JIRAs are going to be ready to commit?
>
> DRILL-4109 (in review)
> DRILL-4125
> DRILL-4145 (in review)
>
> Jinfeng and I discussed about including metastore caching and decided to
> move it to 1.5.
>
> Lets make 5pm as the cutoff time for 1.4 branch.
>
> Thanks
> Venki
>
> On Wed, Dec 2, 2015 at 11:42 AM, Julien Le Dem  wrote:
>
> > DRILL-4124 is ready to merge: https://github.com/apache/drill/pull/281
> > It is a small change.
> >
> > On Wed, Dec 2, 2015 at 7:58 AM, Venki Korukanti <
> venki.koruka...@gmail.com
> > >
> > wrote:
> >
> > > I posted a comment on the commit DRILL-4126 (for some reason no emails
> > are
> > > sent to dev list, may be because of the multiple commits in the pull
> > > request).
> > >
> > > On Wed, Dec 2, 2015 at 7:53 AM, Jinfeng Ni 
> > wrote:
> > >
> > > > If the window is still open, I would like to have DRILL-4126,
> > > > DRILL-4127 merged as well.
> > > >
> > > > Venki, could you please review the revised patch?
> > > >
> > > >
> > > >
> > > > On Wed, Dec 2, 2015 at 7:49 AM, Venki Korukanti
> > > >  wrote:
> > > > > Remaining JIRAs:
> > > > >
> > > > > DRILL-4109 (in review)
> > > > > DRILL-4125
> > > > > DRILL-4145 (in review)
> > > > >
> > > > >
> > > > > On Tue, Dec 1, 2015 at 10:45 PM, Venki Korukanti <
> > > > venki.koruka...@gmail.com>
> > > > > wrote:
> > > > >
> > > > >> Sure, will merge DRILL-4111.
> > > > >>
> > > > >> On Tue, Dec 1, 2015 at 10:43 PM, Sudheesh Katkam <
> > > skat...@maprtech.com>
> > > > >> wrote:
> > > > >>
> > > > >>> Venki, can you please commit DRILL-4111?
> > > > >>>
> > > > >>> Thank you,
> > > > >>> Sudheesh
> > > > >>>
> > > > >>> > On Dec 1, 2015, at 6:22 PM, Jacques Nadeau  >
> > > > wrote:
> > > > >>> >
> > > > >>> > It seems like we should also try to include:
> > > > >>> >
> > > > >>> > DRILL-4109
> > > > >>> > DRILL-4125
> > > > >>> > DRILL-4111
> > > > >>> >
> > > > >>> > 4111 is ready to merge if someone wants to merge it. It looks
> > like
> > > > >>> Sudheesh
> > > > >>> > just needs to commit.
> > > > >>> >
> > > > >>> > For 4125/4109, it seems like we should get Vicki's feedback if
> > > those
> > > > >>> > changes are good for merge or if we should punt.
> > > > >>> >
> > > > >>> >
> > > > >>> >
> > > > >>> >
> > > > >>> > --
> > > > >>> > Jacques Nadeau
> > > > >>> > CTO and Co-Founder, Dremio
> > > > >>> >
> > > > >>> > On Tue, Dec 1, 2015 at 3:59 PM, Venki Korukanti <
> > > > >>> venki.koruka...@gmail.com>
> > > > >>> > wrote:
> > > > >>> >
> > > > >>> >> I can manage the release.
> > > > >>> >>
> > > > >>> >> Here are the pending JIRAs so far:
> > > > >>> >>
> > > > >>> >> 1) DRILL-4053
> > > > >>> >>
> > > > >>> >> Let me know if you have any pending patches and want to get
> them
> > > > into
> > > > >>> 1.4.
> > > > >>> >>
> > > > >>> >> Thanks
> > > > >>> >> Venki
> > > > >>> >>
> > > > >>> >> On Tue, Dec 1, 2015 at 10:43 AM, Parth Chandra <
> > par...@apache.org
> > > >
> > > > >>> wrote:
> > > > >>> >>
> > > > >>> >>> +1 on creating the release branch today.
> > > > >>> >>> I'd like to get DRILL-4053 in. Patch is ready and doing one
> > more
> > > > >>> round of
> > > > >>> >>> regression tests.
> > > > >>> >>>
> > > > >>> >>>
> > > > >>> >>>
> > > > >>> >>> On Tue, Dec 1, 2015 at 9:56 AM, Jacques Nadeau <
> > > jacq...@dremio.com
> > > > >
> > > > >>> >> wrote:
> > > > >>> >>>
> > > > >>>  It is December (already!). Seems like we should create a 1.4
> > > > release
> > > > >>> >>> branch
> > > > >>>  soon so we can get a vote going. Does someone want to
> > volunteer
> > > as
> > > > >>> the
> > > > >>>  release manager?
> > > > >>> 
> > > > >>>  thanks,
> > > > >>>  Jacques
> > > > >>> 
> > > > >>>  --
> > > > >>>  Jacques Nadeau
> > > > >>>  CTO and Co-Founder, Dremio
> > > > >>> 
> > > > >>> >>>
> > > > >>> >>
> > > > >>>
> > > > >>>
> > > > >>
> > > >
> > >
> >
> >
> >
> > --
> > Julien
> >
>

Re: Create 1.4 Release branch soon?

[Venki Korukanti]

Sure, I will trigger a regression run with DRILL-4145.

Thanks
Venki

On Wed, Dec 2, 2015 at 1:36 PM, Jacques Nadeau  wrote:

> I believe 4109 is ready but if I understand correctly, it can't go in
> without 4125. Amit said he needs another hour for that.
>
> On 4145: Venki, can someone on your side running any extended/customer
> tests you have against this to make sure that it doesn't cause any
> regressions. We've run on our side without issue but I'd like to get as
> broad of coverage as possible to ensure no issues.
>
> --
> Jacques Nadeau
> CTO and Co-Founder, Dremio
>
> On Wed, Dec 2, 2015 at 11:52 AM, Venki Korukanti <
> venki.koruka...@gmail.com>
> wrote:
>
> > Sure, we can include DRILL-4124. I will merge it soon. Any idea when the
> > following JIRAs are going to be ready to commit?
> >
> > DRILL-4109 (in review)
> > DRILL-4125
> > DRILL-4145 (in review)
> >
> > Jinfeng and I discussed about including metastore caching and decided to
> > move it to 1.5.
> >
> > Lets make 5pm as the cutoff time for 1.4 branch.
> >
> > Thanks
> > Venki
> >
> > On Wed, Dec 2, 2015 at 11:42 AM, Julien Le Dem 
> wrote:
> >
> > > DRILL-4124 is ready to merge: https://github.com/apache/drill/pull/281
> > > It is a small change.
> > >
> > > On Wed, Dec 2, 2015 at 7:58 AM, Venki Korukanti <
> > venki.koruka...@gmail.com
> > > >
> > > wrote:
> > >
> > > > I posted a comment on the commit DRILL-4126 (for some reason no
> emails
> > > are
> > > > sent to dev list, may be because of the multiple commits in the pull
> > > > request).
> > > >
> > > > On Wed, Dec 2, 2015 at 7:53 AM, Jinfeng Ni 
> > > wrote:
> > > >
> > > > > If the window is still open, I would like to have DRILL-4126,
> > > > > DRILL-4127 merged as well.
> > > > >
> > > > > Venki, could you please review the revised patch?
> > > > >
> > > > >
> > > > >
> > > > > On Wed, Dec 2, 2015 at 7:49 AM, Venki Korukanti
> > > > >  wrote:
> > > > > > Remaining JIRAs:
> > > > > >
> > > > > > DRILL-4109 (in review)
> > > > > > DRILL-4125
> > > > > > DRILL-4145 (in review)
> > > > > >
> > > > > >
> > > > > > On Tue, Dec 1, 2015 at 10:45 PM, Venki Korukanti <
> > > > > venki.koruka...@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > >> Sure, will merge DRILL-4111.
> > > > > >>
> > > > > >> On Tue, Dec 1, 2015 at 10:43 PM, Sudheesh Katkam <
> > > > skat...@maprtech.com>
> > > > > >> wrote:
> > > > > >>
> > > > > >>> Venki, can you please commit DRILL-4111?
> > > > > >>>
> > > > > >>> Thank you,
> > > > > >>> Sudheesh
> > > > > >>>
> > > > > >>> > On Dec 1, 2015, at 6:22 PM, Jacques Nadeau <
> jacq...@dremio.com
> > >
> > > > > wrote:
> > > > > >>> >
> > > > > >>> > It seems like we should also try to include:
> > > > > >>> >
> > > > > >>> > DRILL-4109
> > > > > >>> > DRILL-4125
> > > > > >>> > DRILL-4111
> > > > > >>> >
> > > > > >>> > 4111 is ready to merge if someone wants to merge it. It looks
> > > like
> > > > > >>> Sudheesh
> > > > > >>> > just needs to commit.
> > > > > >>> >
> > > > > >>> > For 4125/4109, it seems like we should get Vicki's feedback
> if
> > > > those
> > > > > >>> > changes are good for merge or if we should punt.
> > > > > >>> >
> > > > > >>> >
> > > > > >>> >
> > > > > >>> >
> > > > > >>> > --
> > > > > >>> > Jacques Nadeau
> > > > > >>> > CTO and Co-Founder, Dremio
> > > > > >>> >
> > > > > >>> > On Tue, Dec 1, 2015 at 3:59 PM, Venki Korukanti <
> > > > > >>> venki.koruka...@gmail.com>
> > > > > >>> > wrote:
> > > > > >>> >
> > > > > >>> >> I can manage the release.
> > > > > >>> >>
> > > > > >>> >> Here are the pending JIRAs so far:
> > > > > >>> >>
> > > > > >>> >> 1) DRILL-4053
> > > > > >>> >>
> > > > > >>> >> Let me know if you have any pending patches and want to get
> > them
> > > > > into
> > > > > >>> 1.4.
> > > > > >>> >>
> > > > > >>> >> Thanks
> > > > > >>> >> Venki
> > > > > >>> >>
> > > > > >>> >> On Tue, Dec 1, 2015 at 10:43 AM, Parth Chandra <
> > > par...@apache.org
> > > > >
> > > > > >>> wrote:
> > > > > >>> >>
> > > > > >>> >>> +1 on creating the release branch today.
> > > > > >>> >>> I'd like to get DRILL-4053 in. Patch is ready and doing one
> > > more
> > > > > >>> round of
> > > > > >>> >>> regression tests.
> > > > > >>> >>>
> > > > > >>> >>>
> > > > > >>> >>>
> > > > > >>> >>> On Tue, Dec 1, 2015 at 9:56 AM, Jacques Nadeau <
> > > > jacq...@dremio.com
> > > > > >
> > > > > >>> >> wrote:
> > > > > >>> >>>
> > > > > >>>  It is December (already!). Seems like we should create a
> 1.4
> > > > > release
> > > > > >>> >>> branch
> > > > > >>>  soon so we can get a vote going. Does someone want to
> > > volunteer
> > > > as
> > > > > >>> the
> > > > > >>>  release manager?
> > > > > >>> 
> > > > > >>>  thanks,
> > > > > >>>  Jacques
> > > > > >>> 
> > > > > >>>  --
> > > > > >>>  Jacques Nadeau
> > > > > >>>  CTO and Co-Founder, Dremio
> > > > > >>> 
> > > > > >>> >>>
> > > > > >>> >>
> > > > > 

Re: Codehale Metrics JMXReporter Disabled?

[Jacques Nadeau]

Afraid not. I think it may have been debug reasons and shouldn't have been
merged.

--
Jacques Nadeau
CTO and Co-Founder, Dremio

On Wed, Dec 2, 2015 at 1:44 PM, Sudheesh Katkam 
wrote:

> Jacques,
>
> Do you happen to remember why JMXReporter was disabled <
> https://github.com/apache/drill/commit/4eea03a052d1b0a9190b9d1512088da9f81cc037#diff-5a33b44e2c23b1f09d338938c8f1e742R47
> >?
>
> Thank you,
> Sudheesh

Re: Create 1.4 Release branch soon?

[Venki Korukanti]

For DRILL-4145: ran the regression suite which includes customer and
extended tests. No regressions found.

On Wed, Dec 2, 2015 at 1:41 PM, Venki Korukanti 
wrote:

> Sure, I will trigger a regression run with DRILL-4145.
>
> Thanks
> Venki
>
> On Wed, Dec 2, 2015 at 1:36 PM, Jacques Nadeau  wrote:
>
>> I believe 4109 is ready but if I understand correctly, it can't go in
>> without 4125. Amit said he needs another hour for that.
>>
>> On 4145: Venki, can someone on your side running any extended/customer
>> tests you have against this to make sure that it doesn't cause any
>> regressions. We've run on our side without issue but I'd like to get as
>> broad of coverage as possible to ensure no issues.
>>
>> --
>> Jacques Nadeau
>> CTO and Co-Founder, Dremio
>>
>> On Wed, Dec 2, 2015 at 11:52 AM, Venki Korukanti <
>> venki.koruka...@gmail.com>
>> wrote:
>>
>> > Sure, we can include DRILL-4124. I will merge it soon. Any idea when the
>> > following JIRAs are going to be ready to commit?
>> >
>> > DRILL-4109 (in review)
>> > DRILL-4125
>> > DRILL-4145 (in review)
>> >
>> > Jinfeng and I discussed about including metastore caching and decided to
>> > move it to 1.5.
>> >
>> > Lets make 5pm as the cutoff time for 1.4 branch.
>> >
>> > Thanks
>> > Venki
>> >
>> > On Wed, Dec 2, 2015 at 11:42 AM, Julien Le Dem 
>> wrote:
>> >
>> > > DRILL-4124 is ready to merge:
>> https://github.com/apache/drill/pull/281
>> > > It is a small change.
>> > >
>> > > On Wed, Dec 2, 2015 at 7:58 AM, Venki Korukanti <
>> > venki.koruka...@gmail.com
>> > > >
>> > > wrote:
>> > >
>> > > > I posted a comment on the commit DRILL-4126 (for some reason no
>> emails
>> > > are
>> > > > sent to dev list, may be because of the multiple commits in the pull
>> > > > request).
>> > > >
>> > > > On Wed, Dec 2, 2015 at 7:53 AM, Jinfeng Ni 
>> > > wrote:
>> > > >
>> > > > > If the window is still open, I would like to have DRILL-4126,
>> > > > > DRILL-4127 merged as well.
>> > > > >
>> > > > > Venki, could you please review the revised patch?
>> > > > >
>> > > > >
>> > > > >
>> > > > > On Wed, Dec 2, 2015 at 7:49 AM, Venki Korukanti
>> > > > >  wrote:
>> > > > > > Remaining JIRAs:
>> > > > > >
>> > > > > > DRILL-4109 (in review)
>> > > > > > DRILL-4125
>> > > > > > DRILL-4145 (in review)
>> > > > > >
>> > > > > >
>> > > > > > On Tue, Dec 1, 2015 at 10:45 PM, Venki Korukanti <
>> > > > > venki.koruka...@gmail.com>
>> > > > > > wrote:
>> > > > > >
>> > > > > >> Sure, will merge DRILL-4111.
>> > > > > >>
>> > > > > >> On Tue, Dec 1, 2015 at 10:43 PM, Sudheesh Katkam <
>> > > > skat...@maprtech.com>
>> > > > > >> wrote:
>> > > > > >>
>> > > > > >>> Venki, can you please commit DRILL-4111?
>> > > > > >>>
>> > > > > >>> Thank you,
>> > > > > >>> Sudheesh
>> > > > > >>>
>> > > > > >>> > On Dec 1, 2015, at 6:22 PM, Jacques Nadeau <
>> jacq...@dremio.com
>> > >
>> > > > > wrote:
>> > > > > >>> >
>> > > > > >>> > It seems like we should also try to include:
>> > > > > >>> >
>> > > > > >>> > DRILL-4109
>> > > > > >>> > DRILL-4125
>> > > > > >>> > DRILL-4111
>> > > > > >>> >
>> > > > > >>> > 4111 is ready to merge if someone wants to merge it. It
>> looks
>> > > like
>> > > > > >>> Sudheesh
>> > > > > >>> > just needs to commit.
>> > > > > >>> >
>> > > > > >>> > For 4125/4109, it seems like we should get Vicki's feedback
>> if
>> > > > those
>> > > > > >>> > changes are good for merge or if we should punt.
>> > > > > >>> >
>> > > > > >>> >
>> > > > > >>> >
>> > > > > >>> >
>> > > > > >>> > --
>> > > > > >>> > Jacques Nadeau
>> > > > > >>> > CTO and Co-Founder, Dremio
>> > > > > >>> >
>> > > > > >>> > On Tue, Dec 1, 2015 at 3:59 PM, Venki Korukanti <
>> > > > > >>> venki.koruka...@gmail.com>
>> > > > > >>> > wrote:
>> > > > > >>> >
>> > > > > >>> >> I can manage the release.
>> > > > > >>> >>
>> > > > > >>> >> Here are the pending JIRAs so far:
>> > > > > >>> >>
>> > > > > >>> >> 1) DRILL-4053
>> > > > > >>> >>
>> > > > > >>> >> Let me know if you have any pending patches and want to get
>> > them
>> > > > > into
>> > > > > >>> 1.4.
>> > > > > >>> >>
>> > > > > >>> >> Thanks
>> > > > > >>> >> Venki
>> > > > > >>> >>
>> > > > > >>> >> On Tue, Dec 1, 2015 at 10:43 AM, Parth Chandra <
>> > > par...@apache.org
>> > > > >
>> > > > > >>> wrote:
>> > > > > >>> >>
>> > > > > >>> >>> +1 on creating the release branch today.
>> > > > > >>> >>> I'd like to get DRILL-4053 in. Patch is ready and doing
>> one
>> > > more
>> > > > > >>> round of
>> > > > > >>> >>> regression tests.
>> > > > > >>> >>>
>> > > > > >>> >>>
>> > > > > >>> >>>
>> > > > > >>> >>> On Tue, Dec 1, 2015 at 9:56 AM, Jacques Nadeau <
>> > > > jacq...@dremio.com
>> > > > > >
>> > > > > >>> >> wrote:
>> > > > > >>> >>>
>> > > > > >>>  It is December (already!). Seems like we should create a
>> 1.4
>> > > > > release
>> > > > > >>> >>> branch
>> > > > > >>>  soon so we can get a vote 

Re: Create 1.4 Release branch soon?

[Steven Phillips]

Okay, I'm going to go ahead and merge DRILL-4145

On Wed, Dec 2, 2015 at 2:45 PM, Venki Korukanti 
wrote:

> For DRILL-4145: ran the regression suite which includes customer and
> extended tests. No regressions found.
>
> On Wed, Dec 2, 2015 at 1:41 PM, Venki Korukanti  >
> wrote:
>
> > Sure, I will trigger a regression run with DRILL-4145.
> >
> > Thanks
> > Venki
> >
> > On Wed, Dec 2, 2015 at 1:36 PM, Jacques Nadeau 
> wrote:
> >
> >> I believe 4109 is ready but if I understand correctly, it can't go in
> >> without 4125. Amit said he needs another hour for that.
> >>
> >> On 4145: Venki, can someone on your side running any extended/customer
> >> tests you have against this to make sure that it doesn't cause any
> >> regressions. We've run on our side without issue but I'd like to get as
> >> broad of coverage as possible to ensure no issues.
> >>
> >> --
> >> Jacques Nadeau
> >> CTO and Co-Founder, Dremio
> >>
> >> On Wed, Dec 2, 2015 at 11:52 AM, Venki Korukanti <
> >> venki.koruka...@gmail.com>
> >> wrote:
> >>
> >> > Sure, we can include DRILL-4124. I will merge it soon. Any idea when
> the
> >> > following JIRAs are going to be ready to commit?
> >> >
> >> > DRILL-4109 (in review)
> >> > DRILL-4125
> >> > DRILL-4145 (in review)
> >> >
> >> > Jinfeng and I discussed about including metastore caching and decided
> to
> >> > move it to 1.5.
> >> >
> >> > Lets make 5pm as the cutoff time for 1.4 branch.
> >> >
> >> > Thanks
> >> > Venki
> >> >
> >> > On Wed, Dec 2, 2015 at 11:42 AM, Julien Le Dem 
> >> wrote:
> >> >
> >> > > DRILL-4124 is ready to merge:
> >> https://github.com/apache/drill/pull/281
> >> > > It is a small change.
> >> > >
> >> > > On Wed, Dec 2, 2015 at 7:58 AM, Venki Korukanti <
> >> > venki.koruka...@gmail.com
> >> > > >
> >> > > wrote:
> >> > >
> >> > > > I posted a comment on the commit DRILL-4126 (for some reason no
> >> emails
> >> > > are
> >> > > > sent to dev list, may be because of the multiple commits in the
> pull
> >> > > > request).
> >> > > >
> >> > > > On Wed, Dec 2, 2015 at 7:53 AM, Jinfeng Ni  >
> >> > > wrote:
> >> > > >
> >> > > > > If the window is still open, I would like to have DRILL-4126,
> >> > > > > DRILL-4127 merged as well.
> >> > > > >
> >> > > > > Venki, could you please review the revised patch?
> >> > > > >
> >> > > > >
> >> > > > >
> >> > > > > On Wed, Dec 2, 2015 at 7:49 AM, Venki Korukanti
> >> > > > >  wrote:
> >> > > > > > Remaining JIRAs:
> >> > > > > >
> >> > > > > > DRILL-4109 (in review)
> >> > > > > > DRILL-4125
> >> > > > > > DRILL-4145 (in review)
> >> > > > > >
> >> > > > > >
> >> > > > > > On Tue, Dec 1, 2015 at 10:45 PM, Venki Korukanti <
> >> > > > > venki.koruka...@gmail.com>
> >> > > > > > wrote:
> >> > > > > >
> >> > > > > >> Sure, will merge DRILL-4111.
> >> > > > > >>
> >> > > > > >> On Tue, Dec 1, 2015 at 10:43 PM, Sudheesh Katkam <
> >> > > > skat...@maprtech.com>
> >> > > > > >> wrote:
> >> > > > > >>
> >> > > > > >>> Venki, can you please commit DRILL-4111?
> >> > > > > >>>
> >> > > > > >>> Thank you,
> >> > > > > >>> Sudheesh
> >> > > > > >>>
> >> > > > > >>> > On Dec 1, 2015, at 6:22 PM, Jacques Nadeau <
> >> jacq...@dremio.com
> >> > >
> >> > > > > wrote:
> >> > > > > >>> >
> >> > > > > >>> > It seems like we should also try to include:
> >> > > > > >>> >
> >> > > > > >>> > DRILL-4109
> >> > > > > >>> > DRILL-4125
> >> > > > > >>> > DRILL-4111
> >> > > > > >>> >
> >> > > > > >>> > 4111 is ready to merge if someone wants to merge it. It
> >> looks
> >> > > like
> >> > > > > >>> Sudheesh
> >> > > > > >>> > just needs to commit.
> >> > > > > >>> >
> >> > > > > >>> > For 4125/4109, it seems like we should get Vicki's
> feedback
> >> if
> >> > > > those
> >> > > > > >>> > changes are good for merge or if we should punt.
> >> > > > > >>> >
> >> > > > > >>> >
> >> > > > > >>> >
> >> > > > > >>> >
> >> > > > > >>> > --
> >> > > > > >>> > Jacques Nadeau
> >> > > > > >>> > CTO and Co-Founder, Dremio
> >> > > > > >>> >
> >> > > > > >>> > On Tue, Dec 1, 2015 at 3:59 PM, Venki Korukanti <
> >> > > > > >>> venki.koruka...@gmail.com>
> >> > > > > >>> > wrote:
> >> > > > > >>> >
> >> > > > > >>> >> I can manage the release.
> >> > > > > >>> >>
> >> > > > > >>> >> Here are the pending JIRAs so far:
> >> > > > > >>> >>
> >> > > > > >>> >> 1) DRILL-4053
> >> > > > > >>> >>
> >> > > > > >>> >> Let me know if you have any pending patches and want to
> get
> >> > them
> >> > > > > into
> >> > > > > >>> 1.4.
> >> > > > > >>> >>
> >> > > > > >>> >> Thanks
> >> > > > > >>> >> Venki
> >> > > > > >>> >>
> >> > > > > >>> >> On Tue, Dec 1, 2015 at 10:43 AM, Parth Chandra <
> >> > > par...@apache.org
> >> > > > >
> >> > > > > >>> wrote:
> >> > > > > >>> >>
> >> > > > > >>> >>> +1 on creating the release branch today.
> >> > > > > >>> >>> I'd like to get DRILL-4053 in. Patch is ready and doing
> >> one
> >> > > 

Re: Codehale Metrics JMXReporter Disabled?

[yuliya Feldman]

I know I was enabling it for my small project I did with Drill for Strata in 
Feb.
If you enable JMX I believe there was a bug somewhere lurking around - I think 
static order init issue. I may have a code somewhere with enabling JMX and 
fixing the issue I described.
  From: Jacques Nadeau 
 To: dev  
 Sent: Wednesday, December 2, 2015 1:54 PM
 Subject: Re: Codehale Metrics JMXReporter Disabled?
   
Afraid not. I think it may have been debug reasons and shouldn't have been
merged.

--
Jacques Nadeau
CTO and Co-Founder, Dremio



On Wed, Dec 2, 2015 at 1:44 PM, Sudheesh Katkam 
wrote:

> Jacques,
>
> Do you happen to remember why JMXReporter was disabled <
> https://github.com/apache/drill/commit/4eea03a052d1b0a9190b9d1512088da9f81cc037#diff-5a33b44e2c23b1f09d338938c8f1e742R47
> >?
>
> Thank you,
> Sudheesh


  

[GitHub] drill pull request: DRILL-4134: Allocator updates

[adeneche]

Github user adeneche commented on a diff in the pull request:

https://github.com/apache/drill/pull/283#discussion_r46477393
  
--- Diff: 
exec/memory/base/src/main/java/org/apache/drill/exec/memory/BaseAllocator.java 
---
@@ -0,0 +1,689 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.memory;
+
+import io.netty.buffer.ByteBufAllocator;
+import io.netty.buffer.DrillBuf;
+import io.netty.buffer.UnsafeDirectLittleEndian;
+
+import java.util.Arrays;
+import java.util.IdentityHashMap;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.concurrent.atomic.AtomicLong;
+
+import org.apache.drill.common.HistoricalLog;
+import org.apache.drill.exec.exception.OutOfMemoryException;
+import org.apache.drill.exec.memory.AllocatorManager.BufferLedger;
+import org.apache.drill.exec.ops.BufferManager;
+import org.apache.drill.exec.util.AssertionUtil;
+
+import com.google.common.base.Preconditions;
+
+public abstract class BaseAllocator extends Accountant implements 
BufferAllocator {
+  private static final org.slf4j.Logger logger = 
org.slf4j.LoggerFactory.getLogger(BaseAllocator.class);
+
+  public static final String DEBUG_ALLOCATOR = 
"drill.memory.debug.allocator";
+
+  private static final AtomicLong ID_GENERATOR = new AtomicLong(0);
+  private static final int CHUNK_SIZE = 
AllocatorManager.INNER_ALLOCATOR.getChunkSize();
+
+  public static final int DEBUG_LOG_LENGTH = 6;
+  public static final boolean DEBUG = AssertionUtil.isAssertionsEnabled()
+  || Boolean.parseBoolean(System.getProperty(DEBUG_ALLOCATOR, 
"false"));
+  private final Object DEBUG_LOCK = DEBUG ? new Object() : null;
+
+  private final BaseAllocator parentAllocator;
+  private final ByteBufAllocator thisAsByteBufAllocator;
+  private final IdentityHashMap childAllocators;
+  private final DrillBuf empty;
+
+  private volatile boolean isClosed = false; // the allocator has been 
closed
+
+  // Package exposed for sharing between AllocatorManger and BaseAllocator 
objects
+  final long id = ID_GENERATOR.incrementAndGet(); // unique ID assigned to 
each allocator
+  final String name;
+  final RootAllocator root;
+
+  // members used purely for debugging
+  private final IdentityHashMap childLedgers;
+  private final IdentityHashMap reservations;
+  private final HistoricalLog historicalLog;
+
+  protected BaseAllocator(
+  final BaseAllocator parentAllocator,
+  final String name,
+  final long initReservation,
+  final long maxAllocation) throws OutOfMemoryException {
+super(parentAllocator, initReservation, maxAllocation);
+
+if (parentAllocator != null) {
+  this.root = parentAllocator.root;
+  empty = parentAllocator.empty;
+} else if (this instanceof RootAllocator) {
+  this.root = (RootAllocator) this;
+  empty = createEmpty();
+} else {
+  throw new IllegalStateException("An parent allocator must either 
carry a root or be the root.");
+}
+
+this.parentAllocator = parentAllocator;
+this.name = name;
+
+// TODO: DRILL-4131
+// this.thisAsByteBufAllocator = new DrillByteBufAllocator(this);
+this.thisAsByteBufAllocator = 
AllocatorManager.INNER_ALLOCATOR.allocator;
+
+if (DEBUG) {
+  childAllocators = new IdentityHashMap<>();
+  reservations = new IdentityHashMap<>();
+  childLedgers = new IdentityHashMap<>();
+  historicalLog = new HistoricalLog(DEBUG_LOG_LENGTH, "allocator[%d]", 
id);
+  hist("created by \"%s\", owned = %d", name.toString(), 
this.getAllocatedMemory());
+} else {
+  childAllocators = null;
+  reservations = null;
+  historicalLog = null;
+  childLedgers = null;
+}
+
+  }
+
+  

[GitHub] drill pull request: DRILL-4145: Handle empty final field in Text r...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/drill/pull/287


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: Create 1.4 Release branch soon?

[Venki Korukanti]

For DRILL-4109 and DRILL-4125, Vicky is not available today to verify. If
the changes are reviewed lets merge them today. Once the branch is cut
today, MapR will do the release sanity for next couple of days before RC0
voting goes out. If any issues are found, we still have time to fix before
the RC0 voting.

Thanks
Venki

On Wed, Dec 2, 2015 at 4:02 PM, Jacques Nadeau  wrote:

> Sounds good.
>
> --
> Jacques Nadeau
> CTO and Co-Founder, Dremio
>
> On Wed, Dec 2, 2015 at 2:47 PM, Steven Phillips  wrote:
>
> > Okay, I'm going to go ahead and merge DRILL-4145
> >
> > On Wed, Dec 2, 2015 at 2:45 PM, Venki Korukanti <
> venki.koruka...@gmail.com
> > >
> > wrote:
> >
> > > For DRILL-4145: ran the regression suite which includes customer and
> > > extended tests. No regressions found.
> > >
> > > On Wed, Dec 2, 2015 at 1:41 PM, Venki Korukanti <
> > venki.koruka...@gmail.com
> > > >
> > > wrote:
> > >
> > > > Sure, I will trigger a regression run with DRILL-4145.
> > > >
> > > > Thanks
> > > > Venki
> > > >
> > > > On Wed, Dec 2, 2015 at 1:36 PM, Jacques Nadeau 
> > > wrote:
> > > >
> > > >> I believe 4109 is ready but if I understand correctly, it can't go
> in
> > > >> without 4125. Amit said he needs another hour for that.
> > > >>
> > > >> On 4145: Venki, can someone on your side running any
> extended/customer
> > > >> tests you have against this to make sure that it doesn't cause any
> > > >> regressions. We've run on our side without issue but I'd like to get
> > as
> > > >> broad of coverage as possible to ensure no issues.
> > > >>
> > > >> --
> > > >> Jacques Nadeau
> > > >> CTO and Co-Founder, Dremio
> > > >>
> > > >> On Wed, Dec 2, 2015 at 11:52 AM, Venki Korukanti <
> > > >> venki.koruka...@gmail.com>
> > > >> wrote:
> > > >>
> > > >> > Sure, we can include DRILL-4124. I will merge it soon. Any idea
> when
> > > the
> > > >> > following JIRAs are going to be ready to commit?
> > > >> >
> > > >> > DRILL-4109 (in review)
> > > >> > DRILL-4125
> > > >> > DRILL-4145 (in review)
> > > >> >
> > > >> > Jinfeng and I discussed about including metastore caching and
> > decided
> > > to
> > > >> > move it to 1.5.
> > > >> >
> > > >> > Lets make 5pm as the cutoff time for 1.4 branch.
> > > >> >
> > > >> > Thanks
> > > >> > Venki
> > > >> >
> > > >> > On Wed, Dec 2, 2015 at 11:42 AM, Julien Le Dem  >
> > > >> wrote:
> > > >> >
> > > >> > > DRILL-4124 is ready to merge:
> > > >> https://github.com/apache/drill/pull/281
> > > >> > > It is a small change.
> > > >> > >
> > > >> > > On Wed, Dec 2, 2015 at 7:58 AM, Venki Korukanti <
> > > >> > venki.koruka...@gmail.com
> > > >> > > >
> > > >> > > wrote:
> > > >> > >
> > > >> > > > I posted a comment on the commit DRILL-4126 (for some reason
> no
> > > >> emails
> > > >> > > are
> > > >> > > > sent to dev list, may be because of the multiple commits in
> the
> > > pull
> > > >> > > > request).
> > > >> > > >
> > > >> > > > On Wed, Dec 2, 2015 at 7:53 AM, Jinfeng Ni <
> > jinfengn...@gmail.com
> > > >
> > > >> > > wrote:
> > > >> > > >
> > > >> > > > > If the window is still open, I would like to have
> DRILL-4126,
> > > >> > > > > DRILL-4127 merged as well.
> > > >> > > > >
> > > >> > > > > Venki, could you please review the revised patch?
> > > >> > > > >
> > > >> > > > >
> > > >> > > > >
> > > >> > > > > On Wed, Dec 2, 2015 at 7:49 AM, Venki Korukanti
> > > >> > > > >  wrote:
> > > >> > > > > > Remaining JIRAs:
> > > >> > > > > >
> > > >> > > > > > DRILL-4109 (in review)
> > > >> > > > > > DRILL-4125
> > > >> > > > > > DRILL-4145 (in review)
> > > >> > > > > >
> > > >> > > > > >
> > > >> > > > > > On Tue, Dec 1, 2015 at 10:45 PM, Venki Korukanti <
> > > >> > > > > venki.koruka...@gmail.com>
> > > >> > > > > > wrote:
> > > >> > > > > >
> > > >> > > > > >> Sure, will merge DRILL-4111.
> > > >> > > > > >>
> > > >> > > > > >> On Tue, Dec 1, 2015 at 10:43 PM, Sudheesh Katkam <
> > > >> > > > skat...@maprtech.com>
> > > >> > > > > >> wrote:
> > > >> > > > > >>
> > > >> > > > > >>> Venki, can you please commit DRILL-4111?
> > > >> > > > > >>>
> > > >> > > > > >>> Thank you,
> > > >> > > > > >>> Sudheesh
> > > >> > > > > >>>
> > > >> > > > > >>> > On Dec 1, 2015, at 6:22 PM, Jacques Nadeau <
> > > >> jacq...@dremio.com
> > > >> > >
> > > >> > > > > wrote:
> > > >> > > > > >>> >
> > > >> > > > > >>> > It seems like we should also try to include:
> > > >> > > > > >>> >
> > > >> > > > > >>> > DRILL-4109
> > > >> > > > > >>> > DRILL-4125
> > > >> > > > > >>> > DRILL-4111
> > > >> > > > > >>> >
> > > >> > > > > >>> > 4111 is ready to merge if someone wants to merge it.
> It
> > > >> looks
> > > >> > > like
> > > >> > > > > >>> Sudheesh
> > > >> > > > > >>> > just needs to commit.
> > > >> > > > > >>> >
> > > >> > > > > >>> > For 4125/4109, it seems like we should get Vicki's
> > > feedback
> > > >> if
> > > >> > > > those
> > > >> > > > > >>> > changes are good for merge or 

[GitHub] drill pull request: DRILL-4109 Fix NPE in RecordIterator.

[StevenMPhillips]

Github user StevenMPhillips commented on the pull request:

https://github.com/apache/drill/pull/282#issuecomment-161478598
  
+1


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request: DRILL-4109 Fix NPE in RecordIterator.

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/drill/pull/282


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: Create 1.4 Release branch soon?

[Jacques Nadeau]

I think we should roll forward to 1.5-S..
On Dec 2, 2015 6:44 PM, "Venki Korukanti"  wrote:

> 1.4.0 branch is cut and available here:
> https://github.com/vkorukanti/drill/tree/1.4.0.
>
> Should I move the master to 1.5.0-SNAPSHOT now or wait until an RC is
> passed?
>
> Thanks
> Venki
>
> On Wed, Dec 2, 2015 at 4:25 PM, Venki Korukanti  >
> wrote:
>
> > For DRILL-4109 and DRILL-4125, Vicky is not available today to verify. If
> > the changes are reviewed lets merge them today. Once the branch is cut
> > today, MapR will do the release sanity for next couple of days before RC0
> > voting goes out. If any issues are found, we still have time to fix
> before
> > the RC0 voting.
> >
> > Thanks
> > Venki
> >
> > On Wed, Dec 2, 2015 at 4:02 PM, Jacques Nadeau 
> wrote:
> >
> >> Sounds good.
> >>
> >> --
> >> Jacques Nadeau
> >> CTO and Co-Founder, Dremio
> >>
> >> On Wed, Dec 2, 2015 at 2:47 PM, Steven Phillips 
> >> wrote:
> >>
> >> > Okay, I'm going to go ahead and merge DRILL-4145
> >> >
> >> > On Wed, Dec 2, 2015 at 2:45 PM, Venki Korukanti <
> >> venki.koruka...@gmail.com
> >> > >
> >> > wrote:
> >> >
> >> > > For DRILL-4145: ran the regression suite which includes customer and
> >> > > extended tests. No regressions found.
> >> > >
> >> > > On Wed, Dec 2, 2015 at 1:41 PM, Venki Korukanti <
> >> > venki.koruka...@gmail.com
> >> > > >
> >> > > wrote:
> >> > >
> >> > > > Sure, I will trigger a regression run with DRILL-4145.
> >> > > >
> >> > > > Thanks
> >> > > > Venki
> >> > > >
> >> > > > On Wed, Dec 2, 2015 at 1:36 PM, Jacques Nadeau <
> jacq...@dremio.com>
> >> > > wrote:
> >> > > >
> >> > > >> I believe 4109 is ready but if I understand correctly, it can't
> go
> >> in
> >> > > >> without 4125. Amit said he needs another hour for that.
> >> > > >>
> >> > > >> On 4145: Venki, can someone on your side running any
> >> extended/customer
> >> > > >> tests you have against this to make sure that it doesn't cause
> any
> >> > > >> regressions. We've run on our side without issue but I'd like to
> >> get
> >> > as
> >> > > >> broad of coverage as possible to ensure no issues.
> >> > > >>
> >> > > >> --
> >> > > >> Jacques Nadeau
> >> > > >> CTO and Co-Founder, Dremio
> >> > > >>
> >> > > >> On Wed, Dec 2, 2015 at 11:52 AM, Venki Korukanti <
> >> > > >> venki.koruka...@gmail.com>
> >> > > >> wrote:
> >> > > >>
> >> > > >> > Sure, we can include DRILL-4124. I will merge it soon. Any idea
> >> when
> >> > > the
> >> > > >> > following JIRAs are going to be ready to commit?
> >> > > >> >
> >> > > >> > DRILL-4109 (in review)
> >> > > >> > DRILL-4125
> >> > > >> > DRILL-4145 (in review)
> >> > > >> >
> >> > > >> > Jinfeng and I discussed about including metastore caching and
> >> > decided
> >> > > to
> >> > > >> > move it to 1.5.
> >> > > >> >
> >> > > >> > Lets make 5pm as the cutoff time for 1.4 branch.
> >> > > >> >
> >> > > >> > Thanks
> >> > > >> > Venki
> >> > > >> >
> >> > > >> > On Wed, Dec 2, 2015 at 11:42 AM, Julien Le Dem <
> >> jul...@dremio.com>
> >> > > >> wrote:
> >> > > >> >
> >> > > >> > > DRILL-4124 is ready to merge:
> >> > > >> https://github.com/apache/drill/pull/281
> >> > > >> > > It is a small change.
> >> > > >> > >
> >> > > >> > > On Wed, Dec 2, 2015 at 7:58 AM, Venki Korukanti <
> >> > > >> > venki.koruka...@gmail.com
> >> > > >> > > >
> >> > > >> > > wrote:
> >> > > >> > >
> >> > > >> > > > I posted a comment on the commit DRILL-4126 (for some
> reason
> >> no
> >> > > >> emails
> >> > > >> > > are
> >> > > >> > > > sent to dev list, may be because of the multiple commits in
> >> the
> >> > > pull
> >> > > >> > > > request).
> >> > > >> > > >
> >> > > >> > > > On Wed, Dec 2, 2015 at 7:53 AM, Jinfeng Ni <
> >> > jinfengn...@gmail.com
> >> > > >
> >> > > >> > > wrote:
> >> > > >> > > >
> >> > > >> > > > > If the window is still open, I would like to have
> >> DRILL-4126,
> >> > > >> > > > > DRILL-4127 merged as well.
> >> > > >> > > > >
> >> > > >> > > > > Venki, could you please review the revised patch?
> >> > > >> > > > >
> >> > > >> > > > >
> >> > > >> > > > >
> >> > > >> > > > > On Wed, Dec 2, 2015 at 7:49 AM, Venki Korukanti
> >> > > >> > > > >  wrote:
> >> > > >> > > > > > Remaining JIRAs:
> >> > > >> > > > > >
> >> > > >> > > > > > DRILL-4109 (in review)
> >> > > >> > > > > > DRILL-4125
> >> > > >> > > > > > DRILL-4145 (in review)
> >> > > >> > > > > >
> >> > > >> > > > > >
> >> > > >> > > > > > On Tue, Dec 1, 2015 at 10:45 PM, Venki Korukanti <
> >> > > >> > > > > venki.koruka...@gmail.com>
> >> > > >> > > > > > wrote:
> >> > > >> > > > > >
> >> > > >> > > > > >> Sure, will merge DRILL-4111.
> >> > > >> > > > > >>
> >> > > >> > > > > >> On Tue, Dec 1, 2015 at 10:43 PM, Sudheesh Katkam <
> >> > > >> > > > skat...@maprtech.com>
> >> > > >> > > > > >> wrote:
> >> > > >> > > > > >>
> >> > > >> > > > > >>> Venki, can you please commit DRILL-4111?
> >> > > >> > > > 

Re: Can we pass the #skipped records with RecordBatch?

[Jacques Nadeau]

Definitely agree that we shouldn't boil the ocean.  That said, I don't
think we should make RecordBatch interface changes without deliberate
design. Same for RPC protocol changes. Part of my internal struggle with
the warning patch is exactly this lack of broader design. I think this is
especially true given the drive to supports backwards compatibility.

I don't think we're talking about a massive undertaking. I'll try to write
up some thoughts later this week to get the ball rolling. Sound good?

--
Jacques Nadeau
CTO and Co-Founder, Dremio
+1 on having a framework.
OTOH, as with the warnings implementation, we might want to go ahead with a
simpler implementation while we get a more generic framework design in
place.

Jacques, do you have any preliminary thoughts on the framework?

On Tue, Dec 1, 2015 at 2:08 PM, Julian Hyde  wrote:

> +1 for a sideband mechanism.
>
> Sideband can also allow correlated restart of sub-queries.
>
> In sideband use cases you described, the messages ran in the opposite
> direction to the data. Would the sideband also run in the same direction
as
> the data? If so it could carry warnings, rejected rows, progress
> indications, and (for online aggregation[1]) notifications that a better
> approximate query result is available.
>
> Julian
>
> [1] https://en.wikipedia.org/wiki/Online_aggregation
>
>
>
> > On Dec 1, 2015, at 1:51 PM, Jacques Nadeau  wrote:
> >
> > This seems like a form of sideband communication. I think we should have
> a
> > framework for this type of thing in general rather than a one-off for
> this
> > particular need. Other forms of sideband might be small table
bloomfilter
> > generation and pushdown into hbase, separate file
assignment/partitioning
> > providers balancing/generating scanner workloads, statistics generation
> for
> > adaptive execution, etc.
> >
> > --
> > Jacques Nadeau
> > CTO and Co-Founder, Dremio
> >
> > On Tue, Dec 1, 2015 at 11:35 AM, Hsuan Yi Chu 
> wrote:
> >
> >> I am trying to deal with the following scenario:
> >>
> >> A bunch of minor fragments are doing things in parallel. Each of them
> could
> >> skip some records. Since the downstream minor fragment needs to know
the
> >> sum of skipped-record-counts (in order to just display or see if the
> number
> >> exceeds the threshold) in the upstreams, each upstream minor fragment
> needs
> >> to pass this scalar with RecordBatch.
> >>
> >> Since this seems impacting the protocol of RecordBatch, I am looking
for
> >> some advice here.
> >>
> >> Thanks.
> >>
>
>

Re: Create 1.4 Release branch soon?

[Venki Korukanti]

1.4.0 branch is cut and available here:
https://github.com/vkorukanti/drill/tree/1.4.0.

Should I move the master to 1.5.0-SNAPSHOT now or wait until an RC is
passed?

Thanks
Venki

On Wed, Dec 2, 2015 at 4:25 PM, Venki Korukanti 
wrote:

> For DRILL-4109 and DRILL-4125, Vicky is not available today to verify. If
> the changes are reviewed lets merge them today. Once the branch is cut
> today, MapR will do the release sanity for next couple of days before RC0
> voting goes out. If any issues are found, we still have time to fix before
> the RC0 voting.
>
> Thanks
> Venki
>
> On Wed, Dec 2, 2015 at 4:02 PM, Jacques Nadeau  wrote:
>
>> Sounds good.
>>
>> --
>> Jacques Nadeau
>> CTO and Co-Founder, Dremio
>>
>> On Wed, Dec 2, 2015 at 2:47 PM, Steven Phillips 
>> wrote:
>>
>> > Okay, I'm going to go ahead and merge DRILL-4145
>> >
>> > On Wed, Dec 2, 2015 at 2:45 PM, Venki Korukanti <
>> venki.koruka...@gmail.com
>> > >
>> > wrote:
>> >
>> > > For DRILL-4145: ran the regression suite which includes customer and
>> > > extended tests. No regressions found.
>> > >
>> > > On Wed, Dec 2, 2015 at 1:41 PM, Venki Korukanti <
>> > venki.koruka...@gmail.com
>> > > >
>> > > wrote:
>> > >
>> > > > Sure, I will trigger a regression run with DRILL-4145.
>> > > >
>> > > > Thanks
>> > > > Venki
>> > > >
>> > > > On Wed, Dec 2, 2015 at 1:36 PM, Jacques Nadeau 
>> > > wrote:
>> > > >
>> > > >> I believe 4109 is ready but if I understand correctly, it can't go
>> in
>> > > >> without 4125. Amit said he needs another hour for that.
>> > > >>
>> > > >> On 4145: Venki, can someone on your side running any
>> extended/customer
>> > > >> tests you have against this to make sure that it doesn't cause any
>> > > >> regressions. We've run on our side without issue but I'd like to
>> get
>> > as
>> > > >> broad of coverage as possible to ensure no issues.
>> > > >>
>> > > >> --
>> > > >> Jacques Nadeau
>> > > >> CTO and Co-Founder, Dremio
>> > > >>
>> > > >> On Wed, Dec 2, 2015 at 11:52 AM, Venki Korukanti <
>> > > >> venki.koruka...@gmail.com>
>> > > >> wrote:
>> > > >>
>> > > >> > Sure, we can include DRILL-4124. I will merge it soon. Any idea
>> when
>> > > the
>> > > >> > following JIRAs are going to be ready to commit?
>> > > >> >
>> > > >> > DRILL-4109 (in review)
>> > > >> > DRILL-4125
>> > > >> > DRILL-4145 (in review)
>> > > >> >
>> > > >> > Jinfeng and I discussed about including metastore caching and
>> > decided
>> > > to
>> > > >> > move it to 1.5.
>> > > >> >
>> > > >> > Lets make 5pm as the cutoff time for 1.4 branch.
>> > > >> >
>> > > >> > Thanks
>> > > >> > Venki
>> > > >> >
>> > > >> > On Wed, Dec 2, 2015 at 11:42 AM, Julien Le Dem <
>> jul...@dremio.com>
>> > > >> wrote:
>> > > >> >
>> > > >> > > DRILL-4124 is ready to merge:
>> > > >> https://github.com/apache/drill/pull/281
>> > > >> > > It is a small change.
>> > > >> > >
>> > > >> > > On Wed, Dec 2, 2015 at 7:58 AM, Venki Korukanti <
>> > > >> > venki.koruka...@gmail.com
>> > > >> > > >
>> > > >> > > wrote:
>> > > >> > >
>> > > >> > > > I posted a comment on the commit DRILL-4126 (for some reason
>> no
>> > > >> emails
>> > > >> > > are
>> > > >> > > > sent to dev list, may be because of the multiple commits in
>> the
>> > > pull
>> > > >> > > > request).
>> > > >> > > >
>> > > >> > > > On Wed, Dec 2, 2015 at 7:53 AM, Jinfeng Ni <
>> > jinfengn...@gmail.com
>> > > >
>> > > >> > > wrote:
>> > > >> > > >
>> > > >> > > > > If the window is still open, I would like to have
>> DRILL-4126,
>> > > >> > > > > DRILL-4127 merged as well.
>> > > >> > > > >
>> > > >> > > > > Venki, could you please review the revised patch?
>> > > >> > > > >
>> > > >> > > > >
>> > > >> > > > >
>> > > >> > > > > On Wed, Dec 2, 2015 at 7:49 AM, Venki Korukanti
>> > > >> > > > >  wrote:
>> > > >> > > > > > Remaining JIRAs:
>> > > >> > > > > >
>> > > >> > > > > > DRILL-4109 (in review)
>> > > >> > > > > > DRILL-4125
>> > > >> > > > > > DRILL-4145 (in review)
>> > > >> > > > > >
>> > > >> > > > > >
>> > > >> > > > > > On Tue, Dec 1, 2015 at 10:45 PM, Venki Korukanti <
>> > > >> > > > > venki.koruka...@gmail.com>
>> > > >> > > > > > wrote:
>> > > >> > > > > >
>> > > >> > > > > >> Sure, will merge DRILL-4111.
>> > > >> > > > > >>
>> > > >> > > > > >> On Tue, Dec 1, 2015 at 10:43 PM, Sudheesh Katkam <
>> > > >> > > > skat...@maprtech.com>
>> > > >> > > > > >> wrote:
>> > > >> > > > > >>
>> > > >> > > > > >>> Venki, can you please commit DRILL-4111?
>> > > >> > > > > >>>
>> > > >> > > > > >>> Thank you,
>> > > >> > > > > >>> Sudheesh
>> > > >> > > > > >>>
>> > > >> > > > > >>> > On Dec 1, 2015, at 6:22 PM, Jacques Nadeau <
>> > > >> jacq...@dremio.com
>> > > >> > >
>> > > >> > > > > wrote:
>> > > >> > > > > >>> >
>> > > >> > > > > >>> > It seems like we should also try to include:
>> > > >> > > > > >>> >
>> > > >> > > > > >>> > DRILL-4109
>> > > >> > > > > >>> > DRILL-4125
>> > > >> > > > 

Re: Create 1.4 Release branch soon?

[Jinfeng Ni]

I run mvn full build on linux box against the latest master branch.
There was one unit test failure. However, on mac, it's successful. Has
anyone experienced the same?

Failed tests:
  TestCsvHeader.testCsvHeaderMismatch:151->validateResults:196 Result mismatch.
Expected:
Year|Model|Category
1999|Venture "Extended Edition"|
1999|Venture "Extended Edition, Very Large"|
Year|Model|Category
1999||Venture "Extended Edition"
1999||Venture "Extended Edition, Very Large"

Received:
Year|Model|Category
1999||Venture "Extended Edition"
1999||Venture "Extended Edition, Very Large"
Year|Model|Category
1999|Venture "Extended Edition"|
1999|Venture "Extended Edition, Very Large"|
 expected:<...Model|Category
1999|[Venture "Extended Edition"|
1999|Venture "Extended Edition, Very Large"|
Year|Model|Category
1999||Venture "Extended Edition"
1999||Venture "Extended Edition, Very Large"]
> but was:<...Model|Category
1999|[|Venture "Extended Edition"
1999||Venture "Extended Edition, Very Large"
Year|Model|Category
1999|Venture "Extended Edition"|
1999|Venture "Extended Edition, Very Large"|]
>

Tests run: 1505, Failures: 1, Errors: 0, Skipped: 121

git log
commit 3ae3bf5e127b4384c7b91d797d36ea4d51a058ae


On Wed, Dec 2, 2015 at 7:21 PM, Jacques Nadeau  wrote:
> I think we should roll forward to 1.5-S..
> On Dec 2, 2015 6:44 PM, "Venki Korukanti"  wrote:
>
>> 1.4.0 branch is cut and available here:
>> https://github.com/vkorukanti/drill/tree/1.4.0.
>>
>> Should I move the master to 1.5.0-SNAPSHOT now or wait until an RC is
>> passed?
>>
>> Thanks
>> Venki
>>
>> On Wed, Dec 2, 2015 at 4:25 PM, Venki Korukanti > >
>> wrote:
>>
>> > For DRILL-4109 and DRILL-4125, Vicky is not available today to verify. If
>> > the changes are reviewed lets merge them today. Once the branch is cut
>> > today, MapR will do the release sanity for next couple of days before RC0
>> > voting goes out. If any issues are found, we still have time to fix
>> before
>> > the RC0 voting.
>> >
>> > Thanks
>> > Venki
>> >
>> > On Wed, Dec 2, 2015 at 4:02 PM, Jacques Nadeau 
>> wrote:
>> >
>> >> Sounds good.
>> >>
>> >> --
>> >> Jacques Nadeau
>> >> CTO and Co-Founder, Dremio
>> >>
>> >> On Wed, Dec 2, 2015 at 2:47 PM, Steven Phillips 
>> >> wrote:
>> >>
>> >> > Okay, I'm going to go ahead and merge DRILL-4145
>> >> >
>> >> > On Wed, Dec 2, 2015 at 2:45 PM, Venki Korukanti <
>> >> venki.koruka...@gmail.com
>> >> > >
>> >> > wrote:
>> >> >
>> >> > > For DRILL-4145: ran the regression suite which includes customer and
>> >> > > extended tests. No regressions found.
>> >> > >
>> >> > > On Wed, Dec 2, 2015 at 1:41 PM, Venki Korukanti <
>> >> > venki.koruka...@gmail.com
>> >> > > >
>> >> > > wrote:
>> >> > >
>> >> > > > Sure, I will trigger a regression run with DRILL-4145.
>> >> > > >
>> >> > > > Thanks
>> >> > > > Venki
>> >> > > >
>> >> > > > On Wed, Dec 2, 2015 at 1:36 PM, Jacques Nadeau <
>> jacq...@dremio.com>
>> >> > > wrote:
>> >> > > >
>> >> > > >> I believe 4109 is ready but if I understand correctly, it can't
>> go
>> >> in
>> >> > > >> without 4125. Amit said he needs another hour for that.
>> >> > > >>
>> >> > > >> On 4145: Venki, can someone on your side running any
>> >> extended/customer
>> >> > > >> tests you have against this to make sure that it doesn't cause
>> any
>> >> > > >> regressions. We've run on our side without issue but I'd like to
>> >> get
>> >> > as
>> >> > > >> broad of coverage as possible to ensure no issues.
>> >> > > >>
>> >> > > >> --
>> >> > > >> Jacques Nadeau
>> >> > > >> CTO and Co-Founder, Dremio
>> >> > > >>
>> >> > > >> On Wed, Dec 2, 2015 at 11:52 AM, Venki Korukanti <
>> >> > > >> venki.koruka...@gmail.com>
>> >> > > >> wrote:
>> >> > > >>
>> >> > > >> > Sure, we can include DRILL-4124. I will merge it soon. Any idea
>> >> when
>> >> > > the
>> >> > > >> > following JIRAs are going to be ready to commit?
>> >> > > >> >
>> >> > > >> > DRILL-4109 (in review)
>> >> > > >> > DRILL-4125
>> >> > > >> > DRILL-4145 (in review)
>> >> > > >> >
>> >> > > >> > Jinfeng and I discussed about including metastore caching and
>> >> > decided
>> >> > > to
>> >> > > >> > move it to 1.5.
>> >> > > >> >
>> >> > > >> > Lets make 5pm as the cutoff time for 1.4 branch.
>> >> > > >> >
>> >> > > >> > Thanks
>> >> > > >> > Venki
>> >> > > >> >
>> >> > > >> > On Wed, Dec 2, 2015 at 11:42 AM, Julien Le Dem <
>> >> jul...@dremio.com>
>> >> > > >> wrote:
>> >> > > >> >
>> >> > > >> > > DRILL-4124 is ready to merge:
>> >> > > >> https://github.com/apache/drill/pull/281
>> >> > > >> > > It is a small change.
>> >> > > >> > >
>> >> > > >> > > On Wed, Dec 2, 2015 at 7:58 AM, Venki Korukanti <
>> >> > > >> > venki.koruka...@gmail.com
>> >> > > >> > > >
>> >> > > >> > > wrote:
>> >> > > >> > >
>> >> > > >> > > > I posted a comment on the commit DRILL-4126 (for some
>> reason
>> >> no
>> >> > > >> emails
>> >> > > >> > > are
>> >> > > >> > > 

Re: Naming the new ValueVector Initiative

[Wes McKinney]

Shall we call the voting closed? Any last stragglers?

On Tue, Dec 1, 2015 at 5:39 PM, Ted Dunning  wrote:
>
> Apache can handle this if we set the groundwork in place.
>
> Also, Twitter's lawyers work for Twitter, not for Apache. As such, their
> opinions can't be taken by Apache as legal advice.  There are issues of
> privilege, conflict of interest and so on.
>
>
>
> On Wed, Dec 2, 2015 at 7:51 AM, Alex Levenson 
> wrote:
>>
>> I can ask about whether Twitter's lawyers can help out -- is that
>> something we need? Or is that something apache helps out with in the next
>> step?
>>
>> On Mon, Nov 30, 2015 at 9:32 PM, Julian Hyde  wrote:
>>>
>>> +1 to have a vote tomorrow.
>>>
>>> Assuming that Vector is out of play, I just did a quick search for the
>>> top 4 remaining, (“arrow”, “honeycomb”, “herringbone”, “joist"), at
>>> sourceforge, open hub, trademarkia, and on google. There are no trademarks
>>> for these in similar subject areas. There is a moderately active project
>>> called “joist” [1].
>>>
>>> I will point out that “Apache Arrow” has native-american connotations
>>> that we may or may not want to live with (just ask the Washington Redskins
>>> how they feel about their name).
>>>
>>> If someone would like to vet other names, use the links on
>>> https://issues.apache.org/jira/browse/PODLINGNAMESEARCH-90, and fill out
>>> column C in the spreadsheet.
>>>
>>> Julian
>>>
>>> [1] https://github.com/stephenh/joist
>>>
>>>
>>> On Nov 30, 2015, at 7:01 PM, Jacques Nadeau  wrote:
>>>
>>> +1
>>>
>>> --
>>> Jacques Nadeau
>>> CTO and Co-Founder, Dremio
>>>
>>> On Mon, Nov 30, 2015 at 6:34 PM, Wes McKinney  wrote:
>>>
>>> Should we have a last call for votes, closing EOD tomorrow (Tuesday)? I
>>> missed this for a few days last week with holiday travel.
>>>
>>> On Thu, Nov 26, 2015 at 3:04 PM, Julian Hyde 
>>> wrote:
>>>
>>> Consulting a lawyer is part of the Apache branding process but the first
>>> stage is to gather a list of potential conflicts -
>>> https://issues.apache.org/jira/browse/PODLINGNAMESEARCH-90 is an example.
>>>
>>> The other part, frankly, is to pick your battles.
>>>
>>> A year or so ago Actian re-branded Vectorwise as Vector.
>>>
>>> http://www.zdnet.com/article/actian-consolidates-its-analytics-portfolio/.
>>> Given that it is an analytic database in the Hadoop space I think that is
>>> as close to a “direct hit” as it gets. I don’t think we need a lawyer to
>>> tell us that. Certainly it makes sense to look for conflicts for the
>>> other
>>> alternatives before consulting lawyers.
>>>
>>> Julian
>>>
>>>
>>>
>>>
>>> On Nov 25, 2015, at 9:42 PM, Marcel Kornacker 
>>> wrote:
>>>
>>>
>>>
>>> On Tue, Nov 24, 2015 at 3:25 PM, Jacques Nadeau 
>>> wrote:
>>>
>>> Ok guys,
>>>
>>> I don't think anyone is doing a thorough analysis of viaability. I did a
>>> quick glance and the top one (Vector) seems like it would have an issue
>>> with conflict of an Actian product. The may be fine. Let's do a second
>>> phase vote.
>>>
>>>
>>> I'm assuming you mean Vectorwise?
>>>
>>> Before we do anything else, could we have a lawyer look into this? Last
>>> time around that I remember (Parquet), Twitter's lawyers did a good job
>>> of
>>> weeding out the potential trademark violations.
>>>
>>> Alex, could Twitter get involved this time around as well?
>>>
>>>
>>>
>>> Pick your top 3 (1,2,3 with 3 being top preference)
>>>
>>> Let's get this done by Friday and then we can do a podling name search
>>> starting with the top one.
>>>
>>> Link again:
>>>
>>>
>>>
>>> https://docs.google.com/spreadsheets/d/1q6UqluW6SLuMKRwW2TBGBzHfYLlXYm37eKJlIxWQGQM/edit#gid=304381532=A1
>>>
>>> thanks
>>>
>>>
>>> --
>>> Jacques Nadeau
>>> CTO and Co-Founder, Dremio
>>>
>>> On Fri, Nov 20, 2015 at 9:24 AM, Jacques Nadeau 
>>> wrote:
>>>
>>> Ok, it looks like we have a candidate list (we actually got 11 since
>>> there was a three-way tie for ninth place):
>>>
>>> VectorArrowhoneycombHerringbonejoistV2Pietcolbufbatonimpulsevictor
>>> Next we need to do trademark searches on each of these to see whether
>>> we're likely to have success. I've moved candidates to a second tab:
>>>
>>>
>>>
>>> https://docs.google.com/spreadsheets/d/1q6UqluW6SLuMKRwW2TBGBzHfYLlXYm37eKJlIxWQGQM/edit#gid=304381532
>>>
>>> Anybody want to give a hand in analyzing potential conflicts?
>>>
>>>
>>>
>>> --
>>> Jacques Nadeau
>>> CTO and Co-Founder, Dremio
>>>
>>> On Mon, Nov 16, 2015 at 12:10 PM, Jacques Nadeau 
>>> wrote:
>>>
>>> Everybody should pick their ten favorites using the numbers 1 to 10.
>>>
>>> 10 is most preferred
>>>
>>>
>>>
>>> --
>>> Jacques Nadeau
>>> CTO and Co-Founder, Dremio
>>>
>>> On Mon, Nov 16, 2015 at 10:17 AM, Ted Dunning 
>>> wrote:
>>>
>>>
>>> Single vote for most preferred?
>>>
>>> Single 

[GitHub] drill pull request: DRILL-4124: Make all uses of AutoCloseables us...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/drill/pull/281


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request: DRILL-4134: Allocator updates

[adeneche]

Github user adeneche commented on a diff in the pull request:

https://github.com/apache/drill/pull/283#discussion_r46493263
  
--- Diff: exec/memory/base/src/main/java/io/netty/buffer/DrillBuf.java ---
@@ -230,20 +249,31 @@ public synchronized boolean release() {
*/
   @Override
   public synchronized boolean release(int decrement) {
+if (isEmpty) {
+  return false;
+}
 
-if(rootBuffer){
-  final long newRefCnt = this.rootRefCnt.addAndGet(-decrement);
-  Preconditions.checkArgument(newRefCnt > -1, "Buffer has negative 
reference count.");
-  if (newRefCnt == 0) {
-b.release(decrement);
-acct.release(this, length);
-return true;
-  }else{
-return false;
-  }
-}else{
-  return b.release(decrement);
+if (decrement < 1) {
+  throw new IllegalStateException(String.format("release(%d) argument 
is not positive. Buffer Info: %s",
+  decrement, toVerboseString()));
 }
+
+if (BaseAllocator.DEBUG) {
+  historicalLog.recordEvent("release(%d)", decrement);
--- End diff --

I think it would be helpful to also record the new `refCnt`


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Resolved] (DRILL-4124) Make all uses of AutoCloseables use addSuppressed exceptions to avoid noise in logs

[Venki Korukanti (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-4124?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Venki Korukanti resolved DRILL-4124.

   Resolution: Fixed
Fix Version/s: 1.4.0

> Make all uses of AutoCloseables use addSuppressed exceptions to avoid noise 
> in logs
> ---
>
> Key: DRILL-4124
> URL: https://issues.apache.org/jira/browse/DRILL-4124
> Project: Apache Drill
>  Issue Type: Improvement
>Reporter: Julien Le Dem
>Assignee: Julien Le Dem
> Fix For: 1.4.0
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Create 1.4 Release branch soon?

[Jacques Nadeau]

Sounds good.

--
Jacques Nadeau
CTO and Co-Founder, Dremio

On Wed, Dec 2, 2015 at 2:47 PM, Steven Phillips  wrote:

> Okay, I'm going to go ahead and merge DRILL-4145
>
> On Wed, Dec 2, 2015 at 2:45 PM, Venki Korukanti  >
> wrote:
>
> > For DRILL-4145: ran the regression suite which includes customer and
> > extended tests. No regressions found.
> >
> > On Wed, Dec 2, 2015 at 1:41 PM, Venki Korukanti <
> venki.koruka...@gmail.com
> > >
> > wrote:
> >
> > > Sure, I will trigger a regression run with DRILL-4145.
> > >
> > > Thanks
> > > Venki
> > >
> > > On Wed, Dec 2, 2015 at 1:36 PM, Jacques Nadeau 
> > wrote:
> > >
> > >> I believe 4109 is ready but if I understand correctly, it can't go in
> > >> without 4125. Amit said he needs another hour for that.
> > >>
> > >> On 4145: Venki, can someone on your side running any extended/customer
> > >> tests you have against this to make sure that it doesn't cause any
> > >> regressions. We've run on our side without issue but I'd like to get
> as
> > >> broad of coverage as possible to ensure no issues.
> > >>
> > >> --
> > >> Jacques Nadeau
> > >> CTO and Co-Founder, Dremio
> > >>
> > >> On Wed, Dec 2, 2015 at 11:52 AM, Venki Korukanti <
> > >> venki.koruka...@gmail.com>
> > >> wrote:
> > >>
> > >> > Sure, we can include DRILL-4124. I will merge it soon. Any idea when
> > the
> > >> > following JIRAs are going to be ready to commit?
> > >> >
> > >> > DRILL-4109 (in review)
> > >> > DRILL-4125
> > >> > DRILL-4145 (in review)
> > >> >
> > >> > Jinfeng and I discussed about including metastore caching and
> decided
> > to
> > >> > move it to 1.5.
> > >> >
> > >> > Lets make 5pm as the cutoff time for 1.4 branch.
> > >> >
> > >> > Thanks
> > >> > Venki
> > >> >
> > >> > On Wed, Dec 2, 2015 at 11:42 AM, Julien Le Dem 
> > >> wrote:
> > >> >
> > >> > > DRILL-4124 is ready to merge:
> > >> https://github.com/apache/drill/pull/281
> > >> > > It is a small change.
> > >> > >
> > >> > > On Wed, Dec 2, 2015 at 7:58 AM, Venki Korukanti <
> > >> > venki.koruka...@gmail.com
> > >> > > >
> > >> > > wrote:
> > >> > >
> > >> > > > I posted a comment on the commit DRILL-4126 (for some reason no
> > >> emails
> > >> > > are
> > >> > > > sent to dev list, may be because of the multiple commits in the
> > pull
> > >> > > > request).
> > >> > > >
> > >> > > > On Wed, Dec 2, 2015 at 7:53 AM, Jinfeng Ni <
> jinfengn...@gmail.com
> > >
> > >> > > wrote:
> > >> > > >
> > >> > > > > If the window is still open, I would like to have DRILL-4126,
> > >> > > > > DRILL-4127 merged as well.
> > >> > > > >
> > >> > > > > Venki, could you please review the revised patch?
> > >> > > > >
> > >> > > > >
> > >> > > > >
> > >> > > > > On Wed, Dec 2, 2015 at 7:49 AM, Venki Korukanti
> > >> > > > >  wrote:
> > >> > > > > > Remaining JIRAs:
> > >> > > > > >
> > >> > > > > > DRILL-4109 (in review)
> > >> > > > > > DRILL-4125
> > >> > > > > > DRILL-4145 (in review)
> > >> > > > > >
> > >> > > > > >
> > >> > > > > > On Tue, Dec 1, 2015 at 10:45 PM, Venki Korukanti <
> > >> > > > > venki.koruka...@gmail.com>
> > >> > > > > > wrote:
> > >> > > > > >
> > >> > > > > >> Sure, will merge DRILL-4111.
> > >> > > > > >>
> > >> > > > > >> On Tue, Dec 1, 2015 at 10:43 PM, Sudheesh Katkam <
> > >> > > > skat...@maprtech.com>
> > >> > > > > >> wrote:
> > >> > > > > >>
> > >> > > > > >>> Venki, can you please commit DRILL-4111?
> > >> > > > > >>>
> > >> > > > > >>> Thank you,
> > >> > > > > >>> Sudheesh
> > >> > > > > >>>
> > >> > > > > >>> > On Dec 1, 2015, at 6:22 PM, Jacques Nadeau <
> > >> jacq...@dremio.com
> > >> > >
> > >> > > > > wrote:
> > >> > > > > >>> >
> > >> > > > > >>> > It seems like we should also try to include:
> > >> > > > > >>> >
> > >> > > > > >>> > DRILL-4109
> > >> > > > > >>> > DRILL-4125
> > >> > > > > >>> > DRILL-4111
> > >> > > > > >>> >
> > >> > > > > >>> > 4111 is ready to merge if someone wants to merge it. It
> > >> looks
> > >> > > like
> > >> > > > > >>> Sudheesh
> > >> > > > > >>> > just needs to commit.
> > >> > > > > >>> >
> > >> > > > > >>> > For 4125/4109, it seems like we should get Vicki's
> > feedback
> > >> if
> > >> > > > those
> > >> > > > > >>> > changes are good for merge or if we should punt.
> > >> > > > > >>> >
> > >> > > > > >>> >
> > >> > > > > >>> >
> > >> > > > > >>> >
> > >> > > > > >>> > --
> > >> > > > > >>> > Jacques Nadeau
> > >> > > > > >>> > CTO and Co-Founder, Dremio
> > >> > > > > >>> >
> > >> > > > > >>> > On Tue, Dec 1, 2015 at 3:59 PM, Venki Korukanti <
> > >> > > > > >>> venki.koruka...@gmail.com>
> > >> > > > > >>> > wrote:
> > >> > > > > >>> >
> > >> > > > > >>> >> I can manage the release.
> > >> > > > > >>> >>
> > >> > > > > >>> >> Here are the pending JIRAs so far:
> > >> > > > > >>> >>
> > >> > > > > >>> >> 1) DRILL-4053
> > >> > > > > >>> >>
> > >> > > > > >>> >> Let me know if you have any pending patches and want to
> > get
> > >> > 

[GitHub] drill pull request: DRILL-4108: Query on csv file w/ header fails ...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/drill/pull/269


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Can Drill distribution be deployed in an existing jetty/tomcat server

[Sudip Mukherjee]

Hi,

Could you please advise if there is a way I can use one of my existing 
jetty/tomcat server and deploy drill as a service?

Thanks,
Sudip



***Legal Disclaimer***
"This communication may contain confidential and privileged material for the
sole use of the intended recipient. Any unauthorized review, use or distribution
by others is strictly prohibited. If you have received the message by mistake,
please advise the sender by reply email and delete the message. Thank you."
**

[jira] [Created] (DRILL-4154) Metadata Caching : Upgrading cache to v2 from v1 corrupts the cache in some scenarios

[Rahul Challapalli (JIRA)]

Rahul Challapalli created DRILL-4154:


 Summary: Metadata Caching : Upgrading cache to v2 from v1 corrupts 
the cache in some scenarios
 Key: DRILL-4154
 URL: https://issues.apache.org/jira/browse/DRILL-4154
 Project: Apache Drill
  Issue Type: Bug
Reporter: Rahul Challapalli
Priority: Critical


git.commit.id.abbrev=46c47a2

I copied the data along with the cache file onto maprfs. Now I ran the upgrade 
tool (https://github.com/parthchandra/drill-upgrade). Now I ran the 
metadata_caching suite from the functional tests (concurrency 10) without the 
datagen phase. I see 3 test failures and when I looked at the cache file it 
seems to be containing wrong information for the varchar column. 

Sample from the cache :
{code}
  {
"name" : [ "varchar_col" ]
  }, {
"name" : [ "float_col" ],
"mxValue" : 68797.22,
"nulls" : 0
  }
{code}

Now I followed the same steps and instead of running the suites I executed the 
"REFRESH TABLE METADATA" command or any query on that folder,  the cache file 
seems to be created properly

I attached the data and cache files required. Let me know if you need anything



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (DRILL-4155) Query with two-way join and flatten fails with "IllegalArgumentException: maxCapacity"

[Abhishek Girish (JIRA)]

Abhishek Girish created DRILL-4155:
--

 Summary: Query with two-way join and flatten fails with 
"IllegalArgumentException: maxCapacity"
 Key: DRILL-4155
 URL: https://issues.apache.org/jira/browse/DRILL-4155
 Project: Apache Drill
  Issue Type: Bug
  Components: Execution - Flow, Storage - JSON
Reporter: Abhishek Girish
 Attachments: drillbit.log.txt

The following query on the Yelp Academic dataset fails to execute:

{code}
select u.name, b.name , flatten(b.categories) from 
maprfs.yelp_tutorial.`yelp_academic_dataset_user.json` u, 
maprfs.yelp_tutorial.`yelp_academic_dataset_business.json` b where 
u.average_stars = b.stars limit 10

Query Failed: An Error Occurred
org.apache.drill.common.exceptions.UserRemoteException: SYSTEM ERROR: 
IllegalArgumentException: maxCapacity: -104845 (expected: >= 0) Fragment 1:0 
[Error Id: b0d99a6c-3434-49ce-8aa6-181993cdd853 on atsqa6c62.qa.lab:31010]
{code}

Tried on multiple setups in distributed mode - consistently fails. 

Dataset can be accessed from : 
https://s3.amazonaws.com/apache-drill/files/yelp.tgz (uncompressed tar archive)
Log attached. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)