Re: Naming the new ValueVector Initiative

[Jake Luciani]

The term redskin is derogatory which is clearly not the same as arrow. I
think if our logo is not a native American arrow there is no real issue.
On Dec 1, 2015 12:32 AM, "Julian Hyde"  wrote:

> +1 to have a vote tomorrow.
>
> Assuming that Vector is out of play, I just did a quick search for the top
> 4 remaining, (“arrow”, “honeycomb”, “herringbone”, “joist"), at
> sourceforge, open hub, trademarkia, and on google. There are no trademarks
> for these in similar subject areas. There is a moderately active project
> called “joist” [1].
>
> I will point out that “Apache Arrow” has native-american connotations that
> we may or may not want to live with (just ask the Washington Redskins how
> they feel about their name).
>
> If someone would like to vet other names, use the links on
> https://issues.apache.org/jira/browse/PODLINGNAMESEARCH-90, and fill out
> column C in the spreadsheet.
>
> Julian
>
> [1] https://github.com/stephenh/joist
>
>
> On Nov 30, 2015, at 7:01 PM, Jacques Nadeau  wrote:
>
> +1
>
> --
> Jacques Nadeau
> CTO and Co-Founder, Dremio
>
> On Mon, Nov 30, 2015 at 6:34 PM, Wes McKinney  wrote:
>
> Should we have a last call for votes, closing EOD tomorrow (Tuesday)? I
> missed this for a few days last week with holiday travel.
>
> On Thu, Nov 26, 2015 at 3:04 PM, Julian Hyde 
> wrote:
>
> Consulting a lawyer is part of the Apache branding process but the first
> stage is to gather a list of potential conflicts -
> https://issues.apache.org/jira/browse/PODLINGNAMESEARCH-90 is an example.
>
> The other part, frankly, is to pick your battles.
>
> A year or so ago Actian re-branded Vectorwise as Vector.
> http://www.zdnet.com/article/actian-consolidates-its-analytics-portfolio/.
> Given that it is an analytic database in the Hadoop space I think that is
> as close to a “direct hit” as it gets. I don’t think we need a lawyer to
> tell us that. Certainly it makes sense to look for conflicts for the other
> alternatives before consulting lawyers.
>
> Julian
>
>
>
>
> On Nov 25, 2015, at 9:42 PM, Marcel Kornacker 
> wrote:
>
>
>
> On Tue, Nov 24, 2015 at 3:25 PM, Jacques Nadeau 
> wrote:
>
> Ok guys,
>
> I don't think anyone is doing a thorough analysis of viaability. I did a
> quick glance and the top one (Vector) seems like it would have an issue
> with conflict of an Actian product. The may be fine. Let's do a second
> phase vote.
>
>
> I'm assuming you mean Vectorwise?
>
> Before we do anything else, could we have a lawyer look into this? Last
> time around that I remember (Parquet), Twitter's lawyers did a good job of
> weeding out the potential trademark violations.
>
> Alex, could Twitter get involved this time around as well?
>
>
>
> Pick your top 3 (1,2,3 with 3 being top preference)
>
> Let's get this done by Friday and then we can do a podling name search
> starting with the top one.
>
> Link again:
>
>
>
> https://docs.google.com/spreadsheets/d/1q6UqluW6SLuMKRwW2TBGBzHfYLlXYm37eKJlIxWQGQM/edit#gid=304381532&vpid=A1
>
> thanks
>
>
> --
> Jacques Nadeau
> CTO and Co-Founder, Dremio
>
> On Fri, Nov 20, 2015 at 9:24 AM, Jacques Nadeau 
> wrote:
>
> Ok, it looks like we have a candidate list (we actually got 11 since
> there was a three-way tie for ninth place):
>
> VectorArrowhoneycombHerringbonejoistV2Pietcolbufbatonimpulsevictor
> Next we need to do trademark searches on each of these to see whether
> we're likely to have success. I've moved candidates to a second tab:
>
>
>
> https://docs.google.com/spreadsheets/d/1q6UqluW6SLuMKRwW2TBGBzHfYLlXYm37eKJlIxWQGQM/edit#gid=304381532
>
> Anybody want to give a hand in analyzing potential conflicts?
>
>
>
> --
> Jacques Nadeau
> CTO and Co-Founder, Dremio
>
> On Mon, Nov 16, 2015 at 12:10 PM, Jacques Nadeau 
> wrote:
>
> Everybody should pick their ten favorites using the numbers 1 to 10.
>
> 10 is most preferred
>
>
>
> --
> Jacques Nadeau
> CTO and Co-Founder, Dremio
>
> On Mon, Nov 16, 2015 at 10:17 AM, Ted Dunning 
> wrote:
>
>
> Single vote for most preferred?
>
> Single transferable vote?
>
>
>
> On Tue, Nov 17, 2015 at 2:50 AM, Jacques Nadeau 
> wrote:
>
> Given that a bunch of people added names to the sheet, I'll take
> that as tacit agreement to the proposed process.
>
> Let's move to the first vote phase. I've added a column for
> everybody's votes. Let's try to wrap up the vote by 10am on Wednesday.
>
> thanks!
> Jacques
>
>
>
> --
> Jacques Nadeau
> CTO and Co-Founder, Dremio
>
> On Thu, Nov 12, 2015 at 12:03 PM, Jacques Nadeau 
> wrote:
>
>
> Hey Guys,
>
> It sounds like we need to do a little more work on the Vector
> proposal
> before the board would like to consider it. The main point of
> contention
> right now is the name of the project. We need to decide on a name
> and get
> it signed off through PODLINGNAMESEARCH.
>
> Naming is extremely subjective so I'd like to propose a process for
> selection that minimizes pain. This is an initial proposal and
>
> We do the naming in the following steps
> - 1: Collect a set of names to

[GitHub] drill pull request: DRILL-4146: Concurrent queries hang in planner...

[jinfengni]

Github user jinfengni commented on the pull request:

https://github.com/apache/drill/pull/285#issuecomment-161017412
  
+1


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request: DRILL-4134: Allocator updates

[julienledem]

Github user julienledem commented on a diff in the pull request:

https://github.com/apache/drill/pull/283#discussion_r46308724
  
--- Diff: exec/memory/base/src/main/java/io/netty/buffer/DrillBuf.java ---
@@ -252,32 +282,25 @@ public int capacity() {
   }
 
   @Override
-  public synchronized ByteBuf capacity(int newCapacity) {
-if (rootBuffer) {
-  if (newCapacity == length) {
-return this;
-  } else if (newCapacity < length) {
-b.capacity(newCapacity);
-int diff = length - b.capacity();
-acct.releasePartial(this, diff);
-this.length = length - diff;
-return this;
-  } else {
-throw new UnsupportedOperationException("Accounting byte buf 
doesn't support increasing allocations.");
-  }
-} else {
-  throw new UnsupportedOperationException("Non root bufs doen't 
support changing allocations.");
+  public synchronized DrillBuf capacity(int newCapacity) {
--- End diff --

```trim(int newCapacity)```?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request: DRILL-4134: Allocator updates

[julienledem]

Github user julienledem commented on a diff in the pull request:

https://github.com/apache/drill/pull/283#discussion_r46308802
  
--- Diff: exec/memory/base/src/main/java/io/netty/buffer/DrillBuf.java ---
@@ -287,14 +310,12 @@ public ByteOrder order() {
 
   @Override
   public ByteBuf order(ByteOrder endianness) {
-// if(endianness != ByteOrder.LITTLE_ENDIAN) throw new
-// UnsupportedOperationException("Drill buffers only support little 
endian.");
--- End diff --

should this throw?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request: DRILL-4134: Allocator updates

[julienledem]

Github user julienledem commented on a diff in the pull request:

https://github.com/apache/drill/pull/283#discussion_r46309586
  
--- Diff: exec/memory/base/src/main/java/io/netty/buffer/DrillBuf.java ---
@@ -431,12 +482,18 @@ public boolean equals(Object obj) {
   }
 
   @Override
-  public synchronized ByteBuf retain(int increment) {
-if(rootBuffer){
-  this.rootRefCnt.addAndGet(increment);
-}else{
-  b.retain(increment);
+  public ByteBuf retain(int increment) {
--- End diff --

how is this retain used compared to the other one (that takes an 
allocator)? Is this one only used when in the same allocator?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request: DRILL-4134: Allocator updates

[julienledem]

Github user julienledem commented on a diff in the pull request:

https://github.com/apache/drill/pull/283#discussion_r46309770
  
--- Diff: exec/memory/base/src/main/java/io/netty/buffer/DrillBuf.java ---
@@ -733,17 +790,98 @@ public byte getByte(int index) {
 return PlatformDependent.getByte(addr(index));
   }
 
-  public static DrillBuf getEmpty(BufferAllocator allocator, Accountor a) {
-return new DrillBuf(allocator, a);
+  @Override
+  public void close() {
+release();
   }
 
-  public boolean isRootBuffer() {
-return rootBuffer;
+  /**
+   * Returns the possible memory consumed by this DrillBuf in the worse 
case scenario. (not shared, connected to larger
+   * underlying buffer of allocated memory)
+   *
+   * @return Size in bytes.
+   */
+  public int getPossibleMemoryConsumed() {
+return ledger.getSize();
   }
 
-  @Override
-  public void close() {
-release();
+  /**
+   * Return that is Accounted for by this buffer (and its potentially 
shared siblings within the context of the
+   * associated allocator).
+   *
+   * @return Size in bytes.
+   */
+  public int getActualMemoryConsumed() {
+return ledger.getAccountedSize();
+  }
+
+  private final static int LOG_BYTES_PER_ROW = 10;
+  /**
+   * Log this buffer's byte contents in the form of a hex dump.
+   *
+   * @param logger where to log to
+   * @param start the starting byte index
+   * @param length how many bytes to log
+   */
+  public void logBytes(final Logger logger, final int start, final int 
length) {
--- End diff --

should this just be ```toString(start, length)``` leaving logging to the 
caller?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request: DRILL-4134: Allocator updates

[julienledem]

Github user julienledem commented on a diff in the pull request:

https://github.com/apache/drill/pull/283#discussion_r46310317
  
--- Diff: exec/memory/base/src/main/java/io/netty/buffer/DrillBuf.java ---
@@ -733,17 +790,98 @@ public byte getByte(int index) {
 return PlatformDependent.getByte(addr(index));
   }
 
-  public static DrillBuf getEmpty(BufferAllocator allocator, Accountor a) {
-return new DrillBuf(allocator, a);
+  @Override
+  public void close() {
+release();
   }
 
-  public boolean isRootBuffer() {
-return rootBuffer;
+  /**
+   * Returns the possible memory consumed by this DrillBuf in the worse 
case scenario. (not shared, connected to larger
+   * underlying buffer of allocated memory)
+   *
+   * @return Size in bytes.
+   */
+  public int getPossibleMemoryConsumed() {
+return ledger.getSize();
   }
 
-  @Override
-  public void close() {
-release();
+  /**
+   * Return that is Accounted for by this buffer (and its potentially 
shared siblings within the context of the
+   * associated allocator).
+   *
+   * @return Size in bytes.
+   */
+  public int getActualMemoryConsumed() {
+return ledger.getAccountedSize();
+  }
+
+  private final static int LOG_BYTES_PER_ROW = 10;
+  /**
+   * Log this buffer's byte contents in the form of a hex dump.
+   *
+   * @param logger where to log to
+   * @param start the starting byte index
+   * @param length how many bytes to log
+   */
+  public void logBytes(final Logger logger, final int start, final int 
length) {
+final int roundedStart = (start / LOG_BYTES_PER_ROW) * 
LOG_BYTES_PER_ROW;
+
+final StringBuilder sb = new StringBuilder("buffer byte dump\n");
+int index = roundedStart;
+for(int nLogged = 0; nLogged < length; nLogged += LOG_BYTES_PER_ROW) {
+  sb.append(String.format(" [%05d-%05d]", index, index + 
LOG_BYTES_PER_ROW - 1));
+  for(int i = 0; i < LOG_BYTES_PER_ROW; ++i) {
+try {
+  final byte b = getByte(index++);
+  sb.append(String.format(" 0x%02x", b));
+} catch(IndexOutOfBoundsException ioob) {
+  sb.append(" ");
+}
+  }
+  sb.append('\n');
+}
+logger.trace(sb.toString());
+  }
+
+  /**
+   * Get the integer id assigned to this DrillBuf for debugging purposes.
+   *
+   * @return integer id
+   */
+  public long getId() {
+return id;
+  }
+
+  /**
+   * Log this buffer's history.
+   *
+   * @param logger the logger to use
+   */
+  public void logHistory(final Logger logger) {
+if (historicalLog == null) {
+  logger.warn("DrillBuf[{}] historicalLog not available", id);
+} else {
+  historicalLog.logHistory(logger);
+}
+  }
+
+  public String toVerboseString() {
+if (isEmpty) {
+  return toString();
+}
+
+StringBuilder sb = new StringBuilder();
+ledger.print(sb, 0, Verbosity.LOG_WITH_STACKTRACE);
+return sb.toString();
+  }
+
+  public void print(StringBuilder sb, int indent, Verbosity verbosity) {
+BaseAllocator.indent(sb, indent).append(toString());
+
+if (BaseAllocator.DEBUG && !isEmpty && verbosity.includeHistoricalLog) 
{
+  sb.append("\n");
+  historicalLog.buildHistory(sb, indent + 1, 
verbosity.includeStackTraces);
+}
   }
--- End diff --

logBytes() logHistory() toVerboseString() and print() have various level of 
details about what they do in their name.
Possibly make print private?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request: DRILL-4124: Make all uses of AutoCloseables us...

[julienledem]

Github user julienledem commented on the pull request:

https://github.com/apache/drill/pull/281#issuecomment-161043400
  
@jaltekruse should be better now :)


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request: DRILL-4134: Allocator updates

[julienledem]

Github user julienledem commented on a diff in the pull request:

https://github.com/apache/drill/pull/283#discussion_r46310977
  
--- Diff: exec/memory/base/src/main/java/io/netty/buffer/DrillBuf.java ---
@@ -252,32 +282,25 @@ public int capacity() {
   }
 
   @Override
-  public synchronized ByteBuf capacity(int newCapacity) {
-if (rootBuffer) {
-  if (newCapacity == length) {
-return this;
-  } else if (newCapacity < length) {
-b.capacity(newCapacity);
-int diff = length - b.capacity();
-acct.releasePartial(this, diff);
-this.length = length - diff;
-return this;
-  } else {
-throw new UnsupportedOperationException("Accounting byte buf 
doesn't support increasing allocations.");
-  }
-} else {
-  throw new UnsupportedOperationException("Non root bufs doen't 
support changing allocations.");
+  public synchronized DrillBuf capacity(int newCapacity) {
--- End diff --

nevermind, it overrides a netty method.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request: DRILL-4134: Allocator updates

[julienledem]

Github user julienledem commented on a diff in the pull request:

https://github.com/apache/drill/pull/283#discussion_r46312830
  
--- Diff: 
exec/memory/base/src/main/java/io/netty/buffer/PooledByteBufAllocatorL.java ---
@@ -23,193 +23,246 @@
 import java.nio.ByteBuffer;
 import java.util.concurrent.atomic.AtomicLong;
 
+import org.apache.drill.exec.exception.OutOfMemoryException;
+
 import com.codahale.metrics.Gauge;
 import com.codahale.metrics.Histogram;
 import com.codahale.metrics.Metric;
 import com.codahale.metrics.MetricFilter;
 import com.codahale.metrics.MetricRegistry;
 
-public class PooledByteBufAllocatorL extends PooledByteBufAllocator{
-  static final org.slf4j.Logger logger = 
org.slf4j.LoggerFactory.getLogger(PooledByteBufAllocatorL.class);
-
+public class PooledByteBufAllocatorL {
   private static final org.slf4j.Logger memoryLogger = 
org.slf4j.LoggerFactory.getLogger("drill.allocator");
+
   private static final int MEMORY_LOGGER_FREQUENCY_SECONDS = 60;
 
+
   private static final String METRIC_PREFIX = "drill.allocator.";
+
   private final MetricRegistry registry;
   private final AtomicLong hugeBufferSize = new AtomicLong(0);
   private final AtomicLong hugeBufferCount = new AtomicLong(0);
   private final AtomicLong normalBufferSize = new AtomicLong(0);
   private final AtomicLong normalBufferCount = new AtomicLong(0);
 
-  private final PoolArena[] directArenas;
-  private final MemoryStatusThread statusThread;
-  private final Histogram largeBuffersHist;
-  private final Histogram normalBuffersHist;
+  public final InnerAllocator allocator;
+  public final UnsafeDirectLittleEndian empty;
 
   public PooledByteBufAllocatorL(MetricRegistry registry) {
-super(true);
 this.registry = registry;
+allocator = new InnerAllocator();
+empty = new UnsafeDirectLittleEndian(new 
DuplicatedByteBuf(Unpooled.EMPTY_BUFFER));
+  }
+
+  public UnsafeDirectLittleEndian allocate(int size) {
 try {
-  Field f = 
PooledByteBufAllocator.class.getDeclaredField("directArenas");
-  f.setAccessible(true);
-  this.directArenas = (PoolArena[]) f.get(this);
-} catch (Exception e) {
-  throw new RuntimeException("Failure while initializing allocator.  
Unable to retrieve direct arenas field.", e);
+  return allocator.directBuffer(size, size);
+} catch (OutOfMemoryError e) {
+  throw new OutOfMemoryException("Failure allocating buffer.", e);
 }
 
-if (memoryLogger.isTraceEnabled()) {
-  statusThread = new MemoryStatusThread();
-  statusThread.start();
-} else {
-  statusThread = null;
-}
-removeOldMetrics();
+  }
 
-registry.register(METRIC_PREFIX + "normal.size", new Gauge() {
-  @Override
-  public Long getValue() {
-return normalBufferSize.get();
-  }
-});
+  public int getChunkSize() {
+return allocator.chunkSize;
+  }
 
-registry.register(METRIC_PREFIX + "normal.count", new Gauge() {
-  @Override
-  public Long getValue() {
-return normalBufferCount.get();
-  }
-});
+  private class InnerAllocator extends PooledByteBufAllocator {
 
-registry.register(METRIC_PREFIX + "huge.size", new Gauge() {
-  @Override
-  public Long getValue() {
-return hugeBufferSize.get();
-  }
-});
 
-registry.register(METRIC_PREFIX + "huge.count", new Gauge() {
-  @Override
-  public Long getValue() {
-return hugeBufferCount.get();
-  }
-});
+private final PoolArena[] directArenas;
+private final MemoryStatusThread statusThread;
+private final Histogram largeBuffersHist;
+private final Histogram normalBuffersHist;
+private final int chunkSize;
 
-largeBuffersHist = registry.histogram(METRIC_PREFIX + "huge.hist");
-normalBuffersHist = registry.histogram(METRIC_PREFIX + "normal.hist");
+public InnerAllocator() {
+  super(true);
 
-  }
+  try {
+Field f = 
PooledByteBufAllocator.class.getDeclaredField("directArenas");
+f.setAccessible(true);
+this.directArenas = (PoolArena[]) f.get(this);
+  } catch (Exception e) {
+throw new RuntimeException("Failure while initializing allocator.  
Unable to retrieve direct arenas field.", e);
+  }
 
-  private synchronized void removeOldMetrics() {
-registry.removeMatching(new MetricFilter() {
-  @Override
-  public boolean matches(String name, Metric metric) {
-return name.startsWith("drill.allocator.");
+  this.chunk

Create 1.4 Release branch soon?

[Jacques Nadeau]

It is December (already!). Seems like we should create a 1.4 release branch
soon so we can get a vote going. Does someone want to volunteer as the
release manager?

thanks,
Jacques

--
Jacques Nadeau
CTO and Co-Founder, Dremio

Hangout Starting Now

[Jacques Nadeau]

https://plus.google.com/hangouts/_/dremio.com/drillhangout?authuser=0

--
Jacques Nadeau
CTO and Co-Founder, Dremio

[GitHub] drill pull request: DRILL-4134: Allocator updates

[julienledem]

Github user julienledem commented on a diff in the pull request:

https://github.com/apache/drill/pull/283#discussion_r46313215
  
--- Diff: 
exec/memory/base/src/main/java/org/apache/drill/exec/memory/Accountant.java ---
@@ -0,0 +1,272 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.memory;
+
+import java.util.concurrent.atomic.AtomicLong;
+
+import javax.annotation.concurrent.ThreadSafe;
+
+import org.apache.drill.exec.exception.OutOfMemoryException;
+
+import com.google.common.base.Preconditions;
+
+/**
+ * Provides a concurrent way to manage account for memory usage without 
locking. Used as basis for Allocators. All
+ * operations are threadsafe (except for close).
+ */
+@ThreadSafe
+class Accountant implements AutoCloseable {
+  // private static final org.slf4j.Logger logger = 
org.slf4j.LoggerFactory.getLogger(Accountant.class);
+
+  /**
+   * The parent allocator
+   */
+  protected final Accountant parent;
+
+  /**
+   * The amount of memory reserved for this allocator. Releases below this 
amount of memory will not be returned to the
+   * parent Accountant until this Accountant is closed.
+   */
+  protected final long reservation;
+
+  private final AtomicLong peakAllocation = new AtomicLong();
+
+  /**
+   * Maximum local memory that can be held. This can be externally 
updated. Changing it won't cause past memory to
+   * change but will change responses to future allocation efforts
+   */
+  private final AtomicLong allocationLimit = new AtomicLong();
+
+  /**
+   * Currently allocated amount of memory;
+   */
+  private final AtomicLong locallyHeldMemory = new AtomicLong();
+
+  public Accountant(Accountant parent, long reservation, long 
maxAllocation) {
+Preconditions.checkArgument(reservation >= 0, "The initial reservation 
size must be non-negative.");
+Preconditions.checkArgument(maxAllocation >= 0, "The maximum 
allocation limit must be non-negative.");
+Preconditions.checkArgument(reservation <= maxAllocation,
+"The initial reservation size must be <= the maximum allocation.");
+Preconditions.checkArgument(reservation == 0 || parent != null, "The 
root accountant can't reserve memory.");
+
+this.parent = parent;
+this.reservation = reservation;
+this.allocationLimit.set(maxAllocation);
+
+if (reservation != 0) {
+  // we will allocate a reservation from our parent.
+  final AllocationOutcome outcome = parent.allocateBytes(reservation);
+  if (!outcome.isOk()) {
+throw new OutOfMemoryException("Failure trying to allocate initial 
reservation for Allocator.");
--- End diff --

does outcome contain more info about what went wrong?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request: DRILL-4134: Allocator updates

[julienledem]

Github user julienledem commented on a diff in the pull request:

https://github.com/apache/drill/pull/283#discussion_r46313687
  
--- Diff: 
exec/memory/base/src/main/java/org/apache/drill/exec/memory/Accountant.java ---
@@ -0,0 +1,272 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.memory;
+
+import java.util.concurrent.atomic.AtomicLong;
+
+import javax.annotation.concurrent.ThreadSafe;
+
+import org.apache.drill.exec.exception.OutOfMemoryException;
+
+import com.google.common.base.Preconditions;
+
+/**
+ * Provides a concurrent way to manage account for memory usage without 
locking. Used as basis for Allocators. All
+ * operations are threadsafe (except for close).
+ */
+@ThreadSafe
+class Accountant implements AutoCloseable {
+  // private static final org.slf4j.Logger logger = 
org.slf4j.LoggerFactory.getLogger(Accountant.class);
+
+  /**
+   * The parent allocator
+   */
+  protected final Accountant parent;
+
+  /**
+   * The amount of memory reserved for this allocator. Releases below this 
amount of memory will not be returned to the
+   * parent Accountant until this Accountant is closed.
+   */
+  protected final long reservation;
+
+  private final AtomicLong peakAllocation = new AtomicLong();
+
+  /**
+   * Maximum local memory that can be held. This can be externally 
updated. Changing it won't cause past memory to
+   * change but will change responses to future allocation efforts
+   */
+  private final AtomicLong allocationLimit = new AtomicLong();
+
+  /**
+   * Currently allocated amount of memory;
+   */
+  private final AtomicLong locallyHeldMemory = new AtomicLong();
+
+  public Accountant(Accountant parent, long reservation, long 
maxAllocation) {
+Preconditions.checkArgument(reservation >= 0, "The initial reservation 
size must be non-negative.");
+Preconditions.checkArgument(maxAllocation >= 0, "The maximum 
allocation limit must be non-negative.");
+Preconditions.checkArgument(reservation <= maxAllocation,
+"The initial reservation size must be <= the maximum allocation.");
+Preconditions.checkArgument(reservation == 0 || parent != null, "The 
root accountant can't reserve memory.");
+
+this.parent = parent;
+this.reservation = reservation;
+this.allocationLimit.set(maxAllocation);
+
+if (reservation != 0) {
+  // we will allocate a reservation from our parent.
+  final AllocationOutcome outcome = parent.allocateBytes(reservation);
+  if (!outcome.isOk()) {
+throw new OutOfMemoryException("Failure trying to allocate initial 
reservation for Allocator.");
+  }
+}
+  }
+
+  /**
+   * Attempt to allocate the requested amount of memory. Either completely 
succeeds or completely fails. Constructs a a
+   * log of delta
+   *
+   * If it fails, no changes are made to accounting.
+   *
+   * @param size
+   *  The amount of memory to reserve in bytes.
+   * @return True if the allocation was successful, false if the 
allocation failed.
+   */
+  AllocationOutcome allocateBytes(long size) {
+final AllocationOutcome outcome = allocate(size, true, false);
+if (!outcome.isOk()) {
+  releaseBytes(size);
+}
+return outcome;
+  }
+
+  private void updatePeak() {
+final long currentMemory = locallyHeldMemory.get();
+while (true) {
+  final long previousPeak = peakAllocation.get();
+  if (currentMemory > previousPeak) {
+if (peakAllocation.compareAndSet(previousPeak, currentMemory)) {
+  // if we're able to update peak, finish.
+  return;
+} else {
+  // peak allocation changed underneath us. try again.
+  continue;
+}
+  } else {
+return;
+  }
+}
--- End diff 

[GitHub] drill pull request: DRILL-4134: Allocator updates

[julienledem]

Github user julienledem commented on a diff in the pull request:

https://github.com/apache/drill/pull/283#discussion_r46312998
  
--- Diff: 
exec/memory/base/src/main/java/io/netty/buffer/PooledByteBufAllocatorL.java ---
@@ -23,193 +23,246 @@
 import java.nio.ByteBuffer;
 import java.util.concurrent.atomic.AtomicLong;
 
+import org.apache.drill.exec.exception.OutOfMemoryException;
+
 import com.codahale.metrics.Gauge;
 import com.codahale.metrics.Histogram;
 import com.codahale.metrics.Metric;
 import com.codahale.metrics.MetricFilter;
 import com.codahale.metrics.MetricRegistry;
 
-public class PooledByteBufAllocatorL extends PooledByteBufAllocator{
-  static final org.slf4j.Logger logger = 
org.slf4j.LoggerFactory.getLogger(PooledByteBufAllocatorL.class);
-
+public class PooledByteBufAllocatorL {
   private static final org.slf4j.Logger memoryLogger = 
org.slf4j.LoggerFactory.getLogger("drill.allocator");
+
   private static final int MEMORY_LOGGER_FREQUENCY_SECONDS = 60;
 
+
   private static final String METRIC_PREFIX = "drill.allocator.";
+
   private final MetricRegistry registry;
   private final AtomicLong hugeBufferSize = new AtomicLong(0);
   private final AtomicLong hugeBufferCount = new AtomicLong(0);
   private final AtomicLong normalBufferSize = new AtomicLong(0);
   private final AtomicLong normalBufferCount = new AtomicLong(0);
 
-  private final PoolArena[] directArenas;
-  private final MemoryStatusThread statusThread;
-  private final Histogram largeBuffersHist;
-  private final Histogram normalBuffersHist;
+  public final InnerAllocator allocator;
+  public final UnsafeDirectLittleEndian empty;
 
   public PooledByteBufAllocatorL(MetricRegistry registry) {
-super(true);
 this.registry = registry;
+allocator = new InnerAllocator();
+empty = new UnsafeDirectLittleEndian(new 
DuplicatedByteBuf(Unpooled.EMPTY_BUFFER));
+  }
+
+  public UnsafeDirectLittleEndian allocate(int size) {
 try {
-  Field f = 
PooledByteBufAllocator.class.getDeclaredField("directArenas");
-  f.setAccessible(true);
-  this.directArenas = (PoolArena[]) f.get(this);
-} catch (Exception e) {
-  throw new RuntimeException("Failure while initializing allocator.  
Unable to retrieve direct arenas field.", e);
+  return allocator.directBuffer(size, size);
+} catch (OutOfMemoryError e) {
+  throw new OutOfMemoryException("Failure allocating buffer.", e);
 }
 
-if (memoryLogger.isTraceEnabled()) {
-  statusThread = new MemoryStatusThread();
-  statusThread.start();
-} else {
-  statusThread = null;
-}
-removeOldMetrics();
+  }
 
-registry.register(METRIC_PREFIX + "normal.size", new Gauge() {
-  @Override
-  public Long getValue() {
-return normalBufferSize.get();
-  }
-});
+  public int getChunkSize() {
+return allocator.chunkSize;
+  }
 
-registry.register(METRIC_PREFIX + "normal.count", new Gauge() {
-  @Override
-  public Long getValue() {
-return normalBufferCount.get();
-  }
-});
+  private class InnerAllocator extends PooledByteBufAllocator {
 
-registry.register(METRIC_PREFIX + "huge.size", new Gauge() {
-  @Override
-  public Long getValue() {
-return hugeBufferSize.get();
-  }
-});
 
-registry.register(METRIC_PREFIX + "huge.count", new Gauge() {
-  @Override
-  public Long getValue() {
-return hugeBufferCount.get();
-  }
-});
+private final PoolArena[] directArenas;
+private final MemoryStatusThread statusThread;
+private final Histogram largeBuffersHist;
+private final Histogram normalBuffersHist;
+private final int chunkSize;
 
-largeBuffersHist = registry.histogram(METRIC_PREFIX + "huge.hist");
-normalBuffersHist = registry.histogram(METRIC_PREFIX + "normal.hist");
+public InnerAllocator() {
+  super(true);
 
-  }
+  try {
+Field f = 
PooledByteBufAllocator.class.getDeclaredField("directArenas");
+f.setAccessible(true);
+this.directArenas = (PoolArena[]) f.get(this);
+  } catch (Exception e) {
+throw new RuntimeException("Failure while initializing allocator.  
Unable to retrieve direct arenas field.", e);
+  }
 
-  private synchronized void removeOldMetrics() {
-registry.removeMatching(new MetricFilter() {
-  @Override
-  public boolean matches(String name, Metric metric) {
-return name.startsWith("drill.allocator.");
+  this.chunk

[GitHub] drill pull request: DRILL-4134: Allocator updates

[julienledem]

Github user julienledem commented on a diff in the pull request:

https://github.com/apache/drill/pull/283#discussion_r46315028
  
--- Diff: 
exec/memory/base/src/main/java/org/apache/drill/exec/memory/AllocatorManager.java
 ---
@@ -0,0 +1,386 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.memory;
+
+import static org.apache.drill.exec.memory.BaseAllocator.indent;
+import io.netty.buffer.DrillBuf;
+import io.netty.buffer.PooledByteBufAllocatorL;
+import io.netty.buffer.UnsafeDirectLittleEndian;
+
+import java.util.IdentityHashMap;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.concurrent.locks.Lock;
+import java.util.concurrent.locks.ReadWriteLock;
+import java.util.concurrent.locks.ReentrantReadWriteLock;
+
+import org.apache.drill.common.HistoricalLog;
+import org.apache.drill.exec.memory.BaseAllocator.Verbosity;
+import org.apache.drill.exec.metrics.DrillMetrics;
+import org.apache.drill.exec.ops.BufferManager;
+
+import com.carrotsearch.hppc.LongObjectOpenHashMap;
+import com.google.common.base.Preconditions;
+
+/**
+ * Manages the relationship between one or more allocators and a 
particular UDLE. Ensures that one allocator owns the
+ * memory that multiple allocators may be referencing. Manages a 
BufferLedger between each of its associated allocators.
+ * This class is also responsible for managing when memory is allocated 
and returned to the Netty-based
+ * PooledByteBufAllocatorL.
+ *
+ * The only reason that this isn't package private is we're forced to put 
DrillBuf in Netty's package which need access
+ * to these objects or methods.
+ *
+ * Threading: AllocatorManager manages thread-safety internally. 
Operations within the context of a single BufferLedger
+ * are lockless in nature and can be leveraged by multiple threads. 
Operations that cross the context of two ledgers
+ * will acquire a lock on the AllocatorManager instance. Important note, 
there is one AllocatorManager per
+ * UnsafeDirectLittleEndian buffer allocation. As such, there will be 
thousands of these in a typical query. The
+ * contention of acquiring a lock on AllocatorManager should be very low.
+ *
+ */
+public class AllocatorManager {
+  // private static final org.slf4j.Logger logger = 
org.slf4j.LoggerFactory.getLogger(AllocatorManager.class);
+
+  private static final AtomicLong LEDGER_ID_GENERATOR = new AtomicLong(0);
+  static final PooledByteBufAllocatorL INNER_ALLOCATOR = new 
PooledByteBufAllocatorL(DrillMetrics.getInstance());
+
+  private final RootAllocator root;
+  private volatile BufferLedger owningLedger;
+  private final int size;
+  private final UnsafeDirectLittleEndian underlying;
+  private final ReadWriteLock lock = new ReentrantReadWriteLock();
+  private final LongObjectOpenHashMap map = new 
LongObjectOpenHashMap<>();
+  private final AutoCloseableLock readLock = new 
AutoCloseableLock(lock.readLock());
+  private final AutoCloseableLock writeLock = new 
AutoCloseableLock(lock.writeLock());
+  private final IdentityHashMap buffers =
+  BaseAllocator.DEBUG ? new IdentityHashMap() : null;
+
+  AllocatorManager(BaseAllocator accountingAllocator, int size) {
+Preconditions.checkNotNull(accountingAllocator);
+this.root = accountingAllocator.root;
+this.underlying = INNER_ALLOCATOR.allocate(size);
+this.owningLedger = associate(accountingAllocator);
+this.size = underlying.capacity();
+  }
+
+  /**
+   * Associate the existing underlying buffer with a new allocator.
+   *
+   * @param allocator
+   *  The target allocator to associate this buffer with.
+   * @return The Ledger (new or existing) that associates the underlying 
buffer to this new ledger.
+   */
+  public BufferLedger associate(BaseAllocator allocator) {
+if (root != allocator.root)

[GitHub] drill pull request: DRILL-4134: Allocator updates

[julienledem]

Github user julienledem commented on a diff in the pull request:

https://github.com/apache/drill/pull/283#discussion_r46315590
  
--- Diff: 
exec/memory/base/src/main/java/org/apache/drill/exec/memory/BaseAllocator.java 
---
@@ -0,0 +1,689 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.memory;
+
+import io.netty.buffer.ByteBufAllocator;
+import io.netty.buffer.DrillBuf;
+import io.netty.buffer.UnsafeDirectLittleEndian;
+
+import java.util.Arrays;
+import java.util.IdentityHashMap;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.concurrent.atomic.AtomicLong;
+
+import org.apache.drill.common.HistoricalLog;
+import org.apache.drill.exec.exception.OutOfMemoryException;
+import org.apache.drill.exec.memory.AllocatorManager.BufferLedger;
+import org.apache.drill.exec.ops.BufferManager;
+import org.apache.drill.exec.util.AssertionUtil;
+
+import com.google.common.base.Preconditions;
+
+public abstract class BaseAllocator extends Accountant implements 
BufferAllocator {
+  private static final org.slf4j.Logger logger = 
org.slf4j.LoggerFactory.getLogger(BaseAllocator.class);
+
+  public static final String DEBUG_ALLOCATOR = 
"drill.memory.debug.allocator";
+
+  private static final AtomicLong ID_GENERATOR = new AtomicLong(0);
+  private static final int CHUNK_SIZE = 
AllocatorManager.INNER_ALLOCATOR.getChunkSize();
+
+  public static final int DEBUG_LOG_LENGTH = 6;
+  public static final boolean DEBUG = AssertionUtil.isAssertionsEnabled()
+  || Boolean.parseBoolean(System.getProperty(DEBUG_ALLOCATOR, 
"false"));
+  private final Object DEBUG_LOCK = DEBUG ? new Object() : null;
+
+  private final BaseAllocator parentAllocator;
+  private final ByteBufAllocator thisAsByteBufAllocator;
+  private final IdentityHashMap childAllocators;
+  private final DrillBuf empty;
+
+  private volatile boolean isClosed = false; // the allocator has been 
closed
+
+  // Package exposed for sharing between AllocatorManger and BaseAllocator 
objects
+  final long id = ID_GENERATOR.incrementAndGet(); // unique ID assigned to 
each allocator
+  final String name;
+  final RootAllocator root;
+
+  // members used purely for debugging
+  private final IdentityHashMap childLedgers;
+  private final IdentityHashMap reservations;
+  private final HistoricalLog historicalLog;
+
+  protected BaseAllocator(
+  final BaseAllocator parentAllocator,
+  final String name,
+  final long initReservation,
+  final long maxAllocation) throws OutOfMemoryException {
+super(parentAllocator, initReservation, maxAllocation);
+
+if (parentAllocator != null) {
+  this.root = parentAllocator.root;
+  empty = parentAllocator.empty;
+} else if (this instanceof RootAllocator) {
+  this.root = (RootAllocator) this;
+  empty = createEmpty();
+} else {
+  throw new IllegalStateException("An parent allocator must either 
carry a root or be the root.");
--- End diff --

add the parent allocator in the message to help with debugging?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request: DRILL-4134: Allocator updates

[julienledem]

Github user julienledem commented on a diff in the pull request:

https://github.com/apache/drill/pull/283#discussion_r46316017
  
--- Diff: 
exec/memory/base/src/main/java/io/netty/buffer/PooledByteBufAllocatorL.java ---
@@ -23,193 +23,246 @@
 import java.nio.ByteBuffer;
 import java.util.concurrent.atomic.AtomicLong;
 
+import org.apache.drill.exec.exception.OutOfMemoryException;
+
 import com.codahale.metrics.Gauge;
 import com.codahale.metrics.Histogram;
 import com.codahale.metrics.Metric;
 import com.codahale.metrics.MetricFilter;
 import com.codahale.metrics.MetricRegistry;
 
-public class PooledByteBufAllocatorL extends PooledByteBufAllocator{
-  static final org.slf4j.Logger logger = 
org.slf4j.LoggerFactory.getLogger(PooledByteBufAllocatorL.class);
-
+public class PooledByteBufAllocatorL {
--- End diff --

javadoc?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request: DRILL-4134: Allocator updates

[julienledem]

Github user julienledem commented on a diff in the pull request:

https://github.com/apache/drill/pull/283#discussion_r46316115
  
--- Diff: 
exec/memory/base/src/main/java/org/apache/drill/exec/memory/README.md ---
@@ -0,0 +1,121 @@
+
+# Memory: Allocation, Accounting and Management
+ 
+The memory management package contains all the memory allocation related 
items that Drill uses to manage memory.
+
+
+## Key Components
+Memory management can be broken into the following main components:
+
+- Memory chunk allocation and fragmentation management
+  - `PooledByteBufAllocatorL` - A LittleEndian clone of Netty's jemalloc 
implementation
+  - `UnsafeDirectLittleEndian` - A base level memory access interface
+  - `LargeBuffer` - A buffer backing implmentation used when working with 
data larger than one Netty chunk (default to 16mb)
--- End diff --

typo: s/implmentation/implementation


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request: DRILL-4134: Allocator updates

[julienledem]

Github user julienledem commented on a diff in the pull request:

https://github.com/apache/drill/pull/283#discussion_r46316081
  
--- Diff: 
exec/memory/base/src/main/java/io/netty/buffer/UnsafeDirectLittleEndian.java ---
@@ -21,21 +21,63 @@
 import io.netty.util.internal.PlatformDependent;
 
 import java.nio.ByteOrder;
+import java.util.Collections;
+import java.util.IdentityHashMap;
+import java.util.LinkedList;
+import java.util.Map;
+import java.util.Set;
 import java.util.concurrent.atomic.AtomicLong;
 
+import org.apache.drill.common.StackTrace;
+import org.slf4j.Logger;
+
 public final class UnsafeDirectLittleEndian extends WrappedByteBuf {
--- End diff --

javadoc?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request: DRILL-4134: Allocator updates

[julienledem]

Github user julienledem commented on a diff in the pull request:

https://github.com/apache/drill/pull/283#discussion_r46316409
  
--- Diff: 
exec/memory/base/src/main/java/org/apache/drill/exec/memory/README.md ---
@@ -0,0 +1,121 @@
+
+# Memory: Allocation, Accounting and Management
+ 
+The memory management package contains all the memory allocation related 
items that Drill uses to manage memory.
+
+
+## Key Components
+Memory management can be broken into the following main components:
+
+- Memory chunk allocation and fragmentation management
+  - `PooledByteBufAllocatorL` - A LittleEndian clone of Netty's jemalloc 
implementation
+  - `UnsafeDirectLittleEndian` - A base level memory access interface
+  - `LargeBuffer` - A buffer backing implmentation used when working with 
data larger than one Netty chunk (default to 16mb)
+- Memory limits & Accounting
+  - `Accountant` - A nestable class of lockfree memory accountors.
+- Application-level memory allocation
+  - `BufferAllocator` - The public interface application users should be 
leveraging
+  - `BaseAllocator` - The base implementation of memory allocation, 
contains the meat of our the Drill allocator implementation
+  - `RootAllocator` - The root allocator. Typically only one created for a 
JVM
+  - `ChildAllocator` - A child allocator that derives from the root 
allocator
+- Buffer ownership and transfer capabilities
+  - `AllocatorManager` - Responsible for managing the relationship between 
multiple allocators and a single chunk of memory
+  - `BufferLedger` - Responsible for allowing maintaining the relationship 
between an `AllocatorManager`, a `BufferAllocator` and one or more individual 
`DrillBuf`s 
+- Memory access
+  - `DrillBuf` - The facade for interacting directly with a chunk of 
memory.
+ 
+
+## Memory Management Overview
+Drill's memory model is based on the following basic concepts:
+
+ - Memory can be allocated up to some limit. That limit could be a real 
limit (OS/JVM) or a locally imposed limit.
+ - Allocation operates in two phases: accounting then actual allocation. 
Allocation could fail at either point.
+ - Allocation failure should be recoverable. In all cases, the Allocator 
infrastructure should expose memory allocation failures (OS or internal 
limit-based) as `OutOfMemoryException`s.
+ - Any allocator can reserve memory when created. This memory shall be 
held such that this allocator will always be able to allocate that amount of 
memory.
+ - A particular application component should work to use a local allocator 
to understand local memory usage and better debug memory leaks.
+ - The same physical memory can be shared by multiple allocators and the 
allocator must provide an accounting paradigm for this purpose.
+
+## Allocator Trees
+
+Drill provides a tree-based model for memory allocation. The RootAllocator 
is created first, then all allocators are created as children of that 
allocator. The RootAllocator is responsible for being the master bookeeper for 
memory allocations. All other allocators are created as children of this tree. 
Each allocator can first determine whether it has enough local memory to 
satisfy a particular request. If not, the allocator can ask its parent for an 
additional memory allocation.
+
+## Reserving Memory
+
+Drill provides two different ways to reserve memory:
+
+  - BufferAllocator accounting reservations: 
+  When a new allocator (other than the `RootAllocator`) is 
initialized, it can set aside memory that it will keep locally for its 
lifetime. This is memory that will never be released back to its parent 
allocator until the allocator
--- End diff --

missing end of sentence?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request: DRILL-4134: Allocator updates

[julienledem]

Github user julienledem commented on a diff in the pull request:

https://github.com/apache/drill/pull/283#discussion_r46316597
  
--- Diff: 
exec/memory/base/src/main/java/org/apache/drill/exec/memory/README.md ---
@@ -0,0 +1,121 @@
+
+# Memory: Allocation, Accounting and Management
+ 
+The memory management package contains all the memory allocation related 
items that Drill uses to manage memory.
+
+
+## Key Components
+Memory management can be broken into the following main components:
+
+- Memory chunk allocation and fragmentation management
+  - `PooledByteBufAllocatorL` - A LittleEndian clone of Netty's jemalloc 
implementation
+  - `UnsafeDirectLittleEndian` - A base level memory access interface
+  - `LargeBuffer` - A buffer backing implmentation used when working with 
data larger than one Netty chunk (default to 16mb)
+- Memory limits & Accounting
+  - `Accountant` - A nestable class of lockfree memory accountors.
+- Application-level memory allocation
+  - `BufferAllocator` - The public interface application users should be 
leveraging
+  - `BaseAllocator` - The base implementation of memory allocation, 
contains the meat of our the Drill allocator implementation
+  - `RootAllocator` - The root allocator. Typically only one created for a 
JVM
+  - `ChildAllocator` - A child allocator that derives from the root 
allocator
+- Buffer ownership and transfer capabilities
+  - `AllocatorManager` - Responsible for managing the relationship between 
multiple allocators and a single chunk of memory
+  - `BufferLedger` - Responsible for allowing maintaining the relationship 
between an `AllocatorManager`, a `BufferAllocator` and one or more individual 
`DrillBuf`s 
+- Memory access
+  - `DrillBuf` - The facade for interacting directly with a chunk of 
memory.
+ 
+
+## Memory Management Overview
+Drill's memory model is based on the following basic concepts:
+
+ - Memory can be allocated up to some limit. That limit could be a real 
limit (OS/JVM) or a locally imposed limit.
+ - Allocation operates in two phases: accounting then actual allocation. 
Allocation could fail at either point.
+ - Allocation failure should be recoverable. In all cases, the Allocator 
infrastructure should expose memory allocation failures (OS or internal 
limit-based) as `OutOfMemoryException`s.
+ - Any allocator can reserve memory when created. This memory shall be 
held such that this allocator will always be able to allocate that amount of 
memory.
+ - A particular application component should work to use a local allocator 
to understand local memory usage and better debug memory leaks.
+ - The same physical memory can be shared by multiple allocators and the 
allocator must provide an accounting paradigm for this purpose.
+
+## Allocator Trees
+
+Drill provides a tree-based model for memory allocation. The RootAllocator 
is created first, then all allocators are created as children of that 
allocator. The RootAllocator is responsible for being the master bookeeper for 
memory allocations. All other allocators are created as children of this tree. 
Each allocator can first determine whether it has enough local memory to 
satisfy a particular request. If not, the allocator can ask its parent for an 
additional memory allocation.
+
+## Reserving Memory
+
+Drill provides two different ways to reserve memory:
+
+  - BufferAllocator accounting reservations: 
+  When a new allocator (other than the `RootAllocator`) is 
initialized, it can set aside memory that it will keep locally for its 
lifetime. This is memory that will never be released back to its parent 
allocator until the allocator
+  - `AllocationReservation` via BufferAllocator.newReservation(): Allows a 
short-term preallocation strategy so that a particular subsystem can ensure 
future memory is available to support a particular request.
+  
+## Memory Ownership, Reference Counts and Sharing
+Many BufferAllocators can reference the same piece of memory at the same 
time. The most common situation for this is in the case of a Broadcast Join: in 
this situation many downstream operators in the same Drillbit will receive the 
same physical memory. Each of these operators will be operating within its own 
Allocator context. We therefore have multiple allocators all pointing at the 
same physical memory. It is the AllocatorManager's responsibility to ensure 
that in this situation, that all memory is accurately accounted for from the 
Root's perspective and also to ensure that the memory is correctly released 
once all BufferAllocators have stopped using that memory.
+
+For simplicity of accounting, we treat that memory as being used by one of 
the BufferAllocators associated with the memory. When that allocator releases 
its claim on that memory, th

[GitHub] drill pull request: DRILL-4134: Allocator updates

[julienledem]

Github user julienledem commented on the pull request:

https://github.com/apache/drill/pull/283#issuecomment-161055115
  
This looks great!
I made some comments above.



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: Create 1.4 Release branch soon?

[Parth Chandra]

+1 on creating the release branch today.
I'd like to get DRILL-4053 in. Patch is ready and doing one more round of
regression tests.



On Tue, Dec 1, 2015 at 9:56 AM, Jacques Nadeau  wrote:

> It is December (already!). Seems like we should create a 1.4 release branch
> soon so we can get a vote going. Does someone want to volunteer as the
> release manager?
>
> thanks,
> Jacques
>
> --
> Jacques Nadeau
> CTO and Co-Founder, Dremio
>

[jira] [Created] (DRILL-4148) NullPointerException in LAG function test

[Khurram Faraaz (JIRA)]

Khurram Faraaz created DRILL-4148:
-

 Summary: NullPointerException in LAG function test
 Key: DRILL-4148
 URL: https://issues.apache.org/jira/browse/DRILL-4148
 Project: Apache Drill
  Issue Type: Bug
  Components: Execution - Relational Operators
Affects Versions: 1.4.0
 Environment: 4 node cluster CentOS
Reporter: Khurram Faraaz


Failing test case : Functional/window_functions/lag_func/lag_Fn_4.q

Tests were run on JDK8 with assertions enabled.
{noformat}
[root@centos-01 lag_func]# java -version
openjdk version "1.8.0_65"
OpenJDK Runtime Environment (build 1.8.0_65-b17)
OpenJDK 64-Bit Server VM (build 25.65-b01, mixed mode)

[root@centos-01 lag_func]# uname -a
Linux centos-01 2.6.32-431.el6.x86_64 #1 SMP Fri Nov 22 03:15:09 UTC 2013 
x86_64 x86_64 x86_64 GNU/Linux
{noformat}

Stack trace from drillbit.log

{noformat}
2015-12-01 03:31:50,133 [29a2eb59-d5a3-f0f2-d2df-9ad4e5d17109:foreman] ERROR 
o.a.drill.exec.work.foreman.Foreman - SYSTEM ERROR: NullPointerException


[Error Id: ab046c45-4a2d-428d-8c72-592a02ea53e5 on centos-02.qa.lab:31010]
org.apache.drill.common.exceptions.UserException: SYSTEM ERROR: 
NullPointerException


[Error Id: ab046c45-4a2d-428d-8c72-592a02ea53e5 on centos-02.qa.lab:31010]
at 
org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:534)
 ~[drill-common-1.4.0-SNAPSHOT.jar:1.4.0-SNAPSHOT]
at 
org.apache.drill.exec.work.foreman.Foreman$ForemanResult.close(Foreman.java:742)
 [drill-java-exec-1.4.0-SNAPSHOT.jar:1.4.0-SNAPSHOT]
at 
org.apache.drill.exec.work.foreman.Foreman$StateSwitch.processEvent(Foreman.java:841)
 [drill-java-exec-1.4.0-SNAPSHOT.jar:1.4.0-SNAPSHOT]
at 
org.apache.drill.exec.work.foreman.Foreman$StateSwitch.processEvent(Foreman.java:786)
 [drill-java-exec-1.4.0-SNAPSHOT.jar:1.4.0-SNAPSHOT]
at 
org.apache.drill.common.EventProcessor.sendEvent(EventProcessor.java:73) 
[drill-common-1.4.0-SNAPSHOT.jar:1.4.0-SNAPSHOT]
at 
org.apache.drill.exec.work.foreman.Foreman$StateSwitch.moveToState(Foreman.java:788)
 [drill-java-exec-1.4.0-SNAPSHOT.jar:1.4.0-SNAPSHOT]
at 
org.apache.drill.exec.work.foreman.Foreman.moveToState(Foreman.java:894) 
[drill-java-exec-1.4.0-SNAPSHOT.jar:1.4.0-SNAPSHOT]
at org.apache.drill.exec.work.foreman.Foreman.run(Foreman.java:255) 
[drill-java-exec-1.4.0-SNAPSHOT.jar:1.4.0-SNAPSHOT]
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 
[na:1.8.0_65]
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
[na:1.8.0_65]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_65]
Caused by: org.apache.drill.exec.work.foreman.ForemanException: Unexpected 
exception during fragment initialization: null
... 4 common frames omitted
Caused by: java.lang.NullPointerException: null
at 
com.google.common.base.Preconditions.checkNotNull(Preconditions.java:191) 
~[guava-14.0.1.jar:na]
at 
org.apache.calcite.rel.metadata.CachingRelMetadataProvider$CachingInvocationHandler.(CachingRelMetadataProvider.java:104)
 ~[calcite-core-1.4.0-drill-r9.jar:1.4.0-drill-r9]
at 
org.apache.calcite.rel.metadata.CachingRelMetadataProvider$2.apply(CachingRelMetadataProvider.java:78)
 ~[calcite-core-1.4.0-drill-r9.jar:1.4.0-drill-r9]
at 
org.apache.calcite.rel.metadata.CachingRelMetadataProvider$2.apply(CachingRelMetadataProvider.java:75)
 ~[calcite-core-1.4.0-drill-r9.jar:1.4.0-drill-r9]
at 
org.apache.calcite.plan.hep.HepRelMetadataProvider$1.apply(HepRelMetadataProvider.java:45)
 ~[calcite-core-1.4.0-drill-r9.jar:1.4.0-drill-r9]
at 
org.apache.calcite.plan.hep.HepRelMetadataProvider$1.apply(HepRelMetadataProvider.java:34)
 ~[calcite-core-1.4.0-drill-r9.jar:1.4.0-drill-r9]
at 
org.apache.calcite.rel.metadata.CachingRelMetadataProvider$2.apply(CachingRelMetadataProvider.java:77)
 ~[calcite-core-1.4.0-drill-r9.jar:1.4.0-drill-r9]
at 
org.apache.calcite.rel.metadata.CachingRelMetadataProvider$2.apply(CachingRelMetadataProvider.java:75)
 ~[calcite-core-1.4.0-drill-r9.jar:1.4.0-drill-r9]
at 
org.apache.calcite.rel.metadata.MetadataFactoryImpl.query(MetadataFactoryImpl.java:69)
 ~[calcite-core-1.4.0-drill-r9.jar:1.4.0-drill-r9]
at 
org.apache.calcite.rel.AbstractRelNode.metadata(AbstractRelNode.java:271) 
~[calcite-core-1.4.0-drill-r9.jar:1.4.0-drill-r9]
at 
org.apache.calcite.rel.metadata.RelMetadataQuery.collations(RelMetadataQuery.java:293)
 ~[calcite-core-1.4.0-drill-r9.jar:1.4.0-drill-r9]
at 
org.apache.calcite.rel.metadata.RelMdCollation.calc(RelMdCollation.java:180) 
~[calcite-core-1.4.0-drill-r9.jar:1.4.0-drill-r9]
at 
org.apache.calcite.rel.logical.LogicalCalc$2.get(LogicalCalc.java:97) 
~[calcite-core-1.4.0-drill-r9.jar:1.4.0-drill-r9]
at 
org.apache.calcite.

Re: Naming the new ValueVector Initiative

[Julien Le Dem]

+1

On Mon, Nov 30, 2015 at 6:34 PM, Wes McKinney  wrote:

> Should we have a last call for votes, closing EOD tomorrow (Tuesday)? I
> missed this for a few days last week with holiday travel.
>
> On Thu, Nov 26, 2015 at 3:04 PM, Julian Hyde 
> wrote:
>
> > Consulting a lawyer is part of the Apache branding process but the first
> > stage is to gather a list of potential conflicts -
> > https://issues.apache.org/jira/browse/PODLINGNAMESEARCH-90 is an
> example.
> >
> > The other part, frankly, is to pick your battles.
> >
> > A year or so ago Actian re-branded Vectorwise as Vector.
> >
> http://www.zdnet.com/article/actian-consolidates-its-analytics-portfolio/.
> > Given that it is an analytic database in the Hadoop space I think that is
> > as close to a “direct hit” as it gets. I don’t think we need a lawyer to
> > tell us that. Certainly it makes sense to look for conflicts for the
> other
> > alternatives before consulting lawyers.
> >
> > Julian
> >
> >
> >
> >
> > On Nov 25, 2015, at 9:42 PM, Marcel Kornacker 
> wrote:
> >
> >
> >
> > On Tue, Nov 24, 2015 at 3:25 PM, Jacques Nadeau 
> > wrote:
> >
> >> Ok guys,
> >>
> >> I don't think anyone is doing a thorough analysis of viaability. I did a
> >> quick glance and the top one (Vector) seems like it would have an issue
> >> with conflict of an Actian product. The may be fine. Let's do a second
> >> phase vote.
> >>
> >
> > I'm assuming you mean Vectorwise?
> >
> > Before we do anything else, could we have a lawyer look into this? Last
> > time around that I remember (Parquet), Twitter's lawyers did a good job
> of
> > weeding out the potential trademark violations.
> >
> > Alex, could Twitter get involved this time around as well?
> >
> >
> >>
> >> Pick your top 3 (1,2,3 with 3 being top preference)
> >>
> >> Let's get this done by Friday and then we can do a podling name search
> >> starting with the top one.
> >>
> >> Link again:
> >>
> >>
> >>
> https://docs.google.com/spreadsheets/d/1q6UqluW6SLuMKRwW2TBGBzHfYLlXYm37eKJlIxWQGQM/edit#gid=304381532&vpid=A1
> >>
> >> thanks
> >>
> >>
> >> --
> >> Jacques Nadeau
> >> CTO and Co-Founder, Dremio
> >>
> >> On Fri, Nov 20, 2015 at 9:24 AM, Jacques Nadeau 
> >> wrote:
> >>
> >>> Ok, it looks like we have a candidate list (we actually got 11 since
> >>> there was a three-way tie for ninth place):
> >>>
> >>> VectorArrowhoneycombHerringbonejoistV2Pietcolbufbatonimpulsevictor
> >>> Next we need to do trademark searches on each of these to see whether
> >>> we're likely to have success. I've moved candidates to a second tab:
> >>>
> >>>
> >>>
> https://docs.google.com/spreadsheets/d/1q6UqluW6SLuMKRwW2TBGBzHfYLlXYm37eKJlIxWQGQM/edit#gid=304381532
> >>>
> >>> Anybody want to give a hand in analyzing potential conflicts?
> >>>
> >>>
> >>>
> >>> --
> >>> Jacques Nadeau
> >>> CTO and Co-Founder, Dremio
> >>>
> >>> On Mon, Nov 16, 2015 at 12:10 PM, Jacques Nadeau 
> >>> wrote:
> >>>
>  Everybody should pick their ten favorites using the numbers 1 to 10.
> 
>  10 is most preferred
> 
> 
> 
>  --
>  Jacques Nadeau
>  CTO and Co-Founder, Dremio
> 
>  On Mon, Nov 16, 2015 at 10:17 AM, Ted Dunning 
>  wrote:
> 
> >
> > Single vote for most preferred?
> >
> > Single transferable vote?
> >
> >
> >
> > On Tue, Nov 17, 2015 at 2:50 AM, Jacques Nadeau 
> > wrote:
> >
> >> Given that a bunch of people added names to the sheet, I'll take
> that
> >> as tacit agreement to the proposed process.
> >>
> >> Let's move to the first vote phase. I've added a column for
> >> everybody's votes. Let's try to wrap up the vote by 10am on
> Wednesday.
> >>
> >> thanks!
> >> Jacques
> >>
> >>
> >>
> >> --
> >> Jacques Nadeau
> >> CTO and Co-Founder, Dremio
> >>
> >> On Thu, Nov 12, 2015 at 12:03 PM, Jacques Nadeau <
> jacq...@apache.org>
> >> wrote:
> >>
> >>> Hey Guys,
> >>>
> >>> It sounds like we need to do a little more work on the Vector
> >>> proposal
> >>> before the board would like to consider it. The main point of
> >>> contention
> >>> right now is the name of the project. We need to decide on a name
> >>> and get
> >>> it signed off through PODLINGNAMESEARCH.
> >>>
> >>> Naming is extremely subjective so I'd like to propose a process for
> >>> selection that minimizes pain. This is an initial proposal and
> >>>
> >>> We do the naming in the following steps
> >>> - 1: Collect a set of names to be considered
> >>> - 2: Run a vote for 2 days where each member ranks their top 10
> >>> options
> >>> 1..10
> >>> - 3: Take the top ten vote getters and do a basic analysis of
> >>> whether we
> >>> think that any have legal issues. Keep dropping names that have
> this
> >>> until
> >>> we get with 10 reasonably solid candidate names
> >>> - 5: Take the top ten names and give people 48 

[GitHub] drill pull request: Drill 4127: Reduce Hive metastore client API c...

[jinfengni]

GitHub user jinfengni opened a pull request:

https://github.com/apache/drill/pull/286

Drill 4127: Reduce Hive metastore client API call in HiveSchema

Also, it has commit for DRILL-4126: Add cache to HiveSchema in order to 
reduce long planning time or execution time caused by slow Hive meta store.

Both DRILL-4127 and DRILL-4126 address the long delay caused by slow hive 
meta store. 
 
Passed unit, pre-commit regression, and additional impersonation test, 
before rebasing onto latest master.

Will re-run the above tests. 

@vkorukanti , could you please review the two patches? Thanks.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/jinfengni/incubator-drill DRILL-4127

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/drill/pull/286.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #286


commit 19a5a4d1c9c23eedcb94c988bd2229680575a118
Author: Jinfeng Ni 
Date:   2015-11-19T04:18:51Z

DRILL-4127: Reduce Hive metastore client API call in HiveSchema.

1) Use lazy loading of tableNames in HiveSchema, in stead of pre-loading 
all table names under each HiveSchema.
2) Do not call get_all_databases for subSchema to check existence if the 
name comes from getSubSchemaNames() directly.

commit 9570319c227649144d3a14f8d5774fbe4a282bc4
Author: Jinfeng Ni 
Date:   2015-11-30T04:15:07Z

DRILL-4126: Add cache to HiveSchema in order to reduce long planning time 
or execution time caused by slow Hive meta store.

1) HiveSchema caching will help in case impersonation is enabled.
2) Use flat level cache for tables in DrillHiveMetaStoreClient.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request: DRILL-4053: Reduce metadata cache file size. S...

[parthchandra]

Github user parthchandra commented on the pull request:

https://github.com/apache/drill/pull/254#issuecomment-161064958
  
Updated the patch. @StevenMPhillips, @jacques-n, if you guys can do a quick 
review. 
The cache code now uses the same file name and reads the metadata 
appropriately. Note that as a result, v1 and v2 of the cache file can not 
co-exist (and also means that Drill 1.3 and earlier are no longer forward 
compatible).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Can we pass the #skipped records with RecordBatch?

[Hsuan Yi Chu]

I am trying to deal with the following scenario:

A bunch of minor fragments are doing things in parallel. Each of them could
skip some records. Since the downstream minor fragment needs to know the
sum of skipped-record-counts (in order to just display or see if the number
exceeds the threshold) in the upstreams, each upstream minor fragment needs
to pass this scalar with RecordBatch.

Since this seems impacting the protocol of RecordBatch, I am looking for
some advice here.

Thanks.

[GitHub] drill pull request: DRILL-3739: Fix issues in reading Hive tables ...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/drill/pull/215


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request: DRILL-3938: Support reading from Hive tables t...

[vkorukanti]

Github user vkorukanti closed the pull request at:

https://github.com/apache/drill/pull/211


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request: DRILL-3893: Change Hive metadata cache invalid...

[vkorukanti]

Github user vkorukanti closed the pull request at:

https://github.com/apache/drill/pull/212


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: Naming the new ValueVector Initiative

[Jason Altekruse]

Jake,

I think that Julian was trying make a point about the use of the complete
name Apache Arrow, not thinking about the project name in isolation. That
being said I completely agree that it is not a derogatory term. I might
make a case that the logo should be a mathematical representation of an
arrow rather than one with a big cartoonish arrowhead if we did go with it.

+1 to a vote

- Jason

On Tue, Dec 1, 2015 at 10:51 AM, Julien Le Dem  wrote:

> +1
>
> On Mon, Nov 30, 2015 at 6:34 PM, Wes McKinney  wrote:
>
> > Should we have a last call for votes, closing EOD tomorrow (Tuesday)? I
> > missed this for a few days last week with holiday travel.
> >
> > On Thu, Nov 26, 2015 at 3:04 PM, Julian Hyde 
> > wrote:
> >
> > > Consulting a lawyer is part of the Apache branding process but the
> first
> > > stage is to gather a list of potential conflicts -
> > > https://issues.apache.org/jira/browse/PODLINGNAMESEARCH-90 is an
> > example.
> > >
> > > The other part, frankly, is to pick your battles.
> > >
> > > A year or so ago Actian re-branded Vectorwise as Vector.
> > >
> >
> http://www.zdnet.com/article/actian-consolidates-its-analytics-portfolio/.
> > > Given that it is an analytic database in the Hadoop space I think that
> is
> > > as close to a “direct hit” as it gets. I don’t think we need a lawyer
> to
> > > tell us that. Certainly it makes sense to look for conflicts for the
> > other
> > > alternatives before consulting lawyers.
> > >
> > > Julian
> > >
> > >
> > >
> > >
> > > On Nov 25, 2015, at 9:42 PM, Marcel Kornacker 
> > wrote:
> > >
> > >
> > >
> > > On Tue, Nov 24, 2015 at 3:25 PM, Jacques Nadeau 
> > > wrote:
> > >
> > >> Ok guys,
> > >>
> > >> I don't think anyone is doing a thorough analysis of viaability. I
> did a
> > >> quick glance and the top one (Vector) seems like it would have an
> issue
> > >> with conflict of an Actian product. The may be fine. Let's do a second
> > >> phase vote.
> > >>
> > >
> > > I'm assuming you mean Vectorwise?
> > >
> > > Before we do anything else, could we have a lawyer look into this? Last
> > > time around that I remember (Parquet), Twitter's lawyers did a good job
> > of
> > > weeding out the potential trademark violations.
> > >
> > > Alex, could Twitter get involved this time around as well?
> > >
> > >
> > >>
> > >> Pick your top 3 (1,2,3 with 3 being top preference)
> > >>
> > >> Let's get this done by Friday and then we can do a podling name search
> > >> starting with the top one.
> > >>
> > >> Link again:
> > >>
> > >>
> > >>
> >
> https://docs.google.com/spreadsheets/d/1q6UqluW6SLuMKRwW2TBGBzHfYLlXYm37eKJlIxWQGQM/edit#gid=304381532&vpid=A1
> > >>
> > >> thanks
> > >>
> > >>
> > >> --
> > >> Jacques Nadeau
> > >> CTO and Co-Founder, Dremio
> > >>
> > >> On Fri, Nov 20, 2015 at 9:24 AM, Jacques Nadeau 
> > >> wrote:
> > >>
> > >>> Ok, it looks like we have a candidate list (we actually got 11 since
> > >>> there was a three-way tie for ninth place):
> > >>>
> > >>> VectorArrowhoneycombHerringbonejoistV2Pietcolbufbatonimpulsevictor
> > >>> Next we need to do trademark searches on each of these to see whether
> > >>> we're likely to have success. I've moved candidates to a second tab:
> > >>>
> > >>>
> > >>>
> >
> https://docs.google.com/spreadsheets/d/1q6UqluW6SLuMKRwW2TBGBzHfYLlXYm37eKJlIxWQGQM/edit#gid=304381532
> > >>>
> > >>> Anybody want to give a hand in analyzing potential conflicts?
> > >>>
> > >>>
> > >>>
> > >>> --
> > >>> Jacques Nadeau
> > >>> CTO and Co-Founder, Dremio
> > >>>
> > >>> On Mon, Nov 16, 2015 at 12:10 PM, Jacques Nadeau  >
> > >>> wrote:
> > >>>
> >  Everybody should pick their ten favorites using the numbers 1 to 10.
> > 
> >  10 is most preferred
> > 
> > 
> > 
> >  --
> >  Jacques Nadeau
> >  CTO and Co-Founder, Dremio
> > 
> >  On Mon, Nov 16, 2015 at 10:17 AM, Ted Dunning <
> ted.dunn...@gmail.com>
> >  wrote:
> > 
> > >
> > > Single vote for most preferred?
> > >
> > > Single transferable vote?
> > >
> > >
> > >
> > > On Tue, Nov 17, 2015 at 2:50 AM, Jacques Nadeau <
> jacq...@dremio.com>
> > > wrote:
> > >
> > >> Given that a bunch of people added names to the sheet, I'll take
> > that
> > >> as tacit agreement to the proposed process.
> > >>
> > >> Let's move to the first vote phase. I've added a column for
> > >> everybody's votes. Let's try to wrap up the vote by 10am on
> > Wednesday.
> > >>
> > >> thanks!
> > >> Jacques
> > >>
> > >>
> > >>
> > >> --
> > >> Jacques Nadeau
> > >> CTO and Co-Founder, Dremio
> > >>
> > >> On Thu, Nov 12, 2015 at 12:03 PM, Jacques Nadeau <
> > jacq...@apache.org>
> > >> wrote:
> > >>
> > >>> Hey Guys,
> > >>>
> > >>> It sounds like we need to do a little more work on the Vector
> > >>> proposal
> > >>> before the board would like to consider it. The main point of
> > >>> contention
> > >>> 

Re: Naming the new ValueVector Initiative

[Alex Levenson]

I can ask about whether Twitter's lawyers can help out -- is that something
we need? Or is that something apache helps out with in the next step?

On Mon, Nov 30, 2015 at 9:32 PM, Julian Hyde  wrote:

> +1 to have a vote tomorrow.
>
> Assuming that Vector is out of play, I just did a quick search for the top
> 4 remaining, (“arrow”, “honeycomb”, “herringbone”, “joist"), at
> sourceforge, open hub, trademarkia, and on google. There are no trademarks
> for these in similar subject areas. There is a moderately active project
> called “joist” [1].
>
> I will point out that “Apache Arrow” has native-american connotations that
> we may or may not want to live with (just ask the Washington Redskins how
> they feel about their name).
>
> If someone would like to vet other names, use the links on
> https://issues.apache.org/jira/browse/PODLINGNAMESEARCH-90, and fill out
> column C in the spreadsheet.
>
> Julian
>
> [1] https://github.com/stephenh/joist
>
>
> On Nov 30, 2015, at 7:01 PM, Jacques Nadeau  wrote:
>
> +1
>
> --
> Jacques Nadeau
> CTO and Co-Founder, Dremio
>
> On Mon, Nov 30, 2015 at 6:34 PM, Wes McKinney  wrote:
>
> Should we have a last call for votes, closing EOD tomorrow (Tuesday)? I
> missed this for a few days last week with holiday travel.
>
> On Thu, Nov 26, 2015 at 3:04 PM, Julian Hyde 
> wrote:
>
> Consulting a lawyer is part of the Apache branding process but the first
> stage is to gather a list of potential conflicts -
> https://issues.apache.org/jira/browse/PODLINGNAMESEARCH-90 is an example.
>
> The other part, frankly, is to pick your battles.
>
> A year or so ago Actian re-branded Vectorwise as Vector.
> http://www.zdnet.com/article/actian-consolidates-its-analytics-portfolio/.
> Given that it is an analytic database in the Hadoop space I think that is
> as close to a “direct hit” as it gets. I don’t think we need a lawyer to
> tell us that. Certainly it makes sense to look for conflicts for the other
> alternatives before consulting lawyers.
>
> Julian
>
>
>
>
> On Nov 25, 2015, at 9:42 PM, Marcel Kornacker 
> wrote:
>
>
>
> On Tue, Nov 24, 2015 at 3:25 PM, Jacques Nadeau 
> wrote:
>
> Ok guys,
>
> I don't think anyone is doing a thorough analysis of viaability. I did a
> quick glance and the top one (Vector) seems like it would have an issue
> with conflict of an Actian product. The may be fine. Let's do a second
> phase vote.
>
>
> I'm assuming you mean Vectorwise?
>
> Before we do anything else, could we have a lawyer look into this? Last
> time around that I remember (Parquet), Twitter's lawyers did a good job of
> weeding out the potential trademark violations.
>
> Alex, could Twitter get involved this time around as well?
>
>
>
> Pick your top 3 (1,2,3 with 3 being top preference)
>
> Let's get this done by Friday and then we can do a podling name search
> starting with the top one.
>
> Link again:
>
>
>
> https://docs.google.com/spreadsheets/d/1q6UqluW6SLuMKRwW2TBGBzHfYLlXYm37eKJlIxWQGQM/edit#gid=304381532&vpid=A1
>
> thanks
>
>
> --
> Jacques Nadeau
> CTO and Co-Founder, Dremio
>
> On Fri, Nov 20, 2015 at 9:24 AM, Jacques Nadeau 
> wrote:
>
> Ok, it looks like we have a candidate list (we actually got 11 since
> there was a three-way tie for ninth place):
>
> VectorArrowhoneycombHerringbonejoistV2Pietcolbufbatonimpulsevictor
> Next we need to do trademark searches on each of these to see whether
> we're likely to have success. I've moved candidates to a second tab:
>
>
>
> https://docs.google.com/spreadsheets/d/1q6UqluW6SLuMKRwW2TBGBzHfYLlXYm37eKJlIxWQGQM/edit#gid=304381532
>
> Anybody want to give a hand in analyzing potential conflicts?
>
>
>
> --
> Jacques Nadeau
> CTO and Co-Founder, Dremio
>
> On Mon, Nov 16, 2015 at 12:10 PM, Jacques Nadeau 
> wrote:
>
> Everybody should pick their ten favorites using the numbers 1 to 10.
>
> 10 is most preferred
>
>
>
> --
> Jacques Nadeau
> CTO and Co-Founder, Dremio
>
> On Mon, Nov 16, 2015 at 10:17 AM, Ted Dunning 
> wrote:
>
>
> Single vote for most preferred?
>
> Single transferable vote?
>
>
>
> On Tue, Nov 17, 2015 at 2:50 AM, Jacques Nadeau 
> wrote:
>
> Given that a bunch of people added names to the sheet, I'll take
> that as tacit agreement to the proposed process.
>
> Let's move to the first vote phase. I've added a column for
> everybody's votes. Let's try to wrap up the vote by 10am on Wednesday.
>
> thanks!
> Jacques
>
>
>
> --
> Jacques Nadeau
> CTO and Co-Founder, Dremio
>
> On Thu, Nov 12, 2015 at 12:03 PM, Jacques Nadeau 
> wrote:
>
>
> Hey Guys,
>
> It sounds like we need to do a little more work on the Vector
> proposal
> before the board would like to consider it. The main point of
> contention
> right now is the name of the project. We need to decide on a name
> and get
> it signed off through PODLINGNAMESEARCH.
>
> Naming is extremely subjective so I'd like to propose a process for
> selection that minimizes pain. This is an initial proposal and
>
> We do the naming in the following steps
> - 1: Collect a set of n

Re: Can we pass the #skipped records with RecordBatch?

[Jacques Nadeau]

This seems like a form of sideband communication. I think we should have a
framework for this type of thing in general rather than a one-off for this
particular need. Other forms of sideband might be small table bloomfilter
generation and pushdown into hbase, separate file assignment/partitioning
providers balancing/generating scanner workloads, statistics generation for
adaptive execution, etc.

--
Jacques Nadeau
CTO and Co-Founder, Dremio

On Tue, Dec 1, 2015 at 11:35 AM, Hsuan Yi Chu  wrote:

> I am trying to deal with the following scenario:
>
> A bunch of minor fragments are doing things in parallel. Each of them could
> skip some records. Since the downstream minor fragment needs to know the
> sum of skipped-record-counts (in order to just display or see if the number
> exceeds the threshold) in the upstreams, each upstream minor fragment needs
> to pass this scalar with RecordBatch.
>
> Since this seems impacting the protocol of RecordBatch, I am looking for
> some advice here.
>
> Thanks.
>

[jira] [Created] (DRILL-4149) Escape Character Not Used for TSVs

[Matt Welsh (JIRA)]

Matt Welsh created DRILL-4149:
-

 Summary: Escape Character Not Used for TSVs
 Key: DRILL-4149
 URL: https://issues.apache.org/jira/browse/DRILL-4149
 Project: Apache Drill
  Issue Type: Bug
  Components: Execution - Data Types
Affects Versions: 1.3.0
Reporter: Matt Welsh
Priority: Minor


Escape Character does not escape tabs in TSVs

For instance query:

With Storage Format configured as:
"tsv": {
  "type": "text",
  "extensions": [
"tsv"
  ],
  "escape": "\\",
  "delimiter": "\t"
},
File:
testval 1   2   3   sometext
testval 4   5   6   some text with a tab between here\  here

This returns 5 columns for first and 6 for second.  Should be 5 for both.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (DRILL-3938) Hive: Failure reading from a partition when a new column is added to the table after the partition creation

[Venki Korukanti (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-3938?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Venki Korukanti resolved DRILL-3938.

Resolution: Fixed

> Hive: Failure reading from a partition when a new column is added to the 
> table after the partition creation
> ---
>
> Key: DRILL-3938
> URL: https://issues.apache.org/jira/browse/DRILL-3938
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Storage - Hive
>Affects Versions: 0.4.0
>Reporter: Venki Korukanti
>Assignee: Venki Korukanti
> Fix For: 1.4.0
>
>
> Repro:
> From Hive:
> {code}
> CREATE TABLE kv(key INT, value STRING);
> LOAD DATA LOCAL INPATH 
> '/Users/hadoop/apache-repos/hive-install/apache-hive-1.0.0-bin/examples/files/kv1.txt'
>  INTO TABLE kv;
> CREATE TABLE kv_p(key INT, value STRING, part1 STRING);
> set hive.exec.dynamic.partition.mode=nonstrict;
> set hive.exec.max.dynamic.partitions=1;
> set hive.exec.max.dynamic.partitions.pernode=1;
> INSERT INTO TABLE kv_p PARTITION (part1) SELECT key, value, value as s FROM 
> kv;
> ALTER TABLE kv_p ADD COLUMNS (newcol STRING);
> {code}
> From Drill:
> {code}
> USE hive;
> DESCRIBE kv_p;
> SELECT newcol FROM kv_p;
> throws column 'newcol' not found error in HiveRecordReader while selecting 
> only the projected columns.
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Can we pass the #skipped records with RecordBatch?

[Julian Hyde]

+1 for a sideband mechanism.

Sideband can also allow correlated restart of sub-queries.

In sideband use cases you described, the messages ran in the opposite direction 
to the data. Would the sideband also run in the same direction as the data? If 
so it could carry warnings, rejected rows, progress indications, and (for 
online aggregation[1]) notifications that a better approximate query result is 
available.

Julian

[1] https://en.wikipedia.org/wiki/Online_aggregation



> On Dec 1, 2015, at 1:51 PM, Jacques Nadeau  wrote:
> 
> This seems like a form of sideband communication. I think we should have a
> framework for this type of thing in general rather than a one-off for this
> particular need. Other forms of sideband might be small table bloomfilter
> generation and pushdown into hbase, separate file assignment/partitioning
> providers balancing/generating scanner workloads, statistics generation for
> adaptive execution, etc.
> 
> --
> Jacques Nadeau
> CTO and Co-Founder, Dremio
> 
> On Tue, Dec 1, 2015 at 11:35 AM, Hsuan Yi Chu  wrote:
> 
>> I am trying to deal with the following scenario:
>> 
>> A bunch of minor fragments are doing things in parallel. Each of them could
>> skip some records. Since the downstream minor fragment needs to know the
>> sum of skipped-record-counts (in order to just display or see if the number
>> exceeds the threshold) in the upstreams, each upstream minor fragment needs
>> to pass this scalar with RecordBatch.
>> 
>> Since this seems impacting the protocol of RecordBatch, I am looking for
>> some advice here.
>> 
>> Thanks.
>> 

[jira] [Resolved] (DRILL-3739) NPE on select from Hive for HBase table

[Venki Korukanti (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-3739?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Venki Korukanti resolved DRILL-3739.

   Resolution: Fixed
 Assignee: Venki Korukanti
Fix Version/s: 1.4.0

> NPE on select from Hive for HBase table
> ---
>
> Key: DRILL-3739
> URL: https://issues.apache.org/jira/browse/DRILL-3739
> Project: Apache Drill
>  Issue Type: Bug
>Affects Versions: 1.1.0
>Reporter: ckran
>Assignee: Venki Korukanti
>Priority: Critical
> Fix For: 1.4.0
>
>
> For a table in HBase or MapR-DB with metadata created in Hive so that it can 
> be accessed through beeline or Hue. From Drill query fail with
> org.apache.drill.common.exceptions.UserRemoteException: SYSTEM ERROR: 
> NullPointerException [Error Id: 1cfd2a36-bc73-4a36-83ee-ac317b8e6cdb]



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (DRILL-3893) Issue with Drill after Hive Alters the Table

[Venki Korukanti (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-3893?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Venki Korukanti resolved DRILL-3893.

   Resolution: Fixed
 Assignee: Venki Korukanti
Fix Version/s: 1.4.0

>  Issue with Drill after Hive Alters the Table
> -
>
> Key: DRILL-3893
> URL: https://issues.apache.org/jira/browse/DRILL-3893
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Functions - Hive, Storage - Hive
>Affects Versions: 1.0.0, 1.1.0
> Environment: DEV
>Reporter: arnab chatterjee
>Assignee: Venki Korukanti
> Fix For: 1.4.0
>
>
> I reproduced this again on another partitioned table with existing data.
> Providing some more details. I have enabled the version mode for errors. 
> Drill is unable to fetch the new column name that was introduced.This most 
> likely to me seems  to me that it’s still picking up the stale metadata of 
> hive.
> if (!tableColumns.contains(columnName)) {
> if (partitionNames.contains(columnName)) {
>   selectedPartitionNames.add(columnName);
> } else {
>   throw new ExecutionSetupException(String.format("Column %s does 
> not exist", columnName));
> }
>   }
> select testdata from testtable;
> Error: SYSTEM ERROR: ExecutionSetupException: Column testdata does not exist
> Fragment 0:0
> [Error Id: be5cccba-97f6-4cc4-94e8-c11a4c53c8f4 on x.x.com:]
>   (org.apache.drill.common.exceptions.ExecutionSetupException) Failure while 
> initializing HiveRecordReader: Column testdata does not exist
> org.apache.drill.exec.store.hive.HiveRecordReader.init():241
> org.apache.drill.exec.store.hive.HiveRecordReader.():138
> org.apache.drill.exec.store.hive.HiveScanBatchCreator.getBatch():58
> org.apache.drill.exec.store.hive.HiveScanBatchCreator.getBatch():34
> org.apache.drill.exec.physical.impl.ImplCreator.getRecordBatch():150
> org.apache.drill.exec.physical.impl.ImplCreator.getChildren():173
> org.apache.drill.exec.physical.impl.ImplCreator.getRootExec():106
> org.apache.drill.exec.physical.impl.ImplCreator.getExec():81
> org.apache.drill.exec.work.fragment.FragmentExecutor.run():235
> org.apache.drill.common.SelfCleaningRunnable.run():38
> java.util.concurrent.ThreadPoolExecutor.runWorker():1142
> java.util.concurrent.ThreadPoolExecutor$Worker.run():617
> java.lang.Thread.run():745
>   Caused By (org.apache.drill.common.exceptions.ExecutionSetupException) 
> Column testdata does not exist
> org.apache.drill.exec.store.hive.HiveRecordReader.init():206
> org.apache.drill.exec.store.hive.HiveRecordReader.():138
> org.apache.drill.exec.store.hive.HiveScanBatchCreator.getBatch():58
> org.apache.drill.exec.store.hive.HiveScanBatchCreator.getBatch():34
> org.apache.drill.exec.physical.impl.ImplCreator.getRecordBatch():150
> org.apache.drill.exec.physical.impl.ImplCreator.getChildren():173
> org.apache.drill.exec.physical.impl.ImplCreator.getRootExec():106
> org.apache.drill.exec.physical.impl.ImplCreator.getExec():81
> org.apache.drill.exec.work.fragment.FragmentExecutor.run():235
> org.apache.drill.common.SelfCleaningRunnable.run():38
> java.util.concurrent.ThreadPoolExecutor.runWorker():1142
> java.util.concurrent.ThreadPoolExecutor$Worker.run():617
> java.lang.Thread.run():745 (state=,code=0)
> #
> Please note that this is a partitioned table with existing data.
> Does Drill Cache the Meta somewhere and hence it’s not getting reflected 
> immediately ?
> DRILL CLI
> > select x from xx;
> Error: SYSTEM ERROR: ExecutionSetupException: Column x does not exist
> Fragment 0:0
> [Error Id: 62086e22-1341-459e-87ce-430a24cc5119 on x.x.com:999] 
> (state=,code=0)
> HIVE CLI
> hive> describe formatted x;
> OK
> # col_name  data_type   comment
> 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] drill pull request: DRILL-4134: Allocator updates

[adeneche]

Github user adeneche commented on a diff in the pull request:

https://github.com/apache/drill/pull/283#discussion_r46347779
  
--- Diff: 
exec/memory/base/src/main/java/io/netty/buffer/UnsafeDirectLittleEndian.java ---
@@ -89,156 +131,139 @@ public ByteBuf order(ByteOrder endianness) {
   }
 
   @Override
-public double getDouble(int index) {
-return Double.longBitsToDouble(getLong(index));
-}
-
-@Override
-public char getChar(int index) {
-return (char) getShort(index);
-}
-
-@Override
-public long getUnsignedInt(int index) {
-return getInt(index) & 0xL;
-}
+  public double getDouble(int index) {
+return Double.longBitsToDouble(getLong(index));
+  }
 
-@Override
-public int getInt(int index) {
-//wrapped.checkIndex(index, 4);
-int v = PlatformDependent.getInt(addr(index));
-return v;
-}
+  @Override
+  public char getChar(int index) {
+return (char) getShort(index);
+  }
 
-@Override
-public int getUnsignedShort(int index) {
-return getShort(index) & 0x;
-}
+  @Override
+  public long getUnsignedInt(int index) {
+return getInt(index) & 0xL;
+  }
 
-@Override
-public short getShort(int index) {
-//wrapped.checkIndex(index, 2);
-short v = PlatformDependent.getShort(addr(index));
-return v;
-}
+  @Override
+  public int getInt(int index) {
+int v = PlatformDependent.getInt(addr(index));
+return v;
+  }
 
-@Override
-public ByteBuf setShort(int index, int value) {
-wrapped.checkIndex(index, 2);
-_setShort(index, value);
-return this;
-}
+  @Override
+  public int getUnsignedShort(int index) {
+return getShort(index) & 0x;
+  }
 
-@Override
-public ByteBuf setInt(int index, int value) {
-wrapped.checkIndex(index, 4);
-_setInt(index, value);
-return this;
-}
+  @Override
+  public short getShort(int index) {
+short v = PlatformDependent.getShort(addr(index));
+return v;
+  }
 
-@Override
-public ByteBuf setLong(int index, long value) {
-wrapped.checkIndex(index, 8);
-_setLong(index, value);
-return this;
-}
+  @Override
+  public ByteBuf setShort(int index, int value) {
+wrapped.checkIndex(index, 2);
+_setShort(index, value);
+return this;
+  }
 
-@Override
-public ByteBuf setChar(int index, int value) {
-setShort(index, value);
-return this;
-}
+  @Override
+  public ByteBuf setInt(int index, int value) {
+wrapped.checkIndex(index, 4);
+_setInt(index, value);
+return this;
+  }
 
-@Override
-public ByteBuf setFloat(int index, float value) {
-setInt(index, Float.floatToRawIntBits(value));
-return this;
-}
+  @Override
+  public ByteBuf setLong(int index, long value) {
+wrapped.checkIndex(index, 8);
+_setLong(index, value);
+return this;
+  }
 
-@Override
-public ByteBuf setDouble(int index, double value) {
-setLong(index, Double.doubleToRawLongBits(value));
-return this;
-}
+  @Override
+  public ByteBuf setChar(int index, int value) {
+setShort(index, value);
+return this;
+  }
 
-@Override
-public ByteBuf writeShort(int value) {
-wrapped.ensureWritable(2);
-_setShort(wrapped.writerIndex, value);
-wrapped.writerIndex += 2;
-return this;
-}
+  @Override
+  public ByteBuf setFloat(int index, float value) {
+setInt(index, Float.floatToRawIntBits(value));
+return this;
+  }
 
-@Override
-public ByteBuf writeInt(int value) {
-wrapped.ensureWritable(4);
-_setInt(wrapped.writerIndex, value);
-wrapped.writerIndex += 4;
-return this;
-}
+  @Override
+  public ByteBuf setDouble(int index, double value) {
+setLong(index, Double.doubleToRawLongBits(value));
+return this;
+  }
 
-@Override
-public ByteBuf writeLong(long value) {
-wrapped.ensureWritable(8);
-_setLong(wrapped.writerIndex, value);
-wrapped.writerIndex += 8;
-return this;
-}
+  @Override
+  public ByteBuf writeShort(int value) {
+wrapped.ensureWritable(2);
+_setShort(wrapped.writerIndex, value);
+wrapped.writerIndex += 2;
+

[GitHub] drill pull request: DRILL-4134: Allocator updates

[adeneche]

Github user adeneche commented on a diff in the pull request:

https://github.com/apache/drill/pull/283#discussion_r46347956
  
--- Diff: 
exec/memory/base/src/main/java/io/netty/buffer/UnsafeDirectLittleEndian.java ---
@@ -21,21 +21,63 @@
 import io.netty.util.internal.PlatformDependent;
 
 import java.nio.ByteOrder;
+import java.util.Collections;
+import java.util.IdentityHashMap;
+import java.util.LinkedList;
+import java.util.Map;
+import java.util.Set;
 import java.util.concurrent.atomic.AtomicLong;
 
+import org.apache.drill.common.StackTrace;
+import org.slf4j.Logger;
+
 public final class UnsafeDirectLittleEndian extends WrappedByteBuf {
   private static final boolean NATIVE_ORDER = ByteOrder.nativeOrder() == 
ByteOrder.LITTLE_ENDIAN;
   private final AbstractByteBuf wrapped;
   private final long memoryAddress;
+  private static final boolean TRACK_BUFFERS = false;
--- End diff --

`TRACK_BUFFERS` is not used anywhere!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: Can we pass the #skipped records with RecordBatch?

[Hsuan Yi Chu]

@Julian:
In the use case, the message flows in the same direction as the data. And
you are right. If we have a sideband, many additional info can be carried
and displayed to the users.

I think we have a pull request specifically dealing with WARNING:
https://github.com/abhipol/drill/commit/137059cd44ec28e8ba3bf2aa73d2c1cbcd55d604

Let me see if there is anything common which can be shared among.


On Tue, Dec 1, 2015 at 2:08 PM, Julian Hyde  wrote:

> +1 for a sideband mechanism.
>
> Sideband can also allow correlated restart of sub-queries.
>
> In sideband use cases you described, the messages ran in the opposite
> direction to the data. Would the sideband also run in the same direction as
> the data? If so it could carry warnings, rejected rows, progress
> indications, and (for online aggregation[1]) notifications that a better
> approximate query result is available.
>
> Julian
>
> [1] https://en.wikipedia.org/wiki/Online_aggregation
>
>
>
> > On Dec 1, 2015, at 1:51 PM, Jacques Nadeau  wrote:
> >
> > This seems like a form of sideband communication. I think we should have
> a
> > framework for this type of thing in general rather than a one-off for
> this
> > particular need. Other forms of sideband might be small table bloomfilter
> > generation and pushdown into hbase, separate file assignment/partitioning
> > providers balancing/generating scanner workloads, statistics generation
> for
> > adaptive execution, etc.
> >
> > --
> > Jacques Nadeau
> > CTO and Co-Founder, Dremio
> >
> > On Tue, Dec 1, 2015 at 11:35 AM, Hsuan Yi Chu 
> wrote:
> >
> >> I am trying to deal with the following scenario:
> >>
> >> A bunch of minor fragments are doing things in parallel. Each of them
> could
> >> skip some records. Since the downstream minor fragment needs to know the
> >> sum of skipped-record-counts (in order to just display or see if the
> number
> >> exceeds the threshold) in the upstreams, each upstream minor fragment
> needs
> >> to pass this scalar with RecordBatch.
> >>
> >> Since this seems impacting the protocol of RecordBatch, I am looking for
> >> some advice here.
> >>
> >> Thanks.
> >>
>
>

[GitHub] drill pull request: DRILL-4134: Allocator updates

[adeneche]

Github user adeneche commented on a diff in the pull request:

https://github.com/apache/drill/pull/283#discussion_r46348640
  
--- Diff: 
exec/memory/base/src/main/java/io/netty/buffer/UnsafeDirectLittleEndian.java ---
@@ -89,156 +131,139 @@ public ByteBuf order(ByteOrder endianness) {
   }
 
   @Override
-public double getDouble(int index) {
-return Double.longBitsToDouble(getLong(index));
-}
-
-@Override
-public char getChar(int index) {
-return (char) getShort(index);
-}
-
-@Override
-public long getUnsignedInt(int index) {
-return getInt(index) & 0xL;
-}
+  public double getDouble(int index) {
+return Double.longBitsToDouble(getLong(index));
+  }
 
-@Override
-public int getInt(int index) {
-//wrapped.checkIndex(index, 4);
-int v = PlatformDependent.getInt(addr(index));
-return v;
-}
+  @Override
+  public char getChar(int index) {
+return (char) getShort(index);
+  }
 
-@Override
-public int getUnsignedShort(int index) {
-return getShort(index) & 0x;
-}
+  @Override
+  public long getUnsignedInt(int index) {
+return getInt(index) & 0xL;
+  }
 
-@Override
-public short getShort(int index) {
-//wrapped.checkIndex(index, 2);
-short v = PlatformDependent.getShort(addr(index));
-return v;
-}
+  @Override
+  public int getInt(int index) {
+int v = PlatformDependent.getInt(addr(index));
+return v;
+  }
 
-@Override
-public ByteBuf setShort(int index, int value) {
-wrapped.checkIndex(index, 2);
-_setShort(index, value);
-return this;
-}
+  @Override
+  public int getUnsignedShort(int index) {
+return getShort(index) & 0x;
+  }
 
-@Override
-public ByteBuf setInt(int index, int value) {
-wrapped.checkIndex(index, 4);
-_setInt(index, value);
-return this;
-}
+  @Override
+  public short getShort(int index) {
+short v = PlatformDependent.getShort(addr(index));
+return v;
+  }
 
-@Override
-public ByteBuf setLong(int index, long value) {
-wrapped.checkIndex(index, 8);
-_setLong(index, value);
-return this;
-}
+  @Override
+  public ByteBuf setShort(int index, int value) {
+wrapped.checkIndex(index, 2);
+_setShort(index, value);
+return this;
+  }
 
-@Override
-public ByteBuf setChar(int index, int value) {
-setShort(index, value);
-return this;
-}
+  @Override
+  public ByteBuf setInt(int index, int value) {
+wrapped.checkIndex(index, 4);
+_setInt(index, value);
+return this;
+  }
 
-@Override
-public ByteBuf setFloat(int index, float value) {
-setInt(index, Float.floatToRawIntBits(value));
-return this;
-}
+  @Override
+  public ByteBuf setLong(int index, long value) {
+wrapped.checkIndex(index, 8);
+_setLong(index, value);
+return this;
+  }
 
-@Override
-public ByteBuf setDouble(int index, double value) {
-setLong(index, Double.doubleToRawLongBits(value));
-return this;
-}
+  @Override
+  public ByteBuf setChar(int index, int value) {
+setShort(index, value);
+return this;
+  }
 
-@Override
-public ByteBuf writeShort(int value) {
-wrapped.ensureWritable(2);
-_setShort(wrapped.writerIndex, value);
-wrapped.writerIndex += 2;
-return this;
-}
+  @Override
+  public ByteBuf setFloat(int index, float value) {
+setInt(index, Float.floatToRawIntBits(value));
+return this;
+  }
 
-@Override
-public ByteBuf writeInt(int value) {
-wrapped.ensureWritable(4);
-_setInt(wrapped.writerIndex, value);
-wrapped.writerIndex += 4;
-return this;
-}
+  @Override
+  public ByteBuf setDouble(int index, double value) {
+setLong(index, Double.doubleToRawLongBits(value));
+return this;
+  }
 
-@Override
-public ByteBuf writeLong(long value) {
-wrapped.ensureWritable(8);
-_setLong(wrapped.writerIndex, value);
-wrapped.writerIndex += 8;
-return this;
-}
+  @Override
+  public ByteBuf writeShort(int value) {
+wrapped.ensureWritable(2);
+_setShort(wrapped.writerIndex, value);
+wrapped.writerIndex += 2;
+

[jira] [Created] (DRILL-4150) AssertionError: no provider found (rel=rel#16:Subset#0.ENUMERABLE.ANY([]).[], m=interface org.apache.calcite.rel.metadata.BuiltInMetadata$Collation); a backstop provider

[Khurram Faraaz (JIRA)]

Khurram Faraaz created DRILL-4150:
-

 Summary: AssertionError: no provider found 
(rel=rel#16:Subset#0.ENUMERABLE.ANY([]).[], m=interface 
org.apache.calcite.rel.metadata.BuiltInMetadata$Collation); a backstop provider 
is recommended
 Key: DRILL-4150
 URL: https://issues.apache.org/jira/browse/DRILL-4150
 Project: Apache Drill
  Issue Type: Bug
  Components: Execution - Relational Operators
Affects Versions: 1.4.0
Reporter: Khurram Faraaz


Test fails with AssertionError on Drill 1.4 with assertions enabled and JDK8, 
on 4 node cluster. Test was executed as part of functional test run on 4 node 
cluster.
Assertion seems to be coming from calcite.

Failing test case : Functional/decimal/decimal101.q
{code}
query => select cast('0.0001' as decimal(9,9)) / 
cast('-' as decimal(28,0)) from data limit 1;
{code}

{noformat}
git.commit.id=ff76078b61a54001da1de3dbd95d01004da4b791
[root@centos-01 drill-output]# java -version
openjdk version "1.8.0_65"
{noformat}

Stack trace from drillbit.log
{noformat}
2015-12-01 03:31:50,248 [29a2eb5a-785d-4e55-6148-b4fd5a5cfc9c:foreman] ERROR 
o.a.drill.exec.work.foreman.Foreman - SYSTEM ERROR: AssertionError: no provider 
found (rel=rel#16:Subset#0.ENUMERABLE.ANY([]).[], m=interface 
org.apache.calcite.rel.metadata.BuiltInMetadata$Collation); a backstop provider 
is recommended


[Error Id: 116e2b9c-e247-44c2-9490-ee6fd22736cc on centos-02.qa.lab:31010]
org.apache.drill.common.exceptions.UserException: SYSTEM ERROR: AssertionError: 
no provider found (rel=rel#16:Subset#0.ENUMERABLE.ANY([]).[], m=interface 
org.apache.calcite.rel.metadata.BuiltInMetadata$Collation); a backstop provider 
is recommended


[Error Id: 116e2b9c-e247-44c2-9490-ee6fd22736cc on centos-02.qa.lab:31010]
at 
org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:534)
 ~[drill-common-1.4.0-SNAPSHOT.jar:1.4.0-SNAPSHOT]
at 
org.apache.drill.exec.work.foreman.Foreman$ForemanResult.close(Foreman.java:742)
 [drill-java-exec-1.4.0-SNAPSHOT.jar:1.4.0-SNAPSHOT]
at 
org.apache.drill.exec.work.foreman.Foreman$StateSwitch.processEvent(Foreman.java:841)
 [drill-java-exec-1.4.0-SNAPSHOT.jar:1.4.0-SNAPSHOT]
at 
org.apache.drill.exec.work.foreman.Foreman$StateSwitch.processEvent(Foreman.java:786)
 [drill-java-exec-1.4.0-SNAPSHOT.jar:1.4.0-SNAPSHOT]
at 
org.apache.drill.common.EventProcessor.sendEvent(EventProcessor.java:73) 
[drill-common-1.4.0-SNAPSHOT.jar:1.4.0-SNAPSHOT]
at 
org.apache.drill.exec.work.foreman.Foreman$StateSwitch.moveToState(Foreman.java:788)
 [drill-java-exec-1.4.0-SNAPSHOT.jar:1.4.0-SNAPSHOT]
at 
org.apache.drill.exec.work.foreman.Foreman.moveToState(Foreman.java:894) 
[drill-java-exec-1.4.0-SNAPSHOT.jar:1.4.0-SNAPSHOT]
at org.apache.drill.exec.work.foreman.Foreman.run(Foreman.java:255) 
[drill-java-exec-1.4.0-SNAPSHOT.jar:1.4.0-SNAPSHOT]
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 
[na:1.8.0_65]
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
[na:1.8.0_65]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_65]
Caused by: org.apache.drill.exec.work.foreman.ForemanException: Unexpected 
exception during fragment initialization: Internal error: Error while applying 
rule ReduceExpressionsRule_Project, args 
[rel#17:LogicalProject.NONE.ANY([]).[](input=rel#16:Subset#0.ENUMERABLE.ANY([]).[],EXPR$0=/(CAST('0.0001'):DECIMAL(9,
 9) NOT NULL, CAST('-'):DECIMAL(28, 0) NOT NULL))]
... 4 common frames omitted
Caused by: java.lang.AssertionError: Internal error: Error while applying rule 
ReduceExpressionsRule_Project, args 
[rel#17:LogicalProject.NONE.ANY([]).[](input=rel#16:Subset#0.ENUMERABLE.ANY([]).[],EXPR$0=/(CAST('0.0001'):DECIMAL(9,
 9) NOT NULL, CAST('-'):DECIMAL(28, 0) NOT NULL))]
at org.apache.calcite.util.Util.newInternal(Util.java:792) 
~[calcite-core-1.4.0-drill-r9.jar:1.4.0-drill-r9]
at 
org.apache.calcite.plan.volcano.VolcanoRuleCall.onMatch(VolcanoRuleCall.java:251)
 ~[calcite-core-1.4.0-drill-r9.jar:1.4.0-drill-r9]
at 
org.apache.calcite.plan.volcano.VolcanoPlanner.findBestExp(VolcanoPlanner.java:808)
 ~[calcite-core-1.4.0-drill-r9.jar:1.4.0-drill-r9]
at 
org.apache.calcite.tools.Programs$RuleSetProgram.run(Programs.java:303) 
~[calcite-core-1.4.0-drill-r9.jar:1.4.0-drill-r9]
at 
org.apache.calcite.prepare.PlannerImpl.transform(PlannerImpl.java:313) 
~[calcite-core-1.4.0-drill-r9.jar:1.4.0-drill-r9]
at 
org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.doLogicalPlanning(DefaultSqlHandler.java:542)
 ~[drill-java-exec-1.4.0-SNAPSHOT.jar:1.4.0-SNAPSHOT]
at 
org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.co

[GitHub] drill pull request: DRILL-4134: Allocator updates

[adeneche]

Github user adeneche commented on a diff in the pull request:

https://github.com/apache/drill/pull/283#discussion_r46355284
  
--- Diff: 
exec/memory/base/src/main/java/org/apache/drill/exec/memory/RootAllocator.java 
---
@@ -0,0 +1,39 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.memory;
+
+import com.google.common.annotations.VisibleForTesting;
+
+/**
+ * The root allocator for using direct memory inside a Drillbit. Supports 
creating a
+ * tree of descendant child allocators.
+ */
+public class RootAllocator extends BaseAllocator {
+
+  public RootAllocator(final long reservation, final long limit) {
--- End diff --

`Accountant1 makes sure that root allocator doesn't have any reservation. 
Why not remove `reservation` parameter from this constructor, and just pass 0 
to parent ?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request: DRILL-4134: Allocator updates

[adeneche]

Github user adeneche commented on a diff in the pull request:

https://github.com/apache/drill/pull/283#discussion_r46355580
  
--- Diff: 
exec/memory/base/src/main/java/org/apache/drill/exec/memory/BaseAllocator.java 
---
@@ -0,0 +1,689 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.memory;
+
+import io.netty.buffer.ByteBufAllocator;
+import io.netty.buffer.DrillBuf;
+import io.netty.buffer.UnsafeDirectLittleEndian;
+
+import java.util.Arrays;
+import java.util.IdentityHashMap;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.concurrent.atomic.AtomicLong;
+
+import org.apache.drill.common.HistoricalLog;
+import org.apache.drill.exec.exception.OutOfMemoryException;
+import org.apache.drill.exec.memory.AllocatorManager.BufferLedger;
+import org.apache.drill.exec.ops.BufferManager;
+import org.apache.drill.exec.util.AssertionUtil;
+
+import com.google.common.base.Preconditions;
+
+public abstract class BaseAllocator extends Accountant implements 
BufferAllocator {
+  private static final org.slf4j.Logger logger = 
org.slf4j.LoggerFactory.getLogger(BaseAllocator.class);
+
+  public static final String DEBUG_ALLOCATOR = 
"drill.memory.debug.allocator";
+
+  private static final AtomicLong ID_GENERATOR = new AtomicLong(0);
+  private static final int CHUNK_SIZE = 
AllocatorManager.INNER_ALLOCATOR.getChunkSize();
+
+  public static final int DEBUG_LOG_LENGTH = 6;
+  public static final boolean DEBUG = AssertionUtil.isAssertionsEnabled()
+  || Boolean.parseBoolean(System.getProperty(DEBUG_ALLOCATOR, 
"false"));
+  private final Object DEBUG_LOCK = DEBUG ? new Object() : null;
+
+  private final BaseAllocator parentAllocator;
+  private final ByteBufAllocator thisAsByteBufAllocator;
+  private final IdentityHashMap childAllocators;
+  private final DrillBuf empty;
+
+  private volatile boolean isClosed = false; // the allocator has been 
closed
+
+  // Package exposed for sharing between AllocatorManger and BaseAllocator 
objects
+  final long id = ID_GENERATOR.incrementAndGet(); // unique ID assigned to 
each allocator
+  final String name;
+  final RootAllocator root;
+
+  // members used purely for debugging
+  private final IdentityHashMap childLedgers;
+  private final IdentityHashMap reservations;
+  private final HistoricalLog historicalLog;
+
+  protected BaseAllocator(
+  final BaseAllocator parentAllocator,
+  final String name,
+  final long initReservation,
+  final long maxAllocation) throws OutOfMemoryException {
+super(parentAllocator, initReservation, maxAllocation);
+
+if (parentAllocator != null) {
+  this.root = parentAllocator.root;
+  empty = parentAllocator.empty;
+} else if (this instanceof RootAllocator) {
+  this.root = (RootAllocator) this;
+  empty = createEmpty();
+} else {
+  throw new IllegalStateException("An parent allocator must either 
carry a root or be the root.");
+}
+
+this.parentAllocator = parentAllocator;
+this.name = name;
+
+// TODO: DRILL-4131
+// this.thisAsByteBufAllocator = new DrillByteBufAllocator(this);
+this.thisAsByteBufAllocator = 
AllocatorManager.INNER_ALLOCATOR.allocator;
+
+if (DEBUG) {
+  childAllocators = new IdentityHashMap<>();
+  reservations = new IdentityHashMap<>();
+  childLedgers = new IdentityHashMap<>();
+  historicalLog = new HistoricalLog(DEBUG_LOG_LENGTH, "allocator[%d]", 
id);
+  hist("created by \"%s\", owned = %d", name.toString(), 
this.getAllocatedMemory());
--- End diff --

no need to call `name.toString()`


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feat

[GitHub] drill pull request: DRILL-4134: Allocator updates

[adeneche]

Github user adeneche commented on a diff in the pull request:

https://github.com/apache/drill/pull/283#discussion_r46355829
  
--- Diff: 
exec/memory/base/src/main/java/org/apache/drill/exec/memory/BaseAllocator.java 
---
@@ -0,0 +1,689 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.memory;
+
+import io.netty.buffer.ByteBufAllocator;
+import io.netty.buffer.DrillBuf;
+import io.netty.buffer.UnsafeDirectLittleEndian;
+
+import java.util.Arrays;
+import java.util.IdentityHashMap;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.concurrent.atomic.AtomicLong;
+
+import org.apache.drill.common.HistoricalLog;
+import org.apache.drill.exec.exception.OutOfMemoryException;
+import org.apache.drill.exec.memory.AllocatorManager.BufferLedger;
+import org.apache.drill.exec.ops.BufferManager;
+import org.apache.drill.exec.util.AssertionUtil;
+
+import com.google.common.base.Preconditions;
+
+public abstract class BaseAllocator extends Accountant implements 
BufferAllocator {
+  private static final org.slf4j.Logger logger = 
org.slf4j.LoggerFactory.getLogger(BaseAllocator.class);
+
+  public static final String DEBUG_ALLOCATOR = 
"drill.memory.debug.allocator";
+
+  private static final AtomicLong ID_GENERATOR = new AtomicLong(0);
+  private static final int CHUNK_SIZE = 
AllocatorManager.INNER_ALLOCATOR.getChunkSize();
+
+  public static final int DEBUG_LOG_LENGTH = 6;
+  public static final boolean DEBUG = AssertionUtil.isAssertionsEnabled()
+  || Boolean.parseBoolean(System.getProperty(DEBUG_ALLOCATOR, 
"false"));
+  private final Object DEBUG_LOCK = DEBUG ? new Object() : null;
+
+  private final BaseAllocator parentAllocator;
+  private final ByteBufAllocator thisAsByteBufAllocator;
+  private final IdentityHashMap childAllocators;
+  private final DrillBuf empty;
+
+  private volatile boolean isClosed = false; // the allocator has been 
closed
+
+  // Package exposed for sharing between AllocatorManger and BaseAllocator 
objects
+  final long id = ID_GENERATOR.incrementAndGet(); // unique ID assigned to 
each allocator
+  final String name;
+  final RootAllocator root;
+
+  // members used purely for debugging
+  private final IdentityHashMap childLedgers;
+  private final IdentityHashMap reservations;
+  private final HistoricalLog historicalLog;
+
+  protected BaseAllocator(
+  final BaseAllocator parentAllocator,
+  final String name,
+  final long initReservation,
+  final long maxAllocation) throws OutOfMemoryException {
+super(parentAllocator, initReservation, maxAllocation);
+
+if (parentAllocator != null) {
+  this.root = parentAllocator.root;
+  empty = parentAllocator.empty;
+} else if (this instanceof RootAllocator) {
+  this.root = (RootAllocator) this;
+  empty = createEmpty();
+} else {
+  throw new IllegalStateException("An parent allocator must either 
carry a root or be the root.");
+}
+
+this.parentAllocator = parentAllocator;
+this.name = name;
+
+// TODO: DRILL-4131
+// this.thisAsByteBufAllocator = new DrillByteBufAllocator(this);
+this.thisAsByteBufAllocator = 
AllocatorManager.INNER_ALLOCATOR.allocator;
+
+if (DEBUG) {
+  childAllocators = new IdentityHashMap<>();
+  reservations = new IdentityHashMap<>();
+  childLedgers = new IdentityHashMap<>();
+  historicalLog = new HistoricalLog(DEBUG_LOG_LENGTH, "allocator[%d]", 
id);
+  hist("created by \"%s\", owned = %d", name.toString(), 
this.getAllocatedMemory());
+} else {
+  childAllocators = null;
+  reservations = null;
+  historicalLog = null;
+  childLedgers = null;
+}
+
+  }
+
+  @Override
+  public String getName() {
+return name;
   

[GitHub] drill pull request: DRILL-4134: Allocator updates

[adeneche]

Github user adeneche commented on a diff in the pull request:

https://github.com/apache/drill/pull/283#discussion_r46355906
  
--- Diff: 
exec/memory/base/src/main/java/org/apache/drill/exec/memory/BaseAllocator.java 
---
@@ -0,0 +1,689 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.memory;
+
+import io.netty.buffer.ByteBufAllocator;
+import io.netty.buffer.DrillBuf;
+import io.netty.buffer.UnsafeDirectLittleEndian;
+
+import java.util.Arrays;
+import java.util.IdentityHashMap;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.concurrent.atomic.AtomicLong;
+
+import org.apache.drill.common.HistoricalLog;
+import org.apache.drill.exec.exception.OutOfMemoryException;
+import org.apache.drill.exec.memory.AllocatorManager.BufferLedger;
+import org.apache.drill.exec.ops.BufferManager;
+import org.apache.drill.exec.util.AssertionUtil;
+
+import com.google.common.base.Preconditions;
+
+public abstract class BaseAllocator extends Accountant implements 
BufferAllocator {
+  private static final org.slf4j.Logger logger = 
org.slf4j.LoggerFactory.getLogger(BaseAllocator.class);
+
+  public static final String DEBUG_ALLOCATOR = 
"drill.memory.debug.allocator";
+
+  private static final AtomicLong ID_GENERATOR = new AtomicLong(0);
+  private static final int CHUNK_SIZE = 
AllocatorManager.INNER_ALLOCATOR.getChunkSize();
+
+  public static final int DEBUG_LOG_LENGTH = 6;
+  public static final boolean DEBUG = AssertionUtil.isAssertionsEnabled()
+  || Boolean.parseBoolean(System.getProperty(DEBUG_ALLOCATOR, 
"false"));
+  private final Object DEBUG_LOCK = DEBUG ? new Object() : null;
+
+  private final BaseAllocator parentAllocator;
+  private final ByteBufAllocator thisAsByteBufAllocator;
+  private final IdentityHashMap childAllocators;
+  private final DrillBuf empty;
+
+  private volatile boolean isClosed = false; // the allocator has been 
closed
+
+  // Package exposed for sharing between AllocatorManger and BaseAllocator 
objects
+  final long id = ID_GENERATOR.incrementAndGet(); // unique ID assigned to 
each allocator
+  final String name;
+  final RootAllocator root;
+
+  // members used purely for debugging
+  private final IdentityHashMap childLedgers;
+  private final IdentityHashMap reservations;
+  private final HistoricalLog historicalLog;
+
+  protected BaseAllocator(
+  final BaseAllocator parentAllocator,
+  final String name,
+  final long initReservation,
+  final long maxAllocation) throws OutOfMemoryException {
+super(parentAllocator, initReservation, maxAllocation);
+
+if (parentAllocator != null) {
+  this.root = parentAllocator.root;
+  empty = parentAllocator.empty;
+} else if (this instanceof RootAllocator) {
+  this.root = (RootAllocator) this;
+  empty = createEmpty();
+} else {
+  throw new IllegalStateException("An parent allocator must either 
carry a root or be the root.");
+}
+
+this.parentAllocator = parentAllocator;
+this.name = name;
+
+// TODO: DRILL-4131
+// this.thisAsByteBufAllocator = new DrillByteBufAllocator(this);
+this.thisAsByteBufAllocator = 
AllocatorManager.INNER_ALLOCATOR.allocator;
+
+if (DEBUG) {
+  childAllocators = new IdentityHashMap<>();
+  reservations = new IdentityHashMap<>();
+  childLedgers = new IdentityHashMap<>();
+  historicalLog = new HistoricalLog(DEBUG_LOG_LENGTH, "allocator[%d]", 
id);
+  hist("created by \"%s\", owned = %d", name.toString(), 
this.getAllocatedMemory());
+} else {
+  childAllocators = null;
+  reservations = null;
+  historicalLog = null;
+  childLedgers = null;
+}
+
+  }
+
+  @Override
+  public String getName() {
+return name;
   

[GitHub] drill pull request: DRILL-4134: Allocator updates

[adeneche]

Github user adeneche commented on a diff in the pull request:

https://github.com/apache/drill/pull/283#discussion_r46356005
  
--- Diff: 
exec/memory/base/src/main/java/org/apache/drill/exec/memory/BaseAllocator.java 
---
@@ -0,0 +1,689 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.memory;
+
+import io.netty.buffer.ByteBufAllocator;
+import io.netty.buffer.DrillBuf;
+import io.netty.buffer.UnsafeDirectLittleEndian;
+
+import java.util.Arrays;
+import java.util.IdentityHashMap;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.concurrent.atomic.AtomicLong;
+
+import org.apache.drill.common.HistoricalLog;
+import org.apache.drill.exec.exception.OutOfMemoryException;
+import org.apache.drill.exec.memory.AllocatorManager.BufferLedger;
+import org.apache.drill.exec.ops.BufferManager;
+import org.apache.drill.exec.util.AssertionUtil;
+
+import com.google.common.base.Preconditions;
+
+public abstract class BaseAllocator extends Accountant implements 
BufferAllocator {
+  private static final org.slf4j.Logger logger = 
org.slf4j.LoggerFactory.getLogger(BaseAllocator.class);
+
+  public static final String DEBUG_ALLOCATOR = 
"drill.memory.debug.allocator";
+
+  private static final AtomicLong ID_GENERATOR = new AtomicLong(0);
+  private static final int CHUNK_SIZE = 
AllocatorManager.INNER_ALLOCATOR.getChunkSize();
+
+  public static final int DEBUG_LOG_LENGTH = 6;
+  public static final boolean DEBUG = AssertionUtil.isAssertionsEnabled()
+  || Boolean.parseBoolean(System.getProperty(DEBUG_ALLOCATOR, 
"false"));
+  private final Object DEBUG_LOCK = DEBUG ? new Object() : null;
+
+  private final BaseAllocator parentAllocator;
+  private final ByteBufAllocator thisAsByteBufAllocator;
+  private final IdentityHashMap childAllocators;
+  private final DrillBuf empty;
+
+  private volatile boolean isClosed = false; // the allocator has been 
closed
+
+  // Package exposed for sharing between AllocatorManger and BaseAllocator 
objects
+  final long id = ID_GENERATOR.incrementAndGet(); // unique ID assigned to 
each allocator
+  final String name;
+  final RootAllocator root;
+
+  // members used purely for debugging
+  private final IdentityHashMap childLedgers;
+  private final IdentityHashMap reservations;
+  private final HistoricalLog historicalLog;
+
+  protected BaseAllocator(
+  final BaseAllocator parentAllocator,
+  final String name,
+  final long initReservation,
+  final long maxAllocation) throws OutOfMemoryException {
+super(parentAllocator, initReservation, maxAllocation);
+
+if (parentAllocator != null) {
+  this.root = parentAllocator.root;
+  empty = parentAllocator.empty;
+} else if (this instanceof RootAllocator) {
+  this.root = (RootAllocator) this;
+  empty = createEmpty();
+} else {
+  throw new IllegalStateException("An parent allocator must either 
carry a root or be the root.");
+}
+
+this.parentAllocator = parentAllocator;
+this.name = name;
+
+// TODO: DRILL-4131
+// this.thisAsByteBufAllocator = new DrillByteBufAllocator(this);
+this.thisAsByteBufAllocator = 
AllocatorManager.INNER_ALLOCATOR.allocator;
+
+if (DEBUG) {
+  childAllocators = new IdentityHashMap<>();
+  reservations = new IdentityHashMap<>();
+  childLedgers = new IdentityHashMap<>();
+  historicalLog = new HistoricalLog(DEBUG_LOG_LENGTH, "allocator[%d]", 
id);
+  hist("created by \"%s\", owned = %d", name.toString(), 
this.getAllocatedMemory());
+} else {
+  childAllocators = null;
+  reservations = null;
+  historicalLog = null;
+  childLedgers = null;
+}
+
+  }
+
+  @Override
+  public String getName() {
+return name;
   

[GitHub] drill pull request: DRILL-4134: Allocator updates

[adeneche]

Github user adeneche commented on a diff in the pull request:

https://github.com/apache/drill/pull/283#discussion_r46356295
  
--- Diff: 
exec/memory/base/src/main/java/org/apache/drill/exec/memory/BaseAllocator.java 
---
@@ -0,0 +1,689 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.memory;
+
+import io.netty.buffer.ByteBufAllocator;
+import io.netty.buffer.DrillBuf;
+import io.netty.buffer.UnsafeDirectLittleEndian;
+
+import java.util.Arrays;
+import java.util.IdentityHashMap;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.concurrent.atomic.AtomicLong;
+
+import org.apache.drill.common.HistoricalLog;
+import org.apache.drill.exec.exception.OutOfMemoryException;
+import org.apache.drill.exec.memory.AllocatorManager.BufferLedger;
+import org.apache.drill.exec.ops.BufferManager;
+import org.apache.drill.exec.util.AssertionUtil;
+
+import com.google.common.base.Preconditions;
+
+public abstract class BaseAllocator extends Accountant implements 
BufferAllocator {
+  private static final org.slf4j.Logger logger = 
org.slf4j.LoggerFactory.getLogger(BaseAllocator.class);
+
+  public static final String DEBUG_ALLOCATOR = 
"drill.memory.debug.allocator";
+
+  private static final AtomicLong ID_GENERATOR = new AtomicLong(0);
+  private static final int CHUNK_SIZE = 
AllocatorManager.INNER_ALLOCATOR.getChunkSize();
+
+  public static final int DEBUG_LOG_LENGTH = 6;
+  public static final boolean DEBUG = AssertionUtil.isAssertionsEnabled()
+  || Boolean.parseBoolean(System.getProperty(DEBUG_ALLOCATOR, 
"false"));
+  private final Object DEBUG_LOCK = DEBUG ? new Object() : null;
+
+  private final BaseAllocator parentAllocator;
+  private final ByteBufAllocator thisAsByteBufAllocator;
+  private final IdentityHashMap childAllocators;
+  private final DrillBuf empty;
+
+  private volatile boolean isClosed = false; // the allocator has been 
closed
+
+  // Package exposed for sharing between AllocatorManger and BaseAllocator 
objects
+  final long id = ID_GENERATOR.incrementAndGet(); // unique ID assigned to 
each allocator
+  final String name;
+  final RootAllocator root;
+
+  // members used purely for debugging
+  private final IdentityHashMap childLedgers;
+  private final IdentityHashMap reservations;
+  private final HistoricalLog historicalLog;
+
+  protected BaseAllocator(
+  final BaseAllocator parentAllocator,
+  final String name,
+  final long initReservation,
+  final long maxAllocation) throws OutOfMemoryException {
+super(parentAllocator, initReservation, maxAllocation);
+
+if (parentAllocator != null) {
+  this.root = parentAllocator.root;
+  empty = parentAllocator.empty;
+} else if (this instanceof RootAllocator) {
+  this.root = (RootAllocator) this;
+  empty = createEmpty();
+} else {
+  throw new IllegalStateException("An parent allocator must either 
carry a root or be the root.");
+}
+
+this.parentAllocator = parentAllocator;
+this.name = name;
+
+// TODO: DRILL-4131
+// this.thisAsByteBufAllocator = new DrillByteBufAllocator(this);
+this.thisAsByteBufAllocator = 
AllocatorManager.INNER_ALLOCATOR.allocator;
+
+if (DEBUG) {
+  childAllocators = new IdentityHashMap<>();
+  reservations = new IdentityHashMap<>();
+  childLedgers = new IdentityHashMap<>();
+  historicalLog = new HistoricalLog(DEBUG_LOG_LENGTH, "allocator[%d]", 
id);
+  hist("created by \"%s\", owned = %d", name.toString(), 
this.getAllocatedMemory());
+} else {
+  childAllocators = null;
+  reservations = null;
+  historicalLog = null;
+  childLedgers = null;
+}
+
+  }
+
+  @Override
+  public String getName() {
+return name;
   

Re: Can we pass the #skipped records with RecordBatch?

[Parth Chandra]

+1 on having a framework.
OTOH, as with the warnings implementation, we might want to go ahead with a
simpler implementation while we get a more generic framework design in
place.

Jacques, do you have any preliminary thoughts on the framework?

On Tue, Dec 1, 2015 at 2:08 PM, Julian Hyde  wrote:

> +1 for a sideband mechanism.
>
> Sideband can also allow correlated restart of sub-queries.
>
> In sideband use cases you described, the messages ran in the opposite
> direction to the data. Would the sideband also run in the same direction as
> the data? If so it could carry warnings, rejected rows, progress
> indications, and (for online aggregation[1]) notifications that a better
> approximate query result is available.
>
> Julian
>
> [1] https://en.wikipedia.org/wiki/Online_aggregation
>
>
>
> > On Dec 1, 2015, at 1:51 PM, Jacques Nadeau  wrote:
> >
> > This seems like a form of sideband communication. I think we should have
> a
> > framework for this type of thing in general rather than a one-off for
> this
> > particular need. Other forms of sideband might be small table bloomfilter
> > generation and pushdown into hbase, separate file assignment/partitioning
> > providers balancing/generating scanner workloads, statistics generation
> for
> > adaptive execution, etc.
> >
> > --
> > Jacques Nadeau
> > CTO and Co-Founder, Dremio
> >
> > On Tue, Dec 1, 2015 at 11:35 AM, Hsuan Yi Chu 
> wrote:
> >
> >> I am trying to deal with the following scenario:
> >>
> >> A bunch of minor fragments are doing things in parallel. Each of them
> could
> >> skip some records. Since the downstream minor fragment needs to know the
> >> sum of skipped-record-counts (in order to just display or see if the
> number
> >> exceeds the threshold) in the upstreams, each upstream minor fragment
> needs
> >> to pass this scalar with RecordBatch.
> >>
> >> Since this seems impacting the protocol of RecordBatch, I am looking for
> >> some advice here.
> >>
> >> Thanks.
> >>
>
>

Re: Create 1.4 Release branch soon?

[Venki Korukanti]

I can manage the release.

Here are the pending JIRAs so far:

1) DRILL-4053

Let me know if you have any pending patches and want to get them into 1.4.

Thanks
Venki

On Tue, Dec 1, 2015 at 10:43 AM, Parth Chandra  wrote:

> +1 on creating the release branch today.
> I'd like to get DRILL-4053 in. Patch is ready and doing one more round of
> regression tests.
>
>
>
> On Tue, Dec 1, 2015 at 9:56 AM, Jacques Nadeau  wrote:
>
> > It is December (already!). Seems like we should create a 1.4 release
> branch
> > soon so we can get a vote going. Does someone want to volunteer as the
> > release manager?
> >
> > thanks,
> > Jacques
> >
> > --
> > Jacques Nadeau
> > CTO and Co-Founder, Dremio
> >
>

Re: Create 1.4 Release branch soon?

[Abdel Hakim Deneche]

Venki, here is a link to an updated version of the release instructions:

https://gist.github.com/adeneche/75ae06a8bb4b7ce24cc6

It worked for me last time I tried.

On Tue, Dec 1, 2015 at 3:59 PM, Venki Korukanti 
wrote:

> I can manage the release.
>
> Here are the pending JIRAs so far:
>
> 1) DRILL-4053
>
> Let me know if you have any pending patches and want to get them into 1.4.
>
> Thanks
> Venki
>
> On Tue, Dec 1, 2015 at 10:43 AM, Parth Chandra  wrote:
>
> > +1 on creating the release branch today.
> > I'd like to get DRILL-4053 in. Patch is ready and doing one more round of
> > regression tests.
> >
> >
> >
> > On Tue, Dec 1, 2015 at 9:56 AM, Jacques Nadeau 
> wrote:
> >
> > > It is December (already!). Seems like we should create a 1.4 release
> > branch
> > > soon so we can get a vote going. Does someone want to volunteer as the
> > > release manager?
> > >
> > > thanks,
> > > Jacques
> > >
> > > --
> > > Jacques Nadeau
> > > CTO and Co-Founder, Dremio
> > >
> >
>



-- 

Abdelhakim Deneche

Software Engineer

  


Now Available - Free Hadoop On-Demand Training

Re: Naming the new ValueVector Initiative

[Ted Dunning]

Apache can handle this if we set the groundwork in place.

Also, Twitter's lawyers work for Twitter, not for Apache. As such, their
opinions can't be taken by Apache as legal advice.  There are issues of
privilege, conflict of interest and so on.



On Wed, Dec 2, 2015 at 7:51 AM, Alex Levenson 
wrote:

> I can ask about whether Twitter's lawyers can help out -- is that
> something we need? Or is that something apache helps out with in the next
> step?
>
> On Mon, Nov 30, 2015 at 9:32 PM, Julian Hyde  wrote:
>
>> +1 to have a vote tomorrow.
>>
>> Assuming that Vector is out of play, I just did a quick search for the
>> top 4 remaining, (“arrow”, “honeycomb”, “herringbone”, “joist"), at
>> sourceforge, open hub, trademarkia, and on google. There are no trademarks
>> for these in similar subject areas. There is a moderately active project
>> called “joist” [1].
>>
>> I will point out that “Apache Arrow” has native-american connotations
>> that we may or may not want to live with (just ask the Washington Redskins
>> how they feel about their name).
>>
>> If someone would like to vet other names, use the links on
>> https://issues.apache.org/jira/browse/PODLINGNAMESEARCH-90, and fill out
>> column C in the spreadsheet.
>>
>> Julian
>>
>> [1] https://github.com/stephenh/joist
>>
>>
>> On Nov 30, 2015, at 7:01 PM, Jacques Nadeau  wrote:
>>
>> +1
>>
>> --
>> Jacques Nadeau
>> CTO and Co-Founder, Dremio
>>
>> On Mon, Nov 30, 2015 at 6:34 PM, Wes McKinney  wrote:
>>
>> Should we have a last call for votes, closing EOD tomorrow (Tuesday)? I
>> missed this for a few days last week with holiday travel.
>>
>> On Thu, Nov 26, 2015 at 3:04 PM, Julian Hyde 
>> wrote:
>>
>> Consulting a lawyer is part of the Apache branding process but the first
>> stage is to gather a list of potential conflicts -
>> https://issues.apache.org/jira/browse/PODLINGNAMESEARCH-90 is an example.
>>
>> The other part, frankly, is to pick your battles.
>>
>> A year or so ago Actian re-branded Vectorwise as Vector.
>> http://www.zdnet.com/article/actian-consolidates-its-analytics-portfolio/
>> .
>> Given that it is an analytic database in the Hadoop space I think that is
>> as close to a “direct hit” as it gets. I don’t think we need a lawyer to
>> tell us that. Certainly it makes sense to look for conflicts for the other
>> alternatives before consulting lawyers.
>>
>> Julian
>>
>>
>>
>>
>> On Nov 25, 2015, at 9:42 PM, Marcel Kornacker 
>> wrote:
>>
>>
>>
>> On Tue, Nov 24, 2015 at 3:25 PM, Jacques Nadeau 
>> wrote:
>>
>> Ok guys,
>>
>> I don't think anyone is doing a thorough analysis of viaability. I did a
>> quick glance and the top one (Vector) seems like it would have an issue
>> with conflict of an Actian product. The may be fine. Let's do a second
>> phase vote.
>>
>>
>> I'm assuming you mean Vectorwise?
>>
>> Before we do anything else, could we have a lawyer look into this? Last
>> time around that I remember (Parquet), Twitter's lawyers did a good job of
>> weeding out the potential trademark violations.
>>
>> Alex, could Twitter get involved this time around as well?
>>
>>
>>
>> Pick your top 3 (1,2,3 with 3 being top preference)
>>
>> Let's get this done by Friday and then we can do a podling name search
>> starting with the top one.
>>
>> Link again:
>>
>>
>>
>> https://docs.google.com/spreadsheets/d/1q6UqluW6SLuMKRwW2TBGBzHfYLlXYm37eKJlIxWQGQM/edit#gid=304381532&vpid=A1
>>
>> thanks
>>
>>
>> --
>> Jacques Nadeau
>> CTO and Co-Founder, Dremio
>>
>> On Fri, Nov 20, 2015 at 9:24 AM, Jacques Nadeau 
>> wrote:
>>
>> Ok, it looks like we have a candidate list (we actually got 11 since
>> there was a three-way tie for ninth place):
>>
>> VectorArrowhoneycombHerringbonejoistV2Pietcolbufbatonimpulsevictor
>> Next we need to do trademark searches on each of these to see whether
>> we're likely to have success. I've moved candidates to a second tab:
>>
>>
>>
>> https://docs.google.com/spreadsheets/d/1q6UqluW6SLuMKRwW2TBGBzHfYLlXYm37eKJlIxWQGQM/edit#gid=304381532
>>
>> Anybody want to give a hand in analyzing potential conflicts?
>>
>>
>>
>> --
>> Jacques Nadeau
>> CTO and Co-Founder, Dremio
>>
>> On Mon, Nov 16, 2015 at 12:10 PM, Jacques Nadeau 
>> wrote:
>>
>> Everybody should pick their ten favorites using the numbers 1 to 10.
>>
>> 10 is most preferred
>>
>>
>>
>> --
>> Jacques Nadeau
>> CTO and Co-Founder, Dremio
>>
>> On Mon, Nov 16, 2015 at 10:17 AM, Ted Dunning 
>> wrote:
>>
>>
>> Single vote for most preferred?
>>
>> Single transferable vote?
>>
>>
>>
>> On Tue, Nov 17, 2015 at 2:50 AM, Jacques Nadeau 
>> wrote:
>>
>> Given that a bunch of people added names to the sheet, I'll take
>> that as tacit agreement to the proposed process.
>>
>> Let's move to the first vote phase. I've added a column for
>> everybody's votes. Let's try to wrap up the vote by 10am on Wednesday.
>>
>> thanks!
>> Jacques
>>
>>
>>
>> --
>> Jacques Nadeau
>> CTO and Co-Founder, Dremio
>>
>> On Thu, Nov 12, 2015 at 12:03 PM, Jacques Nadeau >
>> wrote:
>>
>>
>> Hey 

Re: Create 1.4 Release branch soon?

[Jacques Nadeau]

It seems like we should also try to include:

DRILL-4109
DRILL-4125
DRILL-4111

4111 is ready to merge if someone wants to merge it. It looks like Sudheesh
just needs to commit.

For 4125/4109, it seems like we should get Vicki's feedback if those
changes are good for merge or if we should punt.




--
Jacques Nadeau
CTO and Co-Founder, Dremio

On Tue, Dec 1, 2015 at 3:59 PM, Venki Korukanti 
wrote:

> I can manage the release.
>
> Here are the pending JIRAs so far:
>
> 1) DRILL-4053
>
> Let me know if you have any pending patches and want to get them into 1.4.
>
> Thanks
> Venki
>
> On Tue, Dec 1, 2015 at 10:43 AM, Parth Chandra  wrote:
>
> > +1 on creating the release branch today.
> > I'd like to get DRILL-4053 in. Patch is ready and doing one more round of
> > regression tests.
> >
> >
> >
> > On Tue, Dec 1, 2015 at 9:56 AM, Jacques Nadeau 
> wrote:
> >
> > > It is December (already!). Seems like we should create a 1.4 release
> > branch
> > > soon so we can get a vote going. Does someone want to volunteer as the
> > > release manager?
> > >
> > > thanks,
> > > Jacques
> > >
> > > --
> > > Jacques Nadeau
> > > CTO and Co-Founder, Dremio
> > >
> >
>

Re: Create 1.4 Release branch soon?

[Amit Hadke]

4109 is good to go.
I'm still finishing up 4125, will put it up for review soon.

On Tue, Dec 1, 2015 at 6:22 PM, Jacques Nadeau  wrote:

> It seems like we should also try to include:
>
> DRILL-4109
> DRILL-4125
> DRILL-4111
>
> 4111 is ready to merge if someone wants to merge it. It looks like Sudheesh
> just needs to commit.
>
> For 4125/4109, it seems like we should get Vicki's feedback if those
> changes are good for merge or if we should punt.
>
>
>
>
> --
> Jacques Nadeau
> CTO and Co-Founder, Dremio
>
> On Tue, Dec 1, 2015 at 3:59 PM, Venki Korukanti  >
> wrote:
>
> > I can manage the release.
> >
> > Here are the pending JIRAs so far:
> >
> > 1) DRILL-4053
> >
> > Let me know if you have any pending patches and want to get them into
> 1.4.
> >
> > Thanks
> > Venki
> >
> > On Tue, Dec 1, 2015 at 10:43 AM, Parth Chandra 
> wrote:
> >
> > > +1 on creating the release branch today.
> > > I'd like to get DRILL-4053 in. Patch is ready and doing one more round
> of
> > > regression tests.
> > >
> > >
> > >
> > > On Tue, Dec 1, 2015 at 9:56 AM, Jacques Nadeau 
> > wrote:
> > >
> > > > It is December (already!). Seems like we should create a 1.4 release
> > > branch
> > > > soon so we can get a vote going. Does someone want to volunteer as
> the
> > > > release manager?
> > > >
> > > > thanks,
> > > > Jacques
> > > >
> > > > --
> > > > Jacques Nadeau
> > > > CTO and Co-Founder, Dremio
> > > >
> > >
> >
>

[GitHub] drill pull request: DRILL-4053: Reduce metadata cache file size. S...

[StevenMPhillips]

Github user StevenMPhillips commented on the pull request:

https://github.com/apache/drill/pull/254#issuecomment-161160748
  
+1 LGTM


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request: DRILL-4053: Reduce metadata cache file size. S...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/drill/pull/254


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request: Apache Solr Storage plugin

[sudipmukherjee]

GitHub user sudipmukherjee reopened a pull request:

https://github.com/apache/drill/pull/100

Apache Solr Storage plugin

Basic storage plugin integration of apache solr search engine with
drill.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/sudipmukherjee/drill master

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/drill/pull/100.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #100


commit bebbe8c5ad9134859066fce6e76d952a63e0db88
Author: sudipmukherjee 
Date:   2015-08-03T05:50:17Z

Apache Solr Storage plugin

Basic storage plugin integration of apache solr search engine with
drill.

commit 892ee85fffbbb33aabfc44116c243a32b1c6ca9d
Author: sudipmukherjee 
Date:   2015-08-04T05:43:42Z

Apache Solr Storage plugin

couple of classes for solr schema mapping

commit 26ae6f7b40e552aeb22f107dd4984c683ca60a12
Author: sudipmukherjee 
Date:   2015-08-06T16:14:18Z

Apache solr storage plugin

commit 320983462bd6074d3295584161b3e1493715e4b2
Author: sudipmukherjee 
Date:   2015-08-06T16:18:44Z

Apache solr storage plugin

commit 9b76c12fcb5a72842367372e2788cab9b2825413
Author: sudipmukherjee 
Date:   2015-08-10T04:39:26Z

Apache Solr Storage plugin

Initial code check-ins

commit 232c13b66abe634c851f81c1ec2950747e58f3b1
Author: sudipmukherjee 
Date:   2015-08-10T04:39:56Z

Apache Solr Storage Plugin

Initial code check-in

commit 30ab01ae88a0962313beb6956826f55d9d5ee3c2
Author: sudipmukherjee 
Date:   2015-08-10T10:50:40Z

Apache solr Storage plugin

Initial code check-in

commit ee491c973acc8bee37d79ec9b732dfc3563d05d1
Author: sudipmukherjee 
Date:   2015-08-10T10:52:12Z

Apache Solr Storage

Initial code check-in

commit 39498c905fda8173f6433d4154f151f9ec9b67f5
Author: sudipmukherjee 
Date:   2015-08-10T13:23:10Z

Apache Solr Storage plugin

initial code changes.

commit 9ee6a3bbf9d766f6d9cc8d788d15eb775bd8d1c4
Author: sudipmukherjee 
Date:   2015-08-10T13:23:34Z

Apache Solr Storage plugin

initial code changes

commit 1532706772c2c25ebf1c739ebb8e9e664d00796b
Author: sudipmukherjee 
Date:   2015-08-11T09:14:10Z

Apache Solr Storage plugin

Initial check-in

commit b5f5dacfdb97c43cc0f2f7de65f4c69cb19e2248
Author: sudipmukherjee 
Date:   2015-08-11T09:30:55Z

Apache Solr Storage plugin

commit 04f288d15576207bc9a26971ad9f929b33bb3a61
Author: sudipmukherjee 
Date:   2015-08-27T11:15:35Z

Solr Storage plugin

initial code changes

commit d59b873b1326a6f099e52d6becdc9cdfc0fc
Author: sudipmukherjee 
Date:   2015-08-27T11:16:20Z

Solr Storage Plugin

initial code checkin

commit 30435a1de03066f1db8bb54ec302433031707c31
Author: sudipmukherjee 
Date:   2015-08-27T11:28:21Z

Solr Storage Plugin

initial code checkin

commit 03ab550d51beb64814cc37355c88897a26c37719
Author: sudipmukherjee 
Date:   2015-09-04T11:19:03Z

Apache Solr storage plugin

initial code

commit 7014bba265824e8d14d23cf282c4894c988155bc
Author: sudipmukherjee 
Date:   2015-09-29T20:13:29Z

Initial code check-in

Solr Storage plugin

commit a3f49d50c80de45553bfcfc393195c21be173a15
Author: sudipmukherjee 
Date:   2015-10-03T02:11:35Z

Solr storage plugin

initial code checkin

commit 774b0881a22e0be8644175f7b89dcca49a95c838
Author: sudipmukherjee 
Date:   2015-10-13T12:06:49Z

code cleanup

removing unnecessary eclipse files.

commit b42e09d24123d48e81ad78de19939f59b0d7c602
Author: sudipmukherjee 
Date:   2015-10-14T10:44:04Z

Initial Code check-in

solr storage plugin

commit ba6fd651afdd3acef0284013e6007af24638b77d
Author: sudipmukherjee 
Date:   2015-10-14T11:12:45Z

remove unused classes




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request: DRILL-4134: Allocator updates

[parthchandra]

Github user parthchandra commented on a diff in the pull request:

https://github.com/apache/drill/pull/283#discussion_r46379766
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/ops/FragmentContext.java ---
@@ -465,6 +441,14 @@ public Executor getExecutor(){
 return context.getExecutor();
   }
 
+  public long getFragmentMemoryLimit() {
+return allocator.getLimit();
+  }
+
+  public void setFragmentMemoryLimit(long value) {
--- End diff --

Second that. set/getFragmentMemoryLimit are no longer used and can be 
removed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request: DRILL-4134: Allocator updates

[parthchandra]

Github user parthchandra commented on a diff in the pull request:

https://github.com/apache/drill/pull/283#discussion_r46380109
  
--- Diff: exec/memory/base/src/main/java/io/netty/buffer/DrillBuf.java ---
@@ -230,20 +249,31 @@ public synchronized boolean release() {
*/
   @Override
   public synchronized boolean release(int decrement) {
--- End diff --

Do we still need to pass a decrement count to release? Doesn't look like it 
is used except with a decrement count of 1.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: Create 1.4 Release branch soon?

[Jacques Nadeau]

It seems like 4108 and 4145 should also be addressed. Steven, can you take
a look at trying to get these merged/resolved?  (4145 might be related to
4108 or is otherwise an off-by-one issue it seems.

--
Jacques Nadeau
CTO and Co-Founder, Dremio

On Tue, Dec 1, 2015 at 6:22 PM, Jacques Nadeau  wrote:

> It seems like we should also try to include:
>
> DRILL-4109
> DRILL-4125
> DRILL-4111
>
> 4111 is ready to merge if someone wants to merge it. It looks like
> Sudheesh just needs to commit.
>
> For 4125/4109, it seems like we should get Vicki's feedback if those
> changes are good for merge or if we should punt.
>
>
>
>
> --
> Jacques Nadeau
> CTO and Co-Founder, Dremio
>
> On Tue, Dec 1, 2015 at 3:59 PM, Venki Korukanti  > wrote:
>
>> I can manage the release.
>>
>> Here are the pending JIRAs so far:
>>
>> 1) DRILL-4053
>>
>> Let me know if you have any pending patches and want to get them into 1.4.
>>
>> Thanks
>> Venki
>>
>> On Tue, Dec 1, 2015 at 10:43 AM, Parth Chandra  wrote:
>>
>> > +1 on creating the release branch today.
>> > I'd like to get DRILL-4053 in. Patch is ready and doing one more round
>> of
>> > regression tests.
>> >
>> >
>> >
>> > On Tue, Dec 1, 2015 at 9:56 AM, Jacques Nadeau 
>> wrote:
>> >
>> > > It is December (already!). Seems like we should create a 1.4 release
>> > branch
>> > > soon so we can get a vote going. Does someone want to volunteer as the
>> > > release manager?
>> > >
>> > > thanks,
>> > > Jacques
>> > >
>> > > --
>> > > Jacques Nadeau
>> > > CTO and Co-Founder, Dremio
>> > >
>> >
>>
>
>

Parquet pushdown filtering

[Adam Gilmore]

Hi guys,

I'm trying to (re)implement pushdown filtering for Parquet with the new
Parquet metadata caching implementation.

I've run into a couple of challenges:

   1. Scan batches don't allow empty batches.  This means if a particular
   filter filters out *all* rows, we get an exception.  I haven't read the
   full comments on the relevant JIRA items, but it seems odd that we can't
   query an empty JSON file, for example.  This is a bit of a blocker to
   implement the pushdown filtering properly.
   2. The Parquet metadata doesn't include all the relevant metadata.
   Specifically, count of values is not included, therefore the default
   Parquet statistics filter has issues because it compares the count of
   values with count of nulls to work out if it can drop it.  This isn't
   necessarily a blocker, but it feels ugly simulating there's "1" row in a
   block (just to get around the null comparison).

Also, it feels a bit ugly rehydrating the standard Parquet metadata objects
manually.  I'm not sure I understand why we created our own objects for the
Parquet metadata as opposed to simply writing a custom serializer for those
objects which we store.

Thoughts would be great - I'd love to get a patch out for this.

[GitHub] drill pull request: DRILL-4134: Allocator updates

[parthchandra]

Github user parthchandra commented on the pull request:

https://github.com/apache/drill/pull/283#issuecomment-161190670
  
Overall looks great. 
Note on performance and concurrency numbers : 
 Dechang ran the performance and concurrency tests and we see no 
performance regressions. Numbers are almost the same. Concurrency seems to have 
improved. We see some failures when running 96 concurrent tpch-query6, that may 
be due to other resource constraints. No deadlocks.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: Parquet pushdown filtering

[Parth Chandra]

Parquet metadata has the rowCount for every rowGroup which is also the
value count for every column in the rowGroup. Isn't that what you need?

On Tue, Dec 1, 2015 at 10:10 PM, Adam Gilmore  wrote:

> Hi guys,
>
> I'm trying to (re)implement pushdown filtering for Parquet with the new
> Parquet metadata caching implementation.
>
> I've run into a couple of challenges:
>
>1. Scan batches don't allow empty batches.  This means if a particular
>filter filters out *all* rows, we get an exception.  I haven't read the
>full comments on the relevant JIRA items, but it seems odd that we can't
>query an empty JSON file, for example.  This is a bit of a blocker to
>implement the pushdown filtering properly.
>2. The Parquet metadata doesn't include all the relevant metadata.
>Specifically, count of values is not included, therefore the default
>Parquet statistics filter has issues because it compares the count of
>values with count of nulls to work out if it can drop it.  This isn't
>necessarily a blocker, but it feels ugly simulating there's "1" row in a
>block (just to get around the null comparison).
>
> Also, it feels a bit ugly rehydrating the standard Parquet metadata objects
> manually.  I'm not sure I understand why we created our own objects for the
> Parquet metadata as opposed to simply writing a custom serializer for those
> objects which we store.
>
> Thoughts would be great - I'd love to get a patch out for this.
>

Re: Create 1.4 Release branch soon?

[Steven Phillips]

Sure, I'll see if I can merge 4108, and look into 4145.

On Tue, Dec 1, 2015 at 10:10 PM, Jacques Nadeau  wrote:

> It seems like 4108 and 4145 should also be addressed. Steven, can you take
> a look at trying to get these merged/resolved?  (4145 might be related to
> 4108 or is otherwise an off-by-one issue it seems.
>
> --
> Jacques Nadeau
> CTO and Co-Founder, Dremio
>
> On Tue, Dec 1, 2015 at 6:22 PM, Jacques Nadeau  wrote:
>
> > It seems like we should also try to include:
> >
> > DRILL-4109
> > DRILL-4125
> > DRILL-4111
> >
> > 4111 is ready to merge if someone wants to merge it. It looks like
> > Sudheesh just needs to commit.
> >
> > For 4125/4109, it seems like we should get Vicki's feedback if those
> > changes are good for merge or if we should punt.
> >
> >
> >
> >
> > --
> > Jacques Nadeau
> > CTO and Co-Founder, Dremio
> >
> > On Tue, Dec 1, 2015 at 3:59 PM, Venki Korukanti <
> venki.koruka...@gmail.com
> > > wrote:
> >
> >> I can manage the release.
> >>
> >> Here are the pending JIRAs so far:
> >>
> >> 1) DRILL-4053
> >>
> >> Let me know if you have any pending patches and want to get them into
> 1.4.
> >>
> >> Thanks
> >> Venki
> >>
> >> On Tue, Dec 1, 2015 at 10:43 AM, Parth Chandra 
> wrote:
> >>
> >> > +1 on creating the release branch today.
> >> > I'd like to get DRILL-4053 in. Patch is ready and doing one more round
> >> of
> >> > regression tests.
> >> >
> >> >
> >> >
> >> > On Tue, Dec 1, 2015 at 9:56 AM, Jacques Nadeau 
> >> wrote:
> >> >
> >> > > It is December (already!). Seems like we should create a 1.4 release
> >> > branch
> >> > > soon so we can get a vote going. Does someone want to volunteer as
> the
> >> > > release manager?
> >> > >
> >> > > thanks,
> >> > > Jacques
> >> > >
> >> > > --
> >> > > Jacques Nadeau
> >> > > CTO and Co-Founder, Dremio
> >> > >
> >> >
> >>
> >
> >
>

[GitHub] drill pull request: DRILL-4111: turn tests off in travis as they d...

[sudheeshkatkam]

Github user sudheeshkatkam commented on the pull request:

https://github.com/apache/drill/pull/267#issuecomment-161197244
  
+1


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: Create 1.4 Release branch soon?

[Sudheesh Katkam]

Venki, can you please commit DRILL-4111?

Thank you,
Sudheesh

> On Dec 1, 2015, at 6:22 PM, Jacques Nadeau  wrote:
> 
> It seems like we should also try to include:
> 
> DRILL-4109
> DRILL-4125
> DRILL-4111
> 
> 4111 is ready to merge if someone wants to merge it. It looks like Sudheesh
> just needs to commit.
> 
> For 4125/4109, it seems like we should get Vicki's feedback if those
> changes are good for merge or if we should punt.
> 
> 
> 
> 
> --
> Jacques Nadeau
> CTO and Co-Founder, Dremio
> 
> On Tue, Dec 1, 2015 at 3:59 PM, Venki Korukanti 
> wrote:
> 
>> I can manage the release.
>> 
>> Here are the pending JIRAs so far:
>> 
>> 1) DRILL-4053
>> 
>> Let me know if you have any pending patches and want to get them into 1.4.
>> 
>> Thanks
>> Venki
>> 
>> On Tue, Dec 1, 2015 at 10:43 AM, Parth Chandra  wrote:
>> 
>>> +1 on creating the release branch today.
>>> I'd like to get DRILL-4053 in. Patch is ready and doing one more round of
>>> regression tests.
>>> 
>>> 
>>> 
>>> On Tue, Dec 1, 2015 at 9:56 AM, Jacques Nadeau 
>> wrote:
>>> 
 It is December (already!). Seems like we should create a 1.4 release
>>> branch
 soon so we can get a vote going. Does someone want to volunteer as the
 release manager?
 
 thanks,
 Jacques
 
 --
 Jacques Nadeau
 CTO and Co-Founder, Dremio
 
>>> 
>> 

Re: Create 1.4 Release branch soon?

[Venki Korukanti]

Sure, will merge DRILL-4111.

On Tue, Dec 1, 2015 at 10:43 PM, Sudheesh Katkam 
wrote:

> Venki, can you please commit DRILL-4111?
>
> Thank you,
> Sudheesh
>
> > On Dec 1, 2015, at 6:22 PM, Jacques Nadeau  wrote:
> >
> > It seems like we should also try to include:
> >
> > DRILL-4109
> > DRILL-4125
> > DRILL-4111
> >
> > 4111 is ready to merge if someone wants to merge it. It looks like
> Sudheesh
> > just needs to commit.
> >
> > For 4125/4109, it seems like we should get Vicki's feedback if those
> > changes are good for merge or if we should punt.
> >
> >
> >
> >
> > --
> > Jacques Nadeau
> > CTO and Co-Founder, Dremio
> >
> > On Tue, Dec 1, 2015 at 3:59 PM, Venki Korukanti <
> venki.koruka...@gmail.com>
> > wrote:
> >
> >> I can manage the release.
> >>
> >> Here are the pending JIRAs so far:
> >>
> >> 1) DRILL-4053
> >>
> >> Let me know if you have any pending patches and want to get them into
> 1.4.
> >>
> >> Thanks
> >> Venki
> >>
> >> On Tue, Dec 1, 2015 at 10:43 AM, Parth Chandra 
> wrote:
> >>
> >>> +1 on creating the release branch today.
> >>> I'd like to get DRILL-4053 in. Patch is ready and doing one more round
> of
> >>> regression tests.
> >>>
> >>>
> >>>
> >>> On Tue, Dec 1, 2015 at 9:56 AM, Jacques Nadeau 
> >> wrote:
> >>>
>  It is December (already!). Seems like we should create a 1.4 release
> >>> branch
>  soon so we can get a vote going. Does someone want to volunteer as the
>  release manager?
> 
>  thanks,
>  Jacques
> 
>  --
>  Jacques Nadeau
>  CTO and Co-Founder, Dremio
> 
> >>>
> >>
>
>

[GitHub] drill pull request: DRILL-4111: turn tests off in travis as they d...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/drill/pull/267


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Resolved] (DRILL-4111) turn tests off in travis as they don't work there

[Venki Korukanti (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-4111?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Venki Korukanti resolved DRILL-4111.

   Resolution: Fixed
Fix Version/s: 1.4.0

> turn tests off in travis as they don't work there
> -
>
> Key: DRILL-4111
> URL: https://issues.apache.org/jira/browse/DRILL-4111
> Project: Apache Drill
>  Issue Type: Task
>Reporter: Julien Le Dem
>Assignee: Julien Le Dem
> Fix For: 1.4.0
>
>
> Since the travis build always fails, we should just turn it off for now.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] drill pull request: Drill 4127: Reduce Hive metastore client API c...

[jinfengni]

Github user jinfengni commented on the pull request:

https://github.com/apache/drill/pull/286#issuecomment-161210859
  
@vkorukanti , I revised the patch based on our discussions. 

The cache is enabled when impersonation is turned on. I also put another 
commit which uses flat level cache for tables.  I run the queries which 
originally hit the long planning time/execution time, and seems things are what 
I expected. 

Please take a look at the PR. Thanks!



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---