DRILL-4013:JDBC/ODBC Warnings Design Proposal

[Abhijit Pol]

posted initial design proposal for
https://issues.apache.org/jira/browse/DRILL-4013

direct link:
https://docs.google.com/document/d/14d0t6XW6iLKqFpdzmYqpbrHR6-X-jHgM-VjTQwYybao/

please review and provide feedback/comments on the doc.

[jira] [Created] (DRILL-4086) No planning created

[boris chmiel (JIRA)]

boris chmiel created DRILL-4086:
---

 Summary: No planning created 
 Key: DRILL-4086
 URL: https://issues.apache.org/jira/browse/DRILL-4086
 Project: Apache Drill
  Issue Type: Bug
  Components: Execution - Codegen
Affects Versions: 1.2.0
Reporter: boris chmiel


The query is stuck seems blocked on planning (pending)


View : 
create or replace view View1 AS (
SELECT 
B1.columns[0] c0,
B1.columns[1] c1
FROM dfs.tmp.`TEST\B1.csv` B1
LEFT OUTER JOIN dfs.tmp.`TEST\BK.csv` BK
ON B1.columns[1] = BK.columns[0]
WHERE BK.columns[0] is null AND trim(B1.columns[1]) <> ''
);


create or replace view View2 AS (
SELECT 
View1.c0,
View1.c1
FROM View1
LEFT OUTER JOIN dfs.tmp.`TEST\BK.csv` BK
ON View1.c1 = BK.columns[0]
WHERE BK.columns[0] is null AND trim(View1.c1) <> ''
);


Query :
select * FROM dfs.tmp.View2 
=> Infinite Pending

data set : 
B1 :
A;
B;F
C;A
D;E
E;
F;C

BK:
A;1
B;2
F;4



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] drill pull request: DRILL-4047: Select with options

[jacques-n]

Github user jacques-n commented on a diff in the pull request:

https://github.com/apache/drill/pull/246#discussion_r44821281
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/store/dfs/WorkspaceSchemaFactory.java
 ---
@@ -148,12 +175,427 @@ private Path getViewPath(String name) {
 return DotDrillType.VIEW.getPath(config.getLocation(), name);
   }
 
-  public WorkspaceSchema createSchema(List parentSchemaPath, 
SchemaConfig schemaConfig) throws  IOException {
+  public WorkspaceSchema createSchema(List parentSchemaPath, 
SchemaConfig schemaConfig) throws IOException {
 return new WorkspaceSchema(parentSchemaPath, schemaName, schemaConfig);
   }
 
-  public class WorkspaceSchema extends AbstractSchema implements 
ExpandingConcurrentMap.MapValueFactory {
-private final ExpandingConcurrentMap tables = new 
ExpandingConcurrentMap<>(this);
+  /**
+   * Describes the options for a format plugin
+   * extracted from the FormatPluginConfig subclass
+   */
+  static final class OptionsDescriptor {
--- End diff --

Can you pull this out into a separate file?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request: DRILL-3786: Query with window function fails w...

[parthchandra]

Github user parthchandra commented on the pull request:

https://github.com/apache/drill/pull/239#issuecomment-156506328
  
+1. Looks good.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: drill accumulo

[Jeff Downton]

Hi,

I've done a little investigating, began by taking the HBase storage plugin
and started converting it to work with Accumulo.  Haven't gone too far with
it other than understanding the process flow and exploring what functions
on the Accumulo side would be needed to fill the gaps.  Whereas on HBase
you send the drill query to the specific region server how in Accumulo
would we send the query to the correct tserver or would you use a different
strategy via a batchscanner or the like.

Lots of questions at this point but happy to discuss further if anyone's
interested.

-Jeff


On Fri, Nov 13, 2015 at 11:39 AM, Steven Phillips  wrote:

> There is not currently a plugin available within the Apache Drill codebase
> for Accumulo, nor am I aware of anyone working on creating one.
>
> Is this something you would be interested in working on?
>
> On Fri, Nov 13, 2015 at 6:19 AM, Zen 98052  wrote:
>
> > Hi,
> >
> > Is there already Drill driver talk to Apache Accumulo? If not, do you
> know
> > if someone is actively working on it?
> >
> >
> > Thanks,
> > Z
> >
>

Re: comparing drill and presto

[Jacques Nadeau]

You're asking a community of Drill experts so we know Drill a lot better
than we know Presto.  Drill is focused on a couple of things:

- Extreme flexibility: for example being able to read json & mongo with no
upfront setup or sampling required.
- Extreme performance: runtime compilation, off-heap memory management, etc.

The Drill website covers both of these more eloquently than I would so I
suggest you look there for more detail on Drill's killer capabilities
around both of these categories.

I remember this perf bakeoff from a while ago: (much older versions of both
tech): http://allegro.tech/2015/06/fast-data-hackathon.html

--
Jacques Nadeau
CTO and Co-Founder, Dremio

On Thu, Nov 12, 2015 at 7:49 AM, Zen 98052  wrote:

> Hi,
>
> I just joined this mailing list today, so apologize if the same question
> has been asked before.
>
> I want to know what are main differences between Apache Drill and Presto
> Db (open sourced by Facebook)?
>
>
> Thanks,
>
> Z
>

Re: Java graphical application being launched during the Drill build?

[Steven Phillips]

I actually see it when running without tests as well.

On Fri, Nov 13, 2015 at 10:55 AM, Hsuan Yi Chu  wrote:

> Not bad feature, which gives the visualization of unit test completion.
>
> On Fri, Nov 13, 2015 at 10:27 AM, Parth Chandra  wrote:
>
> > Yes I see it too. Just a minor annoyance I thought.
> >
> > On Mon, Nov 9, 2015 at 2:59 PM, Sudheesh Katkam 
> > wrote:
> >
> > > I did, on my Mac. However, I haven’t looked into it.
> > >
> > > > On Nov 9, 2015, at 2:57 PM, Jason Altekruse <
> altekruseja...@gmail.com>
> > > wrote:
> > > >
> > > > Hello all,
> > > >
> > > > Has anyone else noticed a java graphical application starting up when
> > > > running the full drill build with test? On my mac I can clearly see a
> > new
> > > > icon appear on my task bar for a generic java application after I
> > launch
> > > > the build and tests from the command line. I only started seeing this
> > > > recently, I don't remember seeing any mail about a change in the
> build
> > > that
> > > > would have caused this.
> > > >
> > > > Has anyone else seen this? I'll be looking into it a little more,
> > trying
> > > to
> > > > identify when it comes up and find the test or build phase that is
> > > spawning
> > > > it, but I thought I'd just ask for any ideas to get me started.
> > > >
> > > > Thanks,
> > > > Jason
> > >
> > >
> >
>

Re: Java graphical application being launched during the Drill build?

[Julien Le Dem]

adding -Djava.awt.headless=true to the mvn command line gets rid of it.
I suspect something depends on an awt class that triggers the graphical
interface.
I've seen this before with a library that generated images using awt.
even if it was not displaying anything, that would trigger initializing the
graphical intfc.

On Fri, Nov 13, 2015 at 11:06 AM, Steven Phillips  wrote:

> I actually see it when running without tests as well.
>
> On Fri, Nov 13, 2015 at 10:55 AM, Hsuan Yi Chu 
> wrote:
>
> > Not bad feature, which gives the visualization of unit test completion.
> >
> > On Fri, Nov 13, 2015 at 10:27 AM, Parth Chandra 
> wrote:
> >
> > > Yes I see it too. Just a minor annoyance I thought.
> > >
> > > On Mon, Nov 9, 2015 at 2:59 PM, Sudheesh Katkam 
> > > wrote:
> > >
> > > > I did, on my Mac. However, I haven’t looked into it.
> > > >
> > > > > On Nov 9, 2015, at 2:57 PM, Jason Altekruse <
> > altekruseja...@gmail.com>
> > > > wrote:
> > > > >
> > > > > Hello all,
> > > > >
> > > > > Has anyone else noticed a java graphical application starting up
> when
> > > > > running the full drill build with test? On my mac I can clearly
> see a
> > > new
> > > > > icon appear on my task bar for a generic java application after I
> > > launch
> > > > > the build and tests from the command line. I only started seeing
> this
> > > > > recently, I don't remember seeing any mail about a change in the
> > build
> > > > that
> > > > > would have caused this.
> > > > >
> > > > > Has anyone else seen this? I'll be looking into it a little more,
> > > trying
> > > > to
> > > > > identify when it comes up and find the test or build phase that is
> > > > spawning
> > > > > it, but I thought I'd just ask for any ideas to get me started.
> > > > >
> > > > > Thanks,
> > > > > Jason
> > > >
> > > >
> > >
> >
>



-- 
Julien

[GitHub] drill pull request: DRILL-4047: Select with options

[jacques-n]

Github user jacques-n commented on a diff in the pull request:

https://github.com/apache/drill/pull/246#discussion_r44822446
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/store/dfs/WorkspaceSchemaFactory.java
 ---
@@ -148,12 +175,427 @@ private Path getViewPath(String name) {
 return DotDrillType.VIEW.getPath(config.getLocation(), name);
   }
 
-  public WorkspaceSchema createSchema(List parentSchemaPath, 
SchemaConfig schemaConfig) throws  IOException {
+  public WorkspaceSchema createSchema(List parentSchemaPath, 
SchemaConfig schemaConfig) throws IOException {
 return new WorkspaceSchema(parentSchemaPath, schemaName, schemaConfig);
   }
 
-  public class WorkspaceSchema extends AbstractSchema implements 
ExpandingConcurrentMap.MapValueFactory {
-private final ExpandingConcurrentMap tables = new 
ExpandingConcurrentMap<>(this);
+  /**
+   * Describes the options for a format plugin
+   * extracted from the FormatPluginConfig subclass
+   */
+  static final class OptionsDescriptor {
+final Class pluginConfigClass;
+final String typeName;
+private final Map functionParamsByName;
+
+OptionsDescriptor(Class 
pluginConfigClass) {
+  this.pluginConfigClass = pluginConfigClass;
+  Map paramsByName = new LinkedHashMap<>();
+  Field[] fields = pluginConfigClass.getDeclaredFields();
+  // @JsonTypeName("text")
+  JsonTypeName annotation = 
pluginConfigClass.getAnnotation(JsonTypeName.class);
+  this.typeName = annotation != null ? annotation.value() : null;
+  if (this.typeName != null) {
+paramsByName.put("type", new TableParamDef("type", String.class));
+  }
+  for (Field field : fields) {
+if (Modifier.isStatic(field.getModifiers())
+// we want to deprecate this field
+|| (field.getName().equals("extensions") && field.getType() == 
List.class)) {
+  continue;
+}
+Class fieldType = field.getType();
+if (fieldType == char.class) {
+  // calcite does not like char type. Just use String and enforce 
later that length == 1
+  fieldType = String.class;
+}
+paramsByName.put(field.getName(), new 
TableParamDef(field.getName(), fieldType).optional());
+  }
+  this.functionParamsByName = unmodifiableMap(paramsByName);
+}
+
+TableSignature getTableSignature(String tableName) {
+  return new TableSignature(tableName, params());
+}
+
+private List params() {
+  return new ArrayList<>(functionParamsByName.values());
+}
+
+String presentParams() {
+  StringBuilder sb = new StringBuilder("(");
+  List params = params();
+  for (int i = 0; i < params.size(); i++) {
+TableParamDef paramDef = params.get(i);
+if (i != 0) {
+  sb.append(", ");
+}
+sb.append(paramDef.name).append(": 
").append(paramDef.type.getSimpleName());
+  }
+  sb.append(")");
+  return sb.toString();
+}
+
+FormatPluginConfig eval(TableInstance t) {
--- End diff --

Can you provide a more descriptive method name and short doc?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (DRILL-4087) Error parsing JSON - Invalid numeric value: Leading zeroes not allowed

[Shankar (JIRA)]

Shankar created DRILL-4087:
--

 Summary: Error parsing JSON - Invalid numeric value: Leading 
zeroes not allowed
 Key: DRILL-4087
 URL: https://issues.apache.org/jira/browse/DRILL-4087
 Project: Apache Drill
  Issue Type: Bug
Affects Versions: 1.2.0
 Environment: Hadoop 2.7.1 cluster running on AWS staging instance 
t4.medium 
Apahe Dril - 1.2.0
Reporter: Shankar


jdbc:drill:> SELECT count(`timestamp`) FROM dfs.`/tmp/drill-s/` limit 10;
Error: DATA_READ ERROR: Error parsing JSON - Invalid numeric value: Leading 
zeroes not allowed

is there any solution for this error ?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] drill pull request: DRILL-4047: Select with options

[jacques-n]

Github user jacques-n commented on a diff in the pull request:

https://github.com/apache/drill/pull/246#discussion_r44821481
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/store/dfs/WorkspaceSchemaFactory.java
 ---
@@ -148,12 +175,427 @@ private Path getViewPath(String name) {
 return DotDrillType.VIEW.getPath(config.getLocation(), name);
   }
 
-  public WorkspaceSchema createSchema(List parentSchemaPath, 
SchemaConfig schemaConfig) throws  IOException {
+  public WorkspaceSchema createSchema(List parentSchemaPath, 
SchemaConfig schemaConfig) throws IOException {
 return new WorkspaceSchema(parentSchemaPath, schemaName, schemaConfig);
   }
 
-  public class WorkspaceSchema extends AbstractSchema implements 
ExpandingConcurrentMap.MapValueFactory {
-private final ExpandingConcurrentMap tables = new 
ExpandingConcurrentMap<>(this);
+  /**
+   * Describes the options for a format plugin
+   * extracted from the FormatPluginConfig subclass
+   */
+  static final class OptionsDescriptor {
+final Class pluginConfigClass;
+final String typeName;
+private final Map functionParamsByName;
+
+OptionsDescriptor(Class 
pluginConfigClass) {
+  this.pluginConfigClass = pluginConfigClass;
+  Map paramsByName = new LinkedHashMap<>();
+  Field[] fields = pluginConfigClass.getDeclaredFields();
+  // @JsonTypeName("text")
+  JsonTypeName annotation = 
pluginConfigClass.getAnnotation(JsonTypeName.class);
+  this.typeName = annotation != null ? annotation.value() : null;
+  if (this.typeName != null) {
+paramsByName.put("type", new TableParamDef("type", String.class));
+  }
+  for (Field field : fields) {
+if (Modifier.isStatic(field.getModifiers())
+// we want to deprecate this field
+|| (field.getName().equals("extensions") && field.getType() == 
List.class)) {
+  continue;
+}
+Class fieldType = field.getType();
+if (fieldType == char.class) {
+  // calcite does not like char type. Just use String and enforce 
later that length == 1
+  fieldType = String.class;
+}
+paramsByName.put(field.getName(), new 
TableParamDef(field.getName(), fieldType).optional());
+  }
+  this.functionParamsByName = unmodifiableMap(paramsByName);
+}
+
+TableSignature getTableSignature(String tableName) {
+  return new TableSignature(tableName, params());
+}
+
+private List params() {
+  return new ArrayList<>(functionParamsByName.values());
+}
+
+String presentParams() {
+  StringBuilder sb = new StringBuilder("(");
+  List params = params();
+  for (int i = 0; i < params.size(); i++) {
+TableParamDef paramDef = params.get(i);
+if (i != 0) {
+  sb.append(", ");
+}
+sb.append(paramDef.name).append(": 
").append(paramDef.type.getSimpleName());
+  }
+  sb.append(")");
+  return sb.toString();
+}
+
+FormatPluginConfig eval(TableInstance t) {
+  // Per the constructor, the first param is always "type"
+  TableParamDef typeParamDef = t.sig.params.get(0);
+  Object typeParam = t.params.get(0);
+  if (!typeParamDef.name.equals("type") || typeParamDef.type != 
String.class || !(typeParam instanceof String)
+  || !typeName.equalsIgnoreCase((String) typeParam)) {
+badInput(t);
+  }
+  FormatPluginConfig config;
+  try {
+config = pluginConfigClass.newInstance();
+  } catch (InstantiationException | IllegalAccessException e) {
+throw UserException.parseError(e)
+.message(
+"configuration for format of type %s can not be created 
(class: %s)",
+this.typeName, pluginConfigClass.getName())
+.addContext("table", t.sig.name)
+.build(logger);
+  }
+  for (int i = 1; i < t.params.size(); i++) {
+Object param = t.params.get(i);
+if (param == null)
+  continue;
+TableParamDef paramDef = t.sig.params.get(i);
+TableParamDef expectedParamDef = 
this.functionParamsByName.get(paramDef.name);
+if (expectedParamDef == null || expectedParamDef.type != 
paramDef.type) {
+  badInput(t);
+}
+try {
+  Field field = pluginConfigClass.getField(paramDef.name);
+  field.setAccessible(true);
+  if (field.getType() == char.class && param instanceof String) {
+

Re: Java graphical application being launched during the Drill build?

[Hsuan Yi Chu]

Not bad feature, which gives the visualization of unit test completion.

On Fri, Nov 13, 2015 at 10:27 AM, Parth Chandra  wrote:

> Yes I see it too. Just a minor annoyance I thought.
>
> On Mon, Nov 9, 2015 at 2:59 PM, Sudheesh Katkam 
> wrote:
>
> > I did, on my Mac. However, I haven’t looked into it.
> >
> > > On Nov 9, 2015, at 2:57 PM, Jason Altekruse 
> > wrote:
> > >
> > > Hello all,
> > >
> > > Has anyone else noticed a java graphical application starting up when
> > > running the full drill build with test? On my mac I can clearly see a
> new
> > > icon appear on my task bar for a generic java application after I
> launch
> > > the build and tests from the command line. I only started seeing this
> > > recently, I don't remember seeing any mail about a change in the build
> > that
> > > would have caused this.
> > >
> > > Has anyone else seen this? I'll be looking into it a little more,
> trying
> > to
> > > identify when it comes up and find the test or build phase that is
> > spawning
> > > it, but I thought I'd just ask for any ideas to get me started.
> > >
> > > Thanks,
> > > Jason
> >
> >
>

[GitHub] drill pull request: DRILL-4082: Better error message when multiple...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/drill/pull/252


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: Drill + Kudu Hackathon: Nov 19 @ 10:30am

[Timothy Chen]

How long is the Hackathon? I'm interested in coming.

Tim


> On Nov 13, 2015, at 2:59 PM, Jacques Nadeau  wrote:
> 
> Hey Guys,
> 
> We're going to do a hackathon on integrating Drill and Kudu next week.
> We'll be starting at 10:30 and going until late afternoon. We'll be working
> at 1001 Page Mill Rd @ Cloudera.
> 
> Feel free to join if you'd like to lend a helping hand. Please send me an
> RSVP so we make sure to get a large enough room.
> 
> 
> 
> --
> Jacques Nadeau
> CTO and Co-Founder, Dremio

[GitHub] drill pull request: DRILL-3791: MySQL tests for JDBC plugin

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/drill/pull/251


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: Avro deserialization bug - 1.3-SNAPSHOT

[Stefán Baxter]

So,

Could someone point me to the appropriate place in the Drill code to start
investigating this (We would love to contribute but getting up to speed is
a bit much).

I realize that there are many good things happening and that v. 1.3 is
around the corner but it seems that I incorrectly assumed that data
corruption issues would get a higher priority or that I would, at the very
least, get someone to confirm such a bug.

We are now impeded by this after having moved all our logging from JSON to
Avro to avoid the schema related problems we have been running into with
the JSON reader (null interpreted like double and failing when a string
eventually comes along) .

- Stefan


On Wed, Nov 11, 2015 at 10:14 PM, Stefán Baxter 
wrote:

> Hi,
>
> Can someone please verify that this is in fact a bug so I can rule out our
> own mistakes?
>
> We have recently moved all our logging to Avro to compensate for schema
> differences in JSON that were causing various problems and our latest
> release is now impeded with this.
> Alternatively can someone please point me in the right direction if I was
> to try to fix this myself.
>
> Regards,
>   -Stefán
>
> On Tue, Nov 10, 2015 at 2:41 PM, Stefán Baxter 
> wrote:
>
>> Thank you Kamesh.
>>
>> I have created https://issues.apache.org/jira/browse/DRILL-4056 with the
>> description.
>> I will send you a confidential test file to your private email.
>>
>> Regards,
>>  -Stefan
>>
>> On Tue, Nov 10, 2015 at 2:30 PM, Kamesh  wrote:
>>
>>> Hi Stefán,
>>>  Could you please raise a Jira with sample schema and sample input to
>>> reproduce it. I will look into this.
>>>
>>> On Tue, Nov 10, 2015 at 7:55 PM, Stefán Baxter <
>>> ste...@activitystream.com>
>>> wrote:
>>>
>>> > Hi,
>>> >
>>> > I have an Avro file that support the following data/schema:
>>> >
>>> > {"field":"some", "classification":{"variant":"Gæst"}}
>>> >
>>> > When I select 10 rows from this file I get:
>>> >
>>> > +-+
>>> > |   EXPR$0|
>>> > +-+
>>> > | Gæst|
>>> > | Voksen  |
>>> > | Voksen  |
>>> > | Invitation KIF KBH  |
>>> > | Invitation KIF KBH  |
>>> > | Ordinarie pris KBH  |
>>> > | Ordinarie pris KBH  |
>>> > | Biljetter 200 krBH  |
>>> > | Biljetter 200 krBH  |
>>> > | Biljetter 200 krBH  |
>>> > +-+
>>> >
>>> > The bug is that the field values are incorrectly de-serialized and the
>>> > value from the previous row is retained if the subsequent row is
>>> shorter.
>>> >
>>> > The sql query:
>>> >
>>> > "select s.classification.variant variant from dfs. as s limit
>>> 10;"
>>> >
>>> >
>>> > That way the  "Ordinarie pris" becomes "Ordinarie pris KBH" because the
>>> > previous row had the value "Invitation KIF KBH".
>>> >
>>> > Regards,
>>> >   -Stefán
>>> >
>>>
>>>
>>>
>>> --
>>> Kamesh.
>>>
>>
>>
>

[GitHub] drill pull request: DRILL-4083: ScanPrel's CPU cost should not be ...

[jinfengni]

Github user jinfengni commented on a diff in the pull request:

https://github.com/apache/drill/pull/253#discussion_r44845244
  
--- Diff: 
contrib/storage-hive/core/src/test/java/org/apache/drill/exec/hive/TestHiveStorage.java
 ---
@@ -36,6 +36,19 @@ public static void setupOptions() throws Exception {
 test(String.format("alter session set `%s` = true", 
PlannerSettings.ENABLE_DECIMAL_DATA_TYPE_KEY));
   }
 
+
+  @Test // DRILL-4083
+  public void testNativeScanWhenNoColumnIsRead() throws Exception {
+try {
+  test(String.format("alter session set `%s` = true", 
ExecConstants.HIVE_OPTIMIZE_SCAN_WITH_NATIVE_READERS));
+
+  String query = "SELECT count(*) FROM hive.readtest_parquet";
+  testPhysicalPlan(query, "hive-drill-native-parquet-scan");
+} finally {
+  test(String.format("alter session set `%s` = false", 
ExecConstants.HIVE_OPTIMIZE_SCAN_WITH_NATIVE_READERS));
--- End diff --

Will it make sense to change it back to default value in the finally block 
(in case we change the default value for this option in future release)? 



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request: DRILL-3765: Move partitioning pruning to HepPl...

[jinfengni]

GitHub user jinfengni opened a pull request:

https://github.com/apache/drill/pull/255

DRILL-3765: Move partitioning pruning to HepPlanner to avoid the perf…

…ormance overhead for redundant rule execution.

Add fall back option in planner.

Run the regression/unit with this feature turned on /off.

Please see DRILL-3765 for some performance number.

Regarding the planner options:
1. planner.enable_join_hep_opt is replaced with planner.enable_hep_opt.  
This controls whether HepPlanner would be used in drill logical planning. If 
this option is off, it would fall back to Volcano planner. 
2. Add new planning option : planner.enable_hep_partition_pruning, which 
will turn on/off this feature. If this option is off, it will fall back to the 
code prior to this patch. 

@amansinha100 , could you please review this patch?




You can merge this pull request into a Git repository by running:

$ git pull https://github.com/jinfengni/incubator-drill DRILL-3765

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/drill/pull/255.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #255


commit 739ac9482605cd2593b569ce2ec22fd176f6afbd
Author: Jinfeng Ni 
Date:   2015-11-03T23:51:10Z

DRILL-3765: Move partitioning pruning to HepPlanner to avoid the 
performance overhead for redundant rule execution.

Add fall back option in planner.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request: DRILL-4083: ScanPrel's CPU cost should not be ...

[jinfengni]

Github user jinfengni commented on the pull request:

https://github.com/apache/drill/pull/253#issuecomment-156584940
  
Overall LGTM.

In case that only constants are used and no column is required from scan 
operator, the scan operator still has to go through the data source; the cpu 
costing should not be 0. 

+1



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Build failed in Jenkins: drill-scm #590

[Apache Jenkins Server]

See 

Changes:

[altekrusejason] DRILL-3791: MySQL tests for JDBC plugin

[altekrusejason] DRILL-4082: Better error message when multiple versions of the 
same

--
[...truncated 84899 lines...]
  
Downloaded: 
http://repo.maven.apache.org/maven2/org/hsqldb/sqltool/2.3.2/sqltool-2.3.2.jar 
(144 KB at 1199.8 KB/sec)
4/4 KB   90/548 KB   99/247 KB
4/4 KB   94/548 KB   99/247 KB   
4/4 KB   98/548 KB   99/247 KB   
4/4 KB   99/548 KB   99/247 KB   
4/4 KB   102/548 KB   99/247 KB   
4/4 KB   106/548 KB   99/247 KB   
4/4 KB   110/548 KB   99/247 KB   
4/4 KB   114/548 KB   99/247 KB   
4/4 KB   118/548 KB   99/247 KB   
4/4 KB   122/548 KB   99/247 KB   
4/4 KB   125/548 KB   99/247 KB   
4/4 KB   125/548 KB   4/14 KB   99/247 KB   
4/4 KB   125/548 KB   8/14 KB   99/247 KB   
4/4 KB   125/548 KB   12/14 KB   99/247 KB   
4/4 KB   125/548 KB   14/14 KB   99/247 KB   
 
Downloaded: 
http://repo.maven.apache.org/maven2/org/slf4j/slf4j-nop/1.7.7/slf4j-nop-1.7.7.jar
 (4 KB at 78.3 KB/sec)
125/548 KB   14/14 KB   103/247 KB   
125/548 KB   14/14 KB   107/247 KB   
125/548 KB   14/14 KB   111/247 KB   
125/548 KB   14/14 KB   115/247 KB   
125/548 KB   14/14 KB   119/247 KB   
125/548 KB   14/14 KB   123/247 KB   
125/548 KB   14/14 KB   127/247 KB   
125/548 KB   14/14 KB   131/247 KB   
125/548 KB   14/14 KB   135/247 KB   
125/548 KB   14/14 KB   139/247 KB   
125/548 KB   14/14 KB   143/247 KB   
125/548 KB   14/14 KB   147/247 KB   
125/548 KB   14/14 KB   151/247 KB   
125/548 KB   14/14 KB   155/247 KB   
125/548 KB   14/14 KB   159/247 KB   
125/548 KB   14/14 KB   163/247 KB   
125/548 KB   14/14 KB   167/247 KB   
125/548 KB   14/14 KB   171/247 KB   
125/548 KB   14/14 KB   175/247 KB   
125/548 KB   14/14 KB   179/247 KB   
125/548 KB   14/14 KB   183/247 KB   
125/548 KB   14/14 KB   187/247 KB   
125/548 KB   14/14 KB   191/247 KB   
125/548 KB   14/14 KB   195/247 KB   
125/548 KB   14/14 KB   199/247 KB   
125/548 KB   14/14 KB   203/247 KB   
125/548 KB   14/14 KB   207/247 KB   
125/548 KB   14/14 KB   211/247 KB   
125/548 KB   14/14 KB   212/247 KB   
129/548 KB   14/14 KB   212/247 KB   
133/548 KB   14/14 KB   212/247 KB   
137/548 KB   14/14 KB   212/247 KB   
141/548 KB   14/14 KB   212/247 KB   
145/548 KB   14/14 KB   212/247 KB   
149/548 KB   14/14 KB   212/247 KB   
153/548 KB   14/14 KB   212/247 KB   
156/548 KB   14/14 KB   212/247 KB   
160/548 KB   14/14 KB   212/247 KB   
162/548 KB   14/14 KB   212/247 KB   
166/548 KB   14/14 KB   212/247 KB   
170/548 KB   14/14 KB   212/247 KB   
173/548 KB   14/14 KB   212/247 KB   
177/548 KB   14/14 KB   212/247 KB   
179/548 KB   14/14 KB   212/247 KB   
183/548 KB   14/14 KB   212/247 KB   
187/548 KB   14/14 KB   212/247 KB   
189/548 KB   14/14 KB   212/247 KB   
189/548 KB   14/14 KB   216/247 KB   
189/548 KB   14/14 KB   220/247 KB   
189/548 KB   14/14 KB   224/247 KB   
189/548 KB   14/14 KB   228/247 KB   
189/548 KB   14/14 KB   232/247 KB   
189/548 KB   14/14 KB   236/247 KB   
189/548 KB   14/14 KB   240/247 KB   
189/548 KB   14/14 KB   244/247 KB   
189/548 KB   14/14 KB   247/247 KB   
 
Downloaded: 
http://repo.maven.apache.org/maven2/com/btmatthews/utils/monitor/2.1.3/monitor-2.1.3.jar
 (14 KB at 265.9 KB/sec)
193/548 KB   247/247 KB  
197/548 KB   247/247 KB   
201/548 KB   247/247 KB   
205/548 KB   247/247 KB   
209/548 KB   247/247 KB   
213/548 KB   247/247 KB   
217/548 KB   247/247 KB   
221/548 KB   247/247 KB   
225/548 KB   247/247 KB   
229/548 KB   247/247 KB   
233/548 KB   247/247 KB   
236/548 KB   247/247 KB   
240/548 KB   247/247 KB   
244/548 KB   247/247 KB   
244/548 KB   247/247 KB   
248/548 KB   247/247 KB   
249/548 KB   247/247 KB   
253/548 KB   247/247 KB   
  
Downloaded: 
http://repo.maven.apache.org/maven2/org/apache/derby/derbynet/10.10.2.0/derbynet-10.10.2.0.jar
 (247 KB at 1318.1 KB/sec)
257/548 KB
261/548 KB   
265/548 KB   
269/548 KB   
273/548 KB   
277/548 KB   
281/548 KB   
285/548 KB   
289/548 KB   
291/548 KB   
295/548 KB   
299/548 KB   
300/548 KB   
301/548 KB   
304/548 KB   
308/548 KB   
312/548 KB   
316/548 KB   
317/548 KB   
321/548 KB   
323/548 KB   
327/548 KB   
331/548 KB   
335/548 KB   
339/548 KB   
343/548 KB   
347/548 KB   
347/548 KB   
351/548 KB   
352/548 KB   
356/548 KB   
360/548 KB   
364/548 KB   
368/548 KB   
372/548 KB   
376/548 KB   
380/548 KB   
384/548 KB   
388/548 KB   
392/548 KB   
396/548 KB   
400/548 KB   
404/548 KB   
408/548 KB   
412/548 KB   
416/548 KB   
420/548 KB   
424/548 KB   
428/548 KB   
432/548 KB   
436/548 KB   
440/548 KB   
440/548 KB   
444/548 KB   
446/548 KB   
450/548 KB   
450/548 KB   
454/548 KB   
458/548 KB   

[GitHub] drill pull request: DRILL-3786: Query with window function fails w...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/drill/pull/239


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

maven error when running unit tests on master

[Abdel Hakim Deneche]

If I run

mvn clean install -DskipTests
> mvn test


I get the following error when I run the tests:

[INFO]
> 
> [INFO] Reactor Summary:
> [INFO]
> [INFO] Apache Drill Root POM .. SUCCESS [
> 4.593 s]
> [INFO] ...
> [INFO] exec/Java Execution Engine . FAILURE [
> 2.531 s]
> [INFO] exec/JDBC Driver using dependencies  SKIPPED
> [INFO] ...
> [INFO] contrib/sqlline  SKIPPED
> [INFO]
> 
> [INFO] BUILD FAILURE
> [INFO]
> 
> [INFO] Total time: 36.305 s
> [INFO] Finished at: 2015-11-13T21:05:08+00:00
> [INFO] Final Memory: 118M/1445M
> [INFO]
> 
> [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-dependency-plugin:2.8:unpack
> (unpack-vector-types) on project drill-java-exec: Artifact has not been
> packaged yet. When used on reactor artifact, unpack should be executed
> after packaging: see MDEP-98. -> [Help 1]


I've seen this on both Mac with maven 3.3.3 and CentOS with maven 3.3.1

-- 

Abdelhakim Deneche

Software Engineer

  


Now Available - Free Hadoop On-Demand Training

[GitHub] drill pull request: DRILL-4053: Reduce metadata cache file size. S...

[StevenMPhillips]

Github user StevenMPhillips commented on the pull request:

https://github.com/apache/drill/pull/254#issuecomment-156576322
  
Do you have any numbers to illustrate how much of a performance improvement 
this change provides?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: Avro deserialization bug - 1.3-SNAPSHOT

[Jason Altekruse]

Stefan,

I took a look at the issue and I think I have a fix for the corruption you
are seeing. There have been a number of substantial commits to master
including a refactoring of a number of modules, so I applied this change on
top of the 1.3 branch for you to build and try out. I would like to add
some additional test cases, at which point I will open up and official PR
against master and we will likely be able to pull it back onto the 1.3
branch for inclusion in the release.

Please try this out to see if there are remaining issues reading your data.

https://github.com/jaltekruse/incubator-drill/tree/4056-avro-corruption-bug

Thanks,
Jason



On Fri, Nov 13, 2015 at 2:58 PM, Stefán Baxter 
wrote:

> So,
>
> Could someone point me to the appropriate place in the Drill code to start
> investigating this (We would love to contribute but getting up to speed is
> a bit much).
>
> I realize that there are many good things happening and that v. 1.3 is
> around the corner but it seems that I incorrectly assumed that data
> corruption issues would get a higher priority or that I would, at the very
> least, get someone to confirm such a bug.
>
> We are now impeded by this after having moved all our logging from JSON to
> Avro to avoid the schema related problems we have been running into with
> the JSON reader (null interpreted like double and failing when a string
> eventually comes along) .
>
> - Stefan
>
>
> On Wed, Nov 11, 2015 at 10:14 PM, Stefán Baxter  >
> wrote:
>
> > Hi,
> >
> > Can someone please verify that this is in fact a bug so I can rule out
> our
> > own mistakes?
> >
> > We have recently moved all our logging to Avro to compensate for schema
> > differences in JSON that were causing various problems and our latest
> > release is now impeded with this.
> > Alternatively can someone please point me in the right direction if I was
> > to try to fix this myself.
> >
> > Regards,
> >   -Stefán
> >
> > On Tue, Nov 10, 2015 at 2:41 PM, Stefán Baxter <
> ste...@activitystream.com>
> > wrote:
> >
> >> Thank you Kamesh.
> >>
> >> I have created https://issues.apache.org/jira/browse/DRILL-4056 with
> the
> >> description.
> >> I will send you a confidential test file to your private email.
> >>
> >> Regards,
> >>  -Stefan
> >>
> >> On Tue, Nov 10, 2015 at 2:30 PM, Kamesh 
> wrote:
> >>
> >>> Hi Stefán,
> >>>  Could you please raise a Jira with sample schema and sample input to
> >>> reproduce it. I will look into this.
> >>>
> >>> On Tue, Nov 10, 2015 at 7:55 PM, Stefán Baxter <
> >>> ste...@activitystream.com>
> >>> wrote:
> >>>
> >>> > Hi,
> >>> >
> >>> > I have an Avro file that support the following data/schema:
> >>> >
> >>> > {"field":"some", "classification":{"variant":"Gæst"}}
> >>> >
> >>> > When I select 10 rows from this file I get:
> >>> >
> >>> > +-+
> >>> > |   EXPR$0|
> >>> > +-+
> >>> > | Gæst|
> >>> > | Voksen  |
> >>> > | Voksen  |
> >>> > | Invitation KIF KBH  |
> >>> > | Invitation KIF KBH  |
> >>> > | Ordinarie pris KBH  |
> >>> > | Ordinarie pris KBH  |
> >>> > | Biljetter 200 krBH  |
> >>> > | Biljetter 200 krBH  |
> >>> > | Biljetter 200 krBH  |
> >>> > +-+
> >>> >
> >>> > The bug is that the field values are incorrectly de-serialized and
> the
> >>> > value from the previous row is retained if the subsequent row is
> >>> shorter.
> >>> >
> >>> > The sql query:
> >>> >
> >>> > "select s.classification.variant variant from dfs. as s limit
> >>> 10;"
> >>> >
> >>> >
> >>> > That way the  "Ordinarie pris" becomes "Ordinarie pris KBH" because
> the
> >>> > previous row had the value "Invitation KIF KBH".
> >>> >
> >>> > Regards,
> >>> >   -Stefán
> >>> >
> >>>
> >>>
> >>>
> >>> --
> >>> Kamesh.
> >>>
> >>
> >>
> >
>

Build failed in Jenkins: drill-scm #592

[Apache Jenkins Server]

See 

Changes:

[jni] DRILL-4083: ScanPrel's CPU cost should not be zero even if column count

--
[...truncated 3952 lines...]
[INFO] 
[INFO] --- maven-enforcer-plugin:1.3.1:enforce (avoid_bad_dependencies) @ 
drill-storage-hbase ---
[INFO] 
[INFO] --- maven-install-plugin:2.5.1:install (default-install) @ 
drill-storage-hbase ---
[INFO] Installing 

 to 
/home/jenkins/.m2/repository/org/apache/drill/contrib/drill-storage-hbase/1.4.0-SNAPSHOT/drill-storage-hbase-1.4.0-SNAPSHOT.jar
[INFO] Installing 
 to 
/home/jenkins/.m2/repository/org/apache/drill/contrib/drill-storage-hbase/1.4.0-SNAPSHOT/drill-storage-hbase-1.4.0-SNAPSHOT.pom
[INFO] Installing 

 to 
/home/jenkins/.m2/repository/org/apache/drill/contrib/drill-storage-hbase/1.4.0-SNAPSHOT/drill-storage-hbase-1.4.0-SNAPSHOT-tests.jar
[INFO] 
[INFO] 
[INFO] Building contrib/jdbc-storage-plugin 1.4.0-SNAPSHOT
[INFO] 
[INFO] 
[INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ drill-jdbc-storage ---
[INFO] Deleting 

[INFO] 
[INFO] --- apache-rat-plugin:0.11:check (rat-checks) @ drill-jdbc-storage ---
[INFO] 51 implicit excludes (use -debug for more details).
[INFO] Exclude: **/*.log
[INFO] Exclude: **/*.css
[INFO] Exclude: **/*.js
[INFO] Exclude: **/*.md
[INFO] Exclude: **/*.eps
[INFO] Exclude: **/*.json
[INFO] Exclude: **/*.seq
[INFO] Exclude: **/*.parquet
[INFO] Exclude: **/*.sql
[INFO] Exclude: **/git.properties
[INFO] Exclude: **/*.csv
[INFO] Exclude: **/*.csvh
[INFO] Exclude: **/*.tsv
[INFO] Exclude: **/*.txt
[INFO] Exclude: **/*.ssv
[INFO] Exclude: **/drill-*.conf
[INFO] Exclude: **/.buildpath
[INFO] Exclude: **/*.proto
[INFO] Exclude: **/*.fmpp
[INFO] Exclude: **/target/**
[INFO] Exclude: **/*.iml
[INFO] Exclude: **/*.tdd
[INFO] Exclude: **/*.project
[INFO] Exclude: **/TAGS
[INFO] Exclude: **/*.checkstyle
[INFO] Exclude: **/.classpath
[INFO] Exclude: **/.settings/**
[INFO] Exclude: .*/**
[INFO] Exclude: **/*.patch
[INFO] Exclude: **/*.pb.cc
[INFO] Exclude: **/*.pb.h
[INFO] Exclude: **/*.linux
[INFO] Exclude: **/client/build/**
[INFO] Exclude: **/*.tbl
[INFO] 15 resources included (use -debug for more details)
Warning:  org.apache.xerces.jaxp.SAXParserImpl$JAXPSAXParser: Property 
'http://www.oracle.com/xml/jaxp/properties/entityExpansionLimit' is not 
recognized.
Compiler warnings:
  WARNING:  'org.apache.xerces.jaxp.SAXParserImpl: Property 
'http://javax.xml.XMLConstants/property/accessExternalDTD' is not recognized.'
Warning:  org.apache.xerces.parsers.SAXParser: Feature 
'http://javax.xml.XMLConstants/feature/secure-processing' is not recognized.
Warning:  org.apache.xerces.parsers.SAXParser: Property 
'http://javax.xml.XMLConstants/property/accessExternalDTD' is not recognized.
Warning:  org.apache.xerces.parsers.SAXParser: Property 
'http://www.oracle.com/xml/jaxp/properties/entityExpansionLimit' is not 
recognized.
[INFO] Rat check: Summary of files. Unapproved: 0 unknown: 0 generated: 0 
approved: 15 licence.
[INFO] 
[INFO] --- git-commit-id-plugin:2.1.9:revision (for-jars) @ drill-jdbc-storage 
---
[info] dotGitDirectory 
[info] git.build.user.name Unknown
[info] git.build.user.email Unknown
[info] Using environment variable based branch name. GIT_BRANCH = origin/master
[info] git.branch origin/master
[info] --always = false
[info] --dirty = -dirty
[info] --abbrev = 7
[info] --long = %s true
[info] --match = 
[info] Tag refs [ 
[Ref[refs/tags/0.6.0-incubating=3daef7f553431c5aa76ff3074a8e00943de6dedd], 
Ref[refs/tags/0.9.0=78fd658d20767f8c627e29f58a6ede05a055bfdf], 
Ref[refs/tags/drill-1.0.0-m1=04020a8fca8b287874528d86dc7b8be0269ad788], 
Ref[refs/tags/drill-root-0.4.0-incubating=caa15e2629329cb56903189ff294bbd490a3fca8],
 Ref[refs/tags/drill-root-1.0.0-m1=ad638d9e41aa9efdb1e877cfe7e0a4b910f539fc], 
Ref[refs/tags/oscon_workshop=eaf95ed3c30d7bb147afe337e0e0477be6518d90], 
Ref[refs/tags/pre_exec_merge=a97a22b0a9547f8639e92258c0a3475b01742f15]] ]
[info] Resolved tag [ 0.6.0-incubating ] [ PersonIdent[Steven Phillips, 
sphill...@maprtech.com, Thu Oct 2 02:27:36 2014 -0700] ], points at [ commit 
5fa257b38df77dedf5609952e364ce0cfc7383d2 0 -- ] 
[info] Resolved tag [ 0.9.0 ] [ PersonIdent[Jacques Nadeau, jacq...@apache.org, 
Wed Apr 29 00:33:38 2015 -0700] ], points at [ commit 
c7cb36c8b45613bab299dfe6cafb36d4d2e00add 

[GitHub] drill pull request: DRILL-4053: Reduce metadata cache file size. S...

[jacques-n]

Github user jacques-n commented on the pull request:

https://github.com/apache/drill/pull/254#issuecomment-156589798
  
You're currently using an alternative file name for this. I think it would 
be better if we use the version field and continue to use the same file name. I 
assume we'll have many versions of this file. Also, what is the expected user 
result if they query a directory with an old file? Can we maintain multiple 
classes and dispatch on version?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request: DRILL-4053: Reduce metadata cache file size. S...

[parthchandra]

Github user parthchandra commented on the pull request:

https://github.com/apache/drill/pull/254#issuecomment-156603019
  
Your question suggests that Jackson can take care of deserializing into the 
correct Java objects based on version. I just may not have spent enough time 
figuring it out. I'll take a look but if you have pointers will gladly accept 
the same. In that case I can go back to the old file name and maintain multiple 
versions.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Build failed in Jenkins: drill-scm #591

[Apache Jenkins Server]

See 

Changes:

[asinha] DRILL-2601: log the query text and query id right at the beginning of

--
[...truncated 4014 lines...]
Audit done.

[INFO] 
[INFO] --- maven-enforcer-plugin:1.3.1:enforce (avoid_bad_dependencies) @ 
drill-storage-hbase ---
[INFO] 
[INFO] --- maven-install-plugin:2.5.1:install (default-install) @ 
drill-storage-hbase ---
[INFO] Installing 

 to 
/home/jenkins/.m2/repository/org/apache/drill/contrib/drill-storage-hbase/1.4.0-SNAPSHOT/drill-storage-hbase-1.4.0-SNAPSHOT.jar
[INFO] Installing 
 to 
/home/jenkins/.m2/repository/org/apache/drill/contrib/drill-storage-hbase/1.4.0-SNAPSHOT/drill-storage-hbase-1.4.0-SNAPSHOT.pom
[INFO] Installing 

 to 
/home/jenkins/.m2/repository/org/apache/drill/contrib/drill-storage-hbase/1.4.0-SNAPSHOT/drill-storage-hbase-1.4.0-SNAPSHOT-tests.jar
[INFO] 
[INFO] 
[INFO] Building contrib/jdbc-storage-plugin 1.4.0-SNAPSHOT
[INFO] 
[INFO] 
[INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ drill-jdbc-storage ---
[INFO] Deleting 

[INFO] 
[INFO] --- apache-rat-plugin:0.11:check (rat-checks) @ drill-jdbc-storage ---
[INFO] 51 implicit excludes (use -debug for more details).
[INFO] Exclude: **/*.log
[INFO] Exclude: **/*.css
[INFO] Exclude: **/*.js
[INFO] Exclude: **/*.md
[INFO] Exclude: **/*.eps
[INFO] Exclude: **/*.json
[INFO] Exclude: **/*.seq
[INFO] Exclude: **/*.parquet
[INFO] Exclude: **/*.sql
[INFO] Exclude: **/git.properties
[INFO] Exclude: **/*.csv
[INFO] Exclude: **/*.csvh
[INFO] Exclude: **/*.tsv
[INFO] Exclude: **/*.txt
[INFO] Exclude: **/*.ssv
[INFO] Exclude: **/drill-*.conf
[INFO] Exclude: **/.buildpath
[INFO] Exclude: **/*.proto
[INFO] Exclude: **/*.fmpp
[INFO] Exclude: **/target/**
[INFO] Exclude: **/*.iml
[INFO] Exclude: **/*.tdd
[INFO] Exclude: **/*.project
[INFO] Exclude: **/TAGS
[INFO] Exclude: **/*.checkstyle
[INFO] Exclude: **/.classpath
[INFO] Exclude: **/.settings/**
[INFO] Exclude: .*/**
[INFO] Exclude: **/*.patch
[INFO] Exclude: **/*.pb.cc
[INFO] Exclude: **/*.pb.h
[INFO] Exclude: **/*.linux
[INFO] Exclude: **/client/build/**
[INFO] Exclude: **/*.tbl
[INFO] 15 resources included (use -debug for more details)
Warning:  org.apache.xerces.jaxp.SAXParserImpl$JAXPSAXParser: Property 
'http://www.oracle.com/xml/jaxp/properties/entityExpansionLimit' is not 
recognized.
Compiler warnings:
  WARNING:  'org.apache.xerces.jaxp.SAXParserImpl: Property 
'http://javax.xml.XMLConstants/property/accessExternalDTD' is not recognized.'
Warning:  org.apache.xerces.parsers.SAXParser: Feature 
'http://javax.xml.XMLConstants/feature/secure-processing' is not recognized.
Warning:  org.apache.xerces.parsers.SAXParser: Property 
'http://javax.xml.XMLConstants/property/accessExternalDTD' is not recognized.
Warning:  org.apache.xerces.parsers.SAXParser: Property 
'http://www.oracle.com/xml/jaxp/properties/entityExpansionLimit' is not 
recognized.
[INFO] Rat check: Summary of files. Unapproved: 0 unknown: 0 generated: 0 
approved: 15 licence.
[INFO] 
[INFO] --- git-commit-id-plugin:2.1.9:revision (for-jars) @ drill-jdbc-storage 
---
[info] dotGitDirectory 
[info] git.build.user.name Unknown
[info] git.build.user.email Unknown
[info] Using environment variable based branch name. GIT_BRANCH = origin/master
[info] git.branch origin/master
[info] --always = false
[info] --dirty = -dirty
[info] --abbrev = 7
[info] --long = %s true
[info] --match = 
[info] Tag refs [ 
[Ref[refs/tags/0.6.0-incubating=3daef7f553431c5aa76ff3074a8e00943de6dedd], 
Ref[refs/tags/0.9.0=78fd658d20767f8c627e29f58a6ede05a055bfdf], 
Ref[refs/tags/drill-1.0.0-m1=04020a8fca8b287874528d86dc7b8be0269ad788], 
Ref[refs/tags/drill-root-0.4.0-incubating=caa15e2629329cb56903189ff294bbd490a3fca8],
 Ref[refs/tags/drill-root-1.0.0-m1=ad638d9e41aa9efdb1e877cfe7e0a4b910f539fc], 
Ref[refs/tags/oscon_workshop=eaf95ed3c30d7bb147afe337e0e0477be6518d90], 
Ref[refs/tags/pre_exec_merge=a97a22b0a9547f8639e92258c0a3475b01742f15]] ]
[info] Resolved tag [ 0.6.0-incubating ] [ PersonIdent[Steven Phillips, 
sphill...@maprtech.com, Thu Oct 2 02:27:36 2014 -0700] ], points at [ commit 
5fa257b38df77dedf5609952e364ce0cfc7383d2 0 -- ] 
[info] Resolved tag [ 0.9.0 ] [ PersonIdent[Jacques Nadeau, jacq...@apache.org, 
Wed Apr 29 00:33:38 2015 -0700] ], points at [ commit 

[GitHub] drill pull request: DRILL-4053: Reduce metadata cache file size. S...

[jacques-n]

Github user jacques-n commented on the pull request:

https://github.com/apache/drill/pull/254#issuecomment-156605047
  
As Steven said, one of the design goals I tasked Steven with was to 
multiple versions through Jackson and I believe we structured the file to 
support that effectively. If we can support that, I think it would be a nicer 
user experience.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: Drill + Kudu Hackathon: Nov 19 @ 10:30am

[Jacques Nadeau]

Currently planning to go until 4pm.

--
Jacques Nadeau
CTO and Co-Founder, Dremio

On Fri, Nov 13, 2015 at 3:08 PM, Timothy Chen  wrote:

> How long is the Hackathon? I'm interested in coming.
>
> Tim
>
>
> > On Nov 13, 2015, at 2:59 PM, Jacques Nadeau  wrote:
> >
> > Hey Guys,
> >
> > We're going to do a hackathon on integrating Drill and Kudu next week.
> > We'll be starting at 10:30 and going until late afternoon. We'll be
> working
> > at 1001 Page Mill Rd @ Cloudera.
> >
> > Feel free to join if you'd like to lend a helping hand. Please send me an
> > RSVP so we make sure to get a large enough room.
> >
> >
> >
> > --
> > Jacques Nadeau
> > CTO and Co-Founder, Dremio
>

[GitHub] drill pull request: DRILL-4083: ScanPrel's CPU cost should not be ...

[hsuanyi]

Github user hsuanyi commented on the pull request:

https://github.com/apache/drill/pull/253#issuecomment-156594216
  
Addressed reviewer's comment


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request: DRILL-4053: Reduce metadata cache file size. S...

[StevenMPhillips]

Github user StevenMPhillips commented on the pull request:

https://github.com/apache/drill/pull/254#issuecomment-156603625
  
Yes, that is true, it is able to deserialize based on version, similar to 
what we do when deserializing physical plans or storage plugin configurations. 
See StoragePluginConfig.java for an example. It uses the @JsonTypeInfo 
annotation.
 
I wasn't sure if some other part of the code needs to know the version 
before deserializing.

This was designed to support new versions.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request: DRILL-4053: Reduce metadata cache file size. S...

[parthchandra]

Github user parthchandra commented on the pull request:

https://github.com/apache/drill/pull/254#issuecomment-156589078
  
The last perf run was on 400K parquet files in a 2-layer directory 
hierarchy:
root_dir/ 
  2015-10-01/
 d_01/
 .
 .
 .
   2015-10-31/
 d_25/

That gave us 31x25 directories with 511 parquet files in each dir.

The numbers are with a warm file system cache.

BuildExplain Plan Refresh metadata  
Metadata size
MASTER (e275dc5)145.854 s   225.083 s  
2.9GB
with DRILL-4053   35.430 s   202.133s   
1.3GB


The Explain Plan query - "explain plan for select * from root_dir"
Refresh metadata query - "refresh table metadata dfs.root_dir " 




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: drill accumulo

[Zen 98052]

@Steven: Yes, I am thinking to work on it during my spare time on x-mas 
holidays.
@Jeff: I am thinking the same approach too, looking at HBase storage plugin 
(though I never used HBase, but Accumulo should have similar concept)


From: Jeff Downton 
Sent: Friday, November 13, 2015 2:45 PM
To: dev@drill.apache.org
Subject: Re: drill accumulo

Hi,

I've done a little investigating, began by taking the HBase storage plugin
and started converting it to work with Accumulo.  Haven't gone too far with
it other than understanding the process flow and exploring what functions
on the Accumulo side would be needed to fill the gaps.  Whereas on HBase
you send the drill query to the specific region server how in Accumulo
would we send the query to the correct tserver or would you use a different
strategy via a batchscanner or the like.

Lots of questions at this point but happy to discuss further if anyone's
interested.

-Jeff


On Fri, Nov 13, 2015 at 11:39 AM, Steven Phillips  wrote:

> There is not currently a plugin available within the Apache Drill codebase
> for Accumulo, nor am I aware of anyone working on creating one.
>
> Is this something you would be interested in working on?
>
> On Fri, Nov 13, 2015 at 6:19 AM, Zen 98052  wrote:
>
> > Hi,
> >
> > Is there already Drill driver talk to Apache Accumulo? If not, do you
> know
> > if someone is actively working on it?
> >
> >
> > Thanks,
> > Z
> >
>

[GitHub] drill pull request: DRILL-4053: Reduce metadata cache file size. S...

[StevenMPhillips]

Github user StevenMPhillips commented on the pull request:

https://github.com/apache/drill/pull/254#issuecomment-156602190
  
Why do you need to know the version before reading the file?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request: DRILL-4053: Reduce metadata cache file size. S...

[parthchandra]

Github user parthchandra commented on the pull request:

https://github.com/apache/drill/pull/254#issuecomment-156605210
  
OK. No problem. I'll put both versions in.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request: DRILL-3623: Use shorter query path for LIMIT 0...

[jacques-n]

Github user jacques-n commented on a diff in the pull request:

https://github.com/apache/drill/pull/193#discussion_r44852176
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/planner/logical/DrillDirectScanRel.java
 ---
@@ -0,0 +1,111 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.planner.logical;
+
+import com.google.common.collect.Iterators;
+import org.apache.calcite.plan.RelOptCluster;
+import org.apache.calcite.plan.RelTraitSet;
+import org.apache.calcite.rel.AbstractRelNode;
+import org.apache.calcite.rel.RelWriter;
+import org.apache.calcite.rel.type.RelDataType;
+import org.apache.drill.common.logical.data.LogicalOperator;
+import org.apache.drill.exec.physical.base.PhysicalOperator;
+import org.apache.drill.exec.planner.physical.DrillScanPrel;
+import org.apache.drill.exec.planner.physical.PhysicalPlanCreator;
+import org.apache.drill.exec.planner.physical.PlannerSettings;
+import org.apache.drill.exec.planner.physical.Prel;
+import org.apache.drill.exec.planner.physical.PrelUtil;
+import org.apache.drill.exec.planner.physical.visitor.PrelVisitor;
+import org.apache.drill.exec.record.BatchSchema;
+import org.apache.drill.exec.store.direct.DirectGroupScan;
+
+import java.io.IOException;
+import java.util.Iterator;
+
+/**
+ * Logical and physical RelNode representing a {@link DirectGroupScan}. 
This is not backed by a {@link DrillTable},
+ * unlike {@link DrillScanRel}.
+ */
+public class DrillDirectScanRel extends AbstractRelNode implements 
DrillScanPrel, DrillRel {
--- End diff --

Can we just use the VALUES operator instead? I think that Calcite already 
has the code to do this in the reduceexpressionsrule.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request: DRILL-4083: ScanPrel's CPU cost should not be ...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/drill/pull/253


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: select from table with options

[Julien Le Dem]

Here you go: https://issues.apache.org/jira/browse/CALCITE-967
I was planning on providing patch for both master and the fork, but I
haven't started yet.

On Thu, Nov 12, 2015 at 8:34 PM, Julian Hyde  wrote:

> You’re hitting the grammar ambiguity I expected.
>
> I think that base Calcite should require the full verbose syntax: the
> TABLE keyword for table functions and the EXTEND keyword for extends
> clauses. Then Drill can override to make TABLE optional, and Phoenix can
> override to make EXTEND optional.
>
> Are you changing the parser in your forked copy of Calcite, or are you
> changing Drill’s extensions to that parser?
>
> If the former, you (or I) should add extension points to Calcite’s parser
> make the TABLE keyword optional and to make the EXTEND keyword optional. No
> project should enable both extension points — otherwise they’ll end up with
> an ambiguous grammar. If you agree create a Calcite JIRA case for this.
>
> Julian
>
>
> > On Nov 11, 2015, at 1:55 PM, Julien Le Dem  wrote:
> >
> > Hi,
> > I've been trying to enable this but it looks like in the current grammar
> > (before my change) you can not use table functions and EXTEND together.
> > That's because they are on difference branches of an | in the grammar.
> > So I would suggest that we treat those as two separate improvement in two
> > different pull requests:
> > - not require table(...) to call table functions
> > - allow using table functions and extend together.
> > Does it make sense?
> > Julien
> >
> >
> > On Tue, Nov 10, 2015 at 12:51 PM, Julian Hyde  wrote:
> >
> >> To be clear, it should be possible to use a table function with all of
> >> the options -- EXTENDS clause, OVER clause, AS with alias and column
> >> aliases, TABLESAMPLE.
> >>
> >> I'm surprised that the parser didn't need more lookahead to choose
> >> between 't (x, y)' and 't (x INTEGER, y DATE)'.
> >>
> >> On Tue, Nov 10, 2015 at 12:28 PM, Julien Le Dem 
> wrote:
> >>> In the patch I just sent, probably not.
> >>> I will adjust it and add the corresponding test.
> >>>
> >>> On Tue, Nov 10, 2015 at 11:51 AM, Julian Hyde 
> wrote:
> >>>
>  Can you use both together? Say
> 
>   select columns
>   from dfs.`/path/to/myfile`(type => 'TEXT', fieldDelimiter => '|’)
> >> EXTEND
>  (foo INTEGER)
> 
>  Julian
> 
> 
> 
> > On Nov 10, 2015, at 10:51 AM, Julien Le Dem 
> >> wrote:
> >
> > I took a stab at adding the TableFunction syntax without table(...)
> in
> > Calcite.
> > I have verified that both the table function and extend (with or
> >> without
> > keyword) work
> >
> 
> >>
> https://github.com/julienledem/calcite/commit/b18f335c49e273294c2d475e359c610aaed3da34
> >
> > These work:
> >
> > select columns from dfs.`/path/to/myfile`(type => 'TEXT',
> >> fieldDelimiter
>  =>
> > '|')
> >
> > select columns from table(dfs.`/path/to/myfile`(type => 'TEXT',
> > fieldDelimiter => '|'))
> >
> > select columns from table(dfs.`/path/to/myfile`('JSON'))
> >
> > select columns from dfs.`/path/to/myfile`('JSON')
> >
> > select columns from dfs.`/path/to/myfile`(type => 'JSON')
> >
> > On Sat, Nov 7, 2015 at 5:15 PM, Jacques Nadeau 
>  wrote:
> >
> >> Drill does implicitly what Phoenix does explicitly so I don't think
> >> we
> >> should constrain ourselves to having a union of the two syntaxes.
> >>
> >>
> >> That being said, I think we could make these work together... maybe.
> >>
> >> Remove the EXTENDS without keyword syntax from the grammar.
> >>
> >> Create a new sub block in the table block that requires no keyword.
>  There
> >> would be two paths (and would probably require some lookahead)
> >>
> >> option 1> unnamed parameters (1,2,3)
> >> option 2> named parameters (a => 1, b=>2, c=> 3)
> >> option 3> create table field pattern (favoriteBand VARCHAR(100),
> >> golfHandicap INTEGER)
> >>
> >> Then we create a table function with options 1 & 2, an EXTENDS
> clause
>  for
> >> option 3.
> >>
> >> Best of both worlds?
> >>
> >> On Sat, Nov 7, 2015 at 4:44 PM, James Taylor <
> jamestay...@apache.org
> >>>
> >> wrote:
> >>
> >>> Phoenix already supports columns at read-time using the syntax
> >> without
> >> the
> >>> EXTENDS keyword as Julian indicated:
> >>>  SELECT * FROM Emp (favoriteBand VARCHAR(100), golfHandicap
> >> INTEGER)
> >>>  WHERE goldHandicap < 10;
> >>>
> >>> Changing this by requiring the EXTENDS keyword would create a
> >> backward
> >>> compatibility problem.
> >>>
> >>> I think it'd be good if both of these extensions worked in Drill &
> >> Phoenix
> >>> given our Drillix initiative.
> >>>
> >>> On Sat, Nov 7, 2015 

drill accumulo

[Zen 98052]

Hi,

Is there already Drill driver talk to Apache Accumulo? If not, do you know if 
someone is actively working on it?


Thanks,
Z