Apache Drill Plan... - Add: "Center console for Mac"

[Jingguo Yao (Google Docs)]

Jingguo Yao added a suggestion to Apache Drill Plan Syntax  
(https://docs.google.com/document/d/1QTL8warUYS2KjldQrGUse7zp8eA72VKtLOHwfXy6c7I/edit?disco=Aqd0nlQ&usp=suggestion_email_discussion)


Jingguo Yao
Add: "Center console for Mac"


You received this email because you are subscribed to all comments on  
Apache Drill Plan Syntax.
Change  
(https://docs.google.com/document/docos/notify?id=1QTL8warUYS2KjldQrGUse7zp8eA72VKtLOHwfXy6c7I&title=Apache+Drill+Plan+Syntax)  
what Google sends you.

You can reply to this email to reply to the comment.

[jira] [Created] (DRILL-4596) Drill should do version check among drillbits

[Arina Ielchiieva (JIRA)]

Arina Ielchiieva created DRILL-4596:
---

 Summary: Drill should do version check among drillbits
 Key: DRILL-4596
 URL: https://issues.apache.org/jira/browse/DRILL-4596
 Project: Apache Drill
  Issue Type: New Feature
Affects Versions: 1.6.0
Reporter: Arina Ielchiieva
Assignee: Arina Ielchiieva
 Fix For: Future


Before registering new drillbit in zookeeper, we should do version check, and 
make sure all the running drillbits are in the same version.

Using drillbits of different version can lead to unexpected results.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Jira DRILL-4573

[Jean-Claude Cote]

Hi,

I've created a pull request for issue DRILL-4573. I'm wondering if it's in
the queue of pull request to be reviewed?

Thanks
Jean-Claude

[jira] [Created] (DRILL-4597) Calcite type validation assertions when planner.enable_type_inference is enabled for system tables

[Yuliya Feldman (JIRA)]

Yuliya Feldman created DRILL-4597:
-

 Summary: Calcite type validation assertions when 
planner.enable_type_inference is enabled for system tables
 Key: DRILL-4597
 URL: https://issues.apache.org/jira/browse/DRILL-4597
 Project: Apache Drill
  Issue Type: Bug
  Components: Query Planning & Optimization
Reporter: Yuliya Feldman


With calcite-core-1.4.0-drill-r11.jar:1.4.0-drill-r11 and type inference 
enabled following query fails:
select concat(hostname, ':', user_port) from sys.drillbits where `current`=true;

with below exception

at 
org.apache.calcite.sql.type.SqlTypeFactoryImpl.createSqlType(SqlTypeFactoryImpl.java:62)
 ~[calcite-core-1.4.0-drill-r11.jar:1.4.0-drill-r11]
at 
org.apache.drill.exec.planner.sql.TypeInferenceUtils$DrillConcatSqlReturnTypeInference.inferReturnType(TypeInferenceUtils.java:420)
 ~[classes/:na]
at org.apache.calcite.sql.SqlOperator.inferReturnType(SqlOperator.java:468) 
~[calcite-core-1.4.0-drill-r11.jar:1.4.0-drill-r11]
at org.apache.calcite.sql.SqlOperator.validateOperands(SqlOperator.java:435) 
~[calcite-core-1.4.0-drill-r11.jar:1.4.0-drill-r11]
at org.apache.calcite.sql.SqlFunction.deriveType(SqlFunction.java:287) 
~[calcite-core-1.4.0-drill-r11.jar:1.4.0-drill-r11]
at org.apache.calcite.sql.SqlFunction.deriveType(SqlFunction.java:222) 
~[calcite-core-1.4.0-drill-r11.jar:1.4.0-drill-r11]
at 
org.apache.calcite.sql.validate.SqlValidatorImpl$DeriveTypeVisitor.visit(SqlValidatorImpl.java:4288)
 ~[calcite-core-1.4.0-drill-r11.jar:1.4.0-drill-r11]
at 
org.apache.calcite.sql.validate.SqlValidatorImpl$DeriveTypeVisitor.visit(SqlValidatorImpl.java:4275)
 ~[calcite-core-1.4.0-drill-r11.jar:1.4.0-drill-r11]
at org.apache.calcite.sql.SqlCall.accept(SqlCall.java:130) 
~[calcite-core-1.4.0-drill-r11.jar:1.4.0-drill-r11]
at 
org.apache.calcite.sql.validate.SqlValidatorImpl.deriveTypeImpl(SqlValidatorImpl.java:1495)
 ~[calcite-core-1.4.0-drill-r11.jar:1.4.0-drill-r11]
at 
org.apache.calcite.sql.validate.SqlValidatorImpl.deriveType(SqlValidatorImpl.java:1478)
 ~[calcite-core-1.4.0-drill-r11.jar:1.4.0-drill-r11]
at 
org.apache.calcite.sql.validate.SqlValidatorImpl.expandSelectItem(SqlValidatorImpl.java:440)
 ~[calcite-core-1.4.0-drill-r11.jar:1.4.0-drill-r11]
at 
org.apache.calcite.sql.validate.SqlValidatorImpl.validateSelectList(SqlValidatorImpl.java:3447)
 ~[calcite-core-1.4.0-drill-r11.jar:1.4.0-drill-r11]
at 
org.apache.calcite.sql.validate.SqlValidatorImpl.validateSelect(SqlValidatorImpl.java:2995)
 ~[calcite-core-1.4.0-drill-r11.jar:1.4.0-drill-r11]
at 
org.apache.calcite.sql.validate.SelectNamespace.validateImpl(SelectNamespace.java:60)
 ~[calcite-core-1.4.0-drill-r11.jar:1.4.0-drill-r11]
at 
org.apache.calcite.sql.validate.AbstractNamespace.validate(AbstractNamespace.java:86)
 ~[calcite-core-1.4.0-drill-r11.jar:1.4.0-drill-r11]
at 
org.apache.calcite.sql.validate.SqlValidatorImpl.validateNamespace(SqlValidatorImpl.java:877)
 ~[calcite-core-1.4.0-drill-r11.jar:1.4.0-drill-r11]
at 
org.apache.calcite.sql.validate.SqlValidatorImpl.validateQuery(SqlValidatorImpl.java:863)
 ~[calcite-core-1.4.0-drill-r11.jar:1.4.0-drill-r11]
at org.apache.calcite.sql.SqlSelect.validate(SqlSelect.java:210) 
~[calcite-core-1.4.0-drill-r11.jar:1.4.0-drill-r11]
at 
org.apache.calcite.sql.validate.SqlValidatorImpl.validateScopedExpression(SqlValidatorImpl.java:837)
 ~[calcite-core-1.4.0-drill-r11.jar:1.4.0-drill-r11]
at 
org.apache.calcite.sql.validate.SqlValidatorImpl.validate(SqlValidatorImpl.java:551)
 ~[calcite-core-1.4.0-drill-r11.jar:1.4.0-drill-r11]
at 
org.apache.drill.exec.planner.sql.SqlConverter.validate(SqlConverter.java:155) 
~[classes/:na]
at 
org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.validateNode(DefaultSqlHandler.java:596)
 ~[classes/:na]
at 
org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.validateAndConvert(DefaultSqlHandler.java:192)
 ~[classes/:na]
at 
org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.getPlan(DefaultSqlHandler.java:164)
 ~[classes/:na]
at 
org.apache.drill.exec.planner.sql.DrillSqlWorker.getPlan(DrillSqlWorker.java:94)
 ~[classes/:na]
at org.apache.drill.exec.work.foreman.Foreman.runSQL(Foreman.java:970) 
[classes/:na]
at org.apache.drill.exec.work.foreman.Foreman.run(Foreman.java:254) 
[classes/:na]

So far we could not repro it on non-system tables, but any type inference on 
system table leads to the exception



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] drill pull request: DRILL-4596: Drill should do version check amon...

[arina-ielchiieva]

GitHub user arina-ielchiieva opened a pull request:

https://github.com/apache/drill/pull/474

DRILL-4596: Drill should do version check among drillbits



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/arina-ielchiieva/drill DRILL-4596

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/drill/pull/474.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #474


commit 76e857c29becee22daf116dd58a36fe34a920595
Author: Arina Ielchiieva 
Date:   2016-04-08T14:22:47Z

DRILL-4596: Drill should do version check among drillbits




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: Can this scenario cause a query to hang ?

[François Méthot]

It might just adds up to the mystery of this issue but when we start
getting those hanging CTAS query,
if we restart our drill cluster and the problem goes away.

Next time we start getting this problem I will try to collect the JStack
output of the foreman too.

Thanks for looking into this.

Francois



On Fri, Apr 8, 2016 at 2:20 AM, Abdel Hakim Deneche 
wrote:

> Opened DRILL-4595 [1]  to track this issue.
>
> Thanks
>
> [1] https://issues.apache.org/jira/browse/DRILL-4595
>
> On Fri, Apr 8, 2016 at 6:42 AM, Abdel Hakim Deneche  >
> wrote:
>
> > Hey John, thanks for sharing your experience. If you see this again try
> > collecting the jstack output for the foreman node of the query, and also
> > check in the query profile which fragments are still marked as RUNNING.
> >
> > Thanks
> >
> > On Thu, Apr 7, 2016 at 2:29 PM, John Omernik  wrote:
> >
> >> Abdel -
> >>
> >> I think I've seen this on a MapR cluster I run, especially on CTAS.  For
> >> me, I have not brought it up because the cluster I am running on has
> some
> >> serious personal issues (like being hardware that's near 7 years old,
> its
> >> a
> >> test cluster) and given the "hard to reproduce" nature of the problem,
> >> I've
> >> been reluctant to create noise. Given what you've described, it seems
> very
> >> similar to CTAS hangs I've seen, but couldn't accurately reproduce.
> >>
> >> This didn't add much to your post, but I wanted to give you a +1 for
> >> outlining this potential problem.  Once I move to more robust hardware,
> >> and
> >> I am in similar situations, I will post more verbose details from my
> side.
> >>
> >> John
> >>
> >>
> >>
> >> On Thu, Apr 7, 2016 at 2:29 AM, Abdel Hakim Deneche <
> >> adene...@maprtech.com>
> >> wrote:
> >>
> >> > So, we've been seeing some queries hang, I've come up with a possible
> >> > explanation, but so far it's really difficult to reproduce. Let me
> know
> >> if
> >> > you think this explanation doesn't hold up or if you have any ideas
> how
> >> we
> >> > can reproduce it. Thanks
> >> >
> >> > - generally it's a CTAS running on a large cluster (lot's of writers
> >> > running in parallel)
> >> > - logs show that the user channel was closed and UserServer caused the
> >> root
> >> > fragment to move to a FAILED state [1]
> >> > - jstack shows that the root fragment is blocked in it's receiver
> >> waiting
> >> > for data [2]
> >> > - jstack also shows that ALL other fragments are no longer running,
> and
> >> the
> >> > logs show that all of them succeeded [3]
> >> > - the foreman waits *forever* for the root fragment to finish
> >> >
> >> > [1] the only case I can think off is when the user channel closed
> while
> >> the
> >> > fragment was waiting for an ack from the user client
> >> > [2] if a writer finishes earlier than the others, it will send a data
> >> batch
> >> > to the root fragment that will be sent to the user. The root will then
> >> > immediately block on it's receiver waiting for the remaining writers
> to
> >> > finish
> >> > [3] once the root fragment moves to a failed state, the receiver will
> >> > immediately release any received batch and return an OK to the sender
> >> > without putting the batch in it's blocking queue.
> >> >
> >> > Abdelhakim Deneche
> >> >
> >> > Software Engineer
> >> >
> >> >   
> >> >
> >> >
> >> > Now Available - Free Hadoop On-Demand Training
> >> > <
> >> >
> >>
> http://www.mapr.com/training?utm_source=Email&utm_medium=Signature&utm_campaign=Free%20available
> >> > >
> >> >
> >>
> >
> >
> >
> > --
> >
> > Abdelhakim Deneche
> >
> > Software Engineer
> >
> >   
> >
> >
> > Now Available - Free Hadoop On-Demand Training
> > <
> http://www.mapr.com/training?utm_source=Email&utm_medium=Signature&utm_campaign=Free%20available
> >
> >
>
>
>
> --
>
> Abdelhakim Deneche
>
> Software Engineer
>
>   
>
>
> Now Available - Free Hadoop On-Demand Training
> <
> http://www.mapr.com/training?utm_source=Email&utm_medium=Signature&utm_campaign=Free%20available
> >
>

[GitHub] drill pull request: DRILL-4596: Drill should do version check amon...

[JohnOmernik]

Github user JohnOmernik commented on the pull request:

https://github.com/apache/drill/pull/474#issuecomment-207520784
  
So how is the version actually set? Is it just the bit to register with 
Drill? Is there a config setting that we can set in drill-override that sets 
the cluster version?  Where is the "master" version stored, and how do we move 
away from that? 

Thanks


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request: DRILL-4596: Drill should do version check amon...

[arina-ielchiieva]

Github user arina-ielchiieva commented on the pull request:

https://github.com/apache/drill/pull/474#issuecomment-207529907
  
Drill version is defined from build manifest file [1].
There is no config setting to override it. Though if you want to change 
version in "master", you'll have to update version property in pom.xml files. 
(1.7.0-SNAPSHOT)

[1] 
https://github.com/apache/drill/blob/master/common/src/main/java/org/apache/drill/common/util/DrillVersionInfo.java



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request: DRILL-4596: Drill should do version check amon...

[yufeldman]

Github user yufeldman commented on a diff in the pull request:

https://github.com/apache/drill/pull/474#discussion_r59063793
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/server/Drillbit.java ---
@@ -207,6 +209,28 @@ private void javaPropertiesToSystemOptions() {
   }
 
   /**
+   * Disallow registering drillbit when:
+   * 1. version is unknown;
+   * 2. drillbit with different version has been already registered.
--- End diff --

What if you first DrillBit registered had "bad" version? Somebody would 
have to bring it down, so others can start?
Should it be (at least) option to have soft enforcement, rather than just 
hard enforcement.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request: DRILL-4596: Drill should do version check amon...

[JohnOmernik]

Github user JohnOmernik commented on a diff in the pull request:

https://github.com/apache/drill/pull/474#discussion_r59064857
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/server/Drillbit.java ---
@@ -207,6 +209,28 @@ private void javaPropertiesToSystemOptions() {
   }
 
   /**
+   * Disallow registering drillbit when:
+   * 1. version is unknown;
+   * 2. drillbit with different version has been already registered.
--- End diff --

Agreed.  In addition to potential soft enforcement, I think we should 
consider the differences of a version defined in a pom.xml, the version of the 
running drillbit, and the version of a drill cluster.  By using a setting in 
drill-override to specify what the drill cluster version is, we don't get 
accident "first bit to register is the wrong version" situations.  Are there 
situations where we might want to allow a specific Drill version + all minor 
versions at that level, or Drill version + 1 Major version for rolling upgrades?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request: DRILL-4596: Drill should do version check amon...

[sudheeshkatkam]

Github user sudheeshkatkam commented on a diff in the pull request:

https://github.com/apache/drill/pull/474#discussion_r59065316
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/server/Drillbit.java ---
@@ -207,6 +209,28 @@ private void javaPropertiesToSystemOptions() {
   }
 
   /**
+   * Disallow registering drillbit when:
+   * 1. version is unknown;
+   * 2. drillbit with different version has been already registered.
+   */
+  private void checkVersion(DrillbitEndpoint endpoint) throws 
DrillbitStartupException {
+String currentVersion = endpoint.getVersion();
+if (DrillVersionInfo.UNKNOWN_VERSION.equals(currentVersion)) {
+  throw new DrillbitStartupException("Drillbit version is unknown.");
+}
+
+for (DrillbitEndpoint registeredEnpoint : 
coord.getAvailableEndpoints()) {
+  if (!currentVersion.equals(registeredEnpoint.getVersion())) {
--- End diff --

How does this check work with regards to [protobuf backwards 
compatibility](https://developers.google.com/protocol-buffers/docs/javatutorial#extending-a-protocol-buffer)?
 Endpoints currently do not have a version. So if two bits (A and B), one with 
and one without this change, are started, the two start order cases (A after B, 
and B after A) should pass.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request: DRILL-4596: Drill should do version check amon...

[arina-ielchiieva]

Github user arina-ielchiieva commented on a diff in the pull request:

https://github.com/apache/drill/pull/474#discussion_r59077998
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/server/Drillbit.java ---
@@ -207,6 +209,28 @@ private void javaPropertiesToSystemOptions() {
   }
 
   /**
+   * Disallow registering drillbit when:
+   * 1. version is unknown;
+   * 2. drillbit with different version has been already registered.
+   */
+  private void checkVersion(DrillbitEndpoint endpoint) throws 
DrillbitStartupException {
+String currentVersion = endpoint.getVersion();
+if (DrillVersionInfo.UNKNOWN_VERSION.equals(currentVersion)) {
+  throw new DrillbitStartupException("Drillbit version is unknown.");
+}
+
+for (DrillbitEndpoint registeredEnpoint : 
coord.getAvailableEndpoints()) {
+  if (!currentVersion.equals(registeredEnpoint.getVersion())) {
--- End diff --

Agree. Probably it's better to compare each module major versions.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request: DRILL-4577: Construct a specific path for quer...

[jinfengni]

Github user jinfengni commented on a diff in the pull request:

https://github.com/apache/drill/pull/461#discussion_r59090757
  
--- Diff: 
contrib/storage-hive/core/src/main/java/org/apache/drill/exec/store/hive/schema/HiveDatabaseSchema.java
 ---
@@ -72,4 +80,76 @@ public String getTypeName() {
 return HiveStoragePluginConfig.NAME;
   }
 
+  @Override
+  public List> getTablesByNames(final 
List tableNames) {
+final String schemaName = getName();
+final List> tableNameToTable = 
Lists.newArrayList();
+List tables;
+// Retries once if the first call to fetch the metadata fails
+synchronized(mClient) {
+  final List tableNamesWithAuth = Lists.newArrayList();
+  for(String tableName : tableNames) {
+try {
+  if(mClient.tableExists(schemaName, tableName)) {
--- End diff --

According to [1], under "Sql standard based authorization", Drill will 
return all the tables, even if the user does not have read access. That's the 
behavior before Sean's change to use bulk loading of getTableObjectsByNames(). 
However, under "Storage based authorization", the current expected behavior is 
only list the tables that user has access [2].

@vkorukanti , does this current behavior make sense? Why would Drill show 
different behavior under these two models?

Essentially, looks to me that the bulk loading will make Drill show same 
behavior under both "Sql standard based authorization", and "storage based 
authorization". That is, "show tables" will list all the tables, whether a user 
has access or not. But when a user query the table he does not have read 
access, then error will be raised.

[1] 
https://github.com/apache/drill/blob/master/contrib/storage-hive/core/src/test/java/org/apache/drill/exec/impersonation/hive/TestSqlStdBasedAuthorization.java#L153

[2] 
https://github.com/apache/drill/blob/master/contrib/storage-hive/core/src/test/java/org/apache/drill/exec/impersonation/hive/TestStorageBasedHiveAuthorization.java#L244-L247


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (DRILL-4598) Update Website to Mark Avro Support as Experimental

[John Omernik (JIRA)]

John Omernik created DRILL-4598:
---

 Summary: Update Website to Mark Avro Support as Experimental
 Key: DRILL-4598
 URL: https://issues.apache.org/jira/browse/DRILL-4598
 Project: Apache Drill
  Issue Type: Improvement
  Components: Documentation
Affects Versions: 1.5.0
Reporter: John Omernik


Per mailing list conversation, we should update the Documentation to mark the 
Avro Plugin as experimental.   I think there should be two changes... first on 
the Querying a File System Introduction, 

http://drill.apache.org/docs/querying-a-file-system-introduction/

Have 
Avro(type: avro)

be changed to 

Avro (type:avro) (Experimental See Querying Avro Files) With "Querying Avro 
Files" being a link to the next addition. 

Under Querying a Filesystem, and at the same level as "Querying JSON Files" or 
"Querying Parquet Files" We should add a page with "Querying Avro Files" 

As an initial change, We should have a basic page that states "Querying Avro 
Files is an experimental plugin at this time, and there are known issues.  
Please reference JIRA for more information"

This will be a good stop gap solution.  I will work to try and come up with a 
more complete version once I understand how the website component is pulled 
from the gh-pages site, and how I can do a pull request. In the meantime, if 
someone could update the pages with the "Stop gap" It would be appreciated. 






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)