[jira] [Updated] (DRILL-4976) Querying Parquet files on S3 pulls too much data
[
https://issues.apache.org/jira/browse/DRILL-4976?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Uwe L. Korn updated DRILL-4976:
---
Summary: Querying Parquet files on S3 pulls too much data (was: Querying
Parquet files on S3 pulls )
> Querying Parquet files on S3 pulls too much data
> -
>
> Key: DRILL-4976
> URL: https://issues.apache.org/jira/browse/DRILL-4976
> Project: Apache Drill
> Issue Type: Improvement
> Components: Storage - Parquet
>Affects Versions: 1.8.0
>Reporter: Uwe L. Korn
>
> Currently (Drill 1.8, Hadoop 2.7.2) when queries are executed on files stored
> in S3, the underlying implementation of s3a requests magnitudes too much
> data. Given sufficient seek sizes, the following HTTP pattern is observed:
> * GET bytes=8k-100M
> * GET bytes=2M-100M
> * GET bytes=4M-100M
> Although the HTTP request were normally aborted before all the data was
> send by the server, it was still about 10-15x the size of the input files
> that went over the network, i.e. for a file of the size of 100M, sometimes 1G
> of data is transferred over the network.
> A fix for this is the newly introduced
> {{fs.s3a.experimental.input.fadvise=random}} mode which will be introduced
> with Hadoop 3.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (DRILL-4976) Querying Parquet files on S3 pulls
Uwe L. Korn created DRILL-4976:
--
Summary: Querying Parquet files on S3 pulls
Key: DRILL-4976
URL: https://issues.apache.org/jira/browse/DRILL-4976
Project: Apache Drill
Issue Type: Improvement
Components: Storage - Parquet
Affects Versions: 1.8.0
Reporter: Uwe L. Korn
Currently (Drill 1.8, Hadoop 2.7.2) when queries are executed on files stored
in S3, the underlying implementation of s3a requests magnitudes too much data.
Given sufficient seek sizes, the following HTTP pattern is observed:
* GET bytes=8k-100M
* GET bytes=2M-100M
* GET bytes=4M-100M
Although the HTTP request were normally aborted before all the data was
send by the server, it was still about 10-15x the size of the input files
that went over the network, i.e. for a file of the size of 100M, sometimes 1G
of data is transferred over the network.
A fix for this is the newly introduced
{{fs.s3a.experimental.input.fadvise=random}} mode which will be introduced with
Hadoop 3.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (DRILL-4977) Reading parquet metadata cache from S3 with fadvise=random and Hadoop 3 generates a large number of requests
Uwe L. Korn created DRILL-4977:
--
Summary: Reading parquet metadata cache from S3 with
fadvise=random and Hadoop 3 generates a large number of requests
Key: DRILL-4977
URL: https://issues.apache.org/jira/browse/DRILL-4977
Project: Apache Drill
Issue Type: Improvement
Components: Storage - Parquet
Affects Versions: 1.8.0
Environment: Hadoop 3.0
Reporter: Uwe L. Korn
When using the new {{fs.s3a.experimental.input.fadvise=random}} mode for
accessing Parquet files stored in S3, we see a significant improvement for the
query performance but a slowdown on query planning. This is due to the way the
metadata file is read (each chunk of 8000 bytes generates a new GET request to
S3). Indicating with {{FSDataInputStream.setReadahead(metadata-filesize)}} that
we will read the whole file, this behaviour is circumvented.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (DRILL-4978) Parquet metadata cache on S3 is always renewed
Uwe L. Korn created DRILL-4978: -- Summary: Parquet metadata cache on S3 is always renewed Key: DRILL-4978 URL: https://issues.apache.org/jira/browse/DRILL-4978 Project: Apache Drill Issue Type: Bug Components: Storage - Parquet Affects Versions: 1.8.0 Environment: Hadoop s3a storage Reporter: Uwe L. Korn As dictionary modification times are not tracked by S3 (see https://hadoop.apache.org/docs/r3.0.0-alpha1/hadoop-aws/tools/hadoop-aws/index.html#Warning_2:_Because_Object_stores_dont_track_modification_times_of_directories ) the Parquet metadata is always renewed on query planning. This could either be tuned by: * for the case of s3a, check the modification times of all Parquet files in this directory * deactivate the metadata cache for s3a -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (DRILL-4979) Make dataport configurable
Uwe L. Korn created DRILL-4979: -- Summary: Make dataport configurable Key: DRILL-4979 URL: https://issues.apache.org/jira/browse/DRILL-4979 Project: Apache Drill Issue Type: New Feature Components: Server Affects Versions: 1.8.0 Environment: Scheduling drillbits with Apache Mesos+Aurora Reporter: Uwe L. Korn Currently the dataport of a Drillbit is fixed to +1 on the control port. In a dynamic execution environment like Apache Mesos+Aurora, each port is allocated by the scheduler and then passed on to the application process. There is no possibility or guarantee to allocate two consecutive ports. Therefore, to run Drill in this environment, the dataport of the drillbit also needs to configurable by the scheduler. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (DRILL-4979) Make dataport configurable
[
https://issues.apache.org/jira/browse/DRILL-4979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15614793#comment-15614793
]
Uwe L. Korn commented on DRILL-4979:
Implementation-wise it seems like the only point where the {{+1}} is actually
hardcoded is in
{{exec/java-exec/src/main/java/org/apache/drill/exec/rpc/data/DataConnectionCreator.java:62}},
i.e. {{int port = server.bind(partialEndpoint.getControlPort() + 1,
allowPortHunting)}}. Therefore a fix for this could be to introduce a new
configuration option {{drill.exec.rpc.bit.server.dataport}} and use this if
set, otherwise default back to {{controlport + 1}}.
> Make dataport configurable
> --
>
> Key: DRILL-4979
> URL: https://issues.apache.org/jira/browse/DRILL-4979
> Project: Apache Drill
> Issue Type: New Feature
> Components: Server
>Affects Versions: 1.8.0
> Environment: Scheduling drillbits with Apache Mesos+Aurora
>Reporter: Uwe L. Korn
>
> Currently the dataport of a Drillbit is fixed to +1 on the control port. In a
> dynamic execution environment like Apache Mesos+Aurora, each port is
> allocated by the scheduler and then passed on to the application process.
> There is no possibility or guarantee to allocate two consecutive ports.
> Therefore, to run Drill in this environment, the dataport of the drillbit
> also needs to configurable by the scheduler.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-4373) Drill and Hive have incompatible timestamp representations in parquet
[
https://issues.apache.org/jira/browse/DRILL-4373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15615277#comment-15615277
]
ASF GitHub Bot commented on DRILL-4373:
---
Github user vdiravka commented on a diff in the pull request:
https://github.com/apache/drill/pull/600#discussion_r85522267
--- Diff:
exec/java-exec/src/test/java/org/apache/drill/exec/physical/impl/writer/TestParquetWriter.java
---
@@ -739,30 +741,76 @@ public void runTestAndValidate(String selection,
String validationSelection, Str
}
/*
- Test the reading of an int96 field. Impala encodes timestamps as int96
fields
+Impala encodes timestamp values as int96 fields. Test the reading of
an int96 field with two converters:
+the first one converts parquet INT96 into drill VARBINARY and the
second one (works while
+store.parquet.reader.int96_as_timestamp option is enabled) converts
parquet INT96 into drill TIMESTAMP.
*/
@Test
public void testImpalaParquetInt96() throws Exception {
compareParquetReadersColumnar("field_impala_ts",
"cp.`parquet/int96_impala_1.parquet`");
+try {
+ test("alter session set %s = true",
ExecConstants.PARQUET_READER_INT96_AS_TIMESTAMP);
+ compareParquetReadersColumnar("field_impala_ts",
"cp.`parquet/int96_impala_1.parquet`");
+} finally {
+ test("alter session reset %s",
ExecConstants.PARQUET_READER_INT96_AS_TIMESTAMP);
+}
}
/*
- Test the reading of a binary field where data is in dicationary _and_
non-dictionary encoded pages
+ Test the reading of a binary field as drill varbinary where data is in
dicationary _and_ non-dictionary encoded pages
*/
@Test
- public void testImpalaParquetVarBinary_DictChange() throws Exception {
+ public void testImpalaParquetBinaryAsVarBinary_DictChange() throws
Exception {
compareParquetReadersColumnar("field_impala_ts",
"cp.`parquet/int96_dict_change.parquet`");
}
/*
+ Test the reading of a binary field as drill timestamp where data is in
dicationary _and_ non-dictionary encoded pages
+ */
+ @Test
+ public void testImpalaParquetBinaryAsTimeStamp_DictChange() throws
Exception {
+final String WORKING_PATH = TestTools.getWorkingPath();
+final String TEST_RES_PATH = WORKING_PATH + "/src/test/resources";
+try {
+ testBuilder()
+ .sqlQuery("select int96_ts from
dfs_test.`%s/parquet/int96_dict_change`", TEST_RES_PATH)
+ .optionSettingQueriesForTestQuery(
+ "alter session set `%s` = true",
ExecConstants.PARQUET_READER_INT96_AS_TIMESTAMP)
+ .ordered()
+
.csvBaselineFile("testframework/testParquetReader/testInt96DictChange/q1.tsv")
+ .baselineTypes(TypeProtos.MinorType.TIMESTAMP)
+ .baselineColumns("int96_ts")
+ .build().run();
+} finally {
+ test("alter system reset `%s`",
ExecConstants.PARQUET_READER_INT96_AS_TIMESTAMP);
+}
+ }
+
+ /*
Test the conversion from int96 to impala timestamp
*/
@Test
- public void testImpalaParquetTimestampAsInt96() throws Exception {
+ public void testTimestampImpalaConvertFrom() throws Exception {
compareParquetReadersColumnar("convert_from(field_impala_ts,
'TIMESTAMP_IMPALA')", "cp.`parquet/int96_impala_1.parquet`");
}
/*
+ Test reading parquet Int96 as TimeStamp and comparing obtained values
with the
+ old results (reading the same values as VarBinary and
convert_fromTIMESTAMP_IMPALA function using)
+ */
+ @Test
+ public void testImpalaParquetTimestampInt96AsTimeStamp() throws
Exception {
--- End diff --
This test compares the results between new converter (Int96 to TimeStamp)
and the old one (Int96 to VarBinary) with `convert_fromTIMESTAMP_IMPALA`
function.
The issue was in the `ConvertFromImpalaTimestamp` [link to the code
](https://github.com/apache/drill/pull/600/commits/a45490af2dd663168220cc3bda62a2d79170db62#diff-5d8360c5e3cf7d2f6ac7bfe58b6d319aL57)
Because the timezone changing shouldn't affect on the result timestamp values.
I deleted timezone consideration there, so now all tests passed successfuly
even across different timezones.
> Drill and Hive have incompatible timestamp representations in parquet
> -
>
> Key: DRILL-4373
> URL: https://issues.apache.org/jira/browse/DRILL-4373
> Project: Apache Drill
> Issue Type: Improvement
> Components: Storage - Hive, Storage - Parquet
>Affects Versions: 1.8.0
>Reporter: Rahul Challapalli
>Assignee:
[jira] [Created] (DRILL-4980) Upgrading of the approach of parquet date correctness status detection
Vitalii Diravka created DRILL-4980: -- Summary: Upgrading of the approach of parquet date correctness status detection Key: DRILL-4980 URL: https://issues.apache.org/jira/browse/DRILL-4980 Project: Apache Drill Issue Type: Bug Components: Storage - Parquet Affects Versions: 1.8.0 Reporter: Vitalii Diravka Assignee: Vitalii Diravka Fix For: 1.9.0 This jira is an addition for the [DRILL-4203|https://issues.apache.org/jira/browse/DRILL-4203]. The date correctness label for the new generated parquet files should be upgraded. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (DRILL-4972) Drillbit shuts down immediately after starting if embedded web server is disabled
[
https://issues.apache.org/jira/browse/DRILL-4972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15615702#comment-15615702
]
ASF GitHub Bot commented on DRILL-4972:
---
Github user sudheeshkatkam commented on a diff in the pull request:
https://github.com/apache/drill/pull/633#discussion_r85552294
--- Diff:
exec/java-exec/src/main/java/org/apache/drill/exec/work/WorkManager.java ---
@@ -295,7 +295,7 @@ public FragmentExecutor getFragmentRunner(final
FragmentHandle handle) {
*/
private class StatusThread extends Thread {
public StatusThread() {
- setDaemon(true);
+ setDaemon(false);
--- End diff --
.. but this will make sure that is the case.
> Drillbit shuts down immediately after starting if embedded web server is
> disabled
> -
>
> Key: DRILL-4972
> URL: https://issues.apache.org/jira/browse/DRILL-4972
> Project: Apache Drill
> Issue Type: Bug
> Components: Server
>Affects Versions: 1.8.0
>Reporter: Sudheesh Katkam
>Assignee: Sorabh Hamirwasia
>Priority: Critical
> Fix For: 1.9.0
>
>
> Disable embedded web server by setting "drill.exec.http.enabled" to false.
> Now when drillbit is started, it shuts down immediately after starting.
> JVM exits when the only threads running are all daemon threads. Turns out all
> threads in a drillbit, other than the thread pool started by the web server,
> are daemon. So I suggest WorkManager#StatusThread be made non-daemon.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-4972) Drillbit shuts down immediately after starting if embedded web server is disabled
[
https://issues.apache.org/jira/browse/DRILL-4972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15615764#comment-15615764
]
ASF GitHub Bot commented on DRILL-4972:
---
Github user sohami commented on a diff in the pull request:
https://github.com/apache/drill/pull/633#discussion_r85557510
--- Diff:
exec/java-exec/src/main/java/org/apache/drill/exec/work/WorkManager.java ---
@@ -295,7 +295,7 @@ public FragmentExecutor getFragmentRunner(final
FragmentHandle handle) {
*/
private class StatusThread extends Thread {
public StatusThread() {
- setDaemon(true);
+ setDaemon(false);
--- End diff --
Not sure why removing it will not ensure the same. I think it's redundant
and like overriding a default already set in constructor with default value
using setter method.
> Drillbit shuts down immediately after starting if embedded web server is
> disabled
> -
>
> Key: DRILL-4972
> URL: https://issues.apache.org/jira/browse/DRILL-4972
> Project: Apache Drill
> Issue Type: Bug
> Components: Server
>Affects Versions: 1.8.0
>Reporter: Sudheesh Katkam
>Assignee: Sorabh Hamirwasia
>Priority: Critical
> Fix For: 1.9.0
>
>
> Disable embedded web server by setting "drill.exec.http.enabled" to false.
> Now when drillbit is started, it shuts down immediately after starting.
> JVM exits when the only threads running are all daemon threads. Turns out all
> threads in a drillbit, other than the thread pool started by the web server,
> are daemon. So I suggest WorkManager#StatusThread be made non-daemon.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-4964) Drill fails to connect to hive metastore after hive metastore is restarted unless drillbits are restarted also
[ https://issues.apache.org/jira/browse/DRILL-4964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15615979#comment-15615979 ] ASF GitHub Bot commented on DRILL-4964: --- Github user jinfengni commented on the issue: https://github.com/apache/drill/pull/628 +1 LGTM. > Drill fails to connect to hive metastore after hive metastore is restarted > unless drillbits are restarted also > -- > > Key: DRILL-4964 > URL: https://issues.apache.org/jira/browse/DRILL-4964 > Project: Apache Drill > Issue Type: Bug >Affects Versions: 1.8.0, 1.9.0 >Reporter: Sorabh Hamirwasia >Assignee: Jinfeng Ni > > It is found that if Hive Metastore is restarted then Drillbit also needs to > be restarted to further query and connect to hive metastore. > Repro Steps: > === > 1) Start HiveMetastore and drillbit. > 2) Start Drillbit client with Scheman as hive and run a simple query like > "show databases" >Command to start client: sqlline -u > "jdbc:drill:schema=hive;drillbit=" > 3) restart hive metastore > 4) Execute same query "show databases" on existing drillclient or new one. > You will see that hive default database is not listed. If you query any hive > data then it will fail. > Log snippet from drillbit.log: > == > 2016-10-25 18:32:00,561 [27eff86e-e8fb-3d91-eb88-4af75ff6d174:foreman] INFO > o.a.drill.exec.work.foreman.Foreman - Query text for query id > 27eff86e-e8fb-3d91-eb88-4af75ff6d174: show databases > 2016-10-25 18:32:00,563 [27eff86e-e8fb-3d91-eb88-4af75ff6d174:foreman] DEBUG > o.a.d.e.s.h.HBaseStoragePluginConfig - Initializing HBase StoragePlugin > configuration with zookeeper quorum 'localhost', port '2181'. > 2016-10-25 18:32:00,595 [27eff86e-e8fb-3d91-eb88-4af75ff6d174:foreman] WARN > o.a.d.e.s.h.schema.HiveSchemaFactory - Failure while attempting to access > HiveDatabase 'default'. > java.util.concurrent.ExecutionException: MetaException(message:Got exception: > org.apache.thrift.transport.TTransportException null) > at > com.google.common.util.concurrent.AbstractFuture$Sync.getValue(AbstractFuture.java:299) > ~[guava-18.0.jar:na] > at > com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:286) > ~[guava-18.0.jar:na] > at > com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:116) > ~[guava-18.0.jar:na] > at > com.google.common.util.concurrent.Uninterruptibles.getUninterruptibly(Uninterruptibles.java:137) > ~[guava-18.0.jar:na] > at > com.google.common.cache.LocalCache$Segment.getAndRecordStats(LocalCache.java:2348) > ~[guava-18.0.jar:na] > at > com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2320) > ~[guava-18.0.jar:na] > at > com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2282) > ~[guava-18.0.jar:na] > at > com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2197) > ~[guava-18.0.jar:na] > at com.google.common.cache.LocalCache.get(LocalCache.java:3937) > ~[guava-18.0.jar:na] > at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3941) > ~[guava-18.0.jar:na] > at > com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4824) > ~[guava-18.0.jar:na] > at > org.apache.drill.exec.store.hive.DrillHiveMetaStoreClient$HiveClientWithCaching.getDatabases(DrillHiveMetaStoreClient.java:415) > ~[drill-storage-hive-core-1.9.0-SNAPSHOT.jar:1.9.0-SNAPSHOT] > at > org.apache.drill.exec.store.hive.schema.HiveSchemaFactory$HiveSchema.getSubSchema(HiveSchemaFactory.java:139) > [drill-storage-hive-core-1.9.0-SNAPSHOT.jar:1.9.0-SNAPSHOT] > at > org.apache.drill.exec.store.hive.schema.HiveSchemaFactory$HiveSchema.(HiveSchemaFactory.java:133) > [drill-storage-hive-core-1.9.0-SNAPSHOT.jar:1.9.0-SNAPSHOT] > at > org.apache.drill.exec.store.hive.schema.HiveSchemaFactory.registerSchemas(HiveSchemaFactory.java:118) > [drill-storage-hive-core-1.9.0-SNAPSHOT.jar:1.9.0-SNAPSHOT] > at > org.apache.drill.exec.store.hive.HiveStoragePlugin.registerSchemas(HiveStoragePlugin.java:100) > [drill-storage-hive-core-1.9.0-SNAPSHOT.jar:1.9.0-SNAPSHOT] > at > org.apache.drill.exec.store.StoragePluginRegistryImpl$DrillSchemaFactory.registerSchemas(StoragePluginRegistryImpl.java:365) > [drill-java-exec-1.9.0-SNAPSHOT.jar:1.9.0-SNAPSHOT] > at > org.apache.drill.exec.store.SchemaTreeProvider.createRootSchema(SchemaTreeProvider.java:72) > [drill-java-exec-1.9.0-SNAPSHOT.jar:1.9.0-SNAPSHOT] > at > org.apache.drill.exec.store.SchemaTreeProvider.createRootSchema(SchemaTreeProvider.java:61) > [drill-java-exec-1.9.
[jira] [Commented] (DRILL-4964) Drill fails to connect to hive metastore after hive metastore is restarted unless drillbits are restarted also
[ https://issues.apache.org/jira/browse/DRILL-4964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15616011#comment-15616011 ] ASF GitHub Bot commented on DRILL-4964: --- Github user sohami commented on the issue: https://github.com/apache/drill/pull/628 @jinfengni - Thanks for review. > Drill fails to connect to hive metastore after hive metastore is restarted > unless drillbits are restarted also > -- > > Key: DRILL-4964 > URL: https://issues.apache.org/jira/browse/DRILL-4964 > Project: Apache Drill > Issue Type: Bug >Affects Versions: 1.8.0, 1.9.0 >Reporter: Sorabh Hamirwasia >Assignee: Jinfeng Ni > > It is found that if Hive Metastore is restarted then Drillbit also needs to > be restarted to further query and connect to hive metastore. > Repro Steps: > === > 1) Start HiveMetastore and drillbit. > 2) Start Drillbit client with Scheman as hive and run a simple query like > "show databases" >Command to start client: sqlline -u > "jdbc:drill:schema=hive;drillbit=" > 3) restart hive metastore > 4) Execute same query "show databases" on existing drillclient or new one. > You will see that hive default database is not listed. If you query any hive > data then it will fail. > Log snippet from drillbit.log: > == > 2016-10-25 18:32:00,561 [27eff86e-e8fb-3d91-eb88-4af75ff6d174:foreman] INFO > o.a.drill.exec.work.foreman.Foreman - Query text for query id > 27eff86e-e8fb-3d91-eb88-4af75ff6d174: show databases > 2016-10-25 18:32:00,563 [27eff86e-e8fb-3d91-eb88-4af75ff6d174:foreman] DEBUG > o.a.d.e.s.h.HBaseStoragePluginConfig - Initializing HBase StoragePlugin > configuration with zookeeper quorum 'localhost', port '2181'. > 2016-10-25 18:32:00,595 [27eff86e-e8fb-3d91-eb88-4af75ff6d174:foreman] WARN > o.a.d.e.s.h.schema.HiveSchemaFactory - Failure while attempting to access > HiveDatabase 'default'. > java.util.concurrent.ExecutionException: MetaException(message:Got exception: > org.apache.thrift.transport.TTransportException null) > at > com.google.common.util.concurrent.AbstractFuture$Sync.getValue(AbstractFuture.java:299) > ~[guava-18.0.jar:na] > at > com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:286) > ~[guava-18.0.jar:na] > at > com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:116) > ~[guava-18.0.jar:na] > at > com.google.common.util.concurrent.Uninterruptibles.getUninterruptibly(Uninterruptibles.java:137) > ~[guava-18.0.jar:na] > at > com.google.common.cache.LocalCache$Segment.getAndRecordStats(LocalCache.java:2348) > ~[guava-18.0.jar:na] > at > com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2320) > ~[guava-18.0.jar:na] > at > com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2282) > ~[guava-18.0.jar:na] > at > com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2197) > ~[guava-18.0.jar:na] > at com.google.common.cache.LocalCache.get(LocalCache.java:3937) > ~[guava-18.0.jar:na] > at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3941) > ~[guava-18.0.jar:na] > at > com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4824) > ~[guava-18.0.jar:na] > at > org.apache.drill.exec.store.hive.DrillHiveMetaStoreClient$HiveClientWithCaching.getDatabases(DrillHiveMetaStoreClient.java:415) > ~[drill-storage-hive-core-1.9.0-SNAPSHOT.jar:1.9.0-SNAPSHOT] > at > org.apache.drill.exec.store.hive.schema.HiveSchemaFactory$HiveSchema.getSubSchema(HiveSchemaFactory.java:139) > [drill-storage-hive-core-1.9.0-SNAPSHOT.jar:1.9.0-SNAPSHOT] > at > org.apache.drill.exec.store.hive.schema.HiveSchemaFactory$HiveSchema.(HiveSchemaFactory.java:133) > [drill-storage-hive-core-1.9.0-SNAPSHOT.jar:1.9.0-SNAPSHOT] > at > org.apache.drill.exec.store.hive.schema.HiveSchemaFactory.registerSchemas(HiveSchemaFactory.java:118) > [drill-storage-hive-core-1.9.0-SNAPSHOT.jar:1.9.0-SNAPSHOT] > at > org.apache.drill.exec.store.hive.HiveStoragePlugin.registerSchemas(HiveStoragePlugin.java:100) > [drill-storage-hive-core-1.9.0-SNAPSHOT.jar:1.9.0-SNAPSHOT] > at > org.apache.drill.exec.store.StoragePluginRegistryImpl$DrillSchemaFactory.registerSchemas(StoragePluginRegistryImpl.java:365) > [drill-java-exec-1.9.0-SNAPSHOT.jar:1.9.0-SNAPSHOT] > at > org.apache.drill.exec.store.SchemaTreeProvider.createRootSchema(SchemaTreeProvider.java:72) > [drill-java-exec-1.9.0-SNAPSHOT.jar:1.9.0-SNAPSHOT] > at > org.apache.drill.exec.store.SchemaTreeProvider.createRootSchema(SchemaTreeProvider.java:61) > [drill-java-
[jira] [Commented] (DRILL-4964) Drill fails to connect to hive metastore after hive metastore is restarted unless drillbits are restarted also
[ https://issues.apache.org/jira/browse/DRILL-4964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15616012#comment-15616012 ] ASF GitHub Bot commented on DRILL-4964: --- Github user sohami closed the pull request at: https://github.com/apache/drill/pull/628 > Drill fails to connect to hive metastore after hive metastore is restarted > unless drillbits are restarted also > -- > > Key: DRILL-4964 > URL: https://issues.apache.org/jira/browse/DRILL-4964 > Project: Apache Drill > Issue Type: Bug >Affects Versions: 1.8.0, 1.9.0 >Reporter: Sorabh Hamirwasia >Assignee: Jinfeng Ni > > It is found that if Hive Metastore is restarted then Drillbit also needs to > be restarted to further query and connect to hive metastore. > Repro Steps: > === > 1) Start HiveMetastore and drillbit. > 2) Start Drillbit client with Scheman as hive and run a simple query like > "show databases" >Command to start client: sqlline -u > "jdbc:drill:schema=hive;drillbit=" > 3) restart hive metastore > 4) Execute same query "show databases" on existing drillclient or new one. > You will see that hive default database is not listed. If you query any hive > data then it will fail. > Log snippet from drillbit.log: > == > 2016-10-25 18:32:00,561 [27eff86e-e8fb-3d91-eb88-4af75ff6d174:foreman] INFO > o.a.drill.exec.work.foreman.Foreman - Query text for query id > 27eff86e-e8fb-3d91-eb88-4af75ff6d174: show databases > 2016-10-25 18:32:00,563 [27eff86e-e8fb-3d91-eb88-4af75ff6d174:foreman] DEBUG > o.a.d.e.s.h.HBaseStoragePluginConfig - Initializing HBase StoragePlugin > configuration with zookeeper quorum 'localhost', port '2181'. > 2016-10-25 18:32:00,595 [27eff86e-e8fb-3d91-eb88-4af75ff6d174:foreman] WARN > o.a.d.e.s.h.schema.HiveSchemaFactory - Failure while attempting to access > HiveDatabase 'default'. > java.util.concurrent.ExecutionException: MetaException(message:Got exception: > org.apache.thrift.transport.TTransportException null) > at > com.google.common.util.concurrent.AbstractFuture$Sync.getValue(AbstractFuture.java:299) > ~[guava-18.0.jar:na] > at > com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:286) > ~[guava-18.0.jar:na] > at > com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:116) > ~[guava-18.0.jar:na] > at > com.google.common.util.concurrent.Uninterruptibles.getUninterruptibly(Uninterruptibles.java:137) > ~[guava-18.0.jar:na] > at > com.google.common.cache.LocalCache$Segment.getAndRecordStats(LocalCache.java:2348) > ~[guava-18.0.jar:na] > at > com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2320) > ~[guava-18.0.jar:na] > at > com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2282) > ~[guava-18.0.jar:na] > at > com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2197) > ~[guava-18.0.jar:na] > at com.google.common.cache.LocalCache.get(LocalCache.java:3937) > ~[guava-18.0.jar:na] > at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3941) > ~[guava-18.0.jar:na] > at > com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4824) > ~[guava-18.0.jar:na] > at > org.apache.drill.exec.store.hive.DrillHiveMetaStoreClient$HiveClientWithCaching.getDatabases(DrillHiveMetaStoreClient.java:415) > ~[drill-storage-hive-core-1.9.0-SNAPSHOT.jar:1.9.0-SNAPSHOT] > at > org.apache.drill.exec.store.hive.schema.HiveSchemaFactory$HiveSchema.getSubSchema(HiveSchemaFactory.java:139) > [drill-storage-hive-core-1.9.0-SNAPSHOT.jar:1.9.0-SNAPSHOT] > at > org.apache.drill.exec.store.hive.schema.HiveSchemaFactory$HiveSchema.(HiveSchemaFactory.java:133) > [drill-storage-hive-core-1.9.0-SNAPSHOT.jar:1.9.0-SNAPSHOT] > at > org.apache.drill.exec.store.hive.schema.HiveSchemaFactory.registerSchemas(HiveSchemaFactory.java:118) > [drill-storage-hive-core-1.9.0-SNAPSHOT.jar:1.9.0-SNAPSHOT] > at > org.apache.drill.exec.store.hive.HiveStoragePlugin.registerSchemas(HiveStoragePlugin.java:100) > [drill-storage-hive-core-1.9.0-SNAPSHOT.jar:1.9.0-SNAPSHOT] > at > org.apache.drill.exec.store.StoragePluginRegistryImpl$DrillSchemaFactory.registerSchemas(StoragePluginRegistryImpl.java:365) > [drill-java-exec-1.9.0-SNAPSHOT.jar:1.9.0-SNAPSHOT] > at > org.apache.drill.exec.store.SchemaTreeProvider.createRootSchema(SchemaTreeProvider.java:72) > [drill-java-exec-1.9.0-SNAPSHOT.jar:1.9.0-SNAPSHOT] > at > org.apache.drill.exec.store.SchemaTreeProvider.createRootSchema(SchemaTreeProvider.java:61) > [drill-java-exec-1.9.0-SNAPSHOT.jar:1.9.0-SNAPS
[jira] [Commented] (DRILL-4972) Drillbit shuts down immediately after starting if embedded web server is disabled
[ https://issues.apache.org/jira/browse/DRILL-4972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15616093#comment-15616093 ] ASF GitHub Bot commented on DRILL-4972: --- Github user parthchandra commented on the issue: https://github.com/apache/drill/pull/633 +1. LGTM > Drillbit shuts down immediately after starting if embedded web server is > disabled > - > > Key: DRILL-4972 > URL: https://issues.apache.org/jira/browse/DRILL-4972 > Project: Apache Drill > Issue Type: Bug > Components: Server >Affects Versions: 1.8.0 >Reporter: Sudheesh Katkam >Assignee: Sorabh Hamirwasia >Priority: Critical > Fix For: 1.9.0 > > > Disable embedded web server by setting "drill.exec.http.enabled" to false. > Now when drillbit is started, it shuts down immediately after starting. > JVM exits when the only threads running are all daemon threads. Turns out all > threads in a drillbit, other than the thread pool started by the web server, > are daemon. So I suggest WorkManager#StatusThread be made non-daemon. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (DRILL-4967) Adding template_name to source code generated using freemarker template
[ https://issues.apache.org/jira/browse/DRILL-4967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15616110#comment-15616110 ] ASF GitHub Bot commented on DRILL-4967: --- Github user amansinha100 commented on the issue: https://github.com/apache/drill/pull/629 +1 > Adding template_name to source code generated using freemarker template > --- > > Key: DRILL-4967 > URL: https://issues.apache.org/jira/browse/DRILL-4967 > Project: Apache Drill > Issue Type: Improvement >Affects Versions: 1.8.0 >Reporter: Jinfeng Ni >Assignee: Aman Sinha > Fix For: 1.9.0 > > > Drill uses FreeMarker to generate source code in maven build process. It > would be helpful if we could put the freemarker template name in the > generated source code, so that we could easily locate the template, in case > we have to make change to the template. > Earlier on there was a discussion in the dev list about this idea, and some > templates have been doing so. This JIRA is opened to put the template name in > all the rest of freemarker templates. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (DRILL-4927) Add support for Null Equality Joins
[
https://issues.apache.org/jira/browse/DRILL-4927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15616128#comment-15616128
]
ASF GitHub Bot commented on DRILL-4927:
---
GitHub user KulykRoman opened a pull request:
https://github.com/apache/drill/pull/635
DRILL-4927 (part 2): Add support for Null Equality Joins (mixed compa…
…rators)
This changes are a subset of the original pull request from DRILL-4539
(PR-462).
- Added changes to support mixed comparators;
- Added tests for it.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/KulykRoman/drill DRILL-4927
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/drill/pull/635.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #635
commit 47abbd9aa21443d76bb58b001d3f8a2e36511a7f
Author: Roman Kulyk
Date: 2016-10-28T13:26:53Z
DRILL-4927 (part 2): Add support for Null Equality Joins (mixed comparators)
This changes are a subset of the original pull request from DRILL-4539
(PR-462).
- Added changes to support mixed comparators;
- Added tests for it.
> Add support for Null Equality Joins
> ---
>
> Key: DRILL-4927
> URL: https://issues.apache.org/jira/browse/DRILL-4927
> Project: Apache Drill
> Issue Type: Improvement
> Components: Query Planning & Optimization
>Affects Versions: 1.8.0
>Reporter: Roman
>Assignee: Aman Sinha
> Labels: doc-impacting
> Fix For: 1.9.0
>
>
> Join with an equality condition which allows null=null fails. For example, if
> we use some of this queries:
> {code:sql}
> select ... FROM t1, t2 WHERE t1.c1 = t2.c2 OR (t1.c1 IS NULL AND t2.c2 IS
> NULL);
> select ... FROM t1 INNER JOIN t2 ON t1.c1 = t2.c2 OR (t1.c1 IS NULL AND
> t2.c2 IS NULL);
> {code}
> we got "UNSUPPORTED_OPERATION ERROR". We should add support for this option.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (DRILL-4968) Add column size information to ColumnMetadata
[ https://issues.apache.org/jira/browse/DRILL-4968?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sudheesh Katkam resolved DRILL-4968. Resolution: Fixed Fix Version/s: 1.9.0 Fixed in [c6dbe6a|https://github.com/apache/drill/commit/c6dbe6a2f7033114d6239a4850a9b5092e684589]. > Add column size information to ColumnMetadata > - > > Key: DRILL-4968 > URL: https://issues.apache.org/jira/browse/DRILL-4968 > Project: Apache Drill > Issue Type: Sub-task > Components: Metadata >Reporter: Laurent Goujon >Assignee: Laurent Goujon > Fix For: 1.9.0 > > > Both ODBC and JDBC needs column size information for the column metadata. > Instead of duplicating the logic between C++ and Java (and having to keep in > them sync), column size should be computed on the server so that value is > kept consistent across clients. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (DRILL-4974) NPE in FindPartitionConditions.analyzeCall() for 'holistic' expressions
[ https://issues.apache.org/jira/browse/DRILL-4974?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sudheesh Katkam updated DRILL-4974: --- Fix Version/s: (was: 1.8.0) 1.9.0 > NPE in FindPartitionConditions.analyzeCall() for 'holistic' expressions > --- > > Key: DRILL-4974 > URL: https://issues.apache.org/jira/browse/DRILL-4974 > Project: Apache Drill > Issue Type: Bug > Components: Execution - Relational Operators >Affects Versions: 1.6.0, 1.7.0, 1.8.0 >Reporter: Karthikeyan Manivannan >Assignee: Gautam Kumar Parai > Fix For: 1.9.0 > > Original Estimate: 2h > Remaining Estimate: 2h > > The following query can cause an NPE in FindPartitionConditions.analyzeCall() > if the fileSize column is a partitioned column. > SELECT fileSize FROM dfs.`/drill-data/data/` WHERE compoundId LIKE > 'FOO-1234567%' > This is because, the LIKE is treated as a holistic expression in > FindPartitionConditions.analyzeCall(), causing opStack to be empty, thus > causing opStack.peek() to return a NULL value. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (DRILL-4981) TPC-DS Query 75 fails on MapR-DB JSON Tables
Abhishek Girish created DRILL-4981:
--
Summary: TPC-DS Query 75 fails on MapR-DB JSON Tables
Key: DRILL-4981
URL: https://issues.apache.org/jira/browse/DRILL-4981
Project: Apache Drill
Issue Type: Bug
Components: Storage - MapRDB
Affects Versions: 1.9.0
Reporter: Abhishek Girish
Assignee: Smidth Panchamia
TPC-DS Query 75 fails on MapR-DB JSON Tables, but succeeds on Text, Parquet &
JSON File formats.
I'll work on a simpler repro. Find the original query & error below:
{code}
WITH all_sales
AS (SELECT d_year,
i_brand_id,
i_class_id,
i_category_id,
i_manufact_id,
Sum(sales_cnt) AS sales_cnt,
Sum(sales_amt) AS sales_amt
FROM (SELECT d_year,
i_brand_id,
i_class_id,
i_category_id,
i_manufact_id,
cs_quantity - COALESCE(cr_return_quantity, 0)AS
sales_cnt,
cs_ext_sales_price - COALESCE(cr_return_amount, 0.0) AS
sales_amt
FROM catalog_sales
JOIN item
ON i_item_sk = cs_item_sk
JOIN date_dim
ON d_date_sk = cs_sold_date_sk
LEFT JOIN catalog_returns
ON ( cs_order_number = cr_order_number
AND cs_item_sk = cr_item_sk )
WHERE i_category = 'Men'
UNION
SELECT d_year,
i_brand_id,
i_class_id,
i_category_id,
i_manufact_id,
ss_quantity - COALESCE(sr_return_quantity, 0) AS
sales_cnt,
ss_ext_sales_price - COALESCE(sr_return_amt, 0.0) AS
sales_amt
FROM store_sales
JOIN item
ON i_item_sk = ss_item_sk
JOIN date_dim
ON d_date_sk = ss_sold_date_sk
LEFT JOIN store_returns
ON ( ss_ticket_number = sr_ticket_number
AND ss_item_sk = sr_item_sk )
WHERE i_category = 'Men'
UNION
SELECT d_year,
i_brand_id,
i_class_id,
i_category_id,
i_manufact_id,
ws_quantity - COALESCE(wr_return_quantity, 0) AS
sales_cnt,
ws_ext_sales_price - COALESCE(wr_return_amt, 0.0) AS
sales_amt
FROM web_sales
JOIN item
ON i_item_sk = ws_item_sk
JOIN date_dim
ON d_date_sk = ws_sold_date_sk
LEFT JOIN web_returns
ON ( ws_order_number = wr_order_number
AND ws_item_sk = wr_item_sk )
WHERE i_category = 'Men') sales_detail
GROUP BY d_year,
i_brand_id,
i_class_id,
i_category_id,
i_manufact_id)
SELECT prev_yr.d_yearAS prev_year,
curr_yr.d_yearAS year1,
curr_yr.i_brand_id,
curr_yr.i_class_id,
curr_yr.i_category_id,
curr_yr.i_manufact_id,
prev_yr.sales_cnt AS prev_yr_cnt,
curr_yr.sales_cnt AS curr_yr_cnt,
curr_yr.sales_cnt - prev_yr.sales_cnt AS sales_cnt_diff,
curr_yr.sales_amt - prev_yr.sales_amt AS sales_amt_diff
FROM all_sales curr_yr,
all_sales prev_yr
WHERE curr_yr.i_brand_id = prev_yr.i_brand_id
AND curr_yr.i_class_id = prev_yr.i_class_id
AND curr_yr.i_category_id = prev_yr.i_category_id
AND curr_yr.i_manufact_id = prev_yr.i_manufact_id
AND curr_yr.d_year = 2002
AND prev_yr.d_year = 2002 - 1
AND Cast(curr_yr.sales_cnt AS DECIMAL(17, 2)) / Cast(prev_yr.sales_cnt
AS
DECIMAL(17, 2))
< 0.9
ORDER BY sales_cnt_diff
LIMIT 100
{code}
Error:
{code}
Failed with exception
java.sql.SQLException: SYSTEM ERROR: NullPointerException
[Error Id: 128bb62b-a6af-4b8f-90d5-d9f516b9e3d4 on atsqa6c83
[jira] [Commented] (DRILL-4927) Add support for Null Equality Joins
[
https://issues.apache.org/jira/browse/DRILL-4927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15616941#comment-15616941
]
ASF GitHub Bot commented on DRILL-4927:
---
Github user amansinha100 commented on a diff in the pull request:
https://github.com/apache/drill/pull/635#discussion_r85625856
--- Diff:
exec/java-exec/src/test/java/org/apache/drill/TestJoinNullable.java ---
@@ -493,4 +496,81 @@ public void withNullEqualAdditionFilter() throws
Exception {
.go();
}
+ @Test
+ public void withMixedEqualAndIsNotDistinctHashJoin() throws Exception {
--- End diff --
Could you add a test with the predicate of type:
t1.key = t2.key AND ((t1.data=t2.data) OR (t1.data IS NULL AND t2.data
IS NULL))
where the right side of the AND gets converted to IS NOT DISTINCT FROM.
> Add support for Null Equality Joins
> ---
>
> Key: DRILL-4927
> URL: https://issues.apache.org/jira/browse/DRILL-4927
> Project: Apache Drill
> Issue Type: Improvement
> Components: Query Planning & Optimization
>Affects Versions: 1.8.0
>Reporter: Roman
>Assignee: Aman Sinha
> Labels: doc-impacting
> Fix For: 1.9.0
>
>
> Join with an equality condition which allows null=null fails. For example, if
> we use some of this queries:
> {code:sql}
> select ... FROM t1, t2 WHERE t1.c1 = t2.c2 OR (t1.c1 IS NULL AND t2.c2 IS
> NULL);
> select ... FROM t1 INNER JOIN t2 ON t1.c1 = t2.c2 OR (t1.c1 IS NULL AND
> t2.c2 IS NULL);
> {code}
> we got "UNSUPPORTED_OPERATION ERROR". We should add support for this option.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-4420) C client and ODBC driver should move to using the new metadata methods provided by DRILL-4385
[ https://issues.apache.org/jira/browse/DRILL-4420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15617013#comment-15617013 ] ASF GitHub Bot commented on DRILL-4420: --- Github user laurentgo commented on the issue: https://github.com/apache/drill/pull/602 Branch updated with only commits associated with a JIRA (everything else has been merged into commit for DRILL-4420) > C client and ODBC driver should move to using the new metadata methods > provided by DRILL-4385 > - > > Key: DRILL-4420 > URL: https://issues.apache.org/jira/browse/DRILL-4420 > Project: Apache Drill > Issue Type: Sub-task >Reporter: Jacques Nadeau > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (DRILL-4979) Make dataport configurable
[ https://issues.apache.org/jira/browse/DRILL-4979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15617475#comment-15617475 ] Parth Chandra commented on DRILL-4979: -- This seems like a reasonable approach. (You can follow the pattern used for the control port in ControllerImpl.start() ) > Make dataport configurable > -- > > Key: DRILL-4979 > URL: https://issues.apache.org/jira/browse/DRILL-4979 > Project: Apache Drill > Issue Type: New Feature > Components: Server >Affects Versions: 1.8.0 > Environment: Scheduling drillbits with Apache Mesos+Aurora >Reporter: Uwe L. Korn > > Currently the dataport of a Drillbit is fixed to +1 on the control port. In a > dynamic execution environment like Apache Mesos+Aurora, each port is > allocated by the scheduler and then passed on to the application process. > There is no possibility or guarantee to allocate two consecutive ports. > Therefore, to run Drill in this environment, the dataport of the drillbit > also needs to configurable by the scheduler. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
