[GitHub] drill pull request: DRILL-4287: During initial DrillTable creation...
Github user amansinha100 commented on the pull request: https://github.com/apache/drill/pull/376#issuecomment-185074140 Simplified the state management in FileSelection. @jacques-n is this close enough to what you intended ? Also, want to note that this JIRA is motivated by a performance enhancement, so I haven't added a new unit test. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] drill pull request: DRILL-4287: During initial DrillTable creation...
Github user jacques-n commented on a diff in the pull request: https://github.com/apache/drill/pull/376#discussion_r53121603 --- Diff: exec/java-exec/src/main/java/org/apache/drill/exec/store/dfs/FileSelection.java --- @@ -47,6 +49,14 @@ public List files; public final String selectionRoot; + private enum StatusType { +CHECKED_DIRS,// whether we have already checked for directories +HAS_DIRS,// whether directories were found in the selection +EXPANDED_DIRS// whether this selection has been expanded to files + } + + private final BitSet dirStatus; --- End diff -- Sorry I wasn't clearer. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Created] (DRILL-4392) CTAS with partition writes an internal field into generated parquet files
Jinfeng Ni created DRILL-4392: - Summary: CTAS with partition writes an internal field into generated parquet files Key: DRILL-4392 URL: https://issues.apache.org/jira/browse/DRILL-4392 Project: Apache Drill Issue Type: Bug Reporter: Jinfeng Ni Priority: Blocker On today's master branch: {code} select * from sys.version; +-+---+-++-++ | version | commit_id | commit_message|commit_time | build_email | build_time | +-+---+-++-++ | 1.5.0-SNAPSHOT | 9a3a5c4ff670a50a49f61f97dd838da59a12f976 | DRILL-4382: Remove dependency on drill-logical from vector package | 16.02.2016 @ 11:58:48 PST | j...@apache.org | 16.02.2016 @ 17:40:44 PST | +-+---+-++- {code} Parquet table created by Drill's CTAS statement has one internal field "P_A_R_T_I_T_I_O_N_C_O_M_P_A_R_A_T_O_R". This additional field would not impact non-star query, but would cause incorrect result for star query. {code} use dfs.tmp; create table nation_ctas partition by (n_regionkey) as select * from cp.`tpch/nation.parquet`; select * from dfs.tmp.nation_ctas limit 6; +--++--+-++ | n_nationkey | n_name | n_regionkey | n_comment | P_A_R_T_I_T_I_O_N_C_O_M_P_A_R_A_T_O_R | +--++--+-++ | 5| ETHIOPIA | 0| ven packages wake quickly. regu | true | | 15 | MOROCCO| 0| rns. blithely bold courts among the closely regular packages use furiously bold platelets? | false | | 14 | KENYA | 0| pending excuses haggle furiously deposits. pending, express pinto beans wake fluffily past t | false | | 0| ALGERIA| 0| haggle. carefully final deposits detect slyly agai | false | | 16 | MOZAMBIQUE | 0| s. ironic, unusual asymptotes wake blithely r | false | | 24 | UNITED STATES | 1| y final packages. slow foxes cajole quickly. quickly silent platelets breach ironic accounts. unusual pinto be | true {code} This basically breaks all the parquet files created by Drill's CTAS with partition support. Also, it will also fail one of the Pre-commit functional test [1] [1] https://github.com/mapr/drill-test-framework/blob/master/framework/resources/Functional/ctas/ctas_auto_partition/general/data/drill3361.q -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] drill pull request: DRILL-4287: During initial DrillTable creation...
Github user amansinha100 commented on a diff in the pull request: https://github.com/apache/drill/pull/376#discussion_r53120105 --- Diff: exec/java-exec/src/main/java/org/apache/drill/exec/store/dfs/FileSelection.java --- @@ -47,6 +49,14 @@ public List files; public final String selectionRoot; + private enum StatusType { +CHECKED_DIRS,// whether we have already checked for directories +HAS_DIRS,// whether directories were found in the selection +EXPANDED_DIRS// whether this selection has been expanded to files + } + + private final BitSet dirStatus; --- End diff -- Yeah, there was probably some disconnect since your previous comment was a little succinct :). I was trying to keep track of multiple states, but agree it can be simplified. I may not need to use enumerator constructors. I will post an updated patch once I get a clean test run. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Resolved] (DRILL-4354) Remove sessions in anonymous (user auth disabled) mode in WebUI server
[ https://issues.apache.org/jira/browse/DRILL-4354?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jacques Nadeau resolved DRILL-4354. --- Resolution: Fixed > Remove sessions in anonymous (user auth disabled) mode in WebUI server > -- > > Key: DRILL-4354 > URL: https://issues.apache.org/jira/browse/DRILL-4354 > Project: Apache Drill > Issue Type: Bug > Components: Server >Affects Versions: 1.5.0 >Reporter: Venki Korukanti >Assignee: Venki Korukanti > Fix For: 1.5.0 > > > Currently we open anonymous sessions when user auth disabled. These sessions > are cleaned up when they expire (controlled by boot config > {{drill.exec.http.session_max_idle_secs}}). This may lead to unnecessary > resource accumulation. This JIRA is to remove anonymous sessions and only > have sessions when user authentication is enabled. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[ANNOUNCE] Apache Drill 1.5.0 released
On behalf of *Apache* *Drill* community, I am happy to *announce* the release of *Apache* *Drill* 1.5.0. The source and binary artifacts are available at [1] Review a complete list of fixes and enhancements at [2] This release of *Drill* fixes many issues and introduces a number of enhancements, including the following highlights: Web Authentication Drill 1.5 extends Drill user authentication to the Web Console and underlying REST API so administrators can control the extent of access to the Web Console and REST API client applications. Kudu Support Drill now includes experimental support for querying the Apache Kudu (incubating) scalable columnar database. Improved Memory Allocator Drill uses a new allocator that improves an operator’s use of direct memory and tracks the memory use more accurately. Configurable Caching of Hive Metadata You can now configure the TTL for the Hive metadata client cache depending on how frequently the Hive metadata is updated. Thanks to everyone in the community who contributed in this release. [1] http://drill.apache.org/download/ [2] http://drill.apache.org/docs/apache-drill-1-5-0-release-notes/ - Jason
[GitHub] drill pull request: DRILL-4390: Uses Resource where Drill favicon ...
Github user jacques-n commented on the pull request: https://github.com/apache/drill/pull/378#issuecomment-185001567 LGTM +1 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Created] (DRILL-4391) browsing metadata via SQLSquirrel shows Postgres indexes, primary and foreign keys as tables
N Campbell created DRILL-4391: - Summary: browsing metadata via SQLSquirrel shows Postgres indexes, primary and foreign keys as tables Key: DRILL-4391 URL: https://issues.apache.org/jira/browse/DRILL-4391 Project: Apache Drill Issue Type: Bug Components: Metadata Affects Versions: 1.4.0 Reporter: N Campbell Apache Drill has storage defined to access a Postgres database A schema in the database has several tables which either have indexes, primary keys, foreign keys or combination of them all. When SQLSquirrel presents metadata from the Drill JDBC driver the list of tables will include entries which correspond to the indexes, primary or foreign keys in the schema. The implication being that non-standard JDBC metadata methods to obtain information is being used. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] drill pull request: DRILL-4390: Uses Resource where Drill favicon ...
GitHub user laurentgo opened a pull request: https://github.com/apache/drill/pull/378 DRILL-4390: Uses Resource where Drill favicon is located for static assets Drill Webserver uses the first jar containing a rest/static directory to find its static assets. In case of another jar containing this directory, it might cause the webserver to return 404 errors. This configures the server to use the resource containing the Drill favicon as the place to look for all static resources. You can merge this pull request into a Git repository by running: $ git pull https://github.com/laurentgo/drill laurent/DRILL-4390 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/drill/pull/378.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #378 commit a97c5506cfa800b37258962a8d049fd1b72852e4 Author: Laurent Goujon Date: 2016-02-17T00:47:17Z DRILL-4390: Uses Resource where Drill favicon is located for static assets Drill Webserver uses the first jar containing a rest/static directory to find its static assets. In case of another jar containing this directory, it might cause the webserver to return 404 errors. This configures the server to use the resource containing the Drill favicon as the place to look for all static resources. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Created] (DRILL-4390) Drill webserver cannot get static assets if another jar contains a rest/static directory
Laurent Goujon created DRILL-4390: - Summary: Drill webserver cannot get static assets if another jar contains a rest/static directory Key: DRILL-4390 URL: https://issues.apache.org/jira/browse/DRILL-4390 Project: Apache Drill Issue Type: Bug Components: Web Server Reporter: Laurent Goujon Assignee: Laurent Goujon Priority: Minor WebServer is configured to serve static/ URL from a resource containing "rest/static" but if another jar than drill java-exec jar contains the same directory, all queries to static/ will result most likely in 404s. I propose to be more precise regarding the resource to use, and find the one containing the Drill favicon under rest/static. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] drill pull request: Drill 4372 for review
GitHub user hsuanyi opened a pull request: https://github.com/apache/drill/pull/377 Drill 4372 for review @jinfengni, @amansinha100 I just updated and rebased the initial patch. Can you start review it? You can merge this pull request into a Git repository by running: $ git pull https://github.com/hsuanyi/incubator-drill DRILL-4372_For_Review Alternatively you can review and apply these changes as the patch at: https://github.com/apache/drill/pull/377.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #377 commit 9f6fdca69ef3c652bce0961dd76ad8bbdf1cfaa9 Author: Sudheesh Katkam Date: 2015-12-22T04:38:59Z Validate Drill functions (argument and return types). WIP. commit 3c9bb47f13488bca5b3c528c1c34af4a9b22c5e7 Author: Hsuan-Yi Chu Date: 2015-12-30T22:21:10Z DRILL-4372: Expose the functions return type to Drill --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] drill pull request: DRILL-4372: Expose the functions return type t...
Github user hsuanyi commented on the pull request: https://github.com/apache/drill/pull/370#issuecomment-184940494 Will send out a rebased and updated code shortly --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] drill pull request: DRILL-4372: Expose the functions return type t...
Github user hsuanyi closed the pull request at: https://github.com/apache/drill/pull/370 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] drill pull request: DRILL-4287: During initial DrillTable creation...
Github user jacques-n commented on a diff in the pull request: https://github.com/apache/drill/pull/376#discussion_r53083386 --- Diff: exec/java-exec/src/main/java/org/apache/drill/exec/store/dfs/FileSelection.java --- @@ -47,6 +49,14 @@ public List files; public final String selectionRoot; + private enum StatusType { +CHECKED_DIRS,// whether we have already checked for directories +HAS_DIRS,// whether directories were found in the selection +EXPANDED_DIRS// whether this selection has been expanded to files + } + + private final BitSet dirStatus; --- End diff -- You took this differently than I meant it. My proposal was that FileSelection has various states: NOT_CHECKED_DIRS => (HAS_DIRS | NO_DIRS) => EXPANDED Doesn this lifecycle describe the state of FileSelection? This way you don't have the multi-state-management problem you currently have below with this kind of construct: fileSel.setExpanded(true); fileSel.setCheckedForDirectories(true); fileSel.setHasDirectories(false); // already expanded For each of the enumerations, we can return the right booleans that you need through enumerator constructors. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] drill pull request: DRILL-4383: Allow custom configurations to be ...
Github user jacques-n commented on the pull request: https://github.com/apache/drill/pull/375#issuecomment-184883511 that lgtm => +1 :) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] drill pull request: DRILL-4275: create TransientStore for short-li...
Github user jacques-n commented on the pull request: https://github.com/apache/drill/pull/374#issuecomment-184883435 I'm +1 on my patch assuming the manually testing that Sudheesh is asking about. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] drill pull request: DRILL-4382: Remove dependency on drill-logical...
Github user asfgit closed the pull request at: https://github.com/apache/drill/pull/373 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] drill pull request: DRILL-4287: During initial DrillTable creation...
Github user jinfengni commented on a diff in the pull request: https://github.com/apache/drill/pull/376#discussion_r53068818 --- Diff: exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/ParquetGroupScan.java --- @@ -529,6 +549,36 @@ public long getRowCount() { } + + // Create and return a new file selection based on reading the metadata cache file. + // This function also initializes a few of ParquetGroupScan's fields as appropriate. + private FileSelection + initFromMetadataCache(DrillFileSystem fs, FileSelection selection) throws IOException { +FileStatus metaRootDir = selection.getFirstPath(fs); +Path metaFilePath = new Path(metaRootDir.getPath(), Metadata.METADATA_FILENAME); + +// get (and set internal field) the metadata for the directory by reading the metadata file +this.parquetTableMetadata = Metadata.readBlockMeta(fs, metaFilePath.toString()); +List fileNames = Lists.newArrayList(); +for (Metadata.ParquetFileMetadata file : parquetTableMetadata.getFiles()) { + fileNames.add(file.getPath()); +} +// when creating the file selection, set the selection root in the form /a/b instead of +// file:/a/b. The reason is that the file names above have been created in the form +// /a/b/c.parquet and the format of the selection root must match that of the file names +// otherwise downstream operations such as partition pruning can break. +final Path metaRootPath = Path.getPathWithoutSchemeAndAuthority(metaRootDir.getPath()); +this.selectionRoot = metaRootPath.toString(); + +// Use the FileSelection constructor directly here instead of the FileSelection.create() method +// because create() changes the root to include the scheme and authority; In future, if create() +// is the preferred way to instantiate a file selection, we may need to do something different... +FileSelection newSelection = new FileSelection(selection.getStatuses(fs), fileNames, metaRootPath.toString()); --- End diff -- I see. Agreed that we should prioritize DRILL-4381 to address this inconsistency. Overall, the revised patch looks good to me. +1 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] drill pull request: DRILL-4287: During initial DrillTable creation...
Github user amansinha100 commented on the pull request: https://github.com/apache/drill/pull/376#issuecomment-184848577 Updated PR after addressing review comment from @jacques-n --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] drill pull request: DRILL-4275: create TransientStore for short-li...
Github user sudheeshkatkam commented on the pull request: https://github.com/apache/drill/pull/374#issuecomment-184833965 Since there are no unit tests for web UI, can you do some manual testing to ensure there are no regressions? Is there a simple way to test all persistent and transient stores? I see tests for a few implementations only. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] drill pull request: DRILL-4275: create TransientStore for short-li...
Github user sudheeshkatkam commented on a diff in the pull request: https://github.com/apache/drill/pull/374#discussion_r53059667 --- Diff: exec/java-exec/src/main/java/org/apache/drill/exec/store/sys/store/provider/CachingPersistentStoreProvider.java --- @@ -0,0 +1,77 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.drill.exec.store.sys.store.provider; + +import java.util.List; +import java.util.concurrent.ConcurrentMap; + +import com.google.common.collect.Lists; +import com.google.common.collect.Maps; +import org.apache.drill.common.AutoCloseables; +import org.apache.drill.exec.exception.StoreException; +import org.apache.drill.exec.store.sys.PersistentStore; +import org.apache.drill.exec.store.sys.PersistentStoreConfig; +import org.apache.drill.exec.store.sys.PersistentStoreProvider; + +public class CachingPersistentStoreProvider extends BasePersistentStoreProvider { + private static final org.slf4j.Logger logger = org.slf4j.LoggerFactory.getLogger(CachingPersistentStoreProvider.class); + + private final ConcurrentMap, PersistentStore> storeCache = Maps.newConcurrentMap(); + private final PersistentStoreProvider provider; + + public CachingPersistentStoreProvider(PersistentStoreProvider provider) { +this.provider = provider; + } + + @SuppressWarnings("unchecked") + public PersistentStore getOrCreateStore(final PersistentStoreConfig config) throws StoreException { --- End diff -- override --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] drill pull request: DRILL-4287: During initial DrillTable creation...
Github user amansinha100 commented on a diff in the pull request: https://github.com/apache/drill/pull/376#discussion_r53056521 --- Diff: exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/ParquetGroupScan.java --- @@ -529,6 +549,36 @@ public long getRowCount() { } + + // Create and return a new file selection based on reading the metadata cache file. + // This function also initializes a few of ParquetGroupScan's fields as appropriate. + private FileSelection + initFromMetadataCache(DrillFileSystem fs, FileSelection selection) throws IOException { +FileStatus metaRootDir = selection.getFirstPath(fs); +Path metaFilePath = new Path(metaRootDir.getPath(), Metadata.METADATA_FILENAME); + +// get (and set internal field) the metadata for the directory by reading the metadata file +this.parquetTableMetadata = Metadata.readBlockMeta(fs, metaFilePath.toString()); +List fileNames = Lists.newArrayList(); +for (Metadata.ParquetFileMetadata file : parquetTableMetadata.getFiles()) { + fileNames.add(file.getPath()); +} +// when creating the file selection, set the selection root in the form /a/b instead of +// file:/a/b. The reason is that the file names above have been created in the form +// /a/b/c.parquet and the format of the selection root must match that of the file names +// otherwise downstream operations such as partition pruning can break. +final Path metaRootPath = Path.getPathWithoutSchemeAndAuthority(metaRootDir.getPath()); +this.selectionRoot = metaRootPath.toString(); + +// Use the FileSelection constructor directly here instead of the FileSelection.create() method +// because create() changes the root to include the scheme and authority; In future, if create() +// is the preferred way to instantiate a file selection, we may need to do something different... +FileSelection newSelection = new FileSelection(selection.getStatuses(fs), fileNames, metaRootPath.toString()); --- End diff -- @jinfengni, as part of this PR I moved the expansion from ParquetFormatPlugin to ParquetGroupScan for the metadata cache but did not change the call to create FileSelection. I agree that the inconsistency could be a problem. As @adeneche points out, DRILL-4380 has the background for the change, but we really need to fix the FileSelection.create() interface, which means we should prioritize DRILL-4381. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
Hangout happening now!
Join us to hear the latest news on Drill, anyone with an interest in Drill is welcome to join. https://plus.google.com/hangouts/_/dremio.com/drillhangout - Jason
[jira] [Created] (DRILL-4389) ResultSetMetadata for DatabaseMetadata.getSchemas returns TABLE_CAT and not TABLE_CATALOG
N Campbell created DRILL-4389: - Summary: ResultSetMetadata for DatabaseMetadata.getSchemas returns TABLE_CAT and not TABLE_CATALOG Key: DRILL-4389 URL: https://issues.apache.org/jira/browse/DRILL-4389 Project: Apache Drill Issue Type: Bug Components: Client - JDBC Affects Versions: 1.4.0 Reporter: N Campbell Priority: Minor If you inspect the ResultSetMetadata returned by DatabaseMetadata.getSchemas it returns a column TABLE_CAT instead of TABLE_CATALOG https://docs.oracle.com/javase/7/docs/api/java/sql/DatabaseMetaData.html#getSchemas() -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] drill pull request: DRILL-4287: During initial DrillTable creation...
Github user amansinha100 commented on a diff in the pull request: https://github.com/apache/drill/pull/376#discussion_r53047947 --- Diff: exec/java-exec/src/main/java/org/apache/drill/exec/store/dfs/FileSelection.java --- @@ -68,6 +79,7 @@ protected FileSelection(final FileSelection selection) { this.statuses = selection.statuses; this.files = selection.files; this.selectionRoot = selection.selectionRoot; +this.dirStatus = new BitSet(StatusType.values().length); --- End diff -- Yes, it should...I will change it. Thanks for catching. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Created] (DRILL-4388) A large # of DatabaseMetadata methods return can not access a member of class org.apache.drill.jdbc.impl.DrillDatabaseMetaDataImpl with modifiers "public"
N Campbell created DRILL-4388: - Summary: A large # of DatabaseMetadata methods return can not access a member of class org.apache.drill.jdbc.impl.DrillDatabaseMetaDataImpl with modifiers "public" Key: DRILL-4388 URL: https://issues.apache.org/jira/browse/DRILL-4388 Project: Apache Drill Issue Type: Bug Components: Client - JDBC Affects Versions: 1.4.0 Environment: Drill 1.4.1 Reporter: N Campbell If you were to enumerate all the methods re "get" and "is" on the DatabaseMetadata interface, you will find a large # which cause can not access a member of class org.apache.drill.jdbc.impl.DrillDatabaseMetaDataImpl with modifiers "public" -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] drill pull request: DRILL-4287: During initial DrillTable creation...
Github user adeneche commented on a diff in the pull request: https://github.com/apache/drill/pull/376#discussion_r53043516 --- Diff: exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/ParquetGroupScan.java --- @@ -529,6 +549,36 @@ public long getRowCount() { } + + // Create and return a new file selection based on reading the metadata cache file. + // This function also initializes a few of ParquetGroupScan's fields as appropriate. + private FileSelection + initFromMetadataCache(DrillFileSystem fs, FileSelection selection) throws IOException { +FileStatus metaRootDir = selection.getFirstPath(fs); +Path metaFilePath = new Path(metaRootDir.getPath(), Metadata.METADATA_FILENAME); + +// get (and set internal field) the metadata for the directory by reading the metadata file +this.parquetTableMetadata = Metadata.readBlockMeta(fs, metaFilePath.toString()); +List fileNames = Lists.newArrayList(); +for (Metadata.ParquetFileMetadata file : parquetTableMetadata.getFiles()) { + fileNames.add(file.getPath()); +} +// when creating the file selection, set the selection root in the form /a/b instead of +// file:/a/b. The reason is that the file names above have been created in the form +// /a/b/c.parquet and the format of the selection root must match that of the file names +// otherwise downstream operations such as partition pruning can break. +final Path metaRootPath = Path.getPathWithoutSchemeAndAuthority(metaRootDir.getPath()); +this.selectionRoot = metaRootPath.toString(); + +// Use the FileSelection constructor directly here instead of the FileSelection.create() method +// because create() changes the root to include the scheme and authority; In future, if create() +// is the preferred way to instantiate a file selection, we may need to do something different... +FileSelection newSelection = new FileSelection(selection.getStatuses(fs), fileNames, metaRootPath.toString()); --- End diff -- Unfortunately, trying to fix this will introduce a performance regression, see [DRILL-4380](https://issues.apache.org/jira/browse/DRILL-4380) for more details. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Created] (DRILL-4387) Fix execution side when it handles skip
Jinfeng Ni created DRILL-4387: - Summary: Fix execution side when it handles skip Key: DRILL-4387 URL: https://issues.apache.org/jira/browse/DRILL-4387 Project: Apache Drill Issue Type: Bug Reporter: Jinfeng Ni -- This message was sent by Atlassian JIRA (v6.3.4#6332)