[jira] [Commented] (DRILL-4530) Improve metadata cache performance for queries with single partition
[
https://issues.apache.org/jira/browse/DRILL-4530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15383359#comment-15383359
]
ASF GitHub Bot commented on DRILL-4530:
---
Github user asfgit closed the pull request at:
https://github.com/apache/drill/pull/519
> Improve metadata cache performance for queries with single partition
> -
>
> Key: DRILL-4530
> URL: https://issues.apache.org/jira/browse/DRILL-4530
> Project: Apache Drill
> Issue Type: Improvement
> Components: Query Planning & Optimization
>Affects Versions: 1.6.0
>Reporter: Aman Sinha
>Assignee: Jinfeng Ni
> Fix For: Future
>
>
> Consider two types of queries which are run with Parquet metadata caching:
> {noformat}
> query 1:
> SELECT col FROM `A/B/C`;
> query 2:
> SELECT col FROM `A` WHERE dir0 = 'B' AND dir1 = 'C';
> {noformat}
> For a certain dataset, the query1 elapsed time is 1 sec whereas query2
> elapsed time is 9 sec even though both are accessing the same amount of data.
> The user expectation is that they should perform roughly the same. The main
> difference comes from reading the bigger metadata cache file at the root
> level 'A' for query2 and then applying the partitioning filter. query1 reads
> a much smaller metadata cache file at the subdirectory level.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-4530) Improve metadata cache performance for queries with single partition
[
https://issues.apache.org/jira/browse/DRILL-4530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15382694#comment-15382694
]
ASF GitHub Bot commented on DRILL-4530:
---
Github user jinfengni commented on the issue:
https://github.com/apache/drill/pull/519
+1
LGTM.
> Improve metadata cache performance for queries with single partition
> -
>
> Key: DRILL-4530
> URL: https://issues.apache.org/jira/browse/DRILL-4530
> Project: Apache Drill
> Issue Type: Improvement
> Components: Query Planning & Optimization
>Affects Versions: 1.6.0
>Reporter: Aman Sinha
>Assignee: Jinfeng Ni
> Fix For: Future
>
>
> Consider two types of queries which are run with Parquet metadata caching:
> {noformat}
> query 1:
> SELECT col FROM `A/B/C`;
> query 2:
> SELECT col FROM `A` WHERE dir0 = 'B' AND dir1 = 'C';
> {noformat}
> For a certain dataset, the query1 elapsed time is 1 sec whereas query2
> elapsed time is 9 sec even though both are accessing the same amount of data.
> The user expectation is that they should perform roughly the same. The main
> difference comes from reading the bigger metadata cache file at the root
> level 'A' for query2 and then applying the partitioning filter. query1 reads
> a much smaller metadata cache file at the subdirectory level.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-4530) Improve metadata cache performance for queries with single partition
[
https://issues.apache.org/jira/browse/DRILL-4530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15380438#comment-15380438
]
ASF GitHub Bot commented on DRILL-4530:
---
Github user amansinha100 commented on the issue:
https://github.com/apache/drill/pull/519
@jinfengni , thanks for the review. I have addressed your review
comments. Pls take another look.
> Improve metadata cache performance for queries with single partition
> -
>
> Key: DRILL-4530
> URL: https://issues.apache.org/jira/browse/DRILL-4530
> Project: Apache Drill
> Issue Type: Improvement
> Components: Query Planning & Optimization
>Affects Versions: 1.6.0
>Reporter: Aman Sinha
>Assignee: Aman Sinha
> Fix For: Future
>
>
> Consider two types of queries which are run with Parquet metadata caching:
> {noformat}
> query 1:
> SELECT col FROM `A/B/C`;
> query 2:
> SELECT col FROM `A` WHERE dir0 = 'B' AND dir1 = 'C';
> {noformat}
> For a certain dataset, the query1 elapsed time is 1 sec whereas query2
> elapsed time is 9 sec even though both are accessing the same amount of data.
> The user expectation is that they should perform roughly the same. The main
> difference comes from reading the bigger metadata cache file at the root
> level 'A' for query2 and then applying the partitioning filter. query1 reads
> a much smaller metadata cache file at the subdirectory level.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-4530) Improve metadata cache performance for queries with single partition
[
https://issues.apache.org/jira/browse/DRILL-4530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15380437#comment-15380437
]
ASF GitHub Bot commented on DRILL-4530:
---
Github user amansinha100 commented on a diff in the pull request:
https://github.com/apache/drill/pull/519#discussion_r71059510
--- Diff:
exec/java-exec/src/main/java/org/apache/drill/exec/planner/logical/partition/PruneScanRule.java
---
@@ -143,6 +144,11 @@ public static final RelOptRule
getDirFilterOnScan(OptimizerRulesContext optimize
}
protected void doOnMatch(RelOptRuleCall call, Filter filterRel, Project
projectRel, TableScan scanRel) {
+if (wasAllPartitionsPruned) {
--- End diff --
Pls see commit #ce509af for the changes. Since the 'all partitions pruned'
is a special case, I had to keep track of this state and propagate it during
the new FileSelection creation after partition pruning. Thus, if the first
phase directory pruning eliminated all partitions and we re-added 1 for schema
purposes, this directory selection will be eventually seen by the
ParquetGroupScan which will examine a flag 'wasAllPartitionsPruned' and decide
to retrieve only 1 file from this directory.
> Improve metadata cache performance for queries with single partition
> -
>
> Key: DRILL-4530
> URL: https://issues.apache.org/jira/browse/DRILL-4530
> Project: Apache Drill
> Issue Type: Improvement
> Components: Query Planning & Optimization
>Affects Versions: 1.6.0
>Reporter: Aman Sinha
>Assignee: Aman Sinha
> Fix For: Future
>
>
> Consider two types of queries which are run with Parquet metadata caching:
> {noformat}
> query 1:
> SELECT col FROM `A/B/C`;
> query 2:
> SELECT col FROM `A` WHERE dir0 = 'B' AND dir1 = 'C';
> {noformat}
> For a certain dataset, the query1 elapsed time is 1 sec whereas query2
> elapsed time is 9 sec even though both are accessing the same amount of data.
> The user expectation is that they should perform roughly the same. The main
> difference comes from reading the bigger metadata cache file at the root
> level 'A' for query2 and then applying the partitioning filter. query1 reads
> a much smaller metadata cache file at the subdirectory level.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-4530) Improve metadata cache performance for queries with single partition
[
https://issues.apache.org/jira/browse/DRILL-4530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15380432#comment-15380432
]
ASF GitHub Bot commented on DRILL-4530:
---
Github user amansinha100 commented on a diff in the pull request:
https://github.com/apache/drill/pull/519#discussion_r71059395
--- Diff:
exec/java-exec/src/test/java/org/apache/drill/exec/store/parquet/TestParquetMetadataCache.java
---
@@ -211,9 +217,76 @@ public void testNoSupportedError() throws Exception {
.go();
}
+ @Test // DRILL-4530
+ public void testDrill4530_1() throws Exception {
+// create metadata cache
+test(String.format("refresh table metadata dfs_test.`%s/%s`",
getDfsTestTmpSchemaLocation(), tableName2));
+checkForMetadataFile(tableName2);
+
+// run query and check correctness
+String query1 = String.format("select dir0, dir1, o_custkey,
o_orderdate from dfs_test.`%s/%s` " +
+" where dir0=1995 and dir1='Q3'",
+getDfsTestTmpSchemaLocation(), tableName2);
+int expectedRowCount = 20;
+int expectedNumFiles = 2;
+
+int actualRowCount = testSql(query1);
+assertEquals(expectedRowCount, actualRowCount);
+String numFilesPattern = "numFiles=" + expectedNumFiles;
+String usedMetaPattern = "usedMetadataFile=true";
+String cacheFileRootPattern = String.format("%s/%s/1995/Q3",
getDfsTestTmpSchemaLocation(), tableName2);
--- End diff --
I have changed the test pattern to match 'cacheFileRoot=' string.
> Improve metadata cache performance for queries with single partition
> -
>
> Key: DRILL-4530
> URL: https://issues.apache.org/jira/browse/DRILL-4530
> Project: Apache Drill
> Issue Type: Improvement
> Components: Query Planning & Optimization
>Affects Versions: 1.6.0
>Reporter: Aman Sinha
>Assignee: Aman Sinha
> Fix For: Future
>
>
> Consider two types of queries which are run with Parquet metadata caching:
> {noformat}
> query 1:
> SELECT col FROM `A/B/C`;
> query 2:
> SELECT col FROM `A` WHERE dir0 = 'B' AND dir1 = 'C';
> {noformat}
> For a certain dataset, the query1 elapsed time is 1 sec whereas query2
> elapsed time is 9 sec even though both are accessing the same amount of data.
> The user expectation is that they should perform roughly the same. The main
> difference comes from reading the bigger metadata cache file at the root
> level 'A' for query2 and then applying the partitioning filter. query1 reads
> a much smaller metadata cache file at the subdirectory level.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-4530) Improve metadata cache performance for queries with single partition
[
https://issues.apache.org/jira/browse/DRILL-4530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15380431#comment-15380431
]
ASF GitHub Bot commented on DRILL-4530:
---
Github user amansinha100 commented on a diff in the pull request:
https://github.com/apache/drill/pull/519#discussion_r71059388
--- Diff:
exec/java-exec/src/main/java/org/apache/drill/exec/planner/FileSystemPartitionDescriptor.java
---
@@ -212,8 +223,12 @@ public TableScan
createTableScan(List newPartitionLocation) t
}
if (scanRel instanceof DrillScanRel) {
- final FileSelection newFileSelection = new FileSelection(null,
newFiles, getBaseTableLocation());
- final FileGroupScan newGroupScan =
((FileGroupScan)((DrillScanRel)scanRel).getGroupScan()).clone(newFileSelection);
+// final FormatSelection formatSelection =
(FormatSelection)((DynamicDrillTable)scanRel.getTable()).getSelection();
+ final FormatSelection formatSelection =
(FormatSelection)table.getSelection();
+ final FileSelection newFileSelection = new FileSelection(null,
newFiles, getBaseTableLocation(),
+ cacheFileRoot, formatSelection.getSelection().getDirStatus());
+ final FileGroupScan newGroupScan =
+
((FileGroupScan)((DrillScanRel)scanRel).getGroupScan()).clone(newFileSelection,
cacheFileRoot);
--- End diff --
Fixed in commit #7ae8f7a
> Improve metadata cache performance for queries with single partition
> -
>
> Key: DRILL-4530
> URL: https://issues.apache.org/jira/browse/DRILL-4530
> Project: Apache Drill
> Issue Type: Improvement
> Components: Query Planning & Optimization
>Affects Versions: 1.6.0
>Reporter: Aman Sinha
>Assignee: Aman Sinha
> Fix For: Future
>
>
> Consider two types of queries which are run with Parquet metadata caching:
> {noformat}
> query 1:
> SELECT col FROM `A/B/C`;
> query 2:
> SELECT col FROM `A` WHERE dir0 = 'B' AND dir1 = 'C';
> {noformat}
> For a certain dataset, the query1 elapsed time is 1 sec whereas query2
> elapsed time is 9 sec even though both are accessing the same amount of data.
> The user expectation is that they should perform roughly the same. The main
> difference comes from reading the bigger metadata cache file at the root
> level 'A' for query2 and then applying the partitioning filter. query1 reads
> a much smaller metadata cache file at the subdirectory level.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-4530) Improve metadata cache performance for queries with single partition
[
https://issues.apache.org/jira/browse/DRILL-4530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15380429#comment-15380429
]
ASF GitHub Bot commented on DRILL-4530:
---
Github user amansinha100 commented on a diff in the pull request:
https://github.com/apache/drill/pull/519#discussion_r71059368
--- Diff:
exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/ParquetGroupScan.java
---
@@ -607,7 +607,14 @@ public long getRowCount() {
fileSet = Sets.newHashSet(fileNames);
}
-if (fileNames.isEmpty()) {
+List finalFileNames;
+if (fileSet != null) {
+ finalFileNames = Lists.newArrayList(fileSet);
--- End diff --
Commit #ef37b77 addresses this comment.
> Improve metadata cache performance for queries with single partition
> -
>
> Key: DRILL-4530
> URL: https://issues.apache.org/jira/browse/DRILL-4530
> Project: Apache Drill
> Issue Type: Improvement
> Components: Query Planning & Optimization
>Affects Versions: 1.6.0
>Reporter: Aman Sinha
>Assignee: Aman Sinha
> Fix For: Future
>
>
> Consider two types of queries which are run with Parquet metadata caching:
> {noformat}
> query 1:
> SELECT col FROM `A/B/C`;
> query 2:
> SELECT col FROM `A` WHERE dir0 = 'B' AND dir1 = 'C';
> {noformat}
> For a certain dataset, the query1 elapsed time is 1 sec whereas query2
> elapsed time is 9 sec even though both are accessing the same amount of data.
> The user expectation is that they should perform roughly the same. The main
> difference comes from reading the bigger metadata cache file at the root
> level 'A' for query2 and then applying the partitioning filter. query1 reads
> a much smaller metadata cache file at the subdirectory level.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-4530) Improve metadata cache performance for queries with single partition
[
https://issues.apache.org/jira/browse/DRILL-4530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15375739#comment-15375739
]
ASF GitHub Bot commented on DRILL-4530:
---
Github user amansinha100 commented on a diff in the pull request:
https://github.com/apache/drill/pull/519#discussion_r70706252
--- Diff:
exec/java-exec/src/main/java/org/apache/drill/exec/planner/logical/partition/PruneScanRule.java
---
@@ -143,6 +144,11 @@ public static final RelOptRule
getDirFilterOnScan(OptimizerRulesContext optimize
}
protected void doOnMatch(RelOptRuleCall call, Filter filterRel, Project
projectRel, TableScan scanRel) {
+if (wasAllPartitionsPruned) {
--- End diff --
Agree that this should not be an internal state. I had added it to
overcome an issue where the prunescan rule gets re-applied even when there are
not qualifying partitions (this happens because we always populate the
newPartitions list with at least 1 entry to produce a valid schema during
execution).
I tried removing this flag but will need some associated changes to prevent
the rule from being reapplied without termination. The old code (without my
patch) terminates the pruning because FileSelection.supportDirPruning() returns
FALSE when the selection has directories. Whereas, for this optimization I
want to do the pruning on directories first. So, let me think of a way to
address that.
> Improve metadata cache performance for queries with single partition
> -
>
> Key: DRILL-4530
> URL: https://issues.apache.org/jira/browse/DRILL-4530
> Project: Apache Drill
> Issue Type: Improvement
> Components: Query Planning & Optimization
>Affects Versions: 1.6.0
>Reporter: Aman Sinha
>Assignee: Aman Sinha
> Fix For: Future
>
>
> Consider two types of queries which are run with Parquet metadata caching:
> {noformat}
> query 1:
> SELECT col FROM `A/B/C`;
> query 2:
> SELECT col FROM `A` WHERE dir0 = 'B' AND dir1 = 'C';
> {noformat}
> For a certain dataset, the query1 elapsed time is 1 sec whereas query2
> elapsed time is 9 sec even though both are accessing the same amount of data.
> The user expectation is that they should perform roughly the same. The main
> difference comes from reading the bigger metadata cache file at the root
> level 'A' for query2 and then applying the partitioning filter. query1 reads
> a much smaller metadata cache file at the subdirectory level.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-4530) Improve metadata cache performance for queries with single partition
[
https://issues.apache.org/jira/browse/DRILL-4530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15374090#comment-15374090
]
ASF GitHub Bot commented on DRILL-4530:
---
Github user amansinha100 commented on a diff in the pull request:
https://github.com/apache/drill/pull/519#discussion_r70548891
--- Diff:
exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/ParquetGroupScan.java
---
@@ -607,7 +607,14 @@ public long getRowCount() {
fileSet = Sets.newHashSet(fileNames);
}
-if (fileNames.isEmpty()) {
+List finalFileNames;
+if (fileSet != null) {
+ finalFileNames = Lists.newArrayList(fileSet);
--- End diff --
Yes, I can try to keep only the fileSet and get rid of fileNames (although
I would still need the finalFileNames list since FileSelection only accepts a
List, not a Set).
> Improve metadata cache performance for queries with single partition
> -
>
> Key: DRILL-4530
> URL: https://issues.apache.org/jira/browse/DRILL-4530
> Project: Apache Drill
> Issue Type: Improvement
> Components: Query Planning & Optimization
>Affects Versions: 1.6.0
>Reporter: Aman Sinha
>Assignee: Aman Sinha
> Fix For: Future
>
>
> Consider two types of queries which are run with Parquet metadata caching:
> {noformat}
> query 1:
> SELECT col FROM `A/B/C`;
> query 2:
> SELECT col FROM `A` WHERE dir0 = 'B' AND dir1 = 'C';
> {noformat}
> For a certain dataset, the query1 elapsed time is 1 sec whereas query2
> elapsed time is 9 sec even though both are accessing the same amount of data.
> The user expectation is that they should perform roughly the same. The main
> difference comes from reading the bigger metadata cache file at the root
> level 'A' for query2 and then applying the partitioning filter. query1 reads
> a much smaller metadata cache file at the subdirectory level.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-4530) Improve metadata cache performance for queries with single partition
[
https://issues.apache.org/jira/browse/DRILL-4530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15373879#comment-15373879
]
ASF GitHub Bot commented on DRILL-4530:
---
Github user jinfengni commented on the issue:
https://github.com/apache/drill/pull/519
Other than prior comments, the pruning logic looks good to me.
> Improve metadata cache performance for queries with single partition
> -
>
> Key: DRILL-4530
> URL: https://issues.apache.org/jira/browse/DRILL-4530
> Project: Apache Drill
> Issue Type: Improvement
> Components: Query Planning & Optimization
>Affects Versions: 1.6.0
>Reporter: Aman Sinha
>Assignee: Aman Sinha
> Fix For: Future
>
>
> Consider two types of queries which are run with Parquet metadata caching:
> {noformat}
> query 1:
> SELECT col FROM `A/B/C`;
> query 2:
> SELECT col FROM `A` WHERE dir0 = 'B' AND dir1 = 'C';
> {noformat}
> For a certain dataset, the query1 elapsed time is 1 sec whereas query2
> elapsed time is 9 sec even though both are accessing the same amount of data.
> The user expectation is that they should perform roughly the same. The main
> difference comes from reading the bigger metadata cache file at the root
> level 'A' for query2 and then applying the partitioning filter. query1 reads
> a much smaller metadata cache file at the subdirectory level.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-4530) Improve metadata cache performance for queries with single partition
[
https://issues.apache.org/jira/browse/DRILL-4530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15373878#comment-15373878
]
ASF GitHub Bot commented on DRILL-4530:
---
Github user jinfengni commented on a diff in the pull request:
https://github.com/apache/drill/pull/519#discussion_r70537956
--- Diff:
exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/ParquetGroupScan.java
---
@@ -607,7 +607,14 @@ public long getRowCount() {
fileSet = Sets.newHashSet(fileNames);
}
-if (fileNames.isEmpty()) {
+List finalFileNames;
+if (fileSet != null) {
+ finalFileNames = Lists.newArrayList(fileSet);
--- End diff --
I see the need to remove duplicates and the confusion comes from old code.
Is it reasonable to change existing code, and only keep fileSet in all cases?
Keep fileName and fileSet seems to be a bit confusing.
> Improve metadata cache performance for queries with single partition
> -
>
> Key: DRILL-4530
> URL: https://issues.apache.org/jira/browse/DRILL-4530
> Project: Apache Drill
> Issue Type: Improvement
> Components: Query Planning & Optimization
>Affects Versions: 1.6.0
>Reporter: Aman Sinha
>Assignee: Aman Sinha
> Fix For: Future
>
>
> Consider two types of queries which are run with Parquet metadata caching:
> {noformat}
> query 1:
> SELECT col FROM `A/B/C`;
> query 2:
> SELECT col FROM `A` WHERE dir0 = 'B' AND dir1 = 'C';
> {noformat}
> For a certain dataset, the query1 elapsed time is 1 sec whereas query2
> elapsed time is 9 sec even though both are accessing the same amount of data.
> The user expectation is that they should perform roughly the same. The main
> difference comes from reading the bigger metadata cache file at the root
> level 'A' for query2 and then applying the partitioning filter. query1 reads
> a much smaller metadata cache file at the subdirectory level.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-4530) Improve metadata cache performance for queries with single partition
[
https://issues.apache.org/jira/browse/DRILL-4530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15373847#comment-15373847
]
ASF GitHub Bot commented on DRILL-4530:
---
Github user amansinha100 commented on a diff in the pull request:
https://github.com/apache/drill/pull/519#discussion_r70536259
--- Diff:
exec/java-exec/src/main/java/org/apache/drill/exec/planner/FileSystemPartitionDescriptor.java
---
@@ -212,8 +223,12 @@ public TableScan
createTableScan(List newPartitionLocation) t
}
if (scanRel instanceof DrillScanRel) {
- final FileSelection newFileSelection = new FileSelection(null,
newFiles, getBaseTableLocation());
- final FileGroupScan newGroupScan =
((FileGroupScan)((DrillScanRel)scanRel).getGroupScan()).clone(newFileSelection);
+// final FormatSelection formatSelection =
(FormatSelection)((DynamicDrillTable)scanRel.getTable()).getSelection();
+ final FormatSelection formatSelection =
(FormatSelection)table.getSelection();
+ final FileSelection newFileSelection = new FileSelection(null,
newFiles, getBaseTableLocation(),
+ cacheFileRoot, formatSelection.getSelection().getDirStatus());
+ final FileGroupScan newGroupScan =
+
((FileGroupScan)((DrillScanRel)scanRel).getGroupScan()).clone(newFileSelection,
cacheFileRoot);
--- End diff --
I agree it seems redundant second parameter. I don't recall why I needed
it at some point but let me see if I can get rid of it.
> Improve metadata cache performance for queries with single partition
> -
>
> Key: DRILL-4530
> URL: https://issues.apache.org/jira/browse/DRILL-4530
> Project: Apache Drill
> Issue Type: Improvement
> Components: Query Planning & Optimization
>Affects Versions: 1.6.0
>Reporter: Aman Sinha
>Assignee: Aman Sinha
> Fix For: Future
>
>
> Consider two types of queries which are run with Parquet metadata caching:
> {noformat}
> query 1:
> SELECT col FROM `A/B/C`;
> query 2:
> SELECT col FROM `A` WHERE dir0 = 'B' AND dir1 = 'C';
> {noformat}
> For a certain dataset, the query1 elapsed time is 1 sec whereas query2
> elapsed time is 9 sec even though both are accessing the same amount of data.
> The user expectation is that they should perform roughly the same. The main
> difference comes from reading the bigger metadata cache file at the root
> level 'A' for query2 and then applying the partitioning filter. query1 reads
> a much smaller metadata cache file at the subdirectory level.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-4530) Improve metadata cache performance for queries with single partition
[
https://issues.apache.org/jira/browse/DRILL-4530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15373801#comment-15373801
]
ASF GitHub Bot commented on DRILL-4530:
---
Github user jinfengni commented on a diff in the pull request:
https://github.com/apache/drill/pull/519#discussion_r70532442
--- Diff:
exec/java-exec/src/test/java/org/apache/drill/exec/store/parquet/TestParquetMetadataCache.java
---
@@ -211,9 +217,76 @@ public void testNoSupportedError() throws Exception {
.go();
}
+ @Test // DRILL-4530
+ public void testDrill4530_1() throws Exception {
+// create metadata cache
+test(String.format("refresh table metadata dfs_test.`%s/%s`",
getDfsTestTmpSchemaLocation(), tableName2));
+checkForMetadataFile(tableName2);
+
+// run query and check correctness
+String query1 = String.format("select dir0, dir1, o_custkey,
o_orderdate from dfs_test.`%s/%s` " +
+" where dir0=1995 and dir1='Q3'",
+getDfsTestTmpSchemaLocation(), tableName2);
+int expectedRowCount = 20;
+int expectedNumFiles = 2;
+
+int actualRowCount = testSql(query1);
+assertEquals(expectedRowCount, actualRowCount);
+String numFilesPattern = "numFiles=" + expectedNumFiles;
+String usedMetaPattern = "usedMetadataFile=true";
+String cacheFileRootPattern = String.format("%s/%s/1995/Q3",
getDfsTestTmpSchemaLocation(), tableName2);
--- End diff --
The verification of cacheFileRootPattern probably need put "cacheFileRoot="
as the prefix. Otherwise, the list of files in GroupScan will always find a
match for cacheFileRoot, right?
> Improve metadata cache performance for queries with single partition
> -
>
> Key: DRILL-4530
> URL: https://issues.apache.org/jira/browse/DRILL-4530
> Project: Apache Drill
> Issue Type: Improvement
> Components: Query Planning & Optimization
>Affects Versions: 1.6.0
>Reporter: Aman Sinha
>Assignee: Aman Sinha
> Fix For: Future
>
>
> Consider two types of queries which are run with Parquet metadata caching:
> {noformat}
> query 1:
> SELECT col FROM `A/B/C`;
> query 2:
> SELECT col FROM `A` WHERE dir0 = 'B' AND dir1 = 'C';
> {noformat}
> For a certain dataset, the query1 elapsed time is 1 sec whereas query2
> elapsed time is 9 sec even though both are accessing the same amount of data.
> The user expectation is that they should perform roughly the same. The main
> difference comes from reading the bigger metadata cache file at the root
> level 'A' for query2 and then applying the partitioning filter. query1 reads
> a much smaller metadata cache file at the subdirectory level.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-4530) Improve metadata cache performance for queries with single partition
[
https://issues.apache.org/jira/browse/DRILL-4530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15373772#comment-15373772
]
ASF GitHub Bot commented on DRILL-4530:
---
Github user jinfengni commented on a diff in the pull request:
https://github.com/apache/drill/pull/519#discussion_r70529269
--- Diff:
exec/java-exec/src/main/java/org/apache/drill/exec/planner/logical/partition/PruneScanRule.java
---
@@ -143,6 +144,11 @@ public static final RelOptRule
getDirFilterOnScan(OptimizerRulesContext optimize
}
protected void doOnMatch(RelOptRuleCall call, Filter filterRel, Project
projectRel, TableScan scanRel) {
+if (wasAllPartitionsPruned) {
--- End diff --
I feel this flag may lead to scenarios where pruning does not happen when
it should. The flag is PruneScanRule's internal variable. The same rule could
be applied to pruning for multiple tables. So, if PruneScanRule turns on this
flag for T1, then we might skip pruning logic for T2 later on, when this rule
is fired.
> Improve metadata cache performance for queries with single partition
> -
>
> Key: DRILL-4530
> URL: https://issues.apache.org/jira/browse/DRILL-4530
> Project: Apache Drill
> Issue Type: Improvement
> Components: Query Planning & Optimization
>Affects Versions: 1.6.0
>Reporter: Aman Sinha
>Assignee: Aman Sinha
> Fix For: Future
>
>
> Consider two types of queries which are run with Parquet metadata caching:
> {noformat}
> query 1:
> SELECT col FROM `A/B/C`;
> query 2:
> SELECT col FROM `A` WHERE dir0 = 'B' AND dir1 = 'C';
> {noformat}
> For a certain dataset, the query1 elapsed time is 1 sec whereas query2
> elapsed time is 9 sec even though both are accessing the same amount of data.
> The user expectation is that they should perform roughly the same. The main
> difference comes from reading the bigger metadata cache file at the root
> level 'A' for query2 and then applying the partitioning filter. query1 reads
> a much smaller metadata cache file at the subdirectory level.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-4530) Improve metadata cache performance for queries with single partition
[
https://issues.apache.org/jira/browse/DRILL-4530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15373748#comment-15373748
]
ASF GitHub Bot commented on DRILL-4530:
---
Github user jinfengni commented on a diff in the pull request:
https://github.com/apache/drill/pull/519#discussion_r70527515
--- Diff:
exec/java-exec/src/main/java/org/apache/drill/exec/planner/ParquetPartitionDescriptor.java
---
@@ -130,13 +131,13 @@ protected void createPartitionSublists() {
}
@Override
- public TableScan createTableScan(List
newPartitionLocation) throws Exception {
+ public TableScan createTableScan(List
newPartitionLocation, String cacheFileRoot) throws Exception {
--- End diff --
ParquetPartitionDescriptor does not supportsSinglePartOptimization. Should
it here throws unsupportedException, and implement
createTableScan(newPartitionLocation) only?
> Improve metadata cache performance for queries with single partition
> -
>
> Key: DRILL-4530
> URL: https://issues.apache.org/jira/browse/DRILL-4530
> Project: Apache Drill
> Issue Type: Improvement
> Components: Query Planning & Optimization
>Affects Versions: 1.6.0
>Reporter: Aman Sinha
>Assignee: Aman Sinha
> Fix For: Future
>
>
> Consider two types of queries which are run with Parquet metadata caching:
> {noformat}
> query 1:
> SELECT col FROM `A/B/C`;
> query 2:
> SELECT col FROM `A` WHERE dir0 = 'B' AND dir1 = 'C';
> {noformat}
> For a certain dataset, the query1 elapsed time is 1 sec whereas query2
> elapsed time is 9 sec even though both are accessing the same amount of data.
> The user expectation is that they should perform roughly the same. The main
> difference comes from reading the bigger metadata cache file at the root
> level 'A' for query2 and then applying the partitioning filter. query1 reads
> a much smaller metadata cache file at the subdirectory level.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-4530) Improve metadata cache performance for queries with single partition
[
https://issues.apache.org/jira/browse/DRILL-4530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15373718#comment-15373718
]
ASF GitHub Bot commented on DRILL-4530:
---
Github user jinfengni commented on a diff in the pull request:
https://github.com/apache/drill/pull/519#discussion_r70524663
--- Diff:
exec/java-exec/src/main/java/org/apache/drill/exec/planner/FileSystemPartitionDescriptor.java
---
@@ -212,8 +223,12 @@ public TableScan
createTableScan(List newPartitionLocation) t
}
if (scanRel instanceof DrillScanRel) {
- final FileSelection newFileSelection = new FileSelection(null,
newFiles, getBaseTableLocation());
- final FileGroupScan newGroupScan =
((FileGroupScan)((DrillScanRel)scanRel).getGroupScan()).clone(newFileSelection);
+// final FormatSelection formatSelection =
(FormatSelection)((DynamicDrillTable)scanRel.getTable()).getSelection();
+ final FormatSelection formatSelection =
(FormatSelection)table.getSelection();
+ final FileSelection newFileSelection = new FileSelection(null,
newFiles, getBaseTableLocation(),
+ cacheFileRoot, formatSelection.getSelection().getDirStatus());
+ final FileGroupScan newGroupScan =
+
((FileGroupScan)((DrillScanRel)scanRel).getGroupScan()).clone(newFileSelection,
cacheFileRoot);
--- End diff --
This newFileSelection already has cacheFileRoot. Do you think clone() only
need take FileSelection, since FileSelection has cacheFileRoot infor?
> Improve metadata cache performance for queries with single partition
> -
>
> Key: DRILL-4530
> URL: https://issues.apache.org/jira/browse/DRILL-4530
> Project: Apache Drill
> Issue Type: Improvement
> Components: Query Planning & Optimization
>Affects Versions: 1.6.0
>Reporter: Aman Sinha
>Assignee: Aman Sinha
> Fix For: Future
>
>
> Consider two types of queries which are run with Parquet metadata caching:
> {noformat}
> query 1:
> SELECT col FROM `A/B/C`;
> query 2:
> SELECT col FROM `A` WHERE dir0 = 'B' AND dir1 = 'C';
> {noformat}
> For a certain dataset, the query1 elapsed time is 1 sec whereas query2
> elapsed time is 9 sec even though both are accessing the same amount of data.
> The user expectation is that they should perform roughly the same. The main
> difference comes from reading the bigger metadata cache file at the root
> level 'A' for query2 and then applying the partitioning filter. query1 reads
> a much smaller metadata cache file at the subdirectory level.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-4530) Improve metadata cache performance for queries with single partition
[
https://issues.apache.org/jira/browse/DRILL-4530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15373716#comment-15373716
]
ASF GitHub Bot commented on DRILL-4530:
---
Github user amansinha100 commented on a diff in the pull request:
https://github.com/apache/drill/pull/519#discussion_r70524553
--- Diff:
exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/ParquetGroupScan.java
---
@@ -607,7 +607,14 @@ public long getRowCount() {
fileSet = Sets.newHashSet(fileNames);
}
-if (fileNames.isEmpty()) {
+List finalFileNames;
+if (fileSet != null) {
+ finalFileNames = Lists.newArrayList(fileSet);
--- End diff --
Actually, fileNames can contain duplicates whereas fileSet has a unique
list. fileSet was already present in the code earlier and used by
removeUnneededRowGroups(). I am leveraging fileSet because when expanding
directories it may be possible to end up with duplicate files which we don't
want. I could perhaps add a comment here.
> Improve metadata cache performance for queries with single partition
> -
>
> Key: DRILL-4530
> URL: https://issues.apache.org/jira/browse/DRILL-4530
> Project: Apache Drill
> Issue Type: Improvement
> Components: Query Planning & Optimization
>Affects Versions: 1.6.0
>Reporter: Aman Sinha
>Assignee: Aman Sinha
> Fix For: Future
>
>
> Consider two types of queries which are run with Parquet metadata caching:
> {noformat}
> query 1:
> SELECT col FROM `A/B/C`;
> query 2:
> SELECT col FROM `A` WHERE dir0 = 'B' AND dir1 = 'C';
> {noformat}
> For a certain dataset, the query1 elapsed time is 1 sec whereas query2
> elapsed time is 9 sec even though both are accessing the same amount of data.
> The user expectation is that they should perform roughly the same. The main
> difference comes from reading the bigger metadata cache file at the root
> level 'A' for query2 and then applying the partitioning filter. query1 reads
> a much smaller metadata cache file at the subdirectory level.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-4530) Improve metadata cache performance for queries with single partition
[
https://issues.apache.org/jira/browse/DRILL-4530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15373632#comment-15373632
]
ASF GitHub Bot commented on DRILL-4530:
---
Github user jinfengni commented on a diff in the pull request:
https://github.com/apache/drill/pull/519#discussion_r70514652
--- Diff:
exec/java-exec/src/main/java/org/apache/drill/exec/planner/logical/partition/PruneScanRule.java
---
@@ -269,13 +283,54 @@ protected void doOnMatch(RelOptRuleCall call, Filter
filterRel, Project projectR
int recordCount = 0;
int qualifiedCount = 0;
-// Inner loop: within each batch iterate over the
PartitionLocations
-for(PartitionLocation part: partitions){
- if(!output.getAccessor().isNull(recordCount) &&
output.getAccessor().get(recordCount) == 1){
-newPartitions.add(part);
-qualifiedCount++;
+if (checkForSingle &&
+partitions.get(0).isCompositePartition() /* apply single
partition check only for composite partitions */) {
+ // Inner loop: within each batch iterate over the
PartitionLocations
+ for (PartitionLocation part : partitions) {
--- End diff --
yes, it's ok to put the refactoring effort as an enhancement JIR.
> Improve metadata cache performance for queries with single partition
> -
>
> Key: DRILL-4530
> URL: https://issues.apache.org/jira/browse/DRILL-4530
> Project: Apache Drill
> Issue Type: Improvement
> Components: Query Planning & Optimization
>Affects Versions: 1.6.0
>Reporter: Aman Sinha
>Assignee: Aman Sinha
> Fix For: Future
>
>
> Consider two types of queries which are run with Parquet metadata caching:
> {noformat}
> query 1:
> SELECT col FROM `A/B/C`;
> query 2:
> SELECT col FROM `A` WHERE dir0 = 'B' AND dir1 = 'C';
> {noformat}
> For a certain dataset, the query1 elapsed time is 1 sec whereas query2
> elapsed time is 9 sec even though both are accessing the same amount of data.
> The user expectation is that they should perform roughly the same. The main
> difference comes from reading the bigger metadata cache file at the root
> level 'A' for query2 and then applying the partitioning filter. query1 reads
> a much smaller metadata cache file at the subdirectory level.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-4530) Improve metadata cache performance for queries with single partition
[
https://issues.apache.org/jira/browse/DRILL-4530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15373631#comment-15373631
]
ASF GitHub Bot commented on DRILL-4530:
---
Github user jinfengni commented on a diff in the pull request:
https://github.com/apache/drill/pull/519#discussion_r70514381
--- Diff:
exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/ParquetGroupScan.java
---
@@ -607,7 +607,14 @@ public long getRowCount() {
fileSet = Sets.newHashSet(fileNames);
}
-if (fileNames.isEmpty()) {
+List finalFileNames;
+if (fileSet != null) {
+ finalFileNames = Lists.newArrayList(fileSet);
--- End diff --
Can we use fileNames only? From Line 589 / 607, seems fileSet is assigned
from fileNames; seems they are same under two ELSE branches.
> Improve metadata cache performance for queries with single partition
> -
>
> Key: DRILL-4530
> URL: https://issues.apache.org/jira/browse/DRILL-4530
> Project: Apache Drill
> Issue Type: Improvement
> Components: Query Planning & Optimization
>Affects Versions: 1.6.0
>Reporter: Aman Sinha
>Assignee: Aman Sinha
> Fix For: Future
>
>
> Consider two types of queries which are run with Parquet metadata caching:
> {noformat}
> query 1:
> SELECT col FROM `A/B/C`;
> query 2:
> SELECT col FROM `A` WHERE dir0 = 'B' AND dir1 = 'C';
> {noformat}
> For a certain dataset, the query1 elapsed time is 1 sec whereas query2
> elapsed time is 9 sec even though both are accessing the same amount of data.
> The user expectation is that they should perform roughly the same. The main
> difference comes from reading the bigger metadata cache file at the root
> level 'A' for query2 and then applying the partitioning filter. query1 reads
> a much smaller metadata cache file at the subdirectory level.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-4530) Improve metadata cache performance for queries with single partition
[
https://issues.apache.org/jira/browse/DRILL-4530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15366929#comment-15366929
]
ASF GitHub Bot commented on DRILL-4530:
---
Github user amansinha100 commented on a diff in the pull request:
https://github.com/apache/drill/pull/519#discussion_r70002787
--- Diff:
exec/java-exec/src/main/java/org/apache/drill/exec/planner/logical/partition/PruneScanRule.java
---
@@ -269,13 +283,54 @@ protected void doOnMatch(RelOptRuleCall call, Filter
filterRel, Project projectR
int recordCount = 0;
int qualifiedCount = 0;
-// Inner loop: within each batch iterate over the
PartitionLocations
-for(PartitionLocation part: partitions){
- if(!output.getAccessor().isNull(recordCount) &&
output.getAccessor().get(recordCount) == 1){
-newPartitions.add(part);
-qualifiedCount++;
+if (checkForSingle &&
+partitions.get(0).isCompositePartition() /* apply single
partition check only for composite partitions */) {
+ // Inner loop: within each batch iterate over the
PartitionLocations
+ for (PartitionLocation part : partitions) {
--- End diff --
@jinfengni I started doing the refactoring to move this particular
optimization into a partition descriptor specific class. However, I was having
to propagate several internal states to the doSinglePartOpt() method...such as
partition map, the referenced dirs bitset , the value vector array and others.
Also, since this optimization occurs inside an outer loop, it is not
straightforward without doing a broader restructuring of the
PruneScanRule.doOnMatch() method. Would you be ok to defer this refactoring to
an enhancement JIRA ?
> Improve metadata cache performance for queries with single partition
> -
>
> Key: DRILL-4530
> URL: https://issues.apache.org/jira/browse/DRILL-4530
> Project: Apache Drill
> Issue Type: Improvement
> Components: Query Planning & Optimization
>Affects Versions: 1.6.0
>Reporter: Aman Sinha
>Assignee: Aman Sinha
> Fix For: Future
>
>
> Consider two types of queries which are run with Parquet metadata caching:
> {noformat}
> query 1:
> SELECT col FROM `A/B/C`;
> query 2:
> SELECT col FROM `A` WHERE dir0 = 'B' AND dir1 = 'C';
> {noformat}
> For a certain dataset, the query1 elapsed time is 1 sec whereas query2
> elapsed time is 9 sec even though both are accessing the same amount of data.
> The user expectation is that they should perform roughly the same. The main
> difference comes from reading the bigger metadata cache file at the root
> level 'A' for query2 and then applying the partitioning filter. query1 reads
> a much smaller metadata cache file at the subdirectory level.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-4530) Improve metadata cache performance for queries with single partition
[
https://issues.apache.org/jira/browse/DRILL-4530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15366919#comment-15366919
]
ASF GitHub Bot commented on DRILL-4530:
---
Github user amansinha100 commented on a diff in the pull request:
https://github.com/apache/drill/pull/519#discussion_r70002368
--- Diff:
contrib/storage-hive/core/src/main/java/org/apache/drill/exec/planner/sql/HivePartitionDescriptor.java
---
@@ -151,7 +152,7 @@ protected void createPartitionSublists() {
}
@Override
- public TableScan createTableScan(List newPartitions)
throws Exception {
+ public TableScan createTableScan(List newPartitions,
String cacheFileRoot) throws Exception {
--- End diff --
Refactored this interface method. Updated the PR with a separate commit.
> Improve metadata cache performance for queries with single partition
> -
>
> Key: DRILL-4530
> URL: https://issues.apache.org/jira/browse/DRILL-4530
> Project: Apache Drill
> Issue Type: Improvement
> Components: Query Planning & Optimization
>Affects Versions: 1.6.0
>Reporter: Aman Sinha
>Assignee: Aman Sinha
> Fix For: Future
>
>
> Consider two types of queries which are run with Parquet metadata caching:
> {noformat}
> query 1:
> SELECT col FROM `A/B/C`;
> query 2:
> SELECT col FROM `A` WHERE dir0 = 'B' AND dir1 = 'C';
> {noformat}
> For a certain dataset, the query1 elapsed time is 1 sec whereas query2
> elapsed time is 9 sec even though both are accessing the same amount of data.
> The user expectation is that they should perform roughly the same. The main
> difference comes from reading the bigger metadata cache file at the root
> level 'A' for query2 and then applying the partitioning filter. query1 reads
> a much smaller metadata cache file at the subdirectory level.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-4530) Improve metadata cache performance for queries with single partition
[
https://issues.apache.org/jira/browse/DRILL-4530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15366914#comment-15366914
]
ASF GitHub Bot commented on DRILL-4530:
---
Github user amansinha100 commented on a diff in the pull request:
https://github.com/apache/drill/pull/519#discussion_r70002215
--- Diff:
exec/java-exec/src/main/java/org/apache/drill/exec/planner/logical/partition/PruneScanRule.java
---
@@ -269,13 +283,54 @@ protected void doOnMatch(RelOptRuleCall call, Filter
filterRel, Project projectR
int recordCount = 0;
int qualifiedCount = 0;
-// Inner loop: within each batch iterate over the
PartitionLocations
-for(PartitionLocation part: partitions){
- if(!output.getAccessor().isNull(recordCount) &&
output.getAccessor().get(recordCount) == 1){
-newPartitions.add(part);
-qualifiedCount++;
+if (checkForSingle &&
+partitions.get(0).isCompositePartition() /* apply single
partition check only for composite partitions */) {
+ // Inner loop: within each batch iterate over the
PartitionLocations
+ for (PartitionLocation part : partitions) {
+assert part.isCompositePartition();
+if(!output.getAccessor().isNull(recordCount) &&
output.getAccessor().get(recordCount) == 1) {
+ newPartitions.add(part);
+ if (isSinglePartition) { // only need to do this if we are
already single partition
+// compose the array of partition values for the
directories that are referenced by filter:
+// e.g suppose the dir hierarchy is year/quarter/month and
the query is:
+// SELECT * FROM T WHERE dir0=2015 AND dir1 = 'Q1',
+// then for 2015/Q1/Feb, this will have ['2015', 'Q1',
null]
--- End diff --
I have fixed this case and updated the pull request with a separate commit
(not squashed yet so you can review it separately).
> Improve metadata cache performance for queries with single partition
> -
>
> Key: DRILL-4530
> URL: https://issues.apache.org/jira/browse/DRILL-4530
> Project: Apache Drill
> Issue Type: Improvement
> Components: Query Planning & Optimization
>Affects Versions: 1.6.0
>Reporter: Aman Sinha
>Assignee: Aman Sinha
> Fix For: Future
>
>
> Consider two types of queries which are run with Parquet metadata caching:
> {noformat}
> query 1:
> SELECT col FROM `A/B/C`;
> query 2:
> SELECT col FROM `A` WHERE dir0 = 'B' AND dir1 = 'C';
> {noformat}
> For a certain dataset, the query1 elapsed time is 1 sec whereas query2
> elapsed time is 9 sec even though both are accessing the same amount of data.
> The user expectation is that they should perform roughly the same. The main
> difference comes from reading the bigger metadata cache file at the root
> level 'A' for query2 and then applying the partitioning filter. query1 reads
> a much smaller metadata cache file at the subdirectory level.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-4530) Improve metadata cache performance for queries with single partition
[
https://issues.apache.org/jira/browse/DRILL-4530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15353496#comment-15353496
]
ASF GitHub Bot commented on DRILL-4530:
---
Github user jinfengni commented on a diff in the pull request:
https://github.com/apache/drill/pull/519#discussion_r68814702
--- Diff:
exec/java-exec/src/main/java/org/apache/drill/exec/store/dfs/FileSelection.java
---
@@ -47,16 +47,25 @@
private List statuses;
public List files;
+ /**
+ * root path for the selections
+ */
public final String selectionRoot;
+ /**
+ * root path for the metadata cache file (if any)
+ */
+ public final String cacheFileRoot;
--- End diff --
Thanks for the explanation. It makes sense and I do see why updating
selectionRoot would cause problems for the queries you mentioned.
> Improve metadata cache performance for queries with single partition
> -
>
> Key: DRILL-4530
> URL: https://issues.apache.org/jira/browse/DRILL-4530
> Project: Apache Drill
> Issue Type: Improvement
> Components: Query Planning & Optimization
>Affects Versions: 1.6.0
>Reporter: Aman Sinha
>Assignee: Aman Sinha
> Fix For: Future
>
>
> Consider two types of queries which are run with Parquet metadata caching:
> {noformat}
> query 1:
> SELECT col FROM `A/B/C`;
> query 2:
> SELECT col FROM `A` WHERE dir0 = 'B' AND dir1 = 'C';
> {noformat}
> For a certain dataset, the query1 elapsed time is 1 sec whereas query2
> elapsed time is 9 sec even though both are accessing the same amount of data.
> The user expectation is that they should perform roughly the same. The main
> difference comes from reading the bigger metadata cache file at the root
> level 'A' for query2 and then applying the partitioning filter. query1 reads
> a much smaller metadata cache file at the subdirectory level.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-4530) Improve metadata cache performance for queries with single partition
[
https://issues.apache.org/jira/browse/DRILL-4530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15352088#comment-15352088
]
ASF GitHub Bot commented on DRILL-4530:
---
Github user amansinha100 commented on a diff in the pull request:
https://github.com/apache/drill/pull/519#discussion_r68678168
--- Diff:
exec/java-exec/src/main/java/org/apache/drill/exec/store/dfs/FileSelection.java
---
@@ -47,16 +47,25 @@
private List statuses;
public List files;
+ /**
+ * root path for the selections
+ */
public final String selectionRoot;
+ /**
+ * root path for the metadata cache file (if any)
+ */
+ public final String cacheFileRoot;
--- End diff --
That was my initial approach (updating the selectionRoot without keeping a
separate cacheFileRoot). However, I ran into a few issues. The main one
that I recall is that the dir0, dir1 etc columns are associated with the
selectionRoot, so suppose I run the following query:
SELECT dir0, dir1 FROM dfs.tmp.t2 WHERE dir0=2015 AND dir1='Q1'
and if the selectionRoot gets modified to point to '2015/Q1' then we have
lost the context of the original dir0, dir1 because everything will become
relative to the new selectionRoot.This produces wrong results. The same
problem occurred with a SELECT * query where the directory columns where not
showing up correctly.
> Improve metadata cache performance for queries with single partition
> -
>
> Key: DRILL-4530
> URL: https://issues.apache.org/jira/browse/DRILL-4530
> Project: Apache Drill
> Issue Type: Improvement
> Components: Query Planning & Optimization
>Affects Versions: 1.6.0
>Reporter: Aman Sinha
>Assignee: Aman Sinha
> Fix For: Future
>
>
> Consider two types of queries which are run with Parquet metadata caching:
> {noformat}
> query 1:
> SELECT col FROM `A/B/C`;
> query 2:
> SELECT col FROM `A` WHERE dir0 = 'B' AND dir1 = 'C';
> {noformat}
> For a certain dataset, the query1 elapsed time is 1 sec whereas query2
> elapsed time is 9 sec even though both are accessing the same amount of data.
> The user expectation is that they should perform roughly the same. The main
> difference comes from reading the bigger metadata cache file at the root
> level 'A' for query2 and then applying the partitioning filter. query1 reads
> a much smaller metadata cache file at the subdirectory level.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-4530) Improve metadata cache performance for queries with single partition
[
https://issues.apache.org/jira/browse/DRILL-4530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15352020#comment-15352020
]
ASF GitHub Bot commented on DRILL-4530:
---
Github user amansinha100 commented on a diff in the pull request:
https://github.com/apache/drill/pull/519#discussion_r68673292
--- Diff:
exec/java-exec/src/main/java/org/apache/drill/exec/planner/logical/partition/PruneScanRule.java
---
@@ -269,13 +283,54 @@ protected void doOnMatch(RelOptRuleCall call, Filter
filterRel, Project projectR
int recordCount = 0;
int qualifiedCount = 0;
-// Inner loop: within each batch iterate over the
PartitionLocations
-for(PartitionLocation part: partitions){
- if(!output.getAccessor().isNull(recordCount) &&
output.getAccessor().get(recordCount) == 1){
-newPartitions.add(part);
-qualifiedCount++;
+if (checkForSingle &&
+partitions.get(0).isCompositePartition() /* apply single
partition check only for composite partitions */) {
+ // Inner loop: within each batch iterate over the
PartitionLocations
+ for (PartitionLocation part : partitions) {
+assert part.isCompositePartition();
+if(!output.getAccessor().isNull(recordCount) &&
output.getAccessor().get(recordCount) == 1) {
+ newPartitions.add(part);
+ if (isSinglePartition) { // only need to do this if we are
already single partition
+// compose the array of partition values for the
directories that are referenced by filter:
+// e.g suppose the dir hierarchy is year/quarter/month and
the query is:
+// SELECT * FROM T WHERE dir0=2015 AND dir1 = 'Q1',
+// then for 2015/Q1/Feb, this will have ['2015', 'Q1',
null]
--- End diff --
Good point. The path [2015, null, Jan] should *not* qualify for the
single partition optimization because in the general case there could be
multiple 'Jan' subdirectories under the dir1 directory. We would still use the
metadata cache but at the level of the dir0, so ideally one should get
cacheFileRoot=/tmp/t2/2015. I have a fix for this and will update the PR.
> Improve metadata cache performance for queries with single partition
> -
>
> Key: DRILL-4530
> URL: https://issues.apache.org/jira/browse/DRILL-4530
> Project: Apache Drill
> Issue Type: Improvement
> Components: Query Planning & Optimization
>Affects Versions: 1.6.0
>Reporter: Aman Sinha
>Assignee: Aman Sinha
> Fix For: Future
>
>
> Consider two types of queries which are run with Parquet metadata caching:
> {noformat}
> query 1:
> SELECT col FROM `A/B/C`;
> query 2:
> SELECT col FROM `A` WHERE dir0 = 'B' AND dir1 = 'C';
> {noformat}
> For a certain dataset, the query1 elapsed time is 1 sec whereas query2
> elapsed time is 9 sec even though both are accessing the same amount of data.
> The user expectation is that they should perform roughly the same. The main
> difference comes from reading the bigger metadata cache file at the root
> level 'A' for query2 and then applying the partitioning filter. query1 reads
> a much smaller metadata cache file at the subdirectory level.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-4530) Improve metadata cache performance for queries with single partition
[
https://issues.apache.org/jira/browse/DRILL-4530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15339797#comment-15339797
]
ASF GitHub Bot commented on DRILL-4530:
---
Github user jinfengni commented on a diff in the pull request:
https://github.com/apache/drill/pull/519#discussion_r67716686
--- Diff:
exec/java-exec/src/main/java/org/apache/drill/exec/store/dfs/FileSelection.java
---
@@ -47,16 +47,25 @@
private List statuses;
public List files;
+ /**
+ * root path for the selections
+ */
public final String selectionRoot;
+ /**
+ * root path for the metadata cache file (if any)
+ */
+ public final String cacheFileRoot;
--- End diff --
When singlePartitionOpt is applied, is it possible to update selectionRoot
to be cacheFileRoot? That is, we do not maintain cacheFileRoot separately. In
stead, a FileSelection with updated selectionRoot is used when
singlePartitionOpt is applied.
> Improve metadata cache performance for queries with single partition
> -
>
> Key: DRILL-4530
> URL: https://issues.apache.org/jira/browse/DRILL-4530
> Project: Apache Drill
> Issue Type: Improvement
> Components: Query Planning & Optimization
>Affects Versions: 1.6.0
>Reporter: Aman Sinha
>Assignee: Aman Sinha
> Fix For: 1.7.0
>
>
> Consider two types of queries which are run with Parquet metadata caching:
> {noformat}
> query 1:
> SELECT col FROM `A/B/C`;
> query 2:
> SELECT col FROM `A` WHERE dir0 = 'B' AND dir1 = 'C';
> {noformat}
> For a certain dataset, the query1 elapsed time is 1 sec whereas query2
> elapsed time is 9 sec even though both are accessing the same amount of data.
> The user expectation is that they should perform roughly the same. The main
> difference comes from reading the bigger metadata cache file at the root
> level 'A' for query2 and then applying the partitioning filter. query1 reads
> a much smaller metadata cache file at the subdirectory level.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-4530) Improve metadata cache performance for queries with single partition
[
https://issues.apache.org/jira/browse/DRILL-4530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15339728#comment-15339728
]
ASF GitHub Bot commented on DRILL-4530:
---
Github user jinfengni commented on a diff in the pull request:
https://github.com/apache/drill/pull/519#discussion_r67711244
--- Diff:
exec/java-exec/src/main/java/org/apache/drill/exec/planner/logical/partition/PruneScanRule.java
---
@@ -269,13 +283,54 @@ protected void doOnMatch(RelOptRuleCall call, Filter
filterRel, Project projectR
int recordCount = 0;
int qualifiedCount = 0;
-// Inner loop: within each batch iterate over the
PartitionLocations
-for(PartitionLocation part: partitions){
- if(!output.getAccessor().isNull(recordCount) &&
output.getAccessor().get(recordCount) == 1){
-newPartitions.add(part);
-qualifiedCount++;
+if (checkForSingle &&
+partitions.get(0).isCompositePartition() /* apply single
partition check only for composite partitions */) {
+ // Inner loop: within each batch iterate over the
PartitionLocations
+ for (PartitionLocation part : partitions) {
+assert part.isCompositePartition();
+if(!output.getAccessor().isNull(recordCount) &&
output.getAccessor().get(recordCount) == 1) {
+ newPartitions.add(part);
+ if (isSinglePartition) { // only need to do this if we are
already single partition
+// compose the array of partition values for the
directories that are referenced by filter:
+// e.g suppose the dir hierarchy is year/quarter/month and
the query is:
+// SELECT * FROM T WHERE dir0=2015 AND dir1 = 'Q1',
+// then for 2015/Q1/Feb, this will have ['2015', 'Q1',
null]
--- End diff --
For WHERE condition dir0=2015 and dir2 = 'Jan', if the dataset happens to
have only one 'Jan' under '2015' directory, will this qualify for
singlePartitionOpt?
> Improve metadata cache performance for queries with single partition
> -
>
> Key: DRILL-4530
> URL: https://issues.apache.org/jira/browse/DRILL-4530
> Project: Apache Drill
> Issue Type: Improvement
> Components: Query Planning & Optimization
>Affects Versions: 1.6.0
>Reporter: Aman Sinha
>Assignee: Aman Sinha
> Fix For: 1.7.0
>
>
> Consider two types of queries which are run with Parquet metadata caching:
> {noformat}
> query 1:
> SELECT col FROM `A/B/C`;
> query 2:
> SELECT col FROM `A` WHERE dir0 = 'B' AND dir1 = 'C';
> {noformat}
> For a certain dataset, the query1 elapsed time is 1 sec whereas query2
> elapsed time is 9 sec even though both are accessing the same amount of data.
> The user expectation is that they should perform roughly the same. The main
> difference comes from reading the bigger metadata cache file at the root
> level 'A' for query2 and then applying the partitioning filter. query1 reads
> a much smaller metadata cache file at the subdirectory level.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-4530) Improve metadata cache performance for queries with single partition
[
https://issues.apache.org/jira/browse/DRILL-4530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15339703#comment-15339703
]
ASF GitHub Bot commented on DRILL-4530:
---
Github user jinfengni commented on a diff in the pull request:
https://github.com/apache/drill/pull/519#discussion_r67709299
--- Diff:
exec/java-exec/src/main/java/org/apache/drill/exec/planner/logical/partition/PruneScanRule.java
---
@@ -320,7 +377,17 @@ protected void doOnMatch(RelOptRuleCall call, Filter
filterRel, Project projectR
condition = condition.accept(reverseVisitor);
pruneCondition = pruneCondition.accept(reverseVisitor);
- RelNode inputRel = descriptor.createTableScan(newPartitions);
+ String cacheFileRoot = null;
+ if (checkForSingle && isSinglePartition) {
+// if metadata cache file could potentially be used, then assign a
proper cacheFileRoot
+String path = "";
+for (int j = 0; j <= maxIndex; j++) {
+ path += "/" + spInfo[j];
--- End diff --
Related to Line 313, here we do not check spInfo[j] == null ?
> Improve metadata cache performance for queries with single partition
> -
>
> Key: DRILL-4530
> URL: https://issues.apache.org/jira/browse/DRILL-4530
> Project: Apache Drill
> Issue Type: Improvement
> Components: Query Planning & Optimization
>Affects Versions: 1.6.0
>Reporter: Aman Sinha
>Assignee: Aman Sinha
> Fix For: 1.7.0
>
>
> Consider two types of queries which are run with Parquet metadata caching:
> {noformat}
> query 1:
> SELECT col FROM `A/B/C`;
> query 2:
> SELECT col FROM `A` WHERE dir0 = 'B' AND dir1 = 'C';
> {noformat}
> For a certain dataset, the query1 elapsed time is 1 sec whereas query2
> elapsed time is 9 sec even though both are accessing the same amount of data.
> The user expectation is that they should perform roughly the same. The main
> difference comes from reading the bigger metadata cache file at the root
> level 'A' for query2 and then applying the partitioning filter. query1 reads
> a much smaller metadata cache file at the subdirectory level.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-4530) Improve metadata cache performance for queries with single partition
[
https://issues.apache.org/jira/browse/DRILL-4530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15337980#comment-15337980
]
ASF GitHub Bot commented on DRILL-4530:
---
Github user amansinha100 commented on a diff in the pull request:
https://github.com/apache/drill/pull/519#discussion_r67602442
--- Diff:
contrib/storage-hive/core/src/main/java/org/apache/drill/exec/planner/sql/HivePartitionDescriptor.java
---
@@ -151,7 +152,7 @@ protected void createPartitionSublists() {
}
@Override
- public TableScan createTableScan(List newPartitions)
throws Exception {
+ public TableScan createTableScan(List newPartitions,
String cacheFileRoot) throws Exception {
--- End diff --
I suppose I could add a second interface method for createTableScan that
allows the second parameter and keep the old method as-is.
> Improve metadata cache performance for queries with single partition
> -
>
> Key: DRILL-4530
> URL: https://issues.apache.org/jira/browse/DRILL-4530
> Project: Apache Drill
> Issue Type: Improvement
> Components: Query Planning & Optimization
>Affects Versions: 1.6.0
>Reporter: Aman Sinha
>Assignee: Aman Sinha
> Fix For: 1.7.0
>
>
> Consider two types of queries which are run with Parquet metadata caching:
> {noformat}
> query 1:
> SELECT col FROM `A/B/C`;
> query 2:
> SELECT col FROM `A` WHERE dir0 = 'B' AND dir1 = 'C';
> {noformat}
> For a certain dataset, the query1 elapsed time is 1 sec whereas query2
> elapsed time is 9 sec even though both are accessing the same amount of data.
> The user expectation is that they should perform roughly the same. The main
> difference comes from reading the bigger metadata cache file at the root
> level 'A' for query2 and then applying the partitioning filter. query1 reads
> a much smaller metadata cache file at the subdirectory level.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-4530) Improve metadata cache performance for queries with single partition
[
https://issues.apache.org/jira/browse/DRILL-4530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15337976#comment-15337976
]
ASF GitHub Bot commented on DRILL-4530:
---
Github user amansinha100 commented on a diff in the pull request:
https://github.com/apache/drill/pull/519#discussion_r67602385
--- Diff:
exec/java-exec/src/main/java/org/apache/drill/exec/planner/logical/partition/PruneScanRule.java
---
@@ -269,13 +283,54 @@ protected void doOnMatch(RelOptRuleCall call, Filter
filterRel, Project projectR
int recordCount = 0;
int qualifiedCount = 0;
-// Inner loop: within each batch iterate over the
PartitionLocations
-for(PartitionLocation part: partitions){
- if(!output.getAccessor().isNull(recordCount) &&
output.getAccessor().get(recordCount) == 1){
-newPartitions.add(part);
-qualifiedCount++;
+if (checkForSingle &&
+partitions.get(0).isCompositePartition() /* apply single
partition check only for composite partitions */) {
+ // Inner loop: within each batch iterate over the
PartitionLocations
+ for (PartitionLocation part : partitions) {
--- End diff --
Yes, the rule has become too complex. I will go ahead and at least
refactor this specific optimization.
> Improve metadata cache performance for queries with single partition
> -
>
> Key: DRILL-4530
> URL: https://issues.apache.org/jira/browse/DRILL-4530
> Project: Apache Drill
> Issue Type: Improvement
> Components: Query Planning & Optimization
>Affects Versions: 1.6.0
>Reporter: Aman Sinha
>Assignee: Aman Sinha
> Fix For: 1.7.0
>
>
> Consider two types of queries which are run with Parquet metadata caching:
> {noformat}
> query 1:
> SELECT col FROM `A/B/C`;
> query 2:
> SELECT col FROM `A` WHERE dir0 = 'B' AND dir1 = 'C';
> {noformat}
> For a certain dataset, the query1 elapsed time is 1 sec whereas query2
> elapsed time is 9 sec even though both are accessing the same amount of data.
> The user expectation is that they should perform roughly the same. The main
> difference comes from reading the bigger metadata cache file at the root
> level 'A' for query2 and then applying the partitioning filter. query1 reads
> a much smaller metadata cache file at the subdirectory level.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-4530) Improve metadata cache performance for queries with single partition
[
https://issues.apache.org/jira/browse/DRILL-4530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15337974#comment-15337974
]
ASF GitHub Bot commented on DRILL-4530:
---
Github user amansinha100 commented on a diff in the pull request:
https://github.com/apache/drill/pull/519#discussion_r67602350
--- Diff:
exec/java-exec/src/main/java/org/apache/drill/exec/planner/logical/partition/PruneScanRule.java
---
@@ -269,13 +283,54 @@ protected void doOnMatch(RelOptRuleCall call, Filter
filterRel, Project projectR
int recordCount = 0;
int qualifiedCount = 0;
-// Inner loop: within each batch iterate over the
PartitionLocations
-for(PartitionLocation part: partitions){
- if(!output.getAccessor().isNull(recordCount) &&
output.getAccessor().get(recordCount) == 1){
-newPartitions.add(part);
-qualifiedCount++;
+if (checkForSingle &&
+partitions.get(0).isCompositePartition() /* apply single
partition check only for composite partitions */) {
+ // Inner loop: within each batch iterate over the
PartitionLocations
+ for (PartitionLocation part : partitions) {
+assert part.isCompositePartition();
+if(!output.getAccessor().isNull(recordCount) &&
output.getAccessor().get(recordCount) == 1) {
+ newPartitions.add(part);
+ if (isSinglePartition) { // only need to do this if we are
already single partition
+// compose the array of partition values for the
directories that are referenced by filter:
+// e.g suppose the dir hierarchy is year/quarter/month and
the query is:
+// SELECT * FROM T WHERE dir0=2015 AND dir1 = 'Q1',
+// then for 2015/Q1/Feb, this will have ['2015', 'Q1',
null]
+// Note that we are not using the PartitionLocation here
but composing a different list because
+// we are only interested in the directory columns that
are referenced in the filter condition. not
+// the SELECT list or other parts of the query.
+Pair p =
composePartition(referencedDirsBitSet, partitionMap, vectors, recordCount);
+String[] parts = p.getLeft();
+int tmpIndex = p.getRight();
+if (spInfo == null) {
+ spInfo = parts;
+ maxIndex = tmpIndex;
+} else if (maxIndex != tmpIndex) {
+ isSinglePartition = false;
+ break;
+} else {
+ // we only want to compare until the maxIndex inclusive
since subsequent values would be null
+ for (int j = 0; j <= maxIndex; j++) {
+if (spInfo[j] == null // prefixes should be non-null
--- End diff --
If the query has for example WHERE dir2 = 'January' and does not have any
condition on dir0 or dir1, then the spInfo array itself will be non-null but
will have null elements in it: [null, null, 'January'] and maxIndex = 2. In
this case, the single partition optimization should not be applied. I can add a
comment here to explain.
> Improve metadata cache performance for queries with single partition
> -
>
> Key: DRILL-4530
> URL: https://issues.apache.org/jira/browse/DRILL-4530
> Project: Apache Drill
> Issue Type: Improvement
> Components: Query Planning & Optimization
>Affects Versions: 1.6.0
>Reporter: Aman Sinha
>Assignee: Aman Sinha
> Fix For: 1.7.0
>
>
> Consider two types of queries which are run with Parquet metadata caching:
> {noformat}
> query 1:
> SELECT col FROM `A/B/C`;
> query 2:
> SELECT col FROM `A` WHERE dir0 = 'B' AND dir1 = 'C';
> {noformat}
> For a certain dataset, the query1 elapsed time is 1 sec whereas query2
> elapsed time is 9 sec even though both are accessing the same amount of data.
> The user expectation is that they should perform roughly the same. The main
> difference comes from reading the bigger metadata cache file at the root
> level 'A' for query2 and then applying the partitioning filter. query1 reads
> a much smaller metadata cache file at the subdirectory level.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-4530) Improve metadata cache performance for queries with single partition
[
https://issues.apache.org/jira/browse/DRILL-4530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15336826#comment-15336826
]
ASF GitHub Bot commented on DRILL-4530:
---
Github user jinfengni commented on a diff in the pull request:
https://github.com/apache/drill/pull/519#discussion_r67567993
--- Diff:
exec/java-exec/src/main/java/org/apache/drill/exec/planner/logical/partition/PruneScanRule.java
---
@@ -269,13 +283,54 @@ protected void doOnMatch(RelOptRuleCall call, Filter
filterRel, Project projectR
int recordCount = 0;
int qualifiedCount = 0;
-// Inner loop: within each batch iterate over the
PartitionLocations
-for(PartitionLocation part: partitions){
- if(!output.getAccessor().isNull(recordCount) &&
output.getAccessor().get(recordCount) == 1){
-newPartitions.add(part);
-qualifiedCount++;
+if (checkForSingle &&
+partitions.get(0).isCompositePartition() /* apply single
partition check only for composite partitions */) {
+ // Inner loop: within each batch iterate over the
PartitionLocations
+ for (PartitionLocation part : partitions) {
+assert part.isCompositePartition();
+if(!output.getAccessor().isNull(recordCount) &&
output.getAccessor().get(recordCount) == 1) {
+ newPartitions.add(part);
+ if (isSinglePartition) { // only need to do this if we are
already single partition
+// compose the array of partition values for the
directories that are referenced by filter:
+// e.g suppose the dir hierarchy is year/quarter/month and
the query is:
+// SELECT * FROM T WHERE dir0=2015 AND dir1 = 'Q1',
+// then for 2015/Q1/Feb, this will have ['2015', 'Q1',
null]
+// Note that we are not using the PartitionLocation here
but composing a different list because
+// we are only interested in the directory columns that
are referenced in the filter condition. not
+// the SELECT list or other parts of the query.
+Pair p =
composePartition(referencedDirsBitSet, partitionMap, vectors, recordCount);
+String[] parts = p.getLeft();
+int tmpIndex = p.getRight();
+if (spInfo == null) {
+ spInfo = parts;
+ maxIndex = tmpIndex;
+} else if (maxIndex != tmpIndex) {
+ isSinglePartition = false;
+ break;
+} else {
+ // we only want to compare until the maxIndex inclusive
since subsequent values would be null
+ for (int j = 0; j <= maxIndex; j++) {
+if (spInfo[j] == null // prefixes should be non-null
--- End diff --
Form Line 305-306, spInfo and maxIndex are in sync. Why will we have
spInfo[j] == null, when j <= maxIndex? I thought maxIndex is obtained such that
element in spInfo is not null.
> Improve metadata cache performance for queries with single partition
> -
>
> Key: DRILL-4530
> URL: https://issues.apache.org/jira/browse/DRILL-4530
> Project: Apache Drill
> Issue Type: Improvement
> Components: Query Planning & Optimization
>Affects Versions: 1.6.0
>Reporter: Aman Sinha
>Assignee: Aman Sinha
> Fix For: 1.7.0
>
>
> Consider two types of queries which are run with Parquet metadata caching:
> {noformat}
> query 1:
> SELECT col FROM `A/B/C`;
> query 2:
> SELECT col FROM `A` WHERE dir0 = 'B' AND dir1 = 'C';
> {noformat}
> For a certain dataset, the query1 elapsed time is 1 sec whereas query2
> elapsed time is 9 sec even though both are accessing the same amount of data.
> The user expectation is that they should perform roughly the same. The main
> difference comes from reading the bigger metadata cache file at the root
> level 'A' for query2 and then applying the partitioning filter. query1 reads
> a much smaller metadata cache file at the subdirectory level.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-4530) Improve metadata cache performance for queries with single partition
[
https://issues.apache.org/jira/browse/DRILL-4530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15336821#comment-15336821
]
ASF GitHub Bot commented on DRILL-4530:
---
Github user jinfengni commented on a diff in the pull request:
https://github.com/apache/drill/pull/519#discussion_r67567703
--- Diff:
contrib/storage-hive/core/src/main/java/org/apache/drill/exec/planner/sql/HivePartitionDescriptor.java
---
@@ -151,7 +152,7 @@ protected void createPartitionSublists() {
}
@Override
- public TableScan createTableScan(List newPartitions)
throws Exception {
+ public TableScan createTableScan(List newPartitions,
String cacheFileRoot) throws Exception {
--- End diff --
From the comment, cacheFileRoot could be null. But is it possible to not
adding this parameter to HivePartitionDescriptor? After all, cacheFileRoot will
not be applied to hive partition.
> Improve metadata cache performance for queries with single partition
> -
>
> Key: DRILL-4530
> URL: https://issues.apache.org/jira/browse/DRILL-4530
> Project: Apache Drill
> Issue Type: Improvement
> Components: Query Planning & Optimization
>Affects Versions: 1.6.0
>Reporter: Aman Sinha
>Assignee: Aman Sinha
> Fix For: 1.7.0
>
>
> Consider two types of queries which are run with Parquet metadata caching:
> {noformat}
> query 1:
> SELECT col FROM `A/B/C`;
> query 2:
> SELECT col FROM `A` WHERE dir0 = 'B' AND dir1 = 'C';
> {noformat}
> For a certain dataset, the query1 elapsed time is 1 sec whereas query2
> elapsed time is 9 sec even though both are accessing the same amount of data.
> The user expectation is that they should perform roughly the same. The main
> difference comes from reading the bigger metadata cache file at the root
> level 'A' for query2 and then applying the partitioning filter. query1 reads
> a much smaller metadata cache file at the subdirectory level.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-4530) Improve metadata cache performance for queries with single partition
[
https://issues.apache.org/jira/browse/DRILL-4530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15336817#comment-15336817
]
ASF GitHub Bot commented on DRILL-4530:
---
Github user jinfengni commented on a diff in the pull request:
https://github.com/apache/drill/pull/519#discussion_r67567343
--- Diff:
exec/java-exec/src/main/java/org/apache/drill/exec/planner/logical/partition/PruneScanRule.java
---
@@ -269,13 +283,54 @@ protected void doOnMatch(RelOptRuleCall call, Filter
filterRel, Project projectR
int recordCount = 0;
int qualifiedCount = 0;
-// Inner loop: within each batch iterate over the
PartitionLocations
-for(PartitionLocation part: partitions){
- if(!output.getAccessor().isNull(recordCount) &&
output.getAccessor().get(recordCount) == 1){
-newPartitions.add(part);
-qualifiedCount++;
+if (checkForSingle &&
+partitions.get(0).isCompositePartition() /* apply single
partition check only for composite partitions */) {
+ // Inner loop: within each batch iterate over the
PartitionLocations
+ for (PartitionLocation part : partitions) {
--- End diff --
Do you think it's possible to refactor this part of code such that only
file system prune scan will have this logic? This simplePartition optimization
only applies to file system prune. Also, doOnMatch() method is getting bigger.
Something like: descriptor has supportsSinglePartOptimization(), and prune
scan rule has doSinglePartOpt(), which is by default a no-op, and has
implementation for file system prune rule.
> Improve metadata cache performance for queries with single partition
> -
>
> Key: DRILL-4530
> URL: https://issues.apache.org/jira/browse/DRILL-4530
> Project: Apache Drill
> Issue Type: Improvement
> Components: Query Planning & Optimization
>Affects Versions: 1.6.0
>Reporter: Aman Sinha
>Assignee: Aman Sinha
> Fix For: 1.7.0
>
>
> Consider two types of queries which are run with Parquet metadata caching:
> {noformat}
> query 1:
> SELECT col FROM `A/B/C`;
> query 2:
> SELECT col FROM `A` WHERE dir0 = 'B' AND dir1 = 'C';
> {noformat}
> For a certain dataset, the query1 elapsed time is 1 sec whereas query2
> elapsed time is 9 sec even though both are accessing the same amount of data.
> The user expectation is that they should perform roughly the same. The main
> difference comes from reading the bigger metadata cache file at the root
> level 'A' for query2 and then applying the partitioning filter. query1 reads
> a much smaller metadata cache file at the subdirectory level.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-4530) Improve metadata cache performance for queries with single partition
[
https://issues.apache.org/jira/browse/DRILL-4530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15335374#comment-15335374
]
Aman Sinha commented on DRILL-4530:
---
I got the performance numbers for the query2 (see description) scenario where
there are 400K files in the table.
{noformat}
query2:
current master branch (without the patch): 60 seconds
with the patch: 3 seconds (this time is comparable to that of query1)
{noformat}
> Improve metadata cache performance for queries with single partition
> -
>
> Key: DRILL-4530
> URL: https://issues.apache.org/jira/browse/DRILL-4530
> Project: Apache Drill
> Issue Type: Improvement
> Components: Query Planning & Optimization
>Affects Versions: 1.6.0
>Reporter: Aman Sinha
>Assignee: Aman Sinha
> Fix For: 1.7.0
>
>
> Consider two types of queries which are run with Parquet metadata caching:
> {noformat}
> query 1:
> SELECT col FROM `A/B/C`;
> query 2:
> SELECT col FROM `A` WHERE dir0 = 'B' AND dir1 = 'C';
> {noformat}
> For a certain dataset, the query1 elapsed time is 1 sec whereas query2
> elapsed time is 9 sec even though both are accessing the same amount of data.
> The user expectation is that they should perform roughly the same. The main
> difference comes from reading the bigger metadata cache file at the root
> level 'A' for query2 and then applying the partitioning filter. query1 reads
> a much smaller metadata cache file at the subdirectory level.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-4530) Improve metadata cache performance for queries with single partition
[
https://issues.apache.org/jira/browse/DRILL-4530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15334300#comment-15334300
]
ASF GitHub Bot commented on DRILL-4530:
---
Github user amansinha100 commented on a diff in the pull request:
https://github.com/apache/drill/pull/519#discussion_r67392462
--- Diff:
exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/ParquetFormatPlugin.java
---
@@ -208,8 +209,18 @@ public DrillTable isReadable(DrillFileSystem fs,
FileSelection selection,
FileSystemPlugin fsPlugin, String storageEngineName, String
userName)
throws IOException {
// TODO: we only check the first file for directory reading.
- if(selection.containsDirectories(fs)){
-if(isDirReadable(fs, selection.getFirstPath(fs))){
+ if(selection.containsDirectories(fs)) {
+Path dirMetaPath = new Path(selection.getSelectionRoot(),
Metadata.METADATA_DIRECTORIES_FILENAME);
+if (fs.exists(dirMetaPath)) {
+ ParquetTableMetadataDirs mDirs = Metadata.readMetadataDirs(fs,
dirMetaPath.toString());
+ if (mDirs.getDirectories().size() > 0) {
+FileSelection dirSelection =
FileSelection.createFromDirectories(mDirs.getDirectories(), selection);
+dirSelection.setExpandedPartial();
+return new DynamicDrillTable(fsPlugin, storageEngineName,
userName,
--- End diff --
make sense.. I will add a comment. thanks for reviewing.
> Improve metadata cache performance for queries with single partition
> -
>
> Key: DRILL-4530
> URL: https://issues.apache.org/jira/browse/DRILL-4530
> Project: Apache Drill
> Issue Type: Improvement
> Components: Query Planning & Optimization
>Affects Versions: 1.6.0
>Reporter: Aman Sinha
>Assignee: Aman Sinha
> Fix For: 1.7.0
>
>
> Consider two types of queries which are run with Parquet metadata caching:
> {noformat}
> query 1:
> SELECT col FROM `A/B/C`;
> query 2:
> SELECT col FROM `A` WHERE dir0 = 'B' AND dir1 = 'C';
> {noformat}
> For a certain dataset, the query1 elapsed time is 1 sec whereas query2
> elapsed time is 9 sec even though both are accessing the same amount of data.
> The user expectation is that they should perform roughly the same. The main
> difference comes from reading the bigger metadata cache file at the root
> level 'A' for query2 and then applying the partitioning filter. query1 reads
> a much smaller metadata cache file at the subdirectory level.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-4530) Improve metadata cache performance for queries with single partition
[
https://issues.apache.org/jira/browse/DRILL-4530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15334208#comment-15334208
]
ASF GitHub Bot commented on DRILL-4530:
---
Github user parthchandra commented on a diff in the pull request:
https://github.com/apache/drill/pull/519#discussion_r67387102
--- Diff:
exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/ParquetFormatPlugin.java
---
@@ -208,8 +209,18 @@ public DrillTable isReadable(DrillFileSystem fs,
FileSelection selection,
FileSystemPlugin fsPlugin, String storageEngineName, String
userName)
throws IOException {
// TODO: we only check the first file for directory reading.
- if(selection.containsDirectories(fs)){
-if(isDirReadable(fs, selection.getFirstPath(fs))){
+ if(selection.containsDirectories(fs)) {
+Path dirMetaPath = new Path(selection.getSelectionRoot(),
Metadata.METADATA_DIRECTORIES_FILENAME);
+if (fs.exists(dirMetaPath)) {
+ ParquetTableMetadataDirs mDirs = Metadata.readMetadataDirs(fs,
dirMetaPath.toString());
+ if (mDirs.getDirectories().size() > 0) {
+FileSelection dirSelection =
FileSelection.createFromDirectories(mDirs.getDirectories(), selection);
+dirSelection.setExpandedPartial();
+return new DynamicDrillTable(fsPlugin, storageEngineName,
userName,
--- End diff --
A comment then, perhaps.
> Improve metadata cache performance for queries with single partition
> -
>
> Key: DRILL-4530
> URL: https://issues.apache.org/jira/browse/DRILL-4530
> Project: Apache Drill
> Issue Type: Improvement
> Components: Query Planning & Optimization
>Affects Versions: 1.6.0
>Reporter: Aman Sinha
>Assignee: Aman Sinha
> Fix For: 1.7.0
>
>
> Consider two types of queries which are run with Parquet metadata caching:
> {noformat}
> query 1:
> SELECT col FROM `A/B/C`;
> query 2:
> SELECT col FROM `A` WHERE dir0 = 'B' AND dir1 = 'C';
> {noformat}
> For a certain dataset, the query1 elapsed time is 1 sec whereas query2
> elapsed time is 9 sec even though both are accessing the same amount of data.
> The user expectation is that they should perform roughly the same. The main
> difference comes from reading the bigger metadata cache file at the root
> level 'A' for query2 and then applying the partitioning filter. query1 reads
> a much smaller metadata cache file at the subdirectory level.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-4530) Improve metadata cache performance for queries with single partition
[
https://issues.apache.org/jira/browse/DRILL-4530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15333132#comment-15333132
]
ASF GitHub Bot commented on DRILL-4530:
---
Github user amansinha100 commented on a diff in the pull request:
https://github.com/apache/drill/pull/519#discussion_r67289117
--- Diff:
exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/ParquetFormatPlugin.java
---
@@ -208,8 +209,18 @@ public DrillTable isReadable(DrillFileSystem fs,
FileSelection selection,
FileSystemPlugin fsPlugin, String storageEngineName, String
userName)
throws IOException {
// TODO: we only check the first file for directory reading.
- if(selection.containsDirectories(fs)){
-if(isDirReadable(fs, selection.getFirstPath(fs))){
+ if(selection.containsDirectories(fs)) {
+Path dirMetaPath = new Path(selection.getSelectionRoot(),
Metadata.METADATA_DIRECTORIES_FILENAME);
+if (fs.exists(dirMetaPath)) {
+ ParquetTableMetadataDirs mDirs = Metadata.readMetadataDirs(fs,
dirMetaPath.toString());
+ if (mDirs.getDirectories().size() > 0) {
+FileSelection dirSelection =
FileSelection.createFromDirectories(mDirs.getDirectories(), selection);
+dirSelection.setExpandedPartial();
+return new DynamicDrillTable(fsPlugin, storageEngineName,
userName,
--- End diff --
I intentionally don't call isDirReadable() here because that method returns
true if a metadata cache file exists and I am doing a similar check for the
directories file here with fs.exists(dirMetaPath).If this check fails, we
will fall through to the old code path (line 223) which does check
isDirReadable().
> Improve metadata cache performance for queries with single partition
> -
>
> Key: DRILL-4530
> URL: https://issues.apache.org/jira/browse/DRILL-4530
> Project: Apache Drill
> Issue Type: Improvement
> Components: Query Planning & Optimization
>Affects Versions: 1.6.0
>Reporter: Aman Sinha
>Assignee: Aman Sinha
> Fix For: 1.7.0
>
>
> Consider two types of queries which are run with Parquet metadata caching:
> {noformat}
> query 1:
> SELECT col FROM `A/B/C`;
> query 2:
> SELECT col FROM `A` WHERE dir0 = 'B' AND dir1 = 'C';
> {noformat}
> For a certain dataset, the query1 elapsed time is 1 sec whereas query2
> elapsed time is 9 sec even though both are accessing the same amount of data.
> The user expectation is that they should perform roughly the same. The main
> difference comes from reading the bigger metadata cache file at the root
> level 'A' for query2 and then applying the partitioning filter. query1 reads
> a much smaller metadata cache file at the subdirectory level.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-4530) Improve metadata cache performance for queries with single partition
[
https://issues.apache.org/jira/browse/DRILL-4530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15330924#comment-15330924
]
Aman Sinha commented on DRILL-4530:
---
[~jni] could you pls review the partition pruning and file selection related
changes. [~parthc] could you pls review the metadata changes ? I have
requested performance testing with the large metadata cache files (for 100K
files) and will report back the numbers in the next day or two.
> Improve metadata cache performance for queries with single partition
> -
>
> Key: DRILL-4530
> URL: https://issues.apache.org/jira/browse/DRILL-4530
> Project: Apache Drill
> Issue Type: Improvement
> Components: Query Planning & Optimization
>Affects Versions: 1.6.0
>Reporter: Aman Sinha
>Assignee: Aman Sinha
> Fix For: 1.7.0
>
>
> Consider two types of queries which are run with Parquet metadata caching:
> {noformat}
> query 1:
> SELECT col FROM `A/B/C`;
> query 2:
> SELECT col FROM `A` WHERE dir0 = 'B' AND dir1 = 'C';
> {noformat}
> For a certain dataset, the query1 elapsed time is 1 sec whereas query2
> elapsed time is 9 sec even though both are accessing the same amount of data.
> The user expectation is that they should perform roughly the same. The main
> difference comes from reading the bigger metadata cache file at the root
> level 'A' for query2 and then applying the partitioning filter. query1 reads
> a much smaller metadata cache file at the subdirectory level.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-4530) Improve metadata cache performance for queries with single partition
[
https://issues.apache.org/jira/browse/DRILL-4530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15326690#comment-15326690
]
Aman Sinha commented on DRILL-4530:
---
I have created a PR for this. All unit and functional tests are clean. I
haven't yet run performance tests. The changes are in 3 broad areas: (a)
file/dir selection (b) partition pruning, (c) metadata cache. The changes in
(a) and (b) are mostly independent of the changes in (c) which relies on a
separate directories file. In the future we could swap out the changes in (c)
whenever the metadata cache is enhanced to allow faster access to the
directories field.
Feedback is welcome.
> Improve metadata cache performance for queries with single partition
> -
>
> Key: DRILL-4530
> URL: https://issues.apache.org/jira/browse/DRILL-4530
> Project: Apache Drill
> Issue Type: Improvement
> Components: Query Planning & Optimization
>Affects Versions: 1.6.0
>Reporter: Aman Sinha
>Assignee: Aman Sinha
> Fix For: 1.7.0
>
>
> Consider two types of queries which are run with Parquet metadata caching:
> {noformat}
> query 1:
> SELECT col FROM `A/B/C`;
> query 2:
> SELECT col FROM `A` WHERE dir0 = 'B' AND dir1 = 'C';
> {noformat}
> For a certain dataset, the query1 elapsed time is 1 sec whereas query2
> elapsed time is 9 sec even though both are accessing the same amount of data.
> The user expectation is that they should perform roughly the same. The main
> difference comes from reading the bigger metadata cache file at the root
> level 'A' for query2 and then applying the partitioning filter. query1 reads
> a much smaller metadata cache file at the subdirectory level.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-4530) Improve metadata cache performance for queries with single partition
[
https://issues.apache.org/jira/browse/DRILL-4530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15326679#comment-15326679
]
ASF GitHub Bot commented on DRILL-4530:
---
GitHub user amansinha100 opened a pull request:
https://github.com/apache/drill/pull/519
DRILL-4530: Optimize partition pruning with metadata caching for the …
…single partition case.
- Enhance PruneScanRule to detect single partitions based on referenced
dirs in the filter.
- Keep a new status of EXPANDED_PARTIAL for FileSelection.
- Create separate .directories metadata file to prune directories first
before files.
- Introduce cacheFileRoot attribute to keep track of the parent directory
of the cache file after partition pruning.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/amansinha100/incubator-drill DRILL-4530-1
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/drill/pull/519.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #519
commit 9c9687e804fa05c8f4b7b065738c458cb88bf5c4
Author: Aman Sinha
Date: 2016-03-25T19:55:59Z
DRILL-4530: Optimize partition pruning with metadata caching for the single
partition case.
- Enhance PruneScanRule to detect single partitions based on referenced
dirs in the filter.
- Keep a new status of EXPANDED_PARTIAL for FileSelection.
- Create separate .directories metadata file to prune directories first
before files.
- Introduce cacheFileRoot attribute to keep track of the parent directory
of the cache file after partition pruning.
> Improve metadata cache performance for queries with single partition
> -
>
> Key: DRILL-4530
> URL: https://issues.apache.org/jira/browse/DRILL-4530
> Project: Apache Drill
> Issue Type: Improvement
> Components: Query Planning & Optimization
>Affects Versions: 1.6.0
>Reporter: Aman Sinha
>Assignee: Aman Sinha
> Fix For: 1.7.0
>
>
> Consider two types of queries which are run with Parquet metadata caching:
> {noformat}
> query 1:
> SELECT col FROM `A/B/C`;
> query 2:
> SELECT col FROM `A` WHERE dir0 = 'B' AND dir1 = 'C';
> {noformat}
> For a certain dataset, the query1 elapsed time is 1 sec whereas query2
> elapsed time is 9 sec even though both are accessing the same amount of data.
> The user expectation is that they should perform roughly the same. The main
> difference comes from reading the bigger metadata cache file at the root
> level 'A' for query2 and then applying the partitioning filter. query1 reads
> a much smaller metadata cache file at the subdirectory level.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-4530) Improve metadata cache performance for queries with single partition
[
https://issues.apache.org/jira/browse/DRILL-4530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15228689#comment-15228689
]
John Omernik commented on DRILL-4530:
-
Let me add a big +1 to using protobuf for the cache. We could even include a
simple jar with Drill to decode the protobuf to json for human
reading/troubleshooting. If you consider how many times a human would read the
metadata cache vs. how many times Drill will do it without human eyes, json
does not provide any appreciable advantage over protobuf, especially if we just
include a jar we can use to ready any protobuf file as json when needed.
> Improve metadata cache performance for queries with single partition
> -
>
> Key: DRILL-4530
> URL: https://issues.apache.org/jira/browse/DRILL-4530
> Project: Apache Drill
> Issue Type: Improvement
> Components: Query Planning & Optimization
>Affects Versions: 1.6.0
>Reporter: Aman Sinha
>Assignee: Aman Sinha
> Fix For: 1.7.0
>
>
> Consider two types of queries which are run with Parquet metadata caching:
> {noformat}
> query 1:
> SELECT col FROM `A/B/C`;
> query 2:
> SELECT col FROM `A` WHERE dir0 = 'B' AND dir1 = 'C';
> {noformat}
> For a certain dataset, the query1 elapsed time is 1 sec whereas query2
> elapsed time is 9 sec even though both are accessing the same amount of data.
> The user expectation is that they should perform roughly the same. The main
> difference comes from reading the bigger metadata cache file at the root
> level 'A' for query2 and then applying the partitioning filter. query1 reads
> a much smaller metadata cache file at the subdirectory level.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-4530) Improve metadata cache performance for queries with single partition
[
https://issues.apache.org/jira/browse/DRILL-4530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15228674#comment-15228674
]
Aman Sinha commented on DRILL-4530:
---
Yes, indeed the storage format of the metadata cache has been discussed a few
times and various options are on the table (I believe [~parthc] has done some
analysis of the options). Thanks for the experimentation using protobuf. The
loading time improvements are quite impressive. The advantages of JSON
(simple, human readable etc.) are outweighed by the performance tradeoffs. In
any new option we consider, we must keep in mind the fast incremental refresh
scenario - this feature is highly requested by all users who are using metadata
cache.
> Improve metadata cache performance for queries with single partition
> -
>
> Key: DRILL-4530
> URL: https://issues.apache.org/jira/browse/DRILL-4530
> Project: Apache Drill
> Issue Type: Improvement
> Components: Query Planning & Optimization
>Affects Versions: 1.6.0
>Reporter: Aman Sinha
>Assignee: Aman Sinha
> Fix For: 1.7.0
>
>
> Consider two types of queries which are run with Parquet metadata caching:
> {noformat}
> query 1:
> SELECT col FROM `A/B/C`;
> query 2:
> SELECT col FROM `A` WHERE dir0 = 'B' AND dir1 = 'C';
> {noformat}
> For a certain dataset, the query1 elapsed time is 1 sec whereas query2
> elapsed time is 9 sec even though both are accessing the same amount of data.
> The user expectation is that they should perform roughly the same. The main
> difference comes from reading the bigger metadata cache file at the root
> level 'A' for query2 and then applying the partitioning filter. query1 reads
> a much smaller metadata cache file at the subdirectory level.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-4530) Improve metadata cache performance for queries with single partition
[
https://issues.apache.org/jira/browse/DRILL-4530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15227790#comment-15227790
]
Deneche A. Hakim commented on DRILL-4530:
-
I did an experiment where I *hacked* Drill to use protobuf instead of json for
the metadata cache and for a customer case with a parquet table with 3 levels
of directories and 395250 files, the protobuf cache was 87% smaller than json
and loaded 83% faster.
> Improve metadata cache performance for queries with single partition
> -
>
> Key: DRILL-4530
> URL: https://issues.apache.org/jira/browse/DRILL-4530
> Project: Apache Drill
> Issue Type: Improvement
> Components: Query Planning & Optimization
>Affects Versions: 1.6.0
>Reporter: Aman Sinha
>Assignee: Aman Sinha
> Fix For: 1.7.0
>
>
> Consider two types of queries which are run with Parquet metadata caching:
> {noformat}
> query 1:
> SELECT col FROM `A/B/C`;
> query 2:
> SELECT col FROM `A` WHERE dir0 = 'B' AND dir1 = 'C';
> {noformat}
> For a certain dataset, the query1 elapsed time is 1 sec whereas query2
> elapsed time is 9 sec even though both are accessing the same amount of data.
> The user expectation is that they should perform roughly the same. The main
> difference comes from reading the bigger metadata cache file at the root
> level 'A' for query2 and then applying the partitioning filter. query1 reads
> a much smaller metadata cache file at the subdirectory level.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-4530) Improve metadata cache performance for queries with single partition
[
https://issues.apache.org/jira/browse/DRILL-4530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15227785#comment-15227785
]
Deneche A. Hakim commented on DRILL-4530:
-
Another alternative is to use protobuf for the metadata cache file. Not only it
will use less space and load faster but you can actually only load a subset of
the cache in the beginning (e.g. directory list).
If done right, Drill will be remain backward compatible and use the proper
metadata loader depending on the metadata cache format. The only real drawback
is that it will require a lot more work than adding a separate
.parquet_directories file
> Improve metadata cache performance for queries with single partition
> -
>
> Key: DRILL-4530
> URL: https://issues.apache.org/jira/browse/DRILL-4530
> Project: Apache Drill
> Issue Type: Improvement
> Components: Query Planning & Optimization
>Affects Versions: 1.6.0
>Reporter: Aman Sinha
>Assignee: Aman Sinha
> Fix For: 1.7.0
>
>
> Consider two types of queries which are run with Parquet metadata caching:
> {noformat}
> query 1:
> SELECT col FROM `A/B/C`;
> query 2:
> SELECT col FROM `A` WHERE dir0 = 'B' AND dir1 = 'C';
> {noformat}
> For a certain dataset, the query1 elapsed time is 1 sec whereas query2
> elapsed time is 9 sec even though both are accessing the same amount of data.
> The user expectation is that they should perform roughly the same. The main
> difference comes from reading the bigger metadata cache file at the root
> level 'A' for query2 and then applying the partitioning filter. query1 reads
> a much smaller metadata cache file at the subdirectory level.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-4530) Improve metadata cache performance for queries with single partition
[
https://issues.apache.org/jira/browse/DRILL-4530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15220436#comment-15220436
]
Aman Sinha commented on DRILL-4530:
---
In order to address this in a reasonably efficient way, one option I am
exploring is to keep a separate ".parquet_directories" file similar to the
parquet metadata cache file. The directories file is just a copy of only the
"directories" field of the metadata cache (original metadata cache file remains
intact). The motivation for doing this is the list of directories is much
smaller and I can apply the Partition Pruning only on directories first; this
allows optimizations such as detecting a single partition and reading a smaller
metadata cache file from that partition. [~jnadeau] have you or others
explored keeping the directories list separately ? I am not proposing to break
up the existing cache file (at least not for now ... I am aware based on
discussion with [~parthc] that it could break backward compatibility).
> Improve metadata cache performance for queries with single partition
> -
>
> Key: DRILL-4530
> URL: https://issues.apache.org/jira/browse/DRILL-4530
> Project: Apache Drill
> Issue Type: Improvement
> Components: Query Planning & Optimization
>Affects Versions: 1.6.0
>Reporter: Aman Sinha
>Assignee: Aman Sinha
> Fix For: 1.7.0
>
>
> Consider two types of queries which are run with Parquet metadata caching:
> {noformat}
> query 1:
> SELECT col FROM `A/B/C`;
> query 2:
> SELECT col FROM `A` WHERE dir0 = 'B' AND dir1 = 'C';
> {noformat}
> For a certain dataset, the query1 elapsed time is 1 sec whereas query2
> elapsed time is 9 sec even though both are accessing the same amount of data.
> The user expectation is that they should perform roughly the same. The main
> difference comes from reading the bigger metadata cache file at the root
> level 'A' for query2 and then applying the partitioning filter. query1 reads
> a much smaller metadata cache file at the subdirectory level.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-4530) Improve metadata cache performance for queries with single partition
[
https://issues.apache.org/jira/browse/DRILL-4530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15208363#comment-15208363
]
John Omernik commented on DRILL-4530:
-
This is at least partially related to
https://issues.apache.org/jira/browse/DRILL-3759 and
https://issues.apache.org/jira/browse/DRILL-2517(which was added to 1.6). In
that when you have many partitions, it walks the partitions (and likely
metadata cache files when they exist) and even looks at files in the
directories before the pruning with dir0 and dir1 occurs. I think 2517
addressed some of that, I saw my queries on large parquet tables get much
faster, but still not `A/B/C` fast which requires no walking of directories.
> Improve metadata cache performance for queries with single partition
> -
>
> Key: DRILL-4530
> URL: https://issues.apache.org/jira/browse/DRILL-4530
> Project: Apache Drill
> Issue Type: Improvement
> Components: Query Planning & Optimization
>Affects Versions: 1.6.0
>Reporter: Aman Sinha
>Assignee: Aman Sinha
>
> Consider two types of queries which are run with Parquet metadata caching:
> {noformat}
> query 1:
> SELECT col FROM `A/B/C`;
> query 2:
> SELECT col FROM `A` WHERE dir0 = 'B' AND dir1 = 'C';
> {noformat}
> For a certain dataset, the query1 elapsed time is 1 sec whereas query2
> elapsed time is 9 sec even though both are accessing the same amount of data.
> The user expectation is that they should perform roughly the same. The main
> difference comes from reading the bigger metadata cache file at the root
> level 'A' for query2 and then applying the partitioning filter. query1 reads
> a much smaller metadata cache file at the subdirectory level.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-4530) Improve metadata cache performance for queries with single partition
[
https://issues.apache.org/jira/browse/DRILL-4530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15207648#comment-15207648
]
Aman Sinha commented on DRILL-4530:
---
On a closer look, we can optimize this type of query which has a single
partition by changing the selection root after partition pruning rule is
applied. While populating the partition vectors in ParquetPartitionDescriptor,
we can keep track of the dir0, dir1 etc. values that have been encountered and
if these are unique, then we can change the selection root to 'A/B/C'. This
will subsequently get used by ParquetGroupScan to read the metadata cache file
from the subdirectory instead of top level.
Note that this type of optimization would apply only when we know that a single
partition is being selected. For a general case (e.g IN filter or range
predicate on the directory column) other solutions need to be designed.
However, for the general case it is unlikely that a user will compare its
performance with an equivalent query written without the partition filter (it
is not easy to express arbitrary WHERE condition into a FROM clause).
> Improve metadata cache performance for queries with single partition
> -
>
> Key: DRILL-4530
> URL: https://issues.apache.org/jira/browse/DRILL-4530
> Project: Apache Drill
> Issue Type: Improvement
> Components: Query Planning & Optimization
>Affects Versions: 1.6.0
>Reporter: Aman Sinha
>Assignee: Aman Sinha
>
> Consider two types of queries which are run with Parquet metadata caching:
> {noformat}
> query 1:
> SELECT col FROM `A/B/C`;
> query 2:
> SELECT col FROM `A` WHERE dir0 = 'B' AND dir1 = 'C';
> {noformat}
> For a certain dataset, the query1 elapsed time is 1 sec whereas query1
> elapsed time is 9 sec even though both are accessing the same amount of data.
> The user expectation is that they should perform roughly the same. The main
> difference comes from reading the bigger metadata cache file at the root
> level 'A' for query2 and then applying the partitioning filter. query1 reads
> a much smaller metadata cache file at the subdirectory level.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
