[
https://issues.apache.org/jira/browse/TAJO-1952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15023626#comment-15023626
]
ASF GitHub Bot commented on TAJO-1952:
--------------------------------------
Github user hyunsik commented on a diff in the pull request:
https://github.com/apache/tajo/pull/846#discussion_r45691190
--- Diff:
tajo-plan/src/main/java/org/apache/tajo/plan/rewrite/rules/PartitionedTableRewriter.java
---
@@ -216,7 +228,22 @@ public String toString() {
// Get all file status matched to a ith level path filter.
filteredPaths = toPathArray(fs.listStatus(filteredPaths,
filters[i]));
}
- return filteredPaths;
+
+ // Get partition keys and volume from the list of partition directories
+ partitionKeys = new String[filteredPaths.length];
+ for (int i = 0; i < partitionKeys.length; i++) {
+ Path path = filteredPaths[i];
+ startIdx =
path.toString().indexOf(getColumnPartitionPathPrefix(partitionColumns));
+ partitionKeys[i] = path.toString().substring(startIdx);
+ summary = fs.getContentSummary(path);
+ totalVolume += summary.getLength();
+ }
+
+ partitionContent.setPartitionPaths(filteredPaths);
--- End diff --
It can be refactored to use just the constructor instead of setters.
> Implement PartitionedFileFragment
> ---------------------------------
>
> Key: TAJO-1952
> URL: https://issues.apache.org/jira/browse/TAJO-1952
> Project: Tajo
> Issue Type: Improvement
> Components: Planner/Optimizer, Storage
> Reporter: Jaehwa Jung
> Assignee: Jaehwa Jung
> Fix For: 0.12.0, 0.11.1
>
> Attachments: TAJO-1952.patch
>
>
> Currently, PartitionedTableScanNode contains the list of partitions and it
> seems to me that the list has some problems as following:
> 1. Duplicate Informs: Task contains Fragment which specify target directory
> or target file for scanning. A path of partition lists already would write to
> Fragment.
> 2. Network Resource: When scanning lost of partition, it will occupy network
> resource, for example, several hundred kilobytes or more. It looks like an
> unnecessary resource because Fragment already has the path of partitions.
> I want to improve above problems by implementing new Fragment called
> PartitionedFileFragment. Currently, I'm planning the implementation as
> following:
> * PartitionedFileFragment will borrow FileFragment and it contains the
> partition path and the partition key values.
> * Remove the path array of partitions from PartitionedTableScanNode.
> * Implement a method for getting filtered partition directories in
> FileTableSpace.
> * Implement a method for making PartitionedFileFragment array.
> * Before making splits, call above method and use it for making splits.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
