[GitHub] incubator-carbondata pull request #659: [CARBONDATA-781] Store one SegmentPr...
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/incubator-carbondata/pull/659#discussion_r110810313
--- Diff:
core/src/main/java/org/apache/carbondata/core/datastore/SegmentTaskIndexStore.java
---
@@ -338,14 +339,28 @@ private synchronized Object
addAndGetSegmentLock(String segmentId) {
* @throws IOException
*/
private AbstractIndex loadBlocks(TaskBucketHolder taskBucketHolder,
- List tableBlockInfoList, AbsoluteTableIdentifier
tableIdentifier)
+ List tableBlockInfoList, AbsoluteTableIdentifier
tableIdentifier,
+ TableSegmentUniqueIdentifier tableSegmentUniqueIdentifier)
throws IOException {
// all the block of one task id will be loaded together
// so creating a list which will have all the data file meta data to
of one task
List footerList = CarbonUtil
.readCarbonIndexFile(taskBucketHolder.taskNo,
taskBucketHolder.bucketNumber,
tableBlockInfoList, tableIdentifier);
-AbstractIndex segment = new SegmentTaskIndex();
+
+if (null == tableSegmentUniqueIdentifier.getSegmentProperties()) {
+ // create a metadata details
+ // this will be useful in query handling
+ // all the data file metadata will have common segment properties we
+ // can use first one to get create the segment properties
+ SegmentProperties segmentProperties =
+ new SegmentProperties(footerList.get(0).getColumnInTable(),
--- End diff --
I think the hashmap approach is OK. You can keep a hashmap in this class
and store the mapping from a key (calculated from schema) to the
SegmentProperties object.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-carbondata pull request #659: [CARBONDATA-781] Store one SegmentPr...
Github user QiangCai commented on a diff in the pull request:
https://github.com/apache/incubator-carbondata/pull/659#discussion_r108146058
--- Diff:
core/src/test/java/org/apache/carbondata/core/datastore/block/SegmentTaskIndexTest.java
---
@@ -58,7 +58,9 @@
@Mock public void build(BTreeBuilderInfo segmentBuilderInfos) {}
};
long numberOfRows = 100;
-SegmentTaskIndex segmentTaskIndex = new SegmentTaskIndex();
+SegmentProperties properties = new
SegmentProperties(footerList.get(0).getColumnInTable(),
--- End diff --
should be after the initialization of variable footerList.
move to line 72
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-carbondata pull request #659: [CARBONDATA-781] Store one SegmentPr...
Github user watermen commented on a diff in the pull request:
https://github.com/apache/incubator-carbondata/pull/659#discussion_r108113194
--- Diff:
core/src/main/java/org/apache/carbondata/core/datastore/block/SegmentTaskIndex.java
---
@@ -16,30 +16,52 @@
*/
package org.apache.carbondata.core.datastore.block;
+import java.util.HashMap;
import java.util.List;
+import java.util.Map;
import org.apache.carbondata.core.datastore.BTreeBuilderInfo;
import org.apache.carbondata.core.datastore.BtreeBuilder;
import org.apache.carbondata.core.datastore.impl.btree.BlockBTreeBuilder;
+import org.apache.carbondata.core.metadata.AbsoluteTableIdentifier;
import org.apache.carbondata.core.metadata.blocklet.DataFileFooter;
/**
* Class which is responsible for loading the b+ tree block. This class
will
* persist all the detail of a table segment
*/
public class SegmentTaskIndex extends AbstractIndex {
+ private static Map
segmentPropertiesCached =
--- End diff --
I didn't find the class before, I will try to add a segmentproperites in
TableSegmentUniqueIdentifier.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-carbondata pull request #659: [CARBONDATA-781] Store one SegmentPr...
Github user QiangCai commented on a diff in the pull request:
https://github.com/apache/incubator-carbondata/pull/659#discussion_r108033699
--- Diff:
core/src/main/java/org/apache/carbondata/core/datastore/block/SegmentTaskIndex.java
---
@@ -16,30 +16,52 @@
*/
package org.apache.carbondata.core.datastore.block;
+import java.util.HashMap;
import java.util.List;
+import java.util.Map;
import org.apache.carbondata.core.datastore.BTreeBuilderInfo;
import org.apache.carbondata.core.datastore.BtreeBuilder;
import org.apache.carbondata.core.datastore.impl.btree.BlockBTreeBuilder;
+import org.apache.carbondata.core.metadata.AbsoluteTableIdentifier;
import org.apache.carbondata.core.metadata.blocklet.DataFileFooter;
/**
* Class which is responsible for loading the b+ tree block. This class
will
* persist all the detail of a table segment
*/
public class SegmentTaskIndex extends AbstractIndex {
+ private static Map
segmentPropertiesCached =
--- End diff --
why not use TableSegmentUniqueIdentifier?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
