[GitHub] carbondata pull request #1359: [CARBONDATA-1480]Min Max Index Example for Da...
Github user sounakr commented on a diff in the pull request: https://github.com/apache/carbondata/pull/1359#discussion_r139068734 --- Diff: core/src/main/java/org/apache/carbondata/core/datamap/dev/DataMapWriter.java --- @@ -32,7 +32,12 @@ /** * End of block notification */ - void onBlockEnd(String blockId); + void onBlockEnd(String blockId, String directoryPath); + + /** + * End of block notification when index got created. + */ + void onBlockEndWithIndex(String blockId, String directoryPath); --- End diff -- onBlockEnd Method is called once the block is written. onBlockEndWithIndex is called once the index is also written after the carbondata is written out. ---
[GitHub] carbondata issue #1355: [CARBONDATA-1452] Issue with loading timestamp data ...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1355 Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/163/ ---
[GitHub] carbondata issue #1356: [CARBONDATA-1477] corrected the bug for wrong values...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/1356 SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/788/ ---
[GitHub] carbondata issue #1337: [CARBONDATA-1445] Fix update fail when carbon.update...
Github user zzcclp commented on the issue: https://github.com/apache/carbondata/pull/1337 retest this please ---
[GitHub] carbondata issue #1353: [CARBONDATA-1476] Added Unit Test Case For Presto
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1353 Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/162/ ---
[GitHub] carbondata issue #1355: [CARBONDATA-1452] Issue with loading timestamp data ...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/1355 retest this please ---
[GitHub] carbondata issue #1354: [CARBONDATA-1348] Sort_Column should not supported f...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/1354 @rahulforallp Please check some tests are failing ---
[GitHub] carbondata issue #1281: [CARBONDATA-1326] Fixed findbug issue and univocity-...
Github user pawanmalwal commented on the issue: https://github.com/apache/carbondata/pull/1281 retest this please ---
[GitHub] carbondata issue #1322: [CARBONDATA-1450] Support timestamp more than 68 yea...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1322 Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/161/ ---
[GitHub] carbondata issue #1353: [CARBONDATA-1476] Added Unit Test Case For Presto
Github user anubhav100 commented on the issue: https://github.com/apache/carbondata/pull/1353 restest this please ---
[GitHub] carbondata issue #1324: [CARBONDATA-1291]:carbonData query performance impro...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/1324 SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/787/ ---
[GitHub] carbondata issue #1354: [CARBONDATA-1348] Sort_Column should not supported f...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1354 Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/160/ ---
[GitHub] carbondata issue #1322: [CARBONDATA-1450] Support timestamp more than 68 yea...
Github user dhatchayani commented on the issue: https://github.com/apache/carbondata/pull/1322 retest this please ---
[GitHub] carbondata pull request #809: [WIP] Configured prefetch in query scanner
Github user ravipesala closed the pull request at: https://github.com/apache/carbondata/pull/809 ---
[GitHub] carbondata pull request #677: [WIP] Added lock log
Github user ravipesala closed the pull request at: https://github.com/apache/carbondata/pull/677 ---
[GitHub] carbondata issue #1354: [CARBONDATA-1348] Sort_Column should not supported f...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/1354 retest this please ---
[GitHub] carbondata issue #1357: [CARBONDATA-1478]update compaction documentation
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/1357 SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/786/ ---
[GitHub] carbondata issue #1256: [CARBONDATA-1380] Fixed updation of load fail to tab...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1256 Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/158/ ---
[GitHub] carbondata pull request #1324: [CARBONDATA-1291]:carbonData query performanc...
Github user asfgit closed the pull request at: https://github.com/apache/carbondata/pull/1324 ---
[jira] [Resolved] (CARBONDATA-1291) CarbonData query performace improvement when number of carbon blocks are high
[ https://issues.apache.org/jira/browse/CARBONDATA-1291?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ravindra Pesala resolved CARBONDATA-1291. - Resolution: Fixed Fix Version/s: 1.2.0 > CarbonData query performace improvement when number of carbon blocks are high > - > > Key: CARBONDATA-1291 > URL: https://issues.apache.org/jira/browse/CARBONDATA-1291 > Project: CarbonData > Issue Type: Bug >Reporter: kumar vishal >Assignee: kumar vishal > Fix For: 1.2.0 > > Time Spent: 3h 10m > Remaining Estimate: 0h > > When number of carbon blocks are high query performance is bad -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[GitHub] carbondata issue #1356: [CARBONDATA-1477] corrected the bug for wrong values...
Github user anubhav100 commented on the issue: https://github.com/apache/carbondata/pull/1356 Retest this please ---
[GitHub] carbondata issue #1324: [CARBONDATA-1291]:carbonData query performance impro...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/1324 LGTM ---
[GitHub] carbondata issue #1256: [CARBONDATA-1380] Fixed updation of load fail to tab...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/1256 retest this please ---
[GitHub] carbondata issue #1324: [CARBONDATA-1291]:carbonData query performance impro...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/1324 retest this please ---
[GitHub] carbondata issue #1337: [CARBONDATA-1445] Fix update fail when carbon.update...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/1337 SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/785/ ---
[GitHub] carbondata pull request #1359: [CARBONDATA-1480]Min Max Index Example for Da...
Github user ravipesala commented on a diff in the pull request: https://github.com/apache/carbondata/pull/1359#discussion_r139059518 --- Diff: examples/spark2/src/main/scala/org/apache/carbondata/examples/MinMaxDataMapFactory.java --- @@ -0,0 +1,141 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + *http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.carbondata.examples; + +import java.io.IOException; +import java.util.ArrayList; +import java.util.Arrays; +import java.util.HashMap; +import java.util.List; +import java.util.Map; + +import org.apache.carbondata.core.cache.Cache; +import org.apache.carbondata.core.cache.CacheProvider; +import org.apache.carbondata.core.cache.CacheType; +import org.apache.carbondata.core.datamap.DataMapDistributable; +import org.apache.carbondata.core.datamap.DataMapMeta; +import org.apache.carbondata.core.datamap.TableDataMap; +import org.apache.carbondata.core.datamap.dev.DataMap; +import org.apache.carbondata.core.datamap.dev.DataMapFactory; +import org.apache.carbondata.core.datamap.dev.DataMapWriter; +import org.apache.carbondata.core.datastore.filesystem.CarbonFile; +import org.apache.carbondata.core.datastore.filesystem.CarbonFileFilter; +import org.apache.carbondata.core.datastore.impl.FileFactory; +import org.apache.carbondata.core.events.ChangeEvent; +import org.apache.carbondata.core.indexstore.TableBlockIndexUniqueIdentifier; +import org.apache.carbondata.core.indexstore.blockletindex.BlockletDataMap; +import org.apache.carbondata.core.indexstore.schema.FilterType; +import org.apache.carbondata.core.memory.MemoryException; +import org.apache.carbondata.core.metadata.AbsoluteTableIdentifier; + + +/** + * Table map for blocklet + */ +public class MinMaxDataMapFactory implements DataMapFactory { + + private AbsoluteTableIdentifier identifier; + + // segmentId -> list of index file + private MapsegmentMap = new HashMap<>(); + + private Cache cache; + + @Override + public void init(AbsoluteTableIdentifier identifier, String dataMapName) { +this.identifier = identifier; +cache = CacheProvider.getInstance() --- End diff -- what is the use of this cache when don't use anywhere ---
[GitHub] carbondata pull request #1359: [CARBONDATA-1480]Min Max Index Example for Da...
Github user ravipesala commented on a diff in the pull request: https://github.com/apache/carbondata/pull/1359#discussion_r139059342 --- Diff: core/src/main/java/org/apache/carbondata/core/datamap/dev/DataMap.java --- @@ -31,7 +31,8 @@ /** * It is called to load the data map to memory or to initialize it. */ - void init(String filePath) throws MemoryException, IOException; + void init(String blockletIndexPath, String customIndexPath, String segmentId) --- End diff -- The `filepath` supposed to be either index folder name or index file name, so I don't think this extra information is required here. And also `blockletIndexPath` is not supposed passed as we have carbonIndex exists in other datamap and we supposed to use it. ---
[GitHub] carbondata pull request #1359: [CARBONDATA-1480]Min Max Index Example for Da...
Github user ravipesala commented on a diff in the pull request: https://github.com/apache/carbondata/pull/1359#discussion_r139058897 --- Diff: core/src/main/java/org/apache/carbondata/core/datamap/dev/DataMapWriter.java --- @@ -32,7 +32,12 @@ /** * End of block notification */ - void onBlockEnd(String blockId); + void onBlockEnd(String blockId, String directoryPath); + + /** + * End of block notification when index got created. + */ + void onBlockEndWithIndex(String blockId, String directoryPath); --- End diff -- Why is this method required, why not `onBlockEnd` is enough? ---
[jira] [Resolved] (CARBONDATA-1401) List Info validate Issue
[ https://issues.apache.org/jira/browse/CARBONDATA-1401?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cao, Lionel resolved CARBONDATA-1401. - Resolution: Fixed > List Info validate Issue > > > Key: CARBONDATA-1401 > URL: https://issues.apache.org/jira/browse/CARBONDATA-1401 > Project: CarbonData > Issue Type: Bug > Components: spark-integration, sql >Reporter: Cao, Lionel >Assignee: Cao, Lionel > Time Spent: 1h 40m > Remaining Estimate: 0h > > fix duplicate issue in list info -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[GitHub] carbondata issue #1337: [CARBONDATA-1445] Fix update fail when carbon.update...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1337 Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/157/ ---
[GitHub] carbondata issue #1357: [CARBONDATA-1478]update compaction documentation
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1357 Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/156/ ---
[GitHub] carbondata issue #1321: [CARBONDATA-1438] Unify the sort column and sort sco...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/1321 SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/784/ ---
[GitHub] carbondata pull request #1357: [CARBONDATA-1478]update compaction documentat...
Github user xubo245 commented on a diff in the pull request: https://github.com/apache/carbondata/pull/1357#discussion_r139051979 --- Diff: docs/data-management.md --- @@ -135,7 +135,7 @@ CLEAN FILES FOR TABLE table1 | Parameter | Default | Application | Description | Valid Values | |-|-|-||--| -| carbon.compaction.level.threshold | 4, 3 | Minor | This property is for minor compaction which decides how many segments to be merged. Example: If it is set as 2, 3 then minor compaction will be triggered for every 2 segments. 3 is the number of level 1 compacted segment which is further compacted to new segment. | NA | +| carbon.compaction.level.threshold | 4, 3 | Minor | This property is for minor compaction which decides how many segments to be merged. Example: If it is set as 2, 3, then minor compaction will be triggered for every 2 segments in level 1. 3 is the number of level 1 compacted segment which is further compacted to new segment in level 2. | NA | --- End diff -- I just have changed 2, 3 to "2, 3". " Minor Compaction" part has explained level 1 compaction and and level 2 compaction ---
[GitHub] carbondata issue #1321: [CARBONDATA-1438] Unify the sort column and sort sco...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1321 Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/155/ ---
[GitHub] carbondata issue #1321: [CARBONDATA-1438] Unify the sort column and sort sco...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/1321 SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/783/ ---
[GitHub] carbondata issue #1321: [CARBONDATA-1438] Unify the sort column and sort sco...
Github user chenerlu commented on the issue: https://github.com/apache/carbondata/pull/1321 retest this please ---
[GitHub] carbondata issue #1321: [CARBONDATA-1438] Unify the sort column and sort sco...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1321 Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/154/ ---
[GitHub] carbondata issue #1359: [CARBONDATA-1480]Min Max Index Example for DataMap
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/1359 SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/782/ ---
[GitHub] carbondata issue #1359: [CARBONDATA-1480]Min Max Index Example for DataMap
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1359 Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/153/ ---
[GitHub] carbondata issue #1359: [CARBONDATA-1480]Min Max Index Example for DataMap
Github user QACarbonData commented on the issue: https://github.com/apache/carbondata/pull/1359 Build Success with Spark 1.6, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/32/ ---
[GitHub] carbondata pull request #1329: [WIP] Min Max DataMap Example
Github user sounakr closed the pull request at: https://github.com/apache/carbondata/pull/1329 ---
[GitHub] carbondata pull request #1359: [CARBONDATA-1480]Min Max DataMap
GitHub user sounakr opened a pull request: https://github.com/apache/carbondata/pull/1359 [CARBONDATA-1480]Min Max DataMap Datamap Example. Implementation of Min Max Index through Datamap. And Using the Index while prunning. --- You can merge this pull request into a Git repository by running: $ git pull https://github.com/sounakr/incubator-carbondata minmax Alternatively you can review and apply these changes as the patch at: https://github.com/apache/carbondata/pull/1359.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1359 commit a46e3b7c609e070f052017edabef9355668cf00a Author: sounakrDate: 2017-09-13T11:57:23Z Min Max DataMap ---
[jira] [Created] (CARBONDATA-1480) Datamap Example. Min Max Index implementation.
sounak chakraborty created CARBONDATA-1480: -- Summary: Datamap Example. Min Max Index implementation. Key: CARBONDATA-1480 URL: https://issues.apache.org/jira/browse/CARBONDATA-1480 Project: CarbonData Issue Type: Bug Reporter: sounak chakraborty Example of DataMap Use Case. Creating a Min Max Index through DataMap implementation and using it in Pruning. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[GitHub] carbondata issue #1355: [CARBONDATA-1452] Issue with loading timestamp data ...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/1355 SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/781/ ---
[GitHub] carbondata issue #1322: [CARBONDATA-1450] Support timestamp more than 68 yea...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/1322 SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/780/ ---
[GitHub] carbondata issue #1355: [CARBONDATA-1452] Issue with loading timestamp data ...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1355 Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/152/ ---
[GitHub] carbondata issue #1297: [CARBONDATA-1429] Add a value based compression for ...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/1297 SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/779/ ---
[GitHub] carbondata issue #1355: [CARBONDATA-1452] Issue with loading timestamp data ...
Github user QACarbonData commented on the issue: https://github.com/apache/carbondata/pull/1355 Build Success with Spark 1.6, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/31/ ---
[GitHub] carbondata issue #1355: [CARBONDATA-1452] Issue with loading timestamp data ...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/1355 retest this please ---
[GitHub] carbondata issue #1336: [CARBONDATA-1425] Inappropriate Exception displays w...
Github user QACarbonData commented on the issue: https://github.com/apache/carbondata/pull/1336 Build Success with Spark 1.6, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/30/ ---
[GitHub] carbondata issue #1336: [CARBONDATA-1425] Inappropriate Exception displays w...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1336 Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/151/ ---
[GitHub] carbondata issue #1322: [CARBONDATA-1450] Support timestamp more than 68 yea...
Github user QACarbonData commented on the issue: https://github.com/apache/carbondata/pull/1322 Build Success with Spark 1.6, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/29/ ---
[GitHub] carbondata issue #1336: [CARBONDATA-1425] Inappropriate Exception displays w...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/1336 retest this please ---
[GitHub] carbondata issue #1321: [CARBONDATA-1438] Unify the sort column and sort sco...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/1321 SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/778/ ---
[GitHub] carbondata issue #1336: [CARBONDATA-1425] Inappropriate Exception displays w...
Github user QACarbonData commented on the issue: https://github.com/apache/carbondata/pull/1336 retest this please ---
[GitHub] carbondata issue #1322: [CARBONDATA-1450] Support timestamp more than 68 yea...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1322 Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/150/ ---
[GitHub] carbondata issue #985: [CARBONDATA-1090] added integration test cases for al...
Github user QACarbonData commented on the issue: https://github.com/apache/carbondata/pull/985 Build Success with Spark 1.6, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/28/ ---
[GitHub] carbondata issue #1297: [CARBONDATA-1429] Add a value based compression for ...
Github user QACarbonData commented on the issue: https://github.com/apache/carbondata/pull/1297 Build Success with Spark 1.6, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/27/ ---
[GitHub] carbondata issue #1321: [CARBONDATA-1438] Unify the sort column and sort sco...
Github user QACarbonData commented on the issue: https://github.com/apache/carbondata/pull/1321 test ---
[GitHub] carbondata issue #1354: [CARBONDATA-1348] Sort_Column should not supported f...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/1354 SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/777/ ---
[GitHub] carbondata issue #1336: [CARBONDATA-1425] Inappropriate Exception displays w...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/1336 SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/776/ ---
[GitHub] carbondata pull request #1297: [CARBONDATA-1429] Add a value based compressi...
Github user manishgupta88 commented on a diff in the pull request: https://github.com/apache/carbondata/pull/1297#discussion_r138941702 --- Diff: core/src/main/java/org/apache/carbondata/core/datastore/page/UnsafeDecimalColumnPage.java --- @@ -0,0 +1,274 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + *http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.carbondata.core.datastore.page; + +import java.math.BigDecimal; + +import org.apache.carbondata.core.datastore.TableSpec; +import org.apache.carbondata.core.memory.CarbonUnsafe; +import org.apache.carbondata.core.memory.MemoryException; +import org.apache.carbondata.core.memory.UnsafeMemoryManager; +import org.apache.carbondata.core.metadata.datatype.DataType; +import org.apache.carbondata.core.util.ByteUtil; + +/** + * Represents a columnar data for decimal data type column for one page + */ +public class UnsafeDecimalColumnPage extends DecimalColumnPage { + + UnsafeDecimalColumnPage(TableSpec.ColumnSpec columnSpec, DataType dataType, int pageSize) + throws MemoryException { +super(columnSpec, dataType, pageSize); +capacity = (int) (pageSize * DEFAULT_ROW_SIZE * FACTOR); +initMemory(); + } + + UnsafeDecimalColumnPage(TableSpec.ColumnSpec columnSpec, DataType dataType, int pageSize, + int capacity) throws MemoryException { +super(columnSpec, dataType, pageSize); +this.capacity = capacity; +initMemory(); + } + + private void initMemory() throws MemoryException { +switch (dataType) { + case BYTE: + case SHORT: + case INT: + case LONG: +int size = pageSize << dataType.getSizeBits(); +memoryBlock = UnsafeMemoryManager.allocateMemoryWithRetry(taskId, size); +baseAddress = memoryBlock.getBaseObject(); +baseOffset = memoryBlock.getBaseOffset(); +break; + case SHORT_INT: +size = pageSize * 3; +memoryBlock = UnsafeMemoryManager.allocateMemoryWithRetry(taskId, size); +baseAddress = memoryBlock.getBaseObject(); +baseOffset = memoryBlock.getBaseOffset(); +break; + case DECIMAL: + case STRING: --- End diff -- yes it will not be string..I will remove ---
[GitHub] carbondata pull request #1297: [CARBONDATA-1429] Add a value based compressi...
Github user manishgupta88 commented on a diff in the pull request: https://github.com/apache/carbondata/pull/1297#discussion_r138941633 --- Diff: core/src/main/java/org/apache/carbondata/core/datastore/page/VarLengthColumnPageBase.java --- @@ -22,21 +22,48 @@ import java.util.List; import org.apache.carbondata.core.datastore.TableSpec; +import org.apache.carbondata.core.memory.CarbonUnsafe; +import org.apache.carbondata.core.memory.MemoryBlock; import org.apache.carbondata.core.memory.MemoryException; +import org.apache.carbondata.core.memory.UnsafeMemoryManager; import org.apache.carbondata.core.metadata.datatype.DataType; import org.apache.carbondata.core.metadata.datatype.DecimalConverterFactory; import org.apache.carbondata.core.util.ByteUtil; +import org.apache.carbondata.core.util.ThreadLocalTaskInfo; +import static org.apache.carbondata.core.metadata.datatype.DataType.BYTE; import static org.apache.carbondata.core.metadata.datatype.DataType.DECIMAL; public abstract class VarLengthColumnPageBase extends ColumnPage { + static final int byteBits = BYTE.getSizeBits(); + static final int shortBits = DataType.SHORT.getSizeBits(); + static final int intBits = DataType.INT.getSizeBits(); + static final int longBits = DataType.LONG.getSizeBits(); + // default size for each row, grows as needed + static final int DEFAULT_ROW_SIZE = 8; + + static final double FACTOR = 1.25; + + final long taskId = ThreadLocalTaskInfo.getCarbonTaskInfo().getTaskId(); + + // memory allocated by Unsafe + MemoryBlock memoryBlock; --- End diff -- SafeDecimalColumnPage will handle both fixed and variable implementation of decimal data based on precsion. So it is also required to be extended from VarLengthColumnPageBase ---
[GitHub] carbondata pull request #1297: [CARBONDATA-1429] Add a value based compressi...
Github user manishgupta88 commented on a diff in the pull request: https://github.com/apache/carbondata/pull/1297#discussion_r138941549 --- Diff: core/src/main/java/org/apache/carbondata/core/datastore/page/VarLengthColumnPageBase.java --- @@ -22,21 +22,48 @@ import java.util.List; import org.apache.carbondata.core.datastore.TableSpec; +import org.apache.carbondata.core.memory.CarbonUnsafe; +import org.apache.carbondata.core.memory.MemoryBlock; import org.apache.carbondata.core.memory.MemoryException; +import org.apache.carbondata.core.memory.UnsafeMemoryManager; import org.apache.carbondata.core.metadata.datatype.DataType; import org.apache.carbondata.core.metadata.datatype.DecimalConverterFactory; import org.apache.carbondata.core.util.ByteUtil; +import org.apache.carbondata.core.util.ThreadLocalTaskInfo; +import static org.apache.carbondata.core.metadata.datatype.DataType.BYTE; import static org.apache.carbondata.core.metadata.datatype.DataType.DECIMAL; public abstract class VarLengthColumnPageBase extends ColumnPage { + static final int byteBits = BYTE.getSizeBits(); + static final int shortBits = DataType.SHORT.getSizeBits(); + static final int intBits = DataType.INT.getSizeBits(); + static final int longBits = DataType.LONG.getSizeBits(); + // default size for each row, grows as needed + static final int DEFAULT_ROW_SIZE = 8; + + static final double FACTOR = 1.25; + + final long taskId = ThreadLocalTaskInfo.getCarbonTaskInfo().getTaskId(); + + // memory allocated by Unsafe + MemoryBlock memoryBlock; --- End diff -- SafeDecimalColumnPage will handle both fixed and variable implementation of decimal data based on precsion. So it is also required to be extended from VarLengthColumnPageBase ---
[GitHub] carbondata issue #1321: [CARBONDATA-1438] Unify the sort column and sort sco...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1321 Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/147/ ---
[GitHub] carbondata issue #1322: [CARBONDATA-1450] Support timestamp more than 68 yea...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1322 Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/146/ ---
[GitHub] carbondata issue #1322: [CARBONDATA-1450] Support timestamp more than 68 yea...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1322 Build Success with Spark 1.6, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/20/ ---
[GitHub] carbondata issue #985: [CARBONDATA-1090] added integration test cases for al...
Github user anubhav100 commented on the issue: https://github.com/apache/carbondata/pull/985 retest this please ---
[GitHub] carbondata issue #1297: [CARBONDATA-1429] Add a value based compression for ...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/1297 SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/775/ ---
[GitHub] carbondata issue #985: [CARBONDATA-1090] added integration test cases for al...
Github user anubhav100 commented on the issue: https://github.com/apache/carbondata/pull/985 this pr required #1356 to be merged first because of wrong values of date type issue in hive ---
[GitHub] carbondata issue #1356: [CARBONDATA-1477] corrected the bug for wrong values...
Github user anubhav100 commented on the issue: https://github.com/apache/carbondata/pull/1356 @chenliang613 can you review this pr? ---
[GitHub] carbondata pull request #1297: [CARBONDATA-1429] Add a value based compressi...
Github user jackylk commented on a diff in the pull request: https://github.com/apache/carbondata/pull/1297#discussion_r138936085 --- Diff: core/src/main/java/org/apache/carbondata/core/datastore/page/VarLengthColumnPageBase.java --- @@ -22,21 +22,48 @@ import java.util.List; import org.apache.carbondata.core.datastore.TableSpec; +import org.apache.carbondata.core.memory.CarbonUnsafe; +import org.apache.carbondata.core.memory.MemoryBlock; import org.apache.carbondata.core.memory.MemoryException; +import org.apache.carbondata.core.memory.UnsafeMemoryManager; import org.apache.carbondata.core.metadata.datatype.DataType; import org.apache.carbondata.core.metadata.datatype.DecimalConverterFactory; import org.apache.carbondata.core.util.ByteUtil; +import org.apache.carbondata.core.util.ThreadLocalTaskInfo; +import static org.apache.carbondata.core.metadata.datatype.DataType.BYTE; import static org.apache.carbondata.core.metadata.datatype.DataType.DECIMAL; public abstract class VarLengthColumnPageBase extends ColumnPage { + static final int byteBits = BYTE.getSizeBits(); + static final int shortBits = DataType.SHORT.getSizeBits(); + static final int intBits = DataType.INT.getSizeBits(); + static final int longBits = DataType.LONG.getSizeBits(); + // default size for each row, grows as needed + static final int DEFAULT_ROW_SIZE = 8; + + static final double FACTOR = 1.25; + + final long taskId = ThreadLocalTaskInfo.getCarbonTaskInfo().getTaskId(); + + // memory allocated by Unsafe + MemoryBlock memoryBlock; --- End diff -- This is for Unsafe column page only, right ---
[GitHub] carbondata pull request #1297: [CARBONDATA-1429] Add a value based compressi...
Github user jackylk commented on a diff in the pull request: https://github.com/apache/carbondata/pull/1297#discussion_r138931778 --- Diff: core/src/main/java/org/apache/carbondata/core/datastore/page/UnsafeDecimalColumnPage.java --- @@ -0,0 +1,274 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + *http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.carbondata.core.datastore.page; + +import java.math.BigDecimal; + +import org.apache.carbondata.core.datastore.TableSpec; +import org.apache.carbondata.core.memory.CarbonUnsafe; +import org.apache.carbondata.core.memory.MemoryException; +import org.apache.carbondata.core.memory.UnsafeMemoryManager; +import org.apache.carbondata.core.metadata.datatype.DataType; +import org.apache.carbondata.core.util.ByteUtil; + +/** + * Represents a columnar data for decimal data type column for one page + */ +public class UnsafeDecimalColumnPage extends DecimalColumnPage { + + UnsafeDecimalColumnPage(TableSpec.ColumnSpec columnSpec, DataType dataType, int pageSize) + throws MemoryException { +super(columnSpec, dataType, pageSize); +capacity = (int) (pageSize * DEFAULT_ROW_SIZE * FACTOR); +initMemory(); + } + + UnsafeDecimalColumnPage(TableSpec.ColumnSpec columnSpec, DataType dataType, int pageSize, + int capacity) throws MemoryException { +super(columnSpec, dataType, pageSize); +this.capacity = capacity; +initMemory(); + } + + private void initMemory() throws MemoryException { +switch (dataType) { + case BYTE: + case SHORT: + case INT: + case LONG: +int size = pageSize << dataType.getSizeBits(); +memoryBlock = UnsafeMemoryManager.allocateMemoryWithRetry(taskId, size); +baseAddress = memoryBlock.getBaseObject(); +baseOffset = memoryBlock.getBaseOffset(); +break; + case SHORT_INT: +size = pageSize * 3; +memoryBlock = UnsafeMemoryManager.allocateMemoryWithRetry(taskId, size); +baseAddress = memoryBlock.getBaseObject(); +baseOffset = memoryBlock.getBaseOffset(); +break; + case DECIMAL: + case STRING: --- End diff -- It can not be String, right? ---
[GitHub] carbondata issue #1297: [CARBONDATA-1429] Add a value based compression for ...
Github user manishgupta88 commented on the issue: https://github.com/apache/carbondata/pull/1297 @jackylk ..handled review comments ---
[GitHub] carbondata issue #1321: [CARBONDATA-1438] Unify the sort column and sort sco...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/1321 SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/774/ ---
[GitHub] carbondata issue #1321: [CARBONDATA-1438] Unify the sort column and sort sco...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1321 Build Failed with Spark 1.6, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/19/ ---
[GitHub] carbondata issue #1321: [CARBONDATA-1438] Unify the sort column and sort sco...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1321 Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/145/ ---
[GitHub] carbondata issue #1324: [CARBONDATA-1291]:carbonData query performance impro...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1324 Build Success with Spark 1.6, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/18/ ---
[GitHub] carbondata issue #1324: [CARBONDATA-1291]:carbonData query performance impro...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1324 Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/144/ ---
[GitHub] carbondata issue #1354: [CARBONDATA-1348] Sort_Column should not supported f...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/1354 SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/773/ ---
[GitHub] carbondata pull request #1297: [CARBONDATA-1429] Add a value based compressi...
Github user jackylk commented on a diff in the pull request: https://github.com/apache/carbondata/pull/1297#discussion_r138916072 --- Diff: core/src/main/java/org/apache/carbondata/core/datastore/page/LazyColumnPage.java --- @@ -91,9 +93,24 @@ public float getFloat(int rowId) { throw new UnsupportedOperationException("internal error"); } - @Override - public BigDecimal getDecimal(int rowId) { -return columnPage.getDecimal(rowId); + @Override public BigDecimal getDecimal(int rowId) { --- End diff -- Can you move all @Override to previous line ---
[GitHub] carbondata pull request #1297: [CARBONDATA-1429] Add a value based compressi...
Github user jackylk commented on a diff in the pull request: https://github.com/apache/carbondata/pull/1297#discussion_r138915973 --- Diff: core/src/main/java/org/apache/carbondata/core/datastore/page/DecimalColumnPage.java --- @@ -0,0 +1,96 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + *http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.carbondata.core.datastore.page; + +import org.apache.carbondata.core.datastore.TableSpec; +import org.apache.carbondata.core.metadata.datatype.DataType; +import org.apache.carbondata.core.metadata.datatype.DecimalConverterFactory; + +/** + * Represent a columnar data in one page for one column of decimal data type + */ +public abstract class DecimalColumnPage extends VarLengthColumnPageBase { + + /** + * decimal converter instance + */ + DecimalConverterFactory.DecimalConverter decimalConverter; + + DecimalColumnPage(TableSpec.ColumnSpec columnSpec, DataType dataType, int pageSize) { +super(columnSpec, dataType, pageSize); +decimalConverter = DecimalConverterFactory.INSTANCE +.getDecimalConverter(columnSpec.getPrecision(), columnSpec.getScale()); + } + + public DecimalConverterFactory.DecimalConverter getDecimalConverter() { +return decimalConverter; + } + + @Override public byte[] getBytePage() { +throw new UnsupportedOperationException("invalid data type: " + dataType); + } + + @Override public short[] getShortPage() { +throw new UnsupportedOperationException("invalid data type: " + dataType); + } + + @Override public byte[] getShortIntPage() { +throw new UnsupportedOperationException("invalid data type: " + dataType); + } + + @Override public int[] getIntPage() { +throw new UnsupportedOperationException("invalid data type: " + dataType); + } + + @Override public long[] getLongPage() { +throw new UnsupportedOperationException("invalid data type: " + dataType); + } + + @Override public float[] getFloatPage() { +throw new UnsupportedOperationException("invalid data type: " + dataType); + } + + @Override public double[] getDoublePage() { --- End diff -- Can you move all `@Override` to previous line ---
[GitHub] carbondata issue #1354: [CARBONDATA-1348] Sort_Column should not supported f...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1354 Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/143/ ---
[GitHub] carbondata issue #1322: [CARBONDATA-1450] Support timestamp more than 68 yea...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/1322 SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/772/ ---
[GitHub] carbondata issue #1354: [CARBONDATA-1348] Sort_Column should not supported f...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1354 Build Success with Spark 1.6, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/17/ ---
[GitHub] carbondata issue #1336: [CARBONDATA-1425] Inappropriate Exception displays w...
Github user chenliang613 commented on the issue: https://github.com/apache/carbondata/pull/1336 retest this please ---
[GitHub] carbondata issue #1355: [CARBONDATA-1452] Issue with loading timestamp data ...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1355 Build Success with Spark 1.6, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/16/ ---
[GitHub] carbondata issue #1297: [CARBONDATA-1429] Add a value based compression for ...
Github user jackylk commented on the issue: https://github.com/apache/carbondata/pull/1297 retest this please ---
[GitHub] carbondata issue #1321: [CARBONDATA-1438] Unify the sort column and sort sco...
Github user chenliang613 commented on the issue: https://github.com/apache/carbondata/pull/1321 retest this please ---
[GitHub] carbondata pull request #661: [CARBONDATA-761] remove shutdown dictionary se...
Github user lionelcao closed the pull request at: https://github.com/apache/carbondata/pull/661 ---
[GitHub] carbondata issue #661: [CARBONDATA-761] remove shutdown dictionary server
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/661 SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/771/ ---
[GitHub] carbondata issue #1337: [CARBONDATA-1445] Fix update fail when carbon.update...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1337 Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/141/ ---
[GitHub] carbondata issue #1337: [CARBONDATA-1445] Fix update fail when carbon.update...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1337 Build Success with Spark 1.6, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/15/ ---
[GitHub] carbondata issue #1354: [CARBONDATA-1348] Sort_Column should not supported f...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/1354 SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/769/ ---
[GitHub] carbondata issue #985: [CARBONDATA-1090] added integration test cases for al...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/985 Build Success with Spark 1.6, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/14/ ---
[GitHub] carbondata issue #985: [CARBONDATA-1090] added integration test cases for al...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/985 Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/140/ ---
[GitHub] carbondata issue #1355: [CARBONDATA-1452] Issue with loading timestamp data ...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/1355 SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/768/ ---
[GitHub] carbondata pull request #1358: [SDV]Disable tests in other modules except cl...
Github user asfgit closed the pull request at: https://github.com/apache/carbondata/pull/1358 ---
[GitHub] carbondata issue #1355: [CARBONDATA-1452] Issue with loading timestamp data ...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/1355 SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/767/ ---
[GitHub] carbondata issue #1358: [SDV]Disable tests in other modules except cluster
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/1358 SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/766/ ---