[GitHub] carbondata issue #2915: [CARBONDATA-3095] Optimize the documentation of SDK/...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2915 Build Success with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/1793/ ---
[GitHub] carbondata pull request #2791: [WIP][HOTFIX]correct the exception handling i...
Github user akashrn5 closed the pull request at: https://github.com/apache/carbondata/pull/2791 ---
[GitHub] carbondata pull request #2960: [WIP] Update the Project List
Github user Indhumathi27 closed the pull request at: https://github.com/apache/carbondata/pull/2960 ---
[GitHub] carbondata pull request #2963: [CARBONDATA-3139] Fix bugs in MinMaxDataMap e...
Github user qiuchenjian commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2963#discussion_r237452138
--- Diff:
datamap/example/src/main/java/org/apache/carbondata/datamap/minmax/AbstractMinMaxDataMapWriter.java
---
@@ -0,0 +1,248 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.carbondata.datamap.minmax;
+
+import java.io.DataOutputStream;
+import java.io.IOException;
+import java.math.BigDecimal;
+import java.util.List;
+
+import org.apache.carbondata.common.logging.LogServiceFactory;
+import org.apache.carbondata.core.constants.CarbonCommonConstants;
+import org.apache.carbondata.core.datamap.Segment;
+import org.apache.carbondata.core.datamap.dev.DataMapWriter;
+import org.apache.carbondata.core.datastore.impl.FileFactory;
+import org.apache.carbondata.core.datastore.page.ColumnPage;
+import
org.apache.carbondata.core.datastore.page.encoding.bool.BooleanConvert;
+import
org.apache.carbondata.core.datastore.page.statistics.ColumnPageStatsCollector;
+import
org.apache.carbondata.core.datastore.page.statistics.KeyPageStatsCollector;
+import
org.apache.carbondata.core.datastore.page.statistics.PrimitivePageStatsCollector;
+import org.apache.carbondata.core.metadata.datatype.DataType;
+import org.apache.carbondata.core.metadata.datatype.DataTypes;
+import org.apache.carbondata.core.metadata.encoder.Encoding;
+import
org.apache.carbondata.core.metadata.schema.table.column.CarbonColumn;
+import org.apache.carbondata.core.util.CarbonUtil;
+import org.apache.carbondata.core.util.DataTypeUtil;
+
+import org.apache.log4j.Logger;
+
+/**
+ * We will record the min & max value for each index column in each
blocklet.
+ * Since the size of index is quite small, we will combine the index for
all index columns
+ * in one file.
+ */
+public abstract class AbstractMinMaxDataMapWriter extends DataMapWriter {
+ private static final Logger LOGGER = LogServiceFactory.getLogService(
+ AbstractMinMaxDataMapWriter.class.getName());
+
+ private ColumnPageStatsCollector[] indexColumnMinMaxCollectors;
+ protected int currentBlockletId;
+ private String currentIndexFile;
+ private DataOutputStream currentIndexFileOutStream;
+
+ public AbstractMinMaxDataMapWriter(String tablePath, String dataMapName,
+ List indexColumns, Segment segment, String shardName)
throws IOException {
+super(tablePath, dataMapName, indexColumns, segment, shardName);
+initStatsCollector();
+initDataMapFile();
+ }
+
+ private void initStatsCollector() {
+indexColumnMinMaxCollectors = new
ColumnPageStatsCollector[indexColumns.size()];
+CarbonColumn indexCol;
+for (int i = 0; i < indexColumns.size(); i++) {
+ indexCol = indexColumns.get(i);
+ if (indexCol.isMeasure()
+ || (indexCol.isDimension()
+ && DataTypeUtil.isPrimitiveColumn(indexCol.getDataType())
+ && !indexCol.hasEncoding(Encoding.DICTIONARY)
+ && !indexCol.hasEncoding(Encoding.DIRECT_DICTIONARY))) {
+indexColumnMinMaxCollectors[i] =
PrimitivePageStatsCollector.newInstance(
+indexColumns.get(i).getDataType());
+ } else {
+indexColumnMinMaxCollectors[i] =
KeyPageStatsCollector.newInstance(DataTypes.BYTE_ARRAY);
+ }
+}
+ }
+
+ private void initDataMapFile() throws IOException {
+if (!FileFactory.isFileExist(dataMapPath) &&
+!FileFactory.mkdirs(dataMapPath,
FileFactory.getFileType(dataMapPath))) {
+ throw new IOException("Failed to create directory " + dataMapPath);
+}
+
+try {
+ currentIndexFile = MinMaxIndexDataMap.getIndexFile(dataMapPath,
+ MinMaxIndexHolder.MINMAX_INDEX_PREFFIX + indexColumns.size());
+ FileFactory.createNewFile(currentIndexFile,
FileFactory.getFileType(currentIndexFile));
+ currentIndexFileOutStream =
[GitHub] carbondata pull request #2964: [HOTFIX] Fix ArrayOutOfBound exception when d...
GitHub user ajantha-bhat opened a pull request: https://github.com/apache/carbondata/pull/2964 [HOTFIX] Fix ArrayOutOfBound exception when duplicate measure in projection column problem: ArrayOutOfBound exception when duplicate measure in the projection column cause: In query executor, when the reusable buffer is formed. It was considering only the unique values. Need to consider all the projections. solution: consider all the projections, while forming a reusable buffer. Be sure to do all of the following checklist to help us incorporate your contribution quickly and easily: - [ ] Any interfaces changed? NA - [ ] Any backward compatibility impacted? NA - [ ] Document update required? NA - [ ] Testing done yes, updated UT. - [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA. NA You can merge this pull request into a Git repository by running: $ git pull https://github.com/ajantha-bhat/carbondata sdk Alternatively you can review and apply these changes as the patch at: https://github.com/apache/carbondata/pull/2964.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2964 commit ca6fd01b92844f43f2472ccf3f17d498bbd73216 Author: ajantha-bhat Date: 2018-11-29T11:42:56Z fix ArrayOutOfBound exception when duplicate measure in projection column ---
[GitHub] carbondata issue #2964: [HOTFIX] Fix ArrayOutOfBound exception when duplicat...
Github user ajantha-bhat commented on the issue: https://github.com/apache/carbondata/pull/2964 @kunal642 , @kumarvishal09 : please check this ---
[GitHub] carbondata issue #2964: [HOTFIX] Fix ArrayOutOfBound exception when duplicat...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2964 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/1582/ ---
[GitHub] carbondata issue #2962: [CARBONDATA-3138] Fix random count mismatch with mul...
Github user kumarvishal09 commented on the issue: https://github.com/apache/carbondata/pull/2962 LGTM ---
[GitHub] carbondata pull request #2962: [CARBONDATA-3138] Fix random count mismatch w...
Github user asfgit closed the pull request at: https://github.com/apache/carbondata/pull/2962 ---
[jira] [Resolved] (CARBONDATA-3138) Random count mismatch in query in multi-thread block-pruning scenario
[ https://issues.apache.org/jira/browse/CARBONDATA-3138?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] kumar vishal resolved CARBONDATA-3138. -- Resolution: Fixed > Random count mismatch in query in multi-thread block-pruning scenario > - > > Key: CARBONDATA-3138 > URL: https://issues.apache.org/jira/browse/CARBONDATA-3138 > Project: CarbonData > Issue Type: Bug >Reporter: Ajantha Bhat >Assignee: Ajantha Bhat >Priority: Major > Time Spent: 1h 10m > Remaining Estimate: 0h > > problem: Random count mismatch in query in multi-thread block-pruning > scenario. > cause:Existing prune method not meant for multi-threading as synchronization > was missiing. > only in implicit filter scenario, while preparing the block ID list, > synchronization was missing. Hence pruning was giving wrong result. > solution: syncronize the imlicit filter prepartion, as prune now called in > multi-thread > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] carbondata issue #2964: [HOTFIX] Fix ArrayOutOfBound exception when duplicat...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2964 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/1583/ ---
[GitHub] carbondata pull request #2965: [Documentation] Editorial review
GitHub user sgururajshetty opened a pull request: https://github.com/apache/carbondata/pull/2965 [Documentation] Editorial review Corrected spelling mistakes and grammer You can merge this pull request into a Git repository by running: $ git pull https://github.com/sgururajshetty/carbondata DTS Alternatively you can review and apply these changes as the patch at: https://github.com/apache/carbondata/pull/2965.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2965 commit 2dd9603aece16a53f781265ebc0db6cd482a4d5f Author: sgururajshetty Date: 2018-11-29T13:14:22Z Spelling mistakes corrected ---
[GitHub] carbondata issue #2964: [HOTFIX] Fix ArrayOutOfBound exception when duplicat...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2964 Build Success with Spark 2.3.1, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/9842/ ---
[GitHub] carbondata issue #2965: [Documentation] Editorial review
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2965 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/1584/ ---
[GitHub] carbondata issue #2964: [HOTFIX] Fix ArrayOutOfBound exception when duplicat...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2964 Build Success with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/1795/ ---
[GitHub] carbondata pull request #2964: [HOTFIX] Fix ArrayOutOfBound exception when d...
Github user qiuchenjian commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2964#discussion_r237505549
--- Diff:
store/sdk/src/test/java/org/apache/carbondata/sdk/file/CarbonReaderTest.java ---
@@ -575,7 +575,7 @@ public void testReadColumnTwice() throws IOException,
InterruptedException {
CarbonReader reader = CarbonReader
.builder(path, "_temp")
-.projection(new String[]{"name", "name", "age", "name"})
+.projection(new String[]{"name", "age", "age", "name"})
--- End diff --
the types of four columns are all String, this skip the measure part. I
think another test is needed, which columns has int or short or long type
---
[GitHub] carbondata issue #2965: [Documentation] Editorial review
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2965 Build Success with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/1796/ ---
[GitHub] carbondata issue #2965: [Documentation] Editorial review
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2965 Build Success with Spark 2.3.1, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/9843/ ---
[GitHub] carbondata issue #2914: [WIP][CARBONDATA-3093] Provide property builder for ...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2914 Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/1585/ ---
[GitHub] carbondata issue #2914: [WIP][CARBONDATA-3093] Provide property builder for ...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2914 Build Failed with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/1797/ ---
[GitHub] carbondata pull request #2966: [WIP] test and check no sort by default
GitHub user ajantha-bhat opened a pull request: https://github.com/apache/carbondata/pull/2966 [WIP] test and check no sort by default [WIP] test and check no sort by default Be sure to do all of the following checklist to help us incorporate your contribution quickly and easily: - [ ] Any interfaces changed? - [ ] Any backward compatibility impacted? - [ ] Document update required? - [ ] Testing done Please provide details on - Whether new unit test cases have been added or why no new tests are required? - How it is tested? Please attach test report. - Is it a performance related change? Please attach the performance test report. - Any additional information to help reviewers in testing this change. - [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA. You can merge this pull request into a Git repository by running: $ git pull https://github.com/ajantha-bhat/carbondata optimize Alternatively you can review and apply these changes as the patch at: https://github.com/apache/carbondata/pull/2966.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2966 commit 09410167efee56d69a3c18433c8e6e97a2fe18eb Author: ajantha-bhat Date: 2018-11-29T15:59:34Z no sort by default ---
[GitHub] carbondata issue #2914: [WIP][CARBONDATA-3093] Provide property builder for ...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2914 Build Failed with Spark 2.3.1, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/9844/ ---
[GitHub] carbondata pull request #2964: [HOTFIX] Fix ArrayOutOfBound exception when d...
Github user ajantha-bhat commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2964#discussion_r237548880
--- Diff:
store/sdk/src/test/java/org/apache/carbondata/sdk/file/CarbonReaderTest.java ---
@@ -575,7 +575,7 @@ public void testReadColumnTwice() throws IOException,
InterruptedException {
CarbonReader reader = CarbonReader
.builder(path, "_temp")
-.projection(new String[]{"name", "name", "age", "name"})
+.projection(new String[]{"name", "age", "age", "name"})
--- End diff --
@qiuchenjian : NO, age is an int column (measure), check schema in writer
buildler.
No need of new test case, now this test case handle testing duplicate
measures and dimensions.
---
[GitHub] carbondata issue #1751: [CARBONDATA-1971][Blocklet Prunning] Measure Null va...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1751 Build Failed with Spark 2.3.1, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/9846/ ---
[GitHub] carbondata issue #2966: [WIP] test and check no sort by default
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2966 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/1586/ ---
[GitHub] carbondata issue #2914: [WIP][CARBONDATA-3093] Provide property builder for ...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2914 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/1587/ ---
[GitHub] carbondata issue #2914: [WIP][CARBONDATA-3093] Provide property builder for ...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2914 Build Failed with Spark 2.3.1, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/9847/ ---
[GitHub] carbondata issue #2914: [WIP][CARBONDATA-3093] Provide property builder for ...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2914 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/1588/ ---
[GitHub] carbondata issue #2966: [WIP] test and check no sort by default
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2966 Build Failed with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/1798/ ---
[GitHub] carbondata issue #2966: [WIP] test and check no sort by default
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2966 Build Failed with Spark 2.3.1, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/9845/ ---
[GitHub] carbondata issue #2914: [WIP][CARBONDATA-3093] Provide property builder for ...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2914 Build Failed with Spark 2.3.1, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/9848/ ---
[GitHub] carbondata issue #2914: [WIP][CARBONDATA-3093] Provide property builder for ...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2914 Build Failed with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/1800/ ---
[GitHub] carbondata issue #2161: [CARBONDATA-2218] AlluxioCarbonFile while trying to ...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2161 Build Failed with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/1801/ ---
[GitHub] carbondata issue #2161: [CARBONDATA-2218] AlluxioCarbonFile while trying to ...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2161 Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/1589/ ---
[GitHub] carbondata issue #2161: [CARBONDATA-2218] AlluxioCarbonFile while trying to ...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2161 Build Failed with Spark 2.3.1, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/9849/ ---
[GitHub] carbondata issue #2161: [CARBONDATA-2218] AlluxioCarbonFile while trying to ...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2161 Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/1590/ ---
[GitHub] carbondata issue #2161: [CARBONDATA-2218] AlluxioCarbonFile while trying to ...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2161 Build Failed with Spark 2.3.1, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/9850/ ---
[GitHub] carbondata issue #2161: [CARBONDATA-2218] AlluxioCarbonFile while trying to ...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2161 Build Failed with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/1802/ ---
[GitHub] carbondata issue #2161: [CARBONDATA-2218] AlluxioCarbonFile while trying to ...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2161 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/1591/ ---
[GitHub] carbondata pull request #2964: [HOTFIX] Fix ArrayOutOfBound exception when d...
Github user qiuchenjian commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2964#discussion_r237715058
--- Diff:
store/sdk/src/test/java/org/apache/carbondata/sdk/file/CarbonReaderTest.java ---
@@ -575,7 +575,7 @@ public void testReadColumnTwice() throws IOException,
InterruptedException {
CarbonReader reader = CarbonReader
.builder(path, "_temp")
-.projection(new String[]{"name", "name", "age", "name"})
+.projection(new String[]{"name", "age", "age", "name"})
--- End diff --
OK, I just noticed this âfields[1] = new Field("age", DataTypes.INT);â
---
[GitHub] carbondata issue #2161: [CARBONDATA-2218] AlluxioCarbonFile while trying to ...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2161 Build Failed with Spark 2.3.1, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/9851/ ---
[GitHub] carbondata issue #2161: [CARBONDATA-2218] AlluxioCarbonFile while trying to ...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2161 Build Failed with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/1803/ ---
[GitHub] carbondata pull request #2961: [CARBONDATA-3119] Fixing the getOrCreateCarbo...
Github user xubo245 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2961#discussion_r237722028
--- Diff:
integration/spark2/src/main/scala/org/apache/spark/sql/CarbonSession.scala ---
@@ -180,7 +180,7 @@ object CarbonSession {
val userSuppliedContext: Option[SparkContext] =
getValue("userSuppliedContext",
builder).asInstanceOf[Option[SparkContext]]
- if (metaStorePath != null) {
+ if (metaStorePath != null && !metaStorePath.trim.isEmpty) {
--- End diff --
Can you add some test case to test it?
---
[GitHub] carbondata pull request #2961: [CARBONDATA-3119] Fixing the getOrCreateCarbo...
Github user xubo245 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2961#discussion_r237722035
--- Diff:
integration/spark2/src/main/scala/org/apache/spark/sql/CarbonSession.scala ---
@@ -248,7 +248,7 @@ object CarbonSession {
session = new CarbonSession(sparkContext, None, !enableInMemCatlog)
val carbonProperties = CarbonProperties.getInstance()
-if (storePath != null) {
+if (storePath != null && !storePath.trim.isEmpty) {
--- End diff --
Can you add some test case to test it?
---
[GitHub] carbondata pull request #2954: [CARBONDATA-3128]Fix the HiveExample exceptio...
Github user xubo245 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2954#discussion_r237722288
--- Diff:
integration/hive/src/test/scala/org/apache/carbondata/hiveexampleCI/RunHiveExample.scala
---
@@ -0,0 +1,43 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.carbondata.hiveexampleCI
+
+import org.apache.spark.sql.test.util.QueryTest
+import org.scalatest.BeforeAndAfterAll
+
+import org.apache.carbondata.core.constants.CarbonCommonConstants
+import org.apache.carbondata.core.util.CarbonProperties
+import org.apache.carbondata.hiveexample.HiveExample
+
+class RunHiveExample extends QueryTest with BeforeAndAfterAll {
--- End diff --
Can you change the class name to RunHiveExampleTest?
---
[GitHub] carbondata pull request #2961: [CARBONDATA-3119] Fixing the getOrCreateCarbo...
Github user zzcclp commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2961#discussion_r237722816
--- Diff:
integration/spark2/src/main/scala/org/apache/spark/sql/CarbonSession.scala ---
@@ -248,7 +248,7 @@ object CarbonSession {
session = new CarbonSession(sparkContext, None, !enableInMemCatlog)
val carbonProperties = CarbonProperties.getInstance()
-if (storePath != null) {
+if (storePath != null && !storePath.trim.isEmpty) {
--- End diff --
Use StringUtils.isNotBlank.
---
[GitHub] carbondata pull request #2961: [CARBONDATA-3119] Fixing the getOrCreateCarbo...
Github user zzcclp commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2961#discussion_r237722775
--- Diff:
integration/spark2/src/main/scala/org/apache/spark/sql/CarbonSession.scala ---
@@ -180,7 +180,7 @@ object CarbonSession {
val userSuppliedContext: Option[SparkContext] =
getValue("userSuppliedContext",
builder).asInstanceOf[Option[SparkContext]]
- if (metaStorePath != null) {
+ if (metaStorePath != null && !metaStorePath.trim.isEmpty) {
--- End diff --
Use StringUtils.isNotBlank.
---
[GitHub] carbondata issue #2954: [CARBONDATA-3128]Fix the HiveExample exception
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2954 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/1592/ ---
[GitHub] carbondata pull request #2966: [WIP] test and check no sort by default
Github user qiuchenjian commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2966#discussion_r237726222
--- Diff:
integration/spark-common-test/src/test/scala/org/apache/carbondata/spark/testsuite/datamap/DataMapWriterSuite.scala
---
@@ -156,8 +156,7 @@ class DataMapWriterSuite extends QueryTest with
BeforeAndAfterAll {
CarbonProperties.getInstance()
.addProperty("carbon.blockletgroup.size.in.mb", "16")
CarbonProperties.getInstance()
- .addProperty("carbon.number.of.cores.while.loading",
-CarbonCommonConstants.NUM_CORES_DEFAULT_VAL)
+ .addProperty("carbon.number.of.cores.while.loading", "2")
--- End diff --
CarbonCommonConstants.NUM_CORES_LOADING
---
[GitHub] carbondata pull request #2161: [CARBONDATA-2218] AlluxioCarbonFile while try...
Github user xubo245 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2161#discussion_r237726325
--- Diff:
core/src/test/java/org/apache/carbondata/core/datastore/filesystem/AlluxioCarbonFileTest.java
---
@@ -108,12 +121,12 @@ public void testListFilesForNullListStatus() {
alluxioCarbonFile = new
AlluxioCarbonFile(fileStatusWithOutDirectoryPermission);
new MockUp() {
@Mock
-public FileSystem getFileSystem(Configuration conf) throws
IOException {
-return new DistributedFileSystem();
+public FileSystem get(FileSystemContext context) throws
IOException {
--- End diff --
Please fix the test error
---
[GitHub] carbondata issue #2961: [CARBONDATA-3119] Fixing the getOrCreateCarbonSessio...
Github user xuchuanyin commented on the issue: https://github.com/apache/carbondata/pull/2961 +1 for @zzcclp 's comments ---
[GitHub] carbondata issue #2954: [CARBONDATA-3128]Fix the HiveExample exception
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2954 Build Success with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/1804/ ---
[GitHub] carbondata issue #2954: [CARBONDATA-3128]Fix the HiveExample exception
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2954 Build Success with Spark 2.3.1, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/9852/ ---
[GitHub] carbondata issue #2954: [CARBONDATA-3128]Fix the HiveExample exception
Github user xubo245 commented on the issue: https://github.com/apache/carbondata/pull/2954 LGTM ---
[GitHub] carbondata issue #2914: [WIP][CARBONDATA-3093] Provide property builder for ...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2914 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/1593/ ---
[GitHub] carbondata pull request #2482: [CARBONDATA-2714] Support merge index files f...
Github user xubo245 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2482#discussion_r237739793
--- Diff:
integration/spark-common-test/src/test/scala/org/apache/carbondata/spark/testsuite/datacompaction/CarbonIndexFileMergeTestCase.scala
---
@@ -215,43 +249,215 @@ class CarbonIndexFileMergeTestCase
Assert
.assertEquals(getIndexOrMergeIndexFileSize(table, "0",
CarbonTablePath.INDEX_FILE_EXT),
segment0.head.getIndexSize.toLong)
-new CarbonIndexFileMergeWriter(table)
- .mergeCarbonIndexFilesOfSegment("0", table.getTablePath, false,
String.valueOf(System.currentTimeMillis()))
+sql("Alter table fileSize compact 'segment_index'")
loadMetadataDetails = SegmentStatusManager
.readTableStatusFile(CarbonTablePath.getTableStatusFilePath(table.getTablePath))
segment0 = loadMetadataDetails.filter(x=>
x.getLoadName.equalsIgnoreCase("0"))
Assert
.assertEquals(getIndexOrMergeIndexFileSize(table, "0",
CarbonTablePath.MERGE_INDEX_FILE_EXT),
segment0.head.getIndexSize.toLong)
+
CarbonProperties.getInstance().addProperty(CarbonCommonConstants.CARBON_MERGE_INDEX_IN_SEGMENT,
"true")
sql("DROP TABLE IF EXISTS fileSize")
}
- private def getIndexFileCount(tableName: String, segmentNo: String): Int
= {
-val carbonTable =
CarbonMetadata.getInstance().getCarbonTable(tableName)
-val segmentDir =
CarbonTablePath.getSegmentPath(carbonTable.getTablePath, segmentNo)
-if (FileFactory.isFileExist(segmentDir)) {
- val indexFiles = new
SegmentIndexFileStore().getIndexFilesFromSegment(segmentDir)
- indexFiles.asScala.map { f =>
-if (f._2 == null) {
- 1
-} else {
- 0
-}
- }.sum
-} else {
- val segment = Segment.getSegment(segmentNo, carbonTable.getTablePath)
- if (segment != null) {
-val store = new SegmentFileStore(carbonTable.getTablePath,
segment.getSegmentFileName)
-store.getSegmentFile.getLocationMap.values().asScala.map { f =>
- if (f.getMergeFileName == null) {
-f.getFiles.size()
- } else {
-0
- }
-}.sum
- } else {
-0
+ test("Verify index merge for compacted segments MINOR - level 2") {
+CarbonProperties.getInstance()
+
.addProperty(CarbonCommonConstants.COMPACTION_SEGMENT_LEVEL_THRESHOLD, "2,2")
+CarbonProperties.getInstance()
+ .addProperty(CarbonCommonConstants.CARBON_MERGE_INDEX_IN_SEGMENT,
"false")
+sql("DROP TABLE IF EXISTS nonindexmerge")
+sql(
+ """
+| CREATE TABLE nonindexmerge(id INT, name STRING, city STRING, age
INT)
+| STORED BY 'org.apache.carbondata.format'
+| TBLPROPERTIES('SORT_COLUMNS'='city,name',
'SORT_SCOPE'='GLOBAL_SORT')
+ """.stripMargin)
+sql(s"LOAD DATA LOCAL INPATH '$file2' INTO TABLE nonindexmerge
OPTIONS('header'='false', " +
+s"'GLOBAL_SORT_PARTITIONS'='100')")
+sql(s"LOAD DATA LOCAL INPATH '$file2' INTO TABLE nonindexmerge
OPTIONS('header'='false', " +
+s"'GLOBAL_SORT_PARTITIONS'='100')")
+sql(s"LOAD DATA LOCAL INPATH '$file2' INTO TABLE nonindexmerge
OPTIONS('header'='false', " +
+s"'GLOBAL_SORT_PARTITIONS'='100')")
+sql(s"LOAD DATA LOCAL INPATH '$file2' INTO TABLE nonindexmerge
OPTIONS('header'='false', " +
+s"'GLOBAL_SORT_PARTITIONS'='100')")
+val rows = sql("""Select count(*) from nonindexmerge""").collect()
+assert(getIndexFileCount("default_nonindexmerge", "0") == 100)
+assert(getIndexFileCount("default_nonindexmerge", "1") == 100)
+assert(getIndexFileCount("default_nonindexmerge", "2") == 100)
+assert(getIndexFileCount("default_nonindexmerge", "3") == 100)
+CarbonProperties.getInstance()
+ .addProperty(CarbonCommonConstants.CARBON_MERGE_INDEX_IN_SEGMENT,
"true")
+sql("ALTER TABLE nonindexmerge COMPACT 'minor'").collect()
+assert(getIndexFileCount("default_nonindexmerge", "0") == 100)
+assert(getIndexFileCount("default_nonindexmerge", "1") == 100)
+assert(getIndexFileCount("default_nonindexmerge", "2") == 100)
+assert(getIndexFileCount("default_nonindexmerge", "3") == 100)
+assert(getIndexFileCount("default_nonindexmerge", "0.1") == 100)
+assert(getIndexFileCount("default_nonindexmerge", "2.1") == 100)
+assert(getIndexFileCount("default_nonindexmerge", "0.2") == 0)
+checkAnswer(sql("""Select count(*) from nonindexmerge"""), rows)
+ }
+
+ test("Verify index merge for compacted segments Auto Compaction") {
+CarbonProperties.getInstance()
+
.addProperty(CarbonCommonConstants.COMPACTION_SEGMEN
[GitHub] carbondata pull request #2482: [CARBONDATA-2714] Support merge index files f...
Github user xubo245 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2482#discussion_r237740164
--- Diff:
integration/spark-common-test/src/test/scala/org/apache/carbondata/spark/testsuite/datacompaction/CarbonIndexFileMergeTestCase.scala
---
@@ -215,43 +249,215 @@ class CarbonIndexFileMergeTestCase
Assert
.assertEquals(getIndexOrMergeIndexFileSize(table, "0",
CarbonTablePath.INDEX_FILE_EXT),
segment0.head.getIndexSize.toLong)
-new CarbonIndexFileMergeWriter(table)
- .mergeCarbonIndexFilesOfSegment("0", table.getTablePath, false,
String.valueOf(System.currentTimeMillis()))
+sql("Alter table fileSize compact 'segment_index'")
loadMetadataDetails = SegmentStatusManager
.readTableStatusFile(CarbonTablePath.getTableStatusFilePath(table.getTablePath))
segment0 = loadMetadataDetails.filter(x=>
x.getLoadName.equalsIgnoreCase("0"))
Assert
.assertEquals(getIndexOrMergeIndexFileSize(table, "0",
CarbonTablePath.MERGE_INDEX_FILE_EXT),
segment0.head.getIndexSize.toLong)
+
CarbonProperties.getInstance().addProperty(CarbonCommonConstants.CARBON_MERGE_INDEX_IN_SEGMENT,
"true")
sql("DROP TABLE IF EXISTS fileSize")
}
- private def getIndexFileCount(tableName: String, segmentNo: String): Int
= {
-val carbonTable =
CarbonMetadata.getInstance().getCarbonTable(tableName)
-val segmentDir =
CarbonTablePath.getSegmentPath(carbonTable.getTablePath, segmentNo)
-if (FileFactory.isFileExist(segmentDir)) {
- val indexFiles = new
SegmentIndexFileStore().getIndexFilesFromSegment(segmentDir)
- indexFiles.asScala.map { f =>
-if (f._2 == null) {
- 1
-} else {
- 0
-}
- }.sum
-} else {
- val segment = Segment.getSegment(segmentNo, carbonTable.getTablePath)
- if (segment != null) {
-val store = new SegmentFileStore(carbonTable.getTablePath,
segment.getSegmentFileName)
-store.getSegmentFile.getLocationMap.values().asScala.map { f =>
- if (f.getMergeFileName == null) {
-f.getFiles.size()
- } else {
-0
- }
-}.sum
- } else {
-0
+ test("Verify index merge for compacted segments MINOR - level 2") {
+CarbonProperties.getInstance()
+
.addProperty(CarbonCommonConstants.COMPACTION_SEGMENT_LEVEL_THRESHOLD, "2,2")
+CarbonProperties.getInstance()
+ .addProperty(CarbonCommonConstants.CARBON_MERGE_INDEX_IN_SEGMENT,
"false")
+sql("DROP TABLE IF EXISTS nonindexmerge")
+sql(
+ """
+| CREATE TABLE nonindexmerge(id INT, name STRING, city STRING, age
INT)
+| STORED BY 'org.apache.carbondata.format'
+| TBLPROPERTIES('SORT_COLUMNS'='city,name',
'SORT_SCOPE'='GLOBAL_SORT')
+ """.stripMargin)
+sql(s"LOAD DATA LOCAL INPATH '$file2' INTO TABLE nonindexmerge
OPTIONS('header'='false', " +
+s"'GLOBAL_SORT_PARTITIONS'='100')")
+sql(s"LOAD DATA LOCAL INPATH '$file2' INTO TABLE nonindexmerge
OPTIONS('header'='false', " +
+s"'GLOBAL_SORT_PARTITIONS'='100')")
+sql(s"LOAD DATA LOCAL INPATH '$file2' INTO TABLE nonindexmerge
OPTIONS('header'='false', " +
+s"'GLOBAL_SORT_PARTITIONS'='100')")
+sql(s"LOAD DATA LOCAL INPATH '$file2' INTO TABLE nonindexmerge
OPTIONS('header'='false', " +
+s"'GLOBAL_SORT_PARTITIONS'='100')")
+val rows = sql("""Select count(*) from nonindexmerge""").collect()
+assert(getIndexFileCount("default_nonindexmerge", "0") == 100)
+assert(getIndexFileCount("default_nonindexmerge", "1") == 100)
+assert(getIndexFileCount("default_nonindexmerge", "2") == 100)
+assert(getIndexFileCount("default_nonindexmerge", "3") == 100)
+CarbonProperties.getInstance()
+ .addProperty(CarbonCommonConstants.CARBON_MERGE_INDEX_IN_SEGMENT,
"true")
+sql("ALTER TABLE nonindexmerge COMPACT 'minor'").collect()
+assert(getIndexFileCount("default_nonindexmerge", "0") == 100)
+assert(getIndexFileCount("default_nonindexmerge", "1") == 100)
+assert(getIndexFileCount("default_nonindexmerge", "2") == 100)
+assert(getIndexFileCount("default_nonindexmerge", "3") == 100)
+assert(getIndexFileCount("default_nonindexmerge", "0.1") == 100)
+assert(getIndexFileCount("default_nonindexmerge", "2.1") == 100)
+assert(getIndexFileCount("default_nonindexmerge", "0.2") == 0)
+checkAnswer(sql("""Select count(*) from nonindexmerge"""), rows)
+ }
+
+ test("Verify index merge for compacted segments Auto Compaction") {
+CarbonProperties.getInstance()
+
.addProperty(CarbonCommonConstants.COMPACTION_SEGMEN
[GitHub] carbondata pull request #2482: [CARBONDATA-2714] Support merge index files f...
Github user xubo245 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2482#discussion_r237740112
--- Diff:
integration/spark-common-test/src/test/scala/org/apache/carbondata/spark/testsuite/datacompaction/CarbonIndexFileMergeTestCase.scala
---
@@ -215,43 +249,215 @@ class CarbonIndexFileMergeTestCase
Assert
.assertEquals(getIndexOrMergeIndexFileSize(table, "0",
CarbonTablePath.INDEX_FILE_EXT),
segment0.head.getIndexSize.toLong)
-new CarbonIndexFileMergeWriter(table)
- .mergeCarbonIndexFilesOfSegment("0", table.getTablePath, false,
String.valueOf(System.currentTimeMillis()))
+sql("Alter table fileSize compact 'segment_index'")
loadMetadataDetails = SegmentStatusManager
.readTableStatusFile(CarbonTablePath.getTableStatusFilePath(table.getTablePath))
segment0 = loadMetadataDetails.filter(x=>
x.getLoadName.equalsIgnoreCase("0"))
Assert
.assertEquals(getIndexOrMergeIndexFileSize(table, "0",
CarbonTablePath.MERGE_INDEX_FILE_EXT),
segment0.head.getIndexSize.toLong)
+
CarbonProperties.getInstance().addProperty(CarbonCommonConstants.CARBON_MERGE_INDEX_IN_SEGMENT,
"true")
sql("DROP TABLE IF EXISTS fileSize")
}
- private def getIndexFileCount(tableName: String, segmentNo: String): Int
= {
-val carbonTable =
CarbonMetadata.getInstance().getCarbonTable(tableName)
-val segmentDir =
CarbonTablePath.getSegmentPath(carbonTable.getTablePath, segmentNo)
-if (FileFactory.isFileExist(segmentDir)) {
- val indexFiles = new
SegmentIndexFileStore().getIndexFilesFromSegment(segmentDir)
- indexFiles.asScala.map { f =>
-if (f._2 == null) {
- 1
-} else {
- 0
-}
- }.sum
-} else {
- val segment = Segment.getSegment(segmentNo, carbonTable.getTablePath)
- if (segment != null) {
-val store = new SegmentFileStore(carbonTable.getTablePath,
segment.getSegmentFileName)
-store.getSegmentFile.getLocationMap.values().asScala.map { f =>
- if (f.getMergeFileName == null) {
-f.getFiles.size()
- } else {
-0
- }
-}.sum
- } else {
-0
+ test("Verify index merge for compacted segments MINOR - level 2") {
+CarbonProperties.getInstance()
+
.addProperty(CarbonCommonConstants.COMPACTION_SEGMENT_LEVEL_THRESHOLD, "2,2")
+CarbonProperties.getInstance()
+ .addProperty(CarbonCommonConstants.CARBON_MERGE_INDEX_IN_SEGMENT,
"false")
+sql("DROP TABLE IF EXISTS nonindexmerge")
+sql(
+ """
+| CREATE TABLE nonindexmerge(id INT, name STRING, city STRING, age
INT)
+| STORED BY 'org.apache.carbondata.format'
+| TBLPROPERTIES('SORT_COLUMNS'='city,name',
'SORT_SCOPE'='GLOBAL_SORT')
+ """.stripMargin)
+sql(s"LOAD DATA LOCAL INPATH '$file2' INTO TABLE nonindexmerge
OPTIONS('header'='false', " +
+s"'GLOBAL_SORT_PARTITIONS'='100')")
+sql(s"LOAD DATA LOCAL INPATH '$file2' INTO TABLE nonindexmerge
OPTIONS('header'='false', " +
+s"'GLOBAL_SORT_PARTITIONS'='100')")
+sql(s"LOAD DATA LOCAL INPATH '$file2' INTO TABLE nonindexmerge
OPTIONS('header'='false', " +
+s"'GLOBAL_SORT_PARTITIONS'='100')")
+sql(s"LOAD DATA LOCAL INPATH '$file2' INTO TABLE nonindexmerge
OPTIONS('header'='false', " +
+s"'GLOBAL_SORT_PARTITIONS'='100')")
+val rows = sql("""Select count(*) from nonindexmerge""").collect()
+assert(getIndexFileCount("default_nonindexmerge", "0") == 100)
+assert(getIndexFileCount("default_nonindexmerge", "1") == 100)
+assert(getIndexFileCount("default_nonindexmerge", "2") == 100)
+assert(getIndexFileCount("default_nonindexmerge", "3") == 100)
+CarbonProperties.getInstance()
+ .addProperty(CarbonCommonConstants.CARBON_MERGE_INDEX_IN_SEGMENT,
"true")
+sql("ALTER TABLE nonindexmerge COMPACT 'minor'").collect()
+assert(getIndexFileCount("default_nonindexmerge", "0") == 100)
+assert(getIndexFileCount("default_nonindexmerge", "1") == 100)
+assert(getIndexFileCount("default_nonindexmerge", "2") == 100)
+assert(getIndexFileCount("default_nonindexmerge", "3") == 100)
+assert(getIndexFileCount("default_nonindexmerge", "0.1") == 100)
+assert(getIndexFileCount("default_nonindexmerge", "2.1") == 100)
+assert(getIndexFileCount("default_nonindexmerge", "0.2") == 0)
+checkAnswer(sql("""Select count(*) from nonindexmerge"""), rows)
+ }
+
+ test("Verify index merge for compacted segments Auto Compaction") {
+CarbonProperties.getInstance()
+
.addProperty(CarbonCommonConstants.COMPACTION_SEGMEN
[GitHub] carbondata pull request #2482: [CARBONDATA-2714] Support merge index files f...
Github user xubo245 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2482#discussion_r237740134
--- Diff:
integration/spark-common-test/src/test/scala/org/apache/carbondata/spark/testsuite/datacompaction/CarbonIndexFileMergeTestCase.scala
---
@@ -215,43 +249,215 @@ class CarbonIndexFileMergeTestCase
Assert
.assertEquals(getIndexOrMergeIndexFileSize(table, "0",
CarbonTablePath.INDEX_FILE_EXT),
segment0.head.getIndexSize.toLong)
-new CarbonIndexFileMergeWriter(table)
- .mergeCarbonIndexFilesOfSegment("0", table.getTablePath, false,
String.valueOf(System.currentTimeMillis()))
+sql("Alter table fileSize compact 'segment_index'")
loadMetadataDetails = SegmentStatusManager
.readTableStatusFile(CarbonTablePath.getTableStatusFilePath(table.getTablePath))
segment0 = loadMetadataDetails.filter(x=>
x.getLoadName.equalsIgnoreCase("0"))
Assert
.assertEquals(getIndexOrMergeIndexFileSize(table, "0",
CarbonTablePath.MERGE_INDEX_FILE_EXT),
segment0.head.getIndexSize.toLong)
+
CarbonProperties.getInstance().addProperty(CarbonCommonConstants.CARBON_MERGE_INDEX_IN_SEGMENT,
"true")
sql("DROP TABLE IF EXISTS fileSize")
}
- private def getIndexFileCount(tableName: String, segmentNo: String): Int
= {
-val carbonTable =
CarbonMetadata.getInstance().getCarbonTable(tableName)
-val segmentDir =
CarbonTablePath.getSegmentPath(carbonTable.getTablePath, segmentNo)
-if (FileFactory.isFileExist(segmentDir)) {
- val indexFiles = new
SegmentIndexFileStore().getIndexFilesFromSegment(segmentDir)
- indexFiles.asScala.map { f =>
-if (f._2 == null) {
- 1
-} else {
- 0
-}
- }.sum
-} else {
- val segment = Segment.getSegment(segmentNo, carbonTable.getTablePath)
- if (segment != null) {
-val store = new SegmentFileStore(carbonTable.getTablePath,
segment.getSegmentFileName)
-store.getSegmentFile.getLocationMap.values().asScala.map { f =>
- if (f.getMergeFileName == null) {
-f.getFiles.size()
- } else {
-0
- }
-}.sum
- } else {
-0
+ test("Verify index merge for compacted segments MINOR - level 2") {
+CarbonProperties.getInstance()
+
.addProperty(CarbonCommonConstants.COMPACTION_SEGMENT_LEVEL_THRESHOLD, "2,2")
+CarbonProperties.getInstance()
+ .addProperty(CarbonCommonConstants.CARBON_MERGE_INDEX_IN_SEGMENT,
"false")
+sql("DROP TABLE IF EXISTS nonindexmerge")
+sql(
+ """
+| CREATE TABLE nonindexmerge(id INT, name STRING, city STRING, age
INT)
+| STORED BY 'org.apache.carbondata.format'
+| TBLPROPERTIES('SORT_COLUMNS'='city,name',
'SORT_SCOPE'='GLOBAL_SORT')
+ """.stripMargin)
+sql(s"LOAD DATA LOCAL INPATH '$file2' INTO TABLE nonindexmerge
OPTIONS('header'='false', " +
+s"'GLOBAL_SORT_PARTITIONS'='100')")
+sql(s"LOAD DATA LOCAL INPATH '$file2' INTO TABLE nonindexmerge
OPTIONS('header'='false', " +
+s"'GLOBAL_SORT_PARTITIONS'='100')")
+sql(s"LOAD DATA LOCAL INPATH '$file2' INTO TABLE nonindexmerge
OPTIONS('header'='false', " +
+s"'GLOBAL_SORT_PARTITIONS'='100')")
+sql(s"LOAD DATA LOCAL INPATH '$file2' INTO TABLE nonindexmerge
OPTIONS('header'='false', " +
+s"'GLOBAL_SORT_PARTITIONS'='100')")
+val rows = sql("""Select count(*) from nonindexmerge""").collect()
+assert(getIndexFileCount("default_nonindexmerge", "0") == 100)
+assert(getIndexFileCount("default_nonindexmerge", "1") == 100)
+assert(getIndexFileCount("default_nonindexmerge", "2") == 100)
+assert(getIndexFileCount("default_nonindexmerge", "3") == 100)
+CarbonProperties.getInstance()
+ .addProperty(CarbonCommonConstants.CARBON_MERGE_INDEX_IN_SEGMENT,
"true")
+sql("ALTER TABLE nonindexmerge COMPACT 'minor'").collect()
+assert(getIndexFileCount("default_nonindexmerge", "0") == 100)
+assert(getIndexFileCount("default_nonindexmerge", "1") == 100)
+assert(getIndexFileCount("default_nonindexmerge", "2") == 100)
+assert(getIndexFileCount("default_nonindexmerge", "3") == 100)
+assert(getIndexFileCount("default_nonindexmerge", "0.1") == 100)
+assert(getIndexFileCount("default_nonindexmerge", "2.1") == 100)
+assert(getIndexFileCount("default_nonindexmerge", "0.2") == 0)
+checkAnswer(sql("""Select count(*) from nonindexmerge"""), rows)
+ }
+
+ test("Verify index merge for compacted segments Auto Compaction") {
+CarbonProperties.getInstance()
+
.addProperty(CarbonCommonConstants.COMPACTION_SEGMEN
[GitHub] carbondata issue #2914: [CARBONDATA-3093] Provide property builder for carbo...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2914 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/1594/ ---
[GitHub] carbondata issue #2961: [CARBONDATA-3119] Fixing the getOrCreateCarbonSessio...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2961 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/1595/ ---
[GitHub] carbondata pull request #2966: [WIP] test and check no sort by default
Github user ajantha-bhat commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2966#discussion_r237747693
--- Diff:
integration/spark-common-test/src/test/scala/org/apache/carbondata/spark/testsuite/datamap/DataMapWriterSuite.scala
---
@@ -156,8 +156,7 @@ class DataMapWriterSuite extends QueryTest with
BeforeAndAfterAll {
CarbonProperties.getInstance()
.addProperty("carbon.blockletgroup.size.in.mb", "16")
CarbonProperties.getInstance()
- .addProperty("carbon.number.of.cores.while.loading",
-CarbonCommonConstants.NUM_CORES_DEFAULT_VAL)
+ .addProperty("carbon.number.of.cores.while.loading", "2")
--- End diff --
CarbonCommonConstants.NUM_CORES_DEFAULT_VAL is removed, so setting
CarbonCommonConstants.NUM_CORES_LOADING
as 2. It is a test code.
---
[GitHub] carbondata pull request #2966: [WIP] test and check no sort by default
Github user ajantha-bhat commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2966#discussion_r237747880
--- Diff:
integration/spark-common-test/src/test/scala/org/apache/carbondata/spark/testsuite/datamap/DataMapWriterSuite.scala
---
@@ -156,8 +156,7 @@ class DataMapWriterSuite extends QueryTest with
BeforeAndAfterAll {
CarbonProperties.getInstance()
.addProperty("carbon.blockletgroup.size.in.mb", "16")
CarbonProperties.getInstance()
- .addProperty("carbon.number.of.cores.while.loading",
-CarbonCommonConstants.NUM_CORES_DEFAULT_VAL)
+ .addProperty("carbon.number.of.cores.while.loading", "2")
--- End diff --
And @qiuchenjian : If you have a doubt in PR changes, you can ask it in
comment sections. Don't add as a review comment. Just add as a comment.
---
[GitHub] carbondata issue #2954: [CARBONDATA-3128]Fix the HiveExample exception
Github user zzcclp commented on the issue: https://github.com/apache/carbondata/pull/2954 LGTM ---
[GitHub] carbondata issue #2914: [CARBONDATA-3093] Provide property builder for carbo...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2914 Build Failed with Spark 2.3.1, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/9854/ ---
[GitHub] carbondata issue #2954: [CARBONDATA-3128]Fix the HiveExample exception
Github user zzcclp commented on the issue: https://github.com/apache/carbondata/pull/2954 Merged. Sorry for not closing pr automatically. ---
[GitHub] carbondata pull request #2954: [CARBONDATA-3128]Fix the HiveExample exceptio...
Github user SteNicholas closed the pull request at: https://github.com/apache/carbondata/pull/2954 ---
[GitHub] carbondata issue #2961: [CARBONDATA-3119] Fixing the getOrCreateCarbonSessio...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2961 Build Success with Spark 2.3.1, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/9855/ ---
[GitHub] carbondata issue #2961: [CARBONDATA-3119] Fixing the getOrCreateCarbonSessio...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2961 Build Success with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/1807/ ---
[GitHub] carbondata issue #2914: [CARBONDATA-3093] Provide property builder for carbo...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2914 Build Success with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/1806/ ---
[GitHub] carbondata issue #2914: [CARBONDATA-3093] Provide property builder for carbo...
Github user xubo245 commented on the issue: https://github.com/apache/carbondata/pull/2914 retest this please ---
[GitHub] carbondata issue #2915: [CARBONDATA-3095] Optimize the documentation of SDK/...
Github user xubo245 commented on the issue: https://github.com/apache/carbondata/pull/2915 @sraghunandan @KanakaKumar @kunal642 CI pass, please check it. ---
[GitHub] carbondata issue #2914: [CARBONDATA-3093] Provide property builder for carbo...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2914 Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/1596/ ---
[GitHub] carbondata pull request #2966: [WIP] test and check no sort by default
Github user qiuchenjian commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2966#discussion_r237763480
--- Diff:
integration/spark-common-test/src/test/scala/org/apache/carbondata/spark/testsuite/datamap/DataMapWriterSuite.scala
---
@@ -156,8 +156,7 @@ class DataMapWriterSuite extends QueryTest with
BeforeAndAfterAll {
CarbonProperties.getInstance()
.addProperty("carbon.blockletgroup.size.in.mb", "16")
CarbonProperties.getInstance()
- .addProperty("carbon.number.of.cores.while.loading",
-CarbonCommonConstants.NUM_CORES_DEFAULT_VAL)
+ .addProperty("carbon.number.of.cores.while.loading", "2")
--- End diff --
OK, I mean you can use CarbonCommonConstants.NUM_CORES_LOADING instead of
"carbon.number.of.cores.while.loading"
---
[GitHub] carbondata issue #2964: [HOTFIX] Fix ArrayOutOfBound exception when duplicat...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2964 LGTM ---
[GitHub] carbondata issue #2914: [CARBONDATA-3093] Provide property builder for carbo...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2914 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/1599/ ---
[GitHub] carbondata issue #2966: [WIP] test and check no sort by default
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2966 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/1600/ ---
