[GitHub] carbondata issue #1525: [CARBONDATA-1751] Make the type of exception and mes...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1525 Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/1265/ ---
[GitHub] carbondata issue #1471: [CARBONDATA-1544][Datamap] Datamap FineGrain impleme...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1471 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/1264/ ---
[GitHub] carbondata pull request #1515: [CARBONDATA-1751] Modify sys.err to AnalysisE...
Github user xubo245 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1515#discussion_r151832180
--- Diff:
integration/spark-common/src/main/scala/org/apache/carbondata/spark/util/CommonUtil.scala
---
@@ -662,16 +663,18 @@ object CommonUtil {
val maxColumnsInt = getMaxColumnValue(maxColumns)
if (maxColumnsInt != null) {
if (columnCountInSchema >= maxColumnsInt) {
-sys.error(s"csv headers should be less than the max columns:
$maxColumnsInt")
+CarbonException.analysisException(
+ s"csv headers should be less than the max columns:
$maxColumnsInt")
} else if (maxColumnsInt >
CSVInputFormat.THRESHOLD_MAX_NUMBER_OF_COLUMNS_FOR_PARSING) {
-sys.error(s"max columns cannot be greater than the threshold
value: ${
- CSVInputFormat.THRESHOLD_MAX_NUMBER_OF_COLUMNS_FOR_PARSING
-}")
+CarbonException.analysisException(
+ s"max columns cannot be greater than the threshold value: ${
--- End diff --
Ok, I have modified it.
---
[GitHub] carbondata pull request #1515: [CARBONDATA-1751] Modify sys.err to AnalysisE...
Github user xubo245 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1515#discussion_r151832176
--- Diff:
integration/spark-common-test/src/test/scala/org/apache/spark/sql/execution/command/CarbonTableSchemaCommonSuite.scala
---
@@ -0,0 +1,69 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.command
+
+import org.apache.spark.sql.AnalysisException
+import org.apache.spark.sql.test.util.QueryTest
+import org.junit.Assert
+import org.scalatest.BeforeAndAfterAll
+
+class CarbonTableSchemaCommonSuite extends QueryTest with
BeforeAndAfterAll {
+
+ test("Creating table: Duplicate dimensions found with name, it should
throw AnalysisException") {
+sql("DROP TABLE IF EXISTS carbon_table")
+try {
+ sql(
+s"""
+ | CREATE TABLE carbon_table(
+ | BB INT, bb char(10)
+ | )
+ | STORED BY 'carbondata'
+ """.stripMargin)
+} catch {
+ case ex: AnalysisException => Assert.assertTrue(true)
+ case ex: Exception => Assert.assertTrue(false)
+}
--- End diff --
Ok, I have fixed it.
---
[GitHub] carbondata pull request #1520: [CARBONDATA-1734] Ignore empty line while rea...
Github user ravipesala commented on a diff in the pull request: https://github.com/apache/carbondata/pull/1520#discussion_r151832059 --- Diff: core/src/main/java/org/apache/carbondata/core/constants/CarbonCommonConstants.java --- @@ -1406,6 +1405,13 @@ public static final String LAST_UPDATE_TIME = "Last Update Time"; + /** + * this will be used to skip / ignore empty lines while loading + */ + public static final String CARBON_SKIP_EMPTY_LINE = "carbon.skip.empty.line"; --- End diff -- I think it should be annoted as @CarbonProperty ---
[GitHub] carbondata pull request #1524: [CARBONDATA-1762] Remove existing column leve...
Github user dhatchayani commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1524#discussion_r151832053
--- Diff:
integration/spark-common/src/main/scala/org/apache/carbondata/spark/load/ValidateUtil.scala
---
@@ -17,35 +17,32 @@
package org.apache.carbondata.spark.load
-import scala.collection.JavaConverters._
+import java.text.SimpleDateFormat
-import org.apache.carbondata.core.constants.CarbonCommonConstants
import org.apache.carbondata.core.metadata.schema.table.CarbonTable
-import org.apache.carbondata.processing.loading.model.CarbonLoadModel
import org.apache.carbondata.processing.loading.sort.SortScopeOptions
import
org.apache.carbondata.spark.exception.MalformedCarbonCommandException
object ValidateUtil {
- def validateDateFormat(dateFormat: String, table: CarbonTable,
tableName: String): Unit = {
-val dimensions = table.getDimensionByTableName(tableName).asScala
+
+ /**
+ * validates both timestamp and date for illegal values
--- End diff --
this validates both timestamp and date format
---
[GitHub] carbondata issue #1515: [CARBONDATA-1751] Modify sys.err to AnalysisExceptio...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1515 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/1263/ ---
[GitHub] carbondata pull request #1471: [CARBONDATA-1544][Datamap] Datamap FineGrain ...
Github user ravipesala commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1471#discussion_r151831837
--- Diff:
core/src/main/java/org/apache/carbondata/core/datamap/dev/BlockletSerializer.java
---
@@ -0,0 +1,57 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.carbondata.core.datamap.dev;
+
+import java.io.DataInputStream;
+import java.io.DataOutputStream;
+import java.io.IOException;
+
+import org.apache.carbondata.core.datastore.impl.FileFactory;
+import org.apache.carbondata.core.indexstore.FineGrainBlocklet;
+
+public class BlockletSerializer {
+
+ /**
+ * Serialize and write blocklet to the file.
+ * @param grainBlocklet
+ * @param writePath
+ * @throws IOException
+ */
+ public void serializeBlocklet(FineGrainBlocklet grainBlocklet, String
writePath)
--- End diff --
writePath is part of ExtendedBlocklet and FineGrainBlocklet only used till
data is written in TableDataMap. And also we need to add writepath to
TableBlockinfo as it does not have FineGrainBlocklet.
---
[GitHub] carbondata pull request #1524: [CARBONDATA-1762] Remove existing column leve...
Github user jackylk commented on a diff in the pull request: https://github.com/apache/carbondata/pull/1524#discussion_r151831713 --- Diff: core/src/main/java/org/apache/carbondata/core/constants/CarbonLoadOptionConstants.java --- @@ -52,6 +52,14 @@ public static final String CARBON_OPTIONS_DATEFORMAT = "carbon.options.dateformat"; public static final String CARBON_OPTIONS_DATEFORMAT_DEFAULT = ""; + + /** + * option to specify the load option + */ + @CarbonProperty + public static final String CARBON_OPTIONS_TIMESTAMPFORMAT = --- End diff -- please add comment for these two options to explain how it should be used: `CARBON_OPTIONS_DATEFORMAT` and `CARBON_OPTIONS_TIMESTAMPFORMAT` ---
[GitHub] carbondata pull request #1524: [CARBONDATA-1762] Remove existing column leve...
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1524#discussion_r151831633
--- Diff:
integration/spark-common/src/main/scala/org/apache/carbondata/spark/load/ValidateUtil.scala
---
@@ -17,35 +17,32 @@
package org.apache.carbondata.spark.load
-import scala.collection.JavaConverters._
+import java.text.SimpleDateFormat
-import org.apache.carbondata.core.constants.CarbonCommonConstants
import org.apache.carbondata.core.metadata.schema.table.CarbonTable
-import org.apache.carbondata.processing.loading.model.CarbonLoadModel
import org.apache.carbondata.processing.loading.sort.SortScopeOptions
import
org.apache.carbondata.spark.exception.MalformedCarbonCommandException
object ValidateUtil {
- def validateDateFormat(dateFormat: String, table: CarbonTable,
tableName: String): Unit = {
-val dimensions = table.getDimensionByTableName(tableName).asScala
+
+ /**
+ * validates both timestamp and date for illegal values
--- End diff --
validate timestamp format and date format. please change
---
[GitHub] carbondata pull request #1524: [CARBONDATA-1762] Remove existing column leve...
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1524#discussion_r151831621
--- Diff:
integration/spark-common-test/src/test/scala/org/apache/carbondata/spark/testsuite/dataload/TestLoadDataWithDiffTimestampFormat.scala
---
@@ -75,55 +76,19 @@ class TestLoadDataWithDiffTimestampFormat extends
QueryTest with BeforeAndAfterA
assert(false)
} catch {
case ex: MalformedCarbonCommandException =>
-assertResult(ex.getMessage)("Error: Option DateFormat is not
provided for Column date.")
+assertResult(ex.getMessage)("Error: Wrong option: date is provided
for option DateFormat")
case _: Throwable=> assert(false)
}
try {
sql(s"""
LOAD DATA LOCAL INPATH
'$resourcesPath/timeStampFormatData1.csv' into table t3
- OPTIONS('dateformat' = 'fasfdas:/MM/dd')
--- End diff --
why delete these testcase? `dateformat` is still a valid loading option,
right?
---
[GitHub] carbondata pull request #1524: [CARBONDATA-1762] Remove existing column leve...
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1524#discussion_r151831369
--- Diff:
integration/spark2/src/main/scala/org/apache/carbondata/spark/rdd/CarbonDataRDDFactory.scala
---
@@ -799,18 +799,19 @@ object CarbonDataRDDFactory {
throw new DataLoadingException("Partition column not found.")
}
-val dateFormatMap =
CarbonDataProcessorUtil.getDateFormatMap(carbonLoadModel.getDateFormat)
-val specificFormat =
Option(dateFormatMap.get(partitionColumn.toLowerCase))
-val timeStampFormat = if (specificFormat.isDefined) {
- new SimpleDateFormat(specificFormat.get)
-} else {
- val timestampFormatString =
CarbonProperties.getInstance().getProperty(CarbonCommonConstants
-.CARBON_TIMESTAMP_FORMAT,
CarbonCommonConstants.CARBON_TIMESTAMP_DEFAULT_FORMAT)
- new SimpleDateFormat(timestampFormatString)
-}
+val specificTimestampFormat = carbonLoadModel.getTimestampformat
+val specificDateFormat = carbonLoadModel.getDateFormat
+val timeStampFormat =
+ if (specificTimestampFormat != null &&
!specificTimestampFormat.trim.isEmpty) {
+new SimpleDateFormat(specificTimestampFormat)
+ } else {
+val timestampFormatString =
CarbonProperties.getInstance().getProperty(CarbonCommonConstants
--- End diff --
move `CarbonCommonConstants` to next line, give one parameter in one line
---
[GitHub] carbondata pull request #1471: [CARBONDATA-1544][Datamap] Datamap FineGrain ...
Github user ravipesala commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1471#discussion_r151831374
--- Diff:
core/src/main/java/org/apache/carbondata/core/datamap/TableDataMap.java ---
@@ -75,7 +79,12 @@ public TableDataMap(AbsoluteTableIdentifier identifier,
String dataMapName,
SegmentProperties segmentProperties;
for (String segmentId : segmentIds) {
List pruneBlocklets = new ArrayList<>();
- List dataMaps = dataMapFactory.getDataMaps(segmentId);
+ List dataMaps;
+ if (blockletDetailsFetcher instanceof DataMapFactory && filterExp ==
null) {
+dataMaps =
((DataMapFactory)blockletDetailsFetcher).getDataMaps(segmentId);
--- End diff --
I have added here but placing generic assignment all places is not possible
because we are deriving two types of datamaps through generic .
---
[GitHub] carbondata pull request #1524: [CARBONDATA-1762] Remove existing column leve...
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1524#discussion_r151831356
--- Diff:
integration/spark-common/src/main/scala/org/apache/carbondata/spark/load/ValidateUtil.scala
---
@@ -17,35 +17,32 @@
package org.apache.carbondata.spark.load
-import scala.collection.JavaConverters._
+import java.text.SimpleDateFormat
-import org.apache.carbondata.core.constants.CarbonCommonConstants
import org.apache.carbondata.core.metadata.schema.table.CarbonTable
-import org.apache.carbondata.processing.loading.model.CarbonLoadModel
import org.apache.carbondata.processing.loading.sort.SortScopeOptions
import
org.apache.carbondata.spark.exception.MalformedCarbonCommandException
object ValidateUtil {
- def validateDateFormat(dateFormat: String, table: CarbonTable,
tableName: String): Unit = {
-val dimensions = table.getDimensionByTableName(tableName).asScala
+
+ /**
+ * validates both timestamp and date for illegal values
+ *
+ * @param dateTimeLoadFormat
+ * @param dateTimeLoadOption
+ */
+ def validateDateTimeFormat(dateTimeLoadFormat: String,
dateTimeLoadOption: String): Unit = {
// allowing empty value to be configured for dateformat option.
-if (dateFormat != null && dateFormat.trim != "") {
-val dateFormats: Array[String] =
dateFormat.split(CarbonCommonConstants.COMMA)
-for (singleDateFormat <- dateFormats) {
- val dateFormatSplits: Array[String] =
singleDateFormat.split(":", 2)
- val columnName = dateFormatSplits(0).trim.toLowerCase
- if (!dimensions.exists(_.getColName.equals(columnName))) {
-throw new MalformedCarbonCommandException("Error: Wrong Column
Name " +
- dateFormatSplits(0) +
- " is provided in Option DateFormat.")
- }
- if (dateFormatSplits.length < 2 ||
dateFormatSplits(1).trim.isEmpty) {
-throw new MalformedCarbonCommandException("Error: Option
DateFormat is not provided " +
- "for " + "Column " + dateFormatSplits(0) +
- ".")
- }
-}
+if (dateTimeLoadFormat != null && dateTimeLoadFormat.trim != "") {
+ try {
+new SimpleDateFormat(dateTimeLoadFormat)
}
+ catch {
+case _: IllegalArgumentException =>
+ throw new MalformedCarbonCommandException("Error: Wrong option:
" + dateTimeLoadFormat +
--- End diff --
use $ for variable
---
[GitHub] carbondata pull request #1524: [CARBONDATA-1762] Remove existing column leve...
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1524#discussion_r151831351
--- Diff:
integration/spark-common/src/main/scala/org/apache/carbondata/spark/load/ValidateUtil.scala
---
@@ -17,35 +17,32 @@
package org.apache.carbondata.spark.load
-import scala.collection.JavaConverters._
+import java.text.SimpleDateFormat
-import org.apache.carbondata.core.constants.CarbonCommonConstants
import org.apache.carbondata.core.metadata.schema.table.CarbonTable
-import org.apache.carbondata.processing.loading.model.CarbonLoadModel
import org.apache.carbondata.processing.loading.sort.SortScopeOptions
import
org.apache.carbondata.spark.exception.MalformedCarbonCommandException
object ValidateUtil {
- def validateDateFormat(dateFormat: String, table: CarbonTable,
tableName: String): Unit = {
-val dimensions = table.getDimensionByTableName(tableName).asScala
+
+ /**
+ * validates both timestamp and date for illegal values
+ *
+ * @param dateTimeLoadFormat
+ * @param dateTimeLoadOption
+ */
+ def validateDateTimeFormat(dateTimeLoadFormat: String,
dateTimeLoadOption: String): Unit = {
// allowing empty value to be configured for dateformat option.
-if (dateFormat != null && dateFormat.trim != "") {
-val dateFormats: Array[String] =
dateFormat.split(CarbonCommonConstants.COMMA)
-for (singleDateFormat <- dateFormats) {
- val dateFormatSplits: Array[String] =
singleDateFormat.split(":", 2)
- val columnName = dateFormatSplits(0).trim.toLowerCase
- if (!dimensions.exists(_.getColName.equals(columnName))) {
-throw new MalformedCarbonCommandException("Error: Wrong Column
Name " +
- dateFormatSplits(0) +
- " is provided in Option DateFormat.")
- }
- if (dateFormatSplits.length < 2 ||
dateFormatSplits(1).trim.isEmpty) {
-throw new MalformedCarbonCommandException("Error: Option
DateFormat is not provided " +
- "for " + "Column " + dateFormatSplits(0) +
- ".")
- }
-}
+if (dateTimeLoadFormat != null && dateTimeLoadFormat.trim != "") {
+ try {
+new SimpleDateFormat(dateTimeLoadFormat)
}
+ catch {
--- End diff --
move to previous line
---
[GitHub] carbondata pull request #1525: [CARBONDATA-1751] Make the type of exception ...
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1525#discussion_r151831251
--- Diff:
integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/management/AlterTableCompactionCommand.scala
---
@@ -87,8 +88,10 @@ case class AlterTableCompactionCommand(
} catch {
case e: Exception =>
if (null != e.getMessage) {
+ // TODO: We should modify it when we support compaction later.
sys.error(s"Compaction failed. Please check logs for more info.
${ e.getMessage }")
--- End diff --
Why not change this also?
---
[GitHub] carbondata issue #1424: [CARBONDATA-1602] Remove unused declaration in spark...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1424 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/1262/ ---
[GitHub] carbondata pull request #1471: [CARBONDATA-1544][Datamap] Datamap FineGrain ...
Github user ravipesala commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1471#discussion_r151830437
--- Diff:
hadoop/src/main/java/org/apache/carbondata/hadoop/api/CarbonTableInputFormat.java
---
@@ -687,16 +689,17 @@ protected Expression
getFilterPredicates(Configuration configuration) {
// get tokens for all the required FileSystem for table path
TokenCache.obtainTokensForNamenodes(job.getCredentials(),
new Path[] { new Path(absoluteTableIdentifier.getTablePath()) },
job.getConfiguration());
-
-TableDataMap blockletMap = DataMapStoreManager.getInstance()
-.getDataMap(absoluteTableIdentifier, BlockletDataMap.NAME,
-BlockletDataMapFactory.class.getName());
+boolean distributedCG =
Boolean.parseBoolean(CarbonProperties.getInstance()
+.getProperty(CarbonCommonConstants.USE_DISTRIBUTED_DATAMAP,
+CarbonCommonConstants.USE_DISTRIBUTED_DATAMAP_DEFAULT));
+TableDataMap blockletMap =
+
DataMapStoreManager.getInstance().chooseDataMap(absoluteTableIdentifier);
DataMapJob dataMapJob = getDataMapJob(job.getConfiguration());
List prunedBlocklets;
-if (dataMapJob != null) {
+if (distributedCG || blockletMap.getDataMapFactory().getDataMapType()
== DataMapType.FG) {
--- End diff --
In case of FG datamap the rowid information would be written to temp files
and writepath is given to FGBlocklet so it does not contain any row information
when it returns to the driver.
While executing filter query executor reads rowids from disk and pass to
the scanner.
---
[GitHub] carbondata pull request #1471: [CARBONDATA-1544][Datamap] Datamap FineGrain ...
Github user ravipesala commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1471#discussion_r151830196
--- Diff:
processing/src/main/java/org/apache/carbondata/processing/store/writer/AbstractFactDataWriter.java
---
@@ -463,20 +426,24 @@ protected void writeIndexFile() throws IOException,
CarbonDataWriterException {
* @throws CarbonDataWriterException
*/
protected void closeExecutorService() throws CarbonDataWriterException {
-executorService.shutdown();
try {
- executorService.awaitTermination(2, TimeUnit.HOURS);
-} catch (InterruptedException e) {
- throw new CarbonDataWriterException(e.getMessage());
-}
-for (int i = 0; i < executorServiceSubmitList.size(); i++) {
+ listener.finish();
+ executorService.shutdown();
try {
-executorServiceSubmitList.get(i).get();
+executorService.awaitTermination(2, TimeUnit.HOURS);
} catch (InterruptedException e) {
throw new CarbonDataWriterException(e.getMessage());
- } catch (ExecutionException e) {
-throw new CarbonDataWriterException(e.getMessage());
}
+ for (int i = 0; i < executorServiceSubmitList.size(); i++) {
+executorServiceSubmitList.get(i).get();
+ }
+} catch (InterruptedException e) {
+ throw new CarbonDataWriterException(e.getMessage());
+} catch (ExecutionException e) {
+ throw new CarbonDataWriterException(e.getMessage());
+} catch (IOException e) {
+ LOGGER.error(e, "Error while writing datamap");
+ throw new CarbonDataWriterException(e.getMessage());
--- End diff --
ok
---
[GitHub] carbondata pull request #1471: [CARBONDATA-1544][Datamap] Datamap FineGrain ...
Github user ravipesala commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1471#discussion_r151830168
--- Diff:
processing/src/main/java/org/apache/carbondata/processing/store/writer/AbstractFactDataWriter.java
---
@@ -463,20 +426,24 @@ protected void writeIndexFile() throws IOException,
CarbonDataWriterException {
* @throws CarbonDataWriterException
*/
protected void closeExecutorService() throws CarbonDataWriterException {
-executorService.shutdown();
try {
- executorService.awaitTermination(2, TimeUnit.HOURS);
-} catch (InterruptedException e) {
- throw new CarbonDataWriterException(e.getMessage());
-}
-for (int i = 0; i < executorServiceSubmitList.size(); i++) {
+ listener.finish();
+ executorService.shutdown();
try {
-executorServiceSubmitList.get(i).get();
+executorService.awaitTermination(2, TimeUnit.HOURS);
} catch (InterruptedException e) {
throw new CarbonDataWriterException(e.getMessage());
- } catch (ExecutionException e) {
-throw new CarbonDataWriterException(e.getMessage());
}
+ for (int i = 0; i < executorServiceSubmitList.size(); i++) {
+executorServiceSubmitList.get(i).get();
+ }
+} catch (InterruptedException e) {
+ throw new CarbonDataWriterException(e.getMessage());
+} catch (ExecutionException e) {
--- End diff --
OK
---
[GitHub] carbondata pull request #1471: [CARBONDATA-1544][Datamap] Datamap FineGrain ...
Github user ravipesala commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1471#discussion_r151830137
--- Diff:
integration/spark-common-test/src/test/scala/org/apache/carbondata/spark/testsuite/datamap/FGDataMapTestCase.scala
---
@@ -0,0 +1,434 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.carbondata.spark.testsuite.datamap
+
+import java.io.{ByteArrayInputStream, DataOutputStream, ObjectInputStream,
ObjectOutputStream}
+
+import scala.collection.JavaConverters._
+import scala.collection.mutable.ArrayBuffer
+
+import com.sun.xml.internal.messaging.saaj.util.ByteOutputStream
+import org.apache.spark.sql.test.util.QueryTest
+import org.scalatest.BeforeAndAfterAll
+
+import
org.apache.carbondata.core.datamap.dev.fgdatamap.{AbstractFineGrainDataMap,
AbstractFineGrainDataMapFactory}
+import org.apache.carbondata.core.datamap.dev.{AbstractDataMapWriter,
DataMapModel}
+import org.apache.carbondata.core.datamap.{DataMapDistributable,
DataMapMeta, DataMapStoreManager}
+import org.apache.carbondata.core.datastore.FileHolder
+import org.apache.carbondata.core.datastore.block.SegmentProperties
+import org.apache.carbondata.core.datastore.compression.SnappyCompressor
+import org.apache.carbondata.core.datastore.filesystem.{CarbonFile,
CarbonFileFilter}
+import org.apache.carbondata.core.datastore.impl.FileFactory
+import org.apache.carbondata.core.datastore.page.ColumnPage
+import org.apache.carbondata.core.indexstore.FineGrainBlocklet
+import
org.apache.carbondata.core.indexstore.blockletindex.BlockletDataMapDistributable
+import org.apache.carbondata.core.metadata.{AbsoluteTableIdentifier,
CarbonMetadata}
+import org.apache.carbondata.core.scan.expression.Expression
+import
org.apache.carbondata.core.scan.expression.conditional.EqualToExpression
+import org.apache.carbondata.core.scan.filter.intf.ExpressionType
+import org.apache.carbondata.core.scan.filter.resolver.FilterResolverIntf
+import org.apache.carbondata.core.util.ByteUtil
+import org.apache.carbondata.core.util.path.CarbonTablePath
+import org.apache.carbondata.events.Event
+import
org.apache.carbondata.spark.testsuite.datacompaction.CompactionSupportGlobalSortBigFileTest
+
+class FGDataMapFactory extends AbstractFineGrainDataMapFactory {
+ var identifier: AbsoluteTableIdentifier = _
+ var dataMapName: String = _
+
+ /**
+ * Initialization of Datamap factory with the identifier and datamap name
+ */
+ override def init(identifier: AbsoluteTableIdentifier,
--- End diff --
ok
---
[GitHub] carbondata issue #1524: [CARBONDATA-1762] Remove existing column level datef...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1524 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/1261/ ---
[GitHub] carbondata issue #1520: [CARBONDATA-1734] Ignore empty line while reading CS...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1520 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/1260/ ---
[GitHub] carbondata issue #1515: [CARBONDATA-1751] Modify sys.err to AnalysisExceptio...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1515 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/1259/ ---
[GitHub] carbondata issue #1525: [WIP][CARBONDATA-1751] Make the type of exception an...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1525 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/1258/ ---
[GitHub] carbondata issue #1520: [CARBONDATA-1734] Ignore empty line while reading CS...
Github user dhatchayani commented on the issue: https://github.com/apache/carbondata/pull/1520 Retest this please ---
[GitHub] carbondata issue #1528: [WIP][CARBONDATA-1752] There are some scalastyle err...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1528 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/1257/ ---
[GitHub] carbondata pull request #1471: [CARBONDATA-1544][Datamap] Datamap FineGrain ...
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1471#discussion_r151828034
--- Diff:
core/src/main/java/org/apache/carbondata/core/datamap/dev/BlockletSerializer.java
---
@@ -0,0 +1,57 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.carbondata.core.datamap.dev;
+
+import java.io.DataInputStream;
+import java.io.DataOutputStream;
+import java.io.IOException;
+
+import org.apache.carbondata.core.datastore.impl.FileFactory;
+import org.apache.carbondata.core.indexstore.FineGrainBlocklet;
+
+public class BlockletSerializer {
+
+ /**
+ * Serialize and write blocklet to the file.
+ * @param grainBlocklet
+ * @param writePath
+ * @throws IOException
+ */
+ public void serializeBlocklet(FineGrainBlocklet grainBlocklet, String
writePath)
--- End diff --
Can we move this to FineGrainBlocklet itself? So that we do not need to
keep writePath in many places, like in TableBlockInfo
---
[GitHub] carbondata issue #1525: [WIP][CARBONDATA-1751] Make the type of exception an...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1525 Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/1256/ ---
[GitHub] carbondata pull request #1471: [CARBONDATA-1544][Datamap] Datamap FineGrain ...
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1471#discussion_r151827748
--- Diff:
core/src/main/java/org/apache/carbondata/core/datamap/TableDataMap.java ---
@@ -75,7 +79,12 @@ public TableDataMap(AbsoluteTableIdentifier identifier,
String dataMapName,
SegmentProperties segmentProperties;
for (String segmentId : segmentIds) {
List pruneBlocklets = new ArrayList<>();
- List dataMaps = dataMapFactory.getDataMaps(segmentId);
+ List dataMaps;
+ if (blockletDetailsFetcher instanceof DataMapFactory && filterExp ==
null) {
+dataMaps =
((DataMapFactory)blockletDetailsFetcher).getDataMaps(segmentId);
--- End diff --
Please add correct generic assignment, it is hard to check whether the type
is correct or not
---
[GitHub] carbondata issue #1501: [CARBONDATA-1713] Fixed Aggregate query on main tabl...
Github user jackylk commented on the issue: https://github.com/apache/carbondata/pull/1501 @kumarvishal09 @ravipesala ---
[GitHub] carbondata pull request #1528: [WIP][CARBONDATA-1752] There are some scalast...
GitHub user xubo245 opened a pull request: https://github.com/apache/carbondata/pull/1528 [WIP][CARBONDATA-1752] There are some scalastyle error should be optimized in CarbonData There are some scalastyle error should be optimized in CarbonData, including removing useless import, optimizing method definition and so on We should add it into scala style check. Be sure to do all of the following checklist to help us incorporate your contribution quickly and easily: - [ ] Any interfaces changed? No - [ ] Any backward compatibility impacted? No - [ ] Document update required? No - [ ] Testing done No - [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA. NO You can merge this pull request into a Git repository by running: $ git pull https://github.com/xubo245/carbondata fixStyle2 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/carbondata/pull/1528.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1528 commit 3f8a6d11a0e53ccd0e76ea16ef4167e6b04b9678 Author: xubo245 <601450...@qq.com> Date: 2017-11-18T03:46:59Z [CARBONDATA-1752] There are some scalastyle error should be optimized in CarbonData ---
[GitHub] carbondata pull request #1515: [CARBONDATA-1751] Modify sys.err to AnalysisE...
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1515#discussion_r151827674
--- Diff:
integration/spark-common/src/main/scala/org/apache/carbondata/spark/util/CommonUtil.scala
---
@@ -662,16 +663,18 @@ object CommonUtil {
val maxColumnsInt = getMaxColumnValue(maxColumns)
if (maxColumnsInt != null) {
if (columnCountInSchema >= maxColumnsInt) {
-sys.error(s"csv headers should be less than the max columns:
$maxColumnsInt")
+CarbonException.analysisException(
+ s"csv headers should be less than the max columns:
$maxColumnsInt")
} else if (maxColumnsInt >
CSVInputFormat.THRESHOLD_MAX_NUMBER_OF_COLUMNS_FOR_PARSING) {
-sys.error(s"max columns cannot be greater than the threshold
value: ${
- CSVInputFormat.THRESHOLD_MAX_NUMBER_OF_COLUMNS_FOR_PARSING
-}")
+CarbonException.analysisException(
+ s"max columns cannot be greater than the threshold value: ${
--- End diff --
move
```
${
+CSVInputFormat.THRESHOLD_MAX_NUMBER_OF_COLUMNS_FOR_PARSING
+ }"
```
into one line, like line 667. the same for line 677
---
[GitHub] carbondata pull request #1471: [CARBONDATA-1544][Datamap] Datamap FineGrain ...
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1471#discussion_r151827453
--- Diff:
hadoop/src/main/java/org/apache/carbondata/hadoop/api/CarbonTableInputFormat.java
---
@@ -687,16 +689,17 @@ protected Expression
getFilterPredicates(Configuration configuration) {
// get tokens for all the required FileSystem for table path
TokenCache.obtainTokensForNamenodes(job.getCredentials(),
new Path[] { new Path(absoluteTableIdentifier.getTablePath()) },
job.getConfiguration());
-
-TableDataMap blockletMap = DataMapStoreManager.getInstance()
-.getDataMap(absoluteTableIdentifier, BlockletDataMap.NAME,
-BlockletDataMapFactory.class.getName());
+boolean distributedCG =
Boolean.parseBoolean(CarbonProperties.getInstance()
+.getProperty(CarbonCommonConstants.USE_DISTRIBUTED_DATAMAP,
+CarbonCommonConstants.USE_DISTRIBUTED_DATAMAP_DEFAULT));
+TableDataMap blockletMap =
+
DataMapStoreManager.getInstance().chooseDataMap(absoluteTableIdentifier);
DataMapJob dataMapJob = getDataMapJob(job.getConfiguration());
List prunedBlocklets;
-if (dataMapJob != null) {
+if (distributedCG || blockletMap.getDataMapFactory().getDataMapType()
== DataMapType.FG) {
--- End diff --
It seems distributedCG and FG behave the same, right?
Earlier I though FG datamap will be called in ScanRDD.compute, but it seems
not?
If we collect all pruned blocklet in driver, will it be too many for
driver?
---
[GitHub] carbondata pull request #1471: [CARBONDATA-1544][Datamap] Datamap FineGrain ...
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1471#discussion_r151827371
--- Diff:
processing/src/main/java/org/apache/carbondata/processing/store/writer/AbstractFactDataWriter.java
---
@@ -463,20 +426,24 @@ protected void writeIndexFile() throws IOException,
CarbonDataWriterException {
* @throws CarbonDataWriterException
*/
protected void closeExecutorService() throws CarbonDataWriterException {
-executorService.shutdown();
try {
- executorService.awaitTermination(2, TimeUnit.HOURS);
-} catch (InterruptedException e) {
- throw new CarbonDataWriterException(e.getMessage());
-}
-for (int i = 0; i < executorServiceSubmitList.size(); i++) {
+ listener.finish();
+ executorService.shutdown();
try {
-executorServiceSubmitList.get(i).get();
+executorService.awaitTermination(2, TimeUnit.HOURS);
} catch (InterruptedException e) {
throw new CarbonDataWriterException(e.getMessage());
- } catch (ExecutionException e) {
-throw new CarbonDataWriterException(e.getMessage());
}
+ for (int i = 0; i < executorServiceSubmitList.size(); i++) {
+executorServiceSubmitList.get(i).get();
+ }
+} catch (InterruptedException e) {
+ throw new CarbonDataWriterException(e.getMessage());
+} catch (ExecutionException e) {
+ throw new CarbonDataWriterException(e.getMessage());
+} catch (IOException e) {
+ LOGGER.error(e, "Error while writing datamap");
+ throw new CarbonDataWriterException(e.getMessage());
--- End diff --
merge with line 440
---
[GitHub] carbondata pull request #1471: [CARBONDATA-1544][Datamap] Datamap FineGrain ...
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1471#discussion_r151827362
--- Diff:
processing/src/main/java/org/apache/carbondata/processing/store/writer/AbstractFactDataWriter.java
---
@@ -463,20 +426,24 @@ protected void writeIndexFile() throws IOException,
CarbonDataWriterException {
* @throws CarbonDataWriterException
*/
protected void closeExecutorService() throws CarbonDataWriterException {
-executorService.shutdown();
try {
- executorService.awaitTermination(2, TimeUnit.HOURS);
-} catch (InterruptedException e) {
- throw new CarbonDataWriterException(e.getMessage());
-}
-for (int i = 0; i < executorServiceSubmitList.size(); i++) {
+ listener.finish();
+ executorService.shutdown();
try {
-executorServiceSubmitList.get(i).get();
+executorService.awaitTermination(2, TimeUnit.HOURS);
} catch (InterruptedException e) {
throw new CarbonDataWriterException(e.getMessage());
- } catch (ExecutionException e) {
-throw new CarbonDataWriterException(e.getMessage());
}
+ for (int i = 0; i < executorServiceSubmitList.size(); i++) {
+executorServiceSubmitList.get(i).get();
+ }
+} catch (InterruptedException e) {
+ throw new CarbonDataWriterException(e.getMessage());
+} catch (ExecutionException e) {
--- End diff --
merge with line 440
---
[jira] [Updated] (CARBONDATA-1756) Improve Boolean data compress rate by changing RLE to SNAPPY algorithm
[ https://issues.apache.org/jira/browse/CARBONDATA-1756?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] xubo245 updated CARBONDATA-1756: Summary: Improve Boolean data compress rate by changing RLE to SNAPPY algorithm (was: Improve Boolean data compress rate by changing RLE to SNNAPY algorithm) > Improve Boolean data compress rate by changing RLE to SNAPPY algorithm > -- > > Key: CARBONDATA-1756 > URL: https://issues.apache.org/jira/browse/CARBONDATA-1756 > Project: CarbonData > Issue Type: Improvement > Components: core >Affects Versions: 1.2.0 >Reporter: xubo245 >Assignee: xubo245 > Fix For: 1.3.0 > > Time Spent: 50m > Remaining Estimate: 0h > > Improve Boolean data compress rate by changing RLE to SNNAPY algorithm > Because Boolean data compress rate that uses RLE algorithm is lower than > SNNAPY algorithm in most scenario. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (CARBONDATA-1756) Improve Boolean data compress rate by changing RLE to SNAPPY algorithm
[ https://issues.apache.org/jira/browse/CARBONDATA-1756?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] xubo245 updated CARBONDATA-1756: Description: Improve Boolean data compress rate by changing RLE to SNAPPY algorithm Because Boolean data compress rate that uses RLE algorithm is lower than SNAPPY algorithm in most scenario. was: Improve Boolean data compress rate by changing RLE to SNNAPY algorithm Because Boolean data compress rate that uses RLE algorithm is lower than SNNAPY algorithm in most scenario. > Improve Boolean data compress rate by changing RLE to SNAPPY algorithm > -- > > Key: CARBONDATA-1756 > URL: https://issues.apache.org/jira/browse/CARBONDATA-1756 > Project: CarbonData > Issue Type: Improvement > Components: core >Affects Versions: 1.2.0 >Reporter: xubo245 >Assignee: xubo245 > Fix For: 1.3.0 > > Time Spent: 50m > Remaining Estimate: 0h > > Improve Boolean data compress rate by changing RLE to SNAPPY algorithm > Because Boolean data compress rate that uses RLE algorithm is lower than > SNAPPY algorithm in most scenario. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[GitHub] carbondata pull request #1471: [CARBONDATA-1544][Datamap] Datamap FineGrain ...
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1471#discussion_r151827225
--- Diff:
processing/src/main/java/org/apache/carbondata/processing/store/writer/AbstractFactDataWriter.java
---
@@ -463,20 +426,24 @@ protected void writeIndexFile() throws IOException,
CarbonDataWriterException {
* @throws CarbonDataWriterException
*/
protected void closeExecutorService() throws CarbonDataWriterException {
-executorService.shutdown();
try {
- executorService.awaitTermination(2, TimeUnit.HOURS);
-} catch (InterruptedException e) {
- throw new CarbonDataWriterException(e.getMessage());
-}
-for (int i = 0; i < executorServiceSubmitList.size(); i++) {
+ listener.finish();
+ executorService.shutdown();
try {
--- End diff --
This try is not required, catch in line 440 is enough
---
[GitHub] carbondata pull request #1471: [CARBONDATA-1544][Datamap] Datamap FineGrain ...
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1471#discussion_r151827164
--- Diff:
integration/spark-common-test/src/test/scala/org/apache/carbondata/spark/testsuite/datamap/FGDataMapTestCase.scala
---
@@ -0,0 +1,434 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.carbondata.spark.testsuite.datamap
+
+import java.io.{ByteArrayInputStream, DataOutputStream, ObjectInputStream,
ObjectOutputStream}
+
+import scala.collection.JavaConverters._
+import scala.collection.mutable.ArrayBuffer
+
+import com.sun.xml.internal.messaging.saaj.util.ByteOutputStream
+import org.apache.spark.sql.test.util.QueryTest
+import org.scalatest.BeforeAndAfterAll
+
+import
org.apache.carbondata.core.datamap.dev.fgdatamap.{AbstractFineGrainDataMap,
AbstractFineGrainDataMapFactory}
+import org.apache.carbondata.core.datamap.dev.{AbstractDataMapWriter,
DataMapModel}
+import org.apache.carbondata.core.datamap.{DataMapDistributable,
DataMapMeta, DataMapStoreManager}
+import org.apache.carbondata.core.datastore.FileHolder
+import org.apache.carbondata.core.datastore.block.SegmentProperties
+import org.apache.carbondata.core.datastore.compression.SnappyCompressor
+import org.apache.carbondata.core.datastore.filesystem.{CarbonFile,
CarbonFileFilter}
+import org.apache.carbondata.core.datastore.impl.FileFactory
+import org.apache.carbondata.core.datastore.page.ColumnPage
+import org.apache.carbondata.core.indexstore.FineGrainBlocklet
+import
org.apache.carbondata.core.indexstore.blockletindex.BlockletDataMapDistributable
+import org.apache.carbondata.core.metadata.{AbsoluteTableIdentifier,
CarbonMetadata}
+import org.apache.carbondata.core.scan.expression.Expression
+import
org.apache.carbondata.core.scan.expression.conditional.EqualToExpression
+import org.apache.carbondata.core.scan.filter.intf.ExpressionType
+import org.apache.carbondata.core.scan.filter.resolver.FilterResolverIntf
+import org.apache.carbondata.core.util.ByteUtil
+import org.apache.carbondata.core.util.path.CarbonTablePath
+import org.apache.carbondata.events.Event
+import
org.apache.carbondata.spark.testsuite.datacompaction.CompactionSupportGlobalSortBigFileTest
+
+class FGDataMapFactory extends AbstractFineGrainDataMapFactory {
+ var identifier: AbsoluteTableIdentifier = _
+ var dataMapName: String = _
+
+ /**
+ * Initialization of Datamap factory with the identifier and datamap name
+ */
+ override def init(identifier: AbsoluteTableIdentifier,
--- End diff --
Please add return type in all functions in this file and
CGDataMapTestCase.scala also
---
[GitHub] carbondata issue #1515: [CARBONDATA-1751] Modify sys.err to AnalysisExceptio...
Github user xubo245 commented on the issue: https://github.com/apache/carbondata/pull/1515 Please review it @jackylk ---
[GitHub] carbondata issue #1525: [WIP][CARBONDATA-1751] Make the type of exception an...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1525 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/1255/ ---
[GitHub] carbondata issue #1515: [CARBONDATA-1751] Modify sys.err to AnalysisExceptio...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1515 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/1254/ ---
[jira] [Resolved] (CARBONDATA-1753) Missing 'org.scalatest.tools.Runner' when run test with streaming module
[ https://issues.apache.org/jira/browse/CARBONDATA-1753?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liang Chen resolved CARBONDATA-1753. Resolution: Fixed > Missing 'org.scalatest.tools.Runner' when run test with streaming module > > > Key: CARBONDATA-1753 > URL: https://issues.apache.org/jira/browse/CARBONDATA-1753 > Project: CarbonData > Issue Type: Bug > Components: build >Affects Versions: 1.3.0 >Reporter: Zhichao Zhang >Assignee: Zhichao Zhang >Priority: Minor > Fix For: 1.3.0 > > Time Spent: 1h > Remaining Estimate: 0h > > Missing 'org.scalatest.tools.Runner' when run test with streaming module. > Need to add scalatest to pom.xml of streaming module. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[GitHub] carbondata pull request #1519: [CARBONDATA-1753][Streaming]Fix missing 'org....
Github user asfgit closed the pull request at: https://github.com/apache/carbondata/pull/1519 ---
[GitHub] carbondata issue #1520: [CARBONDATA-1734] Ignore empty line while reading CS...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1520 Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/1253/ ---
[GitHub] carbondata issue #1524: [CARBONDATA-1762] Remove existing column level datef...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1524 Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/1252/ ---
[jira] [Commented] (CARBONDATA-1651) Unsupported Spark2 BooleanType
[ https://issues.apache.org/jira/browse/CARBONDATA-1651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16257867#comment-16257867 ] xubo245 commented on CARBONDATA-1651: - Have you tested other dataframe operation for Boolean data type? > Unsupported Spark2 BooleanType > -- > > Key: CARBONDATA-1651 > URL: https://issues.apache.org/jira/browse/CARBONDATA-1651 > Project: CarbonData > Issue Type: Bug > Components: spark-integration >Affects Versions: 1.3.0 >Reporter: Roman Timrov >Assignee: anubhav tarar > Fix For: 1.3.0 > > Time Spent: 2h 50m > Remaining Estimate: 0h > > Unable to save Dataset if it contains field with BooleanType > class CarbonDataFrameWriter > method convertToCarbonType doesn't support it -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[GitHub] carbondata issue #1523: [CARBONDATA-1756] Improve Boolean data compress rate...
Github user xubo245 commented on the issue: https://github.com/apache/carbondata/pull/1523 @jackylk Please review it ---
[GitHub] carbondata issue #1523: [CARBONDATA-1756] Improve Boolean data compress rate...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1523 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/1251/ ---
[GitHub] carbondata issue #1520: [CARBONDATA-1734] Ignore empty line while reading CS...
Github user dhatchayani commented on the issue: https://github.com/apache/carbondata/pull/1520 Retest this please ---
[GitHub] carbondata issue #1524: [CARBONDATA-1762] Remove existing column level datef...
Github user dhatchayani commented on the issue: https://github.com/apache/carbondata/pull/1524 Retest this please ---
[GitHub] carbondata issue #1471: [CARBONDATA-1544][Datamap] Datamap FineGrain impleme...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1471 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/1250/ ---
[jira] [Resolved] (CARBONDATA-1614) SHOW SEGMENT should include the streaming property
[ https://issues.apache.org/jira/browse/CARBONDATA-1614?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ravindra Pesala resolved CARBONDATA-1614. - Resolution: Fixed Fix Version/s: 1.3.0 > SHOW SEGMENT should include the streaming property > -- > > Key: CARBONDATA-1614 > URL: https://issues.apache.org/jira/browse/CARBONDATA-1614 > Project: CarbonData > Issue Type: Sub-task >Reporter: Jacky Li >Assignee: Jacky Li > Fix For: 1.3.0 > > Time Spent: 1h 50m > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[GitHub] carbondata pull request #1498: [CARBONDATA-1614][Streaming] Show file format...
Github user asfgit closed the pull request at: https://github.com/apache/carbondata/pull/1498 ---
[GitHub] carbondata issue #1498: [CARBONDATA-1614][Streaming] Show file format for se...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/1498 LGTM ---
[GitHub] carbondata issue #1498: [CARBONDATA-1614][Streaming] Show file format for se...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1498 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/1249/ ---
[GitHub] carbondata pull request #1527: [CARBONDATA-1706] Making index merge DDL inse...
Github user dhatchayani closed the pull request at: https://github.com/apache/carbondata/pull/1527 ---
[GitHub] carbondata issue #1527: [CARBONDATA-1706] Making index merge DDL insensitive...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1527 Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/1248/ ---
[GitHub] carbondata issue #1498: [CARBONDATA-1614][Streaming] Show file format for se...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/1498 retest this please ---
[GitHub] carbondata issue #1527: [CARBONDATA-1706] Making index merge DDL insensitive...
Github user dhatchayani commented on the issue: https://github.com/apache/carbondata/pull/1527 retest this please ---
[GitHub] carbondata issue #1527: [CARBONDATA-1706] Making index merge DDL insensitive...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1527 Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/1247/ ---
[GitHub] carbondata issue #1498: [CARBONDATA-1614][Streaming] Show file format for se...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1498 Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/1246/ ---
[GitHub] carbondata issue #1527: [CARBONDATA-1706] Making index merge DDL insensitive...
Github user dhatchayani commented on the issue: https://github.com/apache/carbondata/pull/1527 retest this please ---
[GitHub] carbondata issue #1527: [CARBONDATA-1706] Making index merge DDL insensitive...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1527 Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/1245/ ---
[GitHub] carbondata issue #1514: [CARBONDATA-1746] Count star optimization
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1514 Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/1244/ ---
[jira] [Resolved] (CARBONDATA-1651) Unsupported Spark2 BooleanType
[ https://issues.apache.org/jira/browse/CARBONDATA-1651?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jacky Li resolved CARBONDATA-1651. -- Resolution: Fixed Fix Version/s: 1.3.0 > Unsupported Spark2 BooleanType > -- > > Key: CARBONDATA-1651 > URL: https://issues.apache.org/jira/browse/CARBONDATA-1651 > Project: CarbonData > Issue Type: Bug > Components: spark-integration >Affects Versions: 1.3.0 >Reporter: Roman Timrov >Assignee: anubhav tarar > Fix For: 1.3.0 > > Time Spent: 2h 50m > Remaining Estimate: 0h > > Unable to save Dataset if it contains field with BooleanType > class CarbonDataFrameWriter > method convertToCarbonType doesn't support it -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[GitHub] carbondata pull request #1491: [CARBONDATA-1651] [Supported Boolean Type Whe...
Github user asfgit closed the pull request at: https://github.com/apache/carbondata/pull/1491 ---
[GitHub] carbondata issue #1491: [CARBONDATA-1651] [Supported Boolean Type When Savin...
Github user jackylk commented on the issue: https://github.com/apache/carbondata/pull/1491 LGTM ---
[jira] [Resolved] (CARBONDATA-1764) Fix issue of when create table with short data type
[ https://issues.apache.org/jira/browse/CARBONDATA-1764?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jacky Li resolved CARBONDATA-1764. -- Resolution: Fixed > Fix issue of when create table with short data type > --- > > Key: CARBONDATA-1764 > URL: https://issues.apache.org/jira/browse/CARBONDATA-1764 > Project: CarbonData > Issue Type: Bug > Components: spark-integration >Affects Versions: 1.2.0 >Reporter: xubo245 >Assignee: xubo245 > Fix For: 1.3.0 > > Time Spent: 40m > Remaining Estimate: 0h > > Fix issue of when create table with short data type -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[GitHub] carbondata pull request #1526: [CARBONDATA-1764] Fix issue of when create ta...
Github user asfgit closed the pull request at: https://github.com/apache/carbondata/pull/1526 ---
[GitHub] carbondata issue #1526: [CARBONDATA-1764] Fix issue of when create table wit...
Github user jackylk commented on the issue: https://github.com/apache/carbondata/pull/1526 LGTM ---
[GitHub] carbondata pull request #1527: [CARBONDATA-1706] Making index merge DDL inse...
GitHub user dhatchayani opened a pull request: https://github.com/apache/carbondata/pull/1527 [CARBONDATA-1706] Making index merge DDL insensitive to the property Compilation issue solved - [ ] Any interfaces changed? - [ ] Any backward compatibility impacted? - [ ] Document update required? - [ ] Testing done Please provide details on - Whether new unit test cases have been added or why no new tests are required? - How it is tested? Please attach test report. - Is it a performance related change? Please attach the performance test report. - Any additional information to help reviewers in testing this change. - [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA. You can merge this pull request into a Git repository by running: $ git pull https://github.com/dhatchayani/incubator-carbondata merge_index_1 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/carbondata/pull/1527.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1527 commit 01044ad820f2da1a568bba438b38ccb8b6500b6c Author: dhatchayani Date: 2017-11-17T17:23:21Z [CARBONDATA-1706] Making index merge DDL insensitive to the property Compilation issue solved ---
[GitHub] carbondata issue #1514: [CARBONDATA-1746] Count star optimization
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1514 Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/1243/ ---
[GitHub] carbondata issue #1520: [CARBONDATA-1734] Ignore empty line while reading CS...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1520 Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/1242/ ---
[GitHub] carbondata issue #1524: [CARBONDATA-1762] Remove existing column level datef...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1524 Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/1241/ ---
[GitHub] carbondata issue #1508: [CARBONDATA-1738] Block direct insert/load on pre-ag...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1508 Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/1240/ ---
[GitHub] carbondata issue #1520: [CARBONDATA-1734] Ignore empty line while reading CS...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1520 Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/1238/ ---
[GitHub] carbondata issue #1523: [CARBONDATA-1756] Improve Boolean data compress rate...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1523 Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/1237/ ---
[GitHub] carbondata issue #1515: [CARBONDATA-1751] Modify sys.err to AnalysisExceptio...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1515 Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/1239/ ---
[GitHub] carbondata issue #1524: [CARBONDATA-1762] Remove existing column level datef...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1524 Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/1236/ ---
[GitHub] carbondata issue #1525: [CARBONDATA-1751] Make the type of exception and mes...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1525 Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/1235/ ---
[GitHub] carbondata issue #1526: [CARBONDATA-1764] Fix issue of when create table wit...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1526 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/1234/ ---
[GitHub] carbondata pull request #1435: [CARBONDATA-1626]add data size and index size...
Github user asfgit closed the pull request at: https://github.com/apache/carbondata/pull/1435 ---
[jira] [Resolved] (CARBONDATA-1626) add datasize and index size to table status file
[ https://issues.apache.org/jira/browse/CARBONDATA-1626?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ravindra Pesala resolved CARBONDATA-1626. - Resolution: Fixed Fix Version/s: 1.3.0 > add datasize and index size to table status file > > > Key: CARBONDATA-1626 > URL: https://issues.apache.org/jira/browse/CARBONDATA-1626 > Project: CarbonData > Issue Type: Improvement >Reporter: Akash R Nilugal >Assignee: Akash R Nilugal >Priority: Minor > Fix For: 1.3.0 > > Time Spent: 3h 20m > Remaining Estimate: 0h > > if carbondata is used in cloud which will have charging or billing for the > queries ran, adding datasize and indexsize in table status will help in > billing features. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[GitHub] carbondata issue #1435: [CARBONDATA-1626]add data size and index size in tab...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/1435 LGTM ---
[jira] [Resolved] (CARBONDATA-1706) Making index merge DDL insensitive to the property
[ https://issues.apache.org/jira/browse/CARBONDATA-1706?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ravindra Pesala resolved CARBONDATA-1706. - Resolution: Fixed Fix Version/s: 1.3.0 > Making index merge DDL insensitive to the property > -- > > Key: CARBONDATA-1706 > URL: https://issues.apache.org/jira/browse/CARBONDATA-1706 > Project: CarbonData > Issue Type: Improvement >Reporter: dhatchayani >Assignee: dhatchayani >Priority: Minor > Fix For: 1.3.0 > > Time Spent: 4h 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[GitHub] carbondata pull request #1494: [CARBONDATA-1706] Making index merge DDL inse...
Github user asfgit closed the pull request at: https://github.com/apache/carbondata/pull/1494 ---
[GitHub] carbondata issue #1494: [CARBONDATA-1706] Making index merge DDL insensitive...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/1494 LGTM ---
[GitHub] carbondata issue #1524: [CARBONDATA-1762] Remove existing column level datef...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1524 Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/1233/ ---
[GitHub] carbondata issue #1520: [CARBONDATA-1734] Ignore empty line while reading CS...
Github user kumarvishal09 commented on the issue: https://github.com/apache/carbondata/pull/1520 retest this please ---
[GitHub] carbondata issue #1435: [CARBONDATA-1626]add data size and index size in tab...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1435 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/1232/ ---
[GitHub] carbondata issue #1520: [CARBONDATA-1734] Ignore empty line while reading CS...
Github user kumarvishal09 commented on the issue: https://github.com/apache/carbondata/pull/1520 retest this please ---
[GitHub] carbondata issue #1520: [CARBONDATA-1734] Ignore empty line while reading CS...
Github user kumarvishal09 commented on the issue: https://github.com/apache/carbondata/pull/1520 rest this please ---
[GitHub] carbondata pull request #1526: [CARBONDATA-1764] Fix issue of when create ta...
GitHub user xubo245 opened a pull request: https://github.com/apache/carbondata/pull/1526 [CARBONDATA-1764] Fix issue of when create table with short data type Be sure to do all of the following checklist to help us incorporate your contribution quickly and easily: - [ ] Any interfaces changed? No - [ ] Any backward compatibility impacted? No - [ ] Document update required? No - [ ] Testing done add DataTypeConverterUtilSuite.scala test case for DataTypeConverterUtil - [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA. MR37 You can merge this pull request into a Git repository by running: $ git pull https://github.com/xubo245/carbondata shortTable Alternatively you can review and apply these changes as the patch at: https://github.com/apache/carbondata/pull/1526.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1526 commit de8bf19a36a11e3ca32bb98bf4ff1453ae21c630 Author: xubo245 <601450...@qq.com> Date: 2017-11-17T15:23:31Z [CARBONDATA-1764] Fix issue of when create table with short data type ---
[jira] [Created] (CARBONDATA-1764) Fix issue of when create table with short data type
xubo245 created CARBONDATA-1764: --- Summary: Fix issue of when create table with short data type Key: CARBONDATA-1764 URL: https://issues.apache.org/jira/browse/CARBONDATA-1764 Project: CarbonData Issue Type: Bug Components: spark-integration Affects Versions: 1.2.0 Reporter: xubo245 Assignee: xubo245 Fix For: 1.3.0 Fix issue of when create table with short data type -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[GitHub] carbondata issue #1508: [CARBONDATA-1738] Block direct insert/load on pre-ag...
Github user kunal642 commented on the issue: https://github.com/apache/carbondata/pull/1508 retest this please ---
[GitHub] carbondata pull request #1525: [CARBONDATA-1751] Make the type of exception ...
GitHub user xubo245 opened a pull request: https://github.com/apache/carbondata/pull/1525 [CARBONDATA-1751] Make the type of exception and message correctly Be sure to do all of the following checklist to help us incorporate your contribution quickly and easily: - [ ] Any interfaces changed? No - [ ] Any backward compatibility impacted? No - [ ] Document update required? No - [ ] Testing done change old test cases to adapt changed massage - [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA. MR55 You can merge this pull request into a Git repository by running: $ git pull https://github.com/xubo245/carbondata msgYaDong Alternatively you can review and apply these changes as the patch at: https://github.com/apache/carbondata/pull/1525.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1525 commit 80ea7c95b00680c43fd8d16f5aa51a30311a8930 Author: xubo245 <601450...@qq.com> Date: 2017-11-17T14:36:46Z [CARBONDATA-1751] Make the type of exception and message correctly ---
[GitHub] carbondata issue #1520: [CARBONDATA-1734] Ignore empty line while reading CS...
Github user kumarvishal09 commented on the issue: https://github.com/apache/carbondata/pull/1520 LGTM except one small comment ---
[GitHub] carbondata pull request #1520: [CARBONDATA-1734] Ignore empty line while rea...
Github user kumarvishal09 commented on a diff in the pull request: https://github.com/apache/carbondata/pull/1520#discussion_r151698402 --- Diff: core/src/main/java/org/apache/carbondata/core/constants/CarbonLoadOptionConstants.java --- @@ -45,6 +45,9 @@ "carbon.options.is.empty.data.bad.record"; public static final String CARBON_OPTIONS_IS_EMPTY_DATA_BAD_RECORD_DEFAULT = "false"; + @CarbonProperty public static final String CARBON_OPTIONS_SKIP_EMPTY_LINE = --- End diff -- Please add documentation ---
