[GitHub] carbondata pull request #2209: [CARBONDATA-2388][SDK]Avro Record Complex Typ...
Github user ravipesala commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2209#discussion_r184919901
--- Diff:
processing/src/main/java/org/apache/carbondata/processing/store/TablePage.java
---
@@ -222,8 +222,9 @@ private void addComplexColumn(int index, int rowId,
byte[] complexColumns) {
ByteBuffer byteArrayInput = ByteBuffer.wrap(complexColumns);
ByteArrayOutputStream byteArrayOutput = new ByteArrayOutputStream();
DataOutputStream dataOutputStream = new
DataOutputStream(byteArrayOutput);
- complexDataType.parseAndBitPack(byteArrayInput, dataOutputStream,
- model.getComplexDimensionKeyGenerator());
+ int startOffset = 0;
--- End diff --
What is the use of `startOffset` when not used?
---
[GitHub] carbondata pull request #2209: [CARBONDATA-2388][SDK]Avro Record Complex Typ...
Github user ravipesala commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2209#discussion_r184919620
--- Diff:
processing/src/main/java/org/apache/carbondata/processing/loading/CarbonDataLoadConfiguration.java
---
@@ -348,6 +356,33 @@ public void
setCardinalityFinder(DictionaryCardinalityFinder cardinalityFinder)
return complexKeyGenerators;
}
+ public Boolean[][] createComplexDictionaryFieldIdentification() {
--- End diff --
If not required, please remove this method
---
[GitHub] carbondata pull request #2209: [CARBONDATA-2388][SDK]Avro Record Complex Typ...
Github user ravipesala commented on a diff in the pull request: https://github.com/apache/carbondata/pull/2209#discussion_r184919569 --- Diff: processing/src/main/java/org/apache/carbondata/processing/datatypes/PrimitiveDataType.java --- @@ -245,6 +312,28 @@ public void parseAndBitPack(ByteBuffer byteArrayInput, DataOutputStream dataOutp dataOutputStream.write(v); } + @Override + public int parseComplexValue(ByteBuffer byteArrayInput, DataOutputStream dataOutputStream, + KeyGenerator[] generator, Boolean[][] complexDictionaryIndentification, int startOffset) --- End diff -- What is the use of `complexDictionaryIndentification` if not used ---
[GitHub] carbondata pull request #2209: [CARBONDATA-2388][SDK]Avro Record Complex Typ...
Github user kumarvishal09 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2209#discussion_r184919435
--- Diff:
processing/src/main/java/org/apache/carbondata/processing/util/CarbonDataProcessorUtil.java
---
@@ -267,14 +267,24 @@ private static String
getComplexTypeString(DataField[] dataFields) {
return dimString.toString();
}
+ private static String isDictionaryType(CarbonDimension dimension) {
--- End diff --
why this method return type is string??? Change this to string
---
[GitHub] carbondata pull request #2209: [CARBONDATA-2388][SDK]Avro Record Complex Typ...
Github user ravipesala commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2209#discussion_r184919308
--- Diff:
processing/src/main/java/org/apache/carbondata/processing/datatypes/PrimitiveDataType.java
---
@@ -245,6 +312,28 @@ public void parseAndBitPack(ByteBuffer byteArrayInput,
DataOutputStream dataOutp
dataOutputStream.write(v);
}
+ @Override
+ public int parseComplexValue(ByteBuffer byteArrayInput, DataOutputStream
dataOutputStream,
+ KeyGenerator[] generator, Boolean[][]
complexDictionaryIndentification, int startOffset)
+ throws IOException, KeyGenException {
+if (!this.isDictionary) {
+ int sizeOfData = byteArrayInput.getInt();
+ startOffset += Integer.SIZE / Byte.SIZE;
+ dataOutputStream.writeInt(sizeOfData);
+ byte[] bb = new byte[sizeOfData];
+ byteArrayInput.get(bb, 0, sizeOfData);
+ dataOutputStream.write(bb);
+ startOffset += sizeOfData;
+} else {
+ int data = byteArrayInput.getInt();
+ startOffset += Integer.SIZE / Byte.SIZE;
--- End diff --
I think this is not required, if it is only writeoffset
---
[GitHub] carbondata pull request #2209: [CARBONDATA-2388][SDK]Avro Record Complex Typ...
Github user kumarvishal09 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2209#discussion_r184919245
--- Diff:
processing/src/main/java/org/apache/carbondata/processing/store/CarbonFactDataHandlerModel.java
---
@@ -609,5 +616,13 @@ public short getWritingCoresCount() {
public DataMapWriterListener getDataMapWriterlistener() {
return dataMapWriterlistener;
}
+
+ public Boolean[][] getComplexDictionaryFields() {
--- End diff --
change this to boolean[][] getComplexDictionaryFields()
---
[GitHub] carbondata issue #2224: [CARBONDATA-2393]TaskNo is not working for SDK
Github user ajantha-bhat commented on the issue: https://github.com/apache/carbondata/pull/2224 retest this please ---
[GitHub] carbondata pull request #2209: [CARBONDATA-2388][SDK]Avro Record Complex Typ...
Github user ravipesala commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2209#discussion_r184919017
--- Diff:
processing/src/main/java/org/apache/carbondata/processing/datatypes/PrimitiveDataType.java
---
@@ -245,6 +312,28 @@ public void parseAndBitPack(ByteBuffer byteArrayInput,
DataOutputStream dataOutp
dataOutputStream.write(v);
}
+ @Override
+ public int parseComplexValue(ByteBuffer byteArrayInput, DataOutputStream
dataOutputStream,
+ KeyGenerator[] generator, Boolean[][]
complexDictionaryIndentification, int startOffset)
+ throws IOException, KeyGenException {
+if (!this.isDictionary) {
+ int sizeOfData = byteArrayInput.getInt();
+ startOffset += Integer.SIZE / Byte.SIZE;
--- End diff --
No need to calculate every time , just use 4 and add comment , Do it in all
places
---
[GitHub] carbondata pull request #2209: [CARBONDATA-2388][SDK]Avro Record Complex Typ...
Github user kumarvishal09 commented on a diff in the pull request: https://github.com/apache/carbondata/pull/2209#discussion_r184918803 --- Diff: processing/src/main/java/org/apache/carbondata/processing/datatypes/PrimitiveDataType.java --- @@ -245,6 +312,28 @@ public void parseAndBitPack(ByteBuffer byteArrayInput, DataOutputStream dataOutp dataOutputStream.write(v); } + @Override + public int parseComplexValue(ByteBuffer byteArrayInput, DataOutputStream dataOutputStream, + KeyGenerator[] generator, Boolean[][] complexDictionaryIndentification, int startOffset) --- End diff -- Boolean[][] complexDictionaryIndentification is not required as each Primitive type field knows whether it is of dictionary/ no dictionary type. ---
[GitHub] carbondata pull request #2209: [CARBONDATA-2388][SDK]Avro Record Complex Typ...
Github user ravipesala commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2209#discussion_r184918720
--- Diff:
processing/src/main/java/org/apache/carbondata/processing/datatypes/ArrayDataType.java
---
@@ -183,7 +187,27 @@ public void parseAndBitPack(ByteBuffer byteArrayInput,
DataOutputStream dataOutp
for (int i = 0; i < dataLength; i++) {
children.parseAndBitPack(byteArrayInput, dataOutputStream,
generator);
}
+ }
+
+ @Override
+ public int parseComplexValue(ByteBuffer byteArrayInput, DataOutputStream
dataOutputStream,
+ KeyGenerator[] generator, Boolean[][]
complexDictionaryIndentification, int startOffset)
+ throws IOException, KeyGenException {
+int dataLength = byteArrayInput.getInt();
+startOffset += Integer.SIZE / Byte.SIZE;
--- End diff --
No need to calculate every time , just use `4` and add comment
---
[GitHub] carbondata pull request #2209: [CARBONDATA-2388][SDK]Avro Record Complex Typ...
Github user ravipesala commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2209#discussion_r184918753
--- Diff:
processing/src/main/java/org/apache/carbondata/processing/datatypes/ArrayDataType.java
---
@@ -183,7 +187,27 @@ public void parseAndBitPack(ByteBuffer byteArrayInput,
DataOutputStream dataOutp
for (int i = 0; i < dataLength; i++) {
children.parseAndBitPack(byteArrayInput, dataOutputStream,
generator);
}
+ }
+
+ @Override
+ public int parseComplexValue(ByteBuffer byteArrayInput, DataOutputStream
dataOutputStream,
+ KeyGenerator[] generator, Boolean[][]
complexDictionaryIndentification, int startOffset)
+ throws IOException, KeyGenException {
+int dataLength = byteArrayInput.getInt();
+startOffset += Integer.SIZE / Byte.SIZE;
+dataOutputStream.writeInt(dataLength);
+if (children instanceof PrimitiveDataType) {
+ if (children.getIsColumnDictionary()) {
+
dataOutputStream.writeInt(generator[children.getSurrogateIndex()].getKeySizeInBytes());
+startOffset += Integer.SIZE / Byte.SIZE;
--- End diff --
No need to calculate every time , just use 4 and add comment
---
[GitHub] carbondata pull request #2209: [CARBONDATA-2388][SDK]Avro Record Complex Typ...
Github user kumarvishal09 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2209#discussion_r184918635
--- Diff:
processing/src/main/java/org/apache/carbondata/processing/datatypes/PrimitiveDataType.java
---
@@ -211,29 +241,66 @@ public int getSurrogateIndex() {
/*
* set surrogate index
*/
- @Override
- public void setSurrogateIndex(int surrIndex) {
-index = surrIndex;
+ @Override public void setSurrogateIndex(int surrIndex) {
+if (this.carbonDimension != null &&
!this.carbonDimension.hasEncoding(Encoding.DICTIONARY)) {
+ index = 0;
+} else if (this.carbonDimension == null && isDictionary == false) {
+ index = 0;
+} else {
+ index = surrIndex;
+}
+ }
+
+ @Override public Boolean getIsColumnDictionary() {
+return isDictionary;
}
@Override public void writeByteArray(Object input, DataOutputStream
dataOutputStream)
throws IOException, DictionaryGenerationException {
String parsedValue =
input == null ? null : DataTypeUtil.parseValue(input.toString(),
carbonDimension);
-Integer surrogateKey;
-if (null == parsedValue) {
- surrogateKey =
CarbonCommonConstants.MEMBER_DEFAULT_VAL_SURROGATE_KEY;
-} else {
- surrogateKey = dictionaryGenerator.getOrGenerateKey(parsedValue);
- if (surrogateKey == CarbonCommonConstants.INVALID_SURROGATE_KEY) {
+if (this.isDictionary) {
+ Integer surrogateKey;
+ if (null == parsedValue) {
surrogateKey =
CarbonCommonConstants.MEMBER_DEFAULT_VAL_SURROGATE_KEY;
+ } else {
+surrogateKey = dictionaryGenerator.getOrGenerateKey(parsedValue);
+if (surrogateKey == CarbonCommonConstants.INVALID_SURROGATE_KEY) {
+ surrogateKey =
CarbonCommonConstants.MEMBER_DEFAULT_VAL_SURROGATE_KEY;
+}
+ }
+ dataOutputStream.writeInt(surrogateKey);
+} else {
+ // Transform into ByteArray for No Dictionary.
+ // TODO have to refactor and place all the cases present in
NonDictionaryFieldConverterImpl
+ if (null == parsedValue && this.carbonDimension.getDataType() !=
DataTypes.STRING) {
+updateNullValue(dataOutputStream);
+ } else {
+byte[] value =
DataTypeUtil.getBytesBasedOnDataTypeForNoDictionaryColumn(parsedValue,
--- End diff --
In case of no dictionary primitive type column NoDictionary field converter
will convert and give the parsed value so no need to call
DataTypeUtil.getBytesBasedOnDataTypeForNoDictionaryColumn again, dictionary
write this value in LV format to outputstream
---
[GitHub] carbondata pull request #2209: [CARBONDATA-2388][SDK]Avro Record Complex Typ...
Github user ravipesala commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2209#discussion_r184918548
--- Diff:
processing/src/main/java/org/apache/carbondata/processing/datatypes/ArrayDataType.java
---
@@ -171,9 +175,9 @@ public void fillCardinality(List
dimCardWithComplex) {
/**
* parse byte array and bit pack
*/
- @Override
- public void parseAndBitPack(ByteBuffer byteArrayInput, DataOutputStream
dataOutputStream,
- KeyGenerator[] generator) throws IOException, KeyGenException {
+ @Override public void parseAndBitPack(ByteBuffer byteArrayInput,
--- End diff --
Now this method is not used? if not delete it
---
[GitHub] carbondata pull request #2209: [CARBONDATA-2388][SDK]Avro Record Complex Typ...
Github user ravipesala commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2209#discussion_r184918197
--- Diff:
integration/spark2/src/main/scala/org/apache/spark/util/SparkTypeConverter.scala
---
@@ -123,13 +137,29 @@ private[spark] object SparkTypeConverter {
private def recursiveMethod(
table: CarbonTable, dimName: String, childDim: CarbonDimension) = {
childDim.getDataType.getName.toLowerCase match {
- case "array" => s"${
-childDim.getColName.substring(dimName.length + 1)
- }:array<${ getArrayChildren(table, childDim.getColName) }>"
- case "struct" => s"${
-childDim.getColName.substring(dimName.length + 1)
- }:struct<${ getStructChildren(table, childDim.getColName) }>"
- case dType => s"${ childDim.getColName.substring(dimName.length + 1)
}:${ dType }"
+ case "array" => if (table.isTransactionalTable) {
--- End diff --
Better write another method rather than keeping so many if checks
---
[GitHub] carbondata pull request #2209: [CARBONDATA-2388][SDK]Avro Record Complex Typ...
Github user ravipesala commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2209#discussion_r184918157
--- Diff:
integration/spark2/src/main/scala/org/apache/spark/util/SparkTypeConverter.scala
---
@@ -97,16 +97,30 @@ private[spark] object SparkTypeConverter {
def getStructChildren(table: CarbonTable, dimName: String): String = {
table.getChildren(dimName).asScala.map(childDim => {
childDim.getDataType.getName.toLowerCase match {
-case "array" => s"${
+case "array" => if (table.isTransactionalTable) {s"${
--- End diff --
Could explain in comments why this many checks are required for non
transactional table?
Better write another method rather than keeping so many if checks
---
[GitHub] carbondata pull request #2209: [CARBONDATA-2388][SDK]Avro Record Complex Typ...
Github user kumarvishal09 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2209#discussion_r184918097
--- Diff:
processing/src/main/java/org/apache/carbondata/processing/datatypes/PrimitiveDataType.java
---
@@ -211,29 +241,66 @@ public int getSurrogateIndex() {
/*
* set surrogate index
*/
- @Override
- public void setSurrogateIndex(int surrIndex) {
-index = surrIndex;
+ @Override public void setSurrogateIndex(int surrIndex) {
+if (this.carbonDimension != null &&
!this.carbonDimension.hasEncoding(Encoding.DICTIONARY)) {
+ index = 0;
+} else if (this.carbonDimension == null && isDictionary == false) {
+ index = 0;
+} else {
+ index = surrIndex;
+}
+ }
+
+ @Override public Boolean getIsColumnDictionary() {
+return isDictionary;
}
@Override public void writeByteArray(Object input, DataOutputStream
dataOutputStream)
throws IOException, DictionaryGenerationException {
String parsedValue =
input == null ? null : DataTypeUtil.parseValue(input.toString(),
carbonDimension);
-Integer surrogateKey;
-if (null == parsedValue) {
- surrogateKey =
CarbonCommonConstants.MEMBER_DEFAULT_VAL_SURROGATE_KEY;
-} else {
- surrogateKey = dictionaryGenerator.getOrGenerateKey(parsedValue);
- if (surrogateKey == CarbonCommonConstants.INVALID_SURROGATE_KEY) {
+if (this.isDictionary) {
+ Integer surrogateKey;
+ if (null == parsedValue) {
surrogateKey =
CarbonCommonConstants.MEMBER_DEFAULT_VAL_SURROGATE_KEY;
+ } else {
+surrogateKey = dictionaryGenerator.getOrGenerateKey(parsedValue);
+if (surrogateKey == CarbonCommonConstants.INVALID_SURROGATE_KEY) {
+ surrogateKey =
CarbonCommonConstants.MEMBER_DEFAULT_VAL_SURROGATE_KEY;
+}
+ }
+ dataOutputStream.writeInt(surrogateKey);
+} else {
+ // Transform into ByteArray for No Dictionary.
+ // TODO have to refactor and place all the cases present in
NonDictionaryFieldConverterImpl
+ if (null == parsedValue && this.carbonDimension.getDataType() !=
DataTypes.STRING) {
--- End diff --
null value updation is also required for String data type second condition
in If check is not correct
---
[GitHub] carbondata pull request #2209: [CARBONDATA-2388][SDK]Avro Record Complex Typ...
Github user ravipesala commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2209#discussion_r184917833
--- Diff:
core/src/main/java/org/apache/carbondata/core/scan/complextypes/PrimitiveQueryType.java
---
@@ -108,4 +128,14 @@ public PrimitiveQueryType(String name, String
parentname, int blockIndex,
}
return actualData;
}
+
+ @Override public Object
getDataBasedOnDataTypeFromNoDictionary(ByteBuffer data) {
--- End diff --
I don't think it is required to have one more method, just rename and
handle in same method `getDataBasedOnDataTypeFromSurrogates` , it can avoid
duplicate code and extra handling
---
[GitHub] carbondata pull request #2209: [CARBONDATA-2388][SDK]Avro Record Complex Typ...
Github user kumarvishal09 commented on a diff in the pull request: https://github.com/apache/carbondata/pull/2209#discussion_r184917713 --- Diff: processing/src/main/java/org/apache/carbondata/processing/datatypes/PrimitiveDataType.java --- @@ -94,6 +99,11 @@ private CarbonDimension carbonDimension; + private Boolean isDictionary; + + private FieldConverter fieldConverterForNoDictionary; --- End diff -- FieldConverter is not required in this class, as caller will create for primitive type like existing flow..same way we can handle for no dictionary ---
[GitHub] carbondata issue #1812: [CARBONDATA-2033]Support user specified segments in ...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/1812 SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/4626/ ---
[GitHub] carbondata pull request #2209: [CARBONDATA-2388][SDK]Avro Record Complex Typ...
Github user ravipesala commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2209#discussion_r184917500
--- Diff:
core/src/main/java/org/apache/carbondata/core/scan/complextypes/PrimitiveQueryType.java
---
@@ -84,6 +101,9 @@ public PrimitiveQueryType(String name, String
parentname, int blockIndex,
DimensionRawColumnChunk[] rawColumnChunks, int rowNumber,
int pageNumber, DataOutputStream dataOutputStream) throws
IOException {
byte[] currentVal = copyBlockDataChunk(rawColumnChunks, rowNumber,
pageNumber);
+if (!this.isDictionary) {
+ dataOutputStream.writeInt(currentVal.length);
--- End diff --
Just writing length? where are you writing data?
---
[GitHub] carbondata pull request #2209: [CARBONDATA-2388][SDK]Avro Record Complex Typ...
Github user ravipesala commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2209#discussion_r184917127
--- Diff:
core/src/main/java/org/apache/carbondata/core/scan/complextypes/PrimitiveQueryType.java
---
@@ -53,6 +55,21 @@ public PrimitiveQueryType(String name, String
parentname, int blockIndex,
this.name = name;
this.parentname = parentname;
this.isDirectDictionary = isDirectDictionary;
+this.isDictionary = true;
+ }
+
+
+ public PrimitiveQueryType(String name, String parentname, int blockIndex,
+ org.apache.carbondata.core.metadata.datatype.DataType dataType, int
keySize,
+ Dictionary dictionary, boolean isDirectDictionary, boolean
isDictionary) {
--- End diff --
I don't think it is required to have a separate constructor, if (dictionary
== null && !isDirectDictionary) then it becomes nodictionary
---
[GitHub] carbondata pull request #2209: [CARBONDATA-2388][SDK]Avro Record Complex Typ...
Github user kumarvishal09 commented on a diff in the pull request: https://github.com/apache/carbondata/pull/2209#discussion_r184917041 --- Diff: processing/src/main/java/org/apache/carbondata/processing/datatypes/PrimitiveDataType.java --- @@ -94,6 +99,11 @@ private CarbonDimension carbonDimension; + private Boolean isDictionary; --- End diff -- change this to boolean isDictionary ---
[GitHub] carbondata pull request #2209: [CARBONDATA-2388][SDK]Avro Record Complex Typ...
Github user kumarvishal09 commented on a diff in the pull request: https://github.com/apache/carbondata/pull/2209#discussion_r184916986 --- Diff: processing/src/main/java/org/apache/carbondata/processing/datatypes/GenericDataType.java --- @@ -71,6 +71,8 @@ void writeByteArray(T input, DataOutputStream dataOutputStream) */ void setSurrogateIndex(int surrIndex); + Boolean getIsColumnDictionary(); --- End diff -- change this to boolean isDictionaryEncoded() ---
[GitHub] carbondata pull request #2209: [CARBONDATA-2388][SDK]Avro Record Complex Typ...
Github user kumarvishal09 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2209#discussion_r184916825
--- Diff:
processing/src/main/java/org/apache/carbondata/processing/datatypes/ArrayDataType.java
---
@@ -183,7 +187,27 @@ public void parseAndBitPack(ByteBuffer byteArrayInput,
DataOutputStream dataOutp
for (int i = 0; i < dataLength; i++) {
children.parseAndBitPack(byteArrayInput, dataOutputStream,
generator);
}
+ }
+
+ @Override
+ public int parseComplexValue(ByteBuffer byteArrayInput, DataOutputStream
dataOutputStream,
+ KeyGenerator[] generator, Boolean[][]
complexDictionaryIndentification, int startOffset)
--- End diff --
complexDictionaryIndentification is not required as each primitive children
knows it is not of dictionary type or no dictionary type
---
[GitHub] carbondata pull request #2209: [CARBONDATA-2388][SDK]Avro Record Complex Typ...
Github user ravipesala commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2209#discussion_r184916731
--- Diff:
core/src/main/java/org/apache/carbondata/core/scan/collector/impl/DictionaryBasedResultCollector.java
---
@@ -134,7 +134,13 @@ void fillDimensionData(BlockletScannedResult
scannedResult, int[] surrogateResul
row[order[i]] =
DataTypeUtil.getDataBasedOnDataType(scannedResult.getBlockletId(),
DataTypes.STRING);
}
- } else {
+ } else if (complexDataTypeArray[i]) {
+// Complex Type With No Dictionary Encoding.
+row[order[i]] =
comlexDimensionInfoMap.get(queryDimensions[i].getDimension().getOrdinal())
+.getDataBasedOnDataTypeFromNoDictionary(
+
ByteBuffer.wrap(complexTypeKeyArray[complexTypeColumnIndex++]));
+ } else
+ {
--- End diff --
Move the braces up
---
[GitHub] carbondata pull request #2209: [CARBONDATA-2388][SDK]Avro Record Complex Typ...
Github user kumarvishal09 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2209#discussion_r184916594
--- Diff:
processing/src/main/java/org/apache/carbondata/processing/datatypes/ArrayDataType.java
---
@@ -171,9 +175,9 @@ public void fillCardinality(List
dimCardWithComplex) {
/**
* parse byte array and bit pack
*/
- @Override
- public void parseAndBitPack(ByteBuffer byteArrayInput, DataOutputStream
dataOutputStream,
- KeyGenerator[] generator) throws IOException, KeyGenException {
+ @Override public void parseAndBitPack(ByteBuffer byteArrayInput,
--- End diff --
Please remove this method from interface as below new method will be used
---
[GitHub] carbondata pull request #2244: [CARBONDATA-2310] Refactored code to improve ...
Github user manishgupta88 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2244#discussion_r184915943
--- Diff:
hadoop/src/main/java/org/apache/carbondata/hadoop/CacheClient.java ---
@@ -16,21 +16,40 @@
*/
package org.apache.carbondata.hadoop;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+
+import org.apache.carbondata.common.logging.LogService;
+import org.apache.carbondata.common.logging.LogServiceFactory;
import org.apache.carbondata.core.cache.Cache;
import org.apache.carbondata.core.cache.CacheProvider;
import org.apache.carbondata.core.cache.CacheType;
+import org.apache.carbondata.core.datastore.SegmentTaskIndexStore;
import org.apache.carbondata.core.datastore.TableSegmentUniqueIdentifier;
+import org.apache.carbondata.core.datastore.block.SegmentProperties;
import org.apache.carbondata.core.datastore.block.SegmentTaskIndexWrapper;
+import org.apache.carbondata.core.metadata.AbsoluteTableIdentifier;
+import
org.apache.carbondata.core.metadata.schema.table.column.ColumnSchema;
/**
* CacheClient : Holds all the Cache access clients for Btree, Dictionary
*/
public class CacheClient {
+ private static final LogService LOGGER =
+ LogServiceFactory.getLogService(CacheClient.class.getName());
+
+ private final Object lock = new Object();
+
// segment access client for driver LRU cache
private CacheAccessClient
segmentAccessClient;
+ private static Map
+ segmentProperties =
+ new ConcurrentHashMap<>();
--- End diff --
Move this to above line
---
[GitHub] carbondata pull request #2244: [CARBONDATA-2310] Refactored code to improve ...
Github user manishgupta88 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2244#discussion_r184911430
--- Diff:
core/src/main/java/org/apache/carbondata/core/datamap/dev/expr/DataMapExprWrapperImpl.java
---
@@ -86,4 +87,8 @@ public DataMapExprWrapperImpl(TableDataMap dataMap,
FilterResolverIntf expressio
@Override public DataMapLevel getDataMapType() {
return dataMap.getDataMapFactory().getDataMapType();
}
+
+ public DataMapFactory getDataMapFactory() {
--- End diff --
this is not an interface method...can we try to get the dataMapFactory in
some other way and remove this method from here?
---
[GitHub] carbondata pull request #2244: [CARBONDATA-2310] Refactored code to improve ...
Github user manishgupta88 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2244#discussion_r184915870
--- Diff:
core/src/main/java/org/apache/carbondata/core/util/BlockletDataMapUtil.java ---
@@ -0,0 +1,198 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.carbondata.core.util;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.TreeMap;
+
+import org.apache.carbondata.common.logging.LogService;
+import org.apache.carbondata.common.logging.LogServiceFactory;
+import org.apache.carbondata.core.constants.CarbonCommonConstants;
+import org.apache.carbondata.core.datamap.Segment;
+import
org.apache.carbondata.core.datastore.filesystem.AbstractDFSCarbonFile;
+import org.apache.carbondata.core.datastore.filesystem.CarbonFile;
+import org.apache.carbondata.core.datastore.impl.FileFactory;
+import org.apache.carbondata.core.indexstore.BlockMetaInfo;
+import
org.apache.carbondata.core.indexstore.TableBlockIndexUniqueIdentifier;
+import
org.apache.carbondata.core.indexstore.blockletindex.BlockletDataMapDistributable;
+import
org.apache.carbondata.core.indexstore.blockletindex.SegmentIndexFileStore;
+import org.apache.carbondata.core.metadata.blocklet.DataFileFooter;
+import org.apache.carbondata.core.util.path.CarbonTablePath;
+
+import org.apache.hadoop.fs.LocatedFileStatus;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.PathFilter;
+import org.apache.hadoop.fs.RemoteIterator;
+
+
+public class BlockletDataMapUtil {
+
+ private static final LogService LOGGER =
+ LogServiceFactory.getLogService(BlockletDataMapUtil.class.getName());
+
+ public static Map getBlockMetaInfoMap(
+ TableBlockIndexUniqueIdentifier identifier, SegmentIndexFileStore
indexFileStore,
+ Set filesRead, Map
fileNameToMetaInfoMapping)
+ throws IOException {
+if (identifier.getMergeIndexFileName() != null
+&& indexFileStore.getFileData(identifier.getIndexFileName()) ==
null) {
+ CarbonFile indexMergeFile = FileFactory.getCarbonFile(
+ identifier.getIndexFilePath() +
CarbonCommonConstants.FILE_SEPARATOR + identifier
+ .getMergeIndexFileName());
+ if (indexMergeFile.exists() &&
!filesRead.contains(indexMergeFile.getPath())) {
+indexFileStore.readAllIIndexOfSegment(new CarbonFile[] {
indexMergeFile });
+filesRead.add(indexMergeFile.getPath());
+ }
+}
+if (indexFileStore.getFileData(identifier.getIndexFileName()) == null)
{
+ indexFileStore.readAllIIndexOfSegment(new CarbonFile[] {
FileFactory.getCarbonFile(
+ identifier.getIndexFilePath() +
CarbonCommonConstants.FILE_SEPARATOR + identifier
+ .getIndexFileName()) });
+}
+DataFileFooterConverter fileFooterConverter = new
DataFileFooterConverter();
+Map blockMetaInfoMap = new HashMap<>();
+List indexInfo = fileFooterConverter.getIndexInfo(
+identifier.getIndexFilePath() +
CarbonCommonConstants.FILE_SEPARATOR + identifier
+.getIndexFileName(),
indexFileStore.getFileData(identifier.getIndexFileName()));
+for (DataFileFooter footer : indexInfo) {
+ String blockPath =
footer.getBlockInfo().getTableBlockInfo().getFilePath();
+ if (FileFactory.isFileExist(blockPath)) {
--- End diff --
Is this if check required?.. It will make a namenode call for every
blocklet entry here so better to remove this
---
[GitHub] carbondata pull request #2244: [CARBONDATA-2310] Refactored code to improve ...
Github user manishgupta88 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2244#discussion_r184914643
--- Diff:
core/src/main/java/org/apache/carbondata/core/indexstore/blockletindex/BlockletDataMapFactory.java
---
@@ -277,4 +298,30 @@ public DataMapMeta getMeta() {
}
return blocklets;
}
+
+ @Override public void cache(TableBlockIndexUniqueIdentifier
tableBlockIndexUniqueIdentifier,
+ BlockletDataMapIndexWrapper blockletDataMapIndexWrapper) throws
IOException, MemoryException {
+cache.put(tableBlockIndexUniqueIdentifier,
blockletDataMapIndexWrapper);
+ }
+
+ @Override
+ public List getAllUncachedDistributables(
+ List distributables) throws IOException {
+List distributablesToBeLoaded = new
ArrayList<>(distributables.size());
+for (DataMapDistributable distributable : distributables) {
+ Segment segment = distributable.getSegment();
+ Set
tableBlockIndexUniqueIdentifiers =
+ getTableBlockIndexUniqueIdentifiers(segment);
+ // filter out the tableBlockIndexUniqueIdentifiers based on
distributable
+ TableBlockIndexUniqueIdentifier validIdentifier = BlockletDataMapUtil
+
.filterIdentifiersBasedOnDistributable(tableBlockIndexUniqueIdentifiers,
+ (BlockletDataMapDistributable) distributable);
+ ((BlockletDataMapDistributable) distributable)
+ .setTableBlockIndexUniqueIdentifier(validIdentifier);
--- End diff --
Move this line inside below if check
---
[GitHub] carbondata pull request #2244: [CARBONDATA-2310] Refactored code to improve ...
Github user manishgupta88 commented on a diff in the pull request: https://github.com/apache/carbondata/pull/2244#discussion_r184914866 --- Diff: core/src/main/java/org/apache/carbondata/core/metadata/blocklet/DataFileFooter.java --- @@ -52,7 +52,7 @@ /** * Information about leaf nodes of all columns in this file */ - private List blockletList; + private transient List blockletList; --- End diff -- why transient is required here? ---
[GitHub] carbondata pull request #2209: [CARBONDATA-2388][SDK]Avro Record Complex Typ...
Github user ravipesala commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2209#discussion_r184916441
--- Diff:
core/src/main/java/org/apache/carbondata/core/metadata/schema/table/column/ColumnSchema.java
---
@@ -43,6 +43,9 @@
*/
private static final long serialVersionUID = 7676766554874863763L;
+ public void columnSchema() {
--- End diff --
What is the use of this method
---
[GitHub] carbondata pull request #2209: [CARBONDATA-2388][SDK]Avro Record Complex Typ...
Github user kumarvishal09 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2209#discussion_r184916437
--- Diff:
processing/src/main/java/org/apache/carbondata/processing/datatypes/ArrayDataType.java
---
@@ -171,9 +175,9 @@ public void fillCardinality(List
dimCardWithComplex) {
/**
* parse byte array and bit pack
*/
- @Override
- public void parseAndBitPack(ByteBuffer byteArrayInput, DataOutputStream
dataOutputStream,
- KeyGenerator[] generator) throws IOException, KeyGenException {
+ @Override public void parseAndBitPack(ByteBuffer byteArrayInput,
+ DataOutputStream dataOutputStream, KeyGenerator[] generator)
+ throws IOException, KeyGenException {
int dataLength = byteArrayInput.getInt();
--- End diff --
remove extra space
---
[GitHub] carbondata pull request #2251: [CARBONDATA-2417] SDK writer goes to infinite...
GitHub user ajantha-bhat opened a pull request: https://github.com/apache/carbondata/pull/2251 [CARBONDATA-2417] SDK writer goes to infinite wait when consumer thread is dead [CARBONDATA-2417] problem: SDK writer goes to infinite wait when consumer thread is dead root cause: due to bad record when exception happens at consumer thread during write, this message is not reached producer (sdk writer). So, SDK keeps writing data assuming consumer will consume it. But as consumer is dead. Queue becomes full and queue.put() will be blocked forever. Solution: If cannot be added to queue, check whether consumer is alive or not after every 10 seconds. If not alive throw exception, if alive try again - [ ] Any interfaces changed? no - [ ] Any backward compatibility impacted? no - [ ] Document update required? no - [ ] Testing done updated the testcase in TestNonTransactionalCarbonTable.scala - [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA. NA You can merge this pull request into a Git repository by running: $ git pull https://github.com/ajantha-bhat/carbondata branch3 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/carbondata/pull/2251.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2251 commit 7f19372fc32a6978ac5aa87f20f346e75a9b1e9d Author: ajantha-bhat Date: 2018-04-28T12:47:35Z [CARBONDATA-2417] problem: SDK writer goes to infinite wait when cosumer thread is dead root cause: due to bad record when exception happens at consumer thread during write, this message is not reached producer (sdk writer). So, SDK keeps writing data assuming consumer will consume it. But as consumer is dead. Queue becomes full and queue.put() will be blocked forever. Solution: If cannot be added to queue, check whether consumer is alive or not after every 10 seconds. If not alive throw exception, if alive try again ---
[GitHub] carbondata issue #2195: [CARBONDATA-2366] fixed concurrent datamap creation ...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2195 Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/5524/ ---
[GitHub] carbondata pull request #2209: [CARBONDATA-2388][SDK]Avro Record Complex Typ...
Github user kumarvishal09 commented on a diff in the pull request: https://github.com/apache/carbondata/pull/2209#discussion_r184916011 --- Diff: core/src/main/java/org/apache/carbondata/core/scan/complextypes/PrimitiveQueryType.java --- @@ -53,6 +55,21 @@ public PrimitiveQueryType(String name, String parentname, int blockIndex, this.name = name; this.parentname = parentname; this.isDirectDictionary = isDirectDictionary; +this.isDictionary = true; + } + + + public PrimitiveQueryType(String name, String parentname, int blockIndex, --- End diff -- One more extra constructor is not required. If dictionary is not null then isDictionary=true ---
[GitHub] carbondata issue #2202: [CARBONDATA-2347][LUCENE]change datamap factory inte...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2202 Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/4359/ ---
[GitHub] carbondata issue #2202: [CARBONDATA-2347][LUCENE]change datamap factory inte...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2202 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/5523/ ---
[jira] [Created] (CARBONDATA-2417) SDK writer goes to infinite wait when consumer thread goes dead
Ajantha Bhat created CARBONDATA-2417: Summary: SDK writer goes to infinite wait when consumer thread goes dead Key: CARBONDATA-2417 URL: https://issues.apache.org/jira/browse/CARBONDATA-2417 Project: CarbonData Issue Type: Bug Reporter: Ajantha Bhat Assignee: Ajantha Bhat problem: SDK writer goes to infinite wait when cosumer thread is dead root cause: due to bad record when exception happens at consumer thread during write, this message is not reached producer (sdk writer). So, SDK keeps writing data assuming consumer will consume it. But as consumer is dead. Queue becomes full and queue.put() will be blocked forever. Solution: If cannot be added to queue, check whether consumer is alive or not after every 10 seconds. If not alive throw exception, if alive try again -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] carbondata issue #2210: [CARBONDATA-2381] Improve compaction performance by ...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2210 SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/4625/ ---
[GitHub] carbondata issue #2172: [CARBONDATA-2333] Add validation for insert overwrit...
Github user kunal642 commented on the issue: https://github.com/apache/carbondata/pull/2172 retest this please ---
[GitHub] carbondata issue #2195: [CARBONDATA-2366] fixed concurrent datamap creation ...
Github user kunal642 commented on the issue: https://github.com/apache/carbondata/pull/2195 retest this please ---
[GitHub] carbondata issue #1812: [CARBONDATA-2033]Support user specified segments in ...
Github user manishgupta88 commented on the issue: https://github.com/apache/carbondata/pull/1812 retest sdv please ---
[GitHub] carbondata issue #2210: [CARBONDATA-2381] Improve compaction performance by ...
Github user manishgupta88 commented on the issue: https://github.com/apache/carbondata/pull/2210 retest sdv please ---
[GitHub] carbondata issue #2209: [CARBONDATA-2388][SDK]Avro Record Complex Type Imple...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2209 Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/5522/ ---
[GitHub] carbondata issue #2209: [CARBONDATA-2388][SDK]Avro Record Complex Type Imple...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2209 Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/4358/ ---
[GitHub] carbondata issue #2209: [CARBONDATA-2388][SDK]Avro Record Complex Type Imple...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2209 SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/4624/ ---
[GitHub] carbondata issue #2209: [CARBONDATA-2388][SDK]Avro Record Complex Type Imple...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2209 SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/4623/ ---
[GitHub] carbondata issue #2209: [CARBONDATA-2388][SDK]Avro Record Complex Type Imple...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2209 Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/5521/ ---
[GitHub] carbondata issue #2209: [CARBONDATA-2388][SDK]Avro Record Complex Type Imple...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2209 Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/4357/ ---
[GitHub] carbondata issue #2209: [CARBONDATA-2388][SDK]Avro Record Complex Type Imple...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2209 SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/4622/ ---
[GitHub] carbondata pull request #2209: [CARBONDATA-2388][SDK]Avro Record Complex Typ...
Github user ajantha-bhat commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2209#discussion_r184905554
--- Diff:
core/src/main/java/org/apache/carbondata/core/metadata/schema/SchemaReader.java
---
@@ -81,10 +81,13 @@ public static TableInfo
getTableInfo(AbsoluteTableIdentifier identifier)
}
public static TableInfo inferSchema(AbsoluteTableIdentifier identifier,
- boolean isCarbonFileProvider) throws IOException {
+ boolean isCarbonFileProvider, TableInfo tableInfoFromCache) throws
IOException {
// This routine is going to infer schema from the carbondata file
footer
// Convert the ColumnSchema -> TableSchema -> TableInfo.
// Return the TableInfo.
+if (tableInfoFromCache != null) {
--- End diff --
If tableInfoFromCache is found, No need to call inferschema itself, please
handle this outside infer schema
---
[jira] [Assigned] (CARBONDATA-2313) Support Non Transactional carbon table
[ https://issues.apache.org/jira/browse/CARBONDATA-2313?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sounak chakraborty reassigned CARBONDATA-2313: -- Assignee: sounak chakraborty > Support Non Transactional carbon table > -- > > Key: CARBONDATA-2313 > URL: https://issues.apache.org/jira/browse/CARBONDATA-2313 > Project: CarbonData > Issue Type: New Feature >Reporter: Ajantha Bhat >Assignee: sounak chakraborty >Priority: Major > Attachments: carbon NonTranscational Table.pdf > > Time Spent: 20h 20m > Remaining Estimate: 0h > > h5. Support unmanaged carbon table -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] carbondata pull request #2209: [WIP][Non Transactional Table]Avro Record Com...
Github user ajantha-bhat commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2209#discussion_r184904922
--- Diff:
integration/spark-common-test/src/test/scala/org/apache/carbondata/spark/testsuite/createTable/TestNonTransactionalCarbonTable.scala
---
@@ -175,9 +175,31 @@ class TestNonTransactionalCarbonTable extends
QueryTest with BeforeAndAfterAll {
test("test create External Table with Schema with partition, should
ignore schema and partition")
{
-buildTestDataSingleFile()
+sql("DROP TABLE IF EXISTS sdkOutputTable")
+
+// with partition
+sql("CREATE EXTERNAL TABLE sdkOutputTable(name string) PARTITIONED BY
(age int) STORED BY 'carbondata' LOCATION '/home/root1/avro/files' ")
+//
+//checkAnswer(sql("select * from sdkOutputTable"), Seq(Row("robot0",
0, 0.0),
+// Row("robot1", 1, 0.5),
+// Row("robot2", 2, 1.0)))
--- End diff --
please revert back this test case and add new one
---
[GitHub] carbondata issue #2209: [WIP][Non Transactional Table]Avro Record Complex Ty...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2209 Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/5519/ ---
[GitHub] carbondata issue #2209: [WIP][Non Transactional Table]Avro Record Complex Ty...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2209 Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/4355/ ---
[GitHub] carbondata issue #2209: [WIP][Non Transactional Table]Avro Record Complex Ty...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2209 SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/4621/ ---
[GitHub] carbondata issue #2209: [WIP][Non Transactional Table]Avro Record Complex Ty...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2209 SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/4620/ ---
[GitHub] carbondata issue #2250: [HOTFIX] Fix lucene match limit code
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2250 SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/4619/ ---
[GitHub] carbondata issue #2250: [HOTFIX] Fix lucene match limit code
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2250 Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/4353/ ---
[GitHub] carbondata issue #2207: [WIP] Support flat folder for managed carbon table
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2207 SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/4618/ ---
[GitHub] carbondata issue #2250: [HOTFIX] Fix lucene match limit code
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2250 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/5517/ ---
[GitHub] carbondata issue #2209: [WIP][Non Transactional Table]Avro Record Complex Ty...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2209 Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/5518/ ---
[GitHub] carbondata issue #2209: [WIP][Non Transactional Table]Avro Record Complex Ty...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2209 Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/4354/ ---
[GitHub] carbondata issue #2207: [WIP] Support flat folder for managed carbon table
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2207 SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/4617/ ---
[GitHub] carbondata issue #2207: [WIP] Support flat folder for managed carbon table
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2207 Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/4352/ ---
[GitHub] carbondata issue #2207: [WIP] Support flat folder for managed carbon table
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2207 Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/5516/ ---
[GitHub] carbondata issue #2207: [WIP] Support flat folder for managed carbon table
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2207 SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/4616/ ---
[GitHub] carbondata pull request #2250: [HOTFIX] Fix lucene match limit code
GitHub user ravipesala opened a pull request: https://github.com/apache/carbondata/pull/2250 [HOTFIX] Fix lucene match limit code Problem Currently, Lucene match limit is set as Static in `MatchExpression` it cannot work in concurrent scenarios. Solution: Change to object variable and get the match max limit from expression. Be sure to do all of the following checklist to help us incorporate your contribution quickly and easily: - [ ] Any interfaces changed? - [ ] Any backward compatibility impacted? - [ ] Document update required? - [ ] Testing done Please provide details on - Whether new unit test cases have been added or why no new tests are required? - How it is tested? Please attach test report. - Is it a performance related change? Please attach the performance test report. - Any additional information to help reviewers in testing this change. - [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA. You can merge this pull request into a Git repository by running: $ git pull https://github.com/ravipesala/incubator-carbondata lucene-matchlimit-fix Alternatively you can review and apply these changes as the patch at: https://github.com/apache/carbondata/pull/2250.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2250 commit 32eb738f50b53c76ce95630278ae86216bf4223a Author: ravipesala Date: 2018-04-29T15:19:18Z Fix lucene match limit code ---
[GitHub] carbondata issue #2207: [WIP] Support flat folder for managed carbon table
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2207 Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/4351/ ---
[GitHub] carbondata issue #2207: [WIP] Support flat folder for managed carbon table
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2207 Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/4350/ ---
[GitHub] carbondata issue #2207: [WIP] Support flat folder for managed carbon table
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2207 Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/5515/ ---
[GitHub] carbondata issue #2249: [WIP]sortColumns Order we are getting wrong as we se...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2249 SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/4615/ ---
[GitHub] carbondata issue #2249: [WIP]sortColumns Order we are getting wrong as we se...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2249 Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/4349/ ---
[GitHub] carbondata issue #2249: [WIP]sortColumns Order we are getting wrong as we se...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2249 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/5514/ ---
[GitHub] carbondata pull request #2249: [WIP]sortColumns Order we are getting wrong a...
GitHub user rahulforallp opened a pull request: https://github.com/apache/carbondata/pull/2249 [WIP]sortColumns Order we are getting wrong as we set for external table is fixed Be sure to do all of the following checklist to help us incorporate your contribution quickly and easily: - [ ] Any interfaces changed? - [ ] Any backward compatibility impacted? - [ ] Document update required? - [ ] Testing done Please provide details on - Whether new unit test cases have been added or why no new tests are required? - How it is tested? Please attach test report. - Is it a performance related change? Please attach the performance test report. - Any additional information to help reviewers in testing this change. - [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA. You can merge this pull request into a Git repository by running: $ git pull https://github.com/rahulforallp/incubator-carbondata sort_col_sdk Alternatively you can review and apply these changes as the patch at: https://github.com/apache/carbondata/pull/2249.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2249 ---
[GitHub] carbondata issue #2248: [CARBONDATA-2415][WIP] Support Refresh DataMap comma...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2248 SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/4614/ ---
[jira] [Resolved] (CARBONDATA-2371) Add Profiler output in EXPLAIN command
[ https://issues.apache.org/jira/browse/CARBONDATA-2371?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ravindra Pesala resolved CARBONDATA-2371. - Resolution: Fixed > Add Profiler output in EXPLAIN command > -- > > Key: CARBONDATA-2371 > URL: https://issues.apache.org/jira/browse/CARBONDATA-2371 > Project: CarbonData > Issue Type: New Feature >Reporter: Jacky Li >Priority: Major > Fix For: 1.4.0 > > Time Spent: 8h 40m > Remaining Estimate: 0h > > More information should give in EXPLAIN command to show the effeteness of > datamap -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] carbondata pull request #2197: [CARBONDATA-2371] Add Profiler output in EXPL...
Github user asfgit closed the pull request at: https://github.com/apache/carbondata/pull/2197 ---
[GitHub] carbondata issue #2248: [CARBONDATA-2415][WIP] Support Refresh DataMap comma...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2248 Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/5513/ ---
[GitHub] carbondata issue #2248: [CARBONDATA-2415][WIP] Support Refresh DataMap comma...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2248 Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/4348/ ---
[GitHub] carbondata pull request #2248: [CARBONDATA-2415][WIP] Support Refresh DataMa...
GitHub user jackylk opened a pull request: https://github.com/apache/carbondata/pull/2248 [CARBONDATA-2415][WIP] Support Refresh DataMap command for all Index datamap WIP - [ ] Any interfaces changed? - [ ] Any backward compatibility impacted? - [ ] Document update required? - [ ] Testing done Please provide details on - Whether new unit test cases have been added or why no new tests are required? - How it is tested? Please attach test report. - Is it a performance related change? Please attach the performance test report. - Any additional information to help reviewers in testing this change. - [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA. You can merge this pull request into a Git repository by running: $ git pull https://github.com/jackylk/incubator-carbondata index-datamap-refactory Alternatively you can review and apply these changes as the patch at: https://github.com/apache/carbondata/pull/2248.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2248 ---
[GitHub] carbondata issue #2197: [CARBONDATA-2371] Add Profiler output in EXPLAIN com...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2197 LGTM ---
[GitHub] carbondata pull request #2238: [HOTFIX][Streaming] Enhance streaming aggrega...
Github user asfgit closed the pull request at: https://github.com/apache/carbondata/pull/2238 ---
[GitHub] carbondata issue #2238: [HOTFIX][Streaming] Enhance streaming aggregate tabl...
Github user chenliang613 commented on the issue: https://github.com/apache/carbondata/pull/2238 LGTM ---
