Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/3065
@zzcclp Can you verify the presto with current master. There are changes
related to Hive metastore is done now. So now carbon behaves as a one of the
hive supported format in presto. Please
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/3068
LGTM
---
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/3068
> why does one segment have some blocklet encoded with local dictionary and
some without local dictionary ?
It is because carbon generates a dictionary based on the column va
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/3066
> i think the query performance of carbon_solution is lower than
hive_solution's, because carbon_solution has more segment (insert generates a
segment and update generates more segm
Github user ravipesala commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/3066#discussion_r247119761
--- Diff:
examples/spark2/src/main/scala/org/apache/carbondata/benchmark/CDCBenchmark.scala
---
@@ -0,0 +1,256 @@
+/*
+ * Licensed
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/3060
LGTM
---
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/3056
LGTM
---
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/3001
LGTM
---
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/3001
@QiangCai Please don't add binary files :( . you supposed to generate files
and execute the test
---
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/3055
LGTM
---
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/3029
LGTM
---
Github user ravipesala commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/3055#discussion_r246277431
--- Diff:
integration/presto/src/main/java/org/apache/carbondata/presto/readers/SliceStreamReader.java
---
@@ -142,5 +144,17 @@ public
Github user ravipesala commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/3055#discussion_r246277337
--- Diff:
integration/presto/src/main/java/org/apache/carbondata/presto/readers/SliceStreamReader.java
---
@@ -95,22 +105,14 @@ public
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/3001
@QiangCai Please try to add test case to it, otherwise it will be easy to
break in future commits.
---
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/3001
LGTM
---
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/2971
LGTM @QiangCai I feel it is better to keep in tableproprties as it is not
supposed changed for each load. We can further discuss and raise another PR if
needed, I am merging this now. Thanks
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/3024
LGTM
---
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/3051
LGTM
---
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/3014
LGTM
---
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/2971
@QiangCai we should restrict changing that property from table properties.
I am just explaining about how we can do the compaction on range column
since there are similarities
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/2971
@QiangCai My question how the user can benefit if he chooses a different
range column for each load. I feel range column should be at the table level
not at the load level.
And regarding
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/3039
LGTM
---
Github user ravipesala commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/3029#discussion_r244959393
--- Diff:
processing/src/main/java/org/apache/carbondata/processing/sort/sortdata/SingleThreadFinalSortFilesMerger.java
---
@@ -114,6 +113,31
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/3021
I will check on it
On Wed, 2 Jan 2019 at 10:37 PM, Chandrasekhar Saripaka <
notificati...@github.com> wrote:
> b0733ec
>
<https://github.com/apache/ca
Github user ravipesala commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/3041#discussion_r244692501
--- Diff:
integration/presto/src/main/java/org/apache/carbondata/presto/impl/CarbonTableReader.java
---
@@ -134,23 +127,17
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/2971
@QiangCai @jackylk Adding a `RANGE_COLUMN` at each load level does not
create an issue? If user selects different range column for each load how you
are going to compact when you support
Github user ravipesala commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2971#discussion_r244681963
--- Diff:
integration/spark-common/src/main/scala/org/apache/spark/DataSkewRangePartitioner.scala
---
@@ -0,0 +1,319 @@
+/*
+ * Licensed
Github user ravipesala commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2971#discussion_r244681720
--- Diff:
integration/spark-common/src/main/scala/org/apache/carbondata/spark/load/DataLoadProcessBuilderOnSpark.scala
---
@@ -156,4 +161,206
Github user ravipesala commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2971#discussion_r244681813
--- Diff:
integration/spark-common-test/src/test/scala/org/apache/carbondata/spark/testsuite/dataload/TestGlobalSortDataLoad.scala
---
@@ -106,6
Github user ravipesala commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2971#discussion_r244681164
--- Diff:
integration/spark-common/src/main/scala/org/apache/carbondata/spark/load/DataLoadProcessBuilderOnSpark.scala
---
@@ -156,4 +161,206
Github user ravipesala commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2971#discussion_r244681062
--- Diff:
integration/spark-common/src/main/scala/org/apache/carbondata/spark/load/DataLoadProcessorStepOnSpark.scala
---
@@ -95,6 +96,67 @@ object
Github user ravipesala commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2971#discussion_r244680876
--- Diff:
integration/spark-common/src/main/scala/org/apache/carbondata/spark/load/DataLoadProcessBuilderOnSpark.scala
---
@@ -156,4 +161,206
Github user ravipesala commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2971#discussion_r244670461
--- Diff:
integration/spark-common/src/main/scala/org/apache/carbondata/spark/load/DataLoadProcessBuilderOnSpark.scala
---
@@ -156,4 +161,206
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/3028
LGTM
---
Github user ravipesala commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/3041#discussion_r244665281
--- Diff:
integration/presto/src/main/java/org/apache/carbondata/presto/impl/CarbonTableReader.java
---
@@ -234,30 +219,26 @@ public TBase create
GitHub user ravipesala opened a pull request:
https://github.com/apache/carbondata/pull/3041
[WIP] Fix schema refresh and wrong query result issues in presto.
Problem:
Schema which is updated in spark is not reflecting in presto. which results
in wrong query result in presto
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/3029
@NamanRastogi Lots of code is duplicated here, Please try to unify with
other compactor processor to avoid the duplication.
---
Github user ravipesala commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/3029#discussion_r244319031
--- Diff:
processing/src/main/java/org/apache/carbondata/processing/merger/CarbonCompactionExecutor.java
---
@@ -126,17 +128,24 @@ public
Github user ravipesala commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/3019#discussion_r244278709
--- Diff:
integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/table/CarbonCreateTableCommand.scala
---
@@ -157,7 +157,7
Github user ravipesala commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/3019#discussion_r244278266
--- Diff:
core/src/main/java/org/apache/carbondata/core/datastore/impl/FileFactory.java
---
@@ -369,6 +369,24 @@ public static boolean
Github user ravipesala commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/3019#discussion_r244278116
--- Diff:
integration/presto/src/main/java/org/apache/carbondata/presto/CarbondataModule.java
---
@@ -17,62 +17,150 @@
package
Github user ravipesala commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/3019#discussion_r244276202
--- Diff:
integration/presto/src/main/java/org/apache/carbondata/presto/PrestoFilterUtil.java
---
@@ -78,32 +72,33 @@
private static final
Github user ravipesala commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/3019#discussion_r244276046
--- Diff:
integration/presto/src/test/scala/org/apache/carbondata/presto/util/CarbonDataStoreCreator.scala
---
@@ -80,7 +80,7 @@ object
Github user ravipesala commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/3019#discussion_r244097233
--- Diff:
examples/spark2/src/main/scala/org/apache/carbondata/examples/CarbonSessionExample.scala
---
@@ -72,69 +74,107 @@ object
Github user ravipesala commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/3019#discussion_r244097300
--- Diff:
integration/presto/src/main/java/org/apache/carbondata/presto/CarbondataConnectorFactory.java
---
@@ -17,69 +17,179 @@
package
Github user ravipesala commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/3019#discussion_r244097200
--- Diff:
integration/presto/src/main/java/org/apache/carbondata/presto/CarbondataModule.java
---
@@ -17,62 +17,150 @@
package
Github user ravipesala commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/3019#discussion_r244097091
--- Diff:
integration/presto/src/main/java/org/apache/carbondata/presto/CarbondataPageSourceProvider.java
---
@@ -113,7 +132,7 @@ private
Github user ravipesala commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/3019#discussion_r244097070
--- Diff:
integration/presto/src/main/java/org/apache/carbondata/presto/CarbondataPageSourceProvider.java
---
@@ -43,63 +44,81 @@
import
Github user ravipesala commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/3019#discussion_r244097115
--- Diff:
examples/spark2/src/main/scala/org/apache/carbondata/examples/CarbonSessionExample.scala
---
@@ -72,69 +74,107 @@ object
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/3024
@xubo245 can you more tests related to drop column, rename column , change
datatype of column also here.
---
Github user ravipesala commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/3001#discussion_r244095423
--- Diff:
integration/presto/src/test/scala/org/apache/carbondata/presto/integrationtest/PrestoAllDataTypeLocalDictTest.scala
---
@@ -38,48 +38,16
Github user ravipesala commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/3001#discussion_r244095320
--- Diff:
integration/presto/src/main/java/org/apache/carbondata/presto/stream/CarbondataStreamPageSource.java
---
@@ -0,0 +1,243
Github user ravipesala commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/3001#discussion_r244095288
--- Diff:
integration/presto/src/main/java/org/apache/carbondata/presto/readers/BooleanStreamReader.java
---
@@ -86,4 +86,15 @@ public
Github user ravipesala commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/3001#discussion_r244095118
--- Diff:
integration/presto/src/main/java/org/apache/carbondata/presto/CarbondataPageSourceProvider.java
---
@@ -79,13 +80,31 @@
@Override
Github user ravipesala commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/3001#discussion_r244093200
--- Diff:
integration/presto/src/main/java/org/apache/carbondata/presto/CarbonVectorBatch.java
---
@@ -95,6 +95,9 @@ private CarbonColumnVectorImpl
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/3026
@qiuchenjian Please check the PR description for why carbon need changes
for Spark 2.2 CDH
---
Github user ravipesala commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/3026#discussion_r244090720
--- Diff:
integration/spark-datasource/src/main/spark2.1andspark2.2/org/apache/spark/sql/CarbonDictionaryUtil.java
---
@@ -0,0 +1,116
Github user ravipesala commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/3004#discussion_r244086681
--- Diff:
integration/spark-common-test/src/test/scala/org/apache/carbondata/spark/testsuite/createTable/TestCreateHiveTableWithCarbonDS.scala
Github user ravipesala commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/3004#discussion_r244086693
--- Diff:
integration/spark-common-test/src/test/scala/org/apache/carbondata/spark/testsuite/createTable/TestCreateHiveTableWithCarbonDS.scala
Github user ravipesala commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/3004#discussion_r244086704
--- Diff:
integration/spark-common-test/src/test/scala/org/apache/carbondata/spark/testsuite/createTable/TestCreateHiveTableWithCarbonDS.scala
Github user ravipesala commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/3004#discussion_r244086686
--- Diff:
integration/spark-common-test/src/test/scala/org/apache/carbondata/spark/testsuite/createTable/TestCreateHiveTableWithCarbonDS.scala
Github user ravipesala commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/3004#discussion_r244086671
--- Diff:
integration/spark-common-test/src/test/scala/org/apache/carbondata/spark/testsuite/createTable/TestCreateHiveTableWithCarbonDS.scala
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/3025
@qiuchenjian currently pre-aggregate datamap is not following the datamap
interfaces as it was implemented before datamap framework. That is why not all
datamap DDL works with pre-aggregate
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/3019
@chenliang613 if we are planning to contribute to Presto then it would be
under presto-hive connector, it is just like how ORC, parquet and other formats
supported under this connector
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/3021
@chandrasaripaka Please check the PR
https://github.com/apache/carbondata/pull/3026
---
GitHub user ravipesala opened a pull request:
https://github.com/apache/carbondata/pull/3026
[WIP] Added support to compile carbon CDH spark distribution
Please use `spark-2.2-cdh` profile to compile cdh.
example:
```
mvn -DskipTests -Pspark-2.2-cdh package
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/3021
@chandrasaripaka , I got the issue, but creating many duplicate files may
not be a good idea as it will be difficult to maintain, I will try to do with
reflection.
And one more question
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/3021
I went through this link earlier, but I cannot find spark 2.2 version in
this distribution. I can find only `1.6.0-cdh5.14.4` of spark here.
---
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/3021
@chandrasaripaka I can't find the spark maven dependency for CDH5.14.2, But
I am able to build with CDH spark versions `2.2.0-cdh6.0.1` and
`2.2.0.cloudera3`. Only the problem here I found
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/2897
LGTM
---
Github user ravipesala commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2897#discussion_r243810091
--- Diff:
hadoop/src/main/java/org/apache/carbondata/hadoop/util/CarbonVectorizedRecordReader.java
---
@@ -68,6 +67,10 @@
private
Github user ravipesala commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2897#discussion_r243810009
--- Diff:
core/src/main/java/org/apache/carbondata/core/memory/UnsafeMemoryManager.java
---
@@ -173,6 +174,7 @@ public synchronized void
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/3004
@SteNicholas This PR does not work directly on the Hive integration as we
need to set the right serde, inputformat and output format. We are planning to
refactor the current Hive integration
GitHub user ravipesala opened a pull request:
https://github.com/apache/carbondata/pull/3019
[WIP] Carbon Presto hive metastore
Be sure to do all of the following checklist to help us incorporate
your contribution quickly and easily:
- [ ] Any interfaces changed
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/2998
LGTM
---
Github user ravipesala commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2998#discussion_r243530847
--- Diff:
integration/spark2/src/main/scala/org/apache/spark/sql/CarbonSource.scala ---
@@ -331,7 +333,14 @@ object CarbonSource
Github user ravipesala commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2998#discussion_r243489099
--- Diff:
integration/spark2/src/main/scala/org/apache/spark/sql/CarbonSource.scala ---
@@ -331,7 +333,17 @@ object CarbonSource
GitHub user ravipesala opened a pull request:
https://github.com/apache/carbondata/pull/3004
[WIP] Create carbon table as hive understandable metastore table needed by
Presto and Hive
Problem:
Current carbon table created in spark creates the hive table internally but
it does
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/2966
LGTM
---
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/2995
LGTM
---
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/2985
LGTM
---
Github user ravipesala commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2966#discussion_r242417861
--- Diff:
core/src/main/java/org/apache/carbondata/core/util/DataTypeUtil.java ---
@@ -959,6 +959,14 @@ public static void
setDataTypeConverter
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/2949
LGTM
---
Github user ravipesala commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2161#discussion_r241982006
--- Diff:
core/src/main/java/org/apache/carbondata/core/datastore/filesystem/AlluxioCarbonFile.java
---
@@ -94,14 +93,9 @@ public CarbonFile
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/2161
@chandrasaripaka you can put the CarbonFile test with Alluxio Mini Cluster
but make sure it does not go beyond a few seconds to finish the test as it
impacts the build time
---
Github user ravipesala commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2161#discussion_r241981844
--- Diff:
core/src/main/java/org/apache/carbondata/core/locks/AlluxioFileLock.java ---
@@ -0,0 +1,112 @@
+/*
+ * Licensed to the Apache
Github user ravipesala commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2161#discussion_r241977467
--- Diff:
core/src/main/java/org/apache/carbondata/core/locks/AlluxioFileLock.java ---
@@ -0,0 +1,112 @@
+/*
+ * Licensed to the Apache
Github user ravipesala commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2161#discussion_r241977202
--- Diff:
core/src/main/java/org/apache/carbondata/core/datastore/impl/FileFactory.java
---
@@ -43,7 +44,7 @@
* LOGGER
Github user ravipesala commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2161#discussion_r241977147
--- Diff:
core/src/main/java/org/apache/carbondata/core/datastore/filesystem/AlluxioCarbonFile.java
---
@@ -94,14 +93,9 @@ public CarbonFile
Github user ravipesala commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2161#discussion_r241976975
--- Diff:
core/src/main/java/org/apache/carbondata/core/datastore/filesystem/AbstractDFSCarbonFile.java
---
@@ -550,12 +550,10 @@ public
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/2979
LGTM
---
Github user ravipesala commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2966#discussion_r241437965
--- Diff:
store/sdk/src/main/java/org/apache/carbondata/sdk/file/CarbonWriterBuilder.java
---
@@ -501,6 +502,17 @@ public CarbonLoadModel
Github user ravipesala commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2966#discussion_r241437223
--- Diff:
integration/spark2/src/test/scala/org/apache/spark/carbondata/restructure/AlterTableValidationTestCase.scala
---
@@ -523,7 +523,8
Github user ravipesala commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2966#discussion_r241436353
--- Diff:
integration/spark-common/src/main/scala/org/apache/spark/sql/execution/command/carbonTableSchemaCommon.scala
---
@@ -848,6 +848,19
Github user ravipesala commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2966#discussion_r241433863
--- Diff:
core/src/main/java/org/apache/carbondata/core/util/DataTypeUtil.java ---
@@ -61,7 +61,35 @@
}
};
- private
Github user ravipesala commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2966#discussion_r241432600
--- Diff:
core/src/main/java/org/apache/carbondata/core/scan/filter/FilterUtil.java ---
@@ -2265,7 +2265,8 @@ public static int compareValues(byte
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/2979
retest this please
---
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/2979
LGTM
---
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/2976
retest this please
---
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/2982
LGTM , just a minor comment.
---
1 - 100 of 8676 matches
Mail list logo
