[GitHub] incubator-carbondata pull request #412: [WIP]Added vector reader in Carbon s...
Github user ravipesala commented on a diff in the pull request:
https://github.com/apache/incubator-carbondata/pull/412#discussion_r91835389
--- Diff:
hadoop/src/main/java/org/apache/carbondata/hadoop/CarbonRecordReader.java ---
@@ -78,7 +80,13 @@ public void initialize(InputSplit inputSplit,
TaskAttemptContext context)
readSupport.initialize(queryModel.getProjectionColumns(),
queryModel.getAbsoluteTableIdentifier());
try {
- carbonIterator = new
ChunkRowIterator(queryExecutor.execute(queryModel));
+ if (queryModel.isVectorReader()) {
+carbonIterator = new VectorChunkRowIterator(
--- End diff --
I moved out this logic out of the class. And new
`VectorizedCarbonRecordReader` is created.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-carbondata pull request #412: [WIP]Added vector reader in Carbon s...
Github user ravipesala commented on a diff in the pull request: https://github.com/apache/incubator-carbondata/pull/412#discussion_r91835397 --- Diff: core/src/main/java/org/apache/carbondata/scan/result/vector/CarbonColumnarBatch.java --- @@ -0,0 +1,45 @@ +package org.apache.carbondata.scan.result.vector; --- End diff -- ok --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-carbondata pull request #412: [WIP]Added vector reader in Carbon s...
Github user ravipesala commented on a diff in the pull request: https://github.com/apache/incubator-carbondata/pull/412#discussion_r91835427 --- Diff: core/src/main/java/org/apache/carbondata/scan/result/vector/CarbonColumnVector.java --- @@ -0,0 +1,29 @@ +package org.apache.carbondata.scan.result.vector; --- End diff -- ok --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-carbondata pull request #412: [WIP]Added vector reader in Carbon s...
Github user ravipesala commented on a diff in the pull request:
https://github.com/apache/incubator-carbondata/pull/412#discussion_r91835416
--- Diff:
core/src/main/java/org/apache/carbondata/scan/result/vector/CarbonColumnarBatch.java
---
@@ -0,0 +1,45 @@
+package org.apache.carbondata.scan.result.vector;
+
+public class CarbonColumnarBatch {
--- End diff --
In spark the name is aready `ColumnarBatch` so I just used this name to
avoid confusion
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-carbondata issue #412: [WIP]Added vector reader in Carbon scan.
Github user CarbonDataQA commented on the issue: https://github.com/apache/incubator-carbondata/pull/412 Build Success with Spark 1.5.2, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/86/ --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-carbondata issue #412: [WIP]Added vector reader in Carbon scan.
Github user CarbonDataQA commented on the issue: https://github.com/apache/incubator-carbondata/pull/412 Build Success with Spark 1.5.2, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/87/ --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Assigned] (CARBONDATA-2) Remove kettle for loading data
[ https://issues.apache.org/jira/browse/CARBONDATA-2?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ravindra Pesala reassigned CARBONDATA-2: Assignee: Ravindra Pesala > Remove kettle for loading data > -- > > Key: CARBONDATA-2 > URL: https://issues.apache.org/jira/browse/CARBONDATA-2 > Project: CarbonData > Issue Type: Improvement > Components: data-load >Reporter: Liang Chen >Assignee: Ravindra Pesala >Priority: Critical > Labels: features > Fix For: 1.0.0-incubating > > Attachments: CarbonDataLoadingdesign.pdf > > Time Spent: 5h 10m > Remaining Estimate: 0h > > Remove kettle for loading data module -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (CARBONDATA-519) Enable vector reader in Carbon-Spark 2.0 integration and Carbon layer
Ravindra Pesala created CARBONDATA-519: -- Summary: Enable vector reader in Carbon-Spark 2.0 integration and Carbon layer Key: CARBONDATA-519 URL: https://issues.apache.org/jira/browse/CARBONDATA-519 Project: CarbonData Issue Type: Improvement Reporter: Ravindra Pesala Assignee: Ravindra Pesala Spark 2.0 supports batch reader and uses whole codegen to improve performance, so carbon also can implement vector reader and leverage the features of Spark2.0 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] incubator-carbondata pull request #412: [CARBONDATA-519]Added vector reader ...
Github user ravipesala commented on a diff in the pull request:
https://github.com/apache/incubator-carbondata/pull/412#discussion_r91839108
--- Diff:
core/src/main/java/org/apache/carbondata/scan/result/vector/CarbonColumnVector.java
---
@@ -0,0 +1,29 @@
+package org.apache.carbondata.scan.result.vector;
+
+import org.apache.spark.sql.types.Decimal;
+
+public interface CarbonColumnVector {
+
+ public void putShort(int rowId, short value);
+
+ public void putInt(int rowId, int value);
+
+ public void putLong(int rowId, long value);
+
+ public void putDecimal(int rowId, Decimal value, int precision);
+
+ public void putDouble(int rowId, double value);
+
+ public void putBytes(int rowId, byte[] value);
+
+ public void putBytes(int rowId, int offset, int length, byte[] value);
--- End diff --
`ColumnarVectorWrapper` is the implementation class for this interface
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-carbondata issue #415: Fix integrate with spark 2.1
Github user CarbonDataQA commented on the issue: https://github.com/apache/incubator-carbondata/pull/415 Build Success with Spark 1.5.2, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/88/ --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-carbondata issue #415: Depends on more stable class of spark in sp...
Github user CarbonDataQA commented on the issue: https://github.com/apache/incubator-carbondata/pull/415 Build Success with Spark 1.5.2, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/89/ --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-carbondata pull request #417: fix conf issue for scanrdd
GitHub user scwf opened a pull request: https://github.com/apache/incubator-carbondata/pull/417 fix conf issue for scanrdd we should sync the conf in executor side You can merge this pull request into a Git repository by running: $ git pull https://github.com/KirinKing/incubator-carbondata fix-scan Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-carbondata/pull/417.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #417 commit 1c54cd4ef942be0f5e2eef40c34db98a2233a0dc Author: wangfei Date: 2016-12-10T23:24:03Z fix conf issue for scanrdd --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-carbondata issue #417: fix conf issue for scanrdd
Github user scwf commented on the issue: https://github.com/apache/incubator-carbondata/pull/417 /cc @QiangCai @jackylk plz review this --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-carbondata issue #417: fix conf issue for scanrdd
Github user CarbonDataQA commented on the issue: https://github.com/apache/incubator-carbondata/pull/417 Build Failed with Spark 1.5.2, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/90/ --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-carbondata issue #417: fix conf issue for scanrdd
Github user CarbonDataQA commented on the issue: https://github.com/apache/incubator-carbondata/pull/417 Build Failed with Spark 1.5.2, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/91/ --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-carbondata issue #417: fix conf issue for scanrdd
Github user CarbonDataQA commented on the issue: https://github.com/apache/incubator-carbondata/pull/417 Build Failed with Spark 1.5.2, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/92/ --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-carbondata issue #417: fix conf issue for scanrdd
Github user CarbonDataQA commented on the issue: https://github.com/apache/incubator-carbondata/pull/417 Build Success with Spark 1.5.2, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/93/ --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-carbondata issue #415: Depends on more stable class of spark in sp...
Github user CarbonDataQA commented on the issue: https://github.com/apache/incubator-carbondata/pull/415 Build Success with Spark 1.5.2, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/94/ --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-carbondata issue #415: Depends on more stable class of spark in sp...
Github user CarbonDataQA commented on the issue: https://github.com/apache/incubator-carbondata/pull/415 Build Success with Spark 1.5.2, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/95/ --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-carbondata pull request #416: [CARBONDATA-518]Fix the bug of Carbo...
Github user chenliang613 commented on a diff in the pull request:
https://github.com/apache/incubator-carbondata/pull/416#discussion_r91846766
--- Diff:
examples/spark/src/main/scala/org/apache/carbondata/examples/CarbonExample.scala
---
@@ -26,11 +28,15 @@ object CarbonExample {
def main(args: Array[String]) {
val cc = ExampleUtils.createCarbonContext("CarbonExample")
val testData = ExampleUtils.currentPath +
"/src/main/resources/data.csv"
-
+val rootPath = new File(this.getClass.getResource("/").getPath
--- End diff --
These parameters have been defined in ExampleUtils.scala as below, and
also already been added into CarbonProperties during creation CarbonContext.
def currentPath: String = new File(this.getClass.getResource("/").getPath +
"../../")
.getCanonicalPath
val storeLocation = currentPath + "/target/store"
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-carbondata issue #249: [CARBONDATA-329] constant final class chang...
Github user chenliang613 commented on the issue: https://github.com/apache/incubator-carbondata/pull/249 Can you rebase this PR --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-carbondata issue #200: [CARBONDATA-276]add trim property for cols
Github user chenliang613 commented on the issue: https://github.com/apache/incubator-carbondata/pull/200 @lion-x please rebase it. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-carbondata issue #120: [CARBONDATA-172]Pushdown filters and broadc...
Github user chenliang613 commented on the issue: https://github.com/apache/incubator-carbondata/pull/120 @ashokblend please rebase this PR --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-carbondata issue #415: Depends on more stable class of spark in sp...
Github user jackylk commented on the issue: https://github.com/apache/incubator-carbondata/pull/415 Add to whitelist --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-carbondata issue #417: fix conf issue for scanrdd
Github user jackylk commented on the issue: https://github.com/apache/incubator-carbondata/pull/417 Add to whitelist --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-carbondata issue #417: fix conf issue for scanrdd
Github user CarbonDataQA commented on the issue: https://github.com/apache/incubator-carbondata/pull/417 Build Success with Spark 1.5.2, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/96/ --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-carbondata issue #368: [CARBONDATA-465] Spark streaming dataframe ...
Github user ravipesala commented on the issue: https://github.com/apache/incubator-carbondata/pull/368 LGTM, Good work. But few comments like reflection is not required(can set context from initialize from RDD ) are not addressed, we can fix them in another PR. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-carbondata pull request #368: [CARBONDATA-465] Spark streaming dat...
Github user asfgit closed the pull request at: https://github.com/apache/incubator-carbondata/pull/368 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Resolved] (CARBONDATA-465) Spark streaming dataframe support
[ https://issues.apache.org/jira/browse/CARBONDATA-465?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ravindra Pesala resolved CARBONDATA-465. Resolution: Fixed > Spark streaming dataframe support > - > > Key: CARBONDATA-465 > URL: https://issues.apache.org/jira/browse/CARBONDATA-465 > Project: CarbonData > Issue Type: Improvement > Components: data-load >Affects Versions: 1.0.0-incubating >Reporter: WilliamZhu >Assignee: WilliamZhu >Priority: Minor > Fix For: 1.0.0-incubating > > Time Spent: 5.5h > Remaining Estimate: 0h > > Carbondata-1.0.0 support load data with spark data frame api. There is a > limit that kettle is still required since DataFrameLoaderRDD still depends on > kettle. We provide NewDataFrameLoaderRDD to load data with new flow . > Also,we discovered some bugs: > 1. CarbonMetastoreCatalog.createTableFromThrift > ``` > /** > * schemaFilePath starts with file:// will not create meta files > successfully > * while thriftWriter will have no complains. > * This will cause some weired error eg. No table found. > */ > val thriftWriter = new ThriftWriter(schemaFilePath, false) > thriftWriter.open() > thriftWriter.write(thriftTableInfo) > thriftWriter.close() > ``` > 2. There are some exceptions raised even when you have set useKettle to > false. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] incubator-carbondata issue #348: [CARBONDATA-445] improved integration test-...
Github user ravipesala commented on the issue: https://github.com/apache/incubator-carbondata/pull/348 LGTM --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-carbondata pull request #348: [CARBONDATA-445] improved integratio...
Github user asfgit closed the pull request at: https://github.com/apache/incubator-carbondata/pull/348 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Resolved] (CARBONDATA-445) Improved integration test-case for AllDataTypesTestCase3
[ https://issues.apache.org/jira/browse/CARBONDATA-445?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ravindra Pesala resolved CARBONDATA-445. Resolution: Fixed Fix Version/s: 1.0.0-incubating > Improved integration test-case for AllDataTypesTestCase3 > - > > Key: CARBONDATA-445 > URL: https://issues.apache.org/jira/browse/CARBONDATA-445 > Project: CarbonData > Issue Type: Task >Reporter: SWATI RAO > Fix For: 1.0.0-incubating > > Time Spent: 50m > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] incubator-carbondata issue #394: [CARBONDATA-482 ]AllDataTypeTestCase6 impro...
Github user ravipesala commented on the issue: https://github.com/apache/incubator-carbondata/pull/394 LGTM --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-carbondata pull request #394: [CARBONDATA-482 ]AllDataTypeTestCase...
Github user asfgit closed the pull request at: https://github.com/apache/incubator-carbondata/pull/394 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Resolved] (CARBONDATA-482) improve integration test case of AllDataTypesTestCase6
[ https://issues.apache.org/jira/browse/CARBONDATA-482?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ravindra Pesala resolved CARBONDATA-482. Resolution: Fixed Assignee: Rahul Kumar Fix Version/s: 1.0.0-incubating > improve integration test case of AllDataTypesTestCase6 > -- > > Key: CARBONDATA-482 > URL: https://issues.apache.org/jira/browse/CARBONDATA-482 > Project: CarbonData > Issue Type: Test >Reporter: Rahul Kumar >Assignee: Rahul Kumar > Fix For: 1.0.0-incubating > > Time Spent: 50m > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] incubator-carbondata issue #331: [CARBONDATA-396] Implement test cases for d...
Github user ravipesala commented on the issue: https://github.com/apache/incubator-carbondata/pull/331 LGTM --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-carbondata issue #331: [CARBONDATA-396] Implement test cases for d...
Github user ravipesala commented on the issue: https://github.com/apache/incubator-carbondata/pull/331 @harmeetsingh0013 It seems I could not rebase it because of conflicts in some of your commit. Please squash all commits to 1 commit. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-carbondata pull request #415: Depends on more stable class of spar...
Github user jackylk commented on a diff in the pull request: https://github.com/apache/incubator-carbondata/pull/415#discussion_r91847495 --- Diff: integration/spark2/src/main/scala/org/apache/spark/sql/CarbonDictionaryDecoder.scala --- @@ -223,10 +223,11 @@ case class CarbonDictionaryDecoder( -class CarbonDecoderRDD(relations: Seq[CarbonDecoderRelation], -profile: CarbonProfile, -aliasMap: CarbonAliasDecoderRelation, -prev: RDD[Row], +class CarbonDecoderRDD( +relations: Seq[CarbonDecoderRelation], +profile: CarbonProfile, +aliasMap: CarbonAliasDecoderRelation, +prev: RDD[Row], output: Seq[Attribute]) --- End diff -- incorrect indentation --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-carbondata pull request #415: Depends on more stable class of spar...
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/incubator-carbondata/pull/415#discussion_r91847501
--- Diff:
integration/spark2/src/main/scala/org/apache/spark/sql/execution/CarbonLateDecodeStrategy.scala
---
@@ -101,12 +103,11 @@ private[sql] class CarbonLateDecodeStrategy extends
SparkStrategy {
}
protected def pruneFilterProject(
-relation: LogicalRelation,
-projects: Seq[NamedExpression],
-filterPredicates: Seq[Expression],
-scanBuilder: (Seq[Attribute],
Array[Filter],
-ArrayBuffer[AttributeReference]) =>
-RDD[InternalRow]) = {
+ relation: LogicalRelation,
+ projects: Seq[NamedExpression],
+ filterPredicates: Seq[Expression],
+ scanBuilder: (Seq[Attribute], Array[Filter],
+ ArrayBuffer[AttributeReference]) => RDD[InternalRow]) = {
--- End diff --
incorrect indentation
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-carbondata issue #343: [Carbondata 395] Test cases added for Scan ...
Github user ravipesala commented on the issue: https://github.com/apache/incubator-carbondata/pull/343 @deepakmehra10 Please rename the title from [Carbondata 395] to [CARBONDATA-395] --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-carbondata issue #364: [CarbonData-460]Added Unit Tests For core.w...
Github user ravipesala commented on the issue: https://github.com/apache/incubator-carbondata/pull/364 add to whitelist --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-carbondata issue #364: [CarbonData-460]Added Unit Tests For core.w...
Github user ravipesala commented on the issue: https://github.com/apache/incubator-carbondata/pull/364 @geetikagupta16 Please rename title from [CarbonData-460] to [CARBONDATA-460] --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-carbondata pull request #415: Depends on more stable class of spar...
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/incubator-carbondata/pull/415#discussion_r91847522
--- Diff:
integration/spark2/src/main/scala/org/apache/spark/sql/execution/CarbonLateDecodeStrategy.scala
---
@@ -212,20 +212,20 @@ private[sql] class CarbonLateDecodeStrategy extends
SparkStrategy {
}
attr
}
- val scan = execution.DataSourceScanExec.create(
+ val scan = new execution.RowDataSourceScanExec(
updateProject,
scanBuilder(updateRequestedColumns, candidatePredicates,
pushedFilters, needDecoder),
-relation.relation, metadata, relation.metastoreTableIdentifier)
+relation.relation, UnknownPartitioning(9), metadata, None)
filterCondition.map(execution.FilterExec(_, scan)).getOrElse(scan)
} else {
// Don't request columns that are only referenced by pushed filters.
val requestedColumns =
(projectSet ++ filterSet --
handledSet).map(relation.attributeMap).toSeq
val updateRequestedColumns =
updateRequestedColumnsFunc(requestedColumns, table, needDecoder)
- val scan = execution.DataSourceScanExec.create(
+ val scan = new execution.RowDataSourceScanExec(
updateRequestedColumns,
scanBuilder(updateRequestedColumns, candidatePredicates,
pushedFilters, needDecoder),
-relation.relation, metadata, relation.metastoreTableIdentifier)
+relation.relation, UnknownPartitioning(0), metadata, None)
--- End diff --
give comment for 0
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-carbondata issue #397: [Carbondata -474] Added test Cases for Data...
Github user ravipesala commented on the issue: https://github.com/apache/incubator-carbondata/pull/397 @anubhav100 Please rename the title [Carbondata -474] to [CARBONDATA-474] --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-carbondata pull request #415: Depends on more stable class of spar...
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/incubator-carbondata/pull/415#discussion_r91847518
--- Diff:
integration/spark2/src/main/scala/org/apache/spark/sql/execution/CarbonLateDecodeStrategy.scala
---
@@ -212,20 +212,20 @@ private[sql] class CarbonLateDecodeStrategy extends
SparkStrategy {
}
attr
}
- val scan = execution.DataSourceScanExec.create(
+ val scan = new execution.RowDataSourceScanExec(
updateProject,
scanBuilder(updateRequestedColumns, candidatePredicates,
pushedFilters, needDecoder),
-relation.relation, metadata, relation.metastoreTableIdentifier)
+relation.relation, UnknownPartitioning(9), metadata, None)
--- End diff --
why is it 9? give comment for this.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-carbondata issue #374: [Carbondata 475] Added test cases for org.a...
Github user ravipesala commented on the issue: https://github.com/apache/incubator-carbondata/pull/374 @anubhav100 Please rename PR title from [Carbondata 475] to [CARBONDATA-475] --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-carbondata pull request #417: fix conf issue for scanrdd
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/incubator-carbondata/pull/417#discussion_r91847537
--- Diff:
integration/spark2/src/main/scala/org/apache/spark/sql/CarbonEnv.scala ---
@@ -37,6 +36,9 @@ object CarbonEnv {
@volatile private var carbonEnv: CarbonEnv = _
+ // set readsupport class global so that the executor can get it.
+ SparkReadSupport.readSupportClass = classOf[SparkRowReadSupportImpl]
--- End diff --
move it to init function
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-carbondata pull request #417: fix conf issue for scanrdd
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/incubator-carbondata/pull/417#discussion_r91847539
--- Diff:
integration/spark/src/main/scala/org/apache/spark/sql/CarbonEnv.scala ---
@@ -21,22 +21,24 @@ import
org.apache.spark.scheduler.cluster.CoarseGrainedSchedulerBackend
import org.apache.spark.sql.hive.CarbonMetastore
import org.apache.carbondata.hadoop.readsupport.impl.RawDataReadSupport
-import org.apache.carbondata.spark.rdd.SparkCommonEnv
+import org.apache.carbondata.spark.rdd.SparkReadSupport
case class CarbonEnv(carbonMetastore: CarbonMetastore)
object CarbonEnv {
@volatile private var carbonEnv: CarbonEnv = _
+ // set readsupport class global so that the executor can get it.
+ SparkReadSupport.readSupportClass = classOf[RawDataReadSupport]
--- End diff --
move it to init function
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-carbondata issue #342: [CARBONDATA-438] Added unit test for Filter...
Github user ravipesala commented on the issue: https://github.com/apache/incubator-carbondata/pull/342 retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-carbondata issue #364: [CarbonData-460]Added Unit Tests For core.w...
Github user CarbonDataQA commented on the issue: https://github.com/apache/incubator-carbondata/pull/364 Build Success with Spark 1.5.2, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/97/ --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-carbondata issue #342: [CARBONDATA-438] Added unit test for Filter...
Github user CarbonDataQA commented on the issue: https://github.com/apache/incubator-carbondata/pull/342 Build Failed with Spark 1.5.2, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/98/ --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-carbondata pull request #79: CARBONDATA-163 Tool to merge Github P...
Github user asfgit closed the pull request at: https://github.com/apache/incubator-carbondata/pull/79 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Resolved] (CARBONDATA-163) Tool to merge Github Pull Requests
[ https://issues.apache.org/jira/browse/CARBONDATA-163?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ravindra Pesala resolved CARBONDATA-163. Resolution: Fixed Fix Version/s: 1.0.0-incubating > Tool to merge Github Pull Requests > -- > > Key: CARBONDATA-163 > URL: https://issues.apache.org/jira/browse/CARBONDATA-163 > Project: CarbonData > Issue Type: New Feature >Reporter: Ajay Yadava > Fix For: 1.0.0-incubating > > Time Spent: 20m > Remaining Estimate: 0h > > As per [discussion on the mailing > list|https://www.mail-archive.com/dev@carbondata.incubator.apache.org/msg00608.html], > this JIRA is for creating a tool to help merge pull requests. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] incubator-carbondata pull request #416: [CARBONDATA-518]Fix the bug of Carbo...
Github user Zhangshunyu commented on a diff in the pull request:
https://github.com/apache/incubator-carbondata/pull/416#discussion_r91847903
--- Diff:
examples/spark/src/main/scala/org/apache/carbondata/examples/CarbonExample.scala
---
@@ -26,11 +28,15 @@ object CarbonExample {
def main(args: Array[String]) {
val cc = ExampleUtils.createCarbonContext("CarbonExample")
val testData = ExampleUtils.currentPath +
"/src/main/resources/data.csv"
-
+val rootPath = new File(this.getClass.getResource("/").getPath
--- End diff --
@chenliang613 Hi liang, now the path is not get from carboncontext, they
are got from carbonproperties, the code has been changed by this commit:
https://github.com/apache/incubator-carbondata/commit/f67ec0125247e98d7925866fd3062ee3ddf16e28https://github.com/apache/incubator-carbondata/commit/f67ec0125247e98d7925866fd3062ee3ddf16e28
And you can try carbon example, its not run now.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-carbondata pull request #417: fix conf issue for scanrdd
Github user scwf commented on a diff in the pull request:
https://github.com/apache/incubator-carbondata/pull/417#discussion_r91848571
--- Diff:
integration/spark2/src/main/scala/org/apache/spark/sql/CarbonEnv.scala ---
@@ -37,6 +36,9 @@ object CarbonEnv {
@volatile private var carbonEnv: CarbonEnv = _
+ // set readsupport class global so that the executor can get it.
+ SparkReadSupport.readSupportClass = classOf[SparkRowReadSupportImpl]
--- End diff --
also the same
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-carbondata pull request #417: fix conf issue for scanrdd
Github user scwf commented on a diff in the pull request:
https://github.com/apache/incubator-carbondata/pull/417#discussion_r91848570
--- Diff:
integration/spark/src/main/scala/org/apache/spark/sql/CarbonEnv.scala ---
@@ -21,22 +21,24 @@ import
org.apache.spark.scheduler.cluster.CoarseGrainedSchedulerBackend
import org.apache.spark.sql.hive.CarbonMetastore
import org.apache.carbondata.hadoop.readsupport.impl.RawDataReadSupport
-import org.apache.carbondata.spark.rdd.SparkCommonEnv
+import org.apache.carbondata.spark.rdd.SparkReadSupport
case class CarbonEnv(carbonMetastore: CarbonMetastore)
object CarbonEnv {
@volatile private var carbonEnv: CarbonEnv = _
+ // set readsupport class global so that the executor can get it.
+ SparkReadSupport.readSupportClass = classOf[RawDataReadSupport]
--- End diff --
No, this is the key point, init is called in driver, see the comment, place
here is to let executor get the read support class
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[jira] [Created] (CARBONDATA-520) Executor can not get the read support class
Fei Wang created CARBONDATA-520: --- Summary: Executor can not get the read support class Key: CARBONDATA-520 URL: https://issues.apache.org/jira/browse/CARBONDATA-520 Project: CarbonData Issue Type: Sub-task Components: spark-integration Reporter: Fei Wang Assignee: Fei Wang Executor can not get the read support class, this leads to cast exception when running carbon on spark2 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (CARBONDATA-521) Depends on more stable class of spark in spark2
Fei Wang created CARBONDATA-521: --- Summary: Depends on more stable class of spark in spark2 Key: CARBONDATA-521 URL: https://issues.apache.org/jira/browse/CARBONDATA-521 Project: CarbonData Issue Type: Sub-task Reporter: Fei Wang avoid to use unstable class in spark2, otherwise it leads to compatible issue with spark -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CARBONDATA-521) Depends on more stable class of spark in spark2
[ https://issues.apache.org/jira/browse/CARBONDATA-521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15739172#comment-15739172 ] Fei Wang commented on CARBONDATA-521: - plz not close this, until we refectory all the dependency with spark > Depends on more stable class of spark in spark2 > --- > > Key: CARBONDATA-521 > URL: https://issues.apache.org/jira/browse/CARBONDATA-521 > Project: CarbonData > Issue Type: Sub-task > Components: spark-integration >Reporter: Fei Wang > Fix For: 1.0.0-incubating > > > avoid to use unstable class in spark2, otherwise it leads to compatible issue > with spark -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] incubator-carbondata issue #415: [CARBONDATA-521]Depends on more stable clas...
Github user CarbonDataQA commented on the issue: https://github.com/apache/incubator-carbondata/pull/415 Build Success with Spark 1.5.2, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/99/ --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-carbondata pull request #61: Rename packages, add legal file, and ...
Github user jbonofre closed the pull request at: https://github.com/apache/incubator-carbondata/pull/61 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-carbondata pull request #415: [CARBONDATA-521]Depends on more stab...
Github user asfgit closed the pull request at: https://github.com/apache/incubator-carbondata/pull/415 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-carbondata pull request #417: [CARBONDATA-520] Executor can not ge...
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/incubator-carbondata/pull/417#discussion_r91849094
--- Diff:
integration/spark/src/main/scala/org/apache/spark/sql/CarbonEnv.scala ---
@@ -21,22 +21,24 @@ import
org.apache.spark.scheduler.cluster.CoarseGrainedSchedulerBackend
import org.apache.spark.sql.hive.CarbonMetastore
import org.apache.carbondata.hadoop.readsupport.impl.RawDataReadSupport
-import org.apache.carbondata.spark.rdd.SparkCommonEnv
+import org.apache.carbondata.spark.rdd.SparkReadSupport
case class CarbonEnv(carbonMetastore: CarbonMetastore)
object CarbonEnv {
@volatile private var carbonEnv: CarbonEnv = _
+ // set readsupport class global so that the executor can get it.
+ SparkReadSupport.readSupportClass = classOf[RawDataReadSupport]
--- End diff --
ok
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-carbondata pull request #417: [CARBONDATA-520] Executor can not ge...
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/incubator-carbondata/pull/417#discussion_r91849096
--- Diff:
integration/spark2/src/main/scala/org/apache/spark/sql/CarbonEnv.scala ---
@@ -37,6 +36,9 @@ object CarbonEnv {
@volatile private var carbonEnv: CarbonEnv = _
+ // set readsupport class global so that the executor can get it.
+ SparkReadSupport.readSupportClass = classOf[SparkRowReadSupportImpl]
--- End diff --
ok
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-carbondata pull request #417: [CARBONDATA-520] Executor can not ge...
Github user asfgit closed the pull request at: https://github.com/apache/incubator-carbondata/pull/417 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Resolved] (CARBONDATA-520) Executor can not get the read support class
[ https://issues.apache.org/jira/browse/CARBONDATA-520?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jacky Li resolved CARBONDATA-520. - Resolution: Fixed > Executor can not get the read support class > > > Key: CARBONDATA-520 > URL: https://issues.apache.org/jira/browse/CARBONDATA-520 > Project: CarbonData > Issue Type: Sub-task > Components: spark-integration >Reporter: Fei Wang >Assignee: Fei Wang > Fix For: 1.0.0-incubating > > Time Spent: 0.5h > Remaining Estimate: 0h > > Executor can not get the read support class, this leads to cast exception > when running carbon on spark2 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] incubator-carbondata issue #412: [CARBONDATA-519]Added vector reader in Carb...
Github user jackylk commented on the issue: https://github.com/apache/incubator-carbondata/pull/412 please rebase --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Resolved] (CARBONDATA-521) Depends on more stable class of spark in spark2
[ https://issues.apache.org/jira/browse/CARBONDATA-521?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jacky Li resolved CARBONDATA-521. - Resolution: Fixed Assignee: Fei Wang > Depends on more stable class of spark in spark2 > --- > > Key: CARBONDATA-521 > URL: https://issues.apache.org/jira/browse/CARBONDATA-521 > Project: CarbonData > Issue Type: Sub-task > Components: spark-integration >Reporter: Fei Wang >Assignee: Fei Wang > Fix For: 1.0.0-incubating > > Time Spent: 20m > Remaining Estimate: 0h > > avoid to use unstable class in spark2, otherwise it leads to compatible issue > with spark -- This message was sent by Atlassian JIRA (v6.3.4#6332)
