[GitHub] incubator-carbondata pull request #113: stop compaction flow after metadata ...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-carbondata/pull/113


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-carbondata pull request #128: combine two same name classes

[zuowang]

GitHub user zuowang opened a pull request:

https://github.com/apache/incubator-carbondata/pull/128

combine two same name classes

there are two same name classes, need to optimize :

integration/spark/src/test/scala/org/apache/carbondata/spark/load/CarbonLoaderUtilTest.java

integration/spark/src/test/java/org/carbondata/integration/spark/load/CarbonLoaderUtilTest.java
This PR merged them.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/zuowang/incubator-carbondata master

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-carbondata/pull/128.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #128


commit ca334e5c421d32aaa02019d75742cfaf13d9b7cc
Author: Zuo Wang 
Date:   2016-09-06T06:48:33Z

combine two same name classes




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-carbondata pull request #126: [CARBONDATA-212] Use SQLContext to r...

[QiangCai]

Github user QiangCai commented on a diff in the pull request:

https://github.com/apache/incubator-carbondata/pull/126#discussion_r77598344
  
--- Diff: 
examples/src/main/scala/org/apache/carbondata/examples/DatasourceExample.scala 
---
@@ -0,0 +1,55 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.carbondata.examples
+
+import org.apache.spark.sql.{SaveMode, SQLContext}
+
+import org.apache.carbondata.examples.util.InitForExamples
+
+object DatasourceExample {
+
+  def main(args: Array[String]) {
+// use CarbonContext to write CarbonData files
+val cc = InitForExamples.createCarbonContext("DatasourceExample")
+import cc.implicits._
+val sc = cc.sparkContext
+// create a dataframe, it can be from parquet or hive table
+val df = sc.parallelize(1 to 1000)
+   .map(x => ("a", "b", x))
+   .toDF("c1", "c2", "c3")
+
+// save dataframe to CarbonData files
+df.write
+  .format("carbondata")
+  .option("tableName", "carbon1")
+  .mode(SaveMode.Overwrite)
+  .save()
+
+// use SQLContext to read CarbonData files
+val sqlContext = new SQLContext(sc)
+sqlContext.sql(
+  """
+| CREATE TEMPORARY TABLE source
+| (c1 string, c2 string, c3 long)
+| USING org.apache.spark.sql.CarbonSource
+| OPTIONS (path './examples/target/store',
--- End diff --

please change path to './target/store'


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-carbondata pull request #129: Remove not needed parameters

[Zhangshunyu]

GitHub user Zhangshunyu opened a pull request:

https://github.com/apache/incubator-carbondata/pull/129

Remove not needed parameters

There are many parameters in CarbonCommonConstants we not use now, should 
remove them.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/Zhangshunyu/incubator-carbondata mater

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-carbondata/pull/129.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #129


commit 296a7350a667a13a46c74b2b38d36ee8dc13f53f
Author: Zhangshunyu 
Date:   2016-09-05T06:33:36Z

remove not needed parameters




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-carbondata pull request #126: [CARBONDATA-212] Use SQLContext to r...

[QiangCai]

Github user QiangCai commented on a diff in the pull request:

https://github.com/apache/incubator-carbondata/pull/126#discussion_r77605228
  
--- Diff: 
integration/spark/src/main/scala/org/apache/spark/sql/CarbonDatasourceRelation.scala
 ---
@@ -123,24 +120,24 @@ class CarbonSource
   dataSchema: Option[StructType],
   partitionColumns: Option[StructType],
   parameters: Map[String, String]): HadoopFsRelation = {
-CarbonDatasourceHadoopRelation(sqlContext, paths, parameters)
+CarbonDatasourceHadoopRelation(sqlContext, paths, parameters, 
dataSchema)
--- End diff --

You can validate the parameters here.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-carbondata pull request #118: [CARBONDATA-201]add comment option

[QiangCai]

Github user QiangCai commented on a diff in the pull request:

https://github.com/apache/incubator-carbondata/pull/118#discussion_r77617020
  
--- Diff: 
integration/spark/src/test/scala/org/apache/carbondata/spark/testsuite/dataload/TestLoadDataWithHiveSyntax.scala
 ---
@@ -590,6 +590,23 @@ class TestLoadDataWithHiveSyntax extends QueryTest 
with BeforeAndAfterAll {
 checkAnswer(sql("select * from carbontable1"), sql("select * from 
hivetable1"))
   }
 
+  test("test data loading with comment option") {
+sql("drop table if exists comment_test")
+sql(
+  "create table comment_test(imei string, age int, task bigint, num 
double, level decimal(10," +
+"3), productdate timestamp, mark int, name string) STORED BY 
'org.apache.carbondata.format'"
+)
+sql(
+  "LOAD DATA local inpath './src/test/resources/comment.csv' INTO 
TABLE comment_test " +
+"options('DELIMITER' = ',', 'QUOTECHAR' = '.', 'COMMENTCHAR' = 
'?','FILEHEADER'='imei,age,task,num,level,productdate,mark,name')"
+)
+checkAnswer(sql("select imei from 
comment_test"),Seq(Row("\"huawei"),Row("#huawei"), Row(""),
+  Row("~huawei")))
+sql("drop table if exists comment_test")
--- End diff --

Better to drop table in the method afterAll 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-carbondata pull request #118: [CARBONDATA-201]add comment option

[lion-x]

Github user lion-x commented on a diff in the pull request:

https://github.com/apache/incubator-carbondata/pull/118#discussion_r77618003
  
--- Diff: 
integration/spark/src/test/scala/org/apache/carbondata/spark/testsuite/dataload/TestLoadDataWithHiveSyntax.scala
 ---
@@ -590,6 +590,23 @@ class TestLoadDataWithHiveSyntax extends QueryTest 
with BeforeAndAfterAll {
 checkAnswer(sql("select * from carbontable1"), sql("select * from 
hivetable1"))
   }
 
+  test("test data loading with comment option") {
+sql("drop table if exists comment_test")
+sql(
+  "create table comment_test(imei string, age int, task bigint, num 
double, level decimal(10," +
+"3), productdate timestamp, mark int, name string) STORED BY 
'org.apache.carbondata.format'"
+)
+sql(
+  "LOAD DATA local inpath './src/test/resources/comment.csv' INTO 
TABLE comment_test " +
+"options('DELIMITER' = ',', 'QUOTECHAR' = '.', 'COMMENTCHAR' = 
'?','FILEHEADER'='imei,age,task,num,level,productdate,mark,name')"
+)
+checkAnswer(sql("select imei from 
comment_test"),Seq(Row("\"huawei"),Row("#huawei"), Row(""),
+  Row("~huawei")))
+sql("drop table if exists comment_test")
--- End diff --

ok


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-carbondata pull request #104: [CARBONDATA-188] Compress CSV file b...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-carbondata/pull/104


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (CARBONDATA-214) The binary file need be removed under core/src/test/resources/part-xxx

[ChenLiang (JIRA)]

ChenLiang created CARBONDATA-214:


 Summary: The binary file need be removed under 
core/src/test/resources/part-xxx
 Key: CARBONDATA-214
 URL: https://issues.apache.org/jira/browse/CARBONDATA-214
 Project: CarbonData
  Issue Type: Bug
  Components: core
Affects Versions: 0.2.0-incubating, 0.1.1-incubating
Reporter: ChenLiang
Assignee: ChenLiang
Priority: Minor
 Fix For: 0.2.0-incubating, 0.1.1-incubating


The binary file need be removed under core/src/test/resources/part-xxx



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (CARBONDATA-215) Correct the file headers of classes

[Ravindra Pesala (JIRA)]

Ravindra Pesala created CARBONDATA-215:
--

 Summary: Correct the file headers of classes
 Key: CARBONDATA-215
 URL: https://issues.apache.org/jira/browse/CARBONDATA-215
 Project: CarbonData
  Issue Type: Bug
Affects Versions: 0.1.0-incubating
Reporter: Ravindra Pesala
Priority: Minor
 Fix For: 0.2.0-incubating, 0.1.1-incubating


Extra file header is present in the following classes. These files should not 
have any other file headers apart from Apache's.


./integration/spark/src/main/java/org/apache/carbondata/spark/partition/api/DataPartitioner.java
 
2. 
./integration/spark/src/main/java/org/apache/carbondata/spark/partition/api/impl/CSVFilePartitioner.java
 
3. 
./integration/spark/src/main/java/org/apache/carbondata/spark/partition/api/impl/DefaultLoadBalancer.java
 
4. 
./integration/spark/src/main/java/org/apache/carbondata/spark/partition/api/impl/PartitionImpl.java
 
5. 
./integration/spark/src/main/java/org/apache/carbondata/spark/partition/api/impl/QueryPartitionHelper.java
 
6. 
./integration/spark/src/main/java/org/apache/carbondata/spark/partition/api/Partition.java



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (CARBONDATA-216) Files should be deleted as this feature not supported now.

[Ravindra Pesala (JIRA)]

Ravindra Pesala created CARBONDATA-216:
--

 Summary: Files should be deleted as this feature not supported now.
 Key: CARBONDATA-216
 URL: https://issues.apache.org/jira/browse/CARBONDATA-216
 Project: CarbonData
  Issue Type: Bug
Affects Versions: 0.1.0-incubating
Reporter: Ravindra Pesala
Priority: Minor
 Fix For: 0.2.0-incubating, 0.1.1-incubating


Following files and its references need to deleted from carbon as this feature 
is not used now.
1. ./core/src/test/java/org/apache/carbondata/scan/QueryExecutor_UT.java 
2./integration/spark/src/main/java/org/apache/carbondata/spark/partition/reader/CSVParser.java
 
3. 
./integration/spark/src/main/java/org/apache/carbondata/spark/partition/reader/CSVReader.java
 
4. 
./integration/spark/src/main/java/org/apache/carbondata/spark/partition/reader/CSVWriter.java
 
5. 
./integration/spark/src/main/java/org/apache/carbondata/spark/partition/reader/ResultSetHelper.java
 
6. 
./integration/spark/src/main/java/org/apache/carbondata/spark/partition/reader/ResultSetHelperService.java
 




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-carbondata pull request #128: [CARBONDATA-206] combine two same na...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-carbondata/pull/128


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Discussion for 0.1.1 patch release :

[Liang Big data]

Hi all

Plan to give 0.1.1 patch release in Sep as per the plan.
The below two issues(214,215) reported during 0.1.0 release which will be
fixed , and also includes some other defects which has been fixed

please let us know if there are any other major issues need to be fixed in
0.1.1 patch, which would not report to JIRA yet.


*https://issues.apache.org/jira/browse/CARBONDATA-214
*
https://issues.apache.org/jira/browse/CARBONDATA-215


Regards
Liang
-- Forwarded message --
From: ChenLiang (JIRA) 
Date: 2016-09-06 23:43 GMT+08:00
Subject: [jira] [Created] (CARBONDATA-214) The binary file need be removed
under core/src/test/resources/part-xxx
To: iss...@carbondata.incubator.apache.org


ChenLiang created CARBONDATA-214:


 Summary: The binary file need be removed under
core/src/test/resources/part-xxx
 Key: CARBONDATA-214
 URL: https://issues.apache.org/jira/browse/CARBONDATA-214
 Project: CarbonData
  Issue Type: Bug
  Components: core
Affects Versions: 0.2.0-incubating, 0.1.1-incubating
Reporter: ChenLiang
Assignee: ChenLiang
Priority: Minor
 Fix For: 0.2.0-incubating, 0.1.1-incubating


The binary file need be removed under core/src/test/resources/part-xxx



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)



-- 

Regards
Liang

Podling report for September '16

[Jean-Baptiste Onofré]

Hi all,

I prepared the podling report for this month:

https://wiki.apache.org/incubator/September2016

Please, take a look and let me know if I forgot something or you want to 
do some changes.


Thanks,
Regards
JB
--
Jean-Baptiste Onofré
jbono...@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com

Re: Discussion for 0.1.1 patch release :

[Henry Saputra]

Removing private@ from list.

Hi Liang,

Thanks for the release plan. Looking forward to it.

Just small reminder that let's not do  cross posting to dev@ and private@
at the same time, bc we could accidentally post messages for private@ to
dev@ list.


- Henry

On Tue, Sep 6, 2016 at 7:28 PM, Liang Big data 
wrote:

> Hi all
>
> Plan to give 0.1.1 patch release in Sep as per the plan.
> The below two issues(214,215) reported during 0.1.0 release which will be
> fixed , and also includes some other defects which has been fixed
>
> please let us know if there are any other major issues need to be fixed in
> 0.1.1 patch, which would not report to JIRA yet.
>
>
> *https://issues.apache.org/jira/browse/CARBONDATA-214
> *
> https://issues.apache.org/jira/browse/CARBONDATA-215
>
>
> Regards
> Liang
> -- Forwarded message --
> From: ChenLiang (JIRA) 
> Date: 2016-09-06 23:43 GMT+08:00
> Subject: [jira] [Created] (CARBONDATA-214) The binary file need be removed
> under core/src/test/resources/part-xxx
> To: iss...@carbondata.incubator.apache.org
>
>
> ChenLiang created CARBONDATA-214:
> 
>
>  Summary: The binary file need be removed under
> core/src/test/resources/part-xxx
>  Key: CARBONDATA-214
>  URL: https://issues.apache.org/jira/browse/CARBONDATA-214
>  Project: CarbonData
>   Issue Type: Bug
>   Components: core
> Affects Versions: 0.2.0-incubating, 0.1.1-incubating
> Reporter: ChenLiang
> Assignee: ChenLiang
> Priority: Minor
>  Fix For: 0.2.0-incubating, 0.1.1-incubating
>
>
> The binary file need be removed under core/src/test/resources/part-xxx
>
>
>
> --
> This message was sent by Atlassian JIRA
> (v6.3.4#6332)
>
>
>
> --
>
> Regards
> Liang
>