[jira] [Commented] (CARBONDATA-302) 7. Add DataWriterProcessorStep which reads the data from sort temp files and creates carbondata files.

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/CARBONDATA-302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15590563#comment-15590563
 ] 

ASF GitHub Bot commented on CARBONDATA-302:
---

GitHub user ravipesala opened a pull request:

https://github.com/apache/incubator-carbondata/pull/251

[CARBONDATA-302]Added Writer processor step for dataloading.

Add DataWriterProcessorStep which reads the data from sort processor step 
and apply mdk generator on key and creates carbondata files.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ravipesala/incubator-carbondata 
datawriter-step

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-carbondata/pull/251.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #251


commit 9edbfdd48a9f35c0296703475a01f8cc7b02f8fc
Author: ravipesala 
Date:   2016-10-20T02:41:46Z

Added Writer processor step for dataloading.




> 7. Add DataWriterProcessorStep which reads the data from sort temp files and 
> creates carbondata files.
> --
>
> Key: CARBONDATA-302
> URL: https://issues.apache.org/jira/browse/CARBONDATA-302
> Project: CarbonData
>  Issue Type: Sub-task
>Reporter: Ravindra Pesala
> Fix For: 0.2.0-incubating
>
>
> Add DataWriterProcessorStep which reads the data from sort temp files and 
> merge sort it, and apply mdk generator on key and creates carbondata files.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (CARBONDATA-330) Fix compiler warnings - Java related

[Aniket Adnaik (JIRA)]

Aniket Adnaik created CARBONDATA-330:


 Summary: Fix compiler warnings - Java related
 Key: CARBONDATA-330
 URL: https://issues.apache.org/jira/browse/CARBONDATA-330
 Project: CarbonData
  Issue Type: Improvement
  Components: build, core
Affects Versions: 0.2.0-incubating
Reporter: Aniket Adnaik
Priority: Trivial
 Fix For: 0.2.0-incubating


Fix java compiler warnings and code cleanup.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CARBONDATA-329) constant final class changed to interface

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/CARBONDATA-329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15589562#comment-15589562
 ] 

ASF GitHub Bot commented on CARBONDATA-329:
---

GitHub user abhishekgiri38 opened a pull request:

https://github.com/apache/incubator-carbondata/pull/249

[CARBONDATA-329] constant final class changed to interface

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [x] Make sure the PR title is formatted like:
   `[CARBONDATA-] Description of pull request`
 - [x] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [x] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [x] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).
 - [ ] Testing done
 
Please provide details on 
- Whether new unit test cases have been added or why no new tests 
are required?
- What manual testing you have done?
- Any additional information to help reviewers in testing this 
change.
 
 - [ ] For large changes, please consider breaking it into sub-tasks under 
an umbrella JIRA. 
 
---



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/abhishekgiri38/incubator-carbondata master

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-carbondata/pull/249.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #249


commit e38b01929b16871294effc6eeab37f787e136f00
Author: abhishekgiri38 
Date:   2016-10-19T18:55:21Z

constant file's final class changed to interface




> constant final class changed to interface
> -
>
> Key: CARBONDATA-329
> URL: https://issues.apache.org/jira/browse/CARBONDATA-329
> Project: CarbonData
>  Issue Type: Improvement
>  Components: core
>Reporter: abhishek 
>
> Constant file's is final class and it is now changed to interface. Implicitly 
> fields are static and final.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (CARBONDATA-329) constant final class changed to interface

[abhishek (JIRA)]

abhishek  created CARBONDATA-329:


 Summary: constant final class changed to interface
 Key: CARBONDATA-329
 URL: https://issues.apache.org/jira/browse/CARBONDATA-329
 Project: CarbonData
  Issue Type: Improvement
  Components: core
Reporter: abhishek 


Constant file's is final class and it is now changed to interface. Implicitly 
fields are static and final.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (CARBONDATA-329) constant final class changed to interface

[abhishek (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CARBONDATA-329?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

abhishek  updated CARBONDATA-329:
-
Priority: Trivial  (was: Major)

> constant final class changed to interface
> -
>
> Key: CARBONDATA-329
> URL: https://issues.apache.org/jira/browse/CARBONDATA-329
> Project: CarbonData
>  Issue Type: Improvement
>  Components: core
>Reporter: abhishek 
>Priority: Trivial
>
> Constant file's is final class and it is now changed to interface. Implicitly 
> fields are static and final.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (CARBONDATA-325) Create table with columns contains spaces in name.

[Liang Chen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CARBONDATA-325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Liang Chen updated CARBONDATA-325:
--
Assignee: Harmeet Singh

> Create table with columns contains spaces in name.
> --
>
> Key: CARBONDATA-325
> URL: https://issues.apache.org/jira/browse/CARBONDATA-325
> Project: CarbonData
>  Issue Type: Bug
>Reporter: Harmeet Singh
>Assignee: Harmeet Singh
>
> I want to create table, using columns that contains spaces. I am using Thrift 
> Server and Beeline client for accessing carbon data. Whenever i am trying to 
> create a table, and their columns name contains spaces i am getting an error. 
> Below are the steps:
> Step 1:
> create table three (`first name` string, `age` int) stored by 'carbondata';
> Whenever i am executing above query, i am getting below error:
> Error: org.apache.carbondata.spark.exception.MalformedCarbonCommandException: 
> Unsupported data type : FieldSchema(name:first name, type:string, 
> comment:null).getType (state=,code=0)
> The above error is pretending to be wrong data types are using. 
> If I am removing `stored by 'carbondata'` from query, then this will work 
> fine because it is run on Hive.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CARBONDATA-296) 1.Add CSVInputFormat to read csv files.

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/CARBONDATA-296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15588762#comment-15588762
 ] 

ASF GitHub Bot commented on CARBONDATA-296:
---

Github user QiangCai commented on a diff in the pull request:

https://github.com/apache/incubator-carbondata/pull/233#discussion_r84068389
  
--- Diff: 
hadoop/src/test/java/org/apache/carbondata/hadoop/csv/CSVInputFormatTest.java 
---
@@ -0,0 +1,171 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.carbondata.hadoop.csv;
+
+import java.io.File;
+import java.io.FileInputStream;
+import java.io.FileOutputStream;
+import java.io.IOException;
+
+import org.apache.carbondata.hadoop.io.StringArrayWritable;
+
+import junit.framework.TestCase;
+import org.junit.Assert;
+import org.junit.Test;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.io.NullWritable;
+import org.apache.hadoop.io.compress.BZip2Codec;
+import org.apache.hadoop.io.compress.CompressionOutputStream;
+import org.apache.hadoop.io.compress.GzipCodec;
+import org.apache.hadoop.io.compress.Lz4Codec;
+import org.apache.hadoop.io.compress.SnappyCodec;
+import org.apache.hadoop.mapreduce.Job;
+import org.apache.hadoop.mapreduce.Mapper;
+import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
+import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
+
+public class CSVInputFormatTest extends TestCase {
+
+  /**
+   * generate compressed files, no need to call this method.
+   * @throws Exception
+   */
+  public void testGenerateCompressFiles() throws Exception {
+String pwd = new File("src/test/resources").getCanonicalPath();
+String inputFile = pwd + "/data.csv";
+FileInputStream input = new FileInputStream(inputFile);
+Configuration conf = new Configuration();
+
+// .gz
+String outputFile = pwd + "/data.csv.gz";
+FileOutputStream output = new FileOutputStream(outputFile);
+GzipCodec gzip = new GzipCodec();
+gzip.setConf(conf);
+CompressionOutputStream outputStream = gzip.createOutputStream(output);
+int i = -1;
+while ((i = input.read()) != -1) {
+  outputStream.write(i);
+}
+outputStream.close();
+input.close();
+
+// .bz2
+input = new FileInputStream(inputFile);
+outputFile = pwd + "/data.csv.bz2";
+output = new FileOutputStream(outputFile);
+BZip2Codec bzip2 = new BZip2Codec();
+bzip2.setConf(conf);
+outputStream = bzip2.createOutputStream(output);
+i = -1;
+while ((i = input.read()) != -1) {
+  outputStream.write(i);
+}
+outputStream.close();
+input.close();
+
+// .snappy
+input = new FileInputStream(inputFile);
+outputFile = pwd + "/data.csv.snappy";
+output = new FileOutputStream(outputFile);
+SnappyCodec snappy = new SnappyCodec();
+snappy.setConf(conf);
+outputStream = snappy.createOutputStream(output);
+i = -1;
+while ((i = input.read()) != -1) {
+  outputStream.write(i);
+}
+outputStream.close();
+input.close();
+
+//.lz4
+input = new FileInputStream(inputFile);
+outputFile = pwd + "/data.csv.lz4";
+output = new FileOutputStream(outputFile);
+Lz4Codec lz4 = new Lz4Codec();
+lz4.setConf(conf);
+outputStream = lz4.createOutputStream(output);
+i = -1;
+while ((i = input.read()) != -1) {
+  outputStream.write(i);
+}
+outputStream.close();
+input.close();
+
+  }
+
+  /**
+   * CSVCheckMapper check the content of csv files.
+   */
+  public static class CSVCheckMapper extends Mapper {

[jira] [Created] (CARBONDATA-327) Drop Daabase unexpected behaviour.

[Harmeet Singh (JIRA)]

Harmeet Singh created CARBONDATA-327:


 Summary: Drop Daabase unexpected behaviour. 
 Key: CARBONDATA-327
 URL: https://issues.apache.org/jira/browse/CARBONDATA-327
 Project: CarbonData
  Issue Type: Bug
Reporter: Harmeet Singh


Hey team, I am creating a database as below:

0: jdbc:hive2://127.0.0.1:1> create database Test;
+-+--+
| result  |
+-+--+
+-+--+

After creating an database i am using that database using below command:

0: jdbc:hive2://127.0.0.1:1> use Test;
+-+--+
| result  |
+-+--+
+-+--+

After That, i am drop the database as below:

0: jdbc:hive2://127.0.0.1:1> drop database test;
+-+--+
| result  |
+-+--+
+-+--+

The database drop successfully. I am expecting, after that the carbon data 
automatically switch to the "default" database. But when i trying to execute 
command "show tables" the result return nothing as below :

0: jdbc:hive2://127.0.0.1:1> show tables;
++--+--+
| tableName  | isTemporary  |
++--+--+
++--+--+
No rows selected (0.019 seconds)

But my default database contains some table as below:
0: jdbc:hive2://127.0.0.1:1> use default;
+-+--+
| result  |
+-+--+
+-+--+
No rows selected (0.024 seconds)
0: jdbc:hive2://127.0.0.1:1> show tables;
++--+--+
| tableName  | isTemporary  |
++--+--+
| one| false|
| two| false|
++--+--+
2 rows selected (0.013 seconds)

If I am following all above step on Hive, Hive gave us an error on show tables 
after drop the database as below:

hive> drop database test;
OK
Time taken: 0.628 seconds
hive> show databases;
OK
default
Time taken: 0.022 seconds, Fetched: 1 row(s)
hive> show tables;
FAILED: SemanticException [Error 10072]: Database does not exist: test 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (CARBONDATA-326) Creates wrong table on 'create table like'

[Prabhat Kashyap (JIRA)]

Prabhat Kashyap created CARBONDATA-326:
--

 Summary: Creates wrong table on 'create table like' 
 Key: CARBONDATA-326
 URL: https://issues.apache.org/jira/browse/CARBONDATA-326
 Project: CarbonData
  Issue Type: Bug
Reporter: Prabhat Kashyap


I'm trying to create a table like my old table but it is not creating as 
expected.

0: jdbc:hive2://localhost:1> CREATE TABLE mainTable(id INT, name STRING) 
STORED BY 'carbondata';
+-+--+
| Result  |
+-+--+
+-+--+
No rows selected (0.206 seconds)
0: jdbc:hive2://localhost:1> DESC mainTable;
+---++--+--+
| col_name  | data_type  | comment  |
+---++--+--+
| name  | string |  |
| id| bigint |  |
+---++--+--+
2 rows selected (0.056 seconds)


Above one is my mainTable and I wants to create copiedTable from it but 
everytime it is show something like:

0: jdbc:hive2://localhost:1> CREATE TABLE copiedTable LIKE mainTable;
+-+--+
| result  |
+-+--+
+-+--+
No rows selected (0.101 seconds)
0: jdbc:hive2://localhost:1> DESC copiedTable;
+---+++--+
| col_name  |   data_type|  comment   |
+---+++--+
| col   | array  | from deserializer  |
+---+++--+
1 row selected (0.022 seconds)

0: jdbc:hive2://localhost:1> LOAD DATA LOCAL INPATH 
'hdfs://localhost:54310/user/hduser/datafiles/data.csv' INTO TABLE copiedTable 
OPTIONS('DELIMITER'=',');
Error: java.lang.RuntimeException: Data loading failed. table not found: 
knoldus.copiedtable (state=,code=0)

0: jdbc:hive2://localhost:1> select * from copiedTable;
+--+--+
| col  |
+--+--+
+--+--+
No rows selected (0.11 seconds)




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (CARBONDATA-325) Create table with columns contains spaces in name.

[Harmeet Singh (JIRA)]

Harmeet Singh created CARBONDATA-325:


 Summary: Create table with columns contains spaces in name.
 Key: CARBONDATA-325
 URL: https://issues.apache.org/jira/browse/CARBONDATA-325
 Project: CarbonData
  Issue Type: Bug
Reporter: Harmeet Singh


I want to create table, using columns that contains spaces. I am using Thrift 
Server and Beeline client for accessing carbon data. Whenever i am trying to 
create a table, and their columns name contains spaces i am getting an error. 
Below are the steps:

Step 1:
create table three (`first name` string, `age` int) stored by 'carbondata';

Whenever i am executing above query, i am getting below error:
Error: org.apache.carbondata.spark.exception.MalformedCarbonCommandException: 
Unsupported data type : FieldSchema(name:first name, type:string, 
comment:null).getType (state=,code=0)

The above error is pretending to be wrong data types are using. 

If I am removing `stored by 'carbondata'` from query, then this will work fine 
because it is run on Hive.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)