Re: [DISCUSSION] Propose to move notification of "jira Created" to issues@mailing list from dev

2017-07-04 Thread Raghunandan S 
+1

> On 05-Jul-2017, at 9:07 AM, Jacky Li  wrote:
> 
> +1
> 
>> 在 2017年7月4日，下午9:27，Bhavya Aggarwal  写道：
>> 
>> +1
>> Agreed these should be two seperate mailing lists.
>> 
>> Thanks and Regards
>> Bhavya
>> 
>> On Tue, Jul 4, 2017 at 5:20 PM, Venkata Gollamudi 
>> wrote:
>> 
>>> +1
>>> It is better to be moved
>>> 
>>> Regards,
>>> Venkata Ramana G
>>> 
>>> On Tue, Jul 4, 2017 at 4:40 PM, Kumar Vishal 
>>> wrote:
>>> 
 +1
 Better to move to issue mailing list
 
 Regards
 Kumar Vishal
 
 Sent from my iPhone
 
> On 03-Jul-2017, at 15:02, Ravindra Pesala 
>>> wrote:
> 
> +1
> Yes, we should move to issues mailing list.
> 
> Regards,
> Ravindra.
> 
>> On 30 June 2017 at 07:35, Erlu Chen  wrote:
>> 
>> Agreed, we can separate discussion and created JIRA.
>> 
>> It will be better for develops to filter some unnecessary message and
 focus
>> on discussion.
>> 
>> Regards.
>> Chenerlu.
>> 
>> 
>> 
>> --
>> View this message in context: http://apache-carbondata-dev-
>> mailing-list-archive.1130556.n5.nabble.com/DISCUSSION-
>> Propose-to-move-notification-of-jira-Created-to-issues-
>> mailing-list-from-dev-tp16835p16842.html
>> Sent from the Apache CarbonData Dev Mailing List archive mailing list
>> archive at Nabble.com.
>> 
> 
> 
> 
> --
> Thanks & Regards,
> Ravi
 
>>> 
> 
> 
>

Re: [Discussion] CarbonOutputFormat Implementation

2017-07-04 Thread Jacky Li 
+1.

For carbon data files, I think there should be at least two OutputFormat,
1) FileOutputFormat, which will not do sorting and write to carbondata file 
only. This will be used in GLOBAL_SORT option
2) TableOutputFormat, which will do sorting according to SORT_SCOPE option, and 
use Single Pass to load

And I think dictionary should be another OutputFormat.
So user can combine to use dictionary output format and carbondata file output 
format.

I suggest to firstly check the usage scenario and decide the class hierarchy of 
this feature. 

Regards,
Jacky

> 在 2017年7月4日，下午8:37，Venkata Gollamudi  写道：
> 
> +1
> OutputFormat should be based on single pass and with similar job
> configurations as CarbonInputFormat.
> Please output initial design and code skeleton, for review before
> proceeding for implementation.
> 
> On Tue, Jul 4, 2017 at 4:30 PM, Kumar Vishal 
> wrote:
> 
>> +1
>> It's a long pending task.
>> -Regards
>> Kumar Vishal
>> 
>> Sent from my iPhone
>> 
>>> On 04-Jul-2017, at 16:26, Erlu Chen  wrote:
>>> 
>>> Thanks very much.
>>> 
>>> After you have raised a PR, we can start review.
>>> 
>>> 
>>> Regards.
>>> Chenerlu.
>>> 
>>> 
>>> 
>>> --
>>> View this message in context: http://apache-carbondata-dev-
>> mailing-list-archive.1130556.n5.nabble.com/Discussion-CarbonOutputFormat-
>> Implementation-tp17113p17239.html
>>> Sent from the Apache CarbonData Dev Mailing List archive mailing list
>> archive at Nabble.com.
>>

Re: [DISCUSSION] Propose to move notification of "jira Created" to issues@mailing list from dev

2017-07-04 Thread Jacky Li 
+1

> 在 2017年7月4日，下午9:27，Bhavya Aggarwal  写道：
> 
> +1
> Agreed these should be two seperate mailing lists.
> 
> Thanks and Regards
> Bhavya
> 
> On Tue, Jul 4, 2017 at 5:20 PM, Venkata Gollamudi 
> wrote:
> 
>> +1
>> It is better to be moved
>> 
>> Regards,
>> Venkata Ramana G
>> 
>> On Tue, Jul 4, 2017 at 4:40 PM, Kumar Vishal 
>> wrote:
>> 
>>> +1
>>> Better to move to issue mailing list
>>> 
>>> Regards
>>> Kumar Vishal
>>> 
>>> Sent from my iPhone
>>> 
 On 03-Jul-2017, at 15:02, Ravindra Pesala 
>> wrote:
 
 +1
 Yes, we should move to issues mailing list.
 
 Regards,
 Ravindra.
 
> On 30 June 2017 at 07:35, Erlu Chen  wrote:
> 
> Agreed, we can separate discussion and created JIRA.
> 
> It will be better for develops to filter some unnecessary message and
>>> focus
> on discussion.
> 
> Regards.
> Chenerlu.
> 
> 
> 
> --
> View this message in context: http://apache-carbondata-dev-
> mailing-list-archive.1130556.n5.nabble.com/DISCUSSION-
> Propose-to-move-notification-of-jira-Created-to-issues-
> mailing-list-from-dev-tp16835p16842.html
> Sent from the Apache CarbonData Dev Mailing List archive mailing list
> archive at Nabble.com.
> 
 
 
 
 --
 Thanks & Regards,
 Ravi
>>> 
>>

[jira] [Created] (CARBONDATA-1265) Fix AllDictionaryExample because it is only supported when single_pass is true

2017-07-04 Thread chenerlu (JIRA) 
chenerlu created CARBONDATA-1265:


 Summary: Fix AllDictionaryExample because it is only supported 
when single_pass is true
 Key: CARBONDATA-1265
 URL: https://issues.apache.org/jira/browse/CARBONDATA-1265
 Project: CarbonData
  Issue Type: Bug
Reporter: chenerlu
Assignee: chenerlu
Priority: Minor






--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (CARBONDATA-1264) Fix AllDictionaryExample because it is only supported when single_pass is true

2017-07-04 Thread chenerlu (JIRA) 
chenerlu created CARBONDATA-1264:


 Summary: Fix AllDictionaryExample because it is only supported 
when single_pass is true
 Key: CARBONDATA-1264
 URL: https://issues.apache.org/jira/browse/CARBONDATA-1264
 Project: CarbonData
  Issue Type: Bug
Reporter: chenerlu
Assignee: chenerlu
Priority: Minor






--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (CARBONDATA-1263) Single pass load does not take default value false for blank or invalid single pass value

2017-07-04 Thread Chetan Bhat (JIRA) 
Chetan Bhat created CARBONDATA-1263:
---

 Summary: Single pass load does not take default value false for 
blank or invalid single pass value
 Key: CARBONDATA-1263
 URL: https://issues.apache.org/jira/browse/CARBONDATA-1263
 Project: CarbonData
  Issue Type: Bug
  Components: data-load
Affects Versions: 1.1.0
 Environment: 3 node cluster
Reporter: Chetan Bhat
Priority: Minor


Issue : Single pass load does not take default value false for blank or invalid 
single pass value.

0: jdbc:hive2://10.19.91.224:22550/default> LOAD DATA  inpath 
'hdfs://hacluster/chetan/1.csv' into table flow_carbon_test4 
options('DELIMITER'=',', 
'QUOTECHAR'='"','COLUMNDICT'='test:hdfs://hacluster/chetan/MSISDN.csv','SINGLE_PASS'='','FILEHEADER'='test');
+-+--+
| Result  |
+-+--+
+-+--+
No rows selected (37.628 seconds)

0: jdbc:hive2://10.19.91.224:22550/default> LOAD DATA  inpath 
'hdfs://hacluster/chetan/1.csv' into table flow_carbon_test4 
options('DELIMITER'=',', 
'QUOTECHAR'='"','COLUMNDICT'='test:hdfs://hacluster/chetan/MSISDN.csv','SINGLE_PASS'='1234','FILEHEADER'='test,test1');
+-+--+
| Result  |
+-+--+
+-+--+
No rows selected (39.131 seconds)


Expected : Validation should be provided and the load should fail with error 
message. Default value of Single pass false should be reflected for 
blank/invalid value for single pass.




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: [DISCUSSION] Propose to move notification of "jira Created" to issues@mailing list from dev

2017-07-04 Thread Bhavya Aggarwal 
+1
Agreed these should be two seperate mailing lists.

Thanks and Regards
Bhavya

On Tue, Jul 4, 2017 at 5:20 PM, Venkata Gollamudi 
wrote:

> +1
> It is better to be moved
>
> Regards,
> Venkata Ramana G
>
> On Tue, Jul 4, 2017 at 4:40 PM, Kumar Vishal 
> wrote:
>
> > +1
> > Better to move to issue mailing list
> >
> > Regards
> > Kumar Vishal
> >
> > Sent from my iPhone
> >
> > > On 03-Jul-2017, at 15:02, Ravindra Pesala 
> wrote:
> > >
> > > +1
> > > Yes, we should move to issues mailing list.
> > >
> > > Regards,
> > > Ravindra.
> > >
> > >> On 30 June 2017 at 07:35, Erlu Chen  wrote:
> > >>
> > >> Agreed, we can separate discussion and created JIRA.
> > >>
> > >> It will be better for develops to filter some unnecessary message and
> > focus
> > >> on discussion.
> > >>
> > >> Regards.
> > >> Chenerlu.
> > >>
> > >>
> > >>
> > >> --
> > >> View this message in context: http://apache-carbondata-dev-
> > >> mailing-list-archive.1130556.n5.nabble.com/DISCUSSION-
> > >> Propose-to-move-notification-of-jira-Created-to-issues-
> > >> mailing-list-from-dev-tp16835p16842.html
> > >> Sent from the Apache CarbonData Dev Mailing List archive mailing list
> > >> archive at Nabble.com.
> > >>
> > >
> > >
> > >
> > > --
> > > Thanks & Regards,
> > > Ravi
> >
>

Re: [Discussion] CarbonOutputFormat Implementation

2017-07-04 Thread Venkata Gollamudi 
+1
OutputFormat should be based on single pass and with similar job
configurations as CarbonInputFormat.
Please output initial design and code skeleton, for review before
proceeding for implementation.

On Tue, Jul 4, 2017 at 4:30 PM, Kumar Vishal 
wrote:

> +1
> It's a long pending task.
> -Regards
> Kumar Vishal
>
> Sent from my iPhone
>
> > On 04-Jul-2017, at 16:26, Erlu Chen  wrote:
> >
> > Thanks very much.
> >
> > After you have raised a PR, we can start review.
> >
> >
> > Regards.
> > Chenerlu.
> >
> >
> >
> > --
> > View this message in context: http://apache-carbondata-dev-
> mailing-list-archive.1130556.n5.nabble.com/Discussion-CarbonOutputFormat-
> Implementation-tp17113p17239.html
> > Sent from the Apache CarbonData Dev Mailing List archive mailing list
> archive at Nabble.com.
>

Re: [DISCUSSION] Propose to move notification of "jira Created" to issues@mailing list from dev

2017-07-04 Thread Venkata Gollamudi 
+1
It is better to be moved

Regards,
Venkata Ramana G

On Tue, Jul 4, 2017 at 4:40 PM, Kumar Vishal 
wrote:

> +1
> Better to move to issue mailing list
>
> Regards
> Kumar Vishal
>
> Sent from my iPhone
>
> > On 03-Jul-2017, at 15:02, Ravindra Pesala  wrote:
> >
> > +1
> > Yes, we should move to issues mailing list.
> >
> > Regards,
> > Ravindra.
> >
> >> On 30 June 2017 at 07:35, Erlu Chen  wrote:
> >>
> >> Agreed, we can separate discussion and created JIRA.
> >>
> >> It will be better for develops to filter some unnecessary message and
> focus
> >> on discussion.
> >>
> >> Regards.
> >> Chenerlu.
> >>
> >>
> >>
> >> --
> >> View this message in context: http://apache-carbondata-dev-
> >> mailing-list-archive.1130556.n5.nabble.com/DISCUSSION-
> >> Propose-to-move-notification-of-jira-Created-to-issues-
> >> mailing-list-from-dev-tp16835p16842.html
> >> Sent from the Apache CarbonData Dev Mailing List archive mailing list
> >> archive at Nabble.com.
> >>
> >
> >
> >
> > --
> > Thanks & Regards,
> > Ravi
>

Re: [Discussion] Add HEADER option to load data sql

2017-07-04 Thread Venkata Gollamudi 
I agree that user need not provide columns names if no header present in
file and columns order is same as schema order.

instead of option header=true, will not cover all the cases of header
present, not present, override header etc. I have added added intermediate
approach covering all the cases and also taking care of current default
values and backward compatibility.

csv file without header
1. FILEHEADER="col1,col2,col3",  default: IGNORE_FIRST_LINE="FALSE"
use given header
2. FILEHEADER="" default: IGNORE_FIRST_LINE="FALSE"
use schema order

csv file with header
1. Nonedefault:
IGNORE_FIRST_LINE="FALSE"
 expects CSV first line as header.
2. FILEHEADER="col1,col2,col3",  IGNORE_FIRST_LINE="TRUE"
uses explicitly given header, ignoring header from file.
3. FILEHEADER="",
 IGNORE_FIRST_LINE="TRUE"
uses schema order, ignoring header from file.

Regards,
Ramana

On Tue, Jul 4, 2017 at 6:51 AM, wangbin  wrote:

> I propose the loading the CSV files by explicitly give a table schema,while
> using a option to ignore csv header if has.
>
>
>
> --
> View this message in context: http://apache-carbondata-dev-
> mailing-list-archive.1130556.n5.nabble.com/Discussion-Add-
> HEADER-option-to-load-data-sql-tp17080p17179.html
> Sent from the Apache CarbonData Dev Mailing List archive mailing list
> archive at Nabble.com.
>

[jira] [Created] (CARBONDATA-1262) Remove unnecessary LoadConfiguration creation

2017-07-04 Thread Jacky Li (JIRA) 
Jacky Li created CARBONDATA-1262:


 Summary: Remove unnecessary LoadConfiguration creation
 Key: CARBONDATA-1262
 URL: https://issues.apache.org/jira/browse/CARBONDATA-1262
 Project: CarbonData
  Issue Type: Improvement
Reporter: Jacky Li
 Fix For: 1.2.0


Remove unnecessary LoadConfiguration creation



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: [DISCUSSION] Propose to move notification of "jira Created" to issues@mailing list from dev

2017-07-04 Thread Kumar Vishal 
+1
Better to move to issue mailing list

Regards 
Kumar Vishal

Sent from my iPhone

> On 03-Jul-2017, at 15:02, Ravindra Pesala  wrote:
> 
> +1
> Yes, we should move to issues mailing list.
> 
> Regards,
> Ravindra.
> 
>> On 30 June 2017 at 07:35, Erlu Chen  wrote:
>> 
>> Agreed, we can separate discussion and created JIRA.
>> 
>> It will be better for develops to filter some unnecessary message and focus
>> on discussion.
>> 
>> Regards.
>> Chenerlu.
>> 
>> 
>> 
>> --
>> View this message in context: http://apache-carbondata-dev-
>> mailing-list-archive.1130556.n5.nabble.com/DISCUSSION-
>> Propose-to-move-notification-of-jira-Created-to-issues-
>> mailing-list-from-dev-tp16835p16842.html
>> Sent from the Apache CarbonData Dev Mailing List archive mailing list
>> archive at Nabble.com.
>> 
> 
> 
> 
> -- 
> Thanks & Regards,
> Ravi

Re: [Discussion] Add HEADER option to load data sql

2017-07-04 Thread Kumar Vishal 
+1
It will be useful when csv file header is same as table schema in that case 
it's a pain for user to pass all the csv header. But it's depends completely on 
user scenarios how csv file is getting generated. 

Regards
Kumar Vishal

Sent from my iPhone

> On 04-Jul-2017, at 06:51, wangbin  wrote:
> 
> I propose the loading the CSV files by explicitly give a table schema,while
> using a option to ignore csv header if has.
> 
> 
> 
> --
> View this message in context: 
> http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/Discussion-Add-HEADER-option-to-load-data-sql-tp17080p17179.html
> Sent from the Apache CarbonData Dev Mailing List archive mailing list archive 
> at Nabble.com.

Re: [Discussion] CarbonOutputFormat Implementation

2017-07-04 Thread Kumar Vishal 
+1
It's a long pending task. 
-Regards 
Kumar Vishal

Sent from my iPhone

> On 04-Jul-2017, at 16:26, Erlu Chen  wrote:
> 
> Thanks very much.
> 
> After you have raised a PR, we can start review.
> 
> 
> Regards.
> Chenerlu.
> 
> 
> 
> --
> View this message in context: 
> http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/Discussion-CarbonOutputFormat-Implementation-tp17113p17239.html
> Sent from the Apache CarbonData Dev Mailing List archive mailing list archive 
> at Nabble.com.

Re: [Discussion] CarbonOutputFormat Implementation

2017-07-04 Thread Erlu Chen 
Thanks very much.

After you have raised a PR, we can start review.


Regards.
Chenerlu.



--
View this message in context: 
http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/Discussion-CarbonOutputFormat-Implementation-tp17113p17239.html
Sent from the Apache CarbonData Dev Mailing List archive mailing list archive 
at Nabble.com.

Re: [Discussion] CarbonOutputFormat Implementation

2017-07-04 Thread Divya Gupta 
Thanks for the quick reply Chenerlu.

I would surely like to contribute this feature and will start working
towards CARBONDATA-729.

Regards
Divya Gupta

Regards
Divya Gupta
Project Lead


*Knoldus Software LLP *
India  - US  - Canada


Blog  | Twitter  |
FB  | LinkedIn


On Tue, Jul 4, 2017 at 2:37 PM, Erlu Chen  wrote:

> Hi Divya
>
> Thanks for your suggestion.
>
> Carbondata may support it in the near future.
>
> If you want to contribute this feature, I think it will benefit community a
> lot.
>
>
> Regards.
> Chenerlu.
>
>
>
> --
> View this message in context: http://apache-carbondata-dev-
> mailing-list-archive.1130556.n5.nabble.com/Discussion-CarbonOutputFormat-
> Implementation-tp17113p17214.html
> Sent from the Apache CarbonData Dev Mailing List archive mailing list
> archive at Nabble.com.
>

Re: [Discussion] CarbonOutputFormat Implementation

2017-07-04 Thread Erlu Chen 
Hi Divya

Thanks for your suggestion.

Carbondata may support it in the near future.

If you want to contribute this feature, I think it will benefit community a
lot.


Regards.
Chenerlu.



--
View this message in context: 
http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/Discussion-CarbonOutputFormat-Implementation-tp17113p17214.html
Sent from the Apache CarbonData Dev Mailing List archive mailing list archive 
at Nabble.com.

[jira] [Created] (CARBONDATA-1261) load sql add 'header' option

2017-07-04 Thread QiangCai (JIRA) 
QiangCai created CARBONDATA-1261:


 Summary: load sql add 'header' option
 Key: CARBONDATA-1261
 URL: https://issues.apache.org/jira/browse/CARBONDATA-1261
 Project: CarbonData
  Issue Type: Improvement
Reporter: QiangCai


When we load the CSV files without file header and the file header is the same 
with the table schema,  add 'header'='false' to load data sql, no need to let 
user provide the file header.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: [Discussion] Add HEADER option to load data sql

2017-07-04 Thread wangbin 
I propose the loading the CSV files by explicitly give a table schema,while
using a option to ignore csv header if has.



--
View this message in context: 
http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/Discussion-Add-HEADER-option-to-load-data-sql-tp17080p17179.html
Sent from the Apache CarbonData Dev Mailing List archive mailing list archive 
at Nabble.com.

[jira] [Created] (CARBONDATA-1260) Show Partition for Range partition is not showing the correct range

2017-07-04 Thread ayushmantri (JIRA) 
ayushmantri created CARBONDATA-1260:
---

 Summary: Show Partition for Range partition is not showing the 
correct range 
 Key: CARBONDATA-1260
 URL: https://issues.apache.org/jira/browse/CARBONDATA-1260
 Project: CarbonData
  Issue Type: Bug
Reporter: ayushmantri
Assignee: ayushmantri
Priority: Minor


create table test1(  country string ) PARTITIONED BY (id int)  stored by 
'carbondata' TBLPROPERTIES('DICTIONARY_INCLUDE'='id','PARTITION_TYPE'='RANGE', 
'RANGE_INFO'='10,15,20')"

Actual result 
+--+
|partition |
+--+
|id=default|
|id<10 |
|10<=id<15 |
|15<=id<20 |
+--+

Expected result

+--+
|partition |
+--+
||id<10|
|10<=id<15 |
|15<=id<20 
| id >=20  |
+--+



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)