Re: [DISCUSSION] In 1.2.0, use Spark 2.1 and Hadoop 2.7.2 as default compilation in pom.

2017-06-16 Thread Bhavya Aggarwal 
+1 for Spark 2.1 and Hadoop 2.7.2

Regards
Bhavya

On Sat, Jun 17, 2017 at 8:51 AM, Erlu Chen  wrote:

> +1
> Support use Spark2.1 and Hadoop 2.7.2 as default compilation in pom.
>
>
>
> --
> View this message in context: http://apache-carbondata-dev-
> mailing-list-archive.1130556.n5.nabble.com/DISCUSSION-In-1-
> 2-0-use-Spark-2-1-and-Hadoop-2-7-2-as-default-compilation-
> in-pom-tp15278p15370.html
> Sent from the Apache CarbonData Dev Mailing List archive mailing list
> archive at Nabble.com.
>

Re: [DISCUSSION] In 1.2.0, use Spark 2.1 and Hadoop 2.7.2 as default compilation in pom.

2017-06-16 Thread Erlu Chen 
+1 
Support use Spark2.1 and Hadoop 2.7.2 as default compilation in pom. 



--
View this message in context: 
http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/DISCUSSION-In-1-2-0-use-Spark-2-1-and-Hadoop-2-7-2-as-default-compilation-in-pom-tp15278p15370.html
Sent from the Apache CarbonData Dev Mailing List archive mailing list archive 
at Nabble.com.

Re: 答复: can't apply mappartitions to dataframe generated from carboncontext

2017-06-16 Thread Erlu Chen 
Hi

I think you can debug in windows by adding some debug parameter  when start
spark-shell in linux.

This is what called remote debug.

I tried this method when I use windows, hope my idea can help you.

Regards.
Chenerlu.



--
View this message in context: 
http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/can-t-apply-mappartitions-to-dataframe-generated-from-carboncontext-tp14565p15369.html
Sent from the Apache CarbonData Dev Mailing List archive mailing list archive 
at Nabble.com.

Re: [DISCUSSION] In 1.2.0, use Spark 2.1 and Hadoop 2.7.2 as default compilation in pom.

2017-06-16 Thread Aniket Adnaik 
+1 for Spark 2.1 and Hadoop 2.7.2 as default.

cheers,
Aniket

On Thu, Jun 15, 2017 at 8:42 PM, Liang Chen  wrote:

> Hi Dev
>
> In 1.2, there are many features developing  based on Spark 2.1 and Hadoop
> 2.7.2, so i propose to use Spark2.1 and Hadoop 2.7.2 as default compilation
> in pom.
>
> Please discuss and vote.
>
> Regards
> Liang
>

[jira] [Created] (CARBONDATA-1187) Fix Documentation links pointing to wrong urls in useful-tips-on-carbondata and faq

2017-06-16 Thread Jatin (JIRA) 
Jatin created CARBONDATA-1187:
-

 Summary: Fix Documentation links pointing to wrong urls in 
useful-tips-on-carbondata and faq 
 Key: CARBONDATA-1187
 URL: https://issues.apache.org/jira/browse/CARBONDATA-1187
 Project: CarbonData
  Issue Type: Bug
  Components: docs
Reporter: Jatin
Priority: Minor






--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (CARBONDATA-1186) Class Cast Exception In SortStepRowutil when loading data

2017-06-16 Thread anubhav tarar (JIRA) 
anubhav tarar created CARBONDATA-1186:
-

 Summary: Class Cast Exception In SortStepRowutil when loading data
 Key: CARBONDATA-1186
 URL: https://issues.apache.org/jira/browse/CARBONDATA-1186
 Project: CarbonData
  Issue Type: Bug
  Components: data-load
Affects Versions: 1.2.0
Reporter: anubhav tarar
Assignee: anubhav tarar
Priority: Trivial
 Fix For: 1.2.0






--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (CARBONDATA-1185) Job Aborted due to Stage Failure (In Select Query), when decimalField included to dictionary (for 10lakh data)

2017-06-16 Thread Rahul Kumar (JIRA) 
Rahul Kumar created CARBONDATA-1185:
---

 Summary: Job Aborted due to Stage Failure (In Select Query), when 
decimalField included to dictionary (for 10lakh data)
 Key: CARBONDATA-1185
 URL: https://issues.apache.org/jira/browse/CARBONDATA-1185
 Project: CarbonData
  Issue Type: Bug
Reporter: Rahul Kumar


*Query to reproduce : *

CREATE TABLE IF NOT EXISTS flow_carbon_test4
(
col1 String,
col2 String,
col3 String,
col4 String,
col5 String,
col6 String,
col7 String,
col8 String,
col9 String,
col10 String,
col11 String,
col12 String,
col13 String,
col14 String,
col15 String,
col16 String,
col17 int,
col18 int,
col19 String,
col20 String,
col21 String,
col22 String,
col23 String,
col24 String,
col25 String,
col26 String,
col27 String,
col28 String,
col29 String,
col30 String,
col31 String,
col32 String,
col33 String,
col34 String,
col35 String,
col36 String,
col37 String,
col38 String,
col39 String,
col40 String,
col41 String,
col42 String,
col43 String,
col44 String,
col45 String,
col46 String,
col47 String,
col48 String,
col49 String,
col50 decimal(15,2),
col51 decimal(15,2),
col52 String,
col53 String,
col54 String,
col55 String,
col56 String,
col57 String,
col58 String,
col59 String,
col60 String,
col61 String,
col62 String,
val_dte String,
opp_ac_flg String,
cmb_flg String,
ass_vch_flg String,
col63 String,
col64 String,
vch_bus_rmk String,
tec_rmk_cde String,
vch_tec_rmk String,
rsv_ara String,
col65 String,
col66 String,
col67 String,
col68 String
)
STORED BY 'org.apache.carbondata.format'
TBLPROPERTIES('Dictionary_Include'= 'col50,col51');

 LOAD DATA  inpath 'hdfs://hacluster/BabuStore/cmbc/10LakhData.csv' into table 
flow_carbon_test4 options('DELIMITER'=',', 
'QUOTECHAR'='"','FILEHEADER'='aco_ac,ac_dte,txn_cnt,jrn_par,mfm_jrn_no,cbn_jrn_no,ibs_jrn_no,vch_no,vch_seq,srv_cde,cus_no,bus_cd_no,id_flg,cus_ac,bv_cde,bv_no,txn_dte,txn_time,txn_tlr,txn_bk,txn_br,ety_tlr,ety_bk,ety_br,bus_pss_no,chk_flg,chk_tlr,chk_jrn_no,bus_sys_no,bus_opr_cde,txn_sub_cde,fin_bus_cde,fin_bus_sub_cde,opt_prd_cde,chl,tml_id,sus_no,sus_seq,cho_seq,itm_itm,itm_sub,itm_sss,dc_flg,amt,bal,ccy,spv_flg,vch_vld_dte,pst_bk,pst_br,ec_flg,aco_tlr,opp_ac,opp_ac_nme,opp_bk,gen_flg,his_rec_sum_flg,his_flg,vch_typ,val_dte,opp_ac_flg,cmb_flg,ass_vch_flg,cus_pps_flg,bus_rmk_cde,vch_bus_rmk,tec_rmk_cde,vch_tec_rmk,rsv_ara,own_br,own_bk,gems_last_upd_d,gems_last_upd_d_bat,maps_date,maps_job,dt');

Select * from flow_carbon_test4;






--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

答复: can't apply mappartitions to dataframe generated from carboncontext

2017-06-16 Thread Wangbin (Ben, BigData Design Dept) 
Hi all，
When view or debug carbondata code,shoud I choose windows + IntelliJ  
develop enviorment,the Hadoop is install in a linux machine.Or I must use 
linxu+ IntelliJ,and Hadoop is install in the linux .

-邮件原件-
发件人: Erlu Chen [mailto:chenerl...@gmail.com] 
发送时间: 2017年6月16日 10:39
收件人: dev@carbondata.apache.org
主题: Re: can't apply mappartitions to dataframe generated from carboncontext

Hi,

I check in my IntelliJ, but can not reproduce your issue, please check whether 
it is environment problem ?

Test steps as follows, correct me if I made a mistake


 

Regards.
Chenerlu.



--
View this message in context: 
http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/can-t-apply-mappartitions-to-dataframe-generated-from-carboncontext-tp14565p15277.html
Sent from the Apache CarbonData Dev Mailing List archive mailing list archive 
at Nabble.com.

[jira] [Created] (CARBONDATA-1184) Incorrect value displays in double data type.

2017-06-16 Thread Vinod Rohilla (JIRA) 
Vinod Rohilla created CARBONDATA-1184:
-

 Summary: Incorrect value displays in double data type. 
 Key: CARBONDATA-1184
 URL: https://issues.apache.org/jira/browse/CARBONDATA-1184
 Project: CarbonData
  Issue Type: Bug
  Components: sql
 Environment: Spark 2.1
Reporter: Vinod Rohilla
Priority: Minor
 Attachments: 100_olap_C20.csv

Incorrect value displays to the user in double datatype.

Step to reproduces:

1:Create table:

create table VMALL_DICTIONARY_EXCLUDE (imei string,deviceInformationId int,MAC 
string,deviceColor string,device_backColor string,modelId string,marketName 
string,AMSize string,ROMSize string,CUPAudit string,CPIClocked string,series 
string,productionDate timestamp,bomCode string,internalModels string, 
deliveryTime string, channelsId string, channelsName string , deliveryAreaId 
string, deliveryCountry string, deliveryProvince string, deliveryCity 
string,deliveryDistrict string, deliveryStreet string, oxSingleNumber string, 
ActiveCheckTime string, ActiveAreaId string, ActiveCountry string, 
ActiveProvince string, Activecity string, ActiveDistrict string, ActiveStreet 
string, ActiveOperatorId string, Active_releaseId string, Active_EMUIVersion 
string, Active_operaSysVersion string, Active_BacVerNumber string, 
Active_BacFlashVer string, Active_webUIVersion string, Active_webUITypeCarrVer 
string,Active_webTypeDataVerNumber string, Active_operatorsVersion string, 
Active_phonePADPartitionedVersions string, Latest_YEAR int, Latest_MONTH int, 
Latest_DAY Decimal(30,10), Latest_HOUR string, Latest_areaId string, 
Latest_country string, Latest_province string, Latest_city string, 
Latest_district string, Latest_street string, Latest_releaseId string, 
Latest_EMUIVersion string, Latest_operaSysVersion string, Latest_BacVerNumber 
string, Latest_BacFlashVer string, Latest_webUIVersion string, 
Latest_webUITypeCarrVer string, Latest_webTypeDataVerNumber string, 
Latest_operatorsVersion string, Latest_phonePADPartitionedVersions string, 
Latest_operatorId string, gamePointDescription string,gamePointId 
double,contractNumber BigInt) STORED BY 'org.apache.carbondata.format' 
TBLPROPERTIES('DICTIONARY_EXCLUDE'='imei');

2:Load Data:

LOAD DATA INPATH 'hdfs://localhost:54310/100_olap_C20.csv' INTO table 
VMALL_DICTIONARY_EXCLUDE options ('DELIMITER'=',', 
'QUOTECHAR'='"','BAD_RECORDS_ACTION'='FORCE', 
'FILEHEADER'='imei,deviceInformationId,MAC,deviceColor,device_backColor,modelId,marketName,AMSize,ROMSize,CUPAudit,CPIClocked,series,productionDate,bomCode,internalModels,deliveryTime,channelsId,channelsName,deliveryAreaId,deliveryCountry,deliveryProvince,deliveryCity,deliveryDistrict,deliveryStreet,oxSingleNumber,contractNumber,ActiveCheckTime,ActiveAreaId,ActiveCountry,ActiveProvince,Activecity,ActiveDistrict,ActiveStreet,ActiveOperatorId,Active_releaseId,Active_EMUIVersion,Active_operaSysVersion,Active_BacVerNumber,Active_BacFlashVer,Active_webUIVersion,Active_webUITypeCarrVer,Active_webTypeDataVerNumber,Active_operatorsVersion,Active_phonePADPartitionedVersions,Latest_YEAR,Latest_MONTH,Latest_DAY,Latest_HOUR,Latest_areaId,Latest_country,Latest_province,Latest_city,Latest_district,Latest_street,Latest_releaseId,Latest_EMUIVersion,Latest_operaSysVersion,Latest_BacVerNumber,Latest_BacFlashVer,Latest_webUIVersion,Latest_webUITypeCarrVer,Latest_webTypeDataVerNumber,Latest_operatorsVersion,Latest_phonePADPartitionedVersions,Latest_operatorId,gamePointId,gamePointDescription');

3: Run Select Query.
select gamePointId from VMALL_DICTIONARY_EXCLUDE;

4: Result:
0: jdbc:hive2://localhost:1> select gamePointId from 
VMALL_DICTIONARY_EXCLUDE;
+---+--+
|  gamePointId  |
+---+--+
| 9.223372036854776E18  |
| 9.223372036854776E18  |
| 9.223372036854776E18  |
| 9.223372036854776E18  |
| 9.223372036854776E18  |
| 9.223372036854776E18  |
| 9.223372036854776E18  |
| 9.223372036854776E18  |
| 9.223372036854776E18  |
| 9.223372036854776E18  |
| 9.223372036854776E18  |
| 9.223372036854776E18  |
| 9.223372036854776E18  |
| 9.223372036854776E18  |
| 9.223372036854776E18  |
| 9.223372036854776E18  |
| 9.223372036854776E18  |
| 9.223372036854776E18  |
| 9.223372036854776E18  |
| 9.223372036854776E18  |
| 9.223372036854776E18  |
| 9.223372036854776E18  |
| 9.223372036854776E18  |
| 9.223372036854776E18  |
| 9.223372036854776E18  |
| 9.223372036854776E18  |
| 9.223372036854776E18  |
| 9.223372036854776E18  |
| 9.223372036854776E18  |
| 9.223372036854776E18  |
| 9.223372036854776E18  |
| 9.223372036854776E18  |
| 9.223372036854776E18  |
| 9.223372036854776E18  |
| 9.223372036854776E18  |
| 9.223372036854776E18  |
| 9.223372036854776E18  |
| 9.223372036854776E18  |
| 9.223372036854776E18  |
| 9.223372036854776E18  |
| 9.223372036854776E18  |
| 9.223372036854776E18  |
| 9.223372036854776E18  |
| 9.223372036854776E18  |
|

[jira] [Created] (CARBONDATA-1183) Update CarbonPartitionTable Because partition columns should not be specified in the schema

2017-06-16 Thread chenerlu (JIRA) 
chenerlu created CARBONDATA-1183:


 Summary: Update CarbonPartitionTable Because partition columns 
should not be specified in the schema
 Key: CARBONDATA-1183
 URL: https://issues.apache.org/jira/browse/CARBONDATA-1183
 Project: CarbonData
  Issue Type: Bug
Reporter: chenerlu
Assignee: chenerlu
Priority: Minor






--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (CARBONDATA-1182) Error for packaging carbondata-preto

2017-06-16 Thread lonly (JIRA) 
lonly created CARBONDATA-1182:
-

 Summary: Error for packaging carbondata-preto
 Key: CARBONDATA-1182
 URL: https://issues.apache.org/jira/browse/CARBONDATA-1182
 Project: CarbonData
  Issue Type: Bug
  Components: presto-integration
Affects Versions: 1.2.0
Reporter: lonly
 Attachments: 微信图片_20170616151720.png, 微信图片_20170616151731.png

When I mvn package the carbondata-presto-master, I encountered this problem 
about “ncompatible types: 
org.apache.carbondata.core.metadata.ColumnarFormatVersion cannot be converted 
to org.apache.carbondata.core.datastore.block.BlockletInfos”






--
This message was sent by Atlassian JIRA
(v6.4.14#64029)