Re: Removing of kettle code from Carbondata

2017-03-14 Thread 金铸 

+1


在 2017/3/11 9:51, Ravindra Pesala 写道:

Hi All,

I guess it is time to remove the kettle flow from Carbondata loading. Now
there are two flows to load the data and becomes difficult to maintain the
code.Bug fixing or any feature implementation needs to be done in both the
places so it becomes difficult for developer to implement and test.

Please comment and vote on it.



--
金铸
技术发展部（TDD）
东软集团股份有限公司
沈阳浑南新区新秀街2号东软软件园A2-105A
Postcode:110179
Tel: (86 24)8366 2049
Mobile：13897999526

 




---
Confidentiality Notice: The information contained in this e-mail and any 
accompanying attachment(s)
is intended only for the use of the intended recipient and may be confidential 
and/or privileged of
Neusoft Corporation, its subsidiaries and/or its affiliates. If any reader of 
this communication is
not the intended recipient, unauthorized use, forwarding, printing,  storing, 
disclosure or copying
is strictly prohibited, and may be unlawful.If you have received this 
communication in error,please
immediately notify the sender by return e-mail, and delete the original message 
and all copies from
your system. Thank you.
---

Re: Please vote and advise on building thrift files

2016-11-16 Thread 金铸 

+1 for proposal 1


在 2016/11/17 12:13, 邢冰 写道:

+1 for proposal 1

thx




发自网易邮箱大师
On 11/17/2016 12:09, Ravindra Pesala wrote:
+1 for proposal 1

On 17 November 2016 at 08:23, Xiaoqiao He  wrote:


+1 for proposal 1.

On Thu, Nov 17, 2016 at 10:31 AM, ZhuWilliam 
wrote:


+1 for proposal 1 .

Auto generated code should not be added to project. Also most the of time
,people who dive into carbondata may not touch format code.



--
View this message in context: http://apache-carbondata-
mailing-list-archive.1130556.n5.nabble.com/Please-vote-and-
advise-on-building-thrift-files-tp2952p2957.html
Sent from the Apache CarbonData Mailing List archive mailing list archive
at Nabble.com.




--
Thanks & Regards,
Ravi


 




---
Confidentiality Notice: The information contained in this e-mail and any 
accompanying attachment(s)
is intended only for the use of the intended recipient and may be confidential 
and/or privileged of
Neusoft Corporation, its subsidiaries and/or its affiliates. If any reader of 
this communication is
not the intended recipient, unauthorized use, forwarding, printing,  storing, 
disclosure or copying
is strictly prohibited, and may be unlawful.If you have received this 
communication in error,please
immediately notify the sender by return e-mail, and delete the original message 
and all copies from
your system. Thank you.
---

Re: [VOTE] Apache CarbonData 0.2.0-incubating release

2016-11-10 Thread 金铸 

+1 bingding


jinzhu


在 2016/11/11 0:33, Jacky Li 写道:

+1 binding

Regards,
Jacky

---Original---
From: "Aniket Adnaik"
Date: 2016/11/10 14:43:49
To: 
"dev";"chenliang613";
Subject: Re: [VOTE] Apache CarbonData 0.2.0-incubating release


+1

Regards,
Aniket

On 9 Nov 2016 3:17 p.m., "Liang Chen"  wrote:


Hi all,

I submit the CarbonData 0.2.0-incubating to your vote.

Release Notes:
https://issues.apache.org/jira/secure/ReleaseNote.jspa?
projectId=12320220=12337896

Staging Repository:
https://repository.apache.org/content/repositories/
orgapachecarbondata-1006

Git Tag:
carbondata-0.2.0-incubating

Please vote to approve this release:
[ ] +1 Approve the release
[ ] -1 Don't approve the release (please provide specific comments)

This vote will be open for at least 72 hours. If this vote passes (we need
at least 3 binding votes, meaning three votes from the PPMC), I will
forward to gene...@incubator.apache.org for  the IPMC votes.

Here is my vote : +1 (binding)

Regards
Liang

>


 




---
Confidentiality Notice: The information contained in this e-mail and any 
accompanying attachment(s)
is intended only for the use of the intended recipient and may be confidential 
and/or privileged of
Neusoft Corporation, its subsidiaries and/or its affiliates. If any reader of 
this communication is
not the intended recipient, unauthorized use, forwarding, printing,  storing, 
disclosure or copying
is strictly prohibited, and may be unlawful.If you have received this 
communication in error,please
immediately notify the sender by return e-mail, and delete the original message 
and all copies from
your system. Thank you.
---

Re: [Discuss]Set block_size for table on table level

2016-09-26 Thread 金铸 

+1，agree with jihong.


在 2016/9/27 5:12, chenliang613 写道:

+1, agree with Jihong's comment : make it as optional, usually the default
block size will be used if user don't specially define it.

Regards
Liang


Jihong Ma wrote

+1, To avoid potential compatibility issue, we could introduce this param
as an optional field, as long as it is not a required field, we are fine
with a defined default block size.

Regards.

Jihong

-Original Message-
From: Jacky Li [mailto:
jacky.likun@
]
Sent: Monday, September 26, 2016 7:29 AM
To:
dev@.apache
Subject: Re: [Discuss]Set block_size for table on table level

I am OK with this feature, the only thing I am worrying about is the
compatibility of CarbonData file reader. Can you make it compatible when
you reading old CarbonData file without this property.
We have encountered many times that user need to delete the store and
re-load the data.

Regards,
Jacky


在 2016年9月26日，下午2:15，Ravindra Pesala 

ravi.pesala@
 写道：

+1
At same time max and min block size should be restricted and validated
while creating table.

On 26 September 2016 at 07:36, Zhangshunyu 

zhangshunyu1990@
 wrote:

Purpose:
To configure block file size for each table on column level, so that
each
table could has its own blocksize.
My solution:
Add a new parameter in table properties, when create a table, the user
can
set it in ddl. Add a parameter in thrift format just like other
properties,
and write this info into thrift file so that this info would not lost
when
cluster is restarted.

What's your opinion?



--
View this message in context: http://apache-carbondata-
mailing-list-archive.1130556.n5.nabble.com/Discuss-Set-
block-size-for-table-on-table-level-tp1472.html
Sent from the Apache CarbonData Mailing List archive mailing list
archive
at Nabble.com.



--
Thanks & Regards,
Ravi





--


  



---
Confidentiality Notice: The information contained in this e-mail and any 
accompanying attachment(s)
is intended only for the use of the intended recipient and may be confidential 
and/or privileged of
Neusoft Corporation, its subsidiaries and/or its affiliates. If any reader of 
this communication is
not the intended recipient, unauthorized use, forwarding, printing,  storing, 
disclosure or copying
is strictly prohibited, and may be unlawful.If you have received this 
communication in error,please
immediately notify the sender by return e-mail, and delete the original message 
and all copies from
your system. Thank you.
---

Re: [VOTE] Apache CarbonData 0.1.0-incubating release

2016-08-23 Thread 金铸 

+1


在 2016/8/20 2:57, Jean-Baptiste Onofré 写道:

Hi all,

I submit the first CarbonData release to your vote.

Release Notes:
https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12320220=12337895 



Staging Repository:
https://repository.apache.org/content/repositories/orgapachecarbondata-1000/ 



Git Tag:
carbondata-0.1.0-incubating

Please vote to approve this release:

[ ] +1 Approve the release
[ ] -1 Don't approve the release (please provide specific comments)

This vote will be open for at least 72 hours.

If this vote passes (we need at least 3 binding votes, meaning three 
votes from the PPMC), I will forward to gene...@incubator.apache.org 
for the IPMC votes.


Thanks
Regards
JB


--
金铸
技术发展部（TDD）
东软集团股份有限公司
沈阳浑南新区新秀街2号东软软件园A2-105A
Postcode:110179
Tel: (86 24)8366 2049
Mobile：13897999526

 




---
Confidentiality Notice: The information contained in this e-mail and any 
accompanying attachment(s)
is intended only for the use of the intended recipient and may be confidential 
and/or privileged of
Neusoft Corporation, its subsidiaries and/or its affiliates. If any reader of 
this communication is
not the intended recipient, unauthorized use, forwarding, printing,  storing, 
disclosure or copying
is strictly prohibited, and may be unlawful.If you have received this 
communication in error,please
immediately notify the sender by return e-mail, and delete the original message 
and all copies from
your system. Thank you.
---

Re: Open discussion and Vote: What kind of JIRA issue events need send mail to dev@carbondata.incubator.apache.org

2016-08-18 Thread 金铸 

Option2：+1


在 2016/8/18 15:57, Liang Big data 写道:

Hi all

Please discuss and vote, do you think what kind of JIRA issue events need
send mails to dev@carbondata.incubator.apache.org?

Option1: None, all JIRA issue events don't need send any mails to
dev@carbondata.incubator.apache.org, users can directly go to apache jira
to check issues' content.

Option2:Issue Created and Issue Commented, the two JIRA issue events send
mails to dev@carbondata.incubator.apache.org

Option3:Keep the current notification schema, the below events send mails
to dev@carbondata.incubator.apache.org.

or any other option?

The below is current notification schema
EventsNotifications
Issue Created

- All Watchers
- Current Assignee
- Reporter
- Single Email Address (dev@carbondata.incubator.apache.org)

Issue Updated

- All Watchers
- Current Assignee
- Reporter
- Single Email Address (dev@carbondata.incubator.apache.org)

Issue Assigned

- All Watchers
- Current Assignee
- Reporter
- Single Email Address (dev@carbondata.incubator.apache.org)

Issue Resolved

- All Watchers
- Current Assignee
- Reporter
- Single Email Address (dev@carbondata.incubator.apache.org)

Issue Closed

- All Watchers
- Current Assignee
- Reporter
- Single Email Address (dev@carbondata.incubator.apache.org)

Issue Commented

- All Watchers
- Current Assignee
- Reporter
- Single Email Address (dev@carbondata.incubator.apache.org)

Issue Comment Edited

- All Watchers
- Current Assignee
- Reporter
- Single Email Address (dev@carbondata.incubator.apache.org)

Issue Comment Deleted

- All Watchers
- Current Assignee
- Reporter
- Single Email Address (dev@carbondata.incubator.apache.org)

Issue Reopened

- All Watchers
- Current Assignee
- Reporter
- Single Email Address (dev@carbondata.incubator.apache.org)

Issue Deleted

- All Watchers
- Current Assignee
- Reporter
- Single Email Address (dev@carbondata.incubator.apache.org)

Issue Moved

- Single Email Address (dev@carbondata.incubator.apache.org)



Regards
Liang



 




---
Confidentiality Notice: The information contained in this e-mail and any 
accompanying attachment(s)
is intended only for the use of the intended recipient and may be confidential 
and/or privileged of
Neusoft Corporation, its subsidiaries and/or its affiliates. If any reader of 
this communication is
not the intended recipient, unauthorized use, forwarding, printing,  storing, 
disclosure or copying
is strictly prohibited, and may be unlawful.If you have received this 
communication in error,please
immediately notify the sender by return e-mail, and delete the original message 
and all copies from
your system. Thank you.
---

Re: 答复: load data fail

2016-08-17 Thread 金铸 


i drop table，use $hdc_home/hive/conf/hive-sie.xml replace 
$hdc_home/spark/conf/hive-site.xml,fixed it.


but i do not know the principle inside.


if t4 is exist in hive's defualt,in other words create table t4 in 
hive,then create table in carbondata do not reported exception。





在 2016/8/17 10:35, Chenliang (Liang, CarbonData) 写道:

Can you share the case experience: how did you solve it.

Regards
Liang
-邮件原件-
发件人: 金铸 [mailto:jin...@neusoft.com]
发送时间: 2016年8月17日 10:31
收件人: dev@carbondata.incubator.apache.org
主题: Re: load data fail

thanks a lot，I  solve this。



在 2016/8/17 0:53, Eason 写道:

hi jinzhu,

whether this happen on multiple instance loading the same table?

currently ,it is no support concurrent load on same table.

for this exception

1.please check if any locks are created under system temp folder
with//lockfile, if it exists please delete.

2.try to change the lock ype:
carbon.lock.type =  ZOOKEEPERLOCK Regards, Eason

在 2016年08月12日 14:25, 金铸 写道:

hi ： /usr/hdp/2.4.0.0-169/spark/bin/spark-shell --master yarn-client
--jars
/opt/incubator-carbondata/assembly/target/scala-2.10/carbondata_2.10-
0.1.0-incubating-SNAPSHOT-shade-hadoop2.2.0.jar,/usr/hdp/2.4.0.0-169/
spark/lib/datanucleus-api-jdo-3.2.6.jar,/usr/hdp/2.4.0.0-169/spark/li
b/datanucleus-rdbms-3.2.9.jar,/usr/hdp/2.4.0.0-169/spark/lib/datanucl
eus-core-3.2.10.jar,/opt//mysql-connector-java-5.1.37.jar
scala>import org.apache.spark.sql.CarbonContext scala>import
java.io.File scala>import org.apache.hadoop.hive.conf.HiveConf
scala>val cc = new CarbonContext(sc,
"hdfs://hadoop01/data/carbondata01/store")
scala>cc.setConf("hive.metastore.warehouse.dir",
"/apps/hive/warehouse")
scala>cc.setConf(HiveConf.ConfVars.HIVECHECKFILEFORMAT.varname,
"false")
scala>cc.setConf("carbon.kettle.home","/usr/hdp/2.4.0.0-169/spark/car
scala>bonlib/carbonplugins")
scala>
cc.setConf("carbon.kettle.home","/usr/hdp/2.4.0.0-169/spark/carbonlib
/carbonplugins")
scala> cc.sql(s"load data local inpath 'hdfs://hadoop01/sample.csv'
into table t4 options('FILEHEADER'='id,name,city,age')") INFO  12-08
14:21:24,461 - main Query [LOAD DATA LOCAL INPATH
'HDFS://HADOOP01/SAMPLE.CSV' INTO TABLE T4
OPTIONS('FILEHEADER'='ID,NAME,CITY,AGE')] INFO  12-08 14:21:39,475 -
Table MetaData Unlocked Successfully after data load
java.lang.RuntimeException: Table is locked for updation. Please try
after some time at scala.sys.package$.error(package.scala:27)
at
org.apache.spark.sql.execution.command.LoadTable.run(carbonTableSchema.scala:1049)
 at
org.apache.spark.sql.execution.ExecutedCommand.sideEffectResult$lzycompute(commands.scala:58)
 at
org.apache.spark.sql.execution.ExecutedCommand.sideEffectResult(commands.scala:56)
 at
org.apache.spark.sql.execution.ExecutedCommand.doExecute(commands.scala:70)
 at
org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$5.apply(SparkPlan.scala:132)
 at
org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$5.apply(SparkPlan.scala:130)
 at
org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.s
cala:150)
thanks a lot
-





  



---
Confidentiality Notice: The information contained in this e-mail and any 
accompanying attachment(s)
is intended only for the use of the intended recipient and may be confidential 
and/or privileged of
Neusoft Corporation, its subsidiaries and/or its affiliates. If any reader of 
this communication is
not the intended recipient, unauthorized use, forwarding, printing,  storing, 
disclosure or copying
is strictly prohibited, and may be unlawful.If you have received this 
communication in error,please
immediately notify the sender by return e-mail, and delete the original message 
and all copies from
your system. Thank you.
---

load data fail

2016-08-12 Thread 金铸 

hi ：
/usr/hdp/2.4.0.0-169/spark/bin/spark-shell --master yarn-client --jars 
/opt/incubator-carbondata/assembly/target/scala-2.10/carbondata_2.10-0.1.0-incubating-SNAPSHOT-shade-hadoop2.2.0.jar,/usr/hdp/2.4.0.0-169/spark/lib/datanucleus-api-jdo-3.2.6.jar,/usr/hdp/2.4.0.0-169/spark/lib/datanucleus-rdbms-3.2.9.jar,/usr/hdp/2.4.0.0-169/spark/lib/datanucleus-core-3.2.10.jar,/opt//mysql-connector-java-5.1.37.jar


scala>import org.apache.spark.sql.CarbonContext
scala>import java.io.File
scala>import org.apache.hadoop.hive.conf.HiveConf





scala>val cc = new CarbonContext(sc, 
"hdfs://hadoop01/data/carbondata01/store")


scala>cc.setConf("hive.metastore.warehouse.dir", "/apps/hive/warehouse")
scala>cc.setConf(HiveConf.ConfVars.HIVECHECKFILEFORMAT.varname, "false")
scala>cc.setConf("carbon.kettle.home","/usr/hdp/2.4.0.0-169/spark/carbonlib/carbonplugins")

scala> 
cc.setConf("carbon.kettle.home","/usr/hdp/2.4.0.0-169/spark/carbonlib/carbonplugins")


scala> cc.sql(s"load data local inpath 'hdfs://hadoop01/sample.csv' into 
table t4 options('FILEHEADER'='id,name,city,age')")
INFO  12-08 14:21:24,461 - main Query [LOAD DATA LOCAL INPATH 
'HDFS://HADOOP01/SAMPLE.CSV' INTO TABLE T4 
OPTIONS('FILEHEADER'='ID,NAME,CITY,AGE')]
INFO  12-08 14:21:39,475 - Table MetaData Unlocked Successfully after 
data load
java.lang.RuntimeException: Table is locked for updation. Please try 
after some time

at scala.sys.package$.error(package.scala:27)
at 
org.apache.spark.sql.execution.command.LoadTable.run(carbonTableSchema.scala:1049)
at 
org.apache.spark.sql.execution.ExecutedCommand.sideEffectResult$lzycompute(commands.scala:58)
at 
org.apache.spark.sql.execution.ExecutedCommand.sideEffectResult(commands.scala:56)
at 
org.apache.spark.sql.execution.ExecutedCommand.doExecute(commands.scala:70)
at 
org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$5.apply(SparkPlan.scala:132)
at 
org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$5.apply(SparkPlan.scala:130)
at 
org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:150)


thanks a lot


---
Confidentiality Notice: The information contained in this e-mail and any 
accompanying attachment(s)
is intended only for the use of the intended recipient and may be confidential 
and/or privileged of
Neusoft Corporation, its subsidiaries and/or its affiliates. If any reader of 
this communication is
not the intended recipient, unauthorized use, forwarding, printing,  storing, 
disclosure or copying
is strictly prohibited, and may be unlawful.If you have received this 
communication in error,please
immediately notify the sender by return e-mail, and delete the original message 
and all copies from
your system. Thank you.
---

Re: Discussing to add carbondata-tools module

2016-08-05 Thread 金铸 

+1


在 2016/8/5 17:15, Jean-Baptiste Onofré 写道:
I guess it's where we can put the checkstyle, ... (what we have in the 
dev folder right now), correct ?


Regards
JB

On 08/05/2016 08:31 AM, QiangCai wrote:

Hi all,

  To improve the CarbonData system's usability and maintainability, I
suggest to add carbondata-tools module.
  I think this module should provide some command tools as following.

  1. import
  import a data file/folder to any existing table

  2. export
  export the given columns to a file

  3. schema
  show the detail information of the specified table schema file

  4. metadata
  show tablestatus metadata
  show the history track for dataloading and compaction

  5. footer
  show blocklet metadata list
  show start/end key, min/max value, row number, total size  for the
specified blocklet

  6. blocklet
  show blocket list
  show blocket data, RLE map, Inverted index for the given columns

  7. index
  show BTree node list
  show node information:start/end key, min/max value)

  8. dictionary
  show  key-value list of specified gloabl/local dictionary file
  show sort index
  show dictionary metadata

Thank you and look forward to having your opinion on this carbon-tools
module.

David Cai



--
View this message in context: 
http://apache-carbondata-mailing-list-archive.1130556.n5.nabble.com/Discussing-to-add-carbondata-tools-module-tp4.html
Sent from the Apache CarbonData Mailing List archive mailing list 
archive at Nabble.com.






--
金铸
技术发展部（TDD）
东软集团股份有限公司
沈阳浑南新区新秀街2号东软软件园A2-105A
Postcode:110179
Tel: (86 24)8366 2049
Mobile：13897999526

 




---
Confidentiality Notice: The information contained in this e-mail and any 
accompanying attachment(s)
is intended only for the use of the intended recipient and may be confidential 
and/or privileged of
Neusoft Corporation, its subsidiaries and/or its affiliates. If any reader of 
this communication is
not the intended recipient, unauthorized use, forwarding, printing,  storing, 
disclosure or copying
is strictly prohibited, and may be unlawful.If you have received this 
communication in error,please
immediately notify the sender by return e-mail, and delete the original message 
and all copies from
your system. Thank you.
---

Re: load data error

2016-07-30 Thread 金铸 

hi Liang：

   Thanks your repay。

   I have already used the “/opt/incubator-carbondata/sample.csv” 
，Reported the same error。



在 2016/7/30 22:44, Liang Big data 写道:

Hi jinzhu金铸:


please check the below error:the input path having some issues.
Please use the absolute path to try it again.
---
ERROR 29-07 16:39:46,904 - main
org.apache.hadoop.mapreduce.lib.input.InvalidInputException: Input path
does not exist: /opt/incubator-carbondata/sample.csv

Regards
Liang

2016-07-29 8:47 GMT+08:00 金铸 <jin...@neusoft.com>:


[hadoop@slave2 ~]$ cat /opt/incubator-carbondata/sample.csv
id,name,city,age
1,david,shenzhen,31
2,eason,shenzhen,27
3,jarry,wuhan,35
[hadoop@slave2 ~]$

> load data inpath '../sample.csv' into table test_table;
INFO  29-07 16:39:46,087 - main Property file path:
/opt/incubator-carbondata/bin/../../../conf/carbon.properties
INFO  29-07 16:39:46,087 - main --Using Carbon.properties 
INFO  29-07 16:39:46,087 - main {}
INFO  29-07 16:39:46,088 - main Query [LOAD DATA INPATH '../SAMPLE.CSV'
INTO TABLE TEST_TABLE]
INFO  29-07 16:39:46,527 - Successfully able to get the table metadata
file lock
INFO  29-07 16:39:46,537 - main Initiating Direct Load for the Table :
(default.test_table)
INFO  29-07 16:39:46,541 - Generate global dictionary from source data
files!
INFO  29-07 16:39:46,569 - [Block Distribution]
INFO  29-07 16:39:46,569 - totalInputSpaceConsumed : 74 ,
defaultParallelism : 6
INFO  29-07 16:39:46,569 - mapreduce.input.fileinputformat.split.maxsize :
16777216
INFO  29-07 16:39:46,689 - Block broadcast_0 stored as values in memory
(estimated size 232.6 KB, free 232.6 KB)
INFO  29-07 16:39:46,849 - Block broadcast_0_piece0 stored as bytes in
memory (estimated size 19.7 KB, free 252.3 KB)
INFO  29-07 16:39:46,850 - Added broadcast_0_piece0 in memory on
192.168.241.223:41572 (size: 19.7 KB, free: 511.5 MB)
INFO  29-07 16:39:46,856 - Created broadcast 0 from NewHadoopRDD at
CarbonTextFile.scala:45
ERROR 29-07 16:39:46,904 - generate global dictionary failed
ERROR 29-07 16:39:46,904 - main
org.apache.hadoop.mapreduce.lib.input.InvalidInputException: Input path
does not exist: /opt/incubator-carbondata/sample.csv
 at
org.apache.hadoop.mapreduce.lib.input.FileInputFormat.singleThreadedListStatus(FileInputFormat.java:321)
 at
org.apache.hadoop.mapreduce.lib.input.FileInputFormat.listStatus(FileInputFormat.java:264)
 at
org.apache.hadoop.mapreduce.lib.input.FileInputFormat.getSplits(FileInputFormat.java:385)
 at
org.apache.spark.rdd.NewHadoopRDD.getPartitions(NewHadoopRDD.scala:120)
 at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:239)
 at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:237)
 at scala.Option.getOrElse(Option.scala:120)
 at org.apache.spark.rdd.RDD.partitions(RDD.scala:237)
 at
org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:35)
 at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:239)
 at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:237)
 at scala.Option.getOrElse(Option.scala:120)
 at org.apache.spark.rdd.RDD.partitions(RDD.scala:237)
 at org.apache.spark.rdd.RDD$$anonfun$take$1.apply(RDD.scala:1307)
 at
org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:150)
 at
org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:111)
 at org.apache.spark.rdd.RDD.withScope(RDD.scala:316)
 at org.apache.spark.rdd.RDD.take(RDD.scala:1302)
 at
com.databricks.spark.csv.CarbonCsvRelation.firstLine$lzycompute(CarbonCsvRelation.scala:175)
 at
com.databricks.spark.csv.CarbonCsvRelation.firstLine(CarbonCsvRelation.scala:170)
 at
com.databricks.spark.csv.CarbonCsvRelation.inferSchema(CarbonCsvRelation.scala:141)
 at
com.databricks.spark.csv.CarbonCsvRelation.(CarbonCsvRelation.scala:71)
 at
com.databricks.spark.csv.newapi.DefaultSource.createRelation(DefaultSource.scala:142)
 at
com.databricks.spark.csv.newapi.DefaultSource.createRelation(DefaultSource.scala:44)
 at
org.apache.spark.sql.execution.datasources.ResolvedDataSource$.apply(ResolvedDataSource.scala:158)
 at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:119)
 at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:109)
 at
org.carbondata.spark.util.GlobalDictionaryUtil$.loadDataFrame(GlobalDictionaryUtil.scala:365)
 at
org.carbondata.spark.util.GlobalDictionaryUtil$.generateGlobalDictionary(GlobalDictionaryUtil.scala:676)
 at
org.apache.spark.sql.execution.command.LoadTable.run(carbonTableSchema.scala:1159)
 at
org.apache.spark.sql.execution.ExecutedCommand.sideEffectResult$lzycompute(commands.scala:58)
 at
org.apache.spark.sql.execution.ExecutedCommand.sideEffectResult(commands.scala:56)
 at
org.apache.spark.sql.execution.ExecutedCommand.doExecute(commands

load data error

2016-07-30 Thread 金铸 
ame.(DataFrame.scala:145)
at org.apache.spark.sql.DataFrame.(DataFrame.scala:130)
at 
org.carbondata.spark.rdd.CarbonDataFrameRDD.(CarbonDataFrameRDD.scala:23)

at org.apache.spark.sql.CarbonContext.sql(CarbonContext.scala:131)
at 
org.apache.spark.sql.hive.thriftserver.SparkSQLDriver.run(SparkSQLDriver.scala:63)
at 
org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver.processCmd(SparkSQLCLIDriver.scala:311)

at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:376)
at 
org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver$.main(SparkSQLCLIDriver.scala:226)
at 
org.apache.spark.sql.hive.cli.CarbonSQLCLIDriver$.main(CarbonSQLCLIDriver.scala:40)
at 
org.apache.spark.sql.hive.cli.CarbonSQLCLIDriver.main(CarbonSQLCLIDriver.scala)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:606)
at 
org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:731)
at 
org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:181)

at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:206)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:121)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

--
金铸


 




---
Confidentiality Notice: The information contained in this e-mail and any 
accompanying attachment(s)
is intended only for the use of the intended recipient and may be confidential 
and/or privileged of
Neusoft Corporation, its subsidiaries and/or its affiliates. If any reader of 
this communication is
not the intended recipient, unauthorized use, forwarding, printing,  storing, 
disclosure or copying
is strictly prohibited, and may be unlawful.If you have received this 
communication in error,please
immediately notify the sender by return e-mail, and delete the original message 
and all copies from
your system. Thank you.
---