Re: How to specify file

2016-09-23 Thread Mich Talebzadeh
You can do the following with option("delimiter") .. val df = spark.read.option("header", false).option("delimiter","\t").csv("hdfs://rhes564:9000/tmp/nw_10124772.tsv") HTH Dr Mich Talebzadeh LinkedIn * https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw

?????? How to specify file

2016-09-23 Thread Sea
??9??23??(??) 3:32 ??: "Sea"<261810...@qq.com>; : "user"<user@spark.apache.org>; : Re: How to specify file Check out the READEME on the following page. This is the csv connector that you are using. I think you need to specify the delimiter

Re: How to specify file

2016-09-23 Thread Aditya
Hi Sea, For using Spark SQL you will need to create DataFrame from the file and then execute select * on dataframe. In your case you will need to do something like this JavaRDD DF = context.textFile("path"); JavaRDD rowRDD3 = DF.map(new Function() {

Re: How to specify file

2016-09-23 Thread Hemant Bhanawat
Check out the READEME on the following page. This is the csv connector that you are using. I think you need to specify the delimiter option. https://github.com/databricks/spark-csv Hemant Bhanawat www.snappydata.io On Fri, Sep 23, 2016 at

How to specify file

2016-09-23 Thread Sea
Hi, I want to run sql directly on files, I find that spark has supported sql like select * from csv.`/path/to/file`, but files may not be split by ','. Maybe it is split by '\001', how can I specify delimiter? Thank you!