I think what you might be looking for is the ability to programmatically specify the schema, which is coming in 1.1.
Here's the JIRA: SPARK-2179 <https://issues.apache.org/jira/browse/SPARK-2179> On Wed, Jul 16, 2014 at 8:24 AM, pandees waran <pande...@gmail.com> wrote: > Hi, > > I am newbie to spark sql and i would like to know about how to read all > the columns from a file in spark sql. I have referred the programming guide > here: > http://people.apache.org/~tdas/spark-1.0-docs/sql-programming-guide.html > > The example says: > > val people = > sc.textFile("examples/src/main/resources/people.txt").map(_.split(",")).map(p > => Person(p(0), p(1).trim.toInt)) > > But, instead of explicitly specifying p(0),p(1) I would like to read all > the columns from a file. It would be difficult if my source dataset has > more no of columns. > > Is there any shortcut for that? > > And instead of a single file, i would like to read multiple files which > shares a similar structure from a directory. > > Could you please share your thoughts on this? > > It would be great , if you share any documentation which has details on > these? > > Thanks >
