We can't drop the existing createDataFrame one, since it breaks API
compatibility, and the existing one also automatically infers the column
name for case classes (in that case users most likely won't be declaring
names directly). If this is really a problem, we should just create a new
function
I have the perfect counter example where some of the data scientists
prototype in Python and the production materials is done in Scala.
But I get your point, as a matter of fact I realised the toDF method took
parameters a little while after posting this.
However the toDF still needs you to go
Hi everyone,
SQLContext.createDataFrame has different behaviour in Scala or Python :
l = [('Alice', 1)]
sqlContext.createDataFrame(l).collect()
[Row(_1=u'Alice', _2=1)]
sqlContext.createDataFrame(l, ['name', 'age']).collect()
[Row(name=u'Alice', age=1)]
and in Scala :
scala val data =
Part of the reason is that it is really easy to just call toDF on Scala,
and we already have a lot of createDataFrame functions.
(You might find some of the cross-language differences confusing, but I'd
argue most real users just stick to one language, and developers or
trainers are the only ones
