Scala and Python are not the same in this regard. This isn't related to how
spark works.
On Sun, Feb 6, 2022, 10:04 PM <capitnfrak...@free.fr> wrote:
> Indeed. in spark-shell I ignore the parentheses always,
>
> scala> sc.parallelize(List(3,2,1,4)).toDF.show
> +-----+
> |value|
> +-----+
> | 3|
> | 2|
> | 1|
> | 4|
> +-----+
>
> So I think it would be ok in pyspark.
>
> But this still doesn't work. why?
>
> >>> sc.parallelize([3,2,1,4]).toDF().show()
> Traceback (most recent call last):
> File "<stdin>", line 1, in <module>
> File "/opt/spark/python/pyspark/sql/session.py", line 66, in toDF
> return sparkSession.createDataFrame(self, schema, sampleRatio)
> File "/opt/spark/python/pyspark/sql/session.py", line 675, in
> createDataFrame
> return self._create_dataframe(data, schema, samplingRatio,
> verifySchema)
> File "/opt/spark/python/pyspark/sql/session.py", line 698, in
> _create_dataframe
> rdd, schema = self._createFromRDD(data.map(prepare), schema,
> samplingRatio)
> File "/opt/spark/python/pyspark/sql/session.py", line 486, in
> _createFromRDD
> struct = self._inferSchema(rdd, samplingRatio, names=schema)
> File "/opt/spark/python/pyspark/sql/session.py", line 466, in
> _inferSchema
> schema = _infer_schema(first, names=names)
> File "/opt/spark/python/pyspark/sql/types.py", line 1067, in
> _infer_schema
> raise TypeError("Can not infer schema for type: %s" % type(row))
> TypeError: Can not infer schema for type: <class 'int'>
>
>
> spark 3.2.0
>
>
> On 07/02/2022 11:44, Sean Owen wrote:
> > This is just basic Python - you're missing parentheses on toDF, so you
> > are not calling a function nor getting its result.
> >
> > On Sun, Feb 6, 2022 at 9:39 PM <capitnfrak...@free.fr> wrote:
> >
> >> I am a bit confused why in pyspark this doesn't work?
> >>
> >>>>> x = sc.parallelize([3,2,1,4])
> >>>>> x.toDF.show()
> >> Traceback (most recent call last):
> >> File "<stdin>", line 1, in <module>
> >> AttributeError: 'function' object has no attribute 'show'
> >>
> >> Thank you.
> >>
> >>
> > ---------------------------------------------------------------------
> >> To unsubscribe e-mail: user-unsubscr...@spark.apache.org
>
