Re: coalesce on dataFrame
You can use df.repartition(1) in Spark 1.4. See here <https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/DataFrame.scala#L1396> . Best, Burak On Wed, Jul 1, 2015 at 3:05 AM, Olivier Girardot wrote: > PySpark or Spark (scala) ? > When you use coalesce with anything but a column you must use a literal > like that in PySpark : > > from pyspark.sql import functions as F > > F.coalesce(df.a, F.lit(True)) > > Le mer. 1 juil. 2015 à 12:03, Ewan Leith a > écrit : > >> It's in spark 1.4.0, or should be at least: >> >> https://issues.apache.org/jira/browse/SPARK-6972 >> >> Ewan >> >> -Original Message- >> From: Hafiz Mujadid [mailto:hafizmujadi...@gmail.com] >> Sent: 01 July 2015 08:23 >> To: user@spark.apache.org >> Subject: coalesce on dataFrame >> >> How can we use coalesce(1, true) on dataFrame? >> >> >> Thanks >> >> >> >> -- >> View this message in context: >> http://apache-spark-user-list.1001560.n3.nabble.com/coalesce-on-dataFrame-tp23564.html >> Sent from the Apache Spark User List mailing list archive at Nabble.com. >> >> - >> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional >> commands, e-mail: user-h...@spark.apache.org >> >> >> - >> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org >> For additional commands, e-mail: user-h...@spark.apache.org >> >>
Re: coalesce on dataFrame
PySpark or Spark (scala) ? When you use coalesce with anything but a column you must use a literal like that in PySpark : from pyspark.sql import functions as F F.coalesce(df.a, F.lit(True)) Le mer. 1 juil. 2015 à 12:03, Ewan Leith a écrit : > It's in spark 1.4.0, or should be at least: > > https://issues.apache.org/jira/browse/SPARK-6972 > > Ewan > > -Original Message- > From: Hafiz Mujadid [mailto:hafizmujadi...@gmail.com] > Sent: 01 July 2015 08:23 > To: user@spark.apache.org > Subject: coalesce on dataFrame > > How can we use coalesce(1, true) on dataFrame? > > > Thanks > > > > -- > View this message in context: > http://apache-spark-user-list.1001560.n3.nabble.com/coalesce-on-dataFrame-tp23564.html > Sent from the Apache Spark User List mailing list archive at Nabble.com. > > - > To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional > commands, e-mail: user-h...@spark.apache.org > > > - > To unsubscribe, e-mail: user-unsubscr...@spark.apache.org > For additional commands, e-mail: user-h...@spark.apache.org > >
RE: coalesce on dataFrame
It's in spark 1.4.0, or should be at least: https://issues.apache.org/jira/browse/SPARK-6972 Ewan -Original Message- From: Hafiz Mujadid [mailto:hafizmujadi...@gmail.com] Sent: 01 July 2015 08:23 To: user@spark.apache.org Subject: coalesce on dataFrame How can we use coalesce(1, true) on dataFrame? Thanks -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/coalesce-on-dataFrame-tp23564.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
coalesce on dataFrame
How can we use coalesce(1, true) on dataFrame? Thanks -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/coalesce-on-dataFrame-tp23564.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org