please send an empty email to:
user-unsubscr...@spark.apache.org
to unsubscribe yourself from the list.
On Thu, Feb 9, 2023 at 12:38 PM fuwei...@163.com wrote:
> Unsubscribe
>
Hi Team,
I am running a query in Spark 3.2.
val df1 =
sc.parallelize(List((1,2,3,4,5),(1,2,3,4,5))).toDF("id","col2","col3","col4",
"col5")
val op_cols_same_case = List("id","col2","col3","col4", "col5", "id")
val df2 = df1.select(op_cols_same_case .head, op_cols_same_case .tail: _*)
Hi,
I remember previously that spark.sql() wasn’t a final action
and you would have needed to run something like show() for the query to
actually being performed. Today I noticed that when I do just spark.sql() without show() or anything , lots of executors are being fired and
reading their logs
It's been a few years (so this approach might be out of date) but here's
what I used for PySpark as part of this SO (
https://stackoverflow.com/questions/45717433/stop-structured-streaming-query-gracefully/65708677
)
```
# Helper method to stop a streaming query
def stop_stream_query(query,
