LuciferYang commented on PR #55410:
URL: https://github.com/apache/spark/pull/55410#issuecomment-5509321580
@HyukjinKwon
What changes is Python `foreachBatch` under Connect. The batch DataFrame now
belongs to the session `StreamExecution` clones for the query, so
session-scoped state created from it stops reaching the client session:
```python
def process(batch_df, batch_id):
batch_df.createOrReplaceTempView("v")
...
spark.sql("SELECT * FROM v") # before: resolves. after:
TABLE_OR_VIEW_NOT_FOUND
```
Scala and classic are unaffected; both already hand the user function the
clone. So Connect is catching up here, but working code does break. Not
hypothetically: `branch-4.0`'s `test_nested_dataframes` broke on it and needed
SPARK-59148.
master and `branch-4.x` can absorb that with a migration note. The reason to
go deeper is that on 4.0 through 4.2, whatever the `foreachBatch` body does
with the batch DataFrame is planned under the client session, without the
configs the engine sets on the clone, CBO and AQE among them. But going deeper
means changing session semantics inside a patch release, which I would rather
not do bare.
So: a conf selecting the session, new behavior on master and `branch-4.x`
and old on the maintenance branches? Or conf-free, stopping the backport at
`branch-4.x`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
