This is an automated email from the ASF dual-hosted git repository.
wenchen pushed a commit to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.0 by this push:
new f0a27b5 [SPARK-28199][SPARK-28199][SS][FOLLOWUP] Mention the change
of into the SS migration guide
f0a27b5 is described below
commit f0a27b5488608bf4a4c100c4f26ffa3f7cd6b452
Author: Jungtaek Lim (HeartSaVioR) <kabhwan.opensou...@gmail.com>
AuthorDate: Tue Jun 9 04:52:48 2020 +0000
[SPARK-28199][SPARK-28199][SS][FOLLOWUP] Mention the change of into the SS
migration guide
### What changes were proposed in this pull request?
SPARK-28199 (#24996) made the trigger related public API to be exposed only
from static methods of Trigger class. This is backward incompatible change, so
some users may experience compilation error after upgrading to Spark 3.0.0.
While we plan to mention the change into release note, it's good to mention
the change to the migration guide doc as well, since the purpose of the doc is
to collect the major changes/incompatibilities between versions and end users
would refer the doc.
### Why are the changes needed?
SPARK-28199 is technically backward incompatible change and we should
kindly guide the change.
### Does this PR introduce _any_ user-facing change?
Doc change.
### How was this patch tested?
N/A, as it's just a doc change.
Closes #28763 from HeartSaVioR/SPARK-28199-FOLLOWUP-doc.
Authored-by: Jungtaek Lim (HeartSaVioR) <kabhwan.opensou...@gmail.com>
Signed-off-by: Wenchen Fan <wenc...@databricks.com>
(cherry picked from commit 8305b77796ad45090e9d430e2be59e467fc173d6)
Signed-off-by: Wenchen Fan <wenc...@databricks.com>
---
docs/ss-migration-guide.md | 2 ++
1 file changed, 2 insertions(+)
diff --git a/docs/ss-migration-guide.md b/docs/ss-migration-guide.md
index 963ef07..002058b 100644
--- a/docs/ss-migration-guide.md
+++ b/docs/ss-migration-guide.md
@@ -31,3 +31,5 @@ Please refer [Migration Guide: SQL, Datasets and
DataFrame](sql-migration-guide.
- In Spark 3.0, Structured Streaming forces the source schema into nullable
when file-based datasources such as text, json, csv, parquet and orc are used
via `spark.readStream(...)`. Previously, it respected the nullability in source
schema; however, it caused issues tricky to debug with NPE. To restore the
previous behavior, set `spark.sql.streaming.fileSource.schema.forceNullable` to
`false`.
- Spark 3.0 fixes the correctness issue on Stream-stream outer join, which
changes the schema of state. (See
[SPARK-26154](https://issues.apache.org/jira/browse/SPARK-26154) for more
details). If you start your query from checkpoint constructed from Spark 2.x
which uses stream-stream outer join, Spark 3.0 fails the query. To recalculate
outputs, discard the checkpoint and replay previous inputs.
+
+- In Spark 3.0, the deprecated class
`org.apache.spark.sql.streaming.ProcessingTime` has been removed. Use
`org.apache.spark.sql.streaming.Trigger.ProcessingTime` instead. Likewise,
`org.apache.spark.sql.execution.streaming.continuous.ContinuousTrigger` has
been removed in favor of `Trigger.Continuous`, and
`org.apache.spark.sql.execution.streaming.OneTimeTrigger` has been hidden in
favor of `Trigger.Once`.
\ No newline at end of file
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
