This is an automated email from the ASF dual-hosted git repository.
gurwls223 pushed a commit to branch branch-3.1
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.1 by this push:
new e9bead7 [MINOR][DOCS] Corrected spacing in structured streaming
programming
e9bead7 is described below
commit e9bead79f8555faa8ba6a3b2ca9925a28022bee9
Author: mans2singh <mans2si...@yahoo.com>
AuthorDate: Tue Nov 2 11:01:57 2021 +0900
[MINOR][DOCS] Corrected spacing in structured streaming programming
### What changes were proposed in this pull request?
There is no space between `with` and `<code>` as shown below:
`... configured
with<code>spark.sql.streaming.fileSource.cleaner.numThreads</code> ...`
### Why are the changes needed?
Added space
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Only documentation was changed and no code was change.
Closes #34458 from mans2singh/structured_streaming_programming_guide_space.
Authored-by: mans2singh <mans2si...@yahoo.com>
Signed-off-by: Hyukjin Kwon <gurwls...@apache.org>
(cherry picked from commit 675071a38e47dc2c55cf4f71de7ad0bebc1b4f2b)
Signed-off-by: Hyukjin Kwon <gurwls...@apache.org>
---
docs/structured-streaming-programming-guide.md | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/docs/structured-streaming-programming-guide.md
b/docs/structured-streaming-programming-guide.md
index d88cf91b..28d312e 100644
--- a/docs/structured-streaming-programming-guide.md
+++ b/docs/structured-streaming-programming-guide.md
@@ -553,7 +553,7 @@ Here are the details of all the sources in Spark.
For example, suppose you provide '/hello?/spark/*' as source pattern,
'/hello1/spark/archive/dir' cannot be used as the value of "sourceArchiveDir",
as '/hello?/spark/*' and '/hello1/spark/archive' will be matched.
'/hello1/spark' cannot be also used as the value of "sourceArchiveDir", as
'/hello?/spark' and '/hello1/spark' will be matched. '/archived/here' would be
OK as it doesn't match.<br/>
Spark will move source files respecting their own path. For example,
if the path of source file is <code>/a/b/dataset.txt</code> and the path of
archive directory is <code>/archived/here</code>, file will be moved to
<code>/archived/here/a/b/dataset.txt</code>.<br/>
NOTE: Both archiving (via moving) or deleting completed files will
introduce overhead (slow down, even if it's happening in separate thread) in
each micro-batch, so you need to understand the cost for each operation in your
file system before enabling this option. On the other hand, enabling this
option will reduce the cost to list source files which can be an expensive
operation.<br/>
- Number of threads used in completed file cleaner can be configured
with<code>spark.sql.streaming.fileSource.cleaner.numThreads</code> (default:
1).<br/>
+ Number of threads used in completed file cleaner can be configured
with <code>spark.sql.streaming.fileSource.cleaner.numThreads</code> (default:
1).<br/>
NOTE 2: The source path should not be used from multiple sources or
queries when enabling this option. Similarly, you must ensure the source path
doesn't match to any files in output directory of file stream sink.<br/>
NOTE 3: Both delete and move actions are best effort. Failing to
delete or move files will not fail the streaming query. Spark may not clean up
some source files in some circumstances - e.g. the application doesn't shut
down gracefully, too many files are queued to clean up.
<br/><br/>
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
