Github user ahmed-mahran commented on the issue:
https://github.com/apache/spark/pull/13866
Related PR #512
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so
Github user ahmed-mahran commented on the issue:
https://github.com/apache/spark/pull/13866
@zsxwing, it is great that you have pointed to unallocated blocks, I've
missed those.
**What to clean?**
So, there are three kinds of stored blocks here: *unallocated s
Github user ahmed-mahran commented on a diff in the pull request:
https://github.com/apache/spark/pull/13866#discussion_r84884088
--- Diff:
streaming/src/main/scala/org/apache/spark/streaming/scheduler/JobGenerator.scala
---
@@ -146,6 +146,9 @@ class JobGenerator(jobScheduler
Github user ahmed-mahran commented on the issue:
https://github.com/apache/spark/pull/13866
The idea is about removing persisted RDDs and BlockRDDs data. Setting
StreamingContext to null would make driver-side metadata GCed but not cached
blocks, right?
What makes this
Github user ahmed-mahran closed the pull request at:
https://github.com/apache/spark/pull/13866
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature
Github user ahmed-mahran commented on the issue:
https://github.com/apache/spark/pull/14234
Fine, ignoring this
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and
Github user ahmed-mahran commented on the issue:
https://github.com/apache/spark/pull/14234
The slides renders bad on "libre office" I have; I'll try something else
and see.
---
If your project is set up for it, you can reply to this email and have your
reply appe
Github user ahmed-mahran commented on the issue:
https://github.com/apache/spark/pull/14234
I can find a pptx at "docs/img/structured-streaming.pptx" where there is a
corresponding slide for each image.
---
If your project is set up for it, you can reply to this email and
Github user ahmed-mahran commented on the issue:
https://github.com/apache/spark/pull/14234
Note, "img/structured-streaming-stream-as-a-table.png" needs to be
regenerated; someone changed "new rows appended to **a** unbounded table" to
"new rows appended to *
Github user ahmed-mahran commented on a diff in the pull request:
https://github.com/apache/spark/pull/14234#discussion_r71090897
--- Diff: docs/structured-streaming-programming-guide.md ---
@@ -65,11 +51,13 @@ val words = lines.as[String].flatMap(_.split(" "))
val
Github user ahmed-mahran commented on a diff in the pull request:
https://github.com/apache/spark/pull/14234#discussion_r71090763
--- Diff: docs/structured-streaming-programming-guide.md ---
@@ -14,29 +14,13 @@ Structured Streaming is a scalable and fault-tolerant
stream
Github user ahmed-mahran commented on a diff in the pull request:
https://github.com/apache/spark/pull/14234#discussion_r71081040
--- Diff: docs/structured-streaming-programming-guide.md ---
@@ -14,29 +14,13 @@ Structured Streaming is a scalable and fault-tolerant
stream
Github user ahmed-mahran commented on a diff in the pull request:
https://github.com/apache/spark/pull/14234#discussion_r71080823
--- Diff: docs/structured-streaming-programming-guide.md ---
@@ -14,29 +14,13 @@ Structured Streaming is a scalable and fault-tolerant
stream
Github user ahmed-mahran commented on a diff in the pull request:
https://github.com/apache/spark/pull/14234#discussion_r71080656
--- Diff: docs/structured-streaming-programming-guide.md ---
@@ -14,29 +14,13 @@ Structured Streaming is a scalable and fault-tolerant
stream
Github user ahmed-mahran commented on a diff in the pull request:
https://github.com/apache/spark/pull/14234#discussion_r71073776
--- Diff: docs/structured-streaming-programming-guide.md ---
@@ -1093,12 +1067,10 @@ spark.streams().awaitAnyTermination() # block
until any one of
Github user ahmed-mahran commented on a diff in the pull request:
https://github.com/apache/spark/pull/14234#discussion_r71073767
--- Diff: docs/structured-streaming-programming-guide.md ---
@@ -620,16 +603,14 @@ df.groupBy("type").count()
### Window Operations on
Github user ahmed-mahran commented on a diff in the pull request:
https://github.com/apache/spark/pull/14234#discussion_r71073763
--- Diff: docs/structured-streaming-programming-guide.md ---
@@ -519,10 +502,10 @@ csvDF = spark \
-These examples generate
Github user ahmed-mahran commented on a diff in the pull request:
https://github.com/apache/spark/pull/14234#discussion_r71073751
--- Diff: docs/structured-streaming-programming-guide.md ---
@@ -439,7 +422,7 @@ Here are some examples.
{% highlight scala %}
-val
Github user ahmed-mahran commented on a diff in the pull request:
https://github.com/apache/spark/pull/14234#discussion_r71073746
--- Diff: docs/structured-streaming-programming-guide.md ---
@@ -410,26 +398,21 @@ see how this model handles event-time based
processing and late
Github user ahmed-mahran commented on a diff in the pull request:
https://github.com/apache/spark/pull/14234#discussion_r71073739
--- Diff: docs/structured-streaming-programming-guide.md ---
@@ -410,26 +398,21 @@ see how this model handles event-time based
processing and late
Github user ahmed-mahran commented on a diff in the pull request:
https://github.com/apache/spark/pull/14234#discussion_r71073736
--- Diff: docs/structured-streaming-programming-guide.md ---
@@ -410,26 +398,21 @@ see how this model handles event-time based
processing and late
Github user ahmed-mahran commented on a diff in the pull request:
https://github.com/apache/spark/pull/14234#discussion_r71073721
--- Diff: docs/structured-streaming-programming-guide.md ---
@@ -223,7 +211,7 @@ $ ./bin/run-example
Github user ahmed-mahran commented on a diff in the pull request:
https://github.com/apache/spark/pull/14234#discussion_r71073714
--- Diff: docs/structured-streaming-programming-guide.md ---
@@ -65,11 +51,13 @@ val words = lines.as[String].flatMap(_.split(" "))
val
Github user ahmed-mahran commented on a diff in the pull request:
https://github.com/apache/spark/pull/14234#discussion_r71073700
--- Diff: docs/structured-streaming-programming-guide.md ---
@@ -82,8 +70,6 @@ SparkSession spark = SparkSession
.builder()
.appName
Github user ahmed-mahran commented on a diff in the pull request:
https://github.com/apache/spark/pull/14234#discussion_r71073691
--- Diff: docs/structured-streaming-programming-guide.md ---
@@ -14,29 +14,13 @@ Structured Streaming is a scalable and fault-tolerant
stream
GitHub user ahmed-mahran opened a pull request:
https://github.com/apache/spark/pull/14234
[MINOR][SQL][STREAMING][DOCS] Fix minor typos, punctuations and grammar
## What changes were proposed in this pull request?
Minor fixes correcting some typos, punctuations, grammar
GitHub user ahmed-mahran opened a pull request:
https://github.com/apache/spark/pull/14145
[SPARK-16487] [STREAMING] Fix some batches might not get marked as fully
processed in JobGenerator
## What changes were proposed in this pull request?
In `JobGenerator`, the code
Github user ahmed-mahran commented on a diff in the pull request:
https://github.com/apache/spark/pull/13945#discussion_r68858791
--- Diff: docs/structured-streaming-programming-guide.md ---
@@ -0,0 +1,888 @@
+---
+layout: global
+displayTitle: Structured Streaming
Github user ahmed-mahran commented on a diff in the pull request:
https://github.com/apache/spark/pull/13945#discussion_r68858594
--- Diff: docs/structured-streaming-programming-guide.md ---
@@ -0,0 +1,888 @@
+---
+layout: global
+displayTitle: Structured Streaming
Github user ahmed-mahran commented on a diff in the pull request:
https://github.com/apache/spark/pull/13866#discussion_r68323315
--- Diff:
streaming/src/test/scala/org/apache/spark/streaming/StreamingContextSuite.scala
---
@@ -327,6 +329,50 @@ class StreamingContextSuite extends
Github user ahmed-mahran commented on a diff in the pull request:
https://github.com/apache/spark/pull/13866#discussion_r68323120
--- Diff:
streaming/src/test/scala/org/apache/spark/streaming/StreamingContextSuite.scala
---
@@ -327,6 +329,50 @@ class StreamingContextSuite extends
Github user ahmed-mahran commented on a diff in the pull request:
https://github.com/apache/spark/pull/13866#discussion_r68322736
--- Diff:
streaming/src/test/scala/org/apache/spark/streaming/StreamingContextSuite.scala
---
@@ -327,6 +329,50 @@ class StreamingContextSuite extends
Github user ahmed-mahran commented on a diff in the pull request:
https://github.com/apache/spark/pull/13866#discussion_r68322648
--- Diff:
streaming/src/test/scala/org/apache/spark/streaming/StreamingContextSuite.scala
---
@@ -327,6 +329,50 @@ class StreamingContextSuite extends
Github user ahmed-mahran commented on a diff in the pull request:
https://github.com/apache/spark/pull/13866#discussion_r68322521
--- Diff:
streaming/src/test/scala/org/apache/spark/streaming/StreamingContextSuite.scala
---
@@ -18,10 +18,12 @@
package
GitHub user ahmed-mahran opened a pull request:
https://github.com/apache/spark/pull/13866
[SPARK-16160] [STREAMING] Clear last remembered metadata window per dstream
upon context graceful stop
## What changes were proposed in this pull request?
When stopping a streaming
GitHub user ahmed-mahran opened a pull request:
https://github.com/apache/spark/pull/13825
[SPARK-16120] [STREAMING] getCurrentLogFiles in ReceiverSuite WAL
generating and cleaning case uses external variable instead of the passed
parameter
## What changes were proposed in this
36 matches
Mail list logo
