Github user lw-lin commented on the issue:
https://github.com/apache/spark/pull/16902
@zsxwing would you take a look
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled
Github user lw-lin commented on the issue:
https://github.com/apache/spark/pull/16686
Hi @kayousterhout, a fix to `KafkaSourceSuite.subscribing topic by pattern
with topic deletions` is here: https://github.com/apache/spark/pull/16902
---
If your project is set up for it, you can
Github user lw-lin commented on the issue:
https://github.com/apache/spark/pull/16902
ok all 100 times of `KafkaSourceSuite` passed.
now revert back to one normal pass of testing
---
If your project is set up for it, you can reply to this email and have your
reply appear
GitHub user lw-lin opened a pull request:
https://github.com/apache/spark/pull/16902
[SPARK-19564][SS][KAFKA] KafkaOffsetReader's consumers should not be in the
same group
## What changes were proposed in this pull request?
In `KafkaOffsetReader`, when error occurs, we
Github user lw-lin commented on the issue:
https://github.com/apache/spark/pull/16902
Let's start by running only KafkaSourceSuite for 100 times
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does
Github user lw-lin closed the pull request at:
https://github.com/apache/spark/pull/16900
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature
Github user lw-lin closed the pull request at:
https://github.com/apache/spark/pull/16899
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature
Github user lw-lin commented on the issue:
https://github.com/apache/spark/pull/16900
ok, all 100 times of `KafkaSourceSuite` passed.
let's now do a full pass of testing.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub
Github user lw-lin commented on the issue:
https://github.com/apache/spark/pull/16900
Let's start by running only `KafkaSourceSuite` for 1000 times
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does
GitHub user lw-lin opened a pull request:
https://github.com/apache/spark/pull/16900
[SPARK-19564][SS][KAFKA] KafkaOffsetReader's consumers should not be in the
same group
## What changes were proposed in this pull request?
(Please fill in changes proposed in this fix
GitHub user lw-lin opened a pull request:
https://github.com/apache/spark/pull/16899
[DO_NOT_MERGE][Test] Investigate KafkaSourceSuite flakiness
## What changes were proposed in this pull request?
(Please fill in changes proposed in this fix)
## How was this patch
Github user lw-lin commented on the issue:
https://github.com/apache/spark/pull/16736
@gatorsmile @cloud-fan thank you for the time and efforts you've put in
reviewing this!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub
Github user lw-lin commented on a diff in the pull request:
https://github.com/apache/spark/pull/16736#discussion_r100089378
--- Diff:
sql/core/src/test/scala/org/apache/spark/sql/internal/SQLConfEntrySuite.scala
---
@@ -164,6 +164,18 @@ class SQLConfEntrySuite extends
Github user lw-lin commented on a diff in the pull request:
https://github.com/apache/spark/pull/16736#discussion_r100089218
--- Diff:
core/src/test/scala/org/apache/spark/internal/config/ConfigEntrySuite.scala ---
@@ -128,6 +128,25 @@ class ConfigEntrySuite extends SparkFunSuite
Github user lw-lin commented on a diff in the pull request:
https://github.com/apache/spark/pull/16736#discussion_r13797
--- Diff:
sql/core/src/test/scala/org/apache/spark/sql/internal/SQLConfEntrySuite.scala
---
@@ -164,6 +164,18 @@ class SQLConfEntrySuite extends
Github user lw-lin commented on a diff in the pull request:
https://github.com/apache/spark/pull/16736#discussion_r13250
--- Diff:
core/src/test/scala/org/apache/spark/internal/config/ConfigEntrySuite.scala ---
@@ -128,6 +128,25 @@ class ConfigEntrySuite extends SparkFunSuite
Github user lw-lin commented on the issue:
https://github.com/apache/spark/pull/16736
Jenkins retest this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled
Github user lw-lin commented on the issue:
https://github.com/apache/spark/pull/16736
Jenkins retest this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled
Github user lw-lin commented on the issue:
https://github.com/apache/spark/pull/16736
retest this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so
Github user lw-lin commented on the issue:
https://github.com/apache/spark/pull/16833
Ah this has been done in https://github.com/apache/spark/pull/16736, with
additional sanity-checks. that pr had received 'lgtm' from @smilegator. Do you
want to take a look at that?
---
If your
Github user lw-lin commented on a diff in the pull request:
https://github.com/apache/spark/pull/16758#discussion_r99063259
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/KeyedState.scala ---
@@ -0,0 +1,134 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF
Github user lw-lin commented on a diff in the pull request:
https://github.com/apache/spark/pull/16758#discussion_r99062793
--- Diff:
sql/core/src/test/scala/org/apache/spark/sql/streaming/MapGroupsWithStateSuite.scala
---
@@ -0,0 +1,240 @@
+/*
+ * Licensed to the Apache
Github user lw-lin commented on a diff in the pull request:
https://github.com/apache/spark/pull/16758#discussion_r99062438
--- Diff:
sql/core/src/test/scala/org/apache/spark/sql/streaming/MapGroupsWithStateSuite.scala
---
@@ -0,0 +1,240 @@
+/*
+ * Licensed to the Apache
Github user lw-lin commented on a diff in the pull request:
https://github.com/apache/spark/pull/16758#discussion_r99059679
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/KeyedStateImpl.scala
---
@@ -0,0 +1,57 @@
+/*
+ * Licensed to the Apache
Github user lw-lin commented on the issue:
https://github.com/apache/spark/pull/16736
PR summary & description updated. thanks for comments @gatorsmile !
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your pro
Github user lw-lin commented on the issue:
https://github.com/apache/spark/pull/16731
understood. they really were lots of functions -- let's tell users to use
`expr("approx_percentile(...)")`.
closing this. thanks for the comments!
---
If your project is set up f
Github user lw-lin closed the pull request at:
https://github.com/apache/spark/pull/16731
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature
Github user lw-lin commented on a diff in the pull request:
https://github.com/apache/spark/pull/16736#discussion_r98592980
--- Diff:
sql/core/src/test/scala/org/apache/spark/sql/internal/SQLConfSuite.scala ---
@@ -221,17 +221,8 @@ class SQLConfSuite extends QueryTest
Github user lw-lin commented on a diff in the pull request:
https://github.com/apache/spark/pull/16736#discussion_r98589474
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala ---
@@ -464,6 +464,16 @@ object SQLConf {
.intConf
Github user lw-lin commented on a diff in the pull request:
https://github.com/apache/spark/pull/16736#discussion_r98589278
--- Diff:
sql/core/src/test/scala/org/apache/spark/sql/internal/SQLConfSuite.scala ---
@@ -221,6 +221,19 @@ class SQLConfSuite extends QueryTest
Github user lw-lin commented on the issue:
https://github.com/apache/spark/pull/16736
tests passed for the default `maxSize = 1000`
now let's test against `maxSize = 1`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub
Github user lw-lin commented on the issue:
https://github.com/apache/spark/pull/16736
@gatorsmile fairly good point and thanks for the clarification!
- [Guava checks that maxSize >= 0 must
holds](https://github.com/google/guava/blob/v14.0.1/guava/src/com/google/common/ca
Github user lw-lin commented on a diff in the pull request:
https://github.com/apache/spark/pull/16736#discussion_r98375909
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala ---
@@ -464,6 +464,12 @@ object SQLConf {
.intConf
Github user lw-lin commented on a diff in the pull request:
https://github.com/apache/spark/pull/16736#discussion_r98375736
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/CatalystConf.scala ---
@@ -69,6 +71,7 @@ case class SimpleCatalystConf
Github user lw-lin commented on a diff in the pull request:
https://github.com/apache/spark/pull/16736#discussion_r98374466
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/CatalystConf.scala ---
@@ -69,6 +71,7 @@ case class SimpleCatalystConf
GitHub user lw-lin opened a pull request:
https://github.com/apache/spark/pull/16736
[WIP][SPARK][SQL][Follow-up] Configurable `tableRelationCache` maximum size
## What changes were proposed in this pull request?
(Please fill in changes proposed in this fix)
## How
Github user lw-lin commented on the issue:
https://github.com/apache/spark/pull/16719
Jenkins retest this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled
Github user lw-lin commented on the issue:
https://github.com/apache/spark/pull/16731
also @cloud-fan @gatorsmile could you take a look
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have
Github user lw-lin commented on a diff in the pull request:
https://github.com/apache/spark/pull/16719#discussion_r98347229
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Canonicalize.scala
---
@@ -78,14 +78,18 @@ object Canonicalize extends
Github user lw-lin commented on the issue:
https://github.com/apache/spark/pull/16731
@rxin @hvanhovell could you take a look, thanks!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have
Github user lw-lin commented on a diff in the pull request:
https://github.com/apache/spark/pull/16719#discussion_r98346999
--- Diff:
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/ExpressionSetSuite.scala
---
@@ -32,6 +32,38 @@ class ExpressionSetSuite
Github user lw-lin commented on a diff in the pull request:
https://github.com/apache/spark/pull/16719#discussion_r98346981
--- Diff:
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/ExpressionSetSuite.scala
---
@@ -32,6 +32,38 @@ class ExpressionSetSuite
Github user lw-lin commented on a diff in the pull request:
https://github.com/apache/spark/pull/16719#discussion_r98346453
--- Diff:
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/ExpressionSetSuite.scala
---
@@ -32,6 +32,38 @@ class ExpressionSetSuite
GitHub user lw-lin opened a pull request:
https://github.com/apache/spark/pull/16731
[WIP][SPARK-][SQL] Add `approx_percentile` Dataframe API
## What changes were proposed in this pull request?
(Please fill in changes proposed in this fix)
## How was this patch
Github user lw-lin commented on the issue:
https://github.com/apache/spark/pull/16725
Jenkins test this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes
Github user lw-lin commented on a diff in the pull request:
https://github.com/apache/spark/pull/16719#discussion_r98322977
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Canonicalize.scala
---
@@ -78,14 +78,18 @@ object Canonicalize extends
Github user lw-lin commented on a diff in the pull request:
https://github.com/apache/spark/pull/16719#discussion_r98322886
--- Diff:
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/ExpressionSetSuite.scala
---
@@ -75,10 +107,14 @@ class ExpressionSetSuite
Github user lw-lin commented on the issue:
https://github.com/apache/spark/pull/16719
@cloud-fan @gatorsmile @dongjoon-hyun would you take a look, thanks!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project
GitHub user lw-lin opened a pull request:
https://github.com/apache/spark/pull/16719
[SPARK-19385][SQL] During canonicalization, `NOT(l, r)` should not expect
such cases that l.hashcode > r.hashcode
## What changes were proposed in this pull request?
Dur
Github user lw-lin commented on the issue:
https://github.com/apache/spark/pull/16673
@zsxwing would you take a look, thanks!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user lw-lin commented on the issue:
https://github.com/apache/spark/pull/16673
@uncleGen yea I mean the tooltip won't show for a blue circle (which
represents a successful batch) when the cursor hovers over the blue circle in
the 'processing time' line. Which version of spark
Github user lw-lin commented on a diff in the pull request:
https://github.com/apache/spark/pull/16673#discussion_r97225131
--- Diff:
streaming/src/main/resources/org/apache/spark/streaming/ui/static/streaming-page.js
---
@@ -169,7 +169,7 @@ function drawTimeline(id, data, minX
GitHub user lw-lin opened a pull request:
https://github.com/apache/spark/pull/16673
[SPARK-19330][DStreams] Also show tooltip for successful batches
## What changes were proposed in this pull request?
[Before]
 into this. Also test passed.
So @cloud-fan would you take a look at this again at your convenience
Github user lw-lin commented on the issue:
https://github.com/apache/spark/pull/15480
Jenkins retest this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled
Github user lw-lin commented on the issue:
https://github.com/apache/spark/pull/15480
Hi @mallman, I'll pick up this within this week. Thanks for the feedback! :)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your
Github user lw-lin commented on a diff in the pull request:
https://github.com/apache/spark/pull/16442#discussion_r94271336
--- Diff: docs/structured-streaming-programming-guide.md ---
@@ -537,21 +537,21 @@ Most of the common operations on DataFrame/Dataset
are supported
Github user lw-lin commented on a diff in the pull request:
https://github.com/apache/spark/pull/16442#discussion_r94271332
--- Diff: docs/structured-streaming-programming-guide.md ---
@@ -537,21 +537,21 @@ Most of the common operations on DataFrame/Dataset
are supported
Github user lw-lin commented on the issue:
https://github.com/apache/spark/pull/16442
@srowen would you take a look, thanks!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user lw-lin commented on a diff in the pull request:
https://github.com/apache/spark/pull/16442#discussion_r94271296
--- Diff: docs/structured-streaming-programming-guide.md ---
@@ -1491,14 +1503,17 @@ spark.streams.addListener(new
StreamingQueryListener
Github user lw-lin commented on a diff in the pull request:
https://github.com/apache/spark/pull/16442#discussion_r94271291
--- Diff: docs/structured-streaming-programming-guide.md ---
@@ -1169,11 +1177,11 @@ query.stop() // stop the query
query.awaitTermination
Github user lw-lin commented on a diff in the pull request:
https://github.com/apache/spark/pull/16442#discussion_r94271257
--- Diff: docs/structured-streaming-programming-guide.md ---
@@ -1026,7 +1026,9 @@ noAggDF
// Write new data to Parquet files
noAggDF
Github user lw-lin commented on a diff in the pull request:
https://github.com/apache/spark/pull/16442#discussion_r94271243
--- Diff: docs/structured-streaming-programming-guide.md ---
@@ -973,7 +973,7 @@ Here is a table of all the sinks, and the corresponding
settings
Github user lw-lin commented on a diff in the pull request:
https://github.com/apache/spark/pull/16442#discussion_r94271218
--- Diff: docs/structured-streaming-programming-guide.md ---
@@ -537,21 +537,21 @@ Most of the common operations on DataFrame/Dataset
are supported
GitHub user lw-lin opened a pull request:
https://github.com/apache/spark/pull/16442
[SPARK-19041[SS] Fix code snippet compilation issues in Structured
Streaming Programming Guide
## What changes were proposed in this pull request?
Currently some code snippets
Github user lw-lin commented on the issue:
https://github.com/apache/spark/pull/16423
hi @yhuai, could you also:
add:
```
lw-lin - Liwei Lin
```
update:
```
sharkdtu - Xiaogang Tu
cenyuhai - Yuhai Ceng
```
---
If your project is set up
Github user lw-lin commented on a diff in the pull request:
https://github.com/apache/spark/pull/15480#discussion_r90454770
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/GenerateOrdering.scala
---
@@ -118,7 +118,42 @@ object
Github user lw-lin closed the pull request at:
https://github.com/apache/spark/pull/15946
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature
Github user lw-lin commented on the issue:
https://github.com/apache/spark/pull/15946
sure! closing this.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so
Github user lw-lin commented on a diff in the pull request:
https://github.com/apache/spark/pull/15480#discussion_r89516712
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/GenerateOrdering.scala
---
@@ -118,7 +118,42 @@ object
Github user lw-lin commented on a diff in the pull request:
https://github.com/apache/spark/pull/15480#discussion_r89490417
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/GenerateOrdering.scala
---
@@ -118,7 +118,42 @@ object
Github user lw-lin commented on the issue:
https://github.com/apache/spark/pull/15480
Jenkins retest this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled
Github user lw-lin commented on the issue:
https://github.com/apache/spark/pull/15480
@cloud-fan @hvanhovell would you take a look at this? Seems like it's
targeted for 2.1. Thanks!
---
If your project is set up for it, you can reply to this email and have your
reply appear
Github user lw-lin commented on the issue:
https://github.com/apache/spark/pull/15949
Also cc @marmbrus @tdas @zsxwing; please be aware of
[SPARK-18552](https://issues.apache.org/jira/browse/SPARK-18552), thanks.
---
If your project is set up for it, you can reply to this email
Github user lw-lin commented on the issue:
https://github.com/apache/spark/pull/15949
Hi @tcondie, just want to provide some information regarding watermark
recovery:
After some investigation, I found that even if we do have correctly
recovered the watermark from log
Github user lw-lin commented on the issue:
https://github.com/apache/spark/pull/15946
Ah, I'll keep an eye on that! @zsxwing thanks for the notification.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project
Github user lw-lin commented on the issue:
https://github.com/apache/spark/pull/15946
@marmbrus @zsxwing could you take a look? Thanks!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have
GitHub user lw-lin opened a pull request:
https://github.com/apache/spark/pull/15946
[SPARK-18513][Structured Streaming] Record and recover watermark
## What changes were proposed in this pull request?
(Please fill in changes proposed in this fix)
## How
Github user lw-lin commented on the issue:
https://github.com/apache/spark/pull/15870
@zsxwing thanks. I'm rebasing right now.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user lw-lin closed the pull request at:
https://github.com/apache/spark/pull/15828
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature
101 - 200 of 597 matches
Mail list logo
