Github user HeartSaVioR commented on the issue:
https://github.com/apache/spark/pull/21497
cc. @tdas @jose-torres @jerryshao @arunmahadevan
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
Github user HeartSaVioR commented on the issue:
https://github.com/apache/spark/pull/21469
Also added custom metric for the count of versions stored in loadedMaps.
This is a new screenshot:
https://user-images.githubusercontent.com/1317309/40978481-b46ad324-690e-11e8-9b0f
Github user HeartSaVioR commented on the issue:
https://github.com/apache/spark/pull/21497
retest this please
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail
Github user HeartSaVioR commented on the issue:
https://github.com/apache/spark/pull/21497
Failing tests were below:
* org.apache.spark.sql.hive.client.HiveClientSuites.(It is not a test it is
a sbt.testing.NestedSuiteSelector
GitHub user HeartSaVioR opened a pull request:
https://github.com/apache/spark/pull/21497
[SPARK-24466][SS] Fix TextSocketMicroBatchReader to be compatible with
netcat again
## What changes were proposed in this pull request?
TextSocketMicroBatchReader was no longer
Github user HeartSaVioR commented on the issue:
https://github.com/apache/spark/pull/21469
Looks like the size is added only once for same identity on
SizeEstimator.estimate(), so SizeEstimator.estimate() is working correctly in
this case. There might be other valid cases
Github user HeartSaVioR commented on the issue:
https://github.com/apache/spark/pull/21477
Thanks @HyukjinKwon for cc.ing me. I didn't cover the python part on
structured streaming so would take some time to cover and going through the
code. Hoping I can participate reviewing in time
Github user HeartSaVioR commented on the issue:
https://github.com/apache/spark/pull/21469
@arunmahadevan
I didn't add the metric to StateOperatorProgress cause this behavior is
specific to HDFSBackedStateStoreProvider (though this is only one
implementation available in Apache
Github user HeartSaVioR commented on the issue:
https://github.com/apache/spark/pull/21469
@jose-torres
Ah yes I forgot that shallow copy has been occurring, so while new map
should hold necessary size of map entries but row object will be shared across
versions. Thanks
Github user HeartSaVioR commented on the issue:
https://github.com/apache/spark/pull/21469
Thanks @HyukjinKwon for reviewing. Addressed PR title as well as fixing nit.
---
-
To unsubscribe, e-mail: reviews-unsubscr
Github user HeartSaVioR commented on the issue:
https://github.com/apache/spark/pull/21469
cc. @tdas @jose-torres @jerryshao @HyukjinKwon @arunmahadevan
---
-
To unsubscribe, e-mail: reviews-unsubscr
GitHub user HeartSaVioR opened a pull request:
https://github.com/apache/spark/pull/21469
[SPARK-24441][SS] Expose total size of states in HDFSBackedStateStoreâ¦
â¦Provider
## What changes were proposed in this pull request?
This patch exposes the estimation
Github user HeartSaVioR commented on a diff in the pull request:
https://github.com/apache/spark/pull/21428#discussion_r191629554
--- Diff:
sql/core/src/test/scala/org/apache/spark/sql/streaming/continuous/shuffle/ContinuousShuffleSuite.scala
---
@@ -58,39 +46,29 @@ class
Github user HeartSaVioR commented on a diff in the pull request:
https://github.com/apache/spark/pull/21428#discussion_r191605388
--- Diff:
sql/core/src/test/scala/org/apache/spark/sql/streaming/continuous/shuffle/ContinuousShuffleSuite.scala
---
@@ -58,39 +46,29 @@ class
Github user HeartSaVioR commented on a diff in the pull request:
https://github.com/apache/spark/pull/21428#discussion_r191629272
--- Diff:
sql/core/src/test/scala/org/apache/spark/sql/streaming/continuous/shuffle/ContinuousShuffleSuite.scala
---
@@ -288,4 +267,153 @@ class
Github user HeartSaVioR commented on the issue:
https://github.com/apache/spark/pull/21445
@LiangchangZ
> In the real CP situation, reader and writer may be always in different
tasks, right?
Continuous mode already supports some valid use cases, and putting
Github user HeartSaVioR commented on the issue:
https://github.com/apache/spark/pull/21445
@LiangchangZ
Looks like the patch is needed only with #21353 #21332 #21293 as of now,
right? If then please state the condition in JIRA issue description as well as
PR's description so
Github user HeartSaVioR commented on a diff in the pull request:
https://github.com/apache/spark/pull/21385#discussion_r190131693
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/continuous/shuffle/UnsafeRowReceiver.scala
---
@@ -56,20 +69,73 @@ private
Github user HeartSaVioR commented on a diff in the pull request:
https://github.com/apache/spark/pull/21385#discussion_r190129892
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/continuous/shuffle/UnsafeRowReceiver.scala
---
@@ -56,20 +69,73 @@ private
Github user HeartSaVioR commented on a diff in the pull request:
https://github.com/apache/spark/pull/21385#discussion_r190125731
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/continuous/shuffle/UnsafeRowReceiver.scala
---
@@ -56,20 +69,73 @@ private
Github user HeartSaVioR commented on a diff in the pull request:
https://github.com/apache/spark/pull/21385#discussion_r190120836
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/continuous/shuffle/UnsafeRowReceiver.scala
---
@@ -56,20 +69,73 @@ private
Github user HeartSaVioR commented on the issue:
https://github.com/apache/spark/pull/21388
@hvanhovell
I also think someone might not want to have reflection magic (I was the one
but realized I should do it), so I'm happy to close the PR when others voice
same opinion
Github user HeartSaVioR commented on the issue:
https://github.com/apache/spark/pull/21388
retest this please
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail
Github user HeartSaVioR commented on the issue:
https://github.com/apache/spark/pull/21388
Thanks @HyukjinKwon for reviewing. Addressed review comments.
---
-
To unsubscribe, e-mail: reviews-unsubscr
GitHub user HeartSaVioR opened a pull request:
https://github.com/apache/spark/pull/21388
[SPARK-24336][SQL] Support 'pass through' transformation in BasicOperators
## What changes were proposed in this pull request?
Enable 'pass through' transformation in BasicOperators
Github user HeartSaVioR commented on the issue:
https://github.com/apache/spark/pull/21357
cc. @tdas
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h
GitHub user HeartSaVioR opened a pull request:
https://github.com/apache/spark/pull/21357
[SPARK-24311][SS] Refactor HDFSBackedStateStoreProvider to remove
duplicateâ¦
â¦d logic between operations on delta file and snapshot file
## What changes were proposed in this pull
Github user HeartSaVioR commented on a diff in the pull request:
https://github.com/apache/spark/pull/21337#discussion_r188638980
--- Diff:
sql/core/src/test/scala/org/apache/spark/sql/streaming/continuous/shuffle/ContinuousShuffleReadSuite.scala
---
@@ -0,0 +1,122
Github user HeartSaVioR commented on a diff in the pull request:
https://github.com/apache/spark/pull/21337#discussion_r188632188
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/continuous/shuffle/ContinuousShuffleReadRDD.scala
---
@@ -0,0 +1,64
Github user HeartSaVioR commented on a diff in the pull request:
https://github.com/apache/spark/pull/21337#discussion_r188628202
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/continuous/shuffle/ContinuousShuffleReadRDD.scala
---
@@ -0,0 +1,64
Github user HeartSaVioR commented on a diff in the pull request:
https://github.com/apache/spark/pull/21337#discussion_r188636306
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/continuous/shuffle/UnsafeRowReceiver.scala
---
@@ -0,0 +1,56
Github user HeartSaVioR commented on the issue:
https://github.com/apache/spark/pull/21222
Kindly ping. I guess debugging last batch might not be attractive that
much, but printing codegen would be helpful to someone who want to investigate
or debug in detail
Github user HeartSaVioR commented on a diff in the pull request:
https://github.com/apache/spark/pull/21222#discussion_r186247474
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/debug/package.scala ---
@@ -116,6 +168,30 @@ package object debug
Github user HeartSaVioR commented on a diff in the pull request:
https://github.com/apache/spark/pull/21222#discussion_r186032252
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/debug/package.scala ---
@@ -88,23 +100,62 @@ package object debug
Github user HeartSaVioR commented on the issue:
https://github.com/apache/spark/pull/21207
@tdas @jerryshao @HyukjinKwon
Kindly ping to trigger test and review.
---
-
To unsubscribe, e-mail: reviews-unsubscr
Github user HeartSaVioR commented on the issue:
https://github.com/apache/spark/pull/21222
@tdas @jose-torres @jerryshao @arunmahadevan
Kindly ping to review.
---
-
To unsubscribe, e-mail: reviews-unsubscr
GitHub user HeartSaVioR opened a pull request:
https://github.com/apache/spark/pull/21222
[SPARK-24161][SS] Enable debug package feature on structured streaming
## What changes were proposed in this pull request?
Currently, debug package has a implicit class "Debug
Github user HeartSaVioR commented on a diff in the pull request:
https://github.com/apache/spark/pull/21200#discussion_r18520
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/continuous/ContinuousDataSourceRDD.scala
---
@@ -0,0 +1,153
Github user HeartSaVioR commented on a diff in the pull request:
https://github.com/apache/spark/pull/21200#discussion_r185328820
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/continuous/ContinuousDataSourceRDD.scala
---
@@ -0,0 +1,153
Github user HeartSaVioR commented on a diff in the pull request:
https://github.com/apache/spark/pull/21200#discussion_r185326551
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/continuous/ContinuousDataSourceRDD.scala
---
@@ -0,0 +1,153
Github user HeartSaVioR commented on a diff in the pull request:
https://github.com/apache/spark/pull/21200#discussion_r185317000
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/continuous/ContinuousDataSourceRDD.scala
---
@@ -0,0 +1,153
Github user HeartSaVioR commented on a diff in the pull request:
https://github.com/apache/spark/pull/21200#discussion_r185316062
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/continuous/ContinuousDataSourceRDD.scala
---
@@ -0,0 +1,153
Github user HeartSaVioR commented on a diff in the pull request:
https://github.com/apache/spark/pull/21200#discussion_r185282844
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/continuous/ContinuousDataSourceRDD.scala
---
@@ -0,0 +1,153
Github user HeartSaVioR commented on a diff in the pull request:
https://github.com/apache/spark/pull/21200#discussion_r185201032
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/continuous/WriteToContinuousDataSourceExec.scala
---
@@ -46,28 +46,34
Github user HeartSaVioR commented on a diff in the pull request:
https://github.com/apache/spark/pull/21200#discussion_r18524
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/continuous/EpochPollRunnable.scala
---
@@ -0,0 +1,61
Github user HeartSaVioR commented on a diff in the pull request:
https://github.com/apache/spark/pull/21200#discussion_r185194384
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/continuous/ContinuousDataSourceRDD.scala
---
@@ -0,0 +1,153
Github user HeartSaVioR commented on a diff in the pull request:
https://github.com/apache/spark/pull/21200#discussion_r185198458
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/continuous/ContinuousQueuedDataReader.scala
---
@@ -0,0 +1,90
Github user HeartSaVioR commented on a diff in the pull request:
https://github.com/apache/spark/pull/21200#discussion_r185187424
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/continuous/ContinuousDataSourceRDD.scala
---
@@ -0,0 +1,153
Github user HeartSaVioR commented on the issue:
https://github.com/apache/spark/pull/21152
@jerryshao Thanks for merging! My Apache JIRA ID is âkabhwanâ
---
-
To unsubscribe, e-mail: reviews-unsubscr
Github user HeartSaVioR commented on the issue:
https://github.com/apache/spark/pull/21152
Thanks fore reviewing. I have addressed review comments.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
Github user HeartSaVioR commented on a diff in the pull request:
https://github.com/apache/spark/pull/21152#discussion_r184322699
--- Diff:
sql/core/src/test/scala/org/apache/spark/sql/streaming/continuous/ContinuousSuite.scala
---
@@ -66,157 +66,115 @@ class ContinuousSuite
Github user HeartSaVioR commented on a diff in the pull request:
https://github.com/apache/spark/pull/21152#discussion_r184321762
--- Diff:
sql/core/src/test/scala/org/apache/spark/sql/streaming/continuous/ContinuousSuite.scala
---
@@ -66,157 +66,115 @@ class ContinuousSuite
Github user HeartSaVioR commented on a diff in the pull request:
https://github.com/apache/spark/pull/21063#discussion_r184241625
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/ProgressReporter.scala
---
@@ -111,7 +112,12 @@ trait ProgressReporter
Github user HeartSaVioR commented on a diff in the pull request:
https://github.com/apache/spark/pull/21152#discussion_r184236837
--- Diff:
sql/core/src/test/scala/org/apache/spark/sql/streaming/continuous/ContinuousSuite.scala
---
@@ -66,157 +66,115 @@ class ContinuousSuite
Github user HeartSaVioR commented on the issue:
https://github.com/apache/spark/pull/21152
also cc. to @tdas since he reviews SS related PRs (as well as continuous
mode) so far.
---
-
To unsubscribe, e-mail
Github user HeartSaVioR commented on a diff in the pull request:
https://github.com/apache/spark/pull/21152#discussion_r184213878
--- Diff:
sql/core/src/test/scala/org/apache/spark/sql/streaming/continuous/ContinuousSuite.scala
---
@@ -66,157 +66,115 @@ class ContinuousSuite
Github user HeartSaVioR commented on a diff in the pull request:
https://github.com/apache/spark/pull/21152#discussion_r184006570
--- Diff:
sql/core/src/test/scala/org/apache/spark/sql/streaming/continuous/ContinuousSuite.scala
---
@@ -66,157 +66,115 @@ class ContinuousSuite
Github user HeartSaVioR commented on the issue:
https://github.com/apache/spark/pull/21152
@jose-torres Please review this PR. Thanks!
cc. @jerryshao @HyukjinKwon
---
-
To unsubscribe, e-mail: reviews-unsubscr
GitHub user HeartSaVioR opened a pull request:
https://github.com/apache/spark/pull/21152
[SPARK-23688][SS] Refactor tests away from rate source
## What changes were proposed in this pull request?
Replace rate source with memory source in continuous mode. Keep using
"
Github user HeartSaVioR commented on the issue:
https://github.com/apache/spark/pull/21136
LGTM. This is what I also found so far today.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
Github user HeartSaVioR commented on the pull request:
https://github.com/apache/spark/pull/9812#issuecomment-159347482
@srowen @vanzin @holdenk Thanks for reviewing and merging!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub
Github user HeartSaVioR commented on the pull request:
https://github.com/apache/spark/pull/9812#issuecomment-158338205
@vanzin Thanks for reviewing, I addressed your comment. Please take a look
again.
---
If your project is set up for it, you can reply to this email and have your
Github user HeartSaVioR commented on the pull request:
https://github.com/apache/spark/pull/9812#issuecomment-158377559
Failed tests seems not related.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does
GitHub user HeartSaVioR opened a pull request:
https://github.com/apache/spark/pull/9812
[SPARK-11818][REPL] Fix ExecutorClassLoader to lookup resources from â¦
â¦parent class loader
Without patch, two additional tests of ExecutorClassLoaderSuite fails
Github user HeartSaVioR commented on a diff in the pull request:
https://github.com/apache/spark/pull/9812#discussion_r45288354
--- Diff:
repl/src/main/scala/org/apache/spark/repl/ExecutorClassLoader.scala ---
@@ -55,6 +57,14 @@ class ExecutorClassLoader(conf: SparkConf, classUri
Github user HeartSaVioR commented on a diff in the pull request:
https://github.com/apache/spark/pull/9812#discussion_r45288990
--- Diff:
repl/src/main/scala/org/apache/spark/repl/ExecutorClassLoader.scala ---
@@ -55,6 +57,14 @@ class ExecutorClassLoader(conf: SparkConf, classUri
Github user HeartSaVioR commented on a diff in the pull request:
https://github.com/apache/spark/pull/9812#discussion_r45285698
--- Diff:
repl/src/main/scala/org/apache/spark/repl/ExecutorClassLoader.scala ---
@@ -55,6 +57,14 @@ class ExecutorClassLoader(conf: SparkConf, classUri
Github user HeartSaVioR commented on a diff in the pull request:
https://github.com/apache/spark/pull/9812#discussion_r45260904
--- Diff: core/src/main/scala/org/apache/spark/TestUtils.scala ---
@@ -159,6 +159,16 @@ private[spark] object TestUtils {
createCompiledClass
Github user HeartSaVioR commented on a diff in the pull request:
https://github.com/apache/spark/pull/9812#discussion_r45261349
--- Diff:
repl/src/main/scala/org/apache/spark/repl/ExecutorClassLoader.scala ---
@@ -55,6 +57,14 @@ class ExecutorClassLoader(conf: SparkConf, classUri
401 - 469 of 469 matches
Mail list logo
