[GitHub] spark pull request: [Spark-14687][Core][SQL][MLlib] Call path.getF...

2016-04-17 Thread lw-lin
Github user lw-lin commented on the pull request: https://github.com/apache/spark/pull/12450#issuecomment-211209966 Jenkins retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: [DO_NOT_MERGE][Test] Investigate MemorySinkSui...

2016-04-17 Thread lw-lin
Github user lw-lin closed the pull request at: https://github.com/apache/spark/pull/12443 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature

[GitHub] spark pull request: [SPARK-14672][SQL] Move HiveContext analyze lo...

2016-04-17 Thread lw-lin
Github user lw-lin commented on the pull request: https://github.com/apache/spark/pull/12448#issuecomment-211025518 Seems like `org.apache.spark.sql.hive.HiveSparkSubmitSuite` have been failing since this merging: https://spark-tests.appspot.com/tests

[GitHub] spark pull request: [Spark-14687][Core][SQL][MLlib] Call path.getF...

2016-04-17 Thread lw-lin
GitHub user lw-lin opened a pull request: https://github.com/apache/spark/pull/12450 [Spark-14687][Core][SQL][MLlib] Call path.getFileSystem(conf) instead of call FileSystem.get(conf) ## What changes were proposed in this pull request? - replaced `FileSystem.get(conf

[GitHub] spark pull request: [DO_NOT_MERGE][Test] Investigate MemorySinkSui...

2016-04-16 Thread lw-lin
Github user lw-lin commented on the pull request: https://github.com/apache/spark/pull/12443#issuecomment-210937812 Jenkins retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: [DO_NOT_MERGE][Test] Investigate MemorySinkSui...

2016-04-16 Thread lw-lin
GitHub user lw-lin opened a pull request: https://github.com/apache/spark/pull/12443 [DO_NOT_MERGE][Test] Investigate MemorySinkSuite flakiness You can merge this pull request into a Git repository by running: $ git pull https://github.com/lw-lin/spark fix-memory-sink-suite

[GitHub] spark pull request: [WIP][SPARK-14629][SPARK-14630][Build] Add sup...

2016-04-15 Thread lw-lin
Github user lw-lin closed the pull request at: https://github.com/apache/spark/pull/12396 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature

[GitHub] spark pull request: [WIP][SPARK-14629][SPARK-14630][Build] Add sup...

2016-04-15 Thread lw-lin
Github user lw-lin commented on the pull request: https://github.com/apache/spark/pull/12396#issuecomment-210314939 I see; will follow @rxin 's advise. So I'm closing this. @rxin @vanzin @srowen thank you all for the comments! :-) --- If your project is set up for it, you

[GitHub] spark pull request: [WIP][SPARK-14629][SPARK-14630][Build] Add sup...

2016-04-14 Thread lw-lin
Github user lw-lin commented on the pull request: https://github.com/apache/spark/pull/12396#issuecomment-210262145 hi @vanzin I'm not sure how to build and pre-install some jar for every Spark build on Jenkins's machine -- should some build scripts be modified? Could you give

[GitHub] spark pull request: [WIP][SPARK-14629][SPARK-14630][Build] Add sup...

2016-04-14 Thread lw-lin
Github user lw-lin commented on a diff in the pull request: https://github.com/apache/spark/pull/12396#discussion_r59821242 --- Diff: pom.xml --- @@ -2237,6 +2237,13 @@ ${project.build.sourceEncoding} ${project.reporting.outputEncoding

[GitHub] spark pull request: [SPARK-14630][Build][Core][SQL][Streaming] Cod...

2016-04-14 Thread lw-lin
Github user lw-lin commented on the pull request: https://github.com/apache/spark/pull/12389#issuecomment-210258876 @srowen @rxin thank you for the review & merging! :-) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as

[GitHub] spark pull request: [WIP][SPARK-14629][SPARK-14630] Add support fo...

2016-04-14 Thread lw-lin
Github user lw-lin commented on the pull request: https://github.com/apache/spark/pull/12396#issuecomment-210013991 As per [Scalastyle - Custom rules](http://www.scalastyle.org/custom-rules.html), I installed the jar on my local machine, and verified that the added rule works

[GitHub] spark pull request: [WIP][SPARK-14629][SPARK-14630] Add support fo...

2016-04-14 Thread lw-lin
GitHub user lw-lin opened a pull request: https://github.com/apache/spark/pull/12396 [WIP][SPARK-14629][SPARK-14630] Add support for custom scala style rules & Add rule PublicAbstractMethodsHaveTypeChecker ## What changes were proposed in this pull req

[GitHub] spark pull request: [SPARK-14630][Build][Core][SQL][Streaming] Cod...

2016-04-14 Thread lw-lin
GitHub user lw-lin opened a pull request: https://github.com/apache/spark/pull/12389 [SPARK-14630][Build][Core][SQL][Streaming] Code style: public abstract methods should have explicit return types ## What changes were proposed in this pull request? Currently many public

[GitHub] spark pull request: [SPARK-13992][Core][PySpark][FollowUp] Update ...

2016-04-12 Thread lw-lin
Github user lw-lin commented on the pull request: https://github.com/apache/spark/pull/12126#issuecomment-209217089 Jenkins retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: [SPARK-13992][Core][PySpark][FollowUp] Update ...

2016-04-12 Thread lw-lin
Github user lw-lin commented on the pull request: https://github.com/apache/spark/pull/12126#issuecomment-209216092 @rxin would you mind taking a look, or should I close this PR? Thank you! :-) --- If your project is set up for it, you can reply to this email and have your reply

[GitHub] spark pull request: [SPARK-14556][SQL] Code clean-ups for package ...

2016-04-12 Thread lw-lin
Github user lw-lin commented on the pull request: https://github.com/apache/spark/pull/12323#issuecomment-209201130 @zsxwing thank you for the review & merging ! :-) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as

[GitHub] spark pull request: [SPARK-14556][SQL] Code clean-ups for package ...

2016-04-11 Thread lw-lin
Github user lw-lin commented on the pull request: https://github.com/apache/spark/pull/12323#issuecomment-208715736 @srowen @zsxwing would you mind taking a look at this? Thanks! :-) --- If your project is set up for it, you can reply to this email and have your reply appear

[GitHub] spark pull request: [SPARK-14556][SQL] Code clean-ups for package ...

2016-04-11 Thread lw-lin
Github user lw-lin commented on a diff in the pull request: https://github.com/apache/spark/pull/12323#discussion_r59318208 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/HDFSBackedStateStoreProvider.scala --- @@ -161,24 +163,27 @@ private[state

[GitHub] spark pull request: [SPARK-14556][SQL] Code clean-ups for package ...

2016-04-11 Thread lw-lin
Github user lw-lin commented on a diff in the pull request: https://github.com/apache/spark/pull/12323#discussion_r59317875 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/HDFSBackedStateStoreProvider.scala --- @@ -506,7 +512,6 @@ private[state

[GitHub] spark pull request: [SPARK-14556][SQL] Code clean-ups for package ...

2016-04-11 Thread lw-lin
Github user lw-lin commented on a diff in the pull request: https://github.com/apache/spark/pull/12323#discussion_r59317846 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/StateStoreRDD.scala --- @@ -22,12 +22,12 @@ import scala.reflect.ClassTag

[GitHub] spark pull request: [SPARK-14556][SQL] Code clean-ups for package ...

2016-04-11 Thread lw-lin
Github user lw-lin commented on a diff in the pull request: https://github.com/apache/spark/pull/12323#discussion_r59317781 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/StateStoreConf.scala --- @@ -26,12 +26,11 @@ private[streaming] class

[GitHub] spark pull request: [SPARK-14556][SQL] Code clean-ups for package ...

2016-04-11 Thread lw-lin
GitHub user lw-lin opened a pull request: https://github.com/apache/spark/pull/12323 [SPARK-14556][SQL] Code clean-ups for package o.a.s.sql.execution.streaming.state ## What changes were proposed in this pull request? - `StateStoreConf.**max**DeltasForSnapshot

[GitHub] spark pull request: [SPARK-13211][STREAMING] Deprecate certain con...

2016-04-05 Thread lw-lin
Github user lw-lin closed the pull request at: https://github.com/apache/spark/pull/12035 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature

[GitHub] spark pull request: [SPARK-13211][STREAMING] Deprecate certain con...

2016-04-05 Thread lw-lin
Github user lw-lin commented on the pull request: https://github.com/apache/spark/pull/12035#issuecomment-206123094 @srowen sure, thank you for this review, :-) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] spark pull request: [SPARK-13211] [STREAMING] StreamingContext thr...

2016-04-05 Thread lw-lin
Github user lw-lin commented on the pull request: https://github.com/apache/spark/pull/12174#issuecomment-206107803 @srowen thanks for the fix. However, instead of allowing users to call these obsolete constructors and improving the error messages, I'm inclined to guide users

[GitHub] spark pull request: [DO_NOT_MERGE] Try to reproduce StateStoreSuit...

2016-04-05 Thread lw-lin
Github user lw-lin closed the pull request at: https://github.com/apache/spark/pull/12145 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature

[GitHub] spark pull request: [DO_NOT_MERGE] Try to reproduce StateStoreSuit...

2016-04-04 Thread lw-lin
Github user lw-lin commented on the pull request: https://github.com/apache/spark/pull/12145#issuecomment-205272081 Jenkins retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: [DO_NOT_MERGE] Try to reproduce StateStoreSuit...

2016-04-04 Thread lw-lin
Github user lw-lin commented on the pull request: https://github.com/apache/spark/pull/12145#issuecomment-205219730 Jenkins retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: [DO_NOT_MERGE] Try to reproduce StateStoreSuit...

2016-04-03 Thread lw-lin
GitHub user lw-lin opened a pull request: https://github.com/apache/spark/pull/12145 [DO_NOT_MERGE] Try to reproduce StateStoreSuite.maintenance failure You can merge this pull request into a Git repository by running: $ git pull https://github.com/lw-lin/spark fix

[GitHub] spark pull request: [SPARK-14342][Core][Docs][Tests] Remove stragg...

2016-04-03 Thread lw-lin
Github user lw-lin commented on the pull request: https://github.com/apache/spark/pull/12129#issuecomment-204958822 @rxin thank you always for the review :-) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] spark pull request: [SPARK-13992][Core][PySpark][FollowUp] Update ...

2016-04-03 Thread lw-lin
Github user lw-lin commented on the pull request: https://github.com/apache/spark/pull/12126#issuecomment-204892494 @srowen thank you for pointing this out! Yes sure let's generalize "tachyon" to "off-heap" in these and other files, which has been done

[GitHub] spark pull request: [SPARK-13992][Core][PySpark][FollowUp] Update ...

2016-04-02 Thread lw-lin
Github user lw-lin commented on the pull request: https://github.com/apache/spark/pull/12126#issuecomment-204707515 Jenkins retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: [SPARK-14342][Core][Docs][Tests] Remove stragg...

2016-04-02 Thread lw-lin
Github user lw-lin commented on the pull request: https://github.com/apache/spark/pull/12129#issuecomment-204705307 deep-review this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: [SPARK-14342][Core][Docs][Tests] Remove stragg...

2016-04-02 Thread lw-lin
Github user lw-lin commented on the pull request: https://github.com/apache/spark/pull/12129#issuecomment-204680252 Two places still referring to tachyon are intentionlly left out for backward compibility reasons: - MiMa excludes: https://github.com/apache/spark/blob/master

[GitHub] spark pull request: [SPARK-14342][Core][Docs][Tests] Remove stragg...

2016-04-02 Thread lw-lin
Github user lw-lin commented on a diff in the pull request: https://github.com/apache/spark/pull/12129#discussion_r58290846 --- Diff: core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosSchedulerUtils.scala --- @@ -283,11 +283,11 @@ private[mesos] trait

[GitHub] spark pull request: Remove references to Tachyon

2016-04-02 Thread lw-lin
GitHub user lw-lin opened a pull request: https://github.com/apache/spark/pull/12129 Remove references to Tachyon ## What changes were proposed in this pull request? References to Tachyon were removed: - for docs, `tachyon` has been generalized as `off-heap memory

[GitHub] spark pull request: [SPARK-13992][Core][PySpark][FollowUp] Update ...

2016-04-01 Thread lw-lin
Github user lw-lin commented on the pull request: https://github.com/apache/spark/pull/12126#issuecomment-204645372 hi @JoshRosen could you take a look? Thanks! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] spark pull request: [SPARK-13992][Core][PySpark][FollowUp] Update ...

2016-04-01 Thread lw-lin
GitHub user lw-lin opened a pull request: https://github.com/apache/spark/pull/12126 [SPARK-13992][Core][PySpark][FollowUp] Update storagelevel.py ## What changes were proposed in this pull request? storagelevel.py updates ## How was this patch tested

[GitHub] spark pull request: [SPARK-13618][STREAMING][WEB-UI] Make Streamin...

2016-03-31 Thread lw-lin
Github user lw-lin closed the pull request at: https://github.com/apache/spark/pull/11470 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature

[GitHub] spark pull request: [SPARK-13618][STREAMING][WEB-UI] Make Streamin...

2016-03-31 Thread lw-lin
Github user lw-lin commented on the pull request: https://github.com/apache/spark/pull/11633#issuecomment-203804570 This PR has many conflicts to resolve, so I'm closing this for now and will re-open later, thanks. --- If your project is set up for it, you can reply to this email

[GitHub] spark pull request: [SPARK-13618][STREAMING][WEB-UI] Make Streamin...

2016-03-31 Thread lw-lin
Github user lw-lin commented on the pull request: https://github.com/apache/spark/pull/11470#issuecomment-203804602 This PR has many conflicts to resolve, so I'm closing this for now and will re-open later, thanks. --- If your project is set up for it, you can reply to this email

[GitHub] spark pull request: [SPARK-13618][STREAMING][WEB-UI] Make Streamin...

2016-03-31 Thread lw-lin
Github user lw-lin closed the pull request at: https://github.com/apache/spark/pull/11643 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature

[GitHub] spark pull request: [SPARK-13618][STREAMING][WEB-UI] Make Streamin...

2016-03-31 Thread lw-lin
Github user lw-lin commented on the pull request: https://github.com/apache/spark/pull/11643#issuecomment-203804471 This PR has many conflicts to resolve, so I'm closing this for now and will open later, thanks. --- If your project is set up for it, you can reply to this email

[GitHub] spark pull request: [SPARK-13618][STREAMING][WEB-UI] Make Streamin...

2016-03-31 Thread lw-lin
Github user lw-lin closed the pull request at: https://github.com/apache/spark/pull/11633 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature

[GitHub] spark pull request: [SPARK-12857][STREAMING] Standardize "records"...

2016-03-29 Thread lw-lin
Github user lw-lin commented on the pull request: https://github.com/apache/spark/pull/12032#issuecomment-203259640 @zsxwing please be aware of the changes we made here :-) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well

[GitHub] spark pull request: [SPARK-12857][STREAMING] Standardize "records"...

2016-03-29 Thread lw-lin
Github user lw-lin commented on a diff in the pull request: https://github.com/apache/spark/pull/12032#discussion_r57835381 --- Diff: streaming/src/main/scala/org/apache/spark/streaming/ui/StreamingPage.scala --- @@ -241,24 +241,24 @@ private[ui] class StreamingPage(parent

[GitHub] spark pull request: [SPARK-13211][STREAMING] Deprecate certain con...

2016-03-29 Thread lw-lin
Github user lw-lin commented on the pull request: https://github.com/apache/spark/pull/12035#issuecomment-202881741 As reported in [\[Spark-13211\]](https://issues.apache.org/jira/browse/SPARK-13211), `StreamingContext` would throw `NoSuchElementException` when created from non

[GitHub] spark pull request: [SPARK-13211][STREAMING] Deprecate certain con...

2016-03-29 Thread lw-lin
GitHub user lw-lin opened a pull request: https://github.com/apache/spark/pull/12035 [SPARK-13211][STREAMING] Deprecate certain constructors of StreamingContext's ## What changes were proposed in this pull request? These constructors, each of which takes a checkpoint path

[GitHub] spark pull request: [SPARK-12857][STREAMING] Streaming tab in web ...

2016-03-29 Thread lw-lin
Github user lw-lin commented on the pull request: https://github.com/apache/spark/pull/12032#issuecomment-202808056 No public documents need to change, since we have been using the term "records" consistently. Strings on the Streaming tab has been changed fr

[GitHub] spark pull request: [SPARK-12857][STREAMING] Streaming tab in web ...

2016-03-29 Thread lw-lin
GitHub user lw-lin opened a pull request: https://github.com/apache/spark/pull/12032 [SPARK-12857][STREAMING] Streaming tab in web UI uses records and events interchangeably ## What changes were proposed in this pull request? Currently the Streaming tab in web UI uses

[GitHub] spark pull request: [SPARK-14089][CORE][MLLIB] Remove methods that...

2016-03-26 Thread lw-lin
Github user lw-lin commented on the pull request: https://github.com/apache/spark/pull/11910#issuecomment-201806898 @rxin @srowen @mengxr thank you all for the review. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well

[GitHub] spark pull request: [SPARK-14089][CORE][MLLIB] Remove methods that...

2016-03-25 Thread lw-lin
Github user lw-lin commented on the pull request: https://github.com/apache/spark/pull/11910#issuecomment-201156796 Jenkins test this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: [SPARK-14025][STREAMING][WEBUI] Fix streaming ...

2016-03-23 Thread lw-lin
Github user lw-lin commented on the pull request: https://github.com/apache/spark/pull/11845#issuecomment-200573435 Thank you all for your kind review & comments, @srowen @andrewor14 @zsxwing ! --- If your project is set up for it, you can reply to this email and have your r

[GitHub] spark pull request: [SPARK-14089][CORE][MLLIB] Remove methods that...

2016-03-23 Thread lw-lin
Github user lw-lin commented on a diff in the pull request: https://github.com/apache/spark/pull/11910#discussion_r57166028 --- Diff: core/src/main/java/org/apache/spark/api/java/StorageLevels.java --- @@ -40,19 +40,6 @@ * Create a new StorageLevel object. * @param

[GitHub] spark pull request: [SPARK-14089][CORE][MLLIB] Remove methods that...

2016-03-23 Thread lw-lin
Github user lw-lin commented on the pull request: https://github.com/apache/spark/pull/11910#issuecomment-200360008 Updated with a new commit removing two methods deprecated in `ML` since 1.5.x. Recently deprecated methods (since 1.6) throughout Spark are kept. Please let me know

[GitHub] spark pull request: [SPARK-14089][CORE][MLLIB] Remove methods that...

2016-03-22 Thread lw-lin
Github user lw-lin commented on a diff in the pull request: https://github.com/apache/spark/pull/11910#discussion_r57103353 --- Diff: core/src/main/java/org/apache/spark/api/java/StorageLevels.java --- @@ -40,19 +40,6 @@ * Create a new StorageLevel object. * @param

[GitHub] spark pull request: [SPARK-14089][CORE][MLLIB] Remove methods that...

2016-03-22 Thread lw-lin
Github user lw-lin commented on the pull request: https://github.com/apache/spark/pull/11910#issuecomment-200141425 I also checked that these deprecated-and-should-be-removed methods exist only in core and mllib, i.e. no such methods exist in sql, streaming, yarn, etc., so only

[GitHub] spark pull request: [SPARK-14089][CORE][MLLIB] Remove methods that...

2016-03-22 Thread lw-lin
GitHub user lw-lin opened a pull request: https://github.com/apache/spark/pull/11910 [SPARK-14089][CORE][MLLIB] Remove methods that has been deprecated since 1.1.x, 1.2.x and 1.3.x ## What changes were proposed in this pull request? Removed methods that has been deprecated

[GitHub] spark pull request: [SPARK-14025][STREAMING][WEBUI] Fix streaming ...

2016-03-22 Thread lw-lin
Github user lw-lin commented on a diff in the pull request: https://github.com/apache/spark/pull/11845#discussion_r57100194 --- Diff: core/src/main/scala/org/apache/spark/ui/UIUtils.scala --- @@ -417,7 +417,7 @@ private[spark] object UIUtils extends Logging { * attempts

[GitHub] spark pull request: [SPARK-14025][STREAMING][WEBUI] Fix streaming ...

2016-03-22 Thread lw-lin
Github user lw-lin commented on a diff in the pull request: https://github.com/apache/spark/pull/11845#discussion_r57100182 --- Diff: core/src/main/scala/org/apache/spark/ui/UIUtils.scala --- @@ -445,22 +445,36 @@ private[spark] object UIUtils extends Logging

[GitHub] spark pull request: [SPARK-14025][STREAMING][WEBUI] Fix streaming ...

2016-03-22 Thread lw-lin
Github user lw-lin commented on the pull request: https://github.com/apache/spark/pull/11845#issuecomment-199727209 Updated with new commits. Tests passed on my local machine; I wonder why scalastyle checks failed (this PR doesn't touch this code snippet

[GitHub] spark pull request: [SPARK-14025][STREAMING][WEBUI] Fix streaming ...

2016-03-21 Thread lw-lin
Github user lw-lin commented on the pull request: https://github.com/apache/spark/pull/11845#issuecomment-199583350 @andrewor14 thank you for the informative review. Will soon update this PR accordingly. --- If your project is set up for it, you can reply to this email and have your

[GitHub] spark pull request: [SPARK-14028][STREAMING][KINESIS][TESTS] Remov...

2016-03-21 Thread lw-lin
Github user lw-lin commented on the pull request: https://github.com/apache/spark/pull/11850#issuecomment-199169765 @srowen thanks for the review! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does

[GitHub] spark pull request: [SPARK-14028][STREAMING][KINESIS][TESTS] Remov...

2016-03-20 Thread lw-lin
Github user lw-lin commented on a diff in the pull request: https://github.com/apache/spark/pull/11850#discussion_r56760778 --- Diff: external/kinesis-asl/src/test/scala/org/apache/spark/streaming/kinesis/KinesisStreamSuite.scala --- @@ -154,7 +150,9 @@ abstract class

[GitHub] spark pull request: [SPARK-14028][STREAMING][KINESIS][TESTS] Remov...

2016-03-20 Thread lw-lin
Github user lw-lin commented on a diff in the pull request: https://github.com/apache/spark/pull/11850#discussion_r56760767 --- Diff: external/kinesis-asl/src/test/scala/org/apache/spark/streaming/kinesis/KinesisStreamSuite.scala --- @@ -99,14 +99,10 @@ abstract class

[GitHub] spark pull request: [SPARK-14028][STREAMING][KINESIS][TESTS] Remov...

2016-03-20 Thread lw-lin
Github user lw-lin commented on a diff in the pull request: https://github.com/apache/spark/pull/11850#discussion_r56760750 --- Diff: external/kinesis-asl/src/test/scala/org/apache/spark/streaming/kinesis/KinesisFunSuite.scala --- @@ -40,7 +40,7 @@ trait KinesisFunSuite extends

[GitHub] spark pull request: [SPARK-14028][STREAMING][KINESIS][TESTS] Remov...

2016-03-20 Thread lw-lin
Github user lw-lin commented on a diff in the pull request: https://github.com/apache/spark/pull/11850#discussion_r56760713 --- Diff: external/kinesis-asl/src/test/java/org/apache/spark/streaming/kinesis/JavaKinesisStreamSuite.java --- @@ -34,11 +35,13 @@ public class

[GitHub] spark pull request: [SPARK-14028][STREAMING][KINESIS][TESTS] Remov...

2016-03-20 Thread lw-lin
GitHub user lw-lin opened a pull request: https://github.com/apache/spark/pull/11850 [SPARK-14028][STREAMING][KINESIS][TESTS] Remove deprecated methods; fix two other warnings ## What changes were proposed in this pull request? - Removed two methods that has been

[GitHub] spark pull request: [SPARK-14025][STREAMING][WEBUI] Fix streaming ...

2016-03-19 Thread lw-lin
Github user lw-lin commented on the pull request: https://github.com/apache/spark/pull/11845#issuecomment-198718851 Maybe @andrewor14 will explain better :) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project

[GitHub] spark pull request: [SPARK-14025][STREAMING][WEBUI] Fix streaming ...

2016-03-19 Thread lw-lin
Github user lw-lin commented on the pull request: https://github.com/apache/spark/pull/11845#issuecomment-198718502 Ah, I guess the reason is: - for the blue box in the event timeline, itself is clickable both for streaming/non-streaming jobs, so it's unnecessary to contain any

[GitHub] spark pull request: [SPARK-14025][STREAMING][WEBUI] Fix streaming ...

2016-03-19 Thread lw-lin
Github user lw-lin commented on the pull request: https://github.com/apache/spark/pull/11845#issuecomment-198714597 Maybe let's summarize a little bit: - job/stage descriptions are used at 2 places: the `Event Timeline` and the text tables (`CompletedJobs`/`CompletedStages

[GitHub] spark pull request: [SPARK-14025][STREAMING][WEBUI] Fix streaming ...

2016-03-19 Thread lw-lin
Github user lw-lin commented on the pull request: https://github.com/apache/spark/pull/11845#issuecomment-198710234 Actually we've intentionally escaped the description for the event line, so that it will be rendered as plain texts; please see https://github.com/apache/spark/blob

[GitHub] spark pull request: [SPARK-14025][STREAMING][WEBUI] Fix streaming ...

2016-03-19 Thread lw-lin
Github user lw-lin commented on the pull request: https://github.com/apache/spark/pull/11845#issuecomment-198708658 Besides, the blue/green bar in the event line itself is a clickable, linking to the specific job page. The `` thing is superfluous, let's figure out how to remove

[GitHub] spark pull request: [SPARK-14025][STREAMING][WEBUI] Fix streaming ...

2016-03-19 Thread lw-lin
Github user lw-lin commented on the pull request: https://github.com/apache/spark/pull/11845#issuecomment-198707142 @srowen thanks for looking at this! I believe job descriptions were intended to contains only plain texts at first, but HTMLs were introduced in for streaming

[GitHub] spark pull request: [SPARK-14025][STREAMING][WEBUI] Fix streaming ...

2016-03-19 Thread lw-lin
Github user lw-lin commented on the pull request: https://github.com/apache/spark/pull/11845#issuecomment-198695846 @andrewor14 @zsxwing would you mind taking a look at this when you have time? Thanks! --- If your project is set up for it, you can reply to this email and have your

[GitHub] spark pull request: [SPARK-14025][STREAMING][WEBUI] Fix streaming ...

2016-03-19 Thread lw-lin
GitHub user lw-lin opened a pull request: https://github.com/apache/spark/pull/11845 [SPARK-14025][STREAMING][WEBUI] Fix streaming job descriptions on the event line ## What changes were proposed in this pull request? Removed the extra `...` for each streaming job's

[GitHub] spark pull request: [STREAMING][MINOR] Fix a duplicate "be" in com...

2016-03-11 Thread lw-lin
Github user lw-lin commented on the pull request: https://github.com/apache/spark/pull/11650#issuecomment-195655635 Sure. @rxin thank you for your review and patient guidance! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub

[GitHub] spark pull request: [STREAMING][MINOR] Fix a duplicate "be" in com...

2016-03-11 Thread lw-lin
Github user lw-lin closed the pull request at: https://github.com/apache/spark/pull/11650 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature

[GitHub] spark pull request: [STREAMING][MINOR] Fix a duplicate "be" in com...

2016-03-11 Thread lw-lin
Github user lw-lin commented on the pull request: https://github.com/apache/spark/pull/11650#issuecomment-195335583 Sure, I'll close this for now. Thanks for your time! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well

[GitHub] spark pull request: [STREAMING][MINOR] Fix a duplicate "be" in com...

2016-03-11 Thread lw-lin
Github user lw-lin commented on the pull request: https://github.com/apache/spark/pull/11650#issuecomment-195299373 @rxin @zsxwing Would you take a look when you have time? Thanks! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub

[GitHub] spark pull request: [STREAMING][MINOR] Fix a duplicate "be" in com...

2016-03-10 Thread lw-lin
GitHub user lw-lin opened a pull request: https://github.com/apache/spark/pull/11650 [STREAMING][MINOR] Fix a duplicate "be" in comments You can merge this pull request into a Git repository by running: $ git pull https://github.com/lw-lin/spark typo Alternative

[GitHub] spark pull request: [WIP][SPARK-13809][SQL] State store for stream...

2016-03-10 Thread lw-lin
Github user lw-lin commented on a diff in the pull request: https://github.com/apache/spark/pull/11645#discussion_r55794534 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/StateStore.scala --- @@ -0,0 +1,462 @@ +/* + * Licensed

[GitHub] spark pull request: [WIP][SPARK-13809][SQL] State store for stream...

2016-03-10 Thread lw-lin
Github user lw-lin commented on a diff in the pull request: https://github.com/apache/spark/pull/11645#discussion_r55794512 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/StateStore.scala --- @@ -0,0 +1,462 @@ +/* + * Licensed

[GitHub] spark pull request: [WIP][SPARK-13809][SQL] State store for stream...

2016-03-10 Thread lw-lin
Github user lw-lin commented on a diff in the pull request: https://github.com/apache/spark/pull/11645#discussion_r55794434 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/StateStore.scala --- @@ -0,0 +1,462 @@ +/* + * Licensed

[GitHub] spark pull request: [SPARK-13618][STREAMING][WEB-UI] Make Streamin...

2016-03-10 Thread lw-lin
Github user lw-lin closed the pull request at: https://github.com/apache/spark/pull/11634 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature

[GitHub] spark pull request: Display rate limit on streaming web ui part 3

2016-03-10 Thread lw-lin
Github user lw-lin commented on the pull request: https://github.com/apache/spark/pull/11643#issuecomment-195148951 All three parts are now ready for review. I've also drafted a design doc (please see [Spark-13618](https://issues.apache.org/jira/browse/SPARK-13618)), hopefully it can

[GitHub] spark pull request: Display rate limit on streaming web ui part 3

2016-03-10 Thread lw-lin
GitHub user lw-lin opened a pull request: https://github.com/apache/spark/pull/11643 Display rate limit on streaming web ui part 3 ## What changes were proposed in this pull request? This PR makes Streaming web UI display rate-limit lines in the statistics graph

[GitHub] spark pull request: [SPARK-13618][STREAMING][WEB-UI] Make Streamin...

2016-03-10 Thread lw-lin
Github user lw-lin commented on the pull request: https://github.com/apache/spark/pull/11634#issuecomment-194881087 All three parts are now ready for review. I've also drafted a design doc (please see [Spark-13618](https://issues.apache.org/jira/browse/SPARK-13618)), hopefully it can

[GitHub] spark pull request: [SPARK-13618][STREAMING][WEB-UI] Make Streamin...

2016-03-10 Thread lw-lin
Github user lw-lin commented on the pull request: https://github.com/apache/spark/pull/11633#issuecomment-194881019 All three parts are now ready for review. I've also drafted a design doc (please see [Spark-13618](https://issues.apache.org/jira/browse/SPARK-13618)), hopefully it can

[GitHub] spark pull request: [SPARK-13618][STREAMING][WEB-UI] Make Streamin...

2016-03-10 Thread lw-lin
Github user lw-lin commented on the pull request: https://github.com/apache/spark/pull/11470#issuecomment-194880816 All three parts are now ready for review. I've also drafted a design doc (please see [Spark-13618](https://issues.apache.org/jira/browse/SPARK-13618)), hopefully it can

[GitHub] spark pull request: [SPARK-13618][STREAMING][WEB-UI] Make Streamin...

2016-03-10 Thread lw-lin
Github user lw-lin commented on the pull request: https://github.com/apache/spark/pull/11470#issuecomment-194874328 @zsxwing Points taken -- indeed no need to display rate-limit line when an `InputDStream` instance is not _under rate control_: I've added a field

[GitHub] spark pull request: [SPARK-13618][STREAMING][WEB-UI] Make Streamin...

2016-03-10 Thread lw-lin
GitHub user lw-lin opened a pull request: https://github.com/apache/spark/pull/11634 [SPARK-13618][STREAMING][WEB-UI] Make Streaming web UI page display rate-limit lines on statistics graph - Part 3 ## What changes were proposed in this pull request? (Please fill

[GitHub] spark pull request: [SPARK-13618][STREAMING][WEB-UI] Make Streamin...

2016-03-10 Thread lw-lin
GitHub user lw-lin opened a pull request: https://github.com/apache/spark/pull/11633 [SPARK-13618][STREAMING][WEB-UI] Make Streaming web UI page display rate-limit lines on statistics graph - Part 2 ## What changes were proposed in this pull request? (Please fill

[GitHub] spark pull request: [SPARK-7420][STREAMING][TESTS] Enable test: o....

2016-03-09 Thread lw-lin
Github user lw-lin commented on the pull request: https://github.com/apache/spark/pull/11626#issuecomment-194672903 @rxin thanks ! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: Enable test: o.a.s.streaming.JobGeneratorSuite...

2016-03-09 Thread lw-lin
Github user lw-lin commented on the pull request: https://github.com/apache/spark/pull/11626#issuecomment-194639133 Local tests passed, so maybe we can enable this again for 2.0.0. @rxin @zsxwing would you mind taking a look please? Thanks! --- If your project is set up for it, you

[GitHub] spark pull request: Enable test: o.a.s.streaming.JobGeneratorSuite...

2016-03-09 Thread lw-lin
GitHub user lw-lin opened a pull request: https://github.com/apache/spark/pull/11626 Enable test: o.a.s.streaming.JobGeneratorSuite "Do not clear received… ## How was this patch tested? unit test You can merge this pull request into a Git repository by ru

<    1   2   3   4   5   6