[GitHub] spark issue #15903: [Doc][Minor] Kafka doc: breakup into lines

2016-11-16 Thread lw-lin
Github user lw-lin commented on the issue: https://github.com/apache/spark/pull/15903 thanks @srowen . I've gone though all markdown files in `docs` module, searched by regular expression `[^\n]\n\- `, found 55 occurrences and only this one out of them requires the treatment. thanks

[GitHub] spark pull request #15903: [Doc][Minor] Kafka doc: breakup into lines

2016-11-16 Thread lw-lin
GitHub user lw-lin opened a pull request: https://github.com/apache/spark/pull/15903 [Doc][Minor] Kafka doc: breakup into lines ## Before ![before](https://cloud.githubusercontent.com/assets/15843379/20340231/99b039fe-ac1b-11e6-9ba9-b44582427459.png) ## After

[GitHub] spark issue #15852: Spark-18187 [SQL] CompactibleFileStreamLog should not us...

2016-11-13 Thread lw-lin
Github user lw-lin commented on the issue: https://github.com/apache/spark/pull/15852 @uncleGen @tcondie thanks for working on this. My major concern is this approach might disallow changing the `compactInterval` once there were at least two compact files. Should we disallow

[GitHub] spark issue #15828: [SPARK-18187][SS] CompactibleFileStreamLog should not re...

2016-11-13 Thread lw-lin
Github user lw-lin commented on the issue: https://github.com/apache/spark/pull/15828 Jenkins retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled

[GitHub] spark issue #15828: [WIP][SPARK-18187][SS] CompactibleFileStreamLog should n...

2016-11-13 Thread lw-lin
Github user lw-lin commented on the issue: https://github.com/apache/spark/pull/15828 rebased. - the first & second commits were a simple refactor of the ` CompactibleFileStreamLogSuite.scala` - **the third commit was the actual change of this patch** --- If

[GitHub] spark issue #15870: [SPARK-18425][Structured Streaming][Tests] Test `Compact...

2016-11-13 Thread lw-lin
Github user lw-lin commented on the issue: https://github.com/apache/spark/pull/15870 Hint for reviewers: - the first commit simply moved some code into its own file; other than that nothing was touched; - the second commit did some refactor so review only the second

[GitHub] spark pull request #15870: [SPARK-18425][Structured Streaming][Tests] Test `...

2016-11-12 Thread lw-lin
GitHub user lw-lin opened a pull request: https://github.com/apache/spark/pull/15870 [SPARK-18425][Structured Streaming][Tests] Test `CompactibleFileStreamLog` directly ## What changes were proposed in this pull request? (Please fill in changes proposed in this fix

[GitHub] spark pull request #15828: [WIP][SS] CompactibleFileStreamLog should not rel...

2016-11-09 Thread lw-lin
GitHub user lw-lin opened a pull request: https://github.com/apache/spark/pull/15828 [WIP][SS] CompactibleFileStreamLog should not rely on "compactInterval" to detect a compaction batch ## What changes were proposed in this pull request? Right now CompactibleFil

[GitHub] spark issue #15480: [SPARK-16845][SQL] `GeneratedClass$SpecificOrdering` gro...

2016-11-08 Thread lw-lin
Github user lw-lin commented on the issue: https://github.com/apache/spark/pull/15480 @hvanhovell it'd be great if you can take a look at this, thanks! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does

[GitHub] spark pull request #15786: [SPARK-18261][Structured Streaming] Add statistic...

2016-11-07 Thread lw-lin
Github user lw-lin commented on a diff in the pull request: https://github.com/apache/spark/pull/15786#discussion_r86902077 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/streaming/MemorySinkSuite.scala --- @@ -187,6 +187,22 @@ class MemorySinkSuite extends StreamTest

[GitHub] spark pull request #15786: [SPARK-18261][Structured Streaming] Add statistic...

2016-11-07 Thread lw-lin
Github user lw-lin commented on a diff in the pull request: https://github.com/apache/spark/pull/15786#discussion_r86901995 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/memory.scala --- @@ -212,4 +212,8 @@ class MemorySink(val schema: StructType

[GitHub] spark issue #15786: [SPARK-18261][Structured Streaming] Add statistics to Me...

2016-11-07 Thread lw-lin
Github user lw-lin commented on the issue: https://github.com/apache/spark/pull/15786 @zsxwing could you also take a look --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled

[GitHub] spark issue #15786: [SPARK-18261][Structured Streaming] Add statistics to Me...

2016-11-06 Thread lw-lin
Github user lw-lin commented on the issue: https://github.com/apache/spark/pull/15786 @brkyvz would you take another look --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled

[GitHub] spark issue #15480: [SPARK-16845][SQL] `GeneratedClass$SpecificOrdering` gro...

2016-11-05 Thread lw-lin
Github user lw-lin commented on the issue: https://github.com/apache/spark/pull/15480 Jenkins retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled

[GitHub] spark pull request #15786: [SPARK-18261][SS] Add statistics to MemorySink fo...

2016-11-05 Thread lw-lin
GitHub user lw-lin opened a pull request: https://github.com/apache/spark/pull/15786 [SPARK-18261][SS] Add statistics to MemorySink for joining ## What changes were proposed in this pull request? Right now, there is no way to join the output of a memory sink with any

[GitHub] spark issue #15480: [SPARK-16845][SQL] `GeneratedClass$SpecificOrdering` gro...

2016-11-04 Thread lw-lin
Github user lw-lin commented on the issue: https://github.com/apache/spark/pull/15480 Jenkins retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled

[GitHub] spark pull request #15480: [SPARK-16845][SQL] `GeneratedClass$SpecificOrderi...

2016-11-04 Thread lw-lin
Github user lw-lin commented on a diff in the pull request: https://github.com/apache/spark/pull/15480#discussion_r86494742 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/CodeGenerator.scala --- @@ -537,7 +537,6 @@ class CodegenContext

[GitHub] spark issue #15681: [SPARK-18176][Streaming][Kafka] Kafka010 .createRDD() sc...

2016-11-02 Thread lw-lin
Github user lw-lin commented on the issue: https://github.com/apache/spark/pull/15681 I'm closing this for now; can re-open if other people should demand this. thanks! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well

[GitHub] spark pull request #15681: [SPARK-18176][Streaming][Kafka] Kafka010 .createR...

2016-11-02 Thread lw-lin
Github user lw-lin closed the pull request at: https://github.com/apache/spark/pull/15681 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature

[GitHub] spark issue #15715: [SPARK-18198][Doc][Streaming] Highlight code snippets

2016-11-02 Thread lw-lin
Github user lw-lin commented on the issue: https://github.com/apache/spark/pull/15715 @srowen thanks -- could we also merge this into branch-2.1? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does

[GitHub] spark issue #15715: [SPARK-18198][Doc][Streaming] Highlight code snippets

2016-11-01 Thread lw-lin
Github user lw-lin commented on the issue: https://github.com/apache/spark/pull/15715 Sounds good! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so

[GitHub] spark issue #15715: [SPARK-18198][Doc][Streaming] Highlight code snippets

2016-11-01 Thread lw-lin
Github user lw-lin commented on the issue: https://github.com/apache/spark/pull/15715 Good question! Actually I had tried keeping the indent of the div tags but un-indenting only the code before I open this pr -- and it looked like this: ![snip20161102_8](https

[GitHub] spark issue #15715: [SPARK-18198][Doc][Streaming] Highlight code snippets

2016-11-01 Thread lw-lin
Github user lw-lin commented on the issue: https://github.com/apache/spark/pull/15715 ![snip20161101_7](https://cloud.githubusercontent.com/assets/15843379/19896158/469d72b0-a08e-11e6-972f-5706ab4aa4c4.png) Please see the screenshot above -- this is `flume doc` -- I got

[GitHub] spark issue #15715: [SPARK-18198][Doc][Streaming] Highlight code snippets

2016-11-01 Thread lw-lin
Github user lw-lin commented on the issue: https://github.com/apache/spark/pull/15715 @srowen thanks. I'm afraid the streaming `flume integration doc`, `kinesis integration doc`, as well as `kafka08 integration doc` also need code highlights, but this `{% hightlight

[GitHub] spark issue #15715: [SPARK-18198][Doc][Streaming] Highlight code snippets

2016-11-01 Thread lw-lin
Github user lw-lin commented on the issue: https://github.com/apache/spark/pull/15715 @koeninger @srowen it'd be great if you could take a look at this too :) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] spark issue #15715: [SPARK-18198][Doc][Streaming] Highlight code snippets

2016-11-01 Thread lw-lin
Github user lw-lin commented on the issue: https://github.com/apache/spark/pull/15715 Fix the leading spaces is needed, because without the fixing the code snippet would contain leading spaces as well(see the pic below), which is quite inconsistent with the other programming guides

[GitHub] spark pull request #15715: [SPARK-18198][Doc][Streaming] Highlight code snip...

2016-11-01 Thread lw-lin
GitHub user lw-lin opened a pull request: https://github.com/apache/spark/pull/15715 [SPARK-18198][Doc][Streaming] Highlight code snippets ## What changes were proposed in this pull request? We should use `{% highlight lang %}``{% endhighlight %}` to highlight code

[GitHub] spark issue #15681: [SPARK-18176][Streaming][Kafka] Kafka010 .createRDD() sc...

2016-11-01 Thread lw-lin
Github user lw-lin commented on the issue: https://github.com/apache/spark/pull/15681 Deprecating the existing one would mean we still need to introduce some `createRDDInternal`, and let the deprecated one call it; then we can just remove the deprecated one some time in the future

[GitHub] spark issue #15681: [SPARK-18176][Streaming][Kafka] Kafka010 .createRDD() sc...

2016-11-01 Thread lw-lin
Github user lw-lin commented on the issue: https://github.com/apache/spark/pull/15681 Thank you @koeninger ! Please let me cc @srowen who's been around to also take a look~ --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub

[GitHub] spark issue #15712: [SPARK-18103][SQL] Rename `MetadataLogFileCatalog` to `M...

2016-11-01 Thread lw-lin
Github user lw-lin commented on the issue: https://github.com/apache/spark/pull/15712 @rxin @ericl please take a look thanks! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark pull request #15712: [SPARK-18103][SQL] Rename `MetadataLogFileCatalog...

2016-11-01 Thread lw-lin
GitHub user lw-lin opened a pull request: https://github.com/apache/spark/pull/15712 [SPARK-18103][SQL] Rename `MetadataLogFileCatalog` to `MetadataLogFileIndex` ## What changes were proposed in this pull request? This is a follow-up to https://github.com/apache/spark/pull

[GitHub] spark issue #15681: [SPARK-18176][Streaming][Kafka] Kafka010 .createRDD() sc...

2016-11-01 Thread lw-lin
Github user lw-lin commented on the issue: https://github.com/apache/spark/pull/15681 Reverted all changes and added another createRDD overload ! :-) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does

[GitHub] spark issue #13575: [SPARK-15472][SQL] Add support for writing in `csv`, `js...

2016-11-01 Thread lw-lin
Github user lw-lin commented on the issue: https://github.com/apache/spark/pull/13575 No problem here. Spark-17924 is super great (I've been watching it all the time) and @rxin thank you for bringing that up! :-D --- If your project is set up for it, you can reply to this email

[GitHub] spark issue #13575: [SPARK-15472][SQL] Add support for writing in `csv`, `js...

2016-11-01 Thread lw-lin
Github user lw-lin commented on the issue: https://github.com/apache/spark/pull/13575 No problem here. Spark-17924 is super great (I've been watching it all the time) and thank you for bringing that up! --- If your project is set up for it, you can reply to this email and have your

[GitHub] spark issue #15681: [SPARK-18176][Streaming][Kafka] Kafka010 .createRDD() sc...

2016-10-30 Thread lw-lin
Github user lw-lin commented on the issue: https://github.com/apache/spark/pull/15681 @koeninger well received and thanks! JIRA opened and unrelated changes removed. Let's see what Jenkins would say. --- If your project is set up for it, you can reply to this email and have

[GitHub] spark issue #15681: [Minor][Streaming][Kafka] Kafka010 .createRDD() scala AP...

2016-10-30 Thread lw-lin
Github user lw-lin commented on the issue: https://github.com/apache/spark/pull/15681 @koeninger could you also take a look at this, thanks! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark issue #15679: [SPARK-16312][Follow-up][STREAMING][KAFKA][DOC] Add java...

2016-10-30 Thread lw-lin
Github user lw-lin commented on the issue: https://github.com/apache/spark/pull/15679 @srowen @koeninger thanks for the review! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark pull request #15679: [SPARK-16312][Follow-up][STREAMING][KAFKA][DOC] A...

2016-10-29 Thread lw-lin
Github user lw-lin commented on a diff in the pull request: https://github.com/apache/spark/pull/15679#discussion_r85642639 --- Diff: docs/streaming-kafka-0-10-integration.md --- @@ -141,9 +214,11 @@ For data stores that support transactions, saving offsets in the same transactio

[GitHub] spark pull request #15679: [SPARK-16312][Follow-up][STREAMING][KAFKA][DOC] A...

2016-10-29 Thread lw-lin
Github user lw-lin commented on a diff in the pull request: https://github.com/apache/spark/pull/15679#discussion_r85642607 --- Diff: docs/streaming-kafka-0-10-integration.md --- @@ -165,6 +240,36 @@ For data stores that support transactions, saving offsets in the same transactio

[GitHub] spark pull request #15679: [SPARK-16312][Follow-up][STREAMING][KAFKA][DOC] A...

2016-10-29 Thread lw-lin
Github user lw-lin commented on a diff in the pull request: https://github.com/apache/spark/pull/15679#discussion_r85641731 --- Diff: docs/streaming-kafka-0-10-integration.md --- @@ -120,15 +184,24 @@ Kafka has an offset commit API that stores offsets in a special Kafka topic

[GitHub] spark pull request #15679: [SPARK-16312][Follow-up][STREAMING][KAFKA][DOC] A...

2016-10-29 Thread lw-lin
Github user lw-lin commented on a diff in the pull request: https://github.com/apache/spark/pull/15679#discussion_r85641730 --- Diff: docs/streaming-kafka-0-10-integration.md --- @@ -120,15 +184,24 @@ Kafka has an offset commit API that stores offsets in a special Kafka topic

[GitHub] spark issue #15679: [SPARK-16312][Follow-up][STREAMING][KAFKA][DOC] Add java...

2016-10-29 Thread lw-lin
Github user lw-lin commented on the issue: https://github.com/apache/spark/pull/15679 @koeninger thanks. These were extracted from my local compiled example projects -- no compiled kafka010 examples yet in the Spark repo, right? --- If your project is set up for it, you can reply

[GitHub] spark pull request #15681: [Minor][Streaming][Kafka] Kafka010 .createRDD() s...

2016-10-29 Thread lw-lin
Github user lw-lin commented on a diff in the pull request: https://github.com/apache/spark/pull/15681#discussion_r85640495 --- Diff: external/kafka-0-10/src/main/scala/org/apache/spark/streaming/kafka010/KafkaUtils.scala --- @@ -56,32 +57,18 @@ object KafkaUtils extends Logging

[GitHub] spark pull request #15681: [Minor][Streaming][Kafka] Kafka010 .createRDD() s...

2016-10-29 Thread lw-lin
Github user lw-lin commented on a diff in the pull request: https://github.com/apache/spark/pull/15681#discussion_r85640467 --- Diff: external/kafka-0-10/src/main/scala/org/apache/spark/streaming/kafka010/KafkaUtils.scala --- @@ -56,32 +57,18 @@ object KafkaUtils extends Logging

[GitHub] spark pull request #15681: [Minor][Streaming][Kafka] Kafka010 .createRDD() s...

2016-10-29 Thread lw-lin
Github user lw-lin commented on a diff in the pull request: https://github.com/apache/spark/pull/15681#discussion_r85640450 --- Diff: external/kafka-0-10/src/main/scala/org/apache/spark/streaming/kafka010/KafkaUtils.scala --- @@ -56,32 +57,18 @@ object KafkaUtils extends Logging

[GitHub] spark pull request #15681: [Minor][Streaming][Kafka] Kafka010 .createRDD() s...

2016-10-29 Thread lw-lin
Github user lw-lin commented on a diff in the pull request: https://github.com/apache/spark/pull/15681#discussion_r85640442 --- Diff: external/kafka-0-10/src/main/scala/org/apache/spark/streaming/kafka010/DirectKafkaInputDStream.scala --- @@ -44,11 +44,6 @@ import

[GitHub] spark pull request #15681: [Minor][Streaming][Kafka] Kafka010 .createRDD() s...

2016-10-29 Thread lw-lin
GitHub user lw-lin opened a pull request: https://github.com/apache/spark/pull/15681 [Minor][Streaming][Kafka] Kafka010 .createRDD() scala API should expect scala Map ## What changes were proposed in this pull request? Thoughout `external/kafka-010`, Java APIs

[GitHub] spark pull request #15679: [SPARK-16312][Follow-up][STREAMING][KAFKA][DOC] A...

2016-10-29 Thread lw-lin
Github user lw-lin commented on a diff in the pull request: https://github.com/apache/spark/pull/15679#discussion_r85640168 --- Diff: docs/streaming-kafka-0-10-integration.md --- @@ -165,6 +240,36 @@ For data stores that support transactions, saving offsets in the same transactio

[GitHub] spark pull request #15679: [SPARK-16312][Follow-up][STREAMING][KAFKA][DOC] A...

2016-10-29 Thread lw-lin
Github user lw-lin commented on a diff in the pull request: https://github.com/apache/spark/pull/15679#discussion_r85640157 --- Diff: docs/streaming-kafka-0-10-integration.md --- @@ -103,6 +153,20 @@ Note that you cannot use `PreferBrokers`, because without the stream

[GitHub] spark pull request #15679: [SPARK-16312][Follow-up][STREAMING][KAFKA][DOC] A...

2016-10-29 Thread lw-lin
Github user lw-lin commented on a diff in the pull request: https://github.com/apache/spark/pull/15679#discussion_r85640159 --- Diff: docs/streaming-kafka-0-10-integration.md --- @@ -165,6 +240,36 @@ For data stores that support transactions, saving offsets in the same transactio

[GitHub] spark pull request #15679: [SPARK-16312][Follow-up][STREAMING][KAFKA][DOC] A...

2016-10-29 Thread lw-lin
Github user lw-lin commented on a diff in the pull request: https://github.com/apache/spark/pull/15679#discussion_r85640154 --- Diff: docs/streaming-kafka-0-10-integration.md --- @@ -85,6 +121,20 @@ If you have a use case that is better suited to batch processing, you can create

[GitHub] spark pull request #15679: [SPARK-16312][Follow-up][STREAMING][KAFKA][DOC] A...

2016-10-29 Thread lw-lin
Github user lw-lin commented on a diff in the pull request: https://github.com/apache/spark/pull/15679#discussion_r85640143 --- Diff: docs/streaming-kafka-0-10-integration.md --- @@ -44,6 +44,42 @@ For Scala/Java applications using SBT/Maven project definitions, link your strea

[GitHub] spark issue #15679: [SPARK-16312][Follow-up][STREAMING][KAFKA][DOC] Add java...

2016-10-28 Thread lw-lin
Github user lw-lin commented on the issue: https://github.com/apache/spark/pull/15679 Oh there's a on-going 2.0.2 RC cut. It'd be great if we can make this into 2.0.2. So also cc @srowen to take a look, thanks! --- If your project is set up for it, you can reply to this email

[GitHub] spark issue #15679: [SPARK-16312][Follow-up][STREAMING][KAFKA][DOC] Add java...

2016-10-28 Thread lw-lin
Github user lw-lin commented on the issue: https://github.com/apache/spark/pull/15679 @koeninger @zsxwing it'd be great if you can take a look, thanks! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does

[GitHub] spark pull request #15679: [SPARK-16312][Follow-up][STREAMING][KAFKA][DOC] A...

2016-10-28 Thread lw-lin
GitHub user lw-lin opened a pull request: https://github.com/apache/spark/pull/15679 [SPARK-16312][Follow-up][STREAMING][KAFKA][DOC] Add java code snippet for Kafka 0.10 integration doc ## What changes were proposed in this pull request? added java code snippet for Kafka

[GitHub] spark issue #15588: [SPARK-18039][Scheduler] fix bug maxRegisteredWaitingTim...

2016-10-22 Thread lw-lin
Github user lw-lin commented on the issue: https://github.com/apache/spark/pull/15588 Spark Streaming would do a very simple dummy job ensure that all slaves have registered before scheduling the `Receiver`s; please see https://github.com/apache/spark/blob/v2.0.0/streaming/src/main

[GitHub] spark issue #13705: [SPARK-15472][SQL] Add support for writing in `csv` form...

2016-10-22 Thread lw-lin
Github user lw-lin commented on the issue: https://github.com/apache/spark/pull/13705 closing this in favor of SPARK-17924 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark pull request #13705: [SPARK-15472][SQL] Add support for writing in `cs...

2016-10-22 Thread lw-lin
Github user lw-lin closed the pull request at: https://github.com/apache/spark/pull/13705 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature

[GitHub] spark pull request #13507: [SPARK-15765][SQL][Streaming] Make continuous Par...

2016-10-20 Thread lw-lin
Github user lw-lin closed the pull request at: https://github.com/apache/spark/pull/13507 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature

[GitHub] spark issue #13507: [SPARK-15765][SQL][Streaming] Make continuous Parquet wr...

2016-10-20 Thread lw-lin
Github user lw-lin commented on the issue: https://github.com/apache/spark/pull/13507 I'm closing this in favor of SPARK-17924, thanks! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request #15480: [SPARK-16845][SQL] `GeneratedClass$SpecificOrderi...

2016-10-18 Thread lw-lin
Github user lw-lin commented on a diff in the pull request: https://github.com/apache/spark/pull/15480#discussion_r83990016 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/GenerateOrdering.scala --- @@ -118,7 +118,45 @@ object

[GitHub] spark issue #15519: [WIP][SQL][STREAMING][TEST] Fix flaky tests in Streaming...

2016-10-17 Thread lw-lin
Github user lw-lin commented on the issue: https://github.com/apache/spark/pull/15519 This looks good to me, thanks! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled

[GitHub] spark issue #15497: [Test][SPARK-16002][Follow-up] Fix flaky test in Streami...

2016-10-17 Thread lw-lin
Github user lw-lin commented on the issue: https://github.com/apache/spark/pull/15497 Please go ahead and take over -- let's fix this before 2.0.2, thanks! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project

[GitHub] spark pull request #15497: [Test][SPARK-16002][Follow-up] Fix flaky test in ...

2016-10-17 Thread lw-lin
Github user lw-lin closed the pull request at: https://github.com/apache/spark/pull/15497 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature

[GitHub] spark issue #15497: [Test][SPARK-16002][Follow-up] Fix flaky test in Streami...

2016-10-17 Thread lw-lin
Github user lw-lin commented on the issue: https://github.com/apache/spark/pull/15497 Jenkins retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled

[GitHub] spark pull request #15497: [Test][SPARK-16002][Follow-up] Fix flaky test in ...

2016-10-17 Thread lw-lin
Github user lw-lin commented on a diff in the pull request: https://github.com/apache/spark/pull/15497#discussion_r83584538 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/streaming/StreamingQueryListenerSuite.scala --- @@ -81,7 +81,7 @@ class StreamingQueryListenerSuite

[GitHub] spark pull request #15497: [Test][SPARK-16002][Follow-up] Fix flaky test in ...

2016-10-17 Thread lw-lin
Github user lw-lin commented on a diff in the pull request: https://github.com/apache/spark/pull/15497#discussion_r83584420 --- Diff: core/src/main/scala/org/apache/spark/util/ManualClock.scala --- @@ -27,6 +27,7 @@ package org.apache.spark.util private[spark] class

[GitHub] spark pull request #15480: [SPARK-16845][SQL] `GeneratedClass$SpecificOrderi...

2016-10-16 Thread lw-lin
Github user lw-lin commented on a diff in the pull request: https://github.com/apache/spark/pull/15480#discussion_r83574850 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/GenerateOrdering.scala --- @@ -118,7 +118,45 @@ object

[GitHub] spark pull request #15480: [SPARK-16845][SQL] `GeneratedClass$SpecificOrderi...

2016-10-16 Thread lw-lin
Github user lw-lin commented on a diff in the pull request: https://github.com/apache/spark/pull/15480#discussion_r83574830 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/GenerateOrdering.scala --- @@ -118,7 +118,45 @@ object

[GitHub] spark pull request #15480: [SPARK-16845][SQL] `GeneratedClass$SpecificOrderi...

2016-10-16 Thread lw-lin
Github user lw-lin commented on a diff in the pull request: https://github.com/apache/spark/pull/15480#discussion_r83574113 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/GenerateOrdering.scala --- @@ -118,7 +118,45 @@ object

[GitHub] spark pull request #15497: [Test][SPARK-16002][Follow-up] Fix flaky test in ...

2016-10-16 Thread lw-lin
Github user lw-lin commented on a diff in the pull request: https://github.com/apache/spark/pull/15497#discussion_r83570900 --- Diff: core/src/main/scala/org/apache/spark/util/ManualClock.scala --- @@ -27,6 +27,7 @@ package org.apache.spark.util private[spark] class

[GitHub] spark pull request #15497: [Test][SPARK-16002][Follow-up] Fix flaky test in ...

2016-10-16 Thread lw-lin
Github user lw-lin commented on a diff in the pull request: https://github.com/apache/spark/pull/15497#discussion_r83570798 --- Diff: core/src/main/scala/org/apache/spark/util/ManualClock.scala --- @@ -27,6 +27,7 @@ package org.apache.spark.util private[spark] class

[GitHub] spark issue #15497: [Test][SPARK-16002][Follow-up] Fix flaky test in Streami...

2016-10-14 Thread lw-lin
Github user lw-lin commented on the issue: https://github.com/apache/spark/pull/15497 Jenkins retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled

[GitHub] spark issue #15492: [DO NOT MERGE][TEST] Testing flakiness of StreamingQuery...

2016-10-14 Thread lw-lin
Github user lw-lin commented on the issue: https://github.com/apache/spark/pull/15492 Could you take a look at my PR https://github.com/apache/spark/pull/15497 - sorry I didn't noticed you've already started testing this. --- If your project is set up for it, you can reply

[GitHub] spark issue #15497: [Test][SPARK-16002][Follow-up] Fix flaky test in Streami...

2016-10-14 Thread lw-lin
Github user lw-lin commented on the issue: https://github.com/apache/spark/pull/15497 @tdas @zsxwing could you take a look --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark pull request #15497: [Test][SPARK-16002][Follow-up] Fix flaky test in ...

2016-10-14 Thread lw-lin
GitHub user lw-lin opened a pull request: https://github.com/apache/spark/pull/15497 [Test][SPARK-16002][Follow-up] Fix flaky test in StreamingQueryListenerSuite ## What changes were proposed in this pull request? editing ## How was this patch tested

[GitHub] spark issue #15480: [SPARK-16845][SQL] `GeneratedClass$SpecificOrdering` gro...

2016-10-14 Thread lw-lin
Github user lw-lin commented on the issue: https://github.com/apache/spark/pull/15480 @davies @kiszk it'd be great if you can take a look --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark issue #15480: [SPARK-16845][SQL] `GeneratedClass$SpecificOrdering` gro...

2016-10-14 Thread lw-lin
Github user lw-lin commented on the issue: https://github.com/apache/spark/pull/15480 Jenkins retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled

[GitHub] spark issue #15480: [SPARK-16845][SQL] `GeneratedClass$SpecificOrdering` gro...

2016-10-14 Thread lw-lin
Github user lw-lin commented on the issue: https://github.com/apache/spark/pull/15480 Flaky test I think not related to this PR. Thanks, @HyukjinKwon ! Jenkins retest this please. --- If your project is set up for it, you can reply to this email and have your reply appear

[GitHub] spark pull request #15482: [Hot fix] ignore("single listener, check trigger ...

2016-10-14 Thread lw-lin
Github user lw-lin closed the pull request at: https://github.com/apache/spark/pull/15482 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature

[GitHub] spark issue #15482: [Hot fix] ignore("single listener, check trigger statuse...

2016-10-14 Thread lw-lin
Github user lw-lin commented on the issue: https://github.com/apache/spark/pull/15482 Now that td has issued a commit https://github.com/apache/spark/commit/05800b4b4e7873ebc445dfcd020b76d7539686e1 exactly like this, I'm closing this. Sure `flaky-test` is more proper than `hot-fix

[GitHub] spark pull request #15482: [Hot fix] ignore("single listener, check trigger ...

2016-10-14 Thread lw-lin
GitHub user lw-lin opened a pull request: https://github.com/apache/spark/pull/15482 [Hot fix] ignore("single listener, check trigger statuses") for now You can merge this pull request into a Git repository by running: $ git pull https://github.com/lw-lin/spa

[GitHub] spark pull request #15480: [SPARK-16845][SQL] `GeneratedClass$SpecificOrderi...

2016-10-14 Thread lw-lin
GitHub user lw-lin opened a pull request: https://github.com/apache/spark/pull/15480 [SPARK-16845][SQL] `GeneratedClass$SpecificOrdering` grows beyond 64 KB ## What changes were proposed in this pull request? Prior to this patch, we'll generate `compare

[GitHub] spark issue #15461: [SPARK-16845][SQL] `GeneratedClass$SpecificOrdering` gro...

2016-10-13 Thread lw-lin
Github user lw-lin commented on the issue: https://github.com/apache/spark/pull/15461 Closing for now; will re-open soon. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled

[GitHub] spark pull request #15461: [SPARK-16845][SQL] `GeneratedClass$SpecificOrderi...

2016-10-13 Thread lw-lin
Github user lw-lin closed the pull request at: https://github.com/apache/spark/pull/15461 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature

[GitHub] spark pull request #15461: [SPARK-16845][SQL] `GeneratedClass$SpecificOrderi...

2016-10-13 Thread lw-lin
GitHub user lw-lin opened a pull request: https://github.com/apache/spark/pull/15461 [SPARK-16845][SQL] `GeneratedClass$SpecificOrdering` grows beyond 64 KB ## What changes were proposed in this pull request? Prior to this patch, we'll generate `compare

[GitHub] spark pull request #15195: [SPARK-17632][SQL]make console sink and other sin...

2016-09-27 Thread lw-lin
Github user lw-lin commented on a diff in the pull request: https://github.com/apache/spark/pull/15195#discussion_r80654592 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/streaming/DataStreamWriter.scala --- @@ -290,8 +284,8 @@ final class DataStreamWriter[T] private[sql

[GitHub] spark pull request #14118: [SPARK-16462][SPARK-16460][SPARK-15144][SQL] Make...

2016-09-15 Thread lw-lin
Github user lw-lin commented on a diff in the pull request: https://github.com/apache/spark/pull/14118#discussion_r79088930 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/csv/CSVInferSchema.scala --- @@ -232,66 +232,62 @@ private[csv] object

[GitHub] spark pull request #14118: [SPARK-16462][SPARK-16460][SPARK-15144][SQL] Make...

2016-09-15 Thread lw-lin
Github user lw-lin commented on a diff in the pull request: https://github.com/apache/spark/pull/14118#discussion_r79088096 --- Diff: python/pyspark/sql/readwriter.py --- @@ -329,7 +329,8 @@ def csv(self, path, schema=None, sep=None, encoding=None, quote=None, escape=Non

[GitHub] spark pull request #14118: [SPARK-16462][SPARK-16460][SPARK-15144][SQL] Make...

2016-09-15 Thread lw-lin
Github user lw-lin commented on a diff in the pull request: https://github.com/apache/spark/pull/14118#discussion_r79088141 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/csv/CSVInferSchema.scala --- @@ -232,66 +232,62 @@ private[csv] object

[GitHub] spark issue #14118: [SPARK-16462][SPARK-16460][SPARK-15144][SQL] Make CSV ca...

2016-09-13 Thread lw-lin
Github user lw-lin commented on the issue: https://github.com/apache/spark/pull/14118 @HyukjinKwon thanks for the information! @srowen yea I still think this is good to go. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub

[GitHub] spark issue #14118: [SPARK-16462][SPARK-16460][SPARK-15144][SQL] Make CSV ca...

2016-09-13 Thread lw-lin
Github user lw-lin commented on the issue: https://github.com/apache/spark/pull/14118 Jenkins retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled

[GitHub] spark pull request #14914: [SPARK-17359][SQL][MLLib] Use ArrayBuffer.+=(A) i...

2016-09-01 Thread lw-lin
Github user lw-lin commented on a diff in the pull request: https://github.com/apache/spark/pull/14914#discussion_r77131280 --- Diff: mllib-local/src/main/scala/org/apache/spark/ml/linalg/Matrices.scala --- @@ -999,7 +999,7 @@ object Matrices { val data = new

[GitHub] spark pull request #14914: [SPARK-17359][SQL][MLLib] Use ArrayBuffer.+=(A) i...

2016-09-01 Thread lw-lin
Github user lw-lin commented on a diff in the pull request: https://github.com/apache/spark/pull/14914#discussion_r77130880 --- Diff: mllib-local/src/main/scala/org/apache/spark/ml/linalg/Matrices.scala --- @@ -999,7 +999,7 @@ object Matrices { val data = new

[GitHub] spark pull request #14914: [SPARK-17359][SQL][MLLib] Use ArrayBuffer.+=(A) i...

2016-09-01 Thread lw-lin
Github user lw-lin commented on a diff in the pull request: https://github.com/apache/spark/pull/14914#discussion_r77128029 --- Diff: mllib-local/src/main/scala/org/apache/spark/ml/linalg/Matrices.scala --- @@ -999,7 +999,7 @@ object Matrices { val data = new

[GitHub] spark pull request #14914: [SPARK-17359][SQL][MLLib] Use ArrayBuffer.+=(A) i...

2016-09-01 Thread lw-lin
GitHub user lw-lin opened a pull request: https://github.com/apache/spark/pull/14914 [SPARK-17359][SQL][MLLib] Use ArrayBuffer.+=(A) instead of ArrayBuffer.append(A) in performance critical paths ## What changes were proposed in this pull request? We should generally use

[GitHub] spark pull request #14298: [SPARK-16283][SQL] Implement `percentile_approx` ...

2016-08-29 Thread lw-lin
Github user lw-lin closed the pull request at: https://github.com/apache/spark/pull/14298 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature

[GitHub] spark issue #14298: [SPARK-16283][SQL] Implement `percentile_approx` SQL fun...

2016-08-29 Thread lw-lin
Github user lw-lin commented on the issue: https://github.com/apache/spark/pull/14298 @clockfly Your PR's improvements overall look reasonable to me, so I'm closing this one. Good job! --- If your project is set up for it, you can reply to this email and have your reply appear

[GitHub] spark issue #14118: [SPARK-16462][SPARK-16460][SPARK-15144][SQL] Make CSV ca...

2016-08-29 Thread lw-lin
Github user lw-lin commented on the issue: https://github.com/apache/spark/pull/14118 Jenkins retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled

[GitHub] spark issue #14118: [SPARK-16462][SPARK-16460][SPARK-15144][SQL] Make CSV ca...

2016-08-29 Thread lw-lin
Github user lw-lin commented on the issue: https://github.com/apache/spark/pull/14118 > What if I am writing explicitly an empty string out? Does it become just 1,,2? Yes. It becomes `1,,2` in 2.0, and the same `1,,2` with this patch -- no behavior changes. &g

<    1   2   3   4   5   6   >