[GitHub] spark pull request: [SPARK-8378][Streaming]Add the Python API for ...

[tdas]

Github user tdas commented on a diff in the pull request:

https://github.com/apache/spark/pull/6830#discussion_r33989723
  
--- Diff: external/flume-assembly/pom.xml ---
@@ -0,0 +1,135 @@
+
+
+
+http://maven.apache.org/POM/4.0.0"; 
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"; 
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 
http://maven.apache.org/xsd/maven-4.0.0.xsd";>
+  4.0.0
+  
+org.apache.spark
+spark-parent_2.10
+1.5.0-SNAPSHOT
+../../pom.xml
+  
+
+  org.apache.spark
+  spark-streaming-flume-assembly_2.10
--- End diff --

1. Good point, we can probably exclude scala. Why are the Hadoop libraries
included? Definitely not through spark as spark-streaming is marked as
provided dependency?

2. The whole point of making the assembly JAR is to make it easy to run
spark streaming + flume applications, especially in python where the users
will not be creating mvn/sbt projects to include the dependencies in an
uber jar. The most convenient for python users who want to use flume stream
is to add --jar .jar. Hence flume and its all
its dependencies need to be included.




On Mon, Jul 6, 2015 at 2:19 PM, Marcelo Vanzin 
wrote:

> In external/flume-assembly/pom.xml
> :
>
> > +  ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or 
implied.
> > +  ~ See the License for the specific language governing permissions and
> > +  ~ limitations under the License.
> > +  -->
> > +
> > +http://maven.apache.org/POM/4.0.0"; 
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"; 
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 
http://maven.apache.org/xsd/maven-4.0.0.xsd";>
> > +  4.0.0
> > +  
> > +org.apache.spark
> > +spark-parent_2.10
> > +1.5.0-SNAPSHOT
> > +../../pom.xml
> > +  
> > +
> > +  org.apache.spark
> > +  spark-streaming-flume-assembly_2.10
>
> Well, two things:
>
>- spark-streaming-flume, the existing artifact, has transitive
>dependencies on flume. So if you add it using the ivy support in
>spark-submit, you'd get those.
>- Even if you want to add this assembly, it currently packages way
>more than just flume. It includes all of Scala and Hadoop libraries 
and a
>bunch of other things, as I mentioned above.
>
> So any way you look at it, there is still something to be fixed here.
>
> —
> Reply to this email directly or view it on GitHub
> .
>



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-8378][Streaming]Add the Python API for ...

[tdas]

Github user tdas commented on a diff in the pull request:

https://github.com/apache/spark/pull/6830#discussion_r33983012
  
--- Diff: external/flume-assembly/pom.xml ---
@@ -0,0 +1,135 @@
+
+
+
+http://maven.apache.org/POM/4.0.0"; 
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"; 
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 
http://maven.apache.org/xsd/maven-4.0.0.xsd";>
+  4.0.0
+  
+org.apache.spark
+spark-parent_2.10
+1.5.0-SNAPSHOT
+../../pom.xml
+  
+
+  org.apache.spark
+  spark-streaming-flume-assembly_2.10
--- End diff --

Nope, none of the fume stuff is present in the spark-assembly. That is
precisely why this assembly JAR with spark-streaming-flume and flume+its
dependencies were generated.

On Mon, Jul 6, 2015 at 2:02 PM, Marcelo Vanzin 
wrote:

> In external/flume-assembly/pom.xml
> :
>
> > +  ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or 
implied.
> > +  ~ See the License for the specific language governing permissions and
> > +  ~ limitations under the License.
> > +  -->
> > +
> > +http://maven.apache.org/POM/4.0.0"; 
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"; 
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 
http://maven.apache.org/xsd/maven-4.0.0.xsd";>
> > +  4.0.0
> > +  
> > +org.apache.spark
> > +spark-parent_2.10
> > +1.5.0-SNAPSHOT
> > +../../pom.xml
> > +  
> > +
> > +  org.apache.spark
> > +  spark-streaming-flume-assembly_2.10
>
> Hi guys,
>
> Sorry I'm late to the party, but why is this new assembly necessary?
>
> It creates an 80MB jar file that repackages a bunch of things already
> present in the Spark assembly (e.g. scala.*, org.hadoop.*, and a whole
> lot of other things). If python/pyspark/streaming/flume.py is meant to be
> used inside a Spark application, aren't those dependencies already 
provided
> by the Spark assembly? In which case all that is needed is the existing
> spark-streaming-flume artifact?
>
> —
> Reply to this email directly or view it on GitHub
> .
>



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-8378][Streaming]Add the Python API for ...

[vanzin]

Github user vanzin commented on a diff in the pull request:

https://github.com/apache/spark/pull/6830#discussion_r33983627
  
--- Diff: external/flume-assembly/pom.xml ---
@@ -0,0 +1,135 @@
+
+
+
+http://maven.apache.org/POM/4.0.0"; 
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"; 
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 
http://maven.apache.org/xsd/maven-4.0.0.xsd";>
+  4.0.0
+  
+org.apache.spark
+spark-parent_2.10
+1.5.0-SNAPSHOT
+../../pom.xml
+  
+
+  org.apache.spark
+  spark-streaming-flume-assembly_2.10
--- End diff --

Well, two things:

- spark-streaming-flume, the existing artifact, has transitive dependencies 
on flume. So if you add it using the ivy support in spark-submit, you'd get 
those.
- Even if you want to add this assembly, it currently packages way more 
than just flume. It includes all of Scala and Hadoop libraries and a bunch of 
other things, as I mentioned above.

So any way you look at it, there is still something to be fixed here.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-8378][Streaming]Add the Python API for ...

[vanzin]

Github user vanzin commented on a diff in the pull request:

https://github.com/apache/spark/pull/6830#discussion_r33981901
  
--- Diff: external/flume-assembly/pom.xml ---
@@ -0,0 +1,135 @@
+
+
+
+http://maven.apache.org/POM/4.0.0"; 
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"; 
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 
http://maven.apache.org/xsd/maven-4.0.0.xsd";>
+  4.0.0
+  
+org.apache.spark
+spark-parent_2.10
+1.5.0-SNAPSHOT
+../../pom.xml
+  
+
+  org.apache.spark
+  spark-streaming-flume-assembly_2.10
--- End diff --

Hi guys,

Sorry I'm late to the party, but why is this new assembly necessary?

It creates an 80MB jar file that repackages a bunch of things already 
present in the Spark assembly (e.g. `scala.*`, `org.hadoop.*`, and a whole lot 
of other things). If `python/pyspark/streaming/flume.py` is meant to be used 
inside a Spark application, aren't those dependencies already provided by the 
Spark assembly? In which case all that is needed is the existing 
`spark-streaming-flume` artifact?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-8378][Streaming]Add the Python API for ...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/spark/pull/6830


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-8378][Streaming]Add the Python API for ...

[tdas]

Github user tdas commented on the pull request:

https://github.com/apache/spark/pull/6830#issuecomment-117794987
  
I am merging this to master. Thanks @zsxwing !


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-8378][Streaming]Add the Python API for ...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/6830#issuecomment-117701143
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-8378][Streaming]Add the Python API for ...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/6830#issuecomment-117700768
  
  [Test build #36249 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/36249/console)
 for   PR 6830 at commit 
[`78dfdac`](https://github.com/apache/spark/commit/78dfdac7216c3e68a42c6534f497289212a31a8a).
 * This patch **passes all tests**.
 * This patch merges cleanly.
 * This patch adds the following public classes _(experimental)_:
  * `class FlumeUtils(object):`



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-8378][Streaming]Add the Python API for ...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/6830#issuecomment-117638962
  
  [Test build #36249 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/36249/consoleFull)
 for   PR 6830 at commit 
[`78dfdac`](https://github.com/apache/spark/commit/78dfdac7216c3e68a42c6534f497289212a31a8a).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-8378][Streaming]Add the Python API for ...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/6830#issuecomment-117638654
  
Merged build started.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-8378][Streaming]Add the Python API for ...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/6830#issuecomment-117619061
  
 Merged build triggered.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-8378][Streaming]Add the Python API for ...

[zsxwing]

Github user zsxwing commented on the pull request:

https://github.com/apache/spark/pull/6830#issuecomment-117617495
  
retest this please


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-8378][Streaming]Add the Python API for ...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/6830#issuecomment-117599401
  
  [Test build #36241 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/36241/console)
 for   PR 6830 at commit 
[`78dfdac`](https://github.com/apache/spark/commit/78dfdac7216c3e68a42c6534f497289212a31a8a).
 * This patch **fails Spark unit tests**.
 * This patch merges cleanly.
 * This patch adds the following public classes _(experimental)_:
  * `class FlumeUtils(object):`



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-8378][Streaming]Add the Python API for ...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/6830#issuecomment-117599522
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-8378][Streaming]Add the Python API for ...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/6830#issuecomment-117545192
  
  [Test build #36241 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/36241/consoleFull)
 for   PR 6830 at commit 
[`78dfdac`](https://github.com/apache/spark/commit/78dfdac7216c3e68a42c6534f497289212a31a8a).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-8378][Streaming]Add the Python API for ...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/6830#issuecomment-117544780
  
Merged build started.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-8378][Streaming]Add the Python API for ...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/6830#issuecomment-117544724
  
 Merged build triggered.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-8378][Streaming]Add the Python API for ...

[tdas]

Github user tdas commented on the pull request:

https://github.com/apache/spark/pull/6830#issuecomment-117544824
  
LGTM. I will merge tomorrow morning after the current run passes. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-8378][Streaming]Add the Python API for ...

[tdas]

Github user tdas commented on the pull request:

https://github.com/apache/spark/pull/6830#issuecomment-117542146
  
Jenkins, test this please.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-8378][Streaming]Add the Python API for ...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/6830#issuecomment-117434623
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-8378][Streaming]Add the Python API for ...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/6830#issuecomment-117434415
  
  [Test build #36224 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/36224/console)
 for   PR 6830 at commit 
[`78dfdac`](https://github.com/apache/spark/commit/78dfdac7216c3e68a42c6534f497289212a31a8a).
 * This patch **passes all tests**.
 * This patch merges cleanly.
 * This patch adds the following public classes _(experimental)_:
  * `class FlumeUtils(object):`



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-8378][Streaming]Add the Python API for ...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/6830#issuecomment-117426309
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-8378][Streaming]Add the Python API for ...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/6830#issuecomment-117426260
  
  [Test build #36217 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/36217/console)
 for   PR 6830 at commit 
[`78dfdac`](https://github.com/apache/spark/commit/78dfdac7216c3e68a42c6534f497289212a31a8a).
 * This patch **passes all tests**.
 * This patch merges cleanly.
 * This patch adds the following public classes _(experimental)_:
  * `class FlumeUtils(object):`



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-8378][Streaming]Add the Python API for ...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/6830#issuecomment-117416646
  
  [Test build #36224 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/36224/consoleFull)
 for   PR 6830 at commit 
[`78dfdac`](https://github.com/apache/spark/commit/78dfdac7216c3e68a42c6534f497289212a31a8a).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-8378][Streaming]Add the Python API for ...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/6830#issuecomment-117416157
  
 Merged build triggered.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-8378][Streaming]Add the Python API for ...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/6830#issuecomment-117416211
  
Merged build started.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-8378][Streaming]Add the Python API for ...

[tdas]

Github user tdas commented on the pull request:

https://github.com/apache/spark/pull/6830#issuecomment-117414890
  
Jenkins, test this please.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-8378][Streaming]Add the Python API for ...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/6830#issuecomment-117413815
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-8378][Streaming]Add the Python API for ...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/6830#issuecomment-117413665
  
  [Test build #36215 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/36215/console)
 for   PR 6830 at commit 
[`78dfdac`](https://github.com/apache/spark/commit/78dfdac7216c3e68a42c6534f497289212a31a8a).
 * This patch **passes all tests**.
 * This patch merges cleanly.
 * This patch adds the following public classes _(experimental)_:
  * `class FlumeUtils(object):`



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-8378][Streaming]Add the Python API for ...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/6830#issuecomment-117386301
  
  [Test build #36217 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/36217/consoleFull)
 for   PR 6830 at commit 
[`78dfdac`](https://github.com/apache/spark/commit/78dfdac7216c3e68a42c6534f497289212a31a8a).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-8378][Streaming]Add the Python API for ...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/6830#issuecomment-117385618
  
 Merged build triggered.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-8378][Streaming]Add the Python API for ...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/6830#issuecomment-117385627
  
Merged build started.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-8378][Streaming]Add the Python API for ...

[tdas]

Github user tdas commented on the pull request:

https://github.com/apache/spark/pull/6830#issuecomment-117385396
  
Jenkins, test this please.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-8378][Streaming]Add the Python API for ...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/6830#issuecomment-117378467
  
  [Test build #36215 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/36215/consoleFull)
 for   PR 6830 at commit 
[`78dfdac`](https://github.com/apache/spark/commit/78dfdac7216c3e68a42c6534f497289212a31a8a).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-8378][Streaming]Add the Python API for ...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/6830#issuecomment-117378209
  
 Merged build triggered.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-8378][Streaming]Add the Python API for ...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/6830#issuecomment-117378214
  
Merged build started.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-8378][Streaming]Add the Python API for ...

[tdas]

Github user tdas commented on the pull request:

https://github.com/apache/spark/pull/6830#issuecomment-117377836
  
Jenkins, test this please.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-8378][Streaming]Add the Python API for ...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/6830#issuecomment-117364363
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-8378][Streaming]Add the Python API for ...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/6830#issuecomment-117364265
  
  [Test build #36189 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/36189/console)
 for   PR 6830 at commit 
[`78dfdac`](https://github.com/apache/spark/commit/78dfdac7216c3e68a42c6534f497289212a31a8a).
 * This patch **fails PySpark unit tests**.
 * This patch merges cleanly.
 * This patch adds the following public classes _(experimental)_:
  * `class FlumeUtils(object):`



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-8378][Streaming]Add the Python API for ...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/6830#issuecomment-117360622
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-8378][Streaming]Add the Python API for ...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/6830#issuecomment-117360475
  
  [Test build #36186 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/36186/console)
 for   PR 6830 at commit 
[`78dfdac`](https://github.com/apache/spark/commit/78dfdac7216c3e68a42c6534f497289212a31a8a).
 * This patch **passes all tests**.
 * This patch merges cleanly.
 * This patch adds the following public classes _(experimental)_:
  * `class FlumeUtils(object):`



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-8378][Streaming]Add the Python API for ...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/6830#issuecomment-117360083
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-8378][Streaming]Add the Python API for ...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/6830#issuecomment-117359977
  
  [Test build #36184 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/36184/console)
 for   PR 6830 at commit 
[`78dfdac`](https://github.com/apache/spark/commit/78dfdac7216c3e68a42c6534f497289212a31a8a).
 * This patch **passes all tests**.
 * This patch merges cleanly.
 * This patch adds the following public classes _(experimental)_:
  * `class FlumeUtils(object):`



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-8378][Streaming]Add the Python API for ...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/6830#issuecomment-117333759
  
  [Test build #36189 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/36189/consoleFull)
 for   PR 6830 at commit 
[`78dfdac`](https://github.com/apache/spark/commit/78dfdac7216c3e68a42c6534f497289212a31a8a).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-8378][Streaming]Add the Python API for ...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/6830#issuecomment-11767
  
Merged build started.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-8378][Streaming]Add the Python API for ...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/6830#issuecomment-117333244
  
 Merged build triggered.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-8378][Streaming]Add the Python API for ...

[tdas]

Github user tdas commented on the pull request:

https://github.com/apache/spark/pull/6830#issuecomment-117332072
  
Jenkins, retest this please


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-8378][Streaming]Add the Python API for ...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/6830#issuecomment-117330371
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-8378][Streaming]Add the Python API for ...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/6830#issuecomment-117330368
  
  [Test build #36188 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/36188/console)
 for   PR 6830 at commit 
[`78dfdac`](https://github.com/apache/spark/commit/78dfdac7216c3e68a42c6534f497289212a31a8a).
 * This patch **fails some tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-8378][Streaming]Add the Python API for ...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/6830#issuecomment-117330364
  
  [Test build #36188 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/36188/consoleFull)
 for   PR 6830 at commit 
[`78dfdac`](https://github.com/apache/spark/commit/78dfdac7216c3e68a42c6534f497289212a31a8a).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-8378][Streaming]Add the Python API for ...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/6830#issuecomment-117330170
  
 Merged build triggered.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-8378][Streaming]Add the Python API for ...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/6830#issuecomment-117330208
  
Merged build started.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-8378][Streaming]Add the Python API for ...

[tdas]

Github user tdas commented on the pull request:

https://github.com/apache/spark/pull/6830#issuecomment-117329694
  
test this please.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-8378][Streaming]Add the Python API for ...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/6830#issuecomment-117325488
  
  [Test build #36186 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/36186/consoleFull)
 for   PR 6830 at commit 
[`78dfdac`](https://github.com/apache/spark/commit/78dfdac7216c3e68a42c6534f497289212a31a8a).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-8378][Streaming]Add the Python API for ...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/6830#issuecomment-117325181
  
 Merged build triggered.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-8378][Streaming]Add the Python API for ...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/6830#issuecomment-117325221
  
Merged build started.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-8378][Streaming]Add the Python API for ...

[tdas]

Github user tdas commented on the pull request:

https://github.com/apache/spark/pull/6830#issuecomment-117323076
  
test this please.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-8378][Streaming]Add the Python API for ...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/6830#issuecomment-117320745
  
  [Test build #36184 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/36184/consoleFull)
 for   PR 6830 at commit 
[`78dfdac`](https://github.com/apache/spark/commit/78dfdac7216c3e68a42c6534f497289212a31a8a).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-8378][Streaming]Add the Python API for ...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/6830#issuecomment-117319659
  
Merged build started.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-8378][Streaming]Add the Python API for ...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/6830#issuecomment-117319578
  
 Merged build triggered.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-8378][Streaming]Add the Python API for ...

[JoshRosen]

Github user JoshRosen commented on the pull request:

https://github.com/apache/spark/pull/6830#issuecomment-117318196
  
Jenkins, retest this please.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-8378][Streaming]Add the Python API for ...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/6830#issuecomment-117318039
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-8378][Streaming]Add the Python API for ...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/6830#issuecomment-117314459
  
 Merged build triggered.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-8378][Streaming]Add the Python API for ...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/6830#issuecomment-117314492
  
Merged build started.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-8378][Streaming]Add the Python API for ...

[tdas]

Github user tdas commented on the pull request:

https://github.com/apache/spark/pull/6830#issuecomment-117313845
  
test this please.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-8378][Streaming]Add the Python API for ...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/6830#issuecomment-117249500
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-8378][Streaming]Add the Python API for ...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/6830#issuecomment-117249322
  
  [Test build #36141 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/36141/console)
 for   PR 6830 at commit 
[`78dfdac`](https://github.com/apache/spark/commit/78dfdac7216c3e68a42c6534f497289212a31a8a).
 * This patch **passes all tests**.
 * This patch merges cleanly.
 * This patch adds the following public classes _(experimental)_:
  * `class FlumeUtils(object):`



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-8378][Streaming]Add the Python API for ...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/6830#issuecomment-117206660
  
 Merged build triggered.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-8378][Streaming]Add the Python API for ...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/6830#issuecomment-117206708
  
  [Test build #36141 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/36141/consoleFull)
 for   PR 6830 at commit 
[`78dfdac`](https://github.com/apache/spark/commit/78dfdac7216c3e68a42c6534f497289212a31a8a).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-8378][Streaming]Add the Python API for ...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/6830#issuecomment-117206694
  
Merged build started.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-8378][Streaming]Add the Python API for ...

[zsxwing]

Github user zsxwing commented on the pull request:

https://github.com/apache/spark/pull/6830#issuecomment-117206128
  
retest this please


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-8378][Streaming]Add the Python API for ...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/6830#issuecomment-117190364
  
  [Test build #36137 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/36137/console)
 for   PR 6830 at commit 
[`78dfdac`](https://github.com/apache/spark/commit/78dfdac7216c3e68a42c6534f497289212a31a8a).
 * This patch **fails MiMa tests**.
 * This patch merges cleanly.
 * This patch adds the following public classes _(experimental)_:
  * `class FlumeUtils(object):`



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-8378][Streaming]Add the Python API for ...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/6830#issuecomment-117190420
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-8378][Streaming]Add the Python API for ...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/6830#issuecomment-117187364
  
  [Test build #36137 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/36137/consoleFull)
 for   PR 6830 at commit 
[`78dfdac`](https://github.com/apache/spark/commit/78dfdac7216c3e68a42c6534f497289212a31a8a).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-8378][Streaming]Add the Python API for ...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/6830#issuecomment-117187194
  
 Merged build triggered.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-8378][Streaming]Add the Python API for ...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/6830#issuecomment-117187230
  
Merged build started.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-8378][Streaming]Add the Python API for ...

[zsxwing]

Github user zsxwing commented on the pull request:

https://github.com/apache/spark/pull/6830#issuecomment-117186676
  
retest this please


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-8378][Streaming]Add the Python API for ...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/6830#issuecomment-117168961
  
  [Test build #36132 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/36132/console)
 for   PR 6830 at commit 
[`78dfdac`](https://github.com/apache/spark/commit/78dfdac7216c3e68a42c6534f497289212a31a8a).
 * This patch **fails Scala style tests**.
 * This patch merges cleanly.
 * This patch adds the following public classes _(experimental)_:
  * `class FlumeUtils(object):`



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-8378][Streaming]Add the Python API for ...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/6830#issuecomment-117168975
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-8378][Streaming]Add the Python API for ...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/6830#issuecomment-117167704
  
  [Test build #36132 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/36132/consoleFull)
 for   PR 6830 at commit 
[`78dfdac`](https://github.com/apache/spark/commit/78dfdac7216c3e68a42c6534f497289212a31a8a).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-8378][Streaming]Add the Python API for ...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/6830#issuecomment-117167183
  
Merged build started.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-8378][Streaming]Add the Python API for ...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/6830#issuecomment-117164257
  
 Merged build triggered.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-8378][Streaming]Add the Python API for ...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/6830#issuecomment-117158079
  
  [Test build #36129 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/36129/console)
 for   PR 6830 at commit 
[`f1bf3c0`](https://github.com/apache/spark/commit/f1bf3c05e4582c95025e0bd7e1876062db4be1ed).
 * This patch **fails Spark unit tests**.
 * This patch merges cleanly.
 * This patch adds the following public classes _(experimental)_:
  * `class FlumeUtils(object):`



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-8378][Streaming]Add the Python API for ...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/6830#issuecomment-117158131
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-8378][Streaming]Add the Python API for ...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/6830#issuecomment-117117995
  
  [Test build #36129 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/36129/consoleFull)
 for   PR 6830 at commit 
[`f1bf3c0`](https://github.com/apache/spark/commit/f1bf3c05e4582c95025e0bd7e1876062db4be1ed).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-8378][Streaming]Add the Python API for ...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/6830#issuecomment-117117793
  
 Merged build triggered.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-8378][Streaming]Add the Python API for ...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/6830#issuecomment-117117841
  
Merged build started.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-8378][Streaming]Add the Python API for ...

[zsxwing]

Github user zsxwing commented on the pull request:

https://github.com/apache/spark/pull/6830#issuecomment-117117336
  
@tdas updated as per your comments. I will also check the flakiness locally.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-8378][Streaming]Add the Python API for ...

[zsxwing]

Github user zsxwing commented on a diff in the pull request:

https://github.com/apache/spark/pull/6830#discussion_r33558137
  
--- Diff: 
external/flume/src/test/scala/org/apache/spark/streaming/flume/FlumePollingStreamSuite.scala
 ---
@@ -89,146 +81,55 @@ class FlumePollingStreamSuite extends SparkFunSuite 
with BeforeAndAfter with Log
   }
 
   private def testFlumePolling(): Unit = {
-// Start the channel and sink.
-val context = new Context()
-context.put("capacity", channelCapacity.toString)
-context.put("transactionCapacity", "1000")
-context.put("keep-alive", "0")
-val channel = new MemoryChannel()
-Configurables.configure(channel, context)
-
-val sink = new SparkSink()
-context.put(SparkSinkConfig.CONF_HOSTNAME, "localhost")
-context.put(SparkSinkConfig.CONF_PORT, String.valueOf(0))
-Configurables.configure(sink, context)
-sink.setChannel(channel)
-sink.start()
-
-writeAndVerify(Seq(sink), Seq(channel))
-assertChannelIsEmpty(channel)
-sink.stop()
-channel.stop()
+try {
+  val port = utils.startSingleSink()
+
+  writeAndVerify(Seq(port))
+  utils.assertChannelsAreEmpty()
+} finally {
+  utils.close()
--- End diff --

`testFlumePolling` and `testFlumePollingMultipleHost` will be called 
multiple times in a single test but `utils.close()` should be called every time.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-8378][Streaming]Add the Python API for ...

[tdas]

Github user tdas commented on the pull request:

https://github.com/apache/spark/pull/6830#issuecomment-117095213
  
Overall the patch seems good. Have some small comments. 
Somethings to keep in mind. The flume tests have been historically flaky, 
so I am little concerned that these are being changed. So once the code 
stabilizes, I will do many runs on it to see its flakiness.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-8378][Streaming]Add the Python API for ...

[tdas]

Github user tdas commented on a diff in the pull request:

https://github.com/apache/spark/pull/6830#discussion_r33555262
  
--- Diff: 
external/flume/src/main/scala/org/apache/spark/streaming/flume/PollingFlumeTestUtils.scala
 ---
@@ -0,0 +1,211 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.streaming.flume
+
+import java.util.concurrent._
+import java.util.{List => JList, Map => JMap}
+
+import scala.collection.JavaConversions._
+import scala.collection.mutable.ArrayBuffer
+
+import com.google.common.base.Charsets.UTF_8
+import org.apache.flume.event.EventBuilder
+import org.apache.flume.Context
+import org.apache.flume.channel.MemoryChannel
+import org.apache.flume.conf.Configurables
+
+import org.apache.spark.streaming.flume.sink.{SparkSinkConfig, SparkSink}
+
+/**
+ * Share codes for Scala and Python unit tests
+ */
+private[flume] class PollingFlumeTestUtils {
+
+  private val batchCount = 5
+  private val eventsPerBatch = 100
+  private val totalEventsPerChannel = batchCount * eventsPerBatch
+  private val channelCapacity = 5000
+
+  def getEventsPerBatch: Int = eventsPerBatch
--- End diff --

Is this necessary? Isnt `eventsPerBatch` accessible from Java and Python?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-8378][Streaming]Add the Python API for ...

[tdas]

Github user tdas commented on a diff in the pull request:

https://github.com/apache/spark/pull/6830#discussion_r33554971
  
--- Diff: 
external/flume/src/test/scala/org/apache/spark/streaming/flume/FlumePollingStreamSuite.scala
 ---
@@ -53,6 +43,8 @@ class FlumePollingStreamSuite extends SparkFunSuite with 
BeforeAndAfter with Log
 .setMaster("local[2]")
 .setAppName(this.getClass.getSimpleName)
 
+  var utils = new PollingFlumeTestUtils
+
   def beforeFunction() {
 logInfo("Using manual clock")
 conf.set("spark.streaming.clock", "org.apache.spark.util.ManualClock")
--- End diff --

I think that can be moved into the `conf` above and before can be removed. 
Mind doing this small additional change?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-8378][Streaming]Add the Python API for ...

[tdas]

Github user tdas commented on a diff in the pull request:

https://github.com/apache/spark/pull/6830#discussion_r33554843
  
--- Diff: 
external/flume/src/test/scala/org/apache/spark/streaming/flume/FlumePollingStreamSuite.scala
 ---
@@ -89,146 +81,55 @@ class FlumePollingStreamSuite extends SparkFunSuite 
with BeforeAndAfter with Log
   }
 
   private def testFlumePolling(): Unit = {
-// Start the channel and sink.
-val context = new Context()
-context.put("capacity", channelCapacity.toString)
-context.put("transactionCapacity", "1000")
-context.put("keep-alive", "0")
-val channel = new MemoryChannel()
-Configurables.configure(channel, context)
-
-val sink = new SparkSink()
-context.put(SparkSinkConfig.CONF_HOSTNAME, "localhost")
-context.put(SparkSinkConfig.CONF_PORT, String.valueOf(0))
-Configurables.configure(sink, context)
-sink.setChannel(channel)
-sink.start()
-
-writeAndVerify(Seq(sink), Seq(channel))
-assertChannelIsEmpty(channel)
-sink.stop()
-channel.stop()
+try {
+  val port = utils.startSingleSink()
+
+  writeAndVerify(Seq(port))
+  utils.assertChannelsAreEmpty()
+} finally {
+  utils.close()
--- End diff --

Isnt this better done in scala tests `after`?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-8378][Streaming]Add the Python API for ...

[tdas]

Github user tdas commented on a diff in the pull request:

https://github.com/apache/spark/pull/6830#discussion_r33554587
  
--- Diff: 
external/flume/src/test/scala/org/apache/spark/streaming/flume/FlumePollingStreamSuite.scala
 ---
@@ -53,6 +43,8 @@ class FlumePollingStreamSuite extends SparkFunSuite with 
BeforeAndAfter with Log
 .setMaster("local[2]")
 .setAppName(this.getClass.getSimpleName)
 
+  var utils = new PollingFlumeTestUtils
--- End diff --

Why is this a var not a val?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-8378][Streaming]Add the Python API for ...

[tdas]

Github user tdas commented on a diff in the pull request:

https://github.com/apache/spark/pull/6830#discussion_r33554191
  
--- Diff: 
external/flume/src/main/scala/org/apache/spark/streaming/flume/FlumeUtils.scala 
---
@@ -236,3 +242,71 @@ object FlumeUtils {
 createPollingStream(jssc.ssc, addresses, storageLevel, maxBatchSize, 
parallelism)
   }
 }
+
+/**
+ * This is a helper class that wraps the methods in FlumeUtils into more 
Python-friendly class and
+ * function so that it can be easily instantiated and called from Python's 
FlumeUtils.
+ */
+private class FlumeUtilsPythonHelper {
+
+  def createStream(
+  jssc: JavaStreamingContext,
+  hostname: String,
+  port: Int,
+  storageLevel: StorageLevel,
+  enableDecompression: Boolean
+): JavaPairDStream[Array[Byte], Array[Byte]] = {
+val dstream = FlumeUtils.createStream(jssc, hostname, port, 
storageLevel, enableDecompression)
+FlumeUtilsPythonHelper.toDStreamForPython(dstream)
+  }
+
+  def createPollingStream(
+  jssc: JavaStreamingContext,
+  hosts: JList[String],
+  ports: JList[Int],
+  storageLevel: StorageLevel,
+  maxBatchSize: Int,
+  parallelism: Int
+): JavaPairDStream[Array[Byte], Array[Byte]] = {
+assert(hosts.length == ports.length)
+val addresses = hosts.zip(ports).map {
+  case (host, port) => new InetSocketAddress(host, port)
+}
+val dstream = FlumeUtils.createPollingStream(
+  jssc.ssc, addresses, storageLevel, maxBatchSize, parallelism)
+FlumeUtilsPythonHelper.toDStreamForPython(dstream)
+  }
+
+}
+
+private object FlumeUtilsPythonHelper {
+
+  private def stringMapToByteArray(map: JMap[CharSequence, CharSequence]): 
Array[Byte] = {
+val byteStream = new ByteArrayOutputStream()
+val output = new DataOutputStream(byteStream)
+try {
+  output.writeInt(map.size)
+  map.foreach { kv =>
+PythonRDD.writeUTF(kv._1.toString, output)
+PythonRDD.writeUTF(kv._2.toString, output)
+  }
+  byteStream.toByteArray
+}
+finally {
+  output.close()
+}
+  }
+
+  private def toDStreamForPython(dstream: 
JavaReceiverInputDStream[SparkFlumeEvent]):
--- End diff --

The name isnt very intuitive, could be `toByteArrayPairDStream`.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-8378][Streaming]Add the Python API for ...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/6830#issuecomment-117052846
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-8378][Streaming]Add the Python API for ...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/6830#issuecomment-117052744
  
  [Test build #36099 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/36099/console)
 for   PR 6830 at commit 
[`0449723`](https://github.com/apache/spark/commit/04497231b3d0c1022ece037efd48516ccc6d1995).
 * This patch **passes all tests**.
 * This patch merges cleanly.
 * This patch adds the following public classes _(experimental)_:
  * `class FlumeUtils(object):`



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-8378][Streaming]Add the Python API for ...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/6830#issuecomment-117035109
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-8378][Streaming]Add the Python API for ...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/6830#issuecomment-117034871
  
  [Test build #36087 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/36087/console)
 for   PR 6830 at commit 
[`0449723`](https://github.com/apache/spark/commit/04497231b3d0c1022ece037efd48516ccc6d1995).
 * This patch **passes all tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-8378][Streaming]Add the Python API for ...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/6830#issuecomment-117003809
  
  [Test build #36099 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/36099/consoleFull)
 for   PR 6830 at commit 
[`0449723`](https://github.com/apache/spark/commit/04497231b3d0c1022ece037efd48516ccc6d1995).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org