[
https://issues.apache.org/jira/browse/FLINK-377?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14301888#comment-14301888
]
ASF GitHub Bot commented on FLINK-377:
--------------------------------------
Github user rmetzger commented on the pull request:
https://github.com/apache/flink/pull/202#issuecomment-72533930
Wordcount with build-in data works :+1: nice.
```
robert@robert-tower ...k-0.9-SNAPSHOT-bin/flink-0.9-SNAPSHOT (git)-[papipr]
% ./bin/pyflink3.sh pyflink.py - /home/robert/flink-workdir/yarnLog
/tmp/yarnPyWC
02/02/2015 21:20:34 Job execution switched to status RUNNING.
02/02/2015 21:20:34 DataSource (TextSource)(1/1) switched to SCHEDULED
02/02/2015 21:20:34 DataSource (TextSource)(1/1) switched to DEPLOYING
02/02/2015 21:20:34 DataSource (TextSource)(1/1) switched to RUNNING
02/02/2015 21:20:34 MapPartition (PythonFlatMap -> PythonCombine)(1/1)
switched to SCHEDULED
02/02/2015 21:20:34 MapPartition (PythonFlatMap -> PythonCombine)(1/1)
switched to DEPLOYING
02/02/2015 21:20:34 MapPartition (PythonFlatMap -> PythonCombine)(1/1)
switched to RUNNING
02/02/2015 21:20:34 DataSource (TextSource)(1/1) switched to FINISHED
```
I wanted to run wordcount locally on some serious data, but sadly it seems
that the job somehow deadlocked.
```
"MapPartition (PythonFlatMap -> PythonCombine) (1/1)" #85 daemon prio=5
os_prio=0 tid=0x0000000001b59000 nid=0x855 runnable [0x00007fd73b3f4000]
java.lang.Thread.State: RUNNABLE
at java.net.PlainDatagramSocketImpl.receive0(Native Method)
- locked <0x00000000fad36828> (a java.net.PlainDatagramSocketImpl)
at
java.net.AbstractPlainDatagramSocketImpl.receive(AbstractPlainDatagramSocketImpl.java:143)
- locked <0x00000000fad36828> (a java.net.PlainDatagramSocketImpl)
at java.net.DatagramSocket.receive(DatagramSocket.java:781)
- locked <0x00000000fad367a0> (a java.net.DatagramPacket)
- locked <0x00000000fad367c8> (a java.net.DatagramSocket)
at
org.apache.flink.languagebinding.api.java.common.streaming.Streamer.streamBufferWithoutGroups(Streamer.java:172)
at
org.apache.flink.languagebinding.api.java.python.functions.PythonMapPartition.mapPartition(PythonMapPartition.java:55)
at
org.apache.flink.runtime.operators.MapPartitionDriver.run(MapPartitionDriver.java:98)
at
org.apache.flink.runtime.operators.RegularPactTask.run(RegularPactTask.java:496)
at
org.apache.flink.runtime.operators.RegularPactTask.invoke(RegularPactTask.java:360)
at
org.apache.flink.runtime.execution.RuntimeEnvironment.run(RuntimeEnvironment.java:204)
at java.lang.Thread.run(Thread.java:745)
```
```
"Thread-23" #86 daemon prio=5 os_prio=0 tid=0x00007fd634001800 nid=0x857
runnable [0x00007fd73b4f4000]
java.lang.Thread.State: RUNNABLE
at java.io.FileInputStream.readBytes(Native Method)
at java.io.FileInputStream.read(FileInputStream.java:234)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:284)
at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
- locked <0x00000000fad3a440> (a
java.lang.UNIXProcess$ProcessPipeInputStream)
at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:284)
at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:326)
at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:178)
- locked <0x00000000fad3e968> (a java.io.InputStreamReader)
at java.io.InputStreamReader.read(InputStreamReader.java:184)
at java.io.BufferedReader.fill(BufferedReader.java:161)
at java.io.BufferedReader.readLine(BufferedReader.java:324)
- locked <0x00000000fad3e968> (a java.io.InputStreamReader)
at java.io.BufferedReader.readLine(BufferedReader.java:389)
at
org.apache.flink.languagebinding.api.java.common.streaming.StreamPrinter.run(StreamPrinter.java:34)
```
I have to dig further to understand whats going on.
My understanding of the pull request right now is the following: I see that
this change was a LOT of work and that there had been some iterations of
improvement.
What the code certainly needs are a few more developers. This will probably
automatically lead to cleaner code, better code comments, better error handling
and so on.
I'm still not convinced to merge the code in the state its currently in.
Therefore, I'm just facing too many issues right now. That the example in the
documentation is broken is certainly not the dealbreaker here. Issues like hard
to find error messages or the issues I had with the wordcount (I don't know if
its the runtime or an issue of the Python code)
Please don't get my feedback here wrong. I appreciate and see that this has
been a lot of work, and we are probably close to 100% (we are probably in the
90s already). Whats needed are people to test this in different environments,
with different expectations etc. .. then we'll probably quickly achieve 100%
(for a first prototype!)
In my opinion, we have two options now:
- We try quickly find people (mailing lists, contacts) willing to test the
Python API and @zentol fixes these issues until we have enough confidence to
merge it.
- We go a step back with the Python API and follow the approach similar to
the streaming system and the graph API ("gelly"): We create a separate github
project where the project can exist for some time. Such a repository would
allow @zentol to easily collaborate with other contributors.
I think the two options very much depend on the number of people we find to
test the python API and the willingness of @zentol to fix the issues alone.
If others share my concern with limited number of contributors, we should
probably put the code into a separate directory for a while.
> Create a general purpose framework for language bindings
> --------------------------------------------------------
>
> Key: FLINK-377
> URL: https://issues.apache.org/jira/browse/FLINK-377
> Project: Flink
> Issue Type: Improvement
> Reporter: GitHub Import
> Assignee: Chesnay Schepler
> Labels: github-import
> Fix For: pre-apache
>
>
> A general purpose API to run operators with arbitrary binaries.
> This will allow to run Stratosphere programs written in Python, JavaScript,
> Ruby, Go or whatever you like.
> We suggest using Google Protocol Buffers for data serialization. This is the
> list of languages that currently support ProtoBuf:
> https://code.google.com/p/protobuf/wiki/ThirdPartyAddOns
> Very early prototype with python:
> https://github.com/rmetzger/scratch/tree/learn-protobuf (basically testing
> protobuf)
> For Ruby: https://github.com/infochimps-labs/wukong
> Two new students working at Stratosphere (@skunert and @filiphaase) are
> working on this.
> The reference binding language will be for Python, but other bindings are
> very welcome.
> The best name for this so far is "stratosphere-lang-bindings".
> I created this issue to track the progress (and give everybody a chance to
> comment on this)
> ---------------- Imported from GitHub ----------------
> Url: https://github.com/stratosphere/stratosphere/issues/377
> Created by: [rmetzger|https://github.com/rmetzger]
> Labels: enhancement,
> Assignee: [filiphaase|https://github.com/filiphaase]
> Created at: Tue Jan 07 19:47:20 CET 2014
> State: open
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
