I guess this message from 2016 is very related of what I am looking for ( http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Flink-Execution-Plan-td4290.html). I am posting here for future references.
I am going to implement a toy example to visualize this. Do you guys see this description as actual of latest Flink source code? *--* *-- Felipe Gutierrez* *-- skype: felipe.o.gutierrez* *--* *https://felipeogutierrez.blogspot.com <https://felipeogutierrez.blogspot.com>* On Thu, Nov 29, 2018 at 12:01 PM Felipe Gutierrez < felipe.o.gutier...@gmail.com> wrote: > thanks Kostas for the quick reply, > > yes. It is related to my previous question. > > When you said "But if you know what operation to push down" -> This is > what I am trying to search on Flink code. I want to know the operation on > the fly. > The component on Flink that will say to me that there is a filter on the > query specified by the user. I want to get this metadata and send a message > to my RPi through a Flink connector (I guess this is the way to do) and the > data stream will come to Flink already filtered. > > I intend to start with a simple and naive example. Do you know which > component on Flink I can get the operations on the fly that are > running inside a query? > > thanks > *--* > *-- Felipe Gutierrez* > > *-- skype: felipe.o.gutierrez* > *--* *https://felipeogutierrez.blogspot.com > <https://felipeogutierrez.blogspot.com>* > > > On Thu, Nov 29, 2018 at 11:18 AM Kostas Kloudas < > k.klou...@data-artisans.com> wrote: > >> Hi again, >> >> I forgot to say that, unfortunately, I am not familiar with Apache >> Edgent, but if you can write your filter in Edgent's programming model, >> Then you can push your data from Edgent to a third party storage system >> (e.g. Kafka, HDFS, etc) and use Flink's connectors, instead of >> having to implement a custom source. >> >> Cheers, >> Kostas >> >> On Thu, Nov 29, 2018 at 11:08 AM Kostas Kloudas < >> k.klou...@data-artisans.com> wrote: >> >>> Hi Felipe, >>> >>> This seems related to your previous question about a custom scheduler >>> that knows which task to run on which machine. >>> As Chesnay said, this is a rather involved and laborious task, if you >>> want to do it as a general framework. >>> >>> But if you know what operation to push down, then why not decoupling the >>> two and implementing the filtering as a separate job >>> running on your Raspberry and a new job which consumes the output of the >>> first and does the analytics? >>> >>> Cheers, >>> Kostas >>> >>> On Thu, Nov 29, 2018 at 10:23 AM Felipe Gutierrez < >>> felipe.o.gutier...@gmail.com> wrote: >>> >>>> Hi, >>>> >>>> I am trying to design a little prototype with Flink and Apache Edgent ( >>>> http://edgent.apache.org/) and I would like some help on the direction >>>> for it. I am running Flink at my laptop and Edgent on my Raspberry Pi with >>>> a simple filter for a proximity sensor ( >>>> https://github.com/felipegutierrez/explore-rpi/blob/master/src/main/java/org/sense/edgent/app/UltrasonicEdgentApp.java >>>> ). >>>> >>>> My idea is to push down the filter operator from Flink to the Raspberry >>>> Pi which is running Apache Edgent. With this in mind, where do you guys >>>> advise me to start? >>>> >>>> I have some ideas to study... >>>> 1 - Try to get the list of operators that Flink is about to execute on >>>> the JobManager. source: >>>> https://ci.apache.org/projects/flink/flink-docs-stable/internals/job_scheduling.html >>>> 2 - Implement a connector to Apache Edgent in order to exchange >>>> messages between Flink and Edgent. >>>> >>>> Do you guys think in another source that is interesting regarding my >>>> prototype? >>>> >>>> Thanks, >>>> Felipe >>>> *--* >>>> *-- Felipe Gutierrez* >>>> >>>> *-- skype: felipe.o.gutierrez* >>>> *--* *https://felipeogutierrez.blogspot.com >>>> <https://felipeogutierrez.blogspot.com>* >>>> >>>
