Ok, some initial findings about running Polygene on Flink; This is not going to be easy. Flink by default requires the "operators" (functions/transformations, the blocks in the stream processing chain) to be Serializable with java.io.ObjectOutputStream. That is not going to happen any time soon with Polygene. The alternative (I think ) is to create a custom TaskManager, possibly even a custom JobManager, which seems like a major undertaking, with Scala actors (Akka?) and heavy use of java.io.Serializable. At this point in time, I am not even sure that this is at all possible even at that level of integration.
So, I am not going to bother at this time. That said, having a Kafka integration, as described in this thread is something that I will attempt. Cheers Niclas On Wed, Mar 28, 2018 at 6:29 PM, Niclas Hedhman <[email protected]> wrote: > Gang, > > I have been toying with trying to run Polygene on a Flink cluster (no > success yet), and part of that is coming back to having a pluggable > Messaging/Event SPI, which is pluggable in the similar fashion to > persistence. > > I am also getting a clearer point what is needed, and why we need a SPI > and not just one or more libraries for different messaging systems. > > My initial (and possibly naive) thoughts are as follows; > > It all comes down to UnitOfWork and the fact that Queue readers are in > their own threads. > > 1. A Consumer impl starts a UoW, starts a transaction on the queue* > 2. Polls the Queue (potentially blocking) > 3. Data read is deserialized** > 4. ValueComposite is forwarded to all listeners*** > 5. Any kind of regular Polygene happens > 6. Persistence might be invoked and part of the UoW > 7. Producer impl might be called to forward a message to another > queue/topic > 8. Eventually returns to Consumer and the UoW.complete() happens, which > needs to handle the transaction commit to all individual parts, of > UoW.cancel() to rollback all parts. > > > * Does it need to start transaction on any destination queues as well? > > ** Several issue here; > a. Visibility from where? > b. Multiple types on same queue, how to resolve? > c. Data formats not compatible with ValueComposites, such as raw > binary. > > *** addListener or to all Services with some tagging? > > > Non-trivial I would think, but not excessive by Polygene standards. :-) > > WDYAT? > > > Cheers > -- > Niclas Hedhman, Software Developer > http://polygene.apache.org - New Energy for Java > -- Niclas Hedhman, Software Developer http://polygene.apache.org - New Energy for Java
