entrypoint for executing job in task manager

Here is my understanding of how job submission works in Flink. When submitting a job to job manager via REST API, we provide a entry class. Job manager then evaluate job graph and ship serialized operators to task manager. Task manager then open operators and run tasks. My app would typically requ

Re: entrypoint for executing job in task manager

Hi Steven, Curious how you solved this for you use case. Did you ever find a satisfactory approach? Thanks in advance, Bob -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

Re: entrypoint for executing job in task manager

Please see FLINK-14184 which should fully support such use cases in the future. Feel free to vote for it if you believe it would help your use case. -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

Re: entrypoint for executing job in task manager

Open method is called just before any elements are processed. You can hook in any initialisation logic there, including initialisation of a static context. However keep in mind, that since this context is static, it will be shared between multiple operators (if you are running parallelism > numb

Re: entrypoint for executing job in task manager

We use Guice for dependency injection. We need to install *additional* Guice modules (for bindings) when setting up this static context of Guice injector. Calling the static initializer from operator open method won't really help. Not all operators are implemented by app developer who want to inst

Re: entrypoint for executing job in task manager

I don’t think there is such hook in the Flink code now. You will have to walk around this issue somehow in user space. Maybe you could make a contract that every operator before touching Guice, should call static synchronized method `initializeGuiceContext`. This method could search the classp

Re: entrypoint for executing job in task manager

It would be great to understand a bit more what the exact requirements here are, and what setup you use. I am not a dependency injection expert, so let me know if what I am suggesting here is complete bogus. *(1) Fix set of libraries for Dependency Injection, or dedicated container images per ap

Re: entrypoint for executing job in task manager

Thanks, let me clarify the requirement. Sorry that it wasn't clear in the original email. Here is our setup. these 3 dirs are added to classpath * flink/lib: core flink jars (like flink-dist_2.11, flink-shaded-hadoop2-uber) * spaaslib: many jars pulled in our internal platform * jobs: a single fa

Re: entrypoint for executing job in task manager

Well this thread was super helpful. I'm also looking for ways to integrate DI In particular something like the JVM init hooks for TM and JM would be nice. On Wed, Mar 21, 2018 at 5:08 PM, Steven Wu wrote: > Thanks, let me clarify the requirement. Sorry that it wasn't clear in the > original em