Re: Spring with Apache Beam
-d...@beam.apache.org, +user@beam.apache.org How are you trying to inject your application context? Have you looked at the JvmInitializer.beforeProcessing[1] to create your application context? 1: https://github.com/apache/beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/harness/JvmInitializer.java On Fri, Oct 4, 2019 at 12:32 PM Jitendra kumavat wrote: > Hi, > > I want to add Spring framework in my apache beam project. Somehow i am > unable to inject the Spring Application context to executing ParDo > functions. I couldn't find the way to do so? Can you please let me know how > to integrate Spring runtime application context with Apache Beam pipeline. > > Thanks, > Jitendra >
Re: Spring with Apache Beam
1. won't work since it is happening at pipeline construction time and not pipeline execution time. 2. only works if your application context is scoped to the DoFn instance and doesn't have things you want to possibly share across DoFn instances. You could also try and make it a PipelineOption that is tagged with @JsonIgnore and also has a @Default.InstanceFactory like this[1]. This way when it is accessed by your DoFn it will be initialized for the first time and shared within your process. Making it a PipelineOption would also allow you to pass in preinitialized versions for testing. 1: https://github.com/apache/beam/blob/8267c223425bc201be700babbe596d133b79686e/sdks/java/extensions/google-cloud-platform-core/src/main/java/org/apache/beam/sdk/extensions/gcp/options/GcpOptions.java#L127 On Wed, Oct 9, 2019 at 1:10 PM Jitendra kumavat wrote: > Hi Luke, > > Thanks a lot for your reply. > I tried couple of options which is as follows. > > 1. Initialise the context in main method only. and use it. Creating the > context: > new AnnotationConfigApplicationContext(AppConfig.class); > 2. Creating the context on DoFn.Startup method. > > Unfortunately none of the worked perfectly, later works but it has issue > with @ComponentScan. > Please let me know your comments for the same. > > I will also try this JvmInitializer for context initialisation. > > Thanks, > Jitendra > > On Wed, Oct 9, 2019 at 12:48 PM Luke Cwik wrote: > >> -d...@beam.apache.org, +user@beam.apache.org >> >> How are you trying to inject your application context? >> Have you looked at the JvmInitializer.beforeProcessing[1] to create your >> application context? >> >> 1: >> https://github.com/apache/beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/harness/JvmInitializer.java >> >> >> >> >> On Fri, Oct 4, 2019 at 12:32 PM Jitendra kumavat >> wrote: >> >>> Hi, >>> >>> I want to add Spring framework in my apache beam project. Somehow i am >>> unable to inject the Spring Application context to executing ParDo >>> functions. I couldn't find the way to do so? Can you please let me know how >>> to integrate Spring runtime application context with Apache Beam pipeline. >>> >>> Thanks, >>> Jitendra >>> >>
Re: Spring with Apache Beam
You shouldn't need to call it before running the pipeline as you are doing
(you can if you want but its not necessary).
Have you created a service META-INF entry for the JvmInitializer you have
created or are using @AutoService?
This is the relevant bit of the documentation[1]. Here is some good docs
for how to use @AutoService[2].
1:
https://github.com/apache/beam/blob/f3ce8669b50837d48ab0d0ee9a1298ce3b5bc61c/sdks/java/core/src/main/java/org/apache/beam/sdk/harness/JvmInitializer.java#L30
2: https://github.com/google/auto/tree/master/service
On Thu, Oct 10, 2019 at 5:29 PM Jitendra kumavat
wrote:
> Hi Luke,
>
> I tried the JvmIntializer.beforeProccssing method to initialize the spring
> application context. But it seems not working.
>
>
> * Below is my class definitions. *
>
> JvmInitializer class with context initialization.
>
> public class GcmJvmInitializer implements JvmInitializer {
> @Override
> public void beforeProcessing(PipelineOptions options){
> System.out.println("Starting Custom GcmJvmInitializer");
> ApplicationContext applicationContext = new
> AnnotationConfigApplicationContext(
> AppConfig.class);
> for (String beanName : applicationContext.getBeanDefinitionNames()) {
> System.out.println(beanName);
> }
> System.out.println("Stopping Custom GcmJvmInitializer");
> }
> }
>
>
> *Appconfig*
>
> @Configuration
> @PropertySource("classpath:gcm.properties")
> @ComponentScan(basePackages = {"com.liveramp.intl.gcm"})
> public class AppConfig {
>
> // Bean definitions with @Bean annotation.
>
> }
>
>
> And i am using JvmInitializers to inject the same. Below is the my main
> method for the same.
>
>
>
> public static void main(String[] args) {
>
> PCollection lines = pipeline.apply("Read Lines",
> TextIO.read().from(inputPathProvider));
>
> //.. follows other pipeline transformations.
>
> //run pipeline
> JvmInitializers.runBeforeProcessing(options);
> pipeline.run().waitUntilFinish();
>
> }
>
>
> Is this the right way to use it? Or is there other way round? Please let me
> know.
>
>
> Thanks,
>
> Jitendra
>
>
> On Wed, Oct 9, 2019 at 1:32 PM Luke Cwik wrote:
>
>> 1. won't work since it is happening at pipeline construction time and not
>> pipeline execution time.
>> 2. only works if your application context is scoped to the DoFn instance
>> and doesn't have things you want to possibly share across DoFn instances.
>>
>> You could also try and make it a PipelineOption that is tagged
>> with @JsonIgnore and also has a @Default.InstanceFactory like this[1]. This
>> way when it is accessed by your DoFn it will be initialized for the first
>> time and shared within your process. Making it a PipelineOption would also
>> allow you to pass in preinitialized versions for testing.
>>
>> 1:
>> https://github.com/apache/beam/blob/8267c223425bc201be700babbe596d133b79686e/sdks/java/extensions/google-cloud-platform-core/src/main/java/org/apache/beam/sdk/extensions/gcp/options/GcpOptions.java#L127
>>
>> On Wed, Oct 9, 2019 at 1:10 PM Jitendra kumavat
>> wrote:
>>
>>> Hi Luke,
>>>
>>> Thanks a lot for your reply.
>>> I tried couple of options which is as follows.
>>>
>>> 1. Initialise the context in main method only. and use it. Creating the
>>> context:
>>> new AnnotationConfigApplicationContext(AppConfig.class);
>>> 2. Creating the context on DoFn.Startup method.
>>>
>>> Unfortunately none of the worked perfectly, later works but it has issue
>>> with @ComponentScan.
>>> Please let me know your comments for the same.
>>>
>>> I will also try this JvmInitializer for context initialisation.
>>>
>>> Thanks,
>>> Jitendra
>>>
>>> On Wed, Oct 9, 2019 at 12:48 PM Luke Cwik wrote:
>>>
-d...@beam.apache.org, +user@beam.apache.org
How are you trying to inject your application context?
Have you looked at the JvmInitializer.beforeProcessing[1] to create
your application context?
1:
https://github.com/apache/beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/harness/JvmInitializer.java
On Fri, Oct 4, 2019 at 12:32 PM Jitendra kumavat <
jkumavat1...@gmail.com> wrote:
> Hi,
>
> I want to add Spring framework in my apache beam project. Somehow i
> am unable to inject the Spring Application context to executing ParDo
> functions. I couldn't find the way to do so? Can you please let me know
> how
> to integrate Spring runtime application context with Apache Beam pipeline.
>
> Thanks,
> Jitendra
>
Re: Spring with Apache Beam
+user to see if anyone else can provide guidance.
On Fri, Oct 11, 2019 at 5:31 PM Jitendra kumavat
wrote:
> I have added the plugin and checked my, jar it contains the my service
> registered. Still unable to run my JvmInitializer class.
>
>
> org.apache.maven.plugins
> maven-shade-plugin
> 3.2.1
>
>
>
> shade
>
>
>
>implementation="org.apache.maven.plugins.shade.resource.ServicesResourceTransformer"/>
>
>
>
>
>
>
>
> On Fri, Oct 11, 2019 at 2:56 PM Luke Cwik wrote:
>
>> It is a common issue for users build processes to use the Maven shade
>> plugin or jarjar and to lose the META-INF/services files.
>> Can you verify that the jar's you submitted to Dataflow contain the
>> META-INF/services files?
>>
>> See this stackoverflow question
>> https://stackoverflow.com/questions/44365545/apache-beam-unable-to-find-registrar-for-gs
>> for
>> a similar problem for more details.
>>
>>
>> On Fri, Oct 11, 2019 at 2:38 PM Jitendra kumavat
>> wrote:
>>
>>> Hi Luke,
>>>
>>> I added @Autoservice for my custom intializer/processor and i can see
>>> the entries in META-INF/Services folder in my build class. Please see the
>>> attached screenshot. Still while running the dataflow job on google cloud i
>>> can not see the my System.out statements in logs. It is not picking up
>>> the GcmJvmInitializer, i believe so.
>>>
>>>
>>> Steps which i follow to run the job:
>>>
>>> 1. mvn clean install
>>> 2. create dataflow template in gcs bucket using mvn compile command.
>>> 3. run the job using gcloud dataflow jobs
>>>
>>> Can you please tell me what's wrong i am doing here. Thanks.
>>>
>>> Below is my classes structure:
>>>
>>> @AutoService(Processor.class)
>>> @SupportedAnnotationTypes(
>>> "org.springframework.context.annotation.*")
>>> @SupportedSourceVersion(SourceVersion.RELEASE_8)
>>> public class MyProcessor extends AbstractProcessor {
>>> @Override
>>> public boolean process(Set annotations,
>>> RoundEnvironment roundEnv) {
>>> return false;
>>> }
>>> }
>>>
>>>
>>>
>>>
>>> @AutoService(GcmJvmInitializer.class)
>>> public class GcmJvmInitializer implements JvmInitializer {
>>>
>>> @Override
>>> public void onStartup() {
>>> System.out.println("Starting Custom GcmJvmInitializer");
>>> ApplicationContext applicationContext = new
>>> AnnotationConfigApplicationContext(
>>> AppConfig.class);
>>> for (String beanName : applicationContext.getBeanDefinitionNames()) {
>>> System.out.println(beanName);
>>> }
>>> System.out.println("Stopping Custom GcmJvmInitializer");
>>> }
>>>
>>> @Override
>>> public void beforeProcessing(PipelineOptions options) {
>>> System.out.println("Starting Custom GcmJvmInitializer");
>>> ApplicationContext applicationContext = new
>>> AnnotationConfigApplicationContext(
>>> AppConfig.class);
>>> for (String beanName : applicationContext.getBeanDefinitionNames()) {
>>> System.out.println(beanName);
>>> }
>>> System.out.println("Stopping Custom GcmJvmInitializer");
>>> }
>>>
>>> }
>>>
>>>
>>> Thanks,
>>>
>>> Jitendra
>>>
>>>
>>> On Thu, Oct 10, 2019 at 6:35 PM Luke Cwik wrote:
>>>
You shouldn't need to call it before running the pipeline as you are
doing (you can if you want but its not necessary).
Have you created a service META-INF entry for the JvmInitializer you
have created or are using @AutoService?
This is the relevant bit of the documentation[1]. Here is some good
docs for how to use @AutoService[2].
1:
https://github.com/apache/beam/blob/f3ce8669b50837d48ab0d0ee9a1298ce3b5bc61c/sdks/java/core/src/main/java/org/apache/beam/sdk/harness/JvmInitializer.java#L30
2: https://github.com/google/auto/tree/master/service
On Thu, Oct 10, 2019 at 5:29 PM Jitendra kumavat <
jkumavat1...@gmail.com> wrote:
> Hi Luke,
>
> I tried the JvmIntializer.beforeProccssing method to initialize the
> spring application context. But it seems not working.
>
>
> * Below is my class definitions. *
>
> JvmInitializer class with context initialization.
>
> public class GcmJvmInitializer implements JvmInitializer {
> @Override
> public void beforeProcessing(PipelineOptions options){
> System.out.println("Starting Custom GcmJvmInitializer");
> ApplicationContext applicationContext = new
> AnnotationConfigApplicationContext(
> AppConfig.class);
> for (String beanName : applicationContext.getBeanDefinitionNames()) {
> System.out.println(beanName);
> }
> System.out.println("Stopping Custom GcmJvmInitializer");
> }
> }
>
>
> *Appconfig*
>
> @Configuration
> @PropertySource("classpath:gcm.properties")
> @ComponentScan(basePackages = {"com.liveramp.intl.gcm"})
> public class AppConfig {
>
> // B
