Re: [Question] AWS Credentials Serializability
It seems there is problem with the fat/Uber jar . It might have multiple versions of the classs being overwritten with duplicates . Check this link : https://stackoverflow.com/questions/57039951/beam-failed-to-serialize-and-deserialize-property-awscredentialsprovider On Sat, 23 Sep 2023 at 9:55 PM, Ramya Prasad via user wrote: > Hello, > > I am a developer trying to use Apache Beam, and I'm running into an issue > with trying to provide multiple AWS credentials to the AwsOptions class > within the PipelineOptions class. My use case requires multi-account reads, > and I've tried using the AwsCredentialsProviderChain class within the > options.as(AwsOptions.class).setAwsCredentialsProvider() method, but I > keep getting the error > > Caused by: com.fasterxml.jackson.databind.JsonMappingException: Unexpected > IOException (of type java.io.IOException): Failed to serialize and > deserialize property 'awsCredentialsProvider' with value > 'AwsCredentialsProviderChain. > > Is there anything I'm doing wrong? Any help would be appreciated! > > Thanks and sincerely, > Ramya > -- > > The information contained in this e-mail is confidential and/or > proprietary to Capital One and/or its affiliates and may only be used > solely in performance of work or services for Capital One. The information > transmitted herewith is intended only for use by the individual or entity > to which it is addressed. If the reader of this message is not the intended > recipient, you are hereby notified that any review, retransmission, > dissemination, distribution, copying or other use of, or taking of any > action in reliance upon this information is strictly prohibited. If you > have received this communication in error, please contact the sender and > delete the material from your computer. > > > > >
Re: Firing a dataflow job using REST API
Hi Rajnil, Did you look at the gcloud documentation? https://cloud.google.com/dataflow/docs/reference/rest Thanks Deepak On Mon, Feb 14, 2022 at 3:53 PM Rajnil Guha wrote: > Hi Beam Community, > > I am trying to run a Dataflow pipeline using the REST API. Currently I see > an example for doing the same for Dataflow templates[1]. > My query is can we run a dataflow job using REST API but not create > templates. As per the docs the projects.jobs.create method allows to do so, > but if anyone have tried this before would be very helpful if you could > provide a snippet of your code. > > [1] > https://cloud.google.com/dataflow/docs/guides/templates/running-templates > > Thanks & Regards > Rajnil > > >
Re: Allyship workshops for open source contributors
+1 On Tue, Jun 15, 2021 at 12:31 AM Aizhamal Nurmamat kyzy wrote: > Thank you all! Based on the feedback, I will set up a session for a couple > open source groups. Will share more details soon. Stay tuned. > > On Mon, Jun 7, 2021 at 4:42 PM Kenneth Knowles wrote: > >> Yes please! >> >> On Thu, Jun 3, 2021, 18:32 Ratnakar Malla wrote: >> >>> +1 >>> >>> >>> -- >>> *From:* Austin Bennett >>> *Sent:* Thursday, June 3, 2021 6:20:25 PM >>> *To:* user@beam.apache.org >>> *Cc:* dev >>> *Subject:* Re: Allyship workshops for open source contributors >>> >>> +1, assuming timing can work. >>> >>> On Wed, Jun 2, 2021 at 2:07 PM Aizhamal Nurmamat kyzy < >>> aizha...@apache.org> wrote: >>> >>> If we have a good number of people who express interest in this thread, >>> I will set up training for the Airflow community. >>> >>> >>> I meant Beam ^^' I am organizing it for the Airflow community as well. >>> >>>
Freelancing work
Hi All, Sorry for spamming. If anyone is looking or knows someone who might be looking for freelancers in Apache Beam,please let me know. Thanks.
Ways to learn Beam in detail
Hi There, Can anyone suggest the best way to get started on Beam APIs , so i can start contributing to the codebase? Thanks Deepak
Re: Scheduling dataflow pipelines
We have used composer (airlfow) successfully to schedule Dataflow jobs. Please let me know if you would need details around it. Thanks Deepak On Sun, Apr 5, 2020 at 7:56 PM Joshua B. Harrison wrote: > Hi Marco, > > I've ended using a VM running Luigi to schedule jobs. I use the data flow > Python API to execute stored templates. > > I can give you more details if you’re interested. > > Best, > Joshua > > On Sun, Apr 5, 2020 at 5:02 AM Marco Mistroni wrote: > >> HI all >> sorry for this partially OT but has anyone been successful in scheduling >> dataflow job on GCP? >> I have tried the CloudFunction approach (following few eamples on the >> web) but it didnt work out for me - the cloud function keep on giving me an >> INVALID ARGUMENT - which i could not debug >> >> So i was wondering if anyone has been successful and can provide me an >> example >> >> kind regards >> Marco >> >> -- > Joshua Harrison | Software Engineer | joshharri...@gmail.com > | 404-433-0242 >
Design patterns while using Beam
Hi All I guess we need to put some examples in the documentation around best coding practises , concurrency , non blocking IO and design patterns while writing Apache Beam pipelines. Is there any such guide available ? E.g. when there are lot of options to be used in the pipeline , BuilderPattern should be used. Another use case can be when anyone wants to run complex transformation on incoming objects , visitor pattern should be used. This guide can come from people already running beam in production and written it with all best practices in mind. It will help in greater and wider adaoption. Just a thought. Please let me know if anyone wants to contribute and i can lead this initiative. Thanks Deepak
How to debug dataflow locally
Hi All, I am trying to come up with a framework that would help debugging dataflow job end to end locally. Dataflow can work with 100s of source and sinks. Is there a framework already to setup these sources and sinks locally e.g. if my dataflow reads from BQ and inserts to Bigtable. Please let me know if someone is already doing this. If ues then how. Thanks Deepak
Advanced Java 1.8+ and Beam mentors
Hi I am looking for some advanced level of mentoring in Beam using Java 1.8+. Are there any resources available or if someone does it on professional development level? Please let me know. Thanks Deepak
Is there any job board for Beam with Java?
Hi All, Sorry to spam. I am writing to check if there is any job board where we can post or look for open positions/ jobs in Apache Beam? Thanks Deepak
