Re: Testing a pipeline with external transform

Sounds good, let me try that thank you again! Thanks, Sahith On Mon, Apr 24, 2023 at 2:11 PM Chamikara Jayalath wrote: > > > On Mon, Apr 24, 2023 at 11:06 AM Sahith Nallapareddy > wrote: > >> Hello, >> >> Ah I missed that thank you! I am assuming for java job

Re: Testing a pipeline with external transform

ns for running using DirectRunner ? > > Thanks, > Cham > > [1] > https://beam.apache.org/documentation/sdks/java-multi-language-pipelines/ > [2] > https://beam.apache.org/documentation/sdks/python-multi-language-pipelines/ > > On Mon, Apr 24, 2023 at 10:57 AM

Testing a pipeline with external transform

Hello, I was wondering the best way to test a pipeline with external transform. We were trying to use the DirectRunner but it seemed to hang on the external transform step. I was wondering if someone could point me to somewhere in the beam code where this is done or give some pointers on how to te

Re: Custom Windowing

f we > can find any more evidence of what might be going wrong. > > Thanks, > > > Svetak Sundhar > > Technical Solutions Engineer, Data > s vetaksund...@google.com > > > > On Fri, Mar 17, 2023 at 1:02 PM Sahith Nallapareddy via dev < > dev@beam.apache.org>

Custom Windowing

Hello, We are working on writing a custom windowing function. The functionality is similar to the one described in this book https://www.oreilly.com/library/view/streaming-systems/9781491983867/ch04.html (the bounded session and per key fixed window is what we are trying). However, we are not sure

Re: Adding configurable gRPC channels to External transform

t; On Tue, Jan 24, 2023 at 1:44 PM Sahith Nallapareddy via dev < > dev@beam.apache.org> wrote: > >> Hello, >> >> I made a PR, https://github.com/apache/beam/pull/25151, to add >> configurable gRPC channel to the ArtifactRetrievalService stub. We are >> hostin

Adding configurable gRPC channels to External transform

Hello, I made a PR, https://github.com/apache/beam/pull/25151, to add configurable gRPC channel to the ArtifactRetrievalService stub. We are hosting our external transforms in different environments and we were trying to host them in Google Cloud Run. This requires that the gRPC calls have to use

Re: NullPointerException when getting timestamp from ProcessContext

signature, and will be > available in the upcoming 0.12.0 release :) > > Best, > Claire > > On Wed, Sep 28, 2022 at 12:26 AM Sahith Nallapareddy via dev < > dev@beam.apache.org> wrote: > >> Hello, >> >> We have seen this error popup when running pipelin

NullPointerException when getting timestamp from ProcessContext

Hello, We have seen this error popup when running pipelines on the dataflow runner and not sure exactly why. It seems like we get a null pointer exception when accessing the timestamp via the ProcessContext. This pipeline is using runner_v2 and is a cross transform pipeline so that may be why. Not

Re: Beam BigtableIO versus Google CloudBigtableIO

> more write requests doesn't necessarily mean more cost/nodes. What version > of CloudBigtableIO are you using and are you using an autoscaling CBT > cluster? > > -Diego > > On Tue, Aug 16, 2022 at 11:55 AM Sahith Nallapareddy via dev < > dev@beam.apache.org> wrot

Beam BigtableIO versus Google CloudBigtableIO

Hello, I see that there are two implementations of reading and writing from Bigtable, one in beam and one that is references in Google cloud documentation. Is one preferred over the other? We often use the Beam BigtableIO to write to bigtable but I have found that sometimes the default configurati

ClassNotFoundException when using Java external transforms in a Java job

Hello, I sent an email sometime ago about java external transforms in a java job. Seems like things are for the most part working, but we have run into this error that I have attached. At the base cause it says [debug] Caused by: java.lang.ClassNotFoundException: com.spotify.ingestion.recentactivi

Re: Dataflow java job with java transforms in expansion service

14 PM Sahith Nallapareddy wrote: > Hello, > > I will run another one on the latest beam today and let you know what > happens. The last version I tried this on was I think 2.35. I believe there > were no errors on the dataflow page, but some issues with getting the > workers sta

Re: Dataflow java job with java transforms in expansion service

happens! Thanks, Sahith On Tue, Jun 14, 2022 at 12:26 PM Chamikara Jayalath wrote: > > > On Tue, Jun 14, 2022 at 8:47 AM Sahith Nallapareddy > wrote: > >> Hello, >> >> I was wondering if anyone has run a java job with java external >> transforms in dataflow?

Dataflow java job with java transforms in expansion service

Hello, I was wondering if anyone has run a java job with java external transforms in dataflow? We have had python beam jobs run great with java external transforms. However, we tried to run a java job with java external transforms but this seemed to stall on dataflow (this was done a while ago, ha