Re: [PROPOSAL] Preparing for Beam release 2.40.0

I did a pass this morning. I believe there is only one release blocker that doesn't already have a fix. If I closed your issue or moved it off the milestone, feel free to have a different opinion and revert my action. Kenn On Mon, Jun 13, 2022 at 5:04 PM Ahmet Altay wrote: > > > On Tue, Jun 7,

Re: Jenkins CI currently unavailable

The UI is https://ci-beam.apache.org/ and it is integrated with ASF's LDAP. I don't know if this URL is documented anywhere. Usage of the UI is standard Jenkins. You can select any job and click "build with parameters" and put in a git ref to build from. Kenn On Mon, Jun 13, 2022 at 5:54 PM Reuv

Re: Chained Job Graph Apache Beam | Dataflow

Team, Any update on this? On Mon, Jun 13, 2022 at 8:39 PM Ravi Kapoor wrote: > Hi Team, > > I am currently using Beam in my project with Dataflow Runner. > I am trying to create a pipeline where the data flows from the source to > staging then to target such as: > > A (Source) -> B(Staging) -> C

Re: Jenkins CI currently unavailable

Additionally to what Kenn said, we have some documentation here: https://cwiki.apache.org/confluence/display/BEAM/Jenkins+Tips Though, not sure how up-to-date it is. — Alexey > On 14 Jun 2022, at 16:42, Kenneth Knowles wrote: > >

Re: Chained Job Graph Apache Beam | Dataflow

Hello Ravi, I am not sure I follow what you are trying to do, but BigQueryIO.writeTableRows is a sink and will return only insertion errors. If you already have table_A_records, why bother reading it again from BigQuery? You could use table_A_records to run any intermediary transforms and write t

Dataflow java job with java transforms in expansion service

Hello, I was wondering if anyone has run a java job with java external transforms in dataflow? We have had python beam jobs run great with java external transforms. However, we tried to run a java job with java external transforms but this seemed to stall on dataflow (this was done a while ago, ha

Re: Not Able to Get Code to Work for BigQuery using DataFlow

Juan FYI i am using this, hth cutoff_date_str = (date.today() - BDay(60)).date().strftime('%Y-%m-%d') logging.info('Cutoff is:{}'.format(cutoff_date_str)) bq_sql = """SELECT TICKER, LABEL, COUNT(*) as COUNTER FROM `datascience-projects.gcp_shareloader.stock_selection` WHERE AS_OF_DATE

Re: Null PCollection errors in v2.40 unit tests

I had this happen to me recently as well. After `git bisecting` led to confusing results, I ran my tests again via gradlew adding `--rerun-tasks` to the command. This is an expensive operation, but after I ran that I was able to test again with expected results. YMMV Thanks, Evan On Tue, Jun 1

Re: Dataflow java job with java transforms in expansion service

Hello, I will run another one on the latest beam today and let you know what happens. The last version I tried this on was I think 2.35. I believe there were no errors on the dataflow page, but some issues with getting the workers started. I will try on the latest beam and update to see what happe

Re: Clean Up GitHub Labels

+1 sounds good to me One thing I did a lot of when triaging Jiras was moving them from one component to another, after which people who cared about those components would go through them. Making the labels more straightforward for users would streamline that. Kenn On Sun, Jun 12, 2022 at 9:04 PM

Re: Null PCollection errors in v2.40 unit tests

I had brought up a weird issues I was having with AutoValue awhile ago that looks actually very similar to this: https://lists.apache.org/thread/0sbkykop2gsw71jpf3ln6forbnwq3j4o I never got to the bottom of it, but `--rerun-tasks` always fixes it for me. On Tue, Jun 14, 2022 at 5:11 PM Danny McC

Re: Changing the interface in CassandraIO Mapper

On Mon, May 16, 2022 at 11:29 PM Chamikara Jayalath wrote: > > > On Mon, May 16, 2022 at 12:35 PM Ahmet Altay wrote: > >> Adding folks who might have an opinion : @Alexey Romanenko >> @Chamikara Jayalath >> >> On Wed, May 11, 2022 at 5:47 PM Vincent Marquez < >> vincent.marq...@gmail.com> wrot