Re: PCollection#applyWindowingStrategyInternal

At a top level `setWindowingStrategyInternal` exists to set up the metadata without actually assigning windows. If we were more clever we might have found a way for it to not be public... it is something that can easily lead to an invalid pipeline. I think "compatible windows" today in Beam

Re: PCollection#applyWindowingStrategyInternal

On 4/6/24 21:23, Reuven Lax via dev wrote: So the problem here is that windowFn is a property of the PCollection, not the element, and the result of Flatten is a single PCollection. Yes. That is the cause of why Flatten.pCollections() needs the same windowFn. In various cases, there is a

Beam High Priority Issue Report (62)

This is your daily summary of Beam's current high priority issues that may need attention. See https://beam.apache.org/contribute/issue-priorities for the meaning and expectations around issue priorities. Unassigned P1 Issues: https://github.com/apache/beam/issues/30904 The PostCommit XVR

Re: [RESULT] [VOTE] Release 2.55.1, release candidate #2

Hi everyone, all steps have been completed, so the release is finalized! Please let me know if you encounter any issues Thanks, Danny On Mon, Apr 8, 2024 at 9:06 AM Danny McCormick wrote: > I'm happy to announce that we have unanimously approved this release. > > There are 5 approving votes, 3

Re: [PR] Publish docs for 2.55.1 release [beam-site]

damccorm merged PR #663: URL: https://github.com/apache/beam-site/pull/663 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

Re: [PR] Publish docs for 2.55.1 release [beam-site]

damccorm commented on PR #663: URL: https://github.com/apache/beam-site/pull/663#issuecomment-2042741609 R: @liferoad since I think Yi is out today -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

Re: [PR] Publish docs for 2.55.1 release [beam-site]

damccorm commented on PR #663: URL: https://github.com/apache/beam-site/pull/663#issuecomment-2042721098 R: @Abacn -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

[RESULT] [VOTE] Release 2.55.1, release candidate #2

I'm happy to announce that we have unanimously approved this release. There are 5 approving votes, 3 of which are binding: * Valentyn Tymofieiev (binding) * Kenneth Knowles (binding) * Robert Bradshaw (binding) * Jeff Kinard * XQ Hu There are no disapproving votes. I will now work on finalizing

Beam High Priority Issue Report (59)

This is your daily summary of Beam's current high priority issues that may need attention. See https://beam.apache.org/contribute/issue-priorities for the meaning and expectations around issue priorities. Unassigned P1 Issues: https://github.com/apache/beam/issues/30882 The PostCommit

Re: PCollection#applyWindowingStrategyInternal

So the problem here is that windowFn is a property of the PCollection, not the element, and the result of Flatten is a single PCollection. In various cases, there is a notion of "compatible" windows. Basically given window functions W1 and W2, provide a W3 that "works" with both. Note that Beam

Re: PCollection#applyWindowingStrategyInternal

Immediate self-correction, although setting the strategy directly via setWindowingStrategyInternal() *seemed* to be working during Pipeline construction time, during runtime it obviously does not work, because the PCollection was still windowed using the old windowFn. Make sense to me, but

PCollection#applyWindowingStrategyInternal

Hi, I came across a case where using PCollection#applyWindowingStrategyInternal seems legit in user core. The case is roughly as follows:  a) compute some streaming statistics  b) apply the same transform (say ComputeWindowedAggregation) with different parameters on these statistics

Re: tox issues in dev container

always pin the versions as well. On Fri, Apr 5, 2024 at 5:24 PM Valentyn Tymofieiev via dev < dev@beam.apache.org> wrote: > Could you please provide more info about how you create your environment? > Also what OS do you use? > > On Fri, Apr 5, 2024 at 2:08 PM Joey Tran > wrote: > >> Yeah that

Re: tox issues in dev container

Could you please provide more info about how you create your environment? Also what OS do you use? On Fri, Apr 5, 2024 at 2:08 PM Joey Tran wrote: > Yeah that was the tox command I was running > > On Fri, Apr 5, 2024, 4:37 PM XQ Hu via dev wrote: > >> >>

Re: tox issues in dev container

Yeah that was the tox command I was running On Fri, Apr 5, 2024, 4:37 PM XQ Hu via dev wrote: > > https://cwiki.apache.org/confluence/display/BEAM/Python+Tips#PythonTips-LintandFormattingChecks > > This generally works well. Have you checked this? > > On Fri, Apr 5, 2024 at 4:07 PM Joey Tran >

Re: tox issues in dev container

https://cwiki.apache.org/confluence/display/BEAM/Python+Tips#PythonTips-LintandFormattingChecks This generally works well. Have you checked this? On Fri, Apr 5, 2024 at 4:07 PM Joey Tran wrote: > I think I might be doing something silly with my environment. > > I'm trying to lint using tox in

tox issues in dev container

I think I might be doing something silly with my environment. I'm trying to lint using tox in a dev container, but running tox ends with this error: ``` (env) jtran@[Beam Build Env.]:~/beam {flatmapdefault} ] $ tox File "/usr/lib/python3/dist-packages/tox/reporter.py", line 32, in __init__

Re: 2.56.0 release

Hi, a nice round version number :) Cheers, Maciej On Thu, Apr 4, 2024 at 3:56 PM Danny McCormick via dev wrote: > Hey everyone, > > On top of running the current 2.55.1 patch release, I'd like to volunteer > as release manager for the 2.56.0 release, which will be cut on April 17. > Please

Beam High Priority Issue Report (60)

This is your daily summary of Beam's current high priority issues that may need attention. See https://beam.apache.org/contribute/issue-priorities for the meaning and expectations around issue priorities. Unassigned P1 Issues: https://github.com/apache/beam/issues/30852 [Failing Test]:

2.56.0 release

Hey everyone, On top of running the current 2.55.1 patch release, I'd like to volunteer as release manager for the 2.56.0 release, which will be cut on April 17. Please add any release blockers to the 2.56.0 milestone [1], I will be trying to push on those as the cut date approaches. Thanks,

Beam High Priority Issue Report (59)

This is your daily summary of Beam's current high priority issues that may need attention. See https://beam.apache.org/contribute/issue-priorities for the meaning and expectations around issue priorities. Unassigned P1 Issues: https://github.com/apache/beam/issues/30820 [Bug]: Install

Re: [VOTE] Patch Release 2.55.1, release candidate #2

+1 (binding) The artifacts all look good to me. On Wed, Apr 3, 2024 at 1:35 PM XQ Hu via dev wrote: > +1 (non-binding). Tested this using a simple Dataflow ML pipeline: > https://github.com/google/dataflow-ml-starter/actions/runs/8541848483. > > On Wed, Apr 3, 2024 at 2:35 PM Jeff Kinard

Re: [VOTE] Patch Release 2.55.1, release candidate #2

+1 (non-binding). Tested this using a simple Dataflow ML pipeline: https://github.com/google/dataflow-ml-starter/actions/runs/8541848483. On Wed, Apr 3, 2024 at 2:35 PM Jeff Kinard wrote: > +1. Validated running from local gradle JAR and staged maven JAR for > expansion-service. > > On Wed, Apr

Re: [VOTE] Patch Release 2.55.1, release candidate #2

+1. Validated running from local gradle JAR and staged maven JAR for expansion-service. On Wed, Apr 3, 2024 at 11:08 AM Danny McCormick via dev wrote: > Hi everyone, > > I put together a patch release per the conversation in > https://lists.apache.org/thread/kvq1wsj505pvopkq186dnvc0l6ryyfh0. >

Re: [VOTE] Patch Release 2.55.1, release candidate #2

+1 (binding) Kenn On Wed, Apr 3, 2024 at 12:58 PM Danny McCormick via dev wrote: > > Also noting that there is no PR postsubmit test suite running against > the release branch in the vote email. Given the diff, that's also fine > since previous tests runs didn't detect the breakage, but in

Re: Supporting Dynamic Destinations in a portable context

On Wed, Apr 3, 2024 at 4:15 AM Kenneth Knowles wrote: > Let me summarize the most recent proposal on-list to frame my question > about this last suggestion. It looks like this: > > 1. user has an element, call it `data` > 2. user maps `data` to an arbitrary metadata row, call it `dest` > 3. we

Re: [VOTE] Patch Release 2.55.1, release candidate #2

> Also noting that there is no PR postsubmit test suite running against the release branch in the vote email. Given the diff, that's also fine since previous tests runs didn't detect the breakage, but in general we should include that for patch releases as well. Yeah, it didn't seem useful to

[PR] Publish docs for 2.55.1 release [beam-site]

damccorm opened a new pull request, #663: URL: https://github.com/apache/beam-site/pull/663 Content generated from https://github.com/apache/beam/tree/v2.55.1-RC2. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the

Re: [VOTE] Patch Release 2.55.1, release candidate #2

Hi Danny, Thanks for volunteering to do this patch release. For review convenience, this is the diff: - Diff of release branches: https://github.com/apache/beam/compare/release-2.55.0...release-2.55.1 - The diff of tags v2.55.0-RC3 and v2.55.1-RC2:

Community over Code EU 2024: Start planning your trip!

[Note: You're receiving this email because you are subscribed to one or more project dev@ mailing lists at the Apache Software Foundation.] Dear community, We hope you are doing great, are you ready for Community Over Code EU? Check out the featured sessions, get your tickets with special

[VOTE] Patch Release 2.55.1, release candidate #2

Hi everyone, I put together a patch release per the conversation in https://lists.apache.org/thread/kvq1wsj505pvopkq186dnvc0l6ryyfh0. Please review and vote on the release candidate #2 (I messed up rc1) for the version 2.55.1, as follows: [ ] +1, Approve the release [ ] -1, Do not approve the

Participate in the ASF 25th Anniversary Campaign

Hi everyone, As part of The ASF’s 25th anniversary campaign[1], we will be celebrating projects and communities in multiple ways. We invite all projects and contributors to participate in the following ways: * Individuals - submit your first contribution:

Re: Supporting Dynamic Destinations in a portable context

Let me summarize the most recent proposal on-list to frame my question about this last suggestion. It looks like this: 1. user has an element, call it `data` 2. user maps `data` to an arbitrary metadata row, call it `dest` 3. we can do things like shuffle on `dest` because it isn't too big 4. we

Beam High Priority Issue Report (59)

This is your daily summary of Beam's current high priority issues that may need attention. See https://beam.apache.org/contribute/issue-priorities for the meaning and expectations around issue priorities. Unassigned P1 Issues: https://github.com/apache/beam/issues/30820 [Bug]: Install

Re: Design proposal for Beam YAML templates

It looks like we're converging on an optional jinja preprocessing phase to handle this. I'm in favor of this solution. On Wed, Mar 20, 2024 at 9:23 AM Robert Bradshaw wrote: > Thanks. I think this will be a very powerful feature. Left some comments > on the doc. > > On Tue, Mar 19, 2024 at

Re: Supporting Dynamic Destinations in a portable context

I do suspect that over time we'll find more and more cases we can't express, and will be asked to extend this little templating in more directions. To head that off - could we easily just reuse an existing language (SQL, LUA, something of the form?) instead of creating something new? On Tue, Apr

Re: Supporting Dynamic Destinations in a portable context

I really like this proposal. I think it has narrowed down and solved the essential problem of not shuffling excess redundant data, and also provides the vast majority of the functionality that a lambda would, with significantly better debugability and usability too, since the dynamic destination

Beam High Priority Issue Report (59)

This is your daily summary of Beam's current high priority issues that may need attention. See https://beam.apache.org/contribute/issue-priorities for the meaning and expectations around issue priorities. Unassigned P1 Issues: https://github.com/apache/beam/issues/30820 [Bug]: Install

[GSOC] Build out Beam Use Cases Project Proposal

Hi Everyone, I am Ayush Pandey, interested in the "Build out Beam Use Cases" (Project Link: [GSOC] Build out Beam Use Cases ) Project for GSoC 2024. I worked as a GSoC contributor for Apache Cloudstack in 2023 and was really interested in working

Beam High Priority Issue Report (62)

This is your daily summary of Beam's current high priority issues that may need attention. See https://beam.apache.org/contribute/issue-priorities for the meaning and expectations around issue priorities. Unassigned P1 Issues: https://github.com/apache/beam/issues/30820 [Bug]: Install

[Python SDK] Feedback for deferred side inputs + combiners

I posted a PoC PR [1] for fixing deferred side inputs with combiners in the python SDK. Would someone be willing to take a look at it? I have it working but could use some feedback on where to take it next. It looks like bundle processor combiner operations don't currently support side inputs [2]

Beam High Priority Issue Report (59)

This is your daily summary of Beam's current high priority issues that may need attention. See https://beam.apache.org/contribute/issue-priorities for the meaning and expectations around issue priorities. Unassigned P1 Issues: https://github.com/apache/beam/issues/30799 The PostCommit

Re: Patch release proposal

On Thu, Mar 28, 2024 at 8:57 AM Jan Lukavský wrote: > +1 to either doing full release or deferring to 2.56.0. > +1. Given that validation/testing required for unupdated SDKs should be minimum, I don't think a full release will be that much overhead compared to just releasing Python SDK. Also

Re: Patch release proposal

+1 to either doing full release or deferring to 2.56.0.  Jan On 3/28/24 16:52, Yi Hu via dev wrote: > Just releasing Python can break multi-lang by default (unless expansion service is overridden manually) since we match versions across languages when picking the default expansion service.

Re: Patch release proposal

> Just releasing Python can break multi-lang by default (unless expansion service is overridden manually) since we match versions across languages when picking the default expansion service. Yes, that's why I proposed "the source code of release candidate (e.g. apache_beam/version.py) still reads

Re: Patch release proposal

On Thu, Mar 28, 2024 at 8:36 AM Chamikara Jayalath wrote: > Just releasing Python can break multi-lang by default (unless expansion > service is overridden manually) since we match versions across languages > when picking the default expansion service. > > >

Re: Patch release proposal

Just releasing Python can break multi-lang by default (unless expansion service is overridden manually) since we match versions across languages when picking the default expansion service.

Re: Patch release proposal

> The patch itself [1] is trivial, however, the release process is not trivial. There is little documentation nor practice for a patch release process. I could imagine two options I think there's not a ton of documentation because we haven't done it, but all the release workflows were authored in

Re: Patch release proposal

Thanks Valentyn for raising this. In this case, Python containers will also be included. Different from PyPI wheels, docker tag can override so it can stay with 2.55.0 On Thu, Mar 28, 2024 at 11:15 AM Valentyn Tymofieiev wrote: > If we do a patch release for Python SDK, let's also patch another

Re: Patch release proposal

If we do a patch release for Python SDK, let's also patch another known issue for which fix is available: https://github.com/apache/beam/blob/master/CHANGES.md#known-issues-1 On Thu, Mar 28, 2024 at 8:01 AM Yi Hu via dev wrote: > 2.55.0 release manager here > > The patch itself [1] is trivial,

Re: Patch release proposal

2.55.0 release manager here The patch itself [1] is trivial, however, the release process is not trivial. There is little documentation nor practice for a patch release process. I could imagine two options 1. Do a full "2.55.1" release 2. Do a patch release only for Python SDK, that is a.

Re: Patch release proposal

+1 on a patch release - we've done a fair amount of work to make releasing easier, and one of my hopes is that it will enable quick patches like this. I'd vote we try to fix the underlying Java piece as well, though, doing a patch release for one language shouldn't be significantly cheaper than

Beam High Priority Issue Report (59)

This is your daily summary of Beam's current high priority issues that may need attention. See https://beam.apache.org/contribute/issue-priorities for the meaning and expectations around issue priorities. Unassigned P1 Issues: https://github.com/apache/beam/issues/30776 [Bug]: Beam YAML

Re: Patch release proposal

+1 to a targeted patch release. We did the same for the Go SDK a little while back. It would be good to see what's different for a different SDK. On Wed, Mar 27, 2024, 4:01 PM Robert Bradshaw via dev wrote: > Given the severity of the breakage, and the simplicity of the workaround, > I'm in

Re: Patch release proposal

Given the severity of the breakage, and the simplicity of the workaround, I'm in favor of a patch release. I think we could do Python-only, which would make the process even more lightweight. On Wed, Mar 27, 2024 at 3:48 PM Jeff Kinard wrote: > Hi all, > > Beam 2.55 was released with a bug that

Patch release proposal

Hi all, Beam 2.55 was released with a bug that causes WriteToJson on Beam YAML to fail when using the Java variant. This also affects any user attempting to use the Xlang JsonWriteTransformProvider -

Re: Supporting Dynamic Destinations in a portable context

On Wed, Mar 27, 2024 at 10:20 AM Reuven Lax wrote: > Can the prefix still be generated programmatically at graph creation time? > Yes. It's just a property of the transform passed by the user at configuration time. > On Wed, Mar 27, 2024 at 9:40 AM Robert Bradshaw > wrote: > >> On Wed, Mar

Re: Supporting Dynamic Destinations in a portable context

Can the prefix still be generated programmatically at graph creation time? On Wed, Mar 27, 2024 at 9:40 AM Robert Bradshaw wrote: > On Wed, Mar 27, 2024 at 9:12 AM Reuven Lax wrote: > >> This does seem like the best compromise, though I think there will still >> end up being performance

Re: [PR] Add yaml docs to beam reference documentation website. [beam-site]

robertwb merged PR #662: URL: https://github.com/apache/beam-site/pull/662 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

Re: [PR] Add yaml docs to beam reference documentation website. [beam-site]

robertwb commented on PR #662: URL: https://github.com/apache/beam-site/pull/662#issuecomment-2023256794 Thanks. Yeah, this is mostly a question of placement. (The html generation has been reviewed in previous PRs.) -- This is an automated message from the Apache Git Service. To

Re: Supporting Dynamic Destinations in a portable context

On Wed, Mar 27, 2024 at 9:12 AM Reuven Lax wrote: > This does seem like the best compromise, though I think there will still > end up being performance issues. A common pattern I've seen is that there > is a long common prefix to the dynamic destination followed the dynamic > component. e.g. the

Re: Supporting Dynamic Destinations in a portable context

> This does seem like the best compromise, though I think there will still end up being performance issues. A common pattern I've seen is that there is a long common prefix to the dynamic destination followed the dynamic component. e.g. the destination might be

Re: [PR] Add yaml docs to beam reference documentation website. [beam-site]

brucearctor commented on PR #662: URL: https://github.com/apache/beam-site/pull/662#issuecomment-2023233436 I am assuming since auto-generated, what is generated is what we want ... Didn't verify the `index.html` page. -- This is an automated message from the Apache Git Service. To

[PR] Add yaml docs to beam reference documentation website. [beam-site]

robertwb opened a new pull request, #662: URL: https://github.com/apache/beam-site/pull/662 This was generated with `python -m apache_beam.yaml.generate_yaml_docs`. (Adding this to the beam release process is at https://github.com/apache/beam/pull/30741.) -- This is an automated

Re: [PR] Add yaml docs to beam reference documentation website. [beam-site]

robertwb commented on PR #662: URL: https://github.com/apache/beam-site/pull/662#issuecomment-2023217884 R: @Abacn -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

Re: Supporting Dynamic Destinations in a portable context

This does seem like the best compromise, though I think there will still end up being performance issues. A common pattern I've seen is that there is a long common prefix to the dynamic destination followed the dynamic component. e.g. the destination might be

Re: Supporting Dynamic Destinations in a portable context

Thanks for putting this together, it will be a really useful feature to have. I am in favor of the string-pattern approaches. I think we need to support both the {record=..., dest_info=...} and the elide-fields approaches, as the former is nicer when one has a fixed representation for the output

Supporting Dynamic Destinations in a portable context

Hey all, There have been some conversations lately about how best to enable dynamic destinations in a portable context. Usually, this comes up for cross-language transforms and more recently for Beam YAML. I've started a short doc outlining some routes we could take. The purpose is to establish

Beam High Priority Issue Report (57)

This is your daily summary of Beam's current high priority issues that may need attention. See https://beam.apache.org/contribute/issue-priorities for the meaning and expectations around issue priorities. Unassigned P1 Issues: https://github.com/apache/beam/issues/30760 The PostCommit

Community Over Code NA 2024 Travel Assistance Applications now open!

Hello to all users, contributors and Committers! [ You are receiving this email as a subscriber to one or more ASF project dev or user mailing lists and is not being sent to you directly. It is important that we reach all of our users and contributors/committers so that they may get a chance

[Announce] Beam 2.55.0 Release

We are happy to present the new 2.55.0 release of Beam. This release includes both improvements and new functionality. See https://beam.apache.org/get-started/downloads/ for this release. For more information on changes in 2.55.0, check out the detailed release notes at

Beam High Priority Issue Report (55)

This is your daily summary of Beam's current high priority issues that may need attention. See https://beam.apache.org/contribute/issue-priorities for the meaning and expectations around issue priorities. Unassigned P1 Issues: https://github.com/apache/beam/issues/30737 [Failing Test]:

Re: [VOTE] Release 2.55.0, release candidate #3

Hi, Could a PMC member please help me finalize the source release? That is the steps documented here: https://github.com/apache/beam/blob/master/contributor-docs/release-guide.md#pmc-only-finalization Thanks! On Mon, Mar 25, 2024 at 9:37 AM Yi Hu wrote: > I'm happy to announce that we have

Re: [PR] Publish docs for 2.55.0 release [beam-site]

Abacn merged PR #661: URL: https://github.com/apache/beam-site/pull/661 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@beam.apache.org

Re: [VOTE] Release 2.55.0, release candidate #3

I'm happy to announce that we have unanimously approved this release. There are 7 approving votes, 3 of which are binding: * Ritesh Ghorse * XQ Hu * Chamikara Jayalath (binding) * Jan Lukavský (binding) * Danny McCormick * Yi Hu * Valentyn Tymofieiev There are no disapproving votes. Thanks

Beam High Priority Issue Report (58)

This is your daily summary of Beam's current high priority issues that may need attention. See https://beam.apache.org/contribute/issue-priorities for the meaning and expectations around issue priorities. Unassigned P1 Issues: https://github.com/apache/beam/issues/30730 The PostCommit

Joining the ASF Slack workspace

Hello, Please could someone already part of the workspace add me to the ASF slack workspace. I would like to participate in the Apache Beam channel of the ASF workspace as a guest. I am interested in the following topics related to Beam: I/O connectors and configurable I/O managers, building YAML

Re: [VOTE] Release 2.55.0, release candidate #3

+1 (binding). Checked some of the released artifacts, release blog, and ran a couple Python pipelines on Dataflow. > * GitHub Release notes [1] Is the link correct? It points to the milestone. On Fri, Mar 22, 2024 at 1:10 PM Yi Hu via dev wrote: > +1 (non-binding) > > 1. Checked published

Re: [VOTE] Release 2.55.0, release candidate #3

+1 (non-binding) 1. Checked published Java artifacts 2. Tested with GCP IO performance benchmark 3. Tested with Java PostRelease workflow (including QuickstartJavaDirect, QuickstartJavaDataflow, QuickstartJavaSpark, QuickstartJavaTwister2, QuickstartJavaFlinkLocal, MobileGamingJavaDirect,

Re: container dev environment: go get issue

Excellent! These days go has become much simpler to deal with (nearly any folder with a go.mod is a go project) but legacy GOPATH things remain to confuse matters. When I'm at a computer I'll see how necessary that line was for beam go development with that env script. I'd run one of the

Re: container dev environment: go get issue

Woohoo it works! How could I forget the oldest trick in the book "just delete the problematic line" Thanks for the quick response. I am unblocked now :) On Fri, Mar 22, 2024 at 8:47 AM Robert Burke wrote: > It's not clear to me why that's even requesting that package at all. I > would remove

Re: container dev environment: go get issue

It's not clear to me why that's even requesting that package at all. I would remove that 'go get' line. There's a different issue at play here too since it was written for pre-module Go in mind. I'm unfamiliar with that script though. I'll take a proper look in a few hours. On Fri, Mar 22,

container dev environment: go get issue

Hi, I've been banging my head trying to get a dev environment working. I gave up trying to get a local python environment working after I got some weird clang errors and proto generation issues so I've been trying to just use the docker container by running `bash start-build-env.sh` but I'm

Beam High Priority Issue Report (57)

This is your daily summary of Beam's current high priority issues that may need attention. See https://beam.apache.org/contribute/issue-priorities for the meaning and expectations around issue priorities. Unassigned P1 Issues: https://github.com/apache/beam/issues/30644 The Inference

Re: Python API: FlatMap default -> lambda x:x?

It's fair. if we change the default value, we can perhaps add an error handling logic so that (pcoll) | beam.Flatten() fails with an error that recommends (pcoll) | beam.FlatMap(), instead of saying that input is not an iterable. On Thu, Mar 21, 2024 at 3:41 PM Joey Tran wrote: > +1 > > On Thu,

Re: Python API: FlatMap default -> lambda x:x?

+1 On Thu, Mar 21, 2024 at 6:30 PM Robert Bradshaw via dev wrote: > I would be more comfortable with a default for FlatMap than overloading > Flatten in this way. Distinguishing between > > (pcoll,) | beam.Flatten() > > and > > (pcoll) | beam.Flatten() > > seems a bit error prone. > > >

Re: Python API: FlatMap default -> lambda x:x?

I would be more comfortable with a default for FlatMap than overloading Flatten in this way. Distinguishing between (pcoll,) | beam.Flatten() and (pcoll) | beam.Flatten() seems a bit error prone. On Thu, Mar 21, 2024 at 2:23 PM Joey Tran wrote: > Ah, I misunderstood your original

Re: Python API: FlatMap default -> lambda x:x?

Ah, I misunderstood your original suggestion then. That makes sense then. I have already seen someone get a little confused about the names and surprised that Flatten doesn't do what FlatMap does. On Thu, Mar 21, 2024 at 5:20 PM Valentyn Tymofieiev wrote: > Beam throws an error at submission

Re: Python API: FlatMap default -> lambda x:x?

Beam throws an error at submission time in Python if you pass a single PCollection to Flatten. The scenario you describe concerns a one-element list. On Thu, Mar 21, 2024, 13:43 Joey Tran wrote: > I think it'd be quite surprising if beam.Flatten would become equivalent > to FlatMap if passed

Re: Python API: FlatMap default -> lambda x:x?

I think it'd be quite surprising if beam.Flatten would become equivalent to FlatMap if passed only a single pcollection. One use case that would be broken from that is cases where someone might be flattening a variable number of pcollections, including possibly only one pcollection. In that case,

Re: Python API: FlatMap default -> lambda x:x?

One possible alternative is to define beam.Flatten for a single collection to be functionally equivalent to beam.FlatMap(lambda x: x), but that would be a larger change and such behavior might need to be consistent across SDKs and documented. Adding a default value is a simpler change. I can also

Re: Python API: FlatMap default -> lambda x:x?

IIRC, Java has Flatten.iterables() and Flatten.collections(), the first of which does what you want. Giving FlatMap a default arg of lambda x: x is an interesting idea. The only downside I see is a less clear error if one forgets to provide this (now mandatory) parameter, but maybe that's low

Re: Python API: FlatMap default -> lambda x:x?

That's not really the same thing, is it? `beam.Flatten` combines two or more pcollections into a single pcollection while beam.FlatMap unpacks iterables of elements (i.e. PCollection> -> PCollection) On Thu, Mar 21, 2024 at 2:57 PM Valentyn Tymofieiev via dev < dev@beam.apache.org> wrote: > Hi,

Re: Python API: FlatMap default -> lambda x:x?

Actually, disregard that, Flatten is used in a different context to flatten multiple collections. On Thu, Mar 21, 2024 at 11:55 AM Valentyn Tymofieiev wrote: > Hi, you can use beam.Flatten() instead. > > On Thu, Mar 21, 2024 at 10:55 AM Joey Tran > wrote: > >> Hey all, >> >> Using an identity

Re: Python API: FlatMap default -> lambda x:x?

Hi, you can use beam.Flatten() instead. On Thu, Mar 21, 2024 at 10:55 AM Joey Tran wrote: > Hey all, > > Using an identity function for FlatMap comes up more often than using > FlatMap without an identity function. Would it make sense to use the > identity function as a default? > > > >

Python API: FlatMap default -> lambda x:x?

Hey all, Using an identity function for FlatMap comes up more often than using FlatMap without an identity function. Would it make sense to use the identity function as a default?

Re: [VOTE] Release 2.55.0, release candidate #3

+1 - validated some ML examples with the interactive runner Thanks, Danny On Thu, Mar 21, 2024 at 9:21 AM Jan Lukavský wrote: > +1 (binding) > > Tested Java SDK with FlinkRunner. > > Jan > On 3/20/24 22:40, Chamikara Jayalath via dev wrote: > > +1 (binding) > > Tested multi-lang Java/Python

Re: [VOTE] Release 2.55.0, release candidate #3

+1 (binding) Tested Java SDK with FlinkRunner.  Jan On 3/20/24 22:40, Chamikara Jayalath via dev wrote: +1 (binding) Tested multi-lang Java/Python pipelines and upgrading BQ/Kafka transforms from 2.53.0 to 2.55.0 using the Transform Service. Thanks, Cham On Tue, Mar 19, 2024 at 2:10 PM

Beam High Priority Issue Report (57)

This is your daily summary of Beam's current high priority issues that may need attention. See https://beam.apache.org/contribute/issue-priorities for the meaning and expectations around issue priorities. Unassigned P1 Issues: https://github.com/apache/beam/issues/30683 The PreCommit Java

Re: [VOTE] Release 2.55.0, release candidate #3

+1 (binding) Tested multi-lang Java/Python pipelines and upgrading BQ/Kafka transforms from 2.53.0 to 2.55.0 using the Transform Service. Thanks, Cham On Tue, Mar 19, 2024 at 2:10 PM XQ Hu via dev wrote: > +1 (non-binding). Ran the simple ML pipeline without any issue: >

<    1   2   3   4   5   6   7   8   9   10   >