Re: [Feature proposal] Java Record Schema inference

Thank you Reuven for the response! I'll update the design doc to include adding support for the generic classes. As for the new reflection API that's what I propose in the doc as well, it seems like the right choice. By codegen path you mean using bytebuddy to generate the constructor and the

Beam High Priority Issue Report (62)

This is your daily summary of Beam's current high priority issues that may need attention. See https://beam.apache.org/contribute/issue-priorities for the meaning and expectations around issue priorities. Unassigned P1 Issues: https://github.com/apache/beam/issues/30966 The PreCommit

Re: PCollection#applyWindowingStrategyInternal

Yes, but that's inevitable as stateful ParDo in a sense live outside of most of the window/trigger semantics. Basically a stateful ParDo is the user executing low-level control over these semantics, and controlling output frequency themselves with timers. One could however still propagate the

Re: PCollection#applyWindowingStrategyInternal

On 4/11/24 18:20, Reuven Lax via dev wrote: I'm not sure it would require all that. A "basic" implementation could be done on top of our existing model. Essentially the user would specify triggers at the sink ParDos, then the runner would walk backwards up the graph, reverse-propagating these

Re: Structured Logging in python

Thanks! I opened https://github.com/apache/beam/issues/30978 . Feel free to self-assign when you have time to work on it. On Mon, Apr 15, 2024 at 1:43 PM Geddy Schellevis wrote: > The “custom_data” field didn’t work. > I am Happy with helping with the implementation of this. > > Op ma 15 apr

Re: Structured Logging in python

The “custom_data” field didn’t work. I am Happy with helping with the implementation of this. Op ma 15 apr 2024 om 22:10 schreef Ahmet Altay via dev > Is there an open github issue for this? Perhaps it would be a good project > to implement in Python, using the java version as a > reference

Re: Structured Logging in python

Is there an open github issue for this? Perhaps it would be a good project to implement in Python, using the java version as a reference implementation? On Thu, Apr 11, 2024 at 12:04 PM Udi Meiri wrote: > Hi, > > I believe this wasn't implemented for Python (only Java). You can try > adding

Re: 2.56.0 release

Thank you Danny! On Fri, Apr 5, 2024 at 3:20 AM Maciej Szwaja via dev wrote: > Hi, > > a nice round version number :) > > Cheers, > Maciej > > On Thu, Apr 4, 2024 at 3:56 PM Danny McCormick via dev < > dev@beam.apache.org> wrote: > >> Hey everyone, >> >> On top of running the current 2.55.1

Re: [Feature proposal] Java Record Schema inference

Some initial thoughts: Making schema inference handle generic classes would be a nice improvement - users occasionally bump into this restriction, and there's no reason not to improve it. I would recommend using the new Java reflection APIs (i.e. getRecordComponents) to directly infer the

[Feature proposal] Java Record Schema inference

Hi team, I'd like to propose a new java sdk extension feature, which is adding support for java record schema inference - see the design doc here: https://docs.google.com/document/d/1zSQ9cnqtVM8ttJEuHBDE6hw4qjUuJy1dpZWB6IBTuOs/edit?usp=sharing In short - adding this extension's jar to the

Beam High Priority Issue Report (61)

This is your daily summary of Beam's current high priority issues that may need attention. See https://beam.apache.org/contribute/issue-priorities for the meaning and expectations around issue priorities. Unassigned P1 Issues: https://github.com/apache/beam/issues/30955 [Failing Test]:

Beam High Priority Issue Report (61)

This is your daily summary of Beam's current high priority issues that may need attention. See https://beam.apache.org/contribute/issue-priorities for the meaning and expectations around issue priorities. Unassigned P1 Issues: https://github.com/apache/beam/issues/30927 [Failing Test]:

Re: [Python SDK] Feedback for deferred side inputs + combiners

On Thu, Apr 11, 2024 at 1:00 PM Joey Tran wrote: > Thanks for the feedback! > > Sounds like it'll be a while before this is supported. Would it make sense > to formally _not_ support this functionality by raising an early and clear > exception if someone does try to use a side input with a

Re: [Python SDK] Feedback for deferred side inputs + combiners

Thanks for the feedback! Sounds like it'll be a while before this is supported. Would it make sense to formally _not_ support this functionality by raising an early and clear exception if someone does try to use a side input with a combiner? As it stands now, the exception a user gets is very

Re: [Python SDK] Feedback for deferred side inputs + combiners

I took a look and mentioned the PR to a few folks. Couple of thoughts: - We should avoid Beam adding a high-level functionality only for Batch. - Supporting Windowing/Triggers would likely be non-trivial and worth considering early in the design. - If you'd like to continue working on this, I

Re: Structured Logging in python

Hi, I believe this wasn't implemented for Python (only Java). You can try adding structured data (extra keyword) under the key "custom_data" and that might work. On 2024/04/11 17:49:43 Valentyn Tymofieiev wrote: > Thanks for reaching out. There was a proposal a while back: >

Re: Structured Logging in python

Thanks for reaching out. There was a proposal a while back: https://s.apache.org/beam-structured-logging /cc: @u...@apache.org - do you know the current status? Thanks a lot! On Thu, Apr 11, 2024 at 8:29 AM Geddy Schellevis wrote: > Hi all, > > I would like to know if it is possible to have

Re: PCollection#applyWindowingStrategyInternal

I'm not sure it would require all that. A "basic" implementation could be done on top of our existing model. Essentially the user would specify triggers at the sink ParDos, then the runner would walk backwards up the graph, reverse-propagating these triggers (with some resolution rules aimed at

Structured Logging in python

Hi all, I would like to know if it is possible to have structured logging in Dataflow. In the attached file, you can find the code that I am trying to do. I see the logs are appearing in gcp log explorer, but I cannot see the extra fields. Best regards, from __future__ import annotations import

Re: PCollection#applyWindowingStrategyInternal

I've probably heard about it, but I never read the proposal. Sounds great, but that would require to change our ParDos from the 'directive' style to something more functional, so that processing of elements, state updates and outputting results can be decoupled and managed by the runner

Beam High Priority Issue Report (62)

This is your daily summary of Beam's current high priority issues that may need attention. See https://beam.apache.org/contribute/issue-priorities for the meaning and expectations around issue priorities. Unassigned P1 Issues: https://github.com/apache/beam/issues/30927 [Failing Test]:

Re: tox issues in dev container

based on 'devcontainer', I assume running in/via: https://containers.dev/ ? On Fri, Apr 5, 2024 at 3:46 PM XQ Hu via dev wrote: > always pin the versions as well. > > On Fri, Apr 5, 2024 at 5:24 PM Valentyn Tymofieiev via dev < > dev@beam.apache.org> wrote: > >> Could you please provide more

Re: PCollection#applyWindowingStrategyInternal

Are you familiar with the "sink triggers" proposal? Essentially while windowing is usually a property of the data, and therefore flows downwards through the graph, triggering is usually a property of output (i.e. sink) latency - how much are you willing to wait to see data, and what semantics do

Beam High Priority Issue Report (62)

This is your daily summary of Beam's current high priority issues that may need attention. See https://beam.apache.org/contribute/issue-priorities for the meaning and expectations around issue priorities. Unassigned P1 Issues: https://github.com/apache/beam/issues/30904 The PostCommit XVR

Re: PCollection#applyWindowingStrategyInternal

On 4/9/24 18:33, Kenneth Knowles wrote: At a top level `setWindowingStrategyInternal` exists to set up the metadata without actually assigning windows. If we were more clever we might have found a way for it to not be public... it is something that can easily lead to an invalid pipeline. Yes,

Re: PCollection#applyWindowingStrategyInternal

At a top level `setWindowingStrategyInternal` exists to set up the metadata without actually assigning windows. If we were more clever we might have found a way for it to not be public... it is something that can easily lead to an invalid pipeline. I think "compatible windows" today in Beam

Re: PCollection#applyWindowingStrategyInternal

On 4/6/24 21:23, Reuven Lax via dev wrote: So the problem here is that windowFn is a property of the PCollection, not the element, and the result of Flatten is a single PCollection. Yes. That is the cause of why Flatten.pCollections() needs the same windowFn. In various cases, there is a

Beam High Priority Issue Report (62)

This is your daily summary of Beam's current high priority issues that may need attention. See https://beam.apache.org/contribute/issue-priorities for the meaning and expectations around issue priorities. Unassigned P1 Issues: https://github.com/apache/beam/issues/30904 The PostCommit XVR

Re: [RESULT] [VOTE] Release 2.55.1, release candidate #2

Hi everyone, all steps have been completed, so the release is finalized! Please let me know if you encounter any issues Thanks, Danny On Mon, Apr 8, 2024 at 9:06 AM Danny McCormick wrote: > I'm happy to announce that we have unanimously approved this release. > > There are 5 approving votes, 3

Re: [PR] Publish docs for 2.55.1 release [beam-site]

damccorm merged PR #663: URL: https://github.com/apache/beam-site/pull/663 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

Re: [PR] Publish docs for 2.55.1 release [beam-site]

damccorm commented on PR #663: URL: https://github.com/apache/beam-site/pull/663#issuecomment-2042741609 R: @liferoad since I think Yi is out today -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

Re: [PR] Publish docs for 2.55.1 release [beam-site]

damccorm commented on PR #663: URL: https://github.com/apache/beam-site/pull/663#issuecomment-2042721098 R: @Abacn -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

[RESULT] [VOTE] Release 2.55.1, release candidate #2

I'm happy to announce that we have unanimously approved this release. There are 5 approving votes, 3 of which are binding: * Valentyn Tymofieiev (binding) * Kenneth Knowles (binding) * Robert Bradshaw (binding) * Jeff Kinard * XQ Hu There are no disapproving votes. I will now work on finalizing

Beam High Priority Issue Report (59)

This is your daily summary of Beam's current high priority issues that may need attention. See https://beam.apache.org/contribute/issue-priorities for the meaning and expectations around issue priorities. Unassigned P1 Issues: https://github.com/apache/beam/issues/30882 The PostCommit

Re: PCollection#applyWindowingStrategyInternal

So the problem here is that windowFn is a property of the PCollection, not the element, and the result of Flatten is a single PCollection. In various cases, there is a notion of "compatible" windows. Basically given window functions W1 and W2, provide a W3 that "works" with both. Note that Beam

Re: PCollection#applyWindowingStrategyInternal

Immediate self-correction, although setting the strategy directly via setWindowingStrategyInternal() *seemed* to be working during Pipeline construction time, during runtime it obviously does not work, because the PCollection was still windowed using the old windowFn. Make sense to me, but

PCollection#applyWindowingStrategyInternal

Hi, I came across a case where using PCollection#applyWindowingStrategyInternal seems legit in user core. The case is roughly as follows:  a) compute some streaming statistics  b) apply the same transform (say ComputeWindowedAggregation) with different parameters on these statistics

Re: tox issues in dev container

always pin the versions as well. On Fri, Apr 5, 2024 at 5:24 PM Valentyn Tymofieiev via dev < dev@beam.apache.org> wrote: > Could you please provide more info about how you create your environment? > Also what OS do you use? > > On Fri, Apr 5, 2024 at 2:08 PM Joey Tran > wrote: > >> Yeah that

Re: tox issues in dev container

Could you please provide more info about how you create your environment? Also what OS do you use? On Fri, Apr 5, 2024 at 2:08 PM Joey Tran wrote: > Yeah that was the tox command I was running > > On Fri, Apr 5, 2024, 4:37 PM XQ Hu via dev wrote: > >> >>

Re: tox issues in dev container

Yeah that was the tox command I was running On Fri, Apr 5, 2024, 4:37 PM XQ Hu via dev wrote: > > https://cwiki.apache.org/confluence/display/BEAM/Python+Tips#PythonTips-LintandFormattingChecks > > This generally works well. Have you checked this? > > On Fri, Apr 5, 2024 at 4:07 PM Joey Tran >

Re: tox issues in dev container

https://cwiki.apache.org/confluence/display/BEAM/Python+Tips#PythonTips-LintandFormattingChecks This generally works well. Have you checked this? On Fri, Apr 5, 2024 at 4:07 PM Joey Tran wrote: > I think I might be doing something silly with my environment. > > I'm trying to lint using tox in

tox issues in dev container

I think I might be doing something silly with my environment. I'm trying to lint using tox in a dev container, but running tox ends with this error: ``` (env) jtran@[Beam Build Env.]:~/beam {flatmapdefault} ] $ tox File "/usr/lib/python3/dist-packages/tox/reporter.py", line 32, in __init__

Re: 2.56.0 release

Hi, a nice round version number :) Cheers, Maciej On Thu, Apr 4, 2024 at 3:56 PM Danny McCormick via dev wrote: > Hey everyone, > > On top of running the current 2.55.1 patch release, I'd like to volunteer > as release manager for the 2.56.0 release, which will be cut on April 17. > Please

Beam High Priority Issue Report (60)

This is your daily summary of Beam's current high priority issues that may need attention. See https://beam.apache.org/contribute/issue-priorities for the meaning and expectations around issue priorities. Unassigned P1 Issues: https://github.com/apache/beam/issues/30852 [Failing Test]:

2.56.0 release

Hey everyone, On top of running the current 2.55.1 patch release, I'd like to volunteer as release manager for the 2.56.0 release, which will be cut on April 17. Please add any release blockers to the 2.56.0 milestone [1], I will be trying to push on those as the cut date approaches. Thanks,

Beam High Priority Issue Report (59)

This is your daily summary of Beam's current high priority issues that may need attention. See https://beam.apache.org/contribute/issue-priorities for the meaning and expectations around issue priorities. Unassigned P1 Issues: https://github.com/apache/beam/issues/30820 [Bug]: Install

Re: [VOTE] Patch Release 2.55.1, release candidate #2

+1 (binding) The artifacts all look good to me. On Wed, Apr 3, 2024 at 1:35 PM XQ Hu via dev wrote: > +1 (non-binding). Tested this using a simple Dataflow ML pipeline: > https://github.com/google/dataflow-ml-starter/actions/runs/8541848483. > > On Wed, Apr 3, 2024 at 2:35 PM Jeff Kinard

Re: [VOTE] Patch Release 2.55.1, release candidate #2

+1 (non-binding). Tested this using a simple Dataflow ML pipeline: https://github.com/google/dataflow-ml-starter/actions/runs/8541848483. On Wed, Apr 3, 2024 at 2:35 PM Jeff Kinard wrote: > +1. Validated running from local gradle JAR and staged maven JAR for > expansion-service. > > On Wed, Apr

Re: [VOTE] Patch Release 2.55.1, release candidate #2

+1. Validated running from local gradle JAR and staged maven JAR for expansion-service. On Wed, Apr 3, 2024 at 11:08 AM Danny McCormick via dev wrote: > Hi everyone, > > I put together a patch release per the conversation in > https://lists.apache.org/thread/kvq1wsj505pvopkq186dnvc0l6ryyfh0. >

Re: [VOTE] Patch Release 2.55.1, release candidate #2

+1 (binding) Kenn On Wed, Apr 3, 2024 at 12:58 PM Danny McCormick via dev wrote: > > Also noting that there is no PR postsubmit test suite running against > the release branch in the vote email. Given the diff, that's also fine > since previous tests runs didn't detect the breakage, but in

Re: Supporting Dynamic Destinations in a portable context

On Wed, Apr 3, 2024 at 4:15 AM Kenneth Knowles wrote: > Let me summarize the most recent proposal on-list to frame my question > about this last suggestion. It looks like this: > > 1. user has an element, call it `data` > 2. user maps `data` to an arbitrary metadata row, call it `dest` > 3. we

Re: [VOTE] Patch Release 2.55.1, release candidate #2

> Also noting that there is no PR postsubmit test suite running against the release branch in the vote email. Given the diff, that's also fine since previous tests runs didn't detect the breakage, but in general we should include that for patch releases as well. Yeah, it didn't seem useful to

[PR] Publish docs for 2.55.1 release [beam-site]

damccorm opened a new pull request, #663: URL: https://github.com/apache/beam-site/pull/663 Content generated from https://github.com/apache/beam/tree/v2.55.1-RC2. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the

Re: [VOTE] Patch Release 2.55.1, release candidate #2

Hi Danny, Thanks for volunteering to do this patch release. For review convenience, this is the diff: - Diff of release branches: https://github.com/apache/beam/compare/release-2.55.0...release-2.55.1 - The diff of tags v2.55.0-RC3 and v2.55.1-RC2:

Community over Code EU 2024: Start planning your trip!

[Note: You're receiving this email because you are subscribed to one or more project dev@ mailing lists at the Apache Software Foundation.] Dear community, We hope you are doing great, are you ready for Community Over Code EU? Check out the featured sessions, get your tickets with special

[VOTE] Patch Release 2.55.1, release candidate #2

Hi everyone, I put together a patch release per the conversation in https://lists.apache.org/thread/kvq1wsj505pvopkq186dnvc0l6ryyfh0. Please review and vote on the release candidate #2 (I messed up rc1) for the version 2.55.1, as follows: [ ] +1, Approve the release [ ] -1, Do not approve the

Participate in the ASF 25th Anniversary Campaign

Hi everyone, As part of The ASF’s 25th anniversary campaign[1], we will be celebrating projects and communities in multiple ways. We invite all projects and contributors to participate in the following ways: * Individuals - submit your first contribution:

Re: Supporting Dynamic Destinations in a portable context

Let me summarize the most recent proposal on-list to frame my question about this last suggestion. It looks like this: 1. user has an element, call it `data` 2. user maps `data` to an arbitrary metadata row, call it `dest` 3. we can do things like shuffle on `dest` because it isn't too big 4. we

Beam High Priority Issue Report (59)

This is your daily summary of Beam's current high priority issues that may need attention. See https://beam.apache.org/contribute/issue-priorities for the meaning and expectations around issue priorities. Unassigned P1 Issues: https://github.com/apache/beam/issues/30820 [Bug]: Install

Re: Design proposal for Beam YAML templates

It looks like we're converging on an optional jinja preprocessing phase to handle this. I'm in favor of this solution. On Wed, Mar 20, 2024 at 9:23 AM Robert Bradshaw wrote: > Thanks. I think this will be a very powerful feature. Left some comments > on the doc. > > On Tue, Mar 19, 2024 at

Re: Supporting Dynamic Destinations in a portable context

I do suspect that over time we'll find more and more cases we can't express, and will be asked to extend this little templating in more directions. To head that off - could we easily just reuse an existing language (SQL, LUA, something of the form?) instead of creating something new? On Tue, Apr

Re: Supporting Dynamic Destinations in a portable context

I really like this proposal. I think it has narrowed down and solved the essential problem of not shuffling excess redundant data, and also provides the vast majority of the functionality that a lambda would, with significantly better debugability and usability too, since the dynamic destination

Beam High Priority Issue Report (59)

This is your daily summary of Beam's current high priority issues that may need attention. See https://beam.apache.org/contribute/issue-priorities for the meaning and expectations around issue priorities. Unassigned P1 Issues: https://github.com/apache/beam/issues/30820 [Bug]: Install

[GSOC] Build out Beam Use Cases Project Proposal

Hi Everyone, I am Ayush Pandey, interested in the "Build out Beam Use Cases" (Project Link: [GSOC] Build out Beam Use Cases ) Project for GSoC 2024. I worked as a GSoC contributor for Apache Cloudstack in 2023 and was really interested in working

Beam High Priority Issue Report (62)

This is your daily summary of Beam's current high priority issues that may need attention. See https://beam.apache.org/contribute/issue-priorities for the meaning and expectations around issue priorities. Unassigned P1 Issues: https://github.com/apache/beam/issues/30820 [Bug]: Install

[Python SDK] Feedback for deferred side inputs + combiners

I posted a PoC PR [1] for fixing deferred side inputs with combiners in the python SDK. Would someone be willing to take a look at it? I have it working but could use some feedback on where to take it next. It looks like bundle processor combiner operations don't currently support side inputs [2]

Beam High Priority Issue Report (59)

This is your daily summary of Beam's current high priority issues that may need attention. See https://beam.apache.org/contribute/issue-priorities for the meaning and expectations around issue priorities. Unassigned P1 Issues: https://github.com/apache/beam/issues/30799 The PostCommit

Re: Patch release proposal

On Thu, Mar 28, 2024 at 8:57 AM Jan Lukavský wrote: > +1 to either doing full release or deferring to 2.56.0. > +1. Given that validation/testing required for unupdated SDKs should be minimum, I don't think a full release will be that much overhead compared to just releasing Python SDK. Also

Re: Patch release proposal

+1 to either doing full release or deferring to 2.56.0.  Jan On 3/28/24 16:52, Yi Hu via dev wrote: > Just releasing Python can break multi-lang by default (unless expansion service is overridden manually) since we match versions across languages when picking the default expansion service.

Re: Patch release proposal

> Just releasing Python can break multi-lang by default (unless expansion service is overridden manually) since we match versions across languages when picking the default expansion service. Yes, that's why I proposed "the source code of release candidate (e.g. apache_beam/version.py) still reads

Re: Patch release proposal

On Thu, Mar 28, 2024 at 8:36 AM Chamikara Jayalath wrote: > Just releasing Python can break multi-lang by default (unless expansion > service is overridden manually) since we match versions across languages > when picking the default expansion service. > > >

Re: Patch release proposal

Just releasing Python can break multi-lang by default (unless expansion service is overridden manually) since we match versions across languages when picking the default expansion service.

Re: Patch release proposal

> The patch itself [1] is trivial, however, the release process is not trivial. There is little documentation nor practice for a patch release process. I could imagine two options I think there's not a ton of documentation because we haven't done it, but all the release workflows were authored in

Re: Patch release proposal

Thanks Valentyn for raising this. In this case, Python containers will also be included. Different from PyPI wheels, docker tag can override so it can stay with 2.55.0 On Thu, Mar 28, 2024 at 11:15 AM Valentyn Tymofieiev wrote: > If we do a patch release for Python SDK, let's also patch another

Re: Patch release proposal

If we do a patch release for Python SDK, let's also patch another known issue for which fix is available: https://github.com/apache/beam/blob/master/CHANGES.md#known-issues-1 On Thu, Mar 28, 2024 at 8:01 AM Yi Hu via dev wrote: > 2.55.0 release manager here > > The patch itself [1] is trivial,

Re: Patch release proposal

2.55.0 release manager here The patch itself [1] is trivial, however, the release process is not trivial. There is little documentation nor practice for a patch release process. I could imagine two options 1. Do a full "2.55.1" release 2. Do a patch release only for Python SDK, that is a.

Re: Patch release proposal

+1 on a patch release - we've done a fair amount of work to make releasing easier, and one of my hopes is that it will enable quick patches like this. I'd vote we try to fix the underlying Java piece as well, though, doing a patch release for one language shouldn't be significantly cheaper than

Beam High Priority Issue Report (59)

This is your daily summary of Beam's current high priority issues that may need attention. See https://beam.apache.org/contribute/issue-priorities for the meaning and expectations around issue priorities. Unassigned P1 Issues: https://github.com/apache/beam/issues/30776 [Bug]: Beam YAML

Re: Patch release proposal

+1 to a targeted patch release. We did the same for the Go SDK a little while back. It would be good to see what's different for a different SDK. On Wed, Mar 27, 2024, 4:01 PM Robert Bradshaw via dev wrote: > Given the severity of the breakage, and the simplicity of the workaround, > I'm in

Re: Patch release proposal

Given the severity of the breakage, and the simplicity of the workaround, I'm in favor of a patch release. I think we could do Python-only, which would make the process even more lightweight. On Wed, Mar 27, 2024 at 3:48 PM Jeff Kinard wrote: > Hi all, > > Beam 2.55 was released with a bug that

Patch release proposal

Hi all, Beam 2.55 was released with a bug that causes WriteToJson on Beam YAML to fail when using the Java variant. This also affects any user attempting to use the Xlang JsonWriteTransformProvider -

Re: Supporting Dynamic Destinations in a portable context

On Wed, Mar 27, 2024 at 10:20 AM Reuven Lax wrote: > Can the prefix still be generated programmatically at graph creation time? > Yes. It's just a property of the transform passed by the user at configuration time. > On Wed, Mar 27, 2024 at 9:40 AM Robert Bradshaw > wrote: > >> On Wed, Mar

Re: Supporting Dynamic Destinations in a portable context

Can the prefix still be generated programmatically at graph creation time? On Wed, Mar 27, 2024 at 9:40 AM Robert Bradshaw wrote: > On Wed, Mar 27, 2024 at 9:12 AM Reuven Lax wrote: > >> This does seem like the best compromise, though I think there will still >> end up being performance

Re: [PR] Add yaml docs to beam reference documentation website. [beam-site]

robertwb merged PR #662: URL: https://github.com/apache/beam-site/pull/662 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

Re: [PR] Add yaml docs to beam reference documentation website. [beam-site]

robertwb commented on PR #662: URL: https://github.com/apache/beam-site/pull/662#issuecomment-2023256794 Thanks. Yeah, this is mostly a question of placement. (The html generation has been reviewed in previous PRs.) -- This is an automated message from the Apache Git Service. To

Re: Supporting Dynamic Destinations in a portable context

On Wed, Mar 27, 2024 at 9:12 AM Reuven Lax wrote: > This does seem like the best compromise, though I think there will still > end up being performance issues. A common pattern I've seen is that there > is a long common prefix to the dynamic destination followed the dynamic > component. e.g. the

Re: Supporting Dynamic Destinations in a portable context

> This does seem like the best compromise, though I think there will still end up being performance issues. A common pattern I've seen is that there is a long common prefix to the dynamic destination followed the dynamic component. e.g. the destination might be

Re: [PR] Add yaml docs to beam reference documentation website. [beam-site]

brucearctor commented on PR #662: URL: https://github.com/apache/beam-site/pull/662#issuecomment-2023233436 I am assuming since auto-generated, what is generated is what we want ... Didn't verify the `index.html` page. -- This is an automated message from the Apache Git Service. To

[PR] Add yaml docs to beam reference documentation website. [beam-site]

robertwb opened a new pull request, #662: URL: https://github.com/apache/beam-site/pull/662 This was generated with `python -m apache_beam.yaml.generate_yaml_docs`. (Adding this to the beam release process is at https://github.com/apache/beam/pull/30741.) -- This is an automated

Re: [PR] Add yaml docs to beam reference documentation website. [beam-site]

robertwb commented on PR #662: URL: https://github.com/apache/beam-site/pull/662#issuecomment-2023217884 R: @Abacn -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

Re: Supporting Dynamic Destinations in a portable context

This does seem like the best compromise, though I think there will still end up being performance issues. A common pattern I've seen is that there is a long common prefix to the dynamic destination followed the dynamic component. e.g. the destination might be

Re: Supporting Dynamic Destinations in a portable context

Thanks for putting this together, it will be a really useful feature to have. I am in favor of the string-pattern approaches. I think we need to support both the {record=..., dest_info=...} and the elide-fields approaches, as the former is nicer when one has a fixed representation for the output

Supporting Dynamic Destinations in a portable context

Hey all, There have been some conversations lately about how best to enable dynamic destinations in a portable context. Usually, this comes up for cross-language transforms and more recently for Beam YAML. I've started a short doc outlining some routes we could take. The purpose is to establish

Beam High Priority Issue Report (57)

This is your daily summary of Beam's current high priority issues that may need attention. See https://beam.apache.org/contribute/issue-priorities for the meaning and expectations around issue priorities. Unassigned P1 Issues: https://github.com/apache/beam/issues/30760 The PostCommit

Community Over Code NA 2024 Travel Assistance Applications now open!

Hello to all users, contributors and Committers! [ You are receiving this email as a subscriber to one or more ASF project dev or user mailing lists and is not being sent to you directly. It is important that we reach all of our users and contributors/committers so that they may get a chance

[Announce] Beam 2.55.0 Release

We are happy to present the new 2.55.0 release of Beam. This release includes both improvements and new functionality. See https://beam.apache.org/get-started/downloads/ for this release. For more information on changes in 2.55.0, check out the detailed release notes at

Beam High Priority Issue Report (55)

This is your daily summary of Beam's current high priority issues that may need attention. See https://beam.apache.org/contribute/issue-priorities for the meaning and expectations around issue priorities. Unassigned P1 Issues: https://github.com/apache/beam/issues/30737 [Failing Test]:

Re: [VOTE] Release 2.55.0, release candidate #3

Hi, Could a PMC member please help me finalize the source release? That is the steps documented here: https://github.com/apache/beam/blob/master/contributor-docs/release-guide.md#pmc-only-finalization Thanks! On Mon, Mar 25, 2024 at 9:37 AM Yi Hu wrote: > I'm happy to announce that we have

Re: [PR] Publish docs for 2.55.0 release [beam-site]

Abacn merged PR #661: URL: https://github.com/apache/beam-site/pull/661 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@beam.apache.org

Re: [VOTE] Release 2.55.0, release candidate #3

I'm happy to announce that we have unanimously approved this release. There are 7 approving votes, 3 of which are binding: * Ritesh Ghorse * XQ Hu * Chamikara Jayalath (binding) * Jan Lukavský (binding) * Danny McCormick * Yi Hu * Valentyn Tymofieiev There are no disapproving votes. Thanks

<    1   2   3   4   5   6   7   8   9   10   >