Flaky test issue report (29)

This is your daily summary of Beam's current flaky tests (https://issues.apache.org/jira/issues/?jql=project%20%3D%20BEAM%20AND%20statusCategory%20!%3D%20Done%20AND%20labels%20%3D%20flake) These are P1 issues because they have a major negative impact on the community and make it hard to determin

P1 issues report (42)

This is your daily summary of Beam's current P1 issues, not including flaky tests (https://issues.apache.org/jira/issues/?jql=project%20%3D%20BEAM%20AND%20statusCategory%20!%3D%20Done%20AND%20priority%20%3D%20P1%20AND%20(labels%20is%20EMPTY%20OR%20labels%20!%3D%20flake). See https://beam.apache.

Re: [VOTE] Release 2.31.0, release candidate #1

+1 ran a couple more validation of configuration that aren't in the scripts, etc. On Fri, Jul 2, 2021 at 6:51 PM Ahmet Altay wrote: > > > On Fri, Jul 2, 2021 at 9:57 AM Andrew Pilloud wrote: > >> Thanks for noticing that! The key was updated prior to signature. I need >> a PMC member's help to

Re: [IDEA] Privacy (and security) in Apache Beam

This is a whole area of research that is really cool. Something I read a while ago and think about sometimes: Explaining output in Modern Data Analytics (writeup here https://blog.acolyer.org/2017/02/01/explaining-outputs-in-modern-data-analytics/ ) Kenn On Wed, Jul 7, 2021 at 8:18 AM Reuven Lax

Re: Help: Apache Beam Session Window with limit on number of events and time elapsed from window start

Hi Chandan, I am moving this thread to u...@beam.apache.org. I think that is the best place to discuss. Kenn On Wed, Jul 7, 2021 at 9:32 AM Chandan Bhattad wrote: > Hi Team, > > Hope you are doing well. > > I have a use case around session windowing with some customizations. > > We need to hav

Help: Apache Beam Session Window with limit on number of events and time elapsed from window start

Hi Team, Hope you are doing well. I have a use case around session windowing with some customizations. We need to have create user sessions based on *any *of the 3 conditions below 1. Session Window of 30 minutes (meaning, 30 minutes of inactivity i.e. no event for 30 minutes for a user) 2. Num

Documentation error

Hello everybody, Thanks for the amazing work! I'm loving it! But the python code on the item 2.1.1. Setting PipelineOptions from command-line arguments is wrong. According to Google docs it should be: from apache

Re: Spotbugs issue on project I did not modify

Thank you for the confirmation Alexey. It worked out. On Wed, Jul 7, 2021 at 6:03 AM Alexey Romanenko wrote: > Yes, rebasing against HEAD is almost always a good idea. I did run locally > “./gradlew :sdks:java:harness:check” (which includes “spotbugs" check) > against current HEAD and there is

[IDEA] Privacy (and security) in Apache Beam

Hello! I am wondering if there is anyone interested in exploring the topic of privacy (and potentially security) in the Apache Beam unified programming model. I have been a user of Apache Beam mostly via Tensorflow Transform but also directly and followed its evolution and development early on. H

Re: JavaPrecommit fails

For the reference: https://issues.apache.org/jira/browse/BEAM-12583 > On 6 Jul 2021, at 18:31, Alexey Romanenko wrote: > > Yes, it fails constantly for me too. I briefly did a "git bisect” and > a60aebaac9c90d393ca1e2a7445d45222d908541 is the first bad commit. It’s quite > strange that this te

Re: Spotbugs issue on project I did not modify

Yes, rebasing against HEAD is almost always a good idea. I did run locally “./gradlew :sdks:java:harness:check” (which includes “spotbugs" check) against current HEAD and there is no issue with spotbugsMain task. — Alexey > On 7 Jul 2021, at 04:28, Matthew Ouyang wrote: > > I get the followi

Re: FileIO with custom sharding function

On Sat, Jul 3, 2021 at 7:30 PM Reuven Lax wrote: > > > On Sat, Jul 3, 2021 at 1:02 AM Jozef Vilcek wrote: > >> I don't think this has anything to do with external shuffle services. >> >> Arbitrarily recomputing data is fundamentally incompatible with Beam, >> since Beam does not restrict transfo

Re: FileIO with custom sharding function

On Sat, Jul 3, 2021 at 12:55 PM Jan Lukavský wrote: > > I don't think this has anything to do with external shuffle services. > > > Sorry, for stepping into this discussion again, but I don't think this > statement is 100% correct. What Spark's checkpoint does is that it saves > intermediate data