Re: Aliasing Pub/Sub Lite IO in external repo

2021-07-01 Thread Evan Galpin
Is there any possibility of changing the build cadence allowing for builds released as alpha versions or similar? It’s not too uncommon for projects to have nightly builds for example. Could that help deliver fixes more quickly to customers, while also avoiding the nuisances mentioned in this threa

Re: [Proposal] Portable MultimapState

2021-07-01 Thread Robert Burke
Seems reasonable. Assuming consensus happens, please do add this to the list of design docs on the wiki: https://cwiki.apache.org/confluence/plugins/servlet/mobile?contentId=95653903#content/view/95653903 On Mon, Jun 28, 2021, 4:50 PM Robert Bradshaw wrote: > Thanks. Approach 2 looks reasonab

Re: Specifying environment for cross-language transform expansion

2021-07-01 Thread Jan Lukavský
This really does not match my experience. Passing the correct "use_deprecated_read" flag to the expansion service had the expected impact on the Flink's execution DAG and - most of all - it started to work (at least seems so). The UI in Flink also started to reflect that and stopped using SDF (

Flaky test issue report (29)

2021-07-01 Thread Beam Jira Bot
This is your daily summary of Beam's current flaky tests (https://issues.apache.org/jira/issues/?jql=project%20%3D%20BEAM%20AND%20statusCategory%20!%3D%20Done%20AND%20labels%20%3D%20flake) These are P1 issues because they have a major negative impact on the community and make it hard to determin

Re: Specifying environment for cross-language transform expansion

2021-07-01 Thread Jan Lukavský
I don't have complete comprehension of the topic, but from what I have observed, the runner gets (possibly cross-language) proto description of the pipeline, and the post-processing there might be limited.  That is mainly due to the fact, that we have inverted the expansion flow - we expand Rea

Re: Specifying environment for cross-language transform expansion

2021-07-01 Thread Jan Lukavský
Hi, after today's experience I think I have some arguments about why we *should* pass (at least some) of the PipelineOptions from SDK to expansion service.  1) there are lots, and lots, and lots of bugs around SDF and around the "use_deprecated_read", sorry, but the switch to SDF as the defa

P1 issues report (42)

2021-07-01 Thread Beam Jira Bot
This is your daily summary of Beam's current P1 issues, not including flaky tests (https://issues.apache.org/jira/issues/?jql=project%20%3D%20BEAM%20AND%20statusCategory%20!%3D%20Done%20AND%20priority%20%3D%20P1%20AND%20(labels%20is%20EMPTY%20OR%20labels%20!%3D%20flake). See https://beam.apache.

[ALERT] Disk Space Utilization (Apache Beam Jenkins) on apache-beam-testing apache-ci-beam-jenkins-11

2021-07-01 Thread Google Cloud Alerting
Title: [ALERT] Disk Space Utilization (Apache Beam Jenkins) on apache-beam-testing apache-ci-beam-jenkins-11 VIEW INCIDENT Alert firing Disk Space Utilization (Ap

[ALERT] Disk Space Utilization (Apache Beam Jenkins) on apache-beam-testing apache-ci-beam-jenkins-6

2021-07-01 Thread Google Cloud Alerting
Title: [ALERT] Disk Space Utilization (Apache Beam Jenkins) on apache-beam-testing apache-ci-beam-jenkins-6 VIEW INCIDENT Alert firing Disk Space Utilization (Apa

[ALERT] Disk Space Utilization (Apache Beam Jenkins) on apache-beam-testing apache-ci-beam-jenkins-11

2021-07-01 Thread Google Cloud Alerting
Title: [ALERT] Disk Space Utilization (Apache Beam Jenkins) on apache-beam-testing apache-ci-beam-jenkins-11 VIEW INCIDENT Alert firing Disk Space Utilization (Ap

Re: Specifying environment for cross-language transform expansion

2021-07-01 Thread Jan Lukavský
On 7/1/21 3:26 AM, Kyle Weaver wrote: I think it should accept complete list of PipelineOptions (or at least some defined subset - PortabilityPipelineOptions, ExperimentalOptions, ...?) I'm not totally opposed to redefining some options, either. Using PipelineOptions could be conf