Re: [Proposal] => JMSIO dynamic topic publishing

[Jean-Baptiste Onofré]

Hi Vincent,

It looks interesting. Another possible approach is to have some
implicit (instead of being explicit) but defining the destination on
the JmsRecord. If the JmsRecord contains the destination (that could
be "dynamic"), we use it, overriding the destination provided on the
IO configuration.
Thoughts ?

Regards
JB

NB: I'm the original author of JmsIO ;)

On Fri, Feb 18, 2022 at 7:00 PM BALLADA Vincent
 wrote:
>
> Hi all
>
>
>
> Here is a proposal to implement the ability to publish on dynamic topics with 
> JMSIO:
>
> https://docs.google.com/document/d/1IY4_e5g1g71XvTLL4slHRyVfX7ByiwjD_de3WGsBQXg/edit?usp=sharing
>
>
>
> There is also a JIRA issue:
>
> https://issues.apache.org/jira/browse/BEAM-13608
>
>
>
> Best regards
>
>
>
> Vincent BALLADA
>
>
> Confidential C
>
> -- Disclaimer 
> Ce message ainsi que les eventuelles pieces jointes constituent une 
> correspondance privee et confidentielle a l'attention exclusive du 
> destinataire designe ci-dessus. Si vous n'etes pas le destinataire du present 
> message ou une personne susceptible de pouvoir le lui delivrer, il vous est 
> signifie que toute divulgation, distribution ou copie de cette transmission 
> est strictement interdite. Si vous avez recu ce message par erreur, nous vous 
> remercions d'en informer l'expediteur par telephone ou de lui retourner le 
> present message, puis d'effacer immediatement ce message de votre systeme.
>
> *** This e-mail and any attachments is a confidential correspondence intended 
> only for use of the individual or entity named above. If you are not the 
> intended recipient or the agent responsible for delivering the message to the 
> intended recipient, you are hereby notified that any disclosure, distribution 
> or copying of this communication is strictly prohibited. If you have received 
> this communication in error, please notify the sender by phone or by replying 
> this message, and then delete this message from your system.

[Proposal] => JMSIO dynamic topic publishing

[BALLADA Vincent]

Hi all

Here is a proposal to implement the ability to publish on dynamic topics with 
JMSIO:
https://docs.google.com/document/d/1IY4_e5g1g71XvTLL4slHRyVfX7ByiwjD_de3WGsBQXg/edit?usp=sharing

There is also a JIRA issue:
https://issues.apache.org/jira/browse/BEAM-13608

Best regards

Vincent BALLADA


Confidential C
-- Disclaimer  
Ce message ainsi que les eventuelles pieces jointes constituent une 
correspondance privee et confidentielle a l'attention exclusive du destinataire 
designe ci-dessus. Si vous n'etes pas le destinataire du present message ou une 
personne susceptible de pouvoir le lui delivrer, il vous est signifie que toute 
divulgation, distribution ou copie de cette transmission est strictement 
interdite. Si vous avez recu ce message par erreur, nous vous remercions d'en 
informer l'expediteur par telephone ou de lui retourner le present message, 
puis d'effacer immediatement ce message de votre systeme.

*** This e-mail and any attachments is a confidential correspondence intended 
only for use of the individual or entity named above. If you are not the 
intended recipient or the agent responsible for delivering the message to the 
intended recipient, you are hereby notified that any disclosure, distribution 
or copying of this communication is strictly prohibited. If you have received 
this communication in error, please notify the sender by phone or by replying 
this message, and then delete this message from your system.

P1 issues report (71)

[Beam Jira Bot]

This is your daily summary of Beam's current P1 issues, not including flaky 
tests 
(https://issues.apache.org/jira/issues/?jql=project%20%3D%20BEAM%20AND%20statusCategory%20!%3D%20Done%20AND%20priority%20%3D%20P1%20AND%20(labels%20is%20EMPTY%20OR%20labels%20!%3D%20flake).

See https://beam.apache.org/contribute/jira-priorities/#p1-critical for the 
meaning and expectations around P1 issues.

https://issues.apache.org/jira/browse/BEAM-13959: Unable to write to 
BigQuery tables with column named 'f' (created 2022-02-16)
https://issues.apache.org/jira/browse/BEAM-13952: Dataflow streaming tests 
failing new AfterSynchronizedProcessingTime test (created 2022-02-15)
https://issues.apache.org/jira/browse/BEAM-13950: PVR_Spark2_Streaming 
perma-red (created 2022-02-15)
https://issues.apache.org/jira/browse/BEAM-13920: Beam x-lang Dataflow 
tests failing due to _InactiveRpcError (created 2022-02-10)
https://issues.apache.org/jira/browse/BEAM-13858: Failure of 
:sdks:go:examples:wordCount in check "Mac run local environment shell script" 
(created 2022-02-08)
https://issues.apache.org/jira/browse/BEAM-13850: 
beam_PostCommit_Python_Examples_Dataflow failing (created 2022-02-08)
https://issues.apache.org/jira/browse/BEAM-13830: XVR Direct/Spark/Flink 
tests are timing out (created 2022-02-04)
https://issues.apache.org/jira/browse/BEAM-13822: GBK and CoGBK streaming 
Java load tests failing (created 2022-02-03)
https://issues.apache.org/jira/browse/BEAM-13809: beam_PostCommit_XVR_Flink 
flaky: Connection refused (created 2022-02-03)
https://issues.apache.org/jira/browse/BEAM-13805: Simplify version override 
for Dev versions of the Go SDK. (created 2022-02-02)
https://issues.apache.org/jira/browse/BEAM-13798: Upgrade Kubernetes 
Clusters (created 2022-02-01)
https://issues.apache.org/jira/browse/BEAM-13769: 
beam_PreCommit_Python_Cron failing on test_create_uses_coder_for_pickling 
(created 2022-01-28)
https://issues.apache.org/jira/browse/BEAM-13763: Rotate credentials for 
'io-datastores' Kubernetes cluster (created 2022-01-28)
https://issues.apache.org/jira/browse/BEAM-13741: 
:sdks:java:extensions:sql:hcatalog:compileJava failing in 
beam_Release_NightlySnapshot  (created 2022-01-25)
https://issues.apache.org/jira/browse/BEAM-13715: Kafka commit offset drop 
data on failure for runners that have non-checkpointing shuffle (created 
2022-01-21)
https://issues.apache.org/jira/browse/BEAM-13693: 
beam_PostCommit_Java_ValidatesRunner_Dataflow_Streaming timing out at 9 hours 
(created 2022-01-19)
https://issues.apache.org/jira/browse/BEAM-13582: Beam website precommit 
mentions broken links, but passes. (created 2021-12-30)
https://issues.apache.org/jira/browse/BEAM-13579: Cannot run 
python_xlang_kafka_taxi_dataflow validation script on 2.35.0 (created 
2021-12-29)
https://issues.apache.org/jira/browse/BEAM-13487: WriteToBigQuery Dynamic 
table destinations returns wrong tableId (created 2021-12-17)
https://issues.apache.org/jira/browse/BEAM-13393: GroupIntoBatchesTest is 
failing (created 2021-12-07)
https://issues.apache.org/jira/browse/BEAM-13237: 
org.apache.beam.sdk.transforms.CombineTest$WindowingTests.testWindowedCombineGloballyAsSingletonView
 flaky on Dataflow Runner V2 (created 2021-11-12)
https://issues.apache.org/jira/browse/BEAM-13164: Race between member 
variable being accessed due to leaking uninitialized state via 
OutboundObserverFactory (created 2021-11-01)
https://issues.apache.org/jira/browse/BEAM-13132: WriteToBigQuery submits a 
duplicate BQ load job if a 503 error code is returned from googleapi (created 
2021-10-27)
https://issues.apache.org/jira/browse/BEAM-13087: 
apache_beam.runners.portability.fn_api_runner.translations_test.TranslationsTest.test_run_packable_combine_globally
 'apache_beam.coders.coder_impl._AbstractIterable' object is not reversible 
(created 2021-10-20)
https://issues.apache.org/jira/browse/BEAM-13078: Python DirectRunner does 
not emit data at GC time (created 2021-10-18)
https://issues.apache.org/jira/browse/BEAM-13076: Python AfterAny, AfterAll 
do not follow spec (created 2021-10-18)
https://issues.apache.org/jira/browse/BEAM-13010: Delete orphaned files 
(created 2021-10-06)
https://issues.apache.org/jira/browse/BEAM-12995: Consumer group with 
random prefix (created 2021-10-04)
https://issues.apache.org/jira/browse/BEAM-12959: Dataflow error in 
CombinePerKey operation (created 2021-09-26)
https://issues.apache.org/jira/browse/BEAM-12867: Either Create or 
DirectRunner fails to produce all elements to the following transform (created 
2021-09-09)
https://issues.apache.org/jira/browse/BEAM-12843: (Broken Pipe induced) 
Bricked Dataflow Pipeline  (created 2021-09-06)
https://issues.apache.org/jira/browse/BEAM-12807: Java creates an incorrect 
pipeline proto when core-construction-java jar is not in the CLASSPATH (created 
2021-08-26)
ht

Flaky test issue report (51)

[Beam Jira Bot]

This is your daily summary of Beam's current flaky tests 
(https://issues.apache.org/jira/issues/?jql=project%20%3D%20BEAM%20AND%20statusCategory%20!%3D%20Done%20AND%20labels%20%3D%20flake)

These are P1 issues because they have a major negative impact on the community 
and make it hard to determine the quality of the software.

https://issues.apache.org/jira/browse/BEAM-13952: Dataflow streaming tests 
failing new AfterSynchronizedProcessingTime test (created 2022-02-15)
https://issues.apache.org/jira/browse/BEAM-13859: Test flake: 
test_split_half_sdf (created 2022-02-09)
https://issues.apache.org/jira/browse/BEAM-13858: Failure of 
:sdks:go:examples:wordCount in check "Mac run local environment shell script" 
(created 2022-02-08)
https://issues.apache.org/jira/browse/BEAM-13850: 
beam_PostCommit_Python_Examples_Dataflow failing (created 2022-02-08)
https://issues.apache.org/jira/browse/BEAM-13822: GBK and CoGBK streaming 
Java load tests failing (created 2022-02-03)
https://issues.apache.org/jira/browse/BEAM-13810: Flaky tests: Gradle build 
daemon disappeared unexpectedly (created 2022-02-03)
https://issues.apache.org/jira/browse/BEAM-13797: Flakes: Failed to load 
cache entry (created 2022-02-01)
https://issues.apache.org/jira/browse/BEAM-13783: 
apache_beam.transforms.combinefn_lifecycle_test.LocalCombineFnLifecycleTest.test_combine
 is flaky (created 2022-02-01)
https://issues.apache.org/jira/browse/BEAM-13741: 
:sdks:java:extensions:sql:hcatalog:compileJava failing in 
beam_Release_NightlySnapshot  (created 2022-01-25)
https://issues.apache.org/jira/browse/BEAM-13708: flake: 
FlinkRunnerTest.testEnsureStdoutStdErrIsRestored (created 2022-01-20)
https://issues.apache.org/jira/browse/BEAM-13693: 
beam_PostCommit_Java_ValidatesRunner_Dataflow_Streaming timing out at 9 hours 
(created 2022-01-19)
https://issues.apache.org/jira/browse/BEAM-13575: Flink 
testParDoRequiresStableInput flaky (created 2021-12-28)
https://issues.apache.org/jira/browse/BEAM-13519: Java precommit flaky 
(timing out) (created 2021-12-22)
https://issues.apache.org/jira/browse/BEAM-13500: NPE in Flink Portable 
ValidatesRunner streaming suite (created 2021-12-21)
https://issues.apache.org/jira/browse/BEAM-13453: Flake in 
org.apache.beam.sdk.io.mqtt.MqttIOTest.testReadObject: Address already in use 
(created 2021-12-13)
https://issues.apache.org/jira/browse/BEAM-13393: GroupIntoBatchesTest is 
failing (created 2021-12-07)
https://issues.apache.org/jira/browse/BEAM-13367: 
[beam_PostCommit_Python36] [ 
apache_beam.io.gcp.experimental.spannerio_read_it_test] Failure summary 
(created 2021-12-01)
https://issues.apache.org/jira/browse/BEAM-13312: 
org.apache.beam.sdk.transforms.ParDoLifecycleTest.testTeardownCalledAfterExceptionInStartBundle
 is flaky in Java Spark ValidatesRunner suite  (created 2021-11-23)
https://issues.apache.org/jira/browse/BEAM-13311: 
org.apache.beam.sdk.transforms.ParDoLifecycleTest.testTeardownCalledAfterExceptionInProcessElementStateful
 is flaky in Java ValidatesRunner Flink suite. (created 2021-11-23)
https://issues.apache.org/jira/browse/BEAM-13234: Flake in 
StreamingWordCountIT.test_streaming_wordcount_it (created 2021-11-12)
https://issues.apache.org/jira/browse/BEAM-13025: pubsublite.ReadWriteIT 
flaky in beam_PostCommit_Java_DataflowV2   (created 2021-10-08)
https://issues.apache.org/jira/browse/BEAM-12928: beam_PostCommit_Python36 
- CrossLanguageSpannerIOTest - flakey failing (created 2021-09-21)
https://issues.apache.org/jira/browse/BEAM-12859: 
org.apache.beam.runners.dataflow.worker.fn.logging.BeamFnLoggingServiceTest.testMultipleClientsFailingIsHandledGracefullyByServer
 is flaky (created 2021-09-08)
https://issues.apache.org/jira/browse/BEAM-12858: 
org.apache.beam.sdk.io.gcp.datastore.RampupThrottlingFnTest.testRampupThrottler 
is flaky (created 2021-09-08)
https://issues.apache.org/jira/browse/BEAM-12809: 
testTwoTimersSettingEachOtherWithCreateAsInputBounded flaky (created 2021-08-26)
https://issues.apache.org/jira/browse/BEAM-12794: 
PortableRunnerTestWithExternalEnv.test_pardo_timers flaky (created 2021-08-24)
https://issues.apache.org/jira/browse/BEAM-12793: 
beam_PostRelease_NightlySnapshot failed (created 2021-08-24)
https://issues.apache.org/jira/browse/BEAM-12766: Already Exists: Dataset 
apache-beam-testing:python_bq_file_loads_NNN (created 2021-08-16)
https://issues.apache.org/jira/browse/BEAM-12673: 
apache_beam.examples.streaming_wordcount_it_test.StreamingWordCountIT.test_streaming_wordcount_it
 flakey (created 2021-07-28)
https://issues.apache.org/jira/browse/BEAM-12515: Python PreCommit flaking 
in PipelineOptionsTest.test_display_data (created 2021-06-18)
https://issues.apache.org/jira/browse/BEAM-12322: Python precommit flaky: 
Failed to read inputs in the data plane (created 2021-05-10)
https://issues.apache.org/jira/browse/BEAM-12320: 
PubsubTable

Re: Beam Java starter project template

[Kenneth Knowles]

I always get mixed up myself. The policies are at
https://www.apache.org/legal/src-headers.html#notice and there's some step
by step at https://infra.apache.org/licensing-howto.html

TL;DR the contents should be like so:

Apache Beam
Copyright [2022-] The Apache Software Foundation

This product includes software developed at
The Apache Software Foundation (http://www.apache.org/).

Kenn

On Thu, Feb 17, 2022 at 2:28 PM David Cavazos  wrote:

> I found this example NOTICE
>  file, but
> it doesn't look like it does what we want. It looks like it has to be
> written in a formal legal language and I don't feel comfortable writing it.
> Can I ask for help on writing out the contents of the NOTICE file?
>
> On Thu, Feb 17, 2022 at 11:00 AM David Cavazos 
> wrote:
>
>> Can someone point me to an example on how the NOTICE file should look
>> like? I'm not familiar with it and would like to get it right.
>>
>> On Thu, Feb 17, 2022 at 10:53 AM David Cavazos 
>> wrote:
>>
>>> +1
>>> For the starter projects I like them being "clone and go", but I'd like
>>> to keep them as minimal as possible. We could have another repo like
>>> `beam-working-examples` for more complete examples where each subdirectory
>>> is a self-contained example with all its build files and everything.
>>>
>>> On Wed, Feb 16, 2022 at 5:59 AM Kenneth Knowles  wrote:
>>>
 I like the goal: for things where the build has extra setup, have an
 example that is fully functional on its own. There is of course the problem
 of "where does it end?" since this is infinity things.

 The other piece is that a user wanting to know some of these bits may
 be past the "clone and go" stage of their project. They probably already
 have a project and now they need a working example to read and learn from.
 So it could be just one additional repo `beam-working-examples` where each
 subdirectory is an independent working setup. I do like having it a
 separate repo to avoid the temptation to leverage anything from the Beam
 build. And each subdirectory should be entirely independent and we also
 have to avoid the temptation to share configuration across them, or it
 would defeat the purpose.

 Kenn

 On Tue, Feb 15, 2022 at 9:28 PM Reza Ardeshir Rokni 
 wrote:

> Hi,
>
> This is great!
>
> What do folks think about also having a less minimal set of starters?
> For Java I am thinking about protobuf / autovalue. For Python maybe an
> opinionated setup with tox etc... Again this would just contain 'hello'
> world samples to get folks going.
>
> Regards
> Reza
>
> On Wed, 9 Feb 2022 at 13:56, Robert Burke  wrote:
>
>> SGTM.
>>
>> On Wed, Feb 9, 2022 at 1:09 PM Kenneth Knowles 
>> wrote:
>>
>>> Based on discussion on
>>> https://issues.apache.org/jira/browse/LEGAL-601 I think it will be
>>> simplest to license it under ASL2 and include a NOTICE file. The user 
>>> will
>>> be free to "clone and go".
>>>
>>> I would bring these points back to the dev list:
>>>
>>>  - ASL2 is what people expect from an ASF project, so it is "least
>>> surprise"
>>>  - Dual-licensing is possible (but I think not worthwhile due to its
>>> impact on contributor license agreements)
>>>  - ASL2 says "You must cause any modified files to carry prominent
>>> notices stating that You changed the files" which won't apply to the 
>>> user's
>>> code and I would guess they simply won't bother with for files in the
>>> template. Or maybe there is a clever way to phrase the header so it is
>>> already good to go.
>>>  - ASL2 says if the work includes a NOTICE file, you have to
>>> includes the attributions from it. The NOTICE file is required by ASF
>>> policy. We can easily set it up to be a noop for the user.
>>>
>>> So my overall take is that we should go ahead with ASL2 and a simple
>>> NOTICE file. Check the Jira for details.
>>>
>>> Kenn
>>>
>>> On Mon, Feb 7, 2022 at 10:47 AM Kenneth Knowles 
>>> wrote:
>>>
 And I've created the repos just now.

 Kenn

 On Mon, Feb 7, 2022 at 10:39 AM Kenneth Knowles 
 wrote:

> Legal question asked at
> https://issues.apache.org/jira/browse/LEGAL-601
>
> Kenn
>
> On Fri, Feb 4, 2022 at 7:58 AM Danny McCormick <
> dannymccorm...@google.com> wrote:
>
>> Sure - I'm happy to help out with the Actions setup (and/or with
>> the Go template). I will say though, the Actions config should be 
>> pretty
>> darn simple for these examples -
>> https://github.com/davidcavazos/beam-java/blob/main/.github/workflows/test.yaml
>> seems right, for each language configura