Re: P0 (outage) report

2021-09-03 Thread Kenneth Knowles
Is this so flaky that it is essentially stopping all dev work? Kenn On Fri, Sep 3, 2021 at 11:02 AM Beam Jira Bot wrote: > This is your daily summary of Beam's current outages. See > https://beam.apache.org/contribute/jira-priorities/#p0-outage for the > meaning and expectations around P0 issue

Flaky test issue report (26)

2021-09-03 Thread Beam Jira Bot
This is your daily summary of Beam's current flaky tests (https://issues.apache.org/jira/issues/?jql=project%20%3D%20BEAM%20AND%20statusCategory%20!%3D%20Done%20AND%20labels%20%3D%20flake) These are P1 issues because they have a major negative impact on the community and make it hard to determin

Re: Unexpected in TestStream in Portable Mode

2021-09-03 Thread Jan Lukavský
Hi Robert, There's another hitch here for TestStream. For historical reasons, coders actually represent two encodings: nested (aka self delimiting) and unnested. TestStream elements are given as unnested encoded bytes, but the nested encoding is required for sending data to the SDK. The runner c

Re: Dropping @Experimental from SqlTransform

2021-09-03 Thread Reuven Lax
I think we should drop Experimental. For schemas, I would like to at least rename DATETIME to TIMESTAMP first. There's an argument for going further (removing DATETIME and making it a logical type), but that has been tricky when we've tried it in the past and would be harder for users to adapt to.

P1 issues report (53)

2021-09-03 Thread Beam Jira Bot
This is your daily summary of Beam's current P1 issues, not including flaky tests (https://issues.apache.org/jira/issues/?jql=project%20%3D%20BEAM%20AND%20statusCategory%20!%3D%20Done%20AND%20priority%20%3D%20P1%20AND%20(labels%20is%20EMPTY%20OR%20labels%20!%3D%20flake). See https://beam.apache.

P0 (outage) report

2021-09-03 Thread Beam Jira Bot
This is your daily summary of Beam's current outages. See https://beam.apache.org/contribute/jira-priorities/#p0-outage for the meaning and expectations around P0 issues. BEAM-5172: org.apache.beam.sdk.io.elasticsearch/ElasticsearchIOTest is flaky (https://issues.apache.org/jira/browse/BEAM

Re: Doubts on KafkaIO/SourceIO

2021-09-03 Thread Luke Cwik
https://beam.apache.org/documentation/io/developing-io-java/#implementing-the-reader-subclass is out of date and at the top says IMPORTANT: Use Splittable DoFn to develop your new I/O. For more details, read the new I/O connector overview. On Fri, Sep 3, 2021 at 9:55 AM Alexey Romanenko wrote: >

Re: Unexpected in TestStream in Portable Mode

2021-09-03 Thread Robert Bradshaw
On Fri, Sep 3, 2021 at 2:40 AM Jan Lukavský wrote: > > > On 9/3/21 1:06 AM, Robert Bradshaw wrote: > > On Thu, Sep 2, 2021 at 1:03 AM Jan Lukavský wrote: > >> Hi, > >> > >> I had some more time thinking about this and I'll try to recap that. > >> First some invariants: > >> > >>a) each PColle

Re: Doubts on KafkaIO/SourceIO

2021-09-03 Thread Alexey Romanenko
Hi Marco, I tried to answer your questions and I also CC’ed Boyuan Zhang as initial author of SDF-based Read implementation for KafkaIO. Also, I’d recommend to take a look on related PR’s discussion [1] which perhaps can give more details of some internal decisions. Please, see my answers inli

[no subject]

2021-09-03 Thread Aydar Zaynutdinov
Hi, My name is Aydar. I work as an engineer in Akvelon. I have a recent experience contributing to Apache Beam repository with BEAM-11873 issue. And I would like to continue contributing to the Apache Beam project. Can someone add me as a contributor to Beam's Jira issue tracker? I would like to c

[no subject]

2021-09-03 Thread Aydar Zaynutdinov
Hi, My name is Aydar. I work as an engineer in Akvelon. I have a recent experience contributing to Apache Beam repository with BEAM-11873 issue. And I would like to continue contributing to the Apache Beam project. Can someone add me as a contributor to Beam's Jira issue tracker? I would like to

Flink non-portable runner: unable to restore savepoint in 2.32.0

2021-09-03 Thread Lukáš Drbal
Hi, We are using beam + non-portable flink runner and now we are trying to upgrade from 2.31.0 to 2.32.0. Savepoint restore failed with: Caused by: java.lang.IllegalStateException: Could not Java-deserialize TypeSerializer while restoring checkpoint metadata for serializer snapshot 'org.apache.be

Re: Unexpected in TestStream in Portable Mode

2021-09-03 Thread Jan Lukavský
On 9/3/21 1:06 AM, Robert Bradshaw wrote: On Thu, Sep 2, 2021 at 1:03 AM Jan Lukavský wrote: Hi, I had some more time thinking about this and I'll try to recap that. First some invariants: a) each PCollection has actually two coders - an _SDK coder_ and a _runner coder_. These coders hav