Re: Why there are no certifications for Apache Beam

[Abhishek Patre]

Hi

Thank you for providing the information.

However, I attempted to use the code for "Getting Started with Apache Beam"
but Quicklabs still required 20 credits. It is possible that the promotion
has ended.

I am interested in knowing if there will be *any professional certification
for Apache Beam in the future.* Currently, the closest option I found is
Google's Professional Data Engineer certification, but it is not
specifically focused on Apache Beam. Additionally, it is not very
comprehensive.

Regards
Abhishek Patre

On Fri, Jul 28, 2023 at 12:44 AM Ahmet Altay  wrote:

> Hi Abhishek,
>
> We recently launched a new Beam certificate. You can find more information
> here:
> https://beam.apache.org/get-started/resources/learning-resources/#getting-started-with-apache-beam-quest.
> As part of the launch we also have a code for completing the program for
> free (more info here: https://beam.apache.org/blog/beamquest/ -- the code
> should be valid until Aug 18).
>
> Hope you find this useful.
>
> Ahmet
>
> On Thu, Jul 27, 2023 at 12:08 PM Abhishek Patre 
> wrote:
>
>> Hello Team,
>>
>> I hope this email finds you well. I was wondering if you could provide
>> some clarity on the availability of certification for Apache Beam. I
>> apologize if this isn't the appropriate mailing list for this inquiry.
>> Having a certification option for Apache Beam would be incredibly
>> beneficial in terms of showcasing our skills and expertise.
>>
>> Thank you for your time and assistance.
>>
>> Regards
>> Abhishek Patre
>>
>

Support for Flink 1.17 in Beam Flink Runner

[Vaibhav Gosain]

Hi team,

Are there any plans to add support for Flink 1.17 in the Beam Flink Runner?
The latest supported version is 1.16, and the latest stable release for
Flink is 1.17 .

There are a few fixes in 1.17 related to reactive/adaptive scaling and
operator autoscaler which we are eagerly looking to make use of.

Regards,
Vaibhav Gosain

Re: [RFC] Model Per Key RunInference

[Valentyn Tymofieiev via dev]

Thanks Danny! The narrative is well structured and easy to follow. I
encourage more folks to take a look. I left a couple of comments, mostly
about plans for memory management.

On Thu, Jul 20, 2023 at 7:47 AM Danny McCormick via dev 
wrote:

> Hey everyone! Today, many users have pipelines that choose a single model
> for inference from 100s or 1000s of models based on properties of the data.
> Unfortunately, RunInference does not support this use case. I put
> together a proposal for RunInference that allows a single keyed
> RunInference transform to serve a different model for each key. I'd
> appreciate any thoughts or comments!
>
>
> https://docs.google.com/document/d/1kj3FyWRbJu1KhViX07Z0Gk0MU0842jhYRhI-DMhhcv4/edit?usp=sharing
>
> Thanks,
> Danny
>

[RFC] Throttle Time Counters

[Yasha Ravindra via dev]

Hello everyone,

Throttle time counters were introduced to give clients the option to self
regulate when the service is overwhelmed by requests.
Currently, we have different namespaces for each IO and languages. We would
instead like to have a dedicated namespace for this counter.
We have put together a proposal for the same and would appreciate any
feedback or comments.

https://docs.google.com/document/d/1hUufb3L5jURGeFLaQKeQbPlYK-B2wbaLHtqNVETSOOk/edit?usp=sharing


Thank you

Warm Regards,
Yasha Ravindra

Re: Why there are no certifications for Apache Beam

[Ahmet Altay via dev]

Hi Abhishek,

We recently launched a new Beam certificate. You can find more information
here:
https://beam.apache.org/get-started/resources/learning-resources/#getting-started-with-apache-beam-quest.
As part of the launch we also have a code for completing the program for
free (more info here: https://beam.apache.org/blog/beamquest/ -- the code
should be valid until Aug 18).

Hope you find this useful.

Ahmet

On Thu, Jul 27, 2023 at 12:08 PM Abhishek Patre 
wrote:

> Hello Team,
>
> I hope this email finds you well. I was wondering if you could provide
> some clarity on the availability of certification for Apache Beam. I
> apologize if this isn't the appropriate mailing list for this inquiry.
> Having a certification option for Apache Beam would be incredibly
> beneficial in terms of showcasing our skills and expertise.
>
> Thank you for your time and assistance.
>
> Regards
> Abhishek Patre
>

Re: Why there are no certifications for Apache Beam

[Svetak Sundhar via dev]

Hello Abhishek,

Thank you for your interest. We do have a skill badge you can earn with
Apache Beam (https://www.cloudskillsboost.google/quests/310).

You can use this access code
 (to
earn this for free), search "Getting Started with Apache Beam", and then
complete the labs. Please reach out if you have any questions or run into
any blockers.

Thanks,


Svetak Sundhar

  Data Engineer
s vetaksund...@google.com



On Thu, Jul 27, 2023 at 3:08 PM Abhishek Patre 
wrote:

> Hello Team,
>
> I hope this email finds you well. I was wondering if you could provide
> some clarity on the availability of certification for Apache Beam. I
> apologize if this isn't the appropriate mailing list for this inquiry.
> Having a certification option for Apache Beam would be incredibly
> beneficial in terms of showcasing our skills and expertise.
>
> Thank you for your time and assistance.
>
> Regards
> Abhishek Patre
>

Why there are no certifications for Apache Beam

[Abhishek Patre]

Hello Team,

I hope this email finds you well. I was wondering if you could provide some
clarity on the availability of certification for Apache Beam. I apologize
if this isn't the appropriate mailing list for this inquiry. Having a
certification option for Apache Beam would be incredibly beneficial in
terms of showcasing our skills and expertise.

Thank you for your time and assistance.

Regards
Abhishek Patre

Beam High Priority Issue Report (39)

[beamactions]

This is your daily summary of Beam's current high priority issues that may need 
attention.

See https://beam.apache.org/contribute/issue-priorities for the meaning and 
expectations around issue priorities.

Unassigned P1 Issues:

https://github.com/apache/beam/issues/27648 [Bug]: Python SDFs (e.g. 
PeriodicImpulse) running in Flink and polling using tracker.defer_remainder 
have checkpoint size growing indefinitely 
https://github.com/apache/beam/issues/27616 [Bug]: Unable to use 
applyRowMutations() in bigquery IO apache beam java
https://github.com/apache/beam/issues/27486 [Bug]: Read from datastore with 
inequality filters
https://github.com/apache/beam/issues/27314 [Failing Test]: 
bigquery.StorageApiSinkCreateIfNeededIT.testCreateManyTables[1]
https://github.com/apache/beam/issues/27312 [Bug]: JmsIO create connection 
based on the number of threads
https://github.com/apache/beam/issues/27238 [Bug]: Window trigger has lag when 
using Kafka and GroupByKey on Dataflow Runner
https://github.com/apache/beam/issues/26981 [Bug]: Getting an error related to 
SchemaCoder after upgrading to 2.48
https://github.com/apache/beam/issues/26969 [Failing Test]: Python PostCommit 
is failing due to exceeded rate limits
https://github.com/apache/beam/issues/26911 [Bug]: UNNEST ARRAY with a nested 
ROW (described below)
https://github.com/apache/beam/issues/26354 [Bug]: BigQueryIO direct read not 
reading all rows when set --setEnableBundling=true
https://github.com/apache/beam/issues/26343 [Bug]: 
apache_beam.io.gcp.bigquery_read_it_test.ReadAllBQTests.test_read_queries is 
flaky
https://github.com/apache/beam/issues/26329 [Bug]: BigQuerySourceBase does not 
propagate a Coder to AvroSource
https://github.com/apache/beam/issues/26041 [Bug]: Unable to create 
exactly-once Flink pipeline with stream source and file sink
https://github.com/apache/beam/issues/25975 [Bug]: Reducing parallelism in 
FlinkRunner leads to a data loss
https://github.com/apache/beam/issues/24776 [Bug]: Race condition in Python SDK 
Harness ProcessBundleProgress
https://github.com/apache/beam/issues/24389 [Failing Test]: 
HadoopFormatIOElasticTest.classMethod ExceptionInInitializerError 
ContainerFetchException
https://github.com/apache/beam/issues/24313 [Flaky]: 
apache_beam/runners/portability/portable_runner_test.py::PortableRunnerTestWithSubprocesses::test_pardo_state_with_custom_key_coder
https://github.com/apache/beam/issues/23944  beam_PreCommit_Python_Cron 
regularily failing - test_pardo_large_input flaky
https://github.com/apache/beam/issues/23709 [Flake]: Spark batch flakes in 
ParDoLifecycleTest.testTeardownCalledAfterExceptionInProcessElement and 
ParDoLifecycleTest.testTeardownCalledAfterExceptionInStartBundle
https://github.com/apache/beam/issues/23525 [Bug]: Default PubsubMessage coder 
will drop message id and orderingKey
https://github.com/apache/beam/issues/22913 [Bug]: 
beam_PostCommit_Java_ValidatesRunner_Flink is flakes in 
org.apache.beam.sdk.transforms.GroupByKeyTest$BasicTests.testAfterProcessingTimeContinuationTriggerUsingState
https://github.com/apache/beam/issues/22605 [Bug]: Beam Python failure for 
dataflow_exercise_metrics_pipeline_test.ExerciseMetricsPipelineTest.test_metrics_it
https://github.com/apache/beam/issues/21714 
PulsarIOTest.testReadFromSimpleTopic is very flaky
https://github.com/apache/beam/issues/21708 beam_PostCommit_Java_DataflowV2, 
testBigQueryStorageWrite30MProto failing consistently
https://github.com/apache/beam/issues/21706 Flaky timeout in github Python unit 
test action 
StatefulDoFnOnDirectRunnerTest.test_dynamic_timer_clear_then_set_timer
https://github.com/apache/beam/issues/21643 FnRunnerTest with non-trivial 
(order 1000 elements) numpy input flakes in non-cython environment
https://github.com/apache/beam/issues/21476 WriteToBigQuery Dynamic table 
destinations returns wrong tableId
https://github.com/apache/beam/issues/21469 beam_PostCommit_XVR_Flink flaky: 
Connection refused
https://github.com/apache/beam/issues/21424 Java VR (Dataflow, V2, Streaming) 
failing: ParDoTest$TimestampTests/OnWindowExpirationTests
https://github.com/apache/beam/issues/21262 Python AfterAny, AfterAll do not 
follow spec
https://github.com/apache/beam/issues/21260 Python DirectRunner does not emit 
data at GC time
https://github.com/apache/beam/issues/21121 
apache_beam.examples.streaming_wordcount_it_test.StreamingWordCountIT.test_streaming_wordcount_it
 flakey
https://github.com/apache/beam/issues/21104 Flaky: 
apache_beam.runners.portability.fn_api_runner.fn_runner_test.FnApiRunnerTestWithGrpcAndMultiWorkers
https://github.com/apache/beam/issues/20976 
apache_beam.runners.portability.flink_runner_test.FlinkRunnerTestOptimized.test_flink_metrics
 is flaky
https://github.com/apache/beam/issues/20108 Python direct runner doesn't emit 
empty pane when it should
https://github.com/apache/beam/issues/19814 Flink streaming flakes in 
ParDoLifecycleTest.testTeardownCalledAfterExceptionInStartBundleStateful