Asgarde: Error Handling for Beam?

[Austin Bennett]

Hi Beam Devs,

@Mazlum  was suggested
to consider donating Asgarde  to
Beam for Java/Kotlin error handling to Beam [ see:
https://2022.beamsummit.org/sessions/error-handling-asgarde/ for last
year's Beam Summit talk ], he is also the author of Pasgard
e [ for Python ] and Milgard [ for a
simplified Kotlin API ].

Would Asgarde be a good contribution, something the Beam community would be
willing to accept?  I imagine we might want it to live at
github.com/apache/beam-asgarde ?  Or perhaps there is a good place in
github.com/apache/beam ??

Especially once/if officially part of Beam, I imagine we'd add follow-up
items like getting onto the website/docs, and related.

Cheers,
Austin


P.S.  This might warrant separate/additional conversations for his other
libraries, but let's focus any discussion on Asgarde for now?

Re: [beam-starter-typescript]: Missing place to create issue

[Austin Bennett]

A few additional thoughts:

*  @Anyone --> Should each starter repo allow issues?  Or, better to file
issues in https://github.com/apache/beam/issues ?

* @david-kh...@hotmail.com -- I'd say in general PRs are likely welcome.

* Seems like Contributing.md should get updated.  The text linking to
issues, actually takes people to PRs.



On Wed, Jun 14, 2023 at 10:26 AM Kerry Donny-Clark via dev <
dev@beam.apache.org> wrote:

> Jack may also be able to help you create an issue.
> Kerry
>
> On Wed, Jun 14, 2023, 1:09 PM XQ Hu via dev  wrote:
>
>> I believe Robert is the owner for that project.
>>
>> On Mon, Jun 12, 2023 at 11:30 PM david-kh...@hotmail.com <
>> david-kh...@hotmail.com> wrote:
>>
>>> Hi Beam community,
>>>
>>>
>>>
>>> I am David and new to the community. After tried to tweak some code from
>>> beam-starter-ts, I have found some issues and want to raise. But there is
>>> no way I can create an Github issue in the same project
>>>
>>> apache/beam-starter-typescript: Apache beam (github.com)
>>> .
>>>
>>>
>>>
>>> I also double check the Contribute.md and get no idea still.
>>>
>>>
>>>
>>> Would you mind guide me to the right path?
>>>
>>>
>>>
>>> Regards,
>>>
>>> David L.
>>>
>>

Re: [beam-starter-typescript]: Missing place to create issue

[Kerry Donny-Clark via dev]

Jack may also be able to help you create an issue.
Kerry

On Wed, Jun 14, 2023, 1:09 PM XQ Hu via dev  wrote:

> I believe Robert is the owner for that project.
>
> On Mon, Jun 12, 2023 at 11:30 PM david-kh...@hotmail.com <
> david-kh...@hotmail.com> wrote:
>
>> Hi Beam community,
>>
>>
>>
>> I am David and new to the community. After tried to tweak some code from
>> beam-starter-ts, I have found some issues and want to raise. But there is
>> no way I can create an Github issue in the same project
>>
>> apache/beam-starter-typescript: Apache beam (github.com)
>> .
>>
>>
>>
>> I also double check the Contribute.md and get no idea still.
>>
>>
>>
>> Would you mind guide me to the right path?
>>
>>
>>
>> Regards,
>>
>> David L.
>>
>

Re: [beam-starter-typescript]: Missing place to create issue

[XQ Hu via dev]

I believe Robert is the owner for that project.

On Mon, Jun 12, 2023 at 11:30 PM david-kh...@hotmail.com <
david-kh...@hotmail.com> wrote:

> Hi Beam community,
>
>
>
> I am David and new to the community. After tried to tweak some code from
> beam-starter-ts, I have found some issues and want to raise. But there is
> no way I can create an Github issue in the same project
>
> apache/beam-starter-typescript: Apache beam (github.com)
> .
>
>
>
> I also double check the Contribute.md and get no idea still.
>
>
>
> Would you mind guide me to the right path?
>
>
>
> Regards,
>
> David L.
>

Re: Calcite to Beam mapping, possible?

[deepak kumar]

I would be really interested to contribute as well on calcite to PTransform
kind of conversions

On Wed, Jun 14, 2023 at 7:03 PM Byron Ellis via dev 
wrote:

> Any particular reason you're not using the SqlTransform to do that? That'd
> probably be a good place to start code-wise (perhaps with something a bit
> newer than 2.29) to see how Beam uses Calcite today to produce a PTransform
> from a query plan.
>
> On Wed, Jun 14, 2023 at 5:32 AM Soumyadeep Mukhopadhyay <
> soumyamy...@gmail.com> wrote:
>
>> Hello All,
>>
>> I have a use case where I need Apache Calcite to parse, validate the
>> query and then apply some planner rules.
>>
>> I was hoping to leverage the capabilities of the Apache Beam on top of
>> Calcite so that I can build on top of what we have already done, and apply
>> it at scale. One approach that came to mind was to either convert the
>> RelNode to something equivalent in Beam (like PTransform maybe?
>> https://beam.apache.org/releases/javadoc/2.29.0/org/apache/beam/sdk/extensions/sql/impl/rel/BeamRelNode.html
>> ).
>>
>> Any opinion/suggestion on this will be really appreciated! Thanks for
>> your time and consideration! If you feel this is not intended forum for me
>> to ask, please let me know! :)
>>
>> With regards,
>> Soumyadeep Mukhopadhyay.
>>
>

Re: Calcite to Beam mapping, possible?

[Byron Ellis via dev]

Any particular reason you're not using the SqlTransform to do that? That'd
probably be a good place to start code-wise (perhaps with something a bit
newer than 2.29) to see how Beam uses Calcite today to produce a PTransform
from a query plan.

On Wed, Jun 14, 2023 at 5:32 AM Soumyadeep Mukhopadhyay <
soumyamy...@gmail.com> wrote:

> Hello All,
>
> I have a use case where I need Apache Calcite to parse, validate the query
> and then apply some planner rules.
>
> I was hoping to leverage the capabilities of the Apache Beam on top of
> Calcite so that I can build on top of what we have already done, and apply
> it at scale. One approach that came to mind was to either convert the
> RelNode to something equivalent in Beam (like PTransform maybe?
> https://beam.apache.org/releases/javadoc/2.29.0/org/apache/beam/sdk/extensions/sql/impl/rel/BeamRelNode.html
> ).
>
> Any opinion/suggestion on this will be really appreciated! Thanks for your
> time and consideration! If you feel this is not intended forum for me to
> ask, please let me know! :)
>
> With regards,
> Soumyadeep Mukhopadhyay.
>

Beam High Priority Issue Report (38)

[beamactions]

This is your daily summary of Beam's current high priority issues that may need 
attention.

See https://beam.apache.org/contribute/issue-priorities for the meaning and 
expectations around issue priorities.

Unassigned P1 Issues:

https://github.com/apache/beam/issues/27019 [Failing Test]: Azure Integration 
test is failing in python 3.7 PostCommit
https://github.com/apache/beam/issues/27012 [Bug]: Beam Website cannot run 
locally on Mac
https://github.com/apache/beam/issues/26981 [Bug]: Getting an error related to 
SchemaCoder after upgrading to 2.48
https://github.com/apache/beam/issues/26969 [Failing Test]: Python PostCommit 
is failing due to exceeded rate limits
https://github.com/apache/beam/issues/26911 [Bug]: UNNEST ARRAY with a nested 
ROW (described below)
https://github.com/apache/beam/issues/26547 [Failing Test]: 
beam_PostCommit_Java_DataflowV2
https://github.com/apache/beam/issues/26354 [Bug]: BigQueryIO direct read not 
reading all rows when set --setEnableBundling=true
https://github.com/apache/beam/issues/26343 [Bug]: 
apache_beam.io.gcp.bigquery_read_it_test.ReadAllBQTests.test_read_queries is 
flaky
https://github.com/apache/beam/issues/26329 [Bug]: BigQuerySourceBase does not 
propagate a Coder to AvroSource
https://github.com/apache/beam/issues/26272 [Failing Test]: Python 3.7 
postcommit is red
https://github.com/apache/beam/issues/26041 [Bug]: Unable to create 
exactly-once Flink pipeline with stream source and file sink
https://github.com/apache/beam/issues/25975 [Bug]: Reducing parallelism in 
FlinkRunner leads to a data loss
https://github.com/apache/beam/issues/24776 [Bug]: Race condition in Python SDK 
Harness ProcessBundleProgress
https://github.com/apache/beam/issues/24389 [Failing Test]: 
HadoopFormatIOElasticTest.classMethod ExceptionInInitializerError 
ContainerFetchException
https://github.com/apache/beam/issues/24313 [Flaky]: 
apache_beam/runners/portability/portable_runner_test.py::PortableRunnerTestWithSubprocesses::test_pardo_state_with_custom_key_coder
https://github.com/apache/beam/issues/23944  beam_PreCommit_Python_Cron 
regularily failing - test_pardo_large_input flaky
https://github.com/apache/beam/issues/23709 [Flake]: Spark batch flakes in 
ParDoLifecycleTest.testTeardownCalledAfterExceptionInProcessElement and 
ParDoLifecycleTest.testTeardownCalledAfterExceptionInStartBundle
https://github.com/apache/beam/issues/22913 [Bug]: 
beam_PostCommit_Java_ValidatesRunner_Flink is flakes in 
org.apache.beam.sdk.transforms.GroupByKeyTest$BasicTests.testAfterProcessingTimeContinuationTriggerUsingState
https://github.com/apache/beam/issues/22605 [Bug]: Beam Python failure for 
dataflow_exercise_metrics_pipeline_test.ExerciseMetricsPipelineTest.test_metrics_it
https://github.com/apache/beam/issues/21714 
PulsarIOTest.testReadFromSimpleTopic is very flaky
https://github.com/apache/beam/issues/21708 beam_PostCommit_Java_DataflowV2, 
testBigQueryStorageWrite30MProto failing consistently
https://github.com/apache/beam/issues/21706 Flaky timeout in github Python unit 
test action 
StatefulDoFnOnDirectRunnerTest.test_dynamic_timer_clear_then_set_timer
https://github.com/apache/beam/issues/21643 FnRunnerTest with non-trivial 
(order 1000 elements) numpy input flakes in non-cython environment
https://github.com/apache/beam/issues/21476 WriteToBigQuery Dynamic table 
destinations returns wrong tableId
https://github.com/apache/beam/issues/21469 beam_PostCommit_XVR_Flink flaky: 
Connection refused
https://github.com/apache/beam/issues/21424 Java VR (Dataflow, V2, Streaming) 
failing: ParDoTest$TimestampTests/OnWindowExpirationTests
https://github.com/apache/beam/issues/21262 Python AfterAny, AfterAll do not 
follow spec
https://github.com/apache/beam/issues/21260 Python DirectRunner does not emit 
data at GC time
https://github.com/apache/beam/issues/21121 
apache_beam.examples.streaming_wordcount_it_test.StreamingWordCountIT.test_streaming_wordcount_it
 flakey
https://github.com/apache/beam/issues/21104 Flaky: 
apache_beam.runners.portability.fn_api_runner.fn_runner_test.FnApiRunnerTestWithGrpcAndMultiWorkers
https://github.com/apache/beam/issues/20976 
apache_beam.runners.portability.flink_runner_test.FlinkRunnerTestOptimized.test_flink_metrics
 is flaky
https://github.com/apache/beam/issues/20108 Python direct runner doesn't emit 
empty pane when it should
https://github.com/apache/beam/issues/19814 Flink streaming flakes in 
ParDoLifecycleTest.testTeardownCalledAfterExceptionInStartBundleStateful and 
ParDoLifecycleTest.testTeardownCalledAfterExceptionInProcessElementStateful
https://github.com/apache/beam/issues/19465 Explore possibilities to lower 
in-use IP address quota footprint.


P1 Issues with no update in the last week:

https://github.com/apache/beam/issues/26902 [Bug]: Images built not saved in 
the local image store
https://github.com/apache/beam/issues/26723 [Failing Test]: Tour of Beam 
Frontend Test suite is perma-red on master

Calcite to Beam mapping, possible?

[Soumyadeep Mukhopadhyay]

Hello All,

I have a use case where I need Apache Calcite to parse, validate the query
and then apply some planner rules.

I was hoping to leverage the capabilities of the Apache Beam on top of
Calcite so that I can build on top of what we have already done, and apply
it at scale. One approach that came to mind was to either convert the
RelNode to something equivalent in Beam (like PTransform maybe?
https://beam.apache.org/releases/javadoc/2.29.0/org/apache/beam/sdk/extensions/sql/impl/rel/BeamRelNode.html
).

Any opinion/suggestion on this will be really appreciated! Thanks for your
time and consideration! If you feel this is not intended forum for me to
ask, please let me know! :)

With regards,
Soumyadeep Mukhopadhyay.