[jira] [Created] (BEAM-12585) Consider making NullableCoder as a StandardCoder

2021-07-07 Thread Boyuan Zhang (Jira)
Boyuan Zhang created BEAM-12585: --- Summary: Consider making NullableCoder as a StandardCoder Key: BEAM-12585 URL: https://issues.apache.org/jira/browse/BEAM-12585 Project: Beam Issue Type: Impro

[jira] [Updated] (BEAM-12494) Dataflow Kafka Job not triggering for external subnet

2021-07-02 Thread Boyuan Zhang (Jira)
[ https://issues.apache.org/jira/browse/BEAM-12494?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Boyuan Zhang updated BEAM-12494: Resolution: Won't Fix Status: Resolved (was: Open) > Dataflow Kafka Job not triggering for

[jira] [Commented] (BEAM-12494) Dataflow Kafka Job not triggering for external subnet

2021-07-02 Thread Boyuan Zhang (Jira)
[ https://issues.apache.org/jira/browse/BEAM-12494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17373803#comment-17373803 ] Boyuan Zhang commented on BEAM-12494: - I believe this is an intended behavior on runn

[jira] [Created] (BEAM-12570) SDK should close data output stream when it knows current bundle has failed

2021-07-01 Thread Boyuan Zhang (Jira)
Boyuan Zhang created BEAM-12570: --- Summary: SDK should close data output stream when it knows current bundle has failed Key: BEAM-12570 URL: https://issues.apache.org/jira/browse/BEAM-12570 Project: Beam

[jira] [Created] (BEAM-12522) @InitialRestriction and @SplitRestriction should be able to access sideinputs as @ProcessElement

2021-06-21 Thread Boyuan Zhang (Jira)
Boyuan Zhang created BEAM-12522: --- Summary: @InitialRestriction and @SplitRestriction should be able to access sideinputs as @ProcessElement Key: BEAM-12522 URL: https://issues.apache.org/jira/browse/BEAM-12522

[jira] [Commented] (BEAM-12494) Dataflow Kafka Job not triggering for external subnet

2021-06-21 Thread Boyuan Zhang (Jira)
[ https://issues.apache.org/jira/browse/BEAM-12494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17366794#comment-17366794 ] Boyuan Zhang commented on BEAM-12494: - Are you able to ssh to your kafka cluster? The

[jira] [Commented] (BEAM-12494) Dataflow Kafka Job not triggering for external subnet

2021-06-21 Thread Boyuan Zhang (Jira)
[ https://issues.apache.org/jira/browse/BEAM-12494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17366781#comment-17366781 ] Boyuan Zhang commented on BEAM-12494: - Hi, For the 400 bad job request, as I mention

[jira] [Comment Edited] (BEAM-12494) Dataflow Kafka Job not triggering for external subnet

2021-06-16 Thread Boyuan Zhang (Jira)
[ https://issues.apache.org/jira/browse/BEAM-12494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17364519#comment-17364519 ] Boyuan Zhang edited comment on BEAM-12494 at 6/16/21, 9:07 PM:

[jira] [Commented] (BEAM-12494) Dataflow Kafka Job not triggering for external subnet

2021-06-16 Thread Boyuan Zhang (Jira)
[ https://issues.apache.org/jira/browse/BEAM-12494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17364519#comment-17364519 ] Boyuan Zhang commented on BEAM-12494: - Hi Jasminder, Would you like to share the pip

[jira] [Updated] (BEAM-12475) When bundle processors are re-used, do not respond to splits for previous bundles.

2021-06-16 Thread Boyuan Zhang (Jira)
[ https://issues.apache.org/jira/browse/BEAM-12475?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Boyuan Zhang updated BEAM-12475: Fix Version/s: 2.31.0 > When bundle processors are re-used, do not respond to splits for previous

[jira] [Updated] (BEAM-12475) When bundle processors are re-used, do not respond to splits for previous bundles.

2021-06-16 Thread Boyuan Zhang (Jira)
[ https://issues.apache.org/jira/browse/BEAM-12475?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Boyuan Zhang updated BEAM-12475: Resolution: Fixed Status: Resolved (was: Open) > When bundle processors are re-used, do no

[jira] [Commented] (BEAM-12494) Dataflow Kafka Job not triggering for external subnet

2021-06-16 Thread Boyuan Zhang (Jira)
[ https://issues.apache.org/jira/browse/BEAM-12494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17364367#comment-17364367 ] Boyuan Zhang commented on BEAM-12494: - And this issue will be gone if you are using r

[jira] [Comment Edited] (BEAM-12494) Dataflow Kafka Job not triggering for external subnet

2021-06-15 Thread Boyuan Zhang (Jira)
[ https://issues.apache.org/jira/browse/BEAM-12494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17363994#comment-17363994 ] Boyuan Zhang edited comment on BEAM-12494 at 6/16/21, 12:39 AM: ---

[jira] [Commented] (BEAM-12494) Dataflow Kafka Job not triggering for external subnet

2021-06-15 Thread Boyuan Zhang (Jira)
[ https://issues.apache.org/jira/browse/BEAM-12494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17363994#comment-17363994 ] Boyuan Zhang commented on BEAM-12494: - Hi Jasminder, You investigation is correct an

[jira] [Updated] (BEAM-12459) Watch does not properly advance the watermark by default

2021-06-08 Thread Boyuan Zhang (Jira)
[ https://issues.apache.org/jira/browse/BEAM-12459?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Boyuan Zhang updated BEAM-12459: Priority: P2 (was: P1) > Watch does not properly advance the watermark by default > --

[jira] [Created] (BEAM-12458) Using SubscriberOptions.setPartitions results in pipeline construction error

2021-06-07 Thread Boyuan Zhang (Jira)
Boyuan Zhang created BEAM-12458: --- Summary: Using SubscriberOptions.setPartitions results in pipeline construction error Key: BEAM-12458 URL: https://issues.apache.org/jira/browse/BEAM-12458 Project: Bea

[jira] [Commented] (BEAM-8218) Implement Apache PulsarIO

2021-06-04 Thread Boyuan Zhang (Jira)
[ https://issues.apache.org/jira/browse/BEAM-8218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17357542#comment-17357542 ] Boyuan Zhang commented on BEAM-8218: I don't think there is anyone working on this rig

[jira] [Commented] (BEAM-12384) Read.Bounded typeDescriptor is lost after move to SDF

2021-05-21 Thread Boyuan Zhang (Jira)
[ https://issues.apache.org/jira/browse/BEAM-12384?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17349403#comment-17349403 ] Boyuan Zhang commented on BEAM-12384: - Hi Ismael, would you like to elaborate more on

[jira] [Commented] (BEAM-12353) org.apache.beam.sdk.transforms.GroupByKeyTest$WindowTests.testRewindowWithTimestampCombiner is failng on Dataflow runner v2

2021-05-20 Thread Boyuan Zhang (Jira)
[ https://issues.apache.org/jira/browse/BEAM-12353?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17348613#comment-17348613 ] Boyuan Zhang commented on BEAM-12353: - No, the Dataflow validates runner suite doesn'

[jira] [Created] (BEAM-12358) PR14802 breaks beam_PostCommit_Java_ValidatesRunner_Samza

2021-05-18 Thread Boyuan Zhang (Jira)
Boyuan Zhang created BEAM-12358: --- Summary: PR14802 breaks beam_PostCommit_Java_ValidatesRunner_Samza Key: BEAM-12358 URL: https://issues.apache.org/jira/browse/BEAM-12358 Project: Beam Issue Ty

[jira] [Commented] (BEAM-12353) org.apache.beam.sdk.transforms.GroupByKeyTest$WindowTests.testRewindowWithTimestampCombiner is failng on Dataflow runner v2

2021-05-18 Thread Boyuan Zhang (Jira)
[ https://issues.apache.org/jira/browse/BEAM-12353?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17347093#comment-17347093 ] Boyuan Zhang commented on BEAM-12353: - This test is failed consistently on runner v2

[jira] [Updated] (BEAM-12354) :runners:direct-java:runMobileGamingJavaDirect fails on beam_PostRelease_NightlySnapshot for several times

2021-05-17 Thread Boyuan Zhang (Jira)
[ https://issues.apache.org/jira/browse/BEAM-12354?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Boyuan Zhang updated BEAM-12354: Description: https://ci-beam.apache.org/job/beam_PostRelease_NightlySnapshot is failing on :runner

[jira] [Created] (BEAM-12354) :runners:direct-java:runMobileGamingJavaDirect fails on beam_PostRelease_NightlySnapshot for several times

2021-05-17 Thread Boyuan Zhang (Jira)
Boyuan Zhang created BEAM-12354: --- Summary: :runners:direct-java:runMobileGamingJavaDirect fails on beam_PostRelease_NightlySnapshot for several times Key: BEAM-12354 URL: https://issues.apache.org/jira/browse/BEAM-

[jira] [Created] (BEAM-12353) org.apache.beam.sdk.transforms.GroupByKeyTest$WindowTests.testRewindowWithTimestampCombiner is failng on Dataflow runner v2

2021-05-17 Thread Boyuan Zhang (Jira)
Boyuan Zhang created BEAM-12353: --- Summary: org.apache.beam.sdk.transforms.GroupByKeyTest$WindowTests.testRewindowWithTimestampCombiner is failng on Dataflow runner v2 Key: BEAM-12353 URL: https://issues.apache.org/

[jira] [Created] (BEAM-12352) GcsIOIntegrationTest.test_copy_rewrite_token and GcsIOIntegrationTest.test_copy_batch_rewrite_token starts to fail from 05/15

2021-05-17 Thread Boyuan Zhang (Jira)
Boyuan Zhang created BEAM-12352: --- Summary: GcsIOIntegrationTest.test_copy_rewrite_token and GcsIOIntegrationTest.test_copy_batch_rewrite_token starts to fail from 05/15 Key: BEAM-12352 URL: https://issues.apache.org

[jira] [Updated] (BEAM-10861) Adds URNs and payloads to PubSub transforms

2021-05-17 Thread Boyuan Zhang (Jira)
[ https://issues.apache.org/jira/browse/BEAM-10861?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Boyuan Zhang updated BEAM-10861: Fix Version/s: 2.29.0 Resolution: Fixed Status: Resolved (was: Open) > Adds URNs

[jira] [Created] (BEAM-12336) JmsIO should allow using custom timestamp policy to track watermark

2021-05-13 Thread Boyuan Zhang (Jira)
Boyuan Zhang created BEAM-12336: --- Summary: JmsIO should allow using custom timestamp policy to track watermark Key: BEAM-12336 URL: https://issues.apache.org/jira/browse/BEAM-12336 Project: Beam

[jira] [Created] (BEAM-12335) Apply basic fusion to Java DirectRunner to avoid keeping all intermittent results in memory

2021-05-13 Thread Boyuan Zhang (Jira)
Boyuan Zhang created BEAM-12335: --- Summary: Apply basic fusion to Java DirectRunner to avoid keeping all intermittent results in memory Key: BEAM-12335 URL: https://issues.apache.org/jira/browse/BEAM-12335

[jira] [Commented] (BEAM-10670) Make non-portable Splittable DoFn the only option when executing Java "Read" transforms

2021-05-12 Thread Boyuan Zhang (Jira)
[ https://issues.apache.org/jira/browse/BEAM-10670?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17343398#comment-17343398 ] Boyuan Zhang commented on BEAM-10670: - I can help this. I believe at this moment, the

[jira] [Commented] (BEAM-12287) beam_PerformanceTests_Kafka_IO failing due to :sdks:java:container:pullLicenses failure

2021-05-11 Thread Boyuan Zhang (Jira)
[ https://issues.apache.org/jira/browse/BEAM-12287?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17342931#comment-17342931 ] Boyuan Zhang commented on BEAM-12287: - I can confirm that https://github.com/apache/b

[jira] [Updated] (BEAM-12114) Eliminate beam_fn_api from KafkaIO expansion

2021-05-11 Thread Boyuan Zhang (Jira)
[ https://issues.apache.org/jira/browse/BEAM-12114?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Boyuan Zhang updated BEAM-12114: Fix Version/s: 2.30.0 Resolution: Fixed Status: Resolved (was: Open) > Eliminate

[jira] [Commented] (BEAM-11998) Portable runners should be able to issue checkpoints to Splittable DoFn

2021-05-10 Thread Boyuan Zhang (Jira)
[ https://issues.apache.org/jira/browse/BEAM-11998?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17342150#comment-17342150 ] Boyuan Zhang commented on BEAM-11998: - Hi [~zhao1116], I'm happy to help you on this.

[jira] [Commented] (BEAM-6466) KafkaIO doesn't commit offsets while being used as bounded source

2021-05-10 Thread Boyuan Zhang (Jira)
[ https://issues.apache.org/jira/browse/BEAM-6466?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17342133#comment-17342133 ] Boyuan Zhang commented on BEAM-6466: The unbounded-as-bounded-read goes different expa

[jira] [Commented] (BEAM-11828) JmsIO is not acknowledging messages correctly

2021-04-27 Thread Boyuan Zhang (Jira)
[ https://issues.apache.org/jira/browse/BEAM-11828?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17333402#comment-17333402 ] Boyuan Zhang commented on BEAM-11828: - Talking to one customer regarding to the same

[jira] [Commented] (BEAM-11998) Portable runners should be able to issue checkpoints to Splittable DoFn

2021-04-27 Thread Boyuan Zhang (Jira)
[ https://issues.apache.org/jira/browse/BEAM-11998?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=1750#comment-1750 ] Boyuan Zhang commented on BEAM-11998: - You can always use --experiments=use_deprecate

[jira] [Created] (BEAM-12213) Dataflow should always create v1b3 Steps in runner_v1 flavor

2021-04-22 Thread Boyuan Zhang (Jira)
Boyuan Zhang created BEAM-12213: --- Summary: Dataflow should always create v1b3 Steps in runner_v1 flavor Key: BEAM-12213 URL: https://issues.apache.org/jira/browse/BEAM-12213 Project: Beam Issu

[jira] [Commented] (BEAM-12196) Apache Beam Kafka Source Connector Idle Partition Issue with “CustomTimeStampPolicyWithLimitedDelay”

2021-04-20 Thread Boyuan Zhang (Jira)
[ https://issues.apache.org/jira/browse/BEAM-12196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17326258#comment-17326258 ] Boyuan Zhang commented on BEAM-12196: - Thanks, Jay. And which API are you talking ab

[jira] [Commented] (BEAM-12196) Apache Beam Kafka Source Connector Idle Partition Issue with “CustomTimeStampPolicyWithLimitedDelay”

2021-04-20 Thread Boyuan Zhang (Jira)
[ https://issues.apache.org/jira/browse/BEAM-12196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17326054#comment-17326054 ] Boyuan Zhang commented on BEAM-12196: - Hi Jay, Can you try --experiments=use_depreca

[jira] [Created] (BEAM-12193) WatchKafkaTopicPartitionDoFn reports user counter to indicate which TopicPartition has been emitted to downstream

2021-04-19 Thread Boyuan Zhang (Jira)
Boyuan Zhang created BEAM-12193: --- Summary: WatchKafkaTopicPartitionDoFn reports user counter to indicate which TopicPartition has been emitted to downstream Key: BEAM-12193 URL: https://issues.apache.org/jira/browse

[jira] [Created] (BEAM-12192) WatchKafkaTopicPartitionDoFn should respect given topic from KafkaIO

2021-04-19 Thread Boyuan Zhang (Jira)
Boyuan Zhang created BEAM-12192: --- Summary: WatchKafkaTopicPartitionDoFn should respect given topic from KafkaIO Key: BEAM-12192 URL: https://issues.apache.org/jira/browse/BEAM-12192 Project: Beam

[jira] [Comment Edited] (BEAM-12162) Setup tpcds benchmark for dataflow runner v2

2021-04-13 Thread Boyuan Zhang (Jira)
[ https://issues.apache.org/jira/browse/BEAM-12162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17320468#comment-17320468 ] Boyuan Zhang edited comment on BEAM-12162 at 4/13/21, 6:54 PM:

[jira] [Commented] (BEAM-12162) Setup tpcds benchmark for dataflow runner v2

2021-04-13 Thread Boyuan Zhang (Jira)
[ https://issues.apache.org/jira/browse/BEAM-12162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17320468#comment-17320468 ] Boyuan Zhang commented on BEAM-12162: - Based on the build file, it only runs on java

[jira] [Assigned] (BEAM-12162) Setup tpcds benchmark for dataflow runner v2

2021-04-13 Thread Boyuan Zhang (Jira)
[ https://issues.apache.org/jira/browse/BEAM-12162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Boyuan Zhang reassigned BEAM-12162: --- Assignee: Rui Wang > Setup tpcds benchmark for dataflow runner v2 >

[jira] [Created] (BEAM-12162) Setup tpcds benchmark for dataflow runner v2

2021-04-13 Thread Boyuan Zhang (Jira)
Boyuan Zhang created BEAM-12162: --- Summary: Setup tpcds benchmark for dataflow runner v2 Key: BEAM-12162 URL: https://issues.apache.org/jira/browse/BEAM-12162 Project: Beam Issue Type: Task

[jira] [Updated] (BEAM-12162) Setup tpcds benchmark for dataflow runner v2

2021-04-13 Thread Boyuan Zhang (Jira)
[ https://issues.apache.org/jira/browse/BEAM-12162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Boyuan Zhang updated BEAM-12162: Component/s: runner-dataflow > Setup tpcds benchmark for dataflow runner v2 > -

[jira] [Created] (BEAM-12160) Please fix errorprone, checkstyle and lint warnings for tpcds module

2021-04-12 Thread Boyuan Zhang (Jira)
Boyuan Zhang created BEAM-12160: --- Summary: Please fix errorprone, checkstyle and lint warnings for tpcds module Key: BEAM-12160 URL: https://issues.apache.org/jira/browse/BEAM-12160 Project: Beam

[jira] [Commented] (BEAM-11148) Kafka commitOffsetsInFinalize OOM on Flink

2021-04-09 Thread Boyuan Zhang (Jira)
[ https://issues.apache.org/jira/browse/BEAM-11148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17318179#comment-17318179 ] Boyuan Zhang commented on BEAM-11148: - Workaround: Using --experiments="beam_fn_api_u

[jira] [Created] (BEAM-12114) Eliminate beam_fn_api from KafkaIO expansion

2021-04-06 Thread Boyuan Zhang (Jira)
Boyuan Zhang created BEAM-12114: --- Summary: Eliminate beam_fn_api from KafkaIO expansion Key: BEAM-12114 URL: https://issues.apache.org/jira/browse/BEAM-12114 Project: Beam Issue Type: Improveme

[jira] [Updated] (BEAM-10663) CrossLanguageKafkaIOTest broken on Flink Runner

2021-04-02 Thread Boyuan Zhang (Jira)
[ https://issues.apache.org/jira/browse/BEAM-10663?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Boyuan Zhang updated BEAM-10663: Resolution: Fixed Status: Resolved (was: Open) > CrossLanguageKafkaIOTest broken on Flink

[jira] [Commented] (BEAM-10663) CrossLanguageKafkaIOTest broken on Flink Runner

2021-04-02 Thread Boyuan Zhang (Jira)
[ https://issues.apache.org/jira/browse/BEAM-10663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17314000#comment-17314000 ] Boyuan Zhang commented on BEAM-10663: - Sorry that I made a mistake on the KafkaIO exp

[jira] [Commented] (BEAM-8218) Implement Apache PulsarIO

2021-03-30 Thread Boyuan Zhang (Jira)
[ https://issues.apache.org/jira/browse/BEAM-8218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17311852#comment-17311852 ] Boyuan Zhang commented on BEAM-8218: Hi [~mxm], Do you happen to make some progress

[jira] [Commented] (BEAM-11996) Implement SpannerIO on top of Splittable DoFn

2021-03-30 Thread Boyuan Zhang (Jira)
[ https://issues.apache.org/jira/browse/BEAM-11996?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17311843#comment-17311843 ] Boyuan Zhang commented on BEAM-11996: - h3. Goal Convert [BatchSpannerRead|https://gi

[jira] [Commented] (BEAM-11997) Implement RedisIO on top of Splittable DoFn

2021-03-30 Thread Boyuan Zhang (Jira)
[ https://issues.apache.org/jira/browse/BEAM-11997?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17311809#comment-17311809 ] Boyuan Zhang commented on BEAM-11997: - h3. Goal Convert [ReadFn|https://github.com/a

[jira] [Commented] (BEAM-11995) Implement FileIO/TextIO on top of Splittable DoFn

2021-03-30 Thread Boyuan Zhang (Jira)
[ https://issues.apache.org/jira/browse/BEAM-11995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17311797#comment-17311797 ] Boyuan Zhang commented on BEAM-11995: - h3. Goal Current [FileIO|https://github.com/

[jira] [Updated] (BEAM-11995) Implement FileIO/TextIO on top of Splittable DoFn

2021-03-30 Thread Boyuan Zhang (Jira)
[ https://issues.apache.org/jira/browse/BEAM-11995?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Boyuan Zhang updated BEAM-11995: Summary: Implement FileIO/TextIO on top of Splittable DoFn (was: Implement FileIO on top of Splitt

[jira] [Commented] (BEAM-11998) Portable runners should be able to issue checkpoints to Splittable DoFn

2021-03-30 Thread Boyuan Zhang (Jira)
[ https://issues.apache.org/jira/browse/BEAM-11998?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17311703#comment-17311703 ] Boyuan Zhang commented on BEAM-11998: - The most easy way is to issue ProcessBundleSpl

[jira] [Commented] (BEAM-10995) Java + Universal Local Runner: WindowingTest.testWindowPreservation fails

2021-03-29 Thread Boyuan Zhang (Jira)
[ https://issues.apache.org/jira/browse/BEAM-10995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17310895#comment-17310895 ] Boyuan Zhang commented on BEAM-10995: - No: https://github.com/apache/beam/blob/24312

[jira] [Commented] (BEAM-12008) KafkaIO does not handle null keys

2021-03-29 Thread Boyuan Zhang (Jira)
[ https://issues.apache.org/jira/browse/BEAM-12008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17310850#comment-17310850 ] Boyuan Zhang commented on BEAM-12008: - The issue here is that KafkaRecordCoder doesn'

[jira] [Assigned] (BEAM-12021) PubsubReadIT failures: "Cannot nackAll on persisting checkpoint"

2021-03-19 Thread Boyuan Zhang (Jira)
[ https://issues.apache.org/jira/browse/BEAM-12021?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Boyuan Zhang reassigned BEAM-12021: --- Assignee: Kenneth Knowles (was: Boyuan Zhang) > PubsubReadIT failures: "Cannot nackAll on p

[jira] [Commented] (BEAM-12021) PubsubReadIT failures: "Cannot nackAll on persisting checkpoint"

2021-03-19 Thread Boyuan Zhang (Jira)
[ https://issues.apache.org/jira/browse/BEAM-12021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17305259#comment-17305259 ] Boyuan Zhang commented on BEAM-12021: - https://github.com/apache/beam/pull/14276 shou

[jira] [Comment Edited] (BEAM-10663) CrossLanguageKafkaIOTest broken on Flink Runner

2021-03-17 Thread Boyuan Zhang (Jira)
[ https://issues.apache.org/jira/browse/BEAM-10663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17303613#comment-17303613 ] Boyuan Zhang edited comment on BEAM-10663 at 3/17/21, 5:37 PM:

[jira] [Commented] (BEAM-10663) CrossLanguageKafkaIOTest broken on Flink Runner

2021-03-17 Thread Boyuan Zhang (Jira)
[ https://issues.apache.org/jira/browse/BEAM-10663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17303613#comment-17303613 ] Boyuan Zhang commented on BEAM-10663: - I don't think this test will be broken owing t

[jira] [Commented] (BEAM-11998) Portable runners should be able to issue checkpoints to Splittable DoFn

2021-03-16 Thread Boyuan Zhang (Jira)
[ https://issues.apache.org/jira/browse/BEAM-11998?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17302976#comment-17302976 ] Boyuan Zhang commented on BEAM-11998: - Code pointer for how we executes unbounded Spl

[jira] [Created] (BEAM-11998) Portable runners should be able to issue checkpoints to Splittable DoFn

2021-03-16 Thread Boyuan Zhang (Jira)
Boyuan Zhang created BEAM-11998: --- Summary: Portable runners should be able to issue checkpoints to Splittable DoFn Key: BEAM-11998 URL: https://issues.apache.org/jira/browse/BEAM-11998 Project: Beam

[jira] [Commented] (BEAM-11749) Portable Flink runner skips timers when dynamic timer tags are used

2021-03-16 Thread Boyuan Zhang (Jira)
[ https://issues.apache.org/jira/browse/BEAM-11749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17302950#comment-17302950 ] Boyuan Zhang commented on BEAM-11749: - Hi, we fixed the dynamic timers support in ht

[jira] [Created] (BEAM-11997) Implement RedisIO on top of Splittable DoFn

2021-03-16 Thread Boyuan Zhang (Jira)
Boyuan Zhang created BEAM-11997: --- Summary: Implement RedisIO on top of Splittable DoFn Key: BEAM-11997 URL: https://issues.apache.org/jira/browse/BEAM-11997 Project: Beam Issue Type: Improvemen

[jira] [Created] (BEAM-11996) Implement SpannerIO on top of Splittable DoFn

2021-03-16 Thread Boyuan Zhang (Jira)
Boyuan Zhang created BEAM-11996: --- Summary: Implement SpannerIO on top of Splittable DoFn Key: BEAM-11996 URL: https://issues.apache.org/jira/browse/BEAM-11996 Project: Beam Issue Type: Improvem

[jira] [Created] (BEAM-11995) Implement FileIO on top of Splittable DoFn

2021-03-16 Thread Boyuan Zhang (Jira)
Boyuan Zhang created BEAM-11995: --- Summary: Implement FileIO on top of Splittable DoFn Key: BEAM-11995 URL: https://issues.apache.org/jira/browse/BEAM-11995 Project: Beam Issue Type: Improvement

[jira] [Commented] (BEAM-11991) Python Kafka source not emitting messages for streaming pipelines with Flink Runner

2021-03-16 Thread Boyuan Zhang (Jira)
[ https://issues.apache.org/jira/browse/BEAM-11991?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17302752#comment-17302752 ] Boyuan Zhang commented on BEAM-11991: - Just tried to refresh my memory. I don't thin

[jira] [Commented] (BEAM-11991) Python Kafka source not emitting messages for streaming pipelines with Flink Runner

2021-03-16 Thread Boyuan Zhang (Jira)
[ https://issues.apache.org/jira/browse/BEAM-11991?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17302717#comment-17302717 ] Boyuan Zhang commented on BEAM-11991: - I'm suspecting that the problem pipeline is no

[jira] [Commented] (BEAM-10663) CrossLanguageKafkaIOTest broken on Flink Runner

2021-03-16 Thread Boyuan Zhang (Jira)
[ https://issues.apache.org/jira/browse/BEAM-10663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17302671#comment-17302671 ] Boyuan Zhang commented on BEAM-10663: - I think so. > CrossLanguageKafkaIOTest broke

[jira] [Commented] (BEAM-6868) Flink runner supports Bundle Finalization

2021-03-15 Thread Boyuan Zhang (Jira)
[ https://issues.apache.org/jira/browse/BEAM-6868?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17302161#comment-17302161 ] Boyuan Zhang commented on BEAM-6868: I think the remaining issue here is to support bu

[jira] [Commented] (BEAM-11952) WindowMergingFnRunner unnecessarily keeps previous merged results

2021-03-10 Thread Boyuan Zhang (Jira)
[ https://issues.apache.org/jira/browse/BEAM-11952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17299168#comment-17299168 ] Boyuan Zhang commented on BEAM-11952: - Could you please add more details on which thi

[jira] [Created] (BEAM-11947) KafkaUnboundedReader doesn't support updating pipeline with re-partition case.

2021-03-09 Thread Boyuan Zhang (Jira)
Boyuan Zhang created BEAM-11947: --- Summary: KafkaUnboundedReader doesn't support updating pipeline with re-partition case. Key: BEAM-11947 URL: https://issues.apache.org/jira/browse/BEAM-11947 Project: B

[jira] [Created] (BEAM-11946) Use ReadFromKafkaDoFn for KafkaIO.Read by default when beam_fn_api is enabled

2021-03-09 Thread Boyuan Zhang (Jira)
Boyuan Zhang created BEAM-11946: --- Summary: Use ReadFromKafkaDoFn for KafkaIO.Read by default when beam_fn_api is enabled Key: BEAM-11946 URL: https://issues.apache.org/jira/browse/BEAM-11946 Project: Be

[jira] [Commented] (BEAM-11403) Unbounded SDF wrapper causes performance regression on DirectRunner

2021-03-09 Thread Boyuan Zhang (Jira)
[ https://issues.apache.org/jira/browse/BEAM-11403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17298290#comment-17298290 ] Boyuan Zhang commented on BEAM-11403: - Sorry for the late. One last improvement I ca

[jira] [Commented] (BEAM-11325) KafkaIO should be able to read from new added topic/partition automatically during pipeline execution time

2021-03-09 Thread Boyuan Zhang (Jira)
[ https://issues.apache.org/jira/browse/BEAM-11325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17298285#comment-17298285 ] Boyuan Zhang commented on BEAM-11325: - Missing part is E2E test. I should be able to

[jira] [Assigned] (BEAM-10861) Adds URNs and payloads to PubSub transforms

2021-03-05 Thread Boyuan Zhang (Jira)
[ https://issues.apache.org/jira/browse/BEAM-10861?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Boyuan Zhang reassigned BEAM-10861: --- Assignee: Boyuan Zhang (was: Kenneth Knowles) > Adds URNs and payloads to PubSub transforms

[jira] [Created] (BEAM-11879) SAMPLED_BYTE_SIZE counter should consider recording size per winodw

2021-02-26 Thread Boyuan Zhang (Jira)
Boyuan Zhang created BEAM-11879: --- Summary: SAMPLED_BYTE_SIZE counter should consider recording size per winodw Key: BEAM-11879 URL: https://issues.apache.org/jira/browse/BEAM-11879 Project: Beam

[jira] [Updated] (BEAM-11752) Using LoadingCache instead of Map to cache BundleProcessor

2021-02-08 Thread Boyuan Zhang (Jira)
[ https://issues.apache.org/jira/browse/BEAM-11752?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Boyuan Zhang updated BEAM-11752: Fix Version/s: 2.29.0 Resolution: Fixed Status: Resolved (was: Open) > Using Loa

[jira] [Created] (BEAM-11776) DoFn lifecycle in programming guide should be updated with portable execution

2021-02-08 Thread Boyuan Zhang (Jira)
Boyuan Zhang created BEAM-11776: --- Summary: DoFn lifecycle in programming guide should be updated with portable execution Key: BEAM-11776 URL: https://issues.apache.org/jira/browse/BEAM-11776 Project: Be

[jira] [Created] (BEAM-11752) Using LoadingCache instead of Map to cache BundleProcessor

2021-02-04 Thread Boyuan Zhang (Jira)
Boyuan Zhang created BEAM-11752: --- Summary: Using LoadingCache instead of Map to cache BundleProcessor Key: BEAM-11752 URL: https://issues.apache.org/jira/browse/BEAM-11752 Project: Beam Issue T

[jira] [Created] (BEAM-11734) ReadFromKafkaDoFn should perform seft-checkpoint when there is no offset returned by offsetForTime

2021-02-01 Thread Boyuan Zhang (Jira)
Boyuan Zhang created BEAM-11734: --- Summary: ReadFromKafkaDoFn should perform seft-checkpoint when there is no offset returned by offsetForTime Key: BEAM-11734 URL: https://issues.apache.org/jira/browse/BEAM-11734

[jira] [Updated] (BEAM-10120) Support Dynamic Timers in the Flink Portable Runner

2021-01-28 Thread Boyuan Zhang (Jira)
[ https://issues.apache.org/jira/browse/BEAM-10120?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Boyuan Zhang updated BEAM-10120: Fix Version/s: 2.29.0 Resolution: Fixed Status: Resolved (was: Triage Needed) >

[jira] [Created] (BEAM-11687) Certain exceptions in SDF wrapper should not fail the bundle directly

2021-01-25 Thread Boyuan Zhang (Jira)
Boyuan Zhang created BEAM-11687: --- Summary: Certain exceptions in SDF wrapper should not fail the bundle directly Key: BEAM-11687 URL: https://issues.apache.org/jira/browse/BEAM-11687 Project: Beam

[jira] [Updated] (BEAM-11678) beam_PerformanceTests_Kafka_IO is broken by incorrect docker image cleanup

2021-01-25 Thread Boyuan Zhang (Jira)
[ https://issues.apache.org/jira/browse/BEAM-11678?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Boyuan Zhang updated BEAM-11678: Fix Version/s: 2.28.0 Resolution: Fixed Status: Resolved (was: Triage Needed) >

[jira] [Created] (BEAM-11678) beam_PerformanceTests_Kafka_IO is broken by incorrect docker image cleanup

2021-01-22 Thread Boyuan Zhang (Jira)
Boyuan Zhang created BEAM-11678: --- Summary: beam_PerformanceTests_Kafka_IO is broken by incorrect docker image cleanup Key: BEAM-11678 URL: https://issues.apache.org/jira/browse/BEAM-11678 Project: Beam

[jira] [Updated] (BEAM-11677) Expose commit_offset_in_finalize and timestamp_policy to ReadFromKafka

2021-01-22 Thread Boyuan Zhang (Jira)
[ https://issues.apache.org/jira/browse/BEAM-11677?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Boyuan Zhang updated BEAM-11677: Fix Version/s: 2.28.0 Resolution: Fixed Status: Resolved (was: Open) > Expose co

[jira] [Created] (BEAM-11677) Expose commit_offset_in_finalize and timestamp_policy to ReadFromKafka

2021-01-22 Thread Boyuan Zhang (Jira)
Boyuan Zhang created BEAM-11677: --- Summary: Expose commit_offset_in_finalize and timestamp_policy to ReadFromKafka Key: BEAM-11677 URL: https://issues.apache.org/jira/browse/BEAM-11677 Project: Beam

[jira] [Assigned] (BEAM-10120) Support Dynamic Timers in the Flink Portable Runner

2021-01-21 Thread Boyuan Zhang (Jira)
[ https://issues.apache.org/jira/browse/BEAM-10120?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Boyuan Zhang reassigned BEAM-10120: --- Assignee: Boyuan Zhang > Support Dynamic Timers in the Flink Portable Runner > -

[jira] [Comment Edited] (BEAM-11633) Steer people towards ParDo, SDF, instead of the original Source framework

2021-01-12 Thread Boyuan Zhang (Jira)
[ https://issues.apache.org/jira/browse/BEAM-11633?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17263826#comment-17263826 ] Boyuan Zhang edited comment on BEAM-11633 at 1/13/21, 1:36 AM:

[jira] [Commented] (BEAM-11633) Steer people towards ParDo, SDF, instead of the original Source framework

2021-01-12 Thread Boyuan Zhang (Jira)
[ https://issues.apache.org/jira/browse/BEAM-11633?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17263826#comment-17263826 ] Boyuan Zhang commented on BEAM-11633: - Deprecating source framework is definitely th

[jira] [Commented] (BEAM-11403) Unbounded SDF wrapper causes performance regression on DirectRunner

2021-01-12 Thread Boyuan Zhang (Jira)
[ https://issues.apache.org/jira/browse/BEAM-11403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17263797#comment-17263797 ] Boyuan Zhang commented on BEAM-11403: - Proposed to make the runner-issued checkpoint

[jira] [Comment Edited] (BEAM-11325) KafkaIO should be able to read from new added topic/partition automatically during pipeline execution time

2021-01-07 Thread Boyuan Zhang (Jira)
[ https://issues.apache.org/jira/browse/BEAM-11325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17260976#comment-17260976 ] Boyuan Zhang edited comment on BEAM-11325 at 1/8/21, 3:57 AM: -

[jira] [Commented] (BEAM-11325) KafkaIO should be able to read from new added topic/partition automatically during pipeline execution time

2021-01-07 Thread Boyuan Zhang (Jira)
[ https://issues.apache.org/jira/browse/BEAM-11325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17260976#comment-17260976 ] Boyuan Zhang commented on BEAM-11325: - Started the design: https://github.com/apache

[jira] [Updated] (BEAM-11132) Remove Experiment annotation from SDF API

2021-01-07 Thread Boyuan Zhang (Jira)
[ https://issues.apache.org/jira/browse/BEAM-11132?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Boyuan Zhang updated BEAM-11132: Fix Version/s: 2.26.0 Resolution: Fixed Status: Resolved (was: Open) > Remove Ex

[jira] [Created] (BEAM-11537) PR13598 breaks Java Post Commit

2020-12-29 Thread Boyuan Zhang (Jira)
Boyuan Zhang created BEAM-11537: --- Summary: PR13598 breaks Java Post Commit Key: BEAM-11537 URL: https://issues.apache.org/jira/browse/BEAM-11537 Project: Beam Issue Type: Test Compone

[jira] [Commented] (BEAM-11524) (Deprecated) WriteStringsToPubSub broken on Dataflow

2020-12-24 Thread Boyuan Zhang (Jira)
[ https://issues.apache.org/jira/browse/BEAM-11524?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17254654#comment-17254654 ] Boyuan Zhang commented on BEAM-11524: - Yes. Python doesn't offer custom pubsub soure/

[jira] [Commented] (BEAM-11403) Unbounded SDF wrapper causes performance regression on DirectRunner

2020-12-21 Thread Boyuan Zhang (Jira)
[ https://issues.apache.org/jira/browse/BEAM-11403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17253078#comment-17253078 ] Boyuan Zhang commented on BEAM-11403: - Discussion thread: https://lists.apache.org/t

[jira] [Assigned] (BEAM-11403) Unbounded SDF wrapper causes performance regression on DirectRunner

2020-12-21 Thread Boyuan Zhang (Jira)
[ https://issues.apache.org/jira/browse/BEAM-11403?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Boyuan Zhang reassigned BEAM-11403: --- Assignee: Boyuan Zhang > Unbounded SDF wrapper causes performance regression on DirectRunner

  1   2   3   4   5   >