[jira] [Commented] (BEAM-12942) Dataflow runner specialization of PubsubIO should validate messages

2022-06-01 Thread Sam Whittle (Jira)
[ https://issues.apache.org/jira/browse/BEAM-12942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17544772#comment-17544772 ] Sam Whittle commented on BEAM-12942: Java sdk was fixed with 2.34, reopening to add s

[jira] [Reopened] (BEAM-12942) Dataflow runner specialization of PubsubIO should validate messages

2022-06-01 Thread Sam Whittle (Jira)
[ https://issues.apache.org/jira/browse/BEAM-12942?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sam Whittle reopened BEAM-12942: > Dataflow runner specialization of PubsubIO should validate messages > ---

[jira] [Assigned] (BEAM-14167) Jamm exceptions "JVM prevents jamm from accessing subgraph - cache sizes may be underestimated"

2022-03-25 Thread Sam Whittle (Jira)
[ https://issues.apache.org/jira/browse/BEAM-14167?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sam Whittle reassigned BEAM-14167: -- Assignee: Kiley Sok (was: Luke Cwik) Feel free to mark as duplicate or blocker of BEAM-13695

[jira] [Updated] (BEAM-14167) Jamm exceptions "JVM prevents jamm from accessing subgraph - cache sizes may be underestimated"

2022-03-24 Thread Sam Whittle (Jira)
[ https://issues.apache.org/jira/browse/BEAM-14167?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sam Whittle updated BEAM-14167: --- Labels: Java11 (was: ) > Jamm exceptions "JVM prevents jamm from accessing subgraph - cache sizes ma

[jira] [Comment Edited] (BEAM-14167) Jamm exceptions "JVM prevents jamm from accessing subgraph - cache sizes may be underestimated"

2022-03-24 Thread Sam Whittle (Jira)
[ https://issues.apache.org/jira/browse/BEAM-14167?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17511869#comment-17511869 ] Sam Whittle edited comment on BEAM-14167 at 3/24/22, 1:59 PM: -

[jira] [Commented] (BEAM-14167) Jamm exceptions "JVM prevents jamm from accessing subgraph - cache sizes may be underestimated"

2022-03-24 Thread Sam Whittle (Jira)
[ https://issues.apache.org/jira/browse/BEAM-14167?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17511869#comment-17511869 ] Sam Whittle commented on BEAM-14167: >From >https://stackoverflow.com/questions/4126

[jira] [Created] (BEAM-14167) Jamm exceptions "JVM prevents jamm from accessing subgraph - cache sizes may be underestimated"

2022-03-24 Thread Sam Whittle (Jira)
Sam Whittle created BEAM-14167: -- Summary: Jamm exceptions "JVM prevents jamm from accessing subgraph - cache sizes may be underestimated" Key: BEAM-14167 URL: https://issues.apache.org/jira/browse/BEAM-14167

[jira] [Created] (BEAM-13826) Reduce number of Gax related threads, likely by providing common executor to GAX clients

2022-02-04 Thread Sam Whittle (Jira)
Sam Whittle created BEAM-13826: -- Summary: Reduce number of Gax related threads, likely by providing common executor to GAX clients Key: BEAM-13826 URL: https://issues.apache.org/jira/browse/BEAM-13826 Pr

[jira] [Created] (BEAM-13684) Consider adding information to ProcessBundleRequest indicating there is no existing state

2022-01-18 Thread Sam Whittle (Jira)
Sam Whittle created BEAM-13684: -- Summary: Consider adding information to ProcessBundleRequest indicating there is no existing state Key: BEAM-13684 URL: https://issues.apache.org/jira/browse/BEAM-13684 P

[jira] [Comment Edited] (BEAM-12857) Unable to write to GCS due to IndexOutOfBoundsException in FileSystems

2022-01-17 Thread Sam Whittle (Jira)
[ https://issues.apache.org/jira/browse/BEAM-12857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17477130#comment-17477130 ] Sam Whittle edited comment on BEAM-12857 at 1/17/22, 11:18 AM:

[jira] [Commented] (BEAM-12857) Unable to write to GCS due to IndexOutOfBoundsException in FileSystems

2022-01-17 Thread Sam Whittle (Jira)
[ https://issues.apache.org/jira/browse/BEAM-12857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17477130#comment-17477130 ] Sam Whittle commented on BEAM-12857: >From looking at code it does seem that that suc

[jira] [Reopened] (BEAM-12776) Improve parallelism of closing files in FileIO

2022-01-06 Thread Sam Whittle (Jira)
[ https://issues.apache.org/jira/browse/BEAM-12776?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sam Whittle reopened BEAM-12776: There is a large buffer by default for GCS writes and closing all windows in parallel can increase mem

[jira] [Updated] (BEAM-12776) Improve parallelism of closing files in FileIO

2022-01-06 Thread Sam Whittle (Jira)
[ https://issues.apache.org/jira/browse/BEAM-12776?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sam Whittle updated BEAM-12776: --- Status: Open (was: Triage Needed) > Improve parallelism of closing files in FileIO > ---

[jira] [Work started] (BEAM-12776) Improve parallelism of closing files in FileIO

2022-01-06 Thread Sam Whittle (Jira)
[ https://issues.apache.org/jira/browse/BEAM-12776?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on BEAM-12776 started by Sam Whittle. -- > Improve parallelism of closing files in FileIO > -

[jira] [Updated] (BEAM-12144) Dataflow streaming worker stuck and unable to get work from Streaming Engine

2021-12-01 Thread Sam Whittle (Jira)
[ https://issues.apache.org/jira/browse/BEAM-12144?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sam Whittle updated BEAM-12144: --- Fix Version/s: 2.32.0 (was: 2.31.0) > Dataflow streaming worker stuck and unab

[jira] [Updated] (BEAM-13268) Reduce latency by parallelizing BQ inserts when flushing due to row limit

2021-11-17 Thread Sam Whittle (Jira)
[ https://issues.apache.org/jira/browse/BEAM-13268?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sam Whittle updated BEAM-13268: --- Fix Version/s: 2.35.0 Resolution: Fixed Status: Resolved (was: Triage Needed) > Re

[jira] [Updated] (BEAM-12818) When writing to GCS, spread prefix of temporary files and reuse autoscaling of the temporary directory

2021-11-17 Thread Sam Whittle (Jira)
[ https://issues.apache.org/jira/browse/BEAM-12818?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sam Whittle updated BEAM-12818: --- Resolution: Fixed Status: Resolved (was: Open) > When writing to GCS, spread prefix of tempo

[jira] [Commented] (BEAM-13268) Reduce latency by parallelizing BQ inserts when flushing due to row limit

2021-11-17 Thread Sam Whittle (Jira)
[ https://issues.apache.org/jira/browse/BEAM-13268?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17445154#comment-17445154 ] Sam Whittle commented on BEAM-13268: Actually it looks like Pablo improved the BQ ser

[jira] [Updated] (BEAM-12472) BigQuery streaming writes can be batched beyond request limit with BatchAndInsertElements

2021-11-17 Thread Sam Whittle (Jira)
[ https://issues.apache.org/jira/browse/BEAM-12472?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sam Whittle updated BEAM-12472: --- Fix Version/s: 2.35.0 (was: 2.34.0) > BigQuery streaming writes can be batched

[jira] [Updated] (BEAM-12472) BigQuery streaming writes can be batched beyond request limit with BatchAndInsertElements

2021-11-17 Thread Sam Whittle (Jira)
[ https://issues.apache.org/jira/browse/BEAM-12472?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sam Whittle updated BEAM-12472: --- Fix Version/s: 2.34.0 Resolution: Fixed Status: Resolved (was: Open) > BigQuery st

[jira] [Assigned] (BEAM-12472) BigQuery streaming writes can be batched beyond request limit with BatchAndInsertElements

2021-11-17 Thread Sam Whittle (Jira)
[ https://issues.apache.org/jira/browse/BEAM-12472?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sam Whittle reassigned BEAM-12472: -- Assignee: Pablo Estrada > BigQuery streaming writes can be batched beyond request limit with

[jira] [Commented] (BEAM-12472) BigQuery streaming writes can be batched beyond request limit with BatchAndInsertElements

2021-11-17 Thread Sam Whittle (Jira)
[ https://issues.apache.org/jira/browse/BEAM-12472?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17445136#comment-17445136 ] Sam Whittle commented on BEAM-12472: I believe this would be fixed by Pablo's PR as i

[jira] [Assigned] (BEAM-13268) Reduce latency by parallelizing BQ inserts when flushing due to row limit

2021-11-17 Thread Sam Whittle (Jira)
[ https://issues.apache.org/jira/browse/BEAM-13268?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sam Whittle reassigned BEAM-13268: -- Assignee: Sam Whittle > Reduce latency by parallelizing BQ inserts when flushing due to row li

[jira] [Created] (BEAM-13268) Reduce latency by parallelizing BQ inserts when flushing due to row limit

2021-11-17 Thread Sam Whittle (Jira)
Sam Whittle created BEAM-13268: -- Summary: Reduce latency by parallelizing BQ inserts when flushing due to row limit Key: BEAM-13268 URL: https://issues.apache.org/jira/browse/BEAM-13268 Project: Beam

[jira] [Updated] (BEAM-13042) Prevent unexpected blocking in RegisterAndProcessBundleOperation hasFailed

2021-10-19 Thread Sam Whittle (Jira)
[ https://issues.apache.org/jira/browse/BEAM-13042?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sam Whittle updated BEAM-13042: --- Fix Version/s: 2.35.0 Resolution: Fixed Status: Resolved (was: Open) > Prevent une

[jira] [Updated] (BEAM-12856) Allow for configuration of unbounded reader max elements, read time etc in StreamingDataflowRunner

2021-10-19 Thread Sam Whittle (Jira)
[ https://issues.apache.org/jira/browse/BEAM-12856?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sam Whittle updated BEAM-12856: --- Fix Version/s: 2.34.0 Resolution: Fixed Status: Resolved (was: Open) > Allow for c

[jira] [Commented] (BEAM-12291) org.apache.beam.runners.flink.ReadSourcePortableTest.testExecution[streaming: false] is flaky

2021-10-13 Thread Sam Whittle (Jira)
[ https://issues.apache.org/jira/browse/BEAM-12291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17428227#comment-17428227 ] Sam Whittle commented on BEAM-12291: Sorry I didn't see this was assigned to me earli

[jira] [Assigned] (BEAM-12291) org.apache.beam.runners.flink.ReadSourcePortableTest.testExecution[streaming: false] is flaky

2021-10-13 Thread Sam Whittle (Jira)
[ https://issues.apache.org/jira/browse/BEAM-12291?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sam Whittle reassigned BEAM-12291: -- Assignee: (was: Sam Whittle) > org.apache.beam.runners.flink.ReadSourcePortableTest.testEx

[jira] [Created] (BEAM-13042) Prevent unexpected blocking in RegisterAndProcessBundleOperation hasFailed

2021-10-13 Thread Sam Whittle (Jira)
Sam Whittle created BEAM-13042: -- Summary: Prevent unexpected blocking in RegisterAndProcessBundleOperation hasFailed Key: BEAM-13042 URL: https://issues.apache.org/jira/browse/BEAM-13042 Project: Beam

[jira] [Updated] (BEAM-12942) Dataflow runner specialization of PubsubIO should validate messages

2021-09-29 Thread Sam Whittle (Jira)
[ https://issues.apache.org/jira/browse/BEAM-12942?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sam Whittle updated BEAM-12942: --- Fix Version/s: (was: 2.33.0) 2.34.0 > Dataflow runner specialization of Pubsub

[jira] [Updated] (BEAM-12942) Dataflow runner specialization of PubsubIO should validate messages

2021-09-29 Thread Sam Whittle (Jira)
[ https://issues.apache.org/jira/browse/BEAM-12942?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sam Whittle updated BEAM-12942: --- Fix Version/s: 2.33.0 Resolution: Fixed Status: Resolved (was: Open) > Dataflow ru

[jira] [Commented] (BEAM-12942) Dataflow runner specialization of PubsubIO should validate messages

2021-09-23 Thread Sam Whittle (Jira)
[ https://issues.apache.org/jira/browse/BEAM-12942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17419231#comment-17419231 ] Sam Whittle commented on BEAM-12942: The validation seems like it could be helpful fo

[jira] [Created] (BEAM-12942) Dataflow runner specialization of PubsubIO should validate messages

2021-09-23 Thread Sam Whittle (Jira)
Sam Whittle created BEAM-12942: -- Summary: Dataflow runner specialization of PubsubIO should validate messages Key: BEAM-12942 URL: https://issues.apache.org/jira/browse/BEAM-12942 Project: Beam

[jira] [Work started] (BEAM-7913) Add drain() to DataflowPipelineJob

2021-09-17 Thread Sam Whittle (Jira)
[ https://issues.apache.org/jira/browse/BEAM-7913?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on BEAM-7913 started by null. -- > Add drain() to DataflowPipelineJob > -- > > Key: BE

[jira] [Updated] (BEAM-12740) Reduce and backoff GCS metadata operations when writing to GCS files

2021-09-17 Thread Sam Whittle (Jira)
[ https://issues.apache.org/jira/browse/BEAM-12740?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sam Whittle updated BEAM-12740: --- Resolution: Fixed Status: Resolved (was: Open) > Reduce and backoff GCS metadata operations

[jira] [Updated] (BEAM-12740) Reduce and backoff GCS metadata operations when writing to GCS files

2021-09-17 Thread Sam Whittle (Jira)
[ https://issues.apache.org/jira/browse/BEAM-12740?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sam Whittle updated BEAM-12740: --- Fix Version/s: 2.34.0 > Reduce and backoff GCS metadata operations when writing to GCS files > --

[jira] [Commented] (BEAM-12445) Move Python's BigQuery streaming insert sink to the new BigQuery api client

2021-09-16 Thread Sam Whittle (Jira)
[ https://issues.apache.org/jira/browse/BEAM-12445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17416247#comment-17416247 ] Sam Whittle commented on BEAM-12445: I believe the timeout needs to be set on the new

[jira] [Updated] (BEAM-12776) Improve parallelism of closing files in FileIO

2021-09-08 Thread Sam Whittle (Jira)
[ https://issues.apache.org/jira/browse/BEAM-12776?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sam Whittle updated BEAM-12776: --- Resolution: Fixed Status: Resolved (was: Open) > Improve parallelism of closing files in Fil

[jira] [Updated] (BEAM-12856) Allow for configuration of unbounded reader max elements, read time etc in StreamingDataflowRunner

2021-09-08 Thread Sam Whittle (Jira)
[ https://issues.apache.org/jira/browse/BEAM-12856?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sam Whittle updated BEAM-12856: --- Description: Currently in WorkerCustomSources.java it is hard-coded to 10 seconds, 10k elements, wit

[jira] [Created] (BEAM-12856) Allow for configuration of unbounded reader max elements, read time etc in StreamingDataflowRunner

2021-09-08 Thread Sam Whittle (Jira)
Sam Whittle created BEAM-12856: -- Summary: Allow for configuration of unbounded reader max elements, read time etc in StreamingDataflowRunner Key: BEAM-12856 URL: https://issues.apache.org/jira/browse/BEAM-12856

[jira] [Commented] (BEAM-11994) Java BigQuery - Implement IO Request Count metrics

2021-08-30 Thread Sam Whittle (Jira)
[ https://issues.apache.org/jira/browse/BEAM-11994?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17406716#comment-17406716 ] Sam Whittle commented on BEAM-11994: I observed the following in a test, it seems rel

[jira] [Created] (BEAM-12818) When writing to GCS, spread prefix of temporary files and reuse autoscaling of the temporary directory

2021-08-30 Thread Sam Whittle (Jira)
Sam Whittle created BEAM-12818: -- Summary: When writing to GCS, spread prefix of temporary files and reuse autoscaling of the temporary directory Key: BEAM-12818 URL: https://issues.apache.org/jira/browse/BEAM-12818

[jira] [Updated] (BEAM-12780) StreamingDataflowWorker should limit local retries

2021-08-23 Thread Sam Whittle (Jira)
[ https://issues.apache.org/jira/browse/BEAM-12780?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sam Whittle updated BEAM-12780: --- Fix Version/s: 2.32.0 Resolution: Fixed Status: Resolved (was: In Progress) > Stre

[jira] [Work started] (BEAM-12780) StreamingDataflowWorker should limit local retries

2021-08-20 Thread Sam Whittle (Jira)
[ https://issues.apache.org/jira/browse/BEAM-12780?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on BEAM-12780 started by Sam Whittle. -- > StreamingDataflowWorker should limit local retries > -

[jira] [Updated] (BEAM-12780) StreamingDataflowWorker should limit local retries

2021-08-20 Thread Sam Whittle (Jira)
[ https://issues.apache.org/jira/browse/BEAM-12780?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sam Whittle updated BEAM-12780: --- Status: Open (was: Triage Needed) > StreamingDataflowWorker should limit local retries > ---

[jira] [Assigned] (BEAM-12780) StreamingDataflowWorker should limit local retries

2021-08-20 Thread Sam Whittle (Jira)
[ https://issues.apache.org/jira/browse/BEAM-12780?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sam Whittle reassigned BEAM-12780: -- Assignee: Sam Whittle > StreamingDataflowWorker should limit local retries > -

[jira] [Created] (BEAM-12780) StreamingDataflowWorker should limit local retries

2021-08-20 Thread Sam Whittle (Jira)
Sam Whittle created BEAM-12780: -- Summary: StreamingDataflowWorker should limit local retries Key: BEAM-12780 URL: https://issues.apache.org/jira/browse/BEAM-12780 Project: Beam Issue Type: Bug

[jira] [Created] (BEAM-12776) Improve parallelism of closing files in FileIO

2021-08-19 Thread Sam Whittle (Jira)
Sam Whittle created BEAM-12776: -- Summary: Improve parallelism of closing files in FileIO Key: BEAM-12776 URL: https://issues.apache.org/jira/browse/BEAM-12776 Project: Beam Issue Type: Bug

[jira] [Created] (BEAM-12740) Reduce and backoff GCS metadata operations when writing to GCS files

2021-08-11 Thread Sam Whittle (Jira)
Sam Whittle created BEAM-12740: -- Summary: Reduce and backoff GCS metadata operations when writing to GCS files Key: BEAM-12740 URL: https://issues.apache.org/jira/browse/BEAM-12740 Project: Beam

[jira] [Updated] (BEAM-12144) Dataflow streaming worker stuck and unable to get work from Streaming Engine

2021-06-22 Thread Sam Whittle (Jira)
[ https://issues.apache.org/jira/browse/BEAM-12144?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sam Whittle updated BEAM-12144: --- Fix Version/s: 2.31.0 Resolution: Fixed Status: Resolved (was: Open) > Dataflow st

[jira] [Created] (BEAM-12516) StreamingDataflowWorker.ShardedKey.toString throws exception if key is less than 100 bytes

2021-06-21 Thread Sam Whittle (Jira)
Sam Whittle created BEAM-12516: -- Summary: StreamingDataflowWorker.ShardedKey.toString throws exception if key is less than 100 bytes Key: BEAM-12516 URL: https://issues.apache.org/jira/browse/BEAM-12516

[jira] [Commented] (BEAM-12472) BigQuery streaming writes can be batched beyond request limit with BatchAndInsertElements

2021-06-17 Thread Sam Whittle (Jira)
[ https://issues.apache.org/jira/browse/BEAM-12472?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17364810#comment-17364810 ] Sam Whittle commented on BEAM-12472: I was using withoutInsertIds, which perhaps trig

[jira] [Created] (BEAM-12472) BigQuery streaming writes can be batched beyond request limit with BatchAndInsertElements

2021-06-09 Thread Sam Whittle (Jira)
Sam Whittle created BEAM-12472: -- Summary: BigQuery streaming writes can be batched beyond request limit with BatchAndInsertElements Key: BEAM-12472 URL: https://issues.apache.org/jira/browse/BEAM-12472 P

[jira] [Updated] (BEAM-12402) Optimize PCollectionConsumerRegistry$MultiplexingMetricTrackingFnDataReceiver

2021-05-25 Thread Sam Whittle (Jira)
[ https://issues.apache.org/jira/browse/BEAM-12402?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sam Whittle updated BEAM-12402: --- Resolution: Fixed Status: Resolved (was: Open) > Optimize PCollectionConsumerRegistry$Multip

[jira] [Created] (BEAM-12402) Optimize PCollectionConsumerRegistry$MultiplexingMetricTrackingFnDataReceiver

2021-05-25 Thread Sam Whittle (Jira)
Sam Whittle created BEAM-12402: -- Summary: Optimize PCollectionConsumerRegistry$MultiplexingMetricTrackingFnDataReceiver Key: BEAM-12402 URL: https://issues.apache.org/jira/browse/BEAM-12402 Project: Beam

[jira] [Updated] (BEAM-7717) PubsubIO watermark tracking hovers near start of epoch

2021-05-25 Thread Sam Whittle (Jira)
[ https://issues.apache.org/jira/browse/BEAM-7717?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sam Whittle updated BEAM-7717: -- Fix Version/s: 2.31.0 Resolution: Fixed Status: Resolved (was: Open) > PubsubIO water

[jira] [Assigned] (BEAM-7717) PubsubIO watermark tracking hovers near start of epoch

2021-05-20 Thread Sam Whittle (Jira)
[ https://issues.apache.org/jira/browse/BEAM-7717?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sam Whittle reassigned BEAM-7717: - Assignee: Sam Whittle > PubsubIO watermark tracking hovers near start of epoch >

[jira] [Commented] (BEAM-12144) Dataflow streaming worker stuck and unable to get work from Streaming Engine

2021-05-10 Thread Sam Whittle (Jira)
[ https://issues.apache.org/jira/browse/BEAM-12144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17342140#comment-17342140 ] Sam Whittle commented on BEAM-12144: PR is being reviewed https://github.com/apache/b

[jira] [Updated] (BEAM-12144) Dataflow streaming worker stuck and unable to get work from Streaming Engine

2021-05-10 Thread Sam Whittle (Jira)
[ https://issues.apache.org/jira/browse/BEAM-12144?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sam Whittle updated BEAM-12144: --- Labels: (was: stale-assigned) > Dataflow streaming worker stuck and unable to get work from Streami

[jira] [Updated] (BEAM-12254) Nexmark UnboundedReader does not report backlog correctly

2021-05-03 Thread Sam Whittle (Jira)
[ https://issues.apache.org/jira/browse/BEAM-12254?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sam Whittle updated BEAM-12254: --- Resolution: Fixed Status: Resolved (was: Open) > Nexmark UnboundedReader does not report bac

[jira] [Updated] (BEAM-12118) QueuingBeamFnDataClient adds polling latency to completing bundle processing

2021-05-03 Thread Sam Whittle (Jira)
[ https://issues.apache.org/jira/browse/BEAM-12118?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sam Whittle updated BEAM-12118: --- Resolution: Fixed Status: Resolved (was: Triage Needed) > QueuingBeamFnDataClient adds polli

[jira] [Updated] (BEAM-12209) DirectStreamObserver is not thread-safe as advertised due to racy integer operations

2021-05-03 Thread Sam Whittle (Jira)
[ https://issues.apache.org/jira/browse/BEAM-12209?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sam Whittle updated BEAM-12209: --- Resolution: Fixed Status: Resolved (was: Open) > DirectStreamObserver is not thread-safe as

[jira] [Updated] (BEAM-12203) Reduce thread context switches in BeamFnControlClient

2021-05-03 Thread Sam Whittle (Jira)
[ https://issues.apache.org/jira/browse/BEAM-12203?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sam Whittle updated BEAM-12203: --- Resolution: Fixed Status: Resolved (was: Open) > Reduce thread context switches in BeamFnCon

[jira] [Work started] (BEAM-12253) Read.UnboundedSourceAsSDFRestrictionTracker doesn't use cache for readers in getProgress

2021-05-03 Thread Sam Whittle (Jira)
[ https://issues.apache.org/jira/browse/BEAM-12253?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on BEAM-12253 started by Sam Whittle. -- > Read.UnboundedSourceAsSDFRestrictionTracker doesn't use cache for readers in > ge

[jira] [Created] (BEAM-12254) Nexmark UnboundedReader does not report backlog correctly

2021-04-29 Thread Sam Whittle (Jira)
Sam Whittle created BEAM-12254: -- Summary: Nexmark UnboundedReader does not report backlog correctly Key: BEAM-12254 URL: https://issues.apache.org/jira/browse/BEAM-12254 Project: Beam Issue Type

[jira] [Created] (BEAM-12253) Read.UnboundedSourceAsSDFRestrictionTracker doesn't use cache for readers in getProgress

2021-04-29 Thread Sam Whittle (Jira)
Sam Whittle created BEAM-12253: -- Summary: Read.UnboundedSourceAsSDFRestrictionTracker doesn't use cache for readers in getProgress Key: BEAM-12253 URL: https://issues.apache.org/jira/browse/BEAM-12253 Pr

[jira] [Reopened] (BEAM-12118) QueuingBeamFnDataClient adds polling latency to completing bundle processing

2021-04-28 Thread Sam Whittle (Jira)
[ https://issues.apache.org/jira/browse/BEAM-12118?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sam Whittle reopened BEAM-12118: Reopening to track fixing race triggering precondition > QueuingBeamFnDataClient adds polling latency

[jira] [Assigned] (BEAM-12229) WindmillStateCache has a 0% hit rate in 2.29

2021-04-27 Thread Sam Whittle (Jira)
[ https://issues.apache.org/jira/browse/BEAM-12229?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sam Whittle reassigned BEAM-12229: -- Assignee: Sam Whittle (was: Reuven Lax) > WindmillStateCache has a 0% hit rate in 2.29 >

[jira] [Commented] (BEAM-12229) WindmillStateCache has a 0% hit rate in 2.29

2021-04-27 Thread Sam Whittle (Jira)
[ https://issues.apache.org/jira/browse/BEAM-12229?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17333208#comment-17333208 ] Sam Whittle commented on BEAM-12229: Ugh, I see the issue. I modified how tokens were

[jira] [Created] (BEAM-12209) DirectStreamObserver is not thread-safe as advertised due to racy integer operations

2021-04-22 Thread Sam Whittle (Jira)
Sam Whittle created BEAM-12209: -- Summary: DirectStreamObserver is not thread-safe as advertised due to racy integer operations Key: BEAM-12209 URL: https://issues.apache.org/jira/browse/BEAM-12209 Projec

[jira] [Comment Edited] (BEAM-12203) Reduce thread context switches in BeamFnControlClient

2021-04-21 Thread Sam Whittle (Jira)
[ https://issues.apache.org/jira/browse/BEAM-12203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17326662#comment-17326662 ] Sam Whittle edited comment on BEAM-12203 at 4/21/21, 8:15 PM: -

[jira] [Commented] (BEAM-12203) Reduce thread context switches in BeamFnControlClient

2021-04-21 Thread Sam Whittle (Jira)
[ https://issues.apache.org/jira/browse/BEAM-12203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17326662#comment-17326662 ] Sam Whittle commented on BEAM-12203: Realized that this could further be simplified b

[jira] [Updated] (BEAM-12203) Reduce thread context switches in BeamFnControlClient

2021-04-21 Thread Sam Whittle (Jira)
[ https://issues.apache.org/jira/browse/BEAM-12203?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sam Whittle updated BEAM-12203: --- Summary: Reduce thread context switches in BeamFnControlClient (was: Remove LinkedBlockingQueue from

[jira] [Commented] (BEAM-12203) Remove LinkedBlockingQueue from BeamFnControlClient

2021-04-21 Thread Sam Whittle (Jira)
[ https://issues.apache.org/jira/browse/BEAM-12203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17326606#comment-17326606 ] Sam Whittle commented on BEAM-12203: The put/take on this queue appears to be 2% of c

[jira] [Created] (BEAM-12203) Remove LinkedBlockingQueue from BeamFnControlClient

2021-04-21 Thread Sam Whittle (Jira)
Sam Whittle created BEAM-12203: -- Summary: Remove LinkedBlockingQueue from BeamFnControlClient Key: BEAM-12203 URL: https://issues.apache.org/jira/browse/BEAM-12203 Project: Beam Issue Type: Bug

[jira] [Updated] (BEAM-12118) QueuingBeamFnDataClient adds polling latency to completing bundle processing

2021-04-21 Thread Sam Whittle (Jira)
[ https://issues.apache.org/jira/browse/BEAM-12118?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sam Whittle updated BEAM-12118: --- Fix Version/s: 2.30.0 Resolution: Fixed Status: Resolved (was: Open) > QueuingBeam

[jira] [Updated] (BEAM-11910) Increase subsequent page size for bags after the first

2021-04-15 Thread Sam Whittle (Jira)
[ https://issues.apache.org/jira/browse/BEAM-11910?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sam Whittle updated BEAM-11910: --- Fix Version/s: 2.29.0 > Increase subsequent page size for bags after the first >

[jira] [Updated] (BEAM-12127) Reduce counter overhead in PCollectionConsumerRegistry.accept

2021-04-12 Thread Sam Whittle (Jira)
[ https://issues.apache.org/jira/browse/BEAM-12127?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sam Whittle updated BEAM-12127: --- Resolution: Fixed Status: Resolved (was: Open) > Reduce counter overhead in PCollectionConsu

[jira] [Updated] (BEAM-12142) Reduce overhead of MetricsEnvironment

2021-04-12 Thread Sam Whittle (Jira)
[ https://issues.apache.org/jira/browse/BEAM-12142?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sam Whittle updated BEAM-12142: --- Fix Version/s: 2.30.0 Resolution: Fixed Status: Resolved (was: Open) > Reduce over

[jira] [Updated] (BEAM-12117) QueuingBeamFnDataClient inbound client set grows with BundleProcessor reuse

2021-04-12 Thread Sam Whittle (Jira)
[ https://issues.apache.org/jira/browse/BEAM-12117?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sam Whittle updated BEAM-12117: --- Resolution: Fixed Status: Resolved (was: Open) > QueuingBeamFnDataClient inbound client set

[jira] [Updated] (BEAM-11910) Increase subsequent page size for bags after the first

2021-04-12 Thread Sam Whittle (Jira)
[ https://issues.apache.org/jira/browse/BEAM-11910?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sam Whittle updated BEAM-11910: --- Resolution: Fixed Status: Resolved (was: Open) > Increase subsequent page size for bags afte

[jira] [Created] (BEAM-12144) Dataflow streaming worker stuck and unable to get work from Streaming Engine

2021-04-09 Thread Sam Whittle (Jira)
Sam Whittle created BEAM-12144: -- Summary: Dataflow streaming worker stuck and unable to get work from Streaming Engine Key: BEAM-12144 URL: https://issues.apache.org/jira/browse/BEAM-12144 Project: Beam

[jira] [Created] (BEAM-12142) Reduce overhead of MetricsEnvironment

2021-04-09 Thread Sam Whittle (Jira)
Sam Whittle created BEAM-12142: -- Summary: Reduce overhead of MetricsEnvironment Key: BEAM-12142 URL: https://issues.apache.org/jira/browse/BEAM-12142 Project: Beam Issue Type: Bug Comp

[jira] [Created] (BEAM-12127) Reduce counter overhead in PCollectionConsumerRegistry.accept

2021-04-08 Thread Sam Whittle (Jira)
Sam Whittle created BEAM-12127: -- Summary: Reduce counter overhead in PCollectionConsumerRegistry.accept Key: BEAM-12127 URL: https://issues.apache.org/jira/browse/BEAM-12127 Project: Beam Issue

[jira] [Created] (BEAM-12118) QueuingBeamFnDataClient adds polling latency to completing bundle processing

2021-04-07 Thread Sam Whittle (Jira)
Sam Whittle created BEAM-12118: -- Summary: QueuingBeamFnDataClient adds polling latency to completing bundle processing Key: BEAM-12118 URL: https://issues.apache.org/jira/browse/BEAM-12118 Project: Beam

[jira] [Created] (BEAM-12117) QueuingBeamFnDataClient inbound client set grows with BundleProcessor reuse

2021-04-07 Thread Sam Whittle (Jira)
Sam Whittle created BEAM-12117: -- Summary: QueuingBeamFnDataClient inbound client set grows with BundleProcessor reuse Key: BEAM-12117 URL: https://issues.apache.org/jira/browse/BEAM-12117 Project: Beam

[jira] [Updated] (BEAM-11727) Optimize ExecutionStateSampler

2021-03-12 Thread Sam Whittle (Jira)
[ https://issues.apache.org/jira/browse/BEAM-11727?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sam Whittle updated BEAM-11727: --- Resolution: Fixed Status: Resolved (was: Triage Needed) > Optimize ExecutionStateSampler > -

[jira] [Updated] (BEAM-11730) Reduce context switches for dataflow streaming appliance getdata reads

2021-03-12 Thread Sam Whittle (Jira)
[ https://issues.apache.org/jira/browse/BEAM-11730?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sam Whittle updated BEAM-11730: --- Resolution: Fixed Status: Resolved (was: Open) > Reduce context switches for dataflow stream

[jira] [Updated] (BEAM-11707) Optimize WindmillStateCache CPU usage

2021-03-12 Thread Sam Whittle (Jira)
[ https://issues.apache.org/jira/browse/BEAM-11707?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sam Whittle updated BEAM-11707: --- Resolution: Fixed Status: Resolved (was: Open) > Optimize WindmillStateCache CPU usage > ---

[jira] [Updated] (BEAM-11729) Remove ReduceFnRunner eager class name evaluation for debug logging

2021-03-12 Thread Sam Whittle (Jira)
[ https://issues.apache.org/jira/browse/BEAM-11729?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sam Whittle updated BEAM-11729: --- Resolution: Fixed Status: Resolved (was: Open) > Remove ReduceFnRunner eager class name eval

[jira] [Assigned] (BEAM-11910) Increase subsequent page size for bags after the first

2021-03-02 Thread Sam Whittle (Jira)
[ https://issues.apache.org/jira/browse/BEAM-11910?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sam Whittle reassigned BEAM-11910: -- Assignee: Sam Whittle > Increase subsequent page size for bags after the first > -

[jira] [Created] (BEAM-11910) Increase subsequent page size for bags after the first

2021-03-02 Thread Sam Whittle (Jira)
Sam Whittle created BEAM-11910: -- Summary: Increase subsequent page size for bags after the first Key: BEAM-11910 URL: https://issues.apache.org/jira/browse/BEAM-11910 Project: Beam Issue Type: B

[jira] [Commented] (BEAM-11144) TriggerStateMachine.prefetchOnElement and other prefetch methods use incorrect state for subtriggers

2021-02-23 Thread Sam Whittle (Jira)
[ https://issues.apache.org/jira/browse/BEAM-11144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17289572#comment-17289572 ] Sam Whittle commented on BEAM-11144: Yes, thanks! > TriggerStateMachine.prefetchO

[jira] [Created] (BEAM-11730) Reduce context switches for dataflow streaming appliance getdata reads

2021-02-01 Thread Sam Whittle (Jira)
Sam Whittle created BEAM-11730: -- Summary: Reduce context switches for dataflow streaming appliance getdata reads Key: BEAM-11730 URL: https://issues.apache.org/jira/browse/BEAM-11730 Project: Beam

[jira] [Created] (BEAM-11729) Remove ReduceFnRunner eager class name evaluation for debug logging

2021-02-01 Thread Sam Whittle (Jira)
Sam Whittle created BEAM-11729: -- Summary: Remove ReduceFnRunner eager class name evaluation for debug logging Key: BEAM-11729 URL: https://issues.apache.org/jira/browse/BEAM-11729 Project: Beam

[jira] [Commented] (BEAM-11706) TriggerProto translation shows up as 1% cpu on some benchmarks

2021-02-01 Thread Sam Whittle (Jira)
[ https://issues.apache.org/jira/browse/BEAM-11706?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17276313#comment-17276313 ] Sam Whittle commented on BEAM-11706: I had a typo in the jira issue for the pull requ

[jira] [Updated] (BEAM-11727) Optimize ExecutionStateSampler

2021-02-01 Thread Sam Whittle (Jira)
[ https://issues.apache.org/jira/browse/BEAM-11727?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sam Whittle updated BEAM-11727: --- Component/s: (was: runner-dataflow) runner-core > Optimize ExecutionStateSampler

[jira] [Updated] (BEAM-11727) Optimize ExecutionStateSampler

2021-02-01 Thread Sam Whittle (Jira)
[ https://issues.apache.org/jira/browse/BEAM-11727?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sam Whittle updated BEAM-11727: --- Summary: Optimize ExecutionStateSampler (was: Optimize ExecutionStateTracker) > Optimize ExecutionS

[jira] [Assigned] (BEAM-11727) Optimize ExecutionStateTracker

2021-02-01 Thread Sam Whittle (Jira)
[ https://issues.apache.org/jira/browse/BEAM-11727?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sam Whittle reassigned BEAM-11727: -- Assignee: Sam Whittle > Optimize ExecutionStateTracker > -- > >

[jira] [Created] (BEAM-11727) Optimize ExecutionStateTracker

2021-02-01 Thread Sam Whittle (Jira)
Sam Whittle created BEAM-11727: -- Summary: Optimize ExecutionStateTracker Key: BEAM-11727 URL: https://issues.apache.org/jira/browse/BEAM-11727 Project: Beam Issue Type: Bug Components:

  1   2   >