[
https://issues.apache.org/jira/browse/BEAM-14184?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17521723#comment-17521723
]
Bjorn De Bakker edited comment on BEAM-14184 at 4/13/22 3:22 PM:
-----------------------------------------------------------------
Thanks Luke. I just tested it, but unfortunately the same error came up. The
container img we're using runs Apache Beam version 2.30.0
(apache/beam_python3.7_sdk:2.30.0), so I'm not sure if that's the problem.
This is how I run the pipeline:
{code:java}
args:
- "-m"
- "src.batch_inference_pipeline.pipeline"
- "--padding_x=0"
- "--padding_y=0"
- "--runner=FlinkRunner"
- "--flink_master=beam-flink-cluster-jobmanager:8081"
- "--flink_submit_uber_jar"
- "--environment_type=EXTERNAL"
- "--environment_config=localhost:50000"
- "--experiments=beam_fn_api_buffered_stream" {code}
Maybe I made a mistake here.
was (Author: JIRAUSER288009):
Thanks Luke. I just tested it, but unfortunately the same error came up. The
container img we're using runs Apache Beam version 3.7, so I'm not sure if
that's the problem.
This is how I run the pipeline:
{code:java}
args:
- "-m"
- "src.batch_inference_pipeline.pipeline"
- "--padding_x=0"
- "--padding_y=0"
- "--runner=FlinkRunner"
- "--flink_master=beam-flink-cluster-jobmanager:8081"
- "--flink_submit_uber_jar"
- "--environment_type=EXTERNAL"
- "--environment_config=localhost:50000"
- "--experiments=beam_fn_api_buffered_stream" {code}
Maybe I made a mistake here.
> DirectStreamObserver does not respect channel isReady
> -----------------------------------------------------
>
> Key: BEAM-14184
> URL: https://issues.apache.org/jira/browse/BEAM-14184
> Project: Beam
> Issue Type: Bug
> Components: sdk-java-harness
> Reporter: Luke Cwik
> Assignee: Luke Cwik
> Priority: P2
>
> Leads to OOMs like:
> {noformat}
> Output channel stalled for 1023s, outbound thread CHAIN MapPartition
> (MapPartition at [1]PerformInference) -> FlatMap (FlatMap at
> ExtractOutput[0]) -> Map (Key Extractor) -> GroupCombine (GroupCombine at
> GroupCombine:
> PerformInferenceAndCombineResults_dep_049/GroupPredictionsByImage) -> Map
> (Key Extractor) (1/1). See: https://issues.apache.org/jira/browse/BEAM-4280
> for the history for this issue.
> Feb 18, 2022 11:51:05 AM
> org.apache.beam.vendor.grpc.v1p36p0.io.grpc.netty.NettyServerTransport
> notifyTerminated
> INFO: Transport failed
> org.apache.beam.vendor.grpc.v1p36p0.io.netty.util.internal.OutOfDirectMemoryError:
> failed to allocate 2097152 byte(s) of direct memory (used: 1205862679, max:
> 1207959552)
> at
> org.apache.beam.vendor.grpc.v1p36p0.io.netty.util.internal.PlatformDependent.incrementMemoryCounter(PlatformDependent.java:754)
> at
> org.apache.beam.vendor.grpc.v1p36p0.io.netty.util.internal.PlatformDependent.allocateDirectNoCleaner(PlatformDependent.java:709)
> at
> org.apache.beam.vendor.grpc.v1p36p0.io.netty.buffer.PoolArena$DirectArena.allocateDirect(PoolArena.java:645)
> at
> org.apache.beam.vendor.grpc.v1p36p0.io.netty.buffer.PoolArena$DirectArena.newChunk(PoolArena.java:621)
> at
> org.apache.beam.vendor.grpc.v1p36p0.io.netty.buffer.PoolArena.allocateNormal(PoolArena.java:204)
> at
> org.apache.beam.vendor.grpc.v1p36p0.io.netty.buffer.PoolArena.tcacheAllocateNormal(PoolArena.java:188)
> {noformat}
> See more context in
> https://lists.apache.org/thread/llmxodbmczhn10c98prs8wmd5hy4nvff
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
