Hi Mark, could you please check if you can tune akka? Usually in async I/O, the used library uses a thread pool that becomes the actual bottleneck.
If you configure async I/O to use a capacity of 100 and parallelism of 8 on one node, you also need to have ~800 threads in akka (500 might be enough because of overhead) or else async I/O gets blocked while waiting for akka threads to become available. Best, Arvid On Mon, Jul 6, 2020 at 1:45 PM Benchao Li <libenc...@apache.org> wrote: > Hi Mark, > > According to your data, I think the config of AsyncOperator is OK. > There is one more config that might affect the throughput of > AsyncOperator, it's watermark. > Because unordered async operator still keeps the order between watermarks, > did you use > event time in your job, and if yes, what's the watermark interval in your > job? > > Mark Zitnik <mark.zit...@gmail.com> 于2020年7月5日周日 下午7:44写道： > >> Hi Benchao >> >> The capacity is 100 >> Parallelism is 8 >> Rpc req is 20ms >> >> Thanks >> >> >> On Sun, 5 Jul 2020, 6:16 Benchao Li, <libenc...@apache.org> wrote: >> >>> Hi Mark, >>> >>> Could you give more details about your Flink job? >>> - the capacity of AsyncDataStream >>> - the parallelism of AsyncDataStream operator >>> - the time of per blocked rpc request >>> >>> Mark Zitnik <mark.zit...@gmail.com> 于2020年7月5日周日 上午3:48写道： >>> >>>> Hi >>>> >>>> In my flink application I need to enrich data using >>>> AsyncDataStream.unorderedWait >>>> but I am getting poor perforce at the beginning I was just working with >>>> http call, but I have switched to grpc, I running on 8 core node and >>>> getting total of 3200 events per second my service that I am using is not >>>> fully utilized and can produce up to 10000 req/seq >>>> >>>> Flink job flow >>>> Reading from Kafka ~> some enrichment with unoderedwait ~> map ~> write >>>> to Kafka >>>> >>>> Using Akkad grpc code written in scala >>>> >>>> Thanks >>>> >>> >>> >>> -- >>> >>> Best, >>> Benchao Li >>> >> > > -- > > Best, > Benchao Li > -- Arvid Heise | Senior Java Developer <https://www.ververica.com/> Follow us @VervericaData -- Join Flink Forward <https://flink-forward.org/> - The Apache Flink Conference Stream Processing | Event Driven | Real Time -- Ververica GmbH | Invalidenstrasse 115, 10115 Berlin, Germany -- Ververica GmbH Registered at Amtsgericht Charlottenburg: HRB 158244 B Managing Directors: Timothy Alexander Steinert, Yip Park Tung Jason, Ji (Toni) Cheng
