Re: jobmanager 与taskmanager间的对象传递

Sun, 03 Jul 2022 22:25:10 -0700 

你好。

如果只是需要从各个subtask中收集一些信息，在JobManager中汇总的话，我觉得可以用累加器和计数器[1]。

如果需要双向通信的话，可以考虑一下FLIP-27[2]引入的OperatorCoordinator。如何通过通信来传递对象可以通过自定义算子或函数来实现。

在自定义算子中使用OperatorCoordinator可能还有一些不方便的地方，可以追踪一下相关ticket的进展[3]。

[1]
https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/user_defined_functions/#accumulators--counters
[2]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-27%3A+Refactor+Source+Interface
[3] https://issues.apache.org/jira/browse/FLINK-26029

On Sat, Jul 2, 2022 at 5:43 PM [email protected] <[email protected]>
wrote:

> 各位大佬：<br/> &nbsp;
> &nbsp;请教个问题，有个jobmanager产生的map对象，传递到各个taskmanager的function里，保证每个taskmanager内的对象有且仅有一个，然后会随时根据数据改变map对象的值，我想定时从taskmanager收集这个map对象整合后到jobmanager端，然后进行存储，有什么API或者通讯方式来支持吗？<br/><br/><br/><html>
> <head>
>     <meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
> </head>
> <body>
>         <a href="
> https://maas.mail.163.com/dashi-web-extend/html/proSignature.html?ftlId=1&name=ycbcholy%40gmail.com&uid=ycbcholy%40gmail.com&iconUrl=https%3A%2F%2Fmail-online.nosdn.127.net%2Fqiyelogo%2FdefaultAvatar.png&items=%5B%22%E9%82%AE%E7%AE%B1%EF%BC%9Aycbcholy%40gmail.com%22%5D";
> style="display:block;background:#fff; max-width: 400px; _width:
> 400px;padding:15px 0 10px 0;text-decoration: none;
> outline:none;-webkit-tap-highlight-color:transparent;-webkit-text-size-adjust:none
> !important;text-size-adjust:none !important;">
>             <table cellpadding="0" style='width: 100%; max-width: 100%;
> table-layout: fixed; border-collapse: collapse;color: #9b9ea1;font-size:
> 14px;line-height:1.3;-webkit-text-size-adjust:none
> !important;text-size-adjust:none !important;'>
>                 <tbody style="font-family: 'PingFang SC', 'Hiragino Sans
> GB','WenQuanYi Micro Hei', 'Microsoft Yahei', '微软雅黑', verdana !important;
> word-wrap:break-word; word-break:break-all;-webkit-text-size-adjust:none
> !important;text-size-adjust:none !important;">
>                     <tr>
>                             <td width="38" style="padding:0; box-sizing:
> border-box; width: 38px;">
>                                 <img width="38" height="38"
> style="vertical-align:middle; width: 38px; height: 38px;
> border-radius:50%;" src="
> https://mail-online.nosdn.127.net/qiyelogo/defaultAvatar.png"; />
>                             </td>
>                             <td style='padding: 0 0 0 10px; color:
> #31353b;'>
>                                 <div style="font-size:
> 16px;font-weight:bold; width:100%; white-space: nowrap;
> overflow:hidden;text-overflow: ellipsis;">[email protected]</div>
>                             </td>
>                     </tr>
>                         <tr width="100%" style="font-size: 14px
> !important; width: 100%;">
>                             <td colspan='2' style="padding:10px 0 0 0;
> font-size:14px !important; width: 100%;">
>                                     <div style="width: 100%;font-size:
> 14px !important;word-wrap:break-word;word-break:break-all;">邮箱：
> [email protected]</div>
>                             </td>
>                         </tr>
>                 </tbody>
>             </table>
>         </a>
> </body>
> </html>

回复