实时任务flink on yarn运行,实时同步MySQL表到StarRocks经常遇到的内存问题:十万级别的数据量使用jm
1024mb,tm1024mb启动会报连接超时的问题,tm改成2048mb就会同步成功,但是源表十万级别数据量tm用2048mb内存非常浪费,看jobmanager日志,主要原因应该在chunk上,因为源表主键id的最大值为190562215,最小id为
1,可能是因为chunk太大导致内存不够,这种应该怎么优化下呢?
The information transmitted is intended solely for the use of the addressee and
may contain confidential and/or privileged material. Any unauthorized
disclosure, reproduction, distribution, dissemination, or taking of any action
in reliance upon, this information by persons or entities other than the
intended recipient is prohibited. If you received this in error, please contact
the sender and delete the email together with any material attached (if any)
completely from any device immediately. Unless otherwise stated, any views or
opinions expressed in this email are solely those of the author and do not
necessarily represent those of Haier Group.
本邮件可能包含敏感信息且仅限于发给指定的收件人,任何未经授权泄露、复制、散布或传播此信息的行为将被禁止。如果您错误收到了此邮件,请及时告知发送者并立即从所有设备中完全删除此邮件及附件。除非另有说明,否则邮件中可能包含的观点或建议仅代表发件者本人,并不代表海尔集团。