Hi Dongwon, I see that you are using the latest master (Flink 1.6-SNAPSHOT). This is a known problem in the new FLIP-6 mode. The ResourceManager tries to allocate too many resources, basically on TM per required slot, i.e., it does not take the number of slots per TM into account. The resources are not used and should be returned to YARN after a timeout.
I couldn't find a JIRA issue to point you to. Till (in CC) should know more details about this problem. Best, Fabian 2018-05-05 12:50 GMT+02:00 Dongwon Kim <eastcirc...@gmail.com>: > I'm testing per-job cluster on YARN. > > I just need to launch 7 TMs each with 50GB memory (total 350GB) but Flink > makes more resource request to YARN than necessary. > > All of the remaining memory in YARN, around 370GB, are reserved by the > Flink job, which I can check in YARN UI. > > The remaining memory is not used but reserved; that’s very weird. > > Attached is JM log. > > Any help would be greatly appreciated! > > Thanks, > > - Dongwon > >
