@Sebastian: Getting memory away from the JVM is tricky always, completely independent of pre-allocation of managed memory or lazy allocation.
But here is something that may work: - Start Flink in streaming mode - that will make it allocate managed memory lazily - Set the memory to offheap memory. That way the JVM heap is small. The off-heap memory is returned when no longer used deallocated - this releases memory much better than JVM shrinking the heap. On Wed, Dec 9, 2015 at 10:06 AM, Fabian Hueske <fhue...@gmail.com> wrote: > Hi Sebastian, > > There is no way to return memory from a Flink process except shutting the > process down. > I think YARN could help in your setup. In a YARN setup, you can flexibly > start and stop Flink sessions with different configurations (memory, TMs, > slots) or run a single job. When running a single job, Flink will allocate > resources and free them after the job is done. > > Best, Fabian > > 2015-12-09 9:46 GMT+01:00 Kruse, Sebastian <sebastian.kr...@hpi.de>: > >> Hi everyone, >> >> >> I am currently looking into how Flink can coexist and interoperate with >> other frameworks in a cluster, such as plain single-machine processes >> or Spark. Tachyon seems to be nice solution to exchange data between >> them. >> >> >> However, I think it is a problem that Flink's taskmanagers allocate their >> managed >> memory upfront - in contrast to Spark, as far as I know. If I want a >> taskmanager >> to yield its main memory, so that another process can use that memory, is >> there any other option besides shutting that taskmanager down? Would it be >> beneficial to use YARN? >> >> Thanks for your help! >> >> >> Cheers, >> >> Sebastian >> > >
