Hi Anand, I think that Till is the best person to answer your question.
Cheers, Kostas > On Oct 5, 2018, at 3:44 PM, anand.gopin...@ubs.com wrote: > > Hi , > I had a question with respect flink memory management / overspill to /tmp. > > In the docs > (https://ci.apache.org/projects/flink/flink-docs-release-1.6/ops/config.html#configuring-temporary-io-directories > > <https://ci.apache.org/projects/flink/flink-docs-release-1.6/ops/config.html#configuring-temporary-io-directories>) > it says: Although Flink aims to process as much data in main memory as > possible, it is not uncommon that more data needs to be processed than memory > is available. Flink’s runtime is designed to write temporary data to disk to > handle these situations.... > > In a flink job that processes a couple streams of 1M events in a windowed > co group function with parallelism 8 - we see 8 dirs created in /tmp with > 100s of Meg of data, the name of each dir seems aligned to the data for each > parallel thread windowing against the co-group operator > > e.g. > bash-4.2$ du -sh * > 0 flink-dist-cache-a4a69215-665a-4c3c-8d90-416cbe192f26 > 352M flink-io-9033517c-ac92-4baa-9e59-79bc80c72a9e > 4.0K localState > 7.2M rocksdb-lib-03d9460b15e6bf6af4f3d9b0ff7980c3 > > bash-4.2$ du -sh flink-io-9033517c-ac92-4baa-9e59-79bc80c72a9e/* > ... > 36M > flink-io-9033517c-ac92-4baa-9e59-79bc80c72a9e/job_cf2dca7843dd6b6296aa1a9d15a1d435_op_WindowOperator_014556c228cb5344d41861769d2bbbc1__1_8__uuid_93307150-4f62-4b06-a71e-0230360f7d86 > 36M > flink-io-9033517c-ac92-4baa-9e59-79bc80c72a9e/job_cf2dca7843dd6b6296aa1a9d15a1d435_op_WindowOperator_014556c228cb5344d41861769d2bbbc1__2_8__uuid_7b2f8957-7044-4bb3-869e-28843bd737a1 > 36M > flink-io-9033517c-ac92-4baa-9e59-79bc80c72a9e/job_cf2dca7843dd6b6296aa1a9d15a1d435_op_WindowOperator_014556c228cb5344d41861769d2bbbc1__3_8__uuid_54306a44-7e06-45ae-ba0e-4649887bca7e > ... > > I was wondering can / should this 'over spill' be avoided by increasing the > heap of the task manager or another config or should I not worry about it? > Is there more information/docs on how this data is used/ cleaned up & what is > the cost of this overspill to latency/ checkpointing? Any impact I should be > aware of? > > thanks > Anand > > Visit our website at http://www.ubs.com <http://www.ubs.com/> > > This message contains confidential information and is intended only > for the individual named. If you are not the named addressee you > should not disseminate, distribute or copy this e-mail. Please > notify the sender immediately by e-mail if you have received this > e-mail by mistake and delete this e-mail from your system. > > E-mails are not encrypted and cannot be guaranteed to be secure or > error-free as information could be intercepted, corrupted, lost, > destroyed, arrive late or incomplete, or contain viruses. The sender > therefore does not accept liability for any errors or omissions in the > contents of this message which arise as a result of e-mail transmission. > If verification is required please request a hard-copy version. This > message is provided for informational purposes and should not be > construed as a solicitation or offer to buy or sell any securities > or related financial instruments. > > UBS Limited is a company limited by shares incorporated in the United > Kingdom registered in England and Wales with number 2035362. > Registered Office: 5 Broadgate, London EC2M 2QS > UBS Limited is authorised by the Prudential Regulation Authority > and regulated by the Financial Conduct Authority and the Prudential > Regulation Authority. > > UBS AG is a public company incorporated with limited liability in > Switzerland domiciled in the Canton of Basel-City and the Canton of > Zurich respectively registered at the Commercial Registry offices in > those Cantons with new Identification No: CHE-101.329.561 as from 18 > December 2013 (and prior to 18 December 2013 with Identification > No: CH-270.3.004.646-4) and having respective head offices at > Aeschenvorstadt 1, 4051 Basel and Bahnhofstrasse 45, 8001 Zurich, > Switzerland and is authorised and regulated by the Financial Market > Supervisory Authority in Switzerland. Registered in the United > Kingdom as a foreign company with No: FC021146 and having a UK > Establishment registered at Companies House, Cardiff, with > No: BR 004507. The principal office of UK Establishment: > 5 Broadgate, London EC2M 2QS. In the United Kingdom, UBS AG is > authorised by the Prudential Regulation Authority and subject to > regulation by the Financial Conduct Authority and limited regulation > by the Prudential Regulation Authority. Details about the extent > of our regulation by the Prudential Regulation Authority are > available from us on request. > > UBS Business Solutions AG is a public company incorporated with > limited liability in Switzerland domiciled in the Canton of Zurich > registered at the Commercial Registry office with Identification > No: CHE-262.289.477 and having its head office at Bahnhofstrasse 45, > 8001 Zurich, Switzerland. Registered in the United Kingdom as a > foreign company with No: FC034139 and having a UK Establishment > registered at Companies House, Cardiff, with No: BR019277. The > principal office of UK Establishment: 5 Broadgate London EC2M 2QS. > > UBS reserves the right to retain all messages. Messages are protected > and accessed only in legally justified cases.
