Unfortunately, I didn't have time to work on this yet and it looks like I won't have time this week.
I've created the following issue to track the implementation with two sub tasks: https://issues.apache.org/jira/browse/FLINK-5779 If someone else wants to take this up in the meantime, feel free to do so. Otherwise, I will try to have a look next week. I can certainly help with the initial setup if there are questions. On Tue, Feb 7, 2017 at 3:14 PM, Greg Hogan <c...@greghogan.com> wrote: > +1 and thanks for volunteering for an initial implementation. > > Looking forward to auto-scaling of network buffers. > > On Tue, Feb 7, 2017 at 3:04 AM, Ufuk Celebi <u...@apache.org> wrote: > >> I fully agree with you Greg. >> >> Since this is doomed to get out of sync again very shortly after clean up, >> I vote to automate this. Stephan introduced the ConfigOption type, which >> makes it easy to define the options. It's already planned to migrate all >> configuration options from ConfigConstants to this approach. >> >> For an example see here: https://github.com/apache/flink/blob/master/ >> flink-core/src/main/java/org/apache/flink/configuration/ >> HighAvailabilityOptions.java >> >> I think that it is possible to build the configuration docs page from this >> with reasonable effort. >> >> This would translate the task to: >> 1) Automate ConfigOption to HTML/Markdown generation >> 2) Extend ConfigOption with description fields >> 3) Migrate ConfigConstants to ConfigOptions >> >> I would also volunteer to take a first stab at this. >> >> Regarding the network buffers: +1 to your suggestion. Nico (cc'd) is >> starting to work on automating the network buffer configuration in order to >> get rid of any manual tuning for most users (because of the issues you >> described + streaming and batch jobs require different tuning, which >> complicates things even more). >> >> – Ufuk >> >> On 6 February 2017 at 19:21:28, Greg Hogan (c...@greghogan.com) wrote: >> > > Hi devs, >> > >> > Flink's Configuration page [1] has grown intimidatingly long >> > and complex. >> > Options are described across three main sections: common options >> > (single >> > section), advanced options (multiple sections), and full reference. >> > The >> > trailing "background" section further describes the most impactful >> > options >> > in much greater detail. >> > >> > Several recent tickets, and a few outstanding, have added missing >> > options >> > to the configuration documentation. I'd like to propose a goal >> > of >> > organizing all options in the full reference into alphabetized, >> > tabular >> > form (one table per section), much like the system metrics [2]. >> > Columns >> > would be option name, description, and default value. >> > >> > The common and advanced sections could also be converted to tabular >> > form >> > with the exception of Kerberos-based Security. Missing options >> > would be >> > added to the full reference. >> > >> > Lastly, the simple heuristic for configuring network buffers >> > has prompted >> > many questions on the mailing list. With the 1.3 release the total >> > and >> > number of available buffers is reported through metrics and >> > in the web >> > dashboard. My experience has been that the number of required >> > buffers is >> > highly dependent on job topology and cluster performance. I >> > propose keeping >> > the simple heuristic and description while directing users >> > to monitor the >> > balance of available buffers. >> > >> > Greg >> > >> > [1] https://ci.apache.org/projects/flink/flink-docs- >> master/setup/config.html >> > [2] >> > https://ci.apache.org/projects/flink/flink-docs- >> master/monitoring/metrics.html#system-metrics >> > [3] >> > https://ci.apache.org/projects/flink/flink-docs- >> master/setup/config.html#configuring-the-network-buffers >> >>
