Re: status of autotuning freebsd for 9.2
On 16.08.2013 10:29, Andre Oppermann wrote: On 16.08.2013 08:32, Alfred Perlstein wrote: Andre, I'm kind of bummed out this didn't make it into 9.2, I'm wondering can I commit this to 9-stable now? (or is it already in?) It didn't make it because there was only sparse feedback after the call for testers. There were a couple of replies that it is being tested but no statements either way if it was good or not. Hence I erred on the side of caution and refrained from committing it. Revisiting the history of this after vacation absence actually shows that we straddled the release code freeze deadline and you had provided good testing feedback. However the MFC got rejected by RE on the fear of introducing unknown regressions into the release process. Would you do the honors? Yes, will do later today. Committed to stable/9 as r254515. Let me know if there are any issues. -- Andre ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: status of autotuning freebsd for 9.2
On Aug 19, 2013, at 2:52 AM, Andre Oppermann an...@freebsd.org wrote: On 16.08.2013 10:29, Andre Oppermann wrote: On 16.08.2013 08:32, Alfred Perlstein wrote: Andre, I'm kind of bummed out this didn't make it into 9.2, I'm wondering can I commit this to 9-stable now? (or is it already in?) It didn't make it because there was only sparse feedback after the call for testers. There were a couple of replies that it is being tested but no statements either way if it was good or not. Hence I erred on the side of caution and refrained from committing it. Revisiting the history of this after vacation absence actually shows that we straddled the release code freeze deadline and you had provided good testing feedback. However the MFC got rejected by RE on the fear of introducing unknown regressions into the release process. Would you do the honors? Yes, will do later today. Committed to stable/9 as r254515. Let me know if there are any issues. Thanks Andre. Maybe we can do a point release/patch release with this in a few weeks for 9.2.1 or 9.2p1 because 9.2 out of the box performance is abysmal not only in networking but also disk as maxvnodes is clipped way too small even with plenty of ram. -- Andre ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: status of autotuning freebsd for 9.2
On Mon, Aug 19, 2013 at 10:26 AM, Alfred Perlstein alf...@ixsystems.comwrote: On Aug 19, 2013, at 2:52 AM, Andre Oppermann an...@freebsd.org wrote: On 16.08.2013 10:29, Andre Oppermann wrote: On 16.08.2013 08:32, Alfred Perlstein wrote: Andre, I'm kind of bummed out this didn't make it into 9.2, I'm wondering can I commit this to 9-stable now? (or is it already in?) It didn't make it because there was only sparse feedback after the call for testers. There were a couple of replies that it is being tested but no statements either way if it was good or not. Hence I erred on the side of caution and refrained from committing it. Revisiting the history of this after vacation absence actually shows that we straddled the release code freeze deadline and you had provided good testing feedback. However the MFC got rejected by RE on the fear of introducing unknown regressions into the release process. Would you do the honors? Yes, will do later today. Committed to stable/9 as r254515. Let me know if there are any issues. Thanks Andre. Maybe we can do a point release/patch release with this in a few weeks for 9.2.1 or 9.2p1 because 9.2 out of the box performance is abysmal not only in networking but also disk as maxvnodes is clipped way too small even with plenty of ram. So your saying, 9.2-RELEASE performance suffers degradation against say 9.1 ?? are you referring to with this patch enabled? or just in general 9.2-RELEASE -- Andre ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: status of autotuning freebsd for 9.2
Performance is bad for large memory requirements period. Vnodes and mbufs on a machine with 24gb ram is limited to the same amount as a machine with less than 4GB ram. This was fixed in head but not merged back in time. This results in poor out of the box performance on 10gige and servers with high vnode requirements. Sent from my iPhone On Aug 19, 2013, at 7:30 AM, Outback Dingo outbackdi...@gmail.com wrote: On Mon, Aug 19, 2013 at 10:26 AM, Alfred Perlstein alf...@ixsystems.com wrote: On Aug 19, 2013, at 2:52 AM, Andre Oppermann an...@freebsd.org wrote: On 16.08.2013 10:29, Andre Oppermann wrote: On 16.08.2013 08:32, Alfred Perlstein wrote: Andre, I'm kind of bummed out this didn't make it into 9.2, I'm wondering can I commit this to 9-stable now? (or is it already in?) It didn't make it because there was only sparse feedback after the call for testers. There were a couple of replies that it is being tested but no statements either way if it was good or not. Hence I erred on the side of caution and refrained from committing it. Revisiting the history of this after vacation absence actually shows that we straddled the release code freeze deadline and you had provided good testing feedback. However the MFC got rejected by RE on the fear of introducing unknown regressions into the release process. Would you do the honors? Yes, will do later today. Committed to stable/9 as r254515. Let me know if there are any issues. Thanks Andre. Maybe we can do a point release/patch release with this in a few weeks for 9.2.1 or 9.2p1 because 9.2 out of the box performance is abysmal not only in networking but also disk as maxvnodes is clipped way too small even with plenty of ram. So your saying, 9.2-RELEASE performance suffers degradation against say 9.1 ?? are you referring to with this patch enabled? or just in general 9.2-RELEASE -- Andre ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: status of autotuning freebsd for 9.2
On Mon, Aug 19, 2013 at 12:05 PM, Alfred Perlstein alf...@ixsystems.comwrote: Performance is bad for large memory requirements period. Vnodes and mbufs on a machine with 24gb ram is limited to the same amount as a machine with less than 4GB ram. This was fixed in head but not merged back in time. is there a patch set i can backport on my own, do we know what revision(s) are required? Ive got boxes with 128GB and 10Gbe Intel... so im willing to do some work.. This results in poor out of the box performance on 10gige and servers with high vnode requirements. Sent from my iPhone On Aug 19, 2013, at 7:30 AM, Outback Dingo outbackdi...@gmail.com wrote: On Mon, Aug 19, 2013 at 10:26 AM, Alfred Perlstein alf...@ixsystems.comwrote: On Aug 19, 2013, at 2:52 AM, Andre Oppermann an...@freebsd.org wrote: On 16.08.2013 10:29, Andre Oppermann wrote: On 16.08.2013 08:32, Alfred Perlstein wrote: Andre, I'm kind of bummed out this didn't make it into 9.2, I'm wondering can I commit this to 9-stable now? (or is it already in?) It didn't make it because there was only sparse feedback after the call for testers. There were a couple of replies that it is being tested but no statements either way if it was good or not. Hence I erred on the side of caution and refrained from committing it. Revisiting the history of this after vacation absence actually shows that we straddled the release code freeze deadline and you had provided good testing feedback. However the MFC got rejected by RE on the fear of introducing unknown regressions into the release process. Would you do the honors? Yes, will do later today. Committed to stable/9 as r254515. Let me know if there are any issues. Thanks Andre. Maybe we can do a point release/patch release with this in a few weeks for 9.2.1 or 9.2p1 because 9.2 out of the box performance is abysmal not only in networking but also disk as maxvnodes is clipped way too small even with plenty of ram. So your saying, 9.2-RELEASE performance suffers degradation against say 9.1 ?? are you referring to with this patch enabled? or just in general 9.2-RELEASE -- Andre ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: status of autotuning freebsd for 9.2
On 19.08.2013 18:09, Outback Dingo wrote: On Mon, Aug 19, 2013 at 12:05 PM, Alfred Perlstein alf...@ixsystems.com mailto:alf...@ixsystems.com wrote: Performance is bad for large memory requirements period. Vnodes and mbufs on a machine with 24gb ram is limited to the same amount as a machine with less than 4GB ram. This was fixed in head but not merged back in time. is there a patch set i can backport on my own, do we know what revision(s) are required? Ive got boxes with 128GB and 10Gbe Intel... so im willing to do some work.. I have committed it to 9-stable this morning with r254515. No backporting necessary. -- Andre This results in poor out of the box performance on 10gige and servers with high vnode requirements. Sent from my iPhone On Aug 19, 2013, at 7:30 AM, Outback Dingo outbackdi...@gmail.com mailto:outbackdi...@gmail.com wrote: On Mon, Aug 19, 2013 at 10:26 AM, Alfred Perlstein alf...@ixsystems.com mailto:alf...@ixsystems.com wrote: On Aug 19, 2013, at 2:52 AM, Andre Oppermann an...@freebsd.org mailto:an...@freebsd.org wrote: On 16.08.2013 10:29, Andre Oppermann wrote: On 16.08.2013 08:32, Alfred Perlstein wrote: Andre, I'm kind of bummed out this didn't make it into 9.2, I'm wondering can I commit this to 9-stable now? (or is it already in?) It didn't make it because there was only sparse feedback after the call for testers. There were a couple of replies that it is being tested but no statements either way if it was good or not. Hence I erred on the side of caution and refrained from committing it. Revisiting the history of this after vacation absence actually shows that we straddled the release code freeze deadline and you had provided good testing feedback. However the MFC got rejected by RE on the fear of introducing unknown regressions into the release process. Would you do the honors? Yes, will do later today. Committed to stable/9 as r254515. Let me know if there are any issues. Thanks Andre. Maybe we can do a point release/patch release with this in a few weeks for 9.2.1 or 9.2p1 because 9.2 out of the box performance is abysmal not only in networking but also disk as maxvnodes is clipped way too small even with plenty of ram. So your saying, 9.2-RELEASE performance suffers degradation against say 9.1 ?? are you referring to with this patch enabled? or just in general 9.2-RELEASE -- Andre ___ freebsd-stable@freebsd.org mailto:freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org mailto:freebsd-stable-unsubscr...@freebsd.org ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: status of autotuning freebsd for 9.2
On Mon, Aug 19, 2013 at 12:17 PM, Andre Oppermann an...@freebsd.org wrote: On 19.08.2013 18:09, Outback Dingo wrote: On Mon, Aug 19, 2013 at 12:05 PM, Alfred Perlstein alf...@ixsystems.com mailto:alf...@ixsystems.com wrote: Performance is bad for large memory requirements period. Vnodes and mbufs on a machine with 24gb ram is limited to the same amount as a machine with less than 4GB ram. This was fixed in head but not merged back in time. is there a patch set i can backport on my own, do we know what revision(s) are required? Ive got boxes with 128GB and 10Gbe Intel... so im willing to do some work.. I have committed it to 9-stable this morning with r254515. No backporting necessary. Okay so wait, your saying the autotune commit this morning resolves Alfreds claims of abysmal performance in general, or is there other additional fixes in head aside from the autotune hes mentioning -- Andre This results in poor out of the box performance on 10gige and servers with high vnode requirements. Sent from my iPhone On Aug 19, 2013, at 7:30 AM, Outback Dingo outbackdi...@gmail.com mailto:outbackdi...@gmail.com** wrote: On Mon, Aug 19, 2013 at 10:26 AM, Alfred Perlstein alf...@ixsystems.com mailto:alf...@ixsystems.com wrote: On Aug 19, 2013, at 2:52 AM, Andre Oppermann an...@freebsd.org mailto:an...@freebsd.org wrote: On 16.08.2013 10:29, Andre Oppermann wrote: On 16.08.2013 08:32, Alfred Perlstein wrote: Andre, I'm kind of bummed out this didn't make it into 9.2, I'm wondering can I commit this to 9-stable now? (or is it already in?) It didn't make it because there was only sparse feedback after the call for testers. There were a couple of replies that it is being tested but no statements either way if it was good or not. Hence I erred on the side of caution and refrained from committing it. Revisiting the history of this after vacation absence actually shows that we straddled the release code freeze deadline and you had provided good testing feedback. However the MFC got rejected by RE on the fear of introducing unknown regressions into the release process. Would you do the honors? Yes, will do later today. Committed to stable/9 as r254515. Let me know if there are any issues. Thanks Andre. Maybe we can do a point release/patch release with this in a few weeks for 9.2.1 or 9.2p1 because 9.2 out of the box performance is abysmal not only in networking but also disk as maxvnodes is clipped way too small even with plenty of ram. So your saying, 9.2-RELEASE performance suffers degradation against say 9.1 ?? are you referring to with this patch enabled? or just in general 9.2-RELEASE -- Andre __**_ freebsd-stable@freebsd.org mailto:freebsd-stable@**freebsd.orgfreebsd-stable@freebsd.org mailing list http://lists.freebsd.org/**mailman/listinfo/freebsd-**stablehttp://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscribe@** freebsd.org freebsd-stable-unsubscr...@freebsd.org mailto:freebsd-stable-**unsubscr...@freebsd.orgfreebsd-stable-unsubscr...@freebsd.org ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
RE: status of autotuning freebsd for 9.2
-Original Message- From: owner-freebsd-sta...@freebsd.org [mailto:owner-freebsd-sta...@freebsd.org] On Behalf Of Alfred Perlstein Sent: Tuesday, 20 August 2013 2:05 AM To: Outback Dingo Cc: r...@freebsd.org; sta...@freebsd.org; Andre Oppermann; nones...@longcount.org Subject: Re: status of autotuning freebsd for 9.2 Performance is bad for large memory requirements period. Vnodes and mbufs on a machine with 24gb ram is limited to the same amount as a machine with less than 4GB ram. This was fixed in head but not merged back in time. This results in poor out of the box performance on 10gige and servers with high vnode requirements. Sent from my iPhone On Aug 19, 2013, at 7:30 AM, Outback Dingo outbackdi...@gmail.com wrote: On Mon, Aug 19, 2013 at 10:26 AM, Alfred Perlstein alf...@ixsystems.com wrote: On Aug 19, 2013, at 2:52 AM, Andre Oppermann an...@freebsd.org wrote: On 16.08.2013 10:29, Andre Oppermann wrote: On 16.08.2013 08:32, Alfred Perlstein wrote: Andre, I'm kind of bummed out this didn't make it into 9.2, I'm wondering can I commit this to 9-stable now? (or is it already in?) It didn't make it because there was only sparse feedback after the call for testers. There were a couple of replies that it is being tested but no statements either way if it was good or not. Hence I erred on the side of caution and refrained from committing it. Revisiting the history of this after vacation absence actually shows that we straddled the release code freeze deadline and you had provided good testing feedback. However the MFC got rejected by RE on the fear of introducing unknown regressions into the release process. Would you do the honors? Yes, will do later today. Committed to stable/9 as r254515. Let me know if there are any issues. Thanks Andre. Maybe we can do a point release/patch release with this in a few weeks for 9.2.1 or 9.2p1 because 9.2 out of the box performance is abysmal not only in networking but also disk as maxvnodes is clipped way too small even with plenty of ram. So your saying, 9.2-RELEASE performance suffers degradation against say 9.1 ?? are you referring to with this patch enabled? or just in general 9.2-RELEASE -- Andre ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org It might be relevant that there were performance changes to nullfs (caching) code back in January and updated in May by Kib. Because I use jails and nullfs extensively, the nullfs enhancement demanded an increase in maxvodes, otherwise performance degraded, quite badly. Tripling the default suited my needs on 4GB systems, but I don't have an algorithmic recommendation; as for me it depends on the role/purpose of the server. If vnodes is an issue *and* you use mount_nullfs, another approach is to disable caching via mount_nullfs -o nocache as this may help to narrow the cause. Ref: http://svnweb.freebsd.org/base/stable/9/sys/fs/nullfs/null_subr.c?view=log http://lists.freebsd.org/pipermail/svn-src-stable-9/2013-May/004531.html And thank-you for your work on http://lists.freebsd.org/pipermail/svn-src-stable-9/2013-August/005307.html Regards, Dewayne. ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: status of autotuning freebsd for 9.2
On 8/19/13 9:19 AM, Outback Dingo wrote: On Mon, Aug 19, 2013 at 12:17 PM, Andre Oppermann an...@freebsd.org mailto:an...@freebsd.org wrote: On 19.08.2013 18:09, Outback Dingo wrote: On Mon, Aug 19, 2013 at 12:05 PM, Alfred Perlstein alf...@ixsystems.com mailto:alf...@ixsystems.com mailto:alf...@ixsystems.com mailto:alf...@ixsystems.com wrote: Performance is bad for large memory requirements period. Vnodes and mbufs on a machine with 24gb ram is limited to the same amount as a machine with less than 4GB ram. This was fixed in head but not merged back in time. is there a patch set i can backport on my own, do we know what revision(s) are required? Ive got boxes with 128GB and 10Gbe Intel... so im willing to do some work.. I have committed it to 9-stable this morning with r254515. No backporting necessary. Okay so wait, your saying the autotune commit this morning resolves Alfreds claims of abysmal performance in general, Yes. or is there other additional fixes in head aside from the autotune hes mentioning Well of course head has cool code, but this was a big problem. -Alfred ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: status of autotuning freebsd for 9.2
Andre, I'm kind of bummed out this didn't make it into 9.2, I'm wondering can I commit this to 9-stable now? (or is it already in?) Would you do the honors? -Alfred On 7/8/13 7:37 AM, Andre Oppermann wrote: On 07.07.2013 20:24, Alfred Perlstein wrote: On 7/7/13 1:34 AM, Andre Oppermann wrote: Can you help me with with testing? Yes. Please give me your proposed changes and I'll stand up a machine and give feedback. http://people.freebsd.org/~andre/mfc-autotuning-20130708.diff This is functional bundle MFC of these original commits: MFC r242029 (alfred): Allow autotune maxusers 384 on 64 bit machines. MFC r242847 (alfred): Allow maxusers to scale on machines with large address space. MFC r243631 (andre): Base the mbuf related limits on the available physical memory or kernel memory, whichever is lower. The overall mbuf related memory limit must be set so that mbufs (and clusters of various sizes) can't exhaust physical RAM or KVM. At the same time divorce maxfiles from maxusers and set maxfiles to physpages / 8 with a floor based on maxusers. This way busy servers can make use of the significantly increased mbuf limits with a much larger number of open sockets. MFC r243639 (andre): Complete r243631 by applying the remainder of kern_mbuf.c that got lost while merging into the commit tree. MFC r243668 (andre): Using a long is the wrong type to represent the realmem and maxmbufmem variable as they may overflow on i386/PAE and i386 with 2GB RAM. MFC r243995, r243996, r243997 (pjd): Style cleanups, Make use of the fact that uma_zone_set_max(9) already returns actual limit set. MFC r244080 (andre): Prevent long type overflow of realmem calculation on ILP32 by forcing calculation to be in quad_t space. Fix style issue with second parameter to qmin(). MFC r245469 (alfred): Do not autotune ncallout to be greater than 18508. MFC r245575 (andre): Move the mbuf memory limit calculations from init_param2() to tunable_mbinit() where it is next to where it is used later. MFC r246207 (andre): Remove unused VM_MAX_AUTOTUNE_NMBCLUSTERS define. MFC r249843 (andre): Base the calculation of maxmbufmem in part on kmem_map size instead of kernel_map size to prevent kernel memory exhaustion by mbufs and a subsequent panic on physical page allocation failure. -- Alfred Perlstein ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: status of autotuning freebsd for 9.2
On 16.08.2013 08:32, Alfred Perlstein wrote: Andre, I'm kind of bummed out this didn't make it into 9.2, I'm wondering can I commit this to 9-stable now? (or is it already in?) It didn't make it because there was only sparse feedback after the call for testers. There were a couple of replies that it is being tested but no statements either way if it was good or not. Hence I erred on the side of caution and refrained from committing it. Would you do the honors? Yes, will do later today. -- Andre -Alfred On 7/8/13 7:37 AM, Andre Oppermann wrote: On 07.07.2013 20:24, Alfred Perlstein wrote: On 7/7/13 1:34 AM, Andre Oppermann wrote: Can you help me with with testing? Yes. Please give me your proposed changes and I'll stand up a machine and give feedback. http://people.freebsd.org/~andre/mfc-autotuning-20130708.diff This is functional bundle MFC of these original commits: MFC r242029 (alfred): Allow autotune maxusers 384 on 64 bit machines. MFC r242847 (alfred): Allow maxusers to scale on machines with large address space. MFC r243631 (andre): Base the mbuf related limits on the available physical memory or kernel memory, whichever is lower. The overall mbuf related memory limit must be set so that mbufs (and clusters of various sizes) can't exhaust physical RAM or KVM. At the same time divorce maxfiles from maxusers and set maxfiles to physpages / 8 with a floor based on maxusers. This way busy servers can make use of the significantly increased mbuf limits with a much larger number of open sockets. MFC r243639 (andre): Complete r243631 by applying the remainder of kern_mbuf.c that got lost while merging into the commit tree. MFC r243668 (andre): Using a long is the wrong type to represent the realmem and maxmbufmem variable as they may overflow on i386/PAE and i386 with 2GB RAM. MFC r243995, r243996, r243997 (pjd): Style cleanups, Make use of the fact that uma_zone_set_max(9) already returns actual limit set. MFC r244080 (andre): Prevent long type overflow of realmem calculation on ILP32 by forcing calculation to be in quad_t space. Fix style issue with second parameter to qmin(). MFC r245469 (alfred): Do not autotune ncallout to be greater than 18508. MFC r245575 (andre): Move the mbuf memory limit calculations from init_param2() to tunable_mbinit() where it is next to where it is used later. MFC r246207 (andre): Remove unused VM_MAX_AUTOTUNE_NMBCLUSTERS define. MFC r249843 (andre): Base the calculation of maxmbufmem in part on kmem_map size instead of kernel_map size to prevent kernel memory exhaustion by mbufs and a subsequent panic on physical page allocation failure. ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: status of autotuning freebsd for 9.2
Bezüglich Pascal Drecker's Nachricht vom 16.07.2013 21:42 (localtime): ... IMHO, this is considered a new feature, and not a critical bug fix. re@ asked from the start of the code slush to avoid new features, and at this point, it is too late. It is not worth introducing possible regressions, which will only delay the 9.2-RELEASE. Glen OK, then we need a release notes telling people a sane value for nmbclusters and friends so that they know how to make 10gigE work. I'll poll my team for a value if someone else has one, that would be even better. -- Alfred Perlstein VP Software Engineering, iXsystems Is there a possibility that a separate unofficial patch set could be released for people who want the autotuning but do not want to run 9 stable after 9.2 is released. I would like the autotuning, but i am a little reluctent to use other stable stuff i will get when tracking stable. Regards Johan Hi, I think that's a good point. In our company, it�s not allowed to use the stable tree for any production system. Little and useful patches are still allowed. Having a central point with a description of each patch it would be much easier to update the release version with the needed patches. You're welcome using my deploy-tools patchsets. I'm deploying RELENG only, but with local patchset-policy. Originally, these are automtically handled during build-process with deploy-tools, but of course you can selective/manually apply the desired patches from the local-patches directory: ftp://ftp.omnilan.de/pub/FreeBSD/OmniLAN/deploy-tools/ Best regards, -Harry signature.asc Description: OpenPGP digital signature
Re: status of autotuning freebsd for 9.2
On Fri, Aug 16, 2013 at 10:08 AM, Harald Schmalzbauer h.schmalzba...@omnilan.de wrote: Bezüglich Pascal Drecker's Nachricht vom 16.07.2013 21:42 (localtime): ... IMHO, this is considered a new feature, and not a critical bug fix. re@ asked from the start of the code slush to avoid new features, and at this point, it is too late. It is not worth introducing possible regressions, which will only delay the 9.2-RELEASE. Glen OK, then we need a release notes telling people a sane value for nmbclusters and friends so that they know how to make 10gigE work. I'll poll my team for a value if someone else has one, that would be even better. -- Alfred Perlstein VP Software Engineering, iXsystems Is there a possibility that a separate unofficial patch set could be released for people who want the autotuning but do not want to run 9 stable after 9.2 is released. I would like the autotuning, but i am a little reluctent to use other stable stuff i will get when tracking stable. Regards Johan Hi, I think that's a good point. In our company, it�s not allowed to use the stable tree for any production system. Little and useful patches are still allowed. Having a central point with a description of each patch it would be much easier to update the release version with the needed patches. You're welcome using my deploy-tools patchsets. I'm deploying RELENG only, but with local patchset-policy. Originally, these are automtically handled during build-process with deploy-tools, but of course you can selective/manually apply the desired patches from the local-patches directory: ftp://ftp.omnilan.de/pub/FreeBSD/OmniLAN/deploy-tools/ Best regards, can you maybe define what exactly, this is and or where the specific patches are derived? Ive seen the autotune but whats the rest in here?? -Harry ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: status of autotuning freebsd for 9.2
Op maandag 15 juli 2013 schreef Alfred Perlstein (alf...@ixsystems.com) het volgende: On 7/15/13 7:13 AM, Glen Barber wrote: On Mon, Jul 15, 2013 at 05:48:40AM -0700, Alfred Perlstein wrote: On 7/15/13 5:44 AM, Andre Oppermann wrote: On 15.07.2013 08:38, Andre Oppermann wrote: On 13.07.2013 09:47, Alfred Perlstein wrote: Andre, we have a number of people running this patch in the following configurations: 6-8GB ram + 10gigE ethernet using iozone over NFS. As you haven't seen any problems yet I've asked RE to green light the MFC. RE has rejected the MFC out of fears for unexpected regressions. That is unfortunate. I guess re@ doesn't understand that FreeBSD 9.2 will be unusable out of the box for doing 10gigE for more than a few microseconds. Can we not just do my original patch that has the check for 64bit pointers before unscaling maxusers? That would be dirt simple and just work with minimal risk. IMHO, this is considered a new feature, and not a critical bug fix. re@ asked from the start of the code slush to avoid new features, and at this point, it is too late. It is not worth introducing possible regressions, which will only delay the 9.2-RELEASE. Glen OK, then we need a release notes telling people a sane value for nmbclusters and friends so that they know how to make 10gigE work. I'll poll my team for a value if someone else has one, that would be even better. -- Alfred Perlstein VP Software Engineering, iXsystems Is there a possibility that a separate unofficial patch set could be released for people who want the autotuning but do not want to run 9 stable after 9.2 is released. I would like the autotuning, but i am a little reluctent to use other stable stuff i will get when tracking stable. Regards Johan __**_ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/**mailman/listinfo/freebsd-**stablehttp://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Aw: Re: status of autotuning freebsd for 9.2
Op maandag 15 juli 2013 schreef Alfred Perlstein (alf...@ixsystems.com) het volgende: On 7/15/13 7:13 AM, Glen Barber wrote: On Mon, Jul 15, 2013 at 05:48:40AM -0700, Alfred Perlstein wrote: On 7/15/13 5:44 AM, Andre Oppermann wrote: On 15.07.2013 08:38, Andre Oppermann wrote: On 13.07.2013 09:47, Alfred Perlstein wrote: Andre, we have a number of people running this patch in the following configurations: 6-8GB ram + 10gigE ethernet using iozone over NFS. As you haven't seen any problems yet I've asked RE to green light the MFC. RE has rejected the MFC out of fears for unexpected regressions. That is unfortunate. I guess re@ doesn't understand that FreeBSD 9.2 will be unusable out of the box for doing 10gigE for more than a few microseconds. Can we not just do my original patch that has the check for 64bit pointers before unscaling maxusers? That would be dirt simple and just work with minimal risk. IMHO, this is considered a new feature, and not a critical bug fix. re@ asked from the start of the code slush to avoid new features, and at this point, it is too late. It is not worth introducing possible regressions, which will only delay the 9.2-RELEASE. Glen OK, then we need a release notes telling people a sane value for nmbclusters and friends so that they know how to make 10gigE work. I'll poll my team for a value if someone else has one, that would be even better. -- Alfred Perlstein VP Software Engineering, iXsystems Is there a possibility that a separate unofficial patch set could be released for people who want the autotuning but do not want to run 9 stable after 9.2 is released. I would like the autotuning, but i am a little reluctent to use other stable stuff i will get when tracking stable. Regards Johan Hi, I think that's a good point. In our company, it�s not allowed to use the stable tree for any production system. Little and useful patches are still allowed. Having a central point with a description of each patch it would be much easier to update the release version with the needed patches. Perhaps, each patch could also have a comment section and a state (experimental, almost stable, stable ...) or a counter for successfully and unsuccessfully deployments. Any objections? Regards, Pascal ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: status of autotuning freebsd for 9.2
On 13.07.2013 09:47, Alfred Perlstein wrote: Andre, we have a number of people running this patch in the following configurations: 6-8GB ram + 10gigE ethernet using iozone over NFS. As you haven't seen any problems yet I've asked RE to green light the MFC. -- Andre We're rolling it into the PCBSD rolling update as well. I'm thinking it makes sense to roll this into the 9.2 release to give us more scalability. -Alfred On 7/7/13 1:34 AM, Andre Oppermann wrote: On 07.07.2013 08:32, Alfred Perlstein wrote: Andre, Are you going to have time to MFC things from -current for auto-tuning -stable before 9.2? I simply ran out of time on Friday and MFCing such a big change requires more testing. I fear (maybe unnecessarily?) that we are about to ship yet another release that can't do basic 10gigE when sufficient memory exists. There was some debate with myself whether such a behavior changing MFC would be appropriate for a mid-stream stable release. I guess yes, though a number of people who currently set the parameters manually would have to remove their tuning settings. If you don't have time, then let me know and I'll see what I can do. Can you help me with with testing? ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: status of autotuning freebsd for 9.2
On 15.07.2013 08:38, Andre Oppermann wrote: On 13.07.2013 09:47, Alfred Perlstein wrote: Andre, we have a number of people running this patch in the following configurations: 6-8GB ram + 10gigE ethernet using iozone over NFS. As you haven't seen any problems yet I've asked RE to green light the MFC. RE has rejected the MFC out of fears for unexpected regressions. -- Andre ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: status of autotuning freebsd for 9.2
On 7/15/13 5:44 AM, Andre Oppermann wrote: On 15.07.2013 08:38, Andre Oppermann wrote: On 13.07.2013 09:47, Alfred Perlstein wrote: Andre, we have a number of people running this patch in the following configurations: 6-8GB ram + 10gigE ethernet using iozone over NFS. As you haven't seen any problems yet I've asked RE to green light the MFC. RE has rejected the MFC out of fears for unexpected regressions. That is unfortunate. I guess re@ doesn't understand that FreeBSD 9.2 will be unusable out of the box for doing 10gigE for more than a few microseconds. Can we not just do my original patch that has the check for 64bit pointers before unscaling maxusers? That would be dirt simple and just work with minimal risk. -- Alfred Perlstein VP Software Engineering, iXsystems ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: status of autotuning freebsd for 9.2
On Mon, Jul 15, 2013 at 05:48:40AM -0700, Alfred Perlstein wrote: On 7/15/13 5:44 AM, Andre Oppermann wrote: On 15.07.2013 08:38, Andre Oppermann wrote: On 13.07.2013 09:47, Alfred Perlstein wrote: Andre, we have a number of people running this patch in the following configurations: 6-8GB ram + 10gigE ethernet using iozone over NFS. As you haven't seen any problems yet I've asked RE to green light the MFC. RE has rejected the MFC out of fears for unexpected regressions. That is unfortunate. I guess re@ doesn't understand that FreeBSD 9.2 will be unusable out of the box for doing 10gigE for more than a few microseconds. Can we not just do my original patch that has the check for 64bit pointers before unscaling maxusers? That would be dirt simple and just work with minimal risk. IMHO, this is considered a new feature, and not a critical bug fix. re@ asked from the start of the code slush to avoid new features, and at this point, it is too late. It is not worth introducing possible regressions, which will only delay the 9.2-RELEASE. Glen pgpvtvpJkKESM.pgp Description: PGP signature
Re: status of autotuning freebsd for 9.2
On Mon, Jul 15, 2013 at 10:13 AM, Glen Barber g...@freebsd.org wrote: On Mon, Jul 15, 2013 at 05:48:40AM -0700, Alfred Perlstein wrote: On 7/15/13 5:44 AM, Andre Oppermann wrote: On 15.07.2013 08:38, Andre Oppermann wrote: On 13.07.2013 09:47, Alfred Perlstein wrote: Andre, we have a number of people running this patch in the following configurations: 6-8GB ram + 10gigE ethernet using iozone over NFS. As you haven't seen any problems yet I've asked RE to green light the MFC. RE has rejected the MFC out of fears for unexpected regressions. That is unfortunate. I guess re@ doesn't understand that FreeBSD 9.2 will be unusable out of the box for doing 10gigE for more than a few microseconds. Can we not just do my original patch that has the check for 64bit pointers before unscaling maxusers? That would be dirt simple and just work with minimal risk. IMHO, this is considered a new feature, and not a critical bug fix. re@ asked from the start of the code slush to avoid new features, and at this point, it is too late. It is not worth introducing possible regressions, which will only delay the 9.2-RELEASE. Its kinda sad that it wount be MFC'd though I understand, it does help 10Gbe environments Id give it a vote, and its perceived to have possible regressions, though it might need more testing other then a handful of users. we can always patch until its MFCd Glen ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: status of autotuning freebsd for 9.2
On 7/15/13 7:13 AM, Glen Barber wrote: On Mon, Jul 15, 2013 at 05:48:40AM -0700, Alfred Perlstein wrote: On 7/15/13 5:44 AM, Andre Oppermann wrote: On 15.07.2013 08:38, Andre Oppermann wrote: On 13.07.2013 09:47, Alfred Perlstein wrote: Andre, we have a number of people running this patch in the following configurations: 6-8GB ram + 10gigE ethernet using iozone over NFS. As you haven't seen any problems yet I've asked RE to green light the MFC. RE has rejected the MFC out of fears for unexpected regressions. That is unfortunate. I guess re@ doesn't understand that FreeBSD 9.2 will be unusable out of the box for doing 10gigE for more than a few microseconds. Can we not just do my original patch that has the check for 64bit pointers before unscaling maxusers? That would be dirt simple and just work with minimal risk. IMHO, this is considered a new feature, and not a critical bug fix. re@ asked from the start of the code slush to avoid new features, and at this point, it is too late. It is not worth introducing possible regressions, which will only delay the 9.2-RELEASE. Glen OK, then we need a release notes telling people a sane value for nmbclusters and friends so that they know how to make 10gigE work. I'll poll my team for a value if someone else has one, that would be even better. -- Alfred Perlstein VP Software Engineering, iXsystems ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: status of autotuning freebsd for 9.2
Andre, we have a number of people running this patch in the following configurations: 6-8GB ram + 10gigE ethernet using iozone over NFS. We're rolling it into the PCBSD rolling update as well. I'm thinking it makes sense to roll this into the 9.2 release to give us more scalability. -Alfred On 7/7/13 1:34 AM, Andre Oppermann wrote: On 07.07.2013 08:32, Alfred Perlstein wrote: Andre, Are you going to have time to MFC things from -current for auto-tuning -stable before 9.2? I simply ran out of time on Friday and MFCing such a big change requires more testing. I fear (maybe unnecessarily?) that we are about to ship yet another release that can't do basic 10gigE when sufficient memory exists. There was some debate with myself whether such a behavior changing MFC would be appropriate for a mid-stream stable release. I guess yes, though a number of people who currently set the parameters manually would have to remove their tuning settings. If you don't have time, then let me know and I'll see what I can do. Can you help me with with testing? -- Alfred Perlstein VP Software Engineering, iXsystems ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: status of autotuning freebsd for 9.2
On 08.07.2013 16:37, Andre Oppermann wrote: On 07.07.2013 20:24, Alfred Perlstein wrote: On 7/7/13 1:34 AM, Andre Oppermann wrote: Can you help me with with testing? Yes. Please give me your proposed changes and I'll stand up a machine and give feedback. http://people.freebsd.org/~andre/mfc-autotuning-20130708.diff Any feedback from testers on this? The MFC window is closing soon. -- Andre This is functional bundle MFC of these original commits: MFC r242029 (alfred): Allow autotune maxusers 384 on 64 bit machines. MFC r242847 (alfred): Allow maxusers to scale on machines with large address space. MFC r243631 (andre): Base the mbuf related limits on the available physical memory or kernel memory, whichever is lower. The overall mbuf related memory limit must be set so that mbufs (and clusters of various sizes) can't exhaust physical RAM or KVM. At the same time divorce maxfiles from maxusers and set maxfiles to physpages / 8 with a floor based on maxusers. This way busy servers can make use of the significantly increased mbuf limits with a much larger number of open sockets. MFC r243639 (andre): Complete r243631 by applying the remainder of kern_mbuf.c that got lost while merging into the commit tree. MFC r243668 (andre): Using a long is the wrong type to represent the realmem and maxmbufmem variable as they may overflow on i386/PAE and i386 with 2GB RAM. MFC r243995, r243996, r243997 (pjd): Style cleanups, Make use of the fact that uma_zone_set_max(9) already returns actual limit set. MFC r244080 (andre): Prevent long type overflow of realmem calculation on ILP32 by forcing calculation to be in quad_t space. Fix style issue with second parameter to qmin(). MFC r245469 (alfred): Do not autotune ncallout to be greater than 18508. MFC r245575 (andre): Move the mbuf memory limit calculations from init_param2() to tunable_mbinit() where it is next to where it is used later. MFC r246207 (andre): Remove unused VM_MAX_AUTOTUNE_NMBCLUSTERS define. MFC r249843 (andre): Base the calculation of maxmbufmem in part on kmem_map size instead of kernel_map size to prevent kernel memory exhaustion by mbufs and a subsequent panic on physical page allocation failure. ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: status of autotuning freebsd for 9.2
On 7/11/13 12:44 AM, Andre Oppermann wrote: On 08.07.2013 16:37, Andre Oppermann wrote: On 07.07.2013 20:24, Alfred Perlstein wrote: On 7/7/13 1:34 AM, Andre Oppermann wrote: Can you help me with with testing? Yes. Please give me your proposed changes and I'll stand up a machine and give feedback. http://people.freebsd.org/~andre/mfc-autotuning-20130708.diff Any feedback from testers on this? The MFC window is closing soon. This week has been pretty intense. I've sent out a request to the team to apply this patch to as many machines as possible. I'm going to talk to the rest of the FreeNAS team about it as well shortly. -- Alfred Perlstein VP Software Engineering, iXsystems ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: status of autotuning freebsd for 9.2
- Original Message - From: Andre Oppermann an...@freebsd.org On 08.07.2013 16:37, Andre Oppermann wrote: On 07.07.2013 20:24, Alfred Perlstein wrote: On 7/7/13 1:34 AM, Andre Oppermann wrote: Can you help me with with testing? Yes. Please give me your proposed changes and I'll stand up a machine and give feedback. http://people.freebsd.org/~andre/mfc-autotuning-20130708.diff Any feedback from testers on this? The MFC window is closing soon. Few things I've noticed most of which look like issues against the original patch and not the MFC but worth mentioning. 1. You've introduced a new tunable kern.maxmbufmem which is autosized but doesnt seem to be exposed via a sysctl so it looks like there is no way to determine what its actually set to? 2. There's a missmatch between the tuneable kern.ipc.nmbufs in tunable_mbinit and the sysctl kern.ipc.nmbuf i.e. no 's'. 3. Should kern.maxmbufmem be kern.ipc.maxmbufmem to sit along side all of the other sysctls? 4. style issues: * @@ -178,11 +202,13 @@ ... if (newnmbjumbo9 nmbjumbo9 Finally out of interest what made us arrive at the various defaults for each type as it looks like the ratios have changed? Regards Steve This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or entity to whom it is addressed. In the event of misdirection, the recipient is prohibited from using, copying, printing or otherwise disseminating it or any information contained in it. In the event of misdirection, illegible or incomplete transmission please telephone +44 845 868 1337 or return the E.mail to postmas...@multiplay.co.uk. ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: status of autotuning freebsd for 9.2
On 11.07.2013 11:08, Steven Hartland wrote: - Original Message - From: Andre Oppermann an...@freebsd.org On 08.07.2013 16:37, Andre Oppermann wrote: On 07.07.2013 20:24, Alfred Perlstein wrote: On 7/7/13 1:34 AM, Andre Oppermann wrote: Can you help me with with testing? Yes. Please give me your proposed changes and I'll stand up a machine and give feedback. http://people.freebsd.org/~andre/mfc-autotuning-20130708.diff Any feedback from testers on this? The MFC window is closing soon. Few things I've noticed most of which look like issues against the original patch and not the MFC but worth mentioning. 1. You've introduced a new tunable kern.maxmbufmem which is autosized but doesnt seem to be exposed via a sysctl so it looks like there is no way to determine what its actually set to? Good point. I've made it global and exposed as kern.ipc.maxmbufmem (RDTUN). 2. There's a missmatch between the tuneable kern.ipc.nmbufs in tunable_mbinit and the sysctl kern.ipc.nmbuf i.e. no 's'. That's a typo, fixed. 3. Should kern.maxmbufmem be kern.ipc.maxmbufmem to sit along side all of the other sysctls? Yes, see above. 4. style issues: * @@ -178,11 +202,13 @@ ... if (newnmbjumbo9 nmbjumbo9 Thanks. All fixed in r253204. Finally out of interest what made us arrive at the various defaults for each type as it looks like the ratios have changed? Before it was an arbitrary mess. Mbufs were not limited at all and the others to some random multiple of maxusers with the net limit ending up at some 25,000 mbuf clusters by default. Now default overall limit is set at 50% of all available min(physical, kmem_map) memory to prevent mbufs from monopolizing kernel memory and leave some space for other kernel structures and buffers as well as user-space programs. It can be raised to 3/4 of available memory by the tunable. 2K and 4K (page size) mbuf clusters can each go up to 25% of this mbuf memory. The former is dominantly used on the receive path and the latter in the send path. 9K and 16K jumbo mbuf clusters can each go up to 12.5% of mbuf memory. They are only used in the receive path if large jumbo MTUs on a network interface are active. Both are special in that their memory is contiguous in KVM and physical memory. This becomes problematic due to memory fragmentation after a short amount of heavy system use. I hope to deprecate them for 10.0. Network interfaces should use 4K clusters instead and chain them together for larger packets. All modern NICs support that. Only the early and limited DMA engines without scatter-gather capabilities required contiguous physical memory. They are long gone by now. The limit for mbufs itselfs is 12.5% of mbuf memory and should be at least as many as the sum of the cluster types. Each cluster requires an mbuf to which it is attached. Two examples on the revised mbuf sizing limits: 1GB KVM: 512MB limit for mbufs 419,430 mbufs 65,536 2K mbuf clusters 32,768 4K mbuf clusters 9,709 9K mbuf clusters 5,461 16K mbuf clusters 16GB RAM: 8GB limit for mbufs 33,554,432 mbufs 1,048,576 2K mbuf clusters 524,288 4K mbuf clusters 155,344 9K mbuf clusters 87,381 16K mbuf clusters These defaults should be sufficient for even the most demanding network loads. For additional information see: http://svnweb.freebsd.org/changeset/base/243631 -- Andre ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: status of autotuning freebsd for 9.2
Andre, Peter what about i386? Ever since I touched this Peter has been worried about i386 and said we've broken the platform. I'm going to boot some vms but maybe we ought to get some testing from Peter on i386? Sent from my iPhone On Jul 11, 2013, at 5:47 AM, Andre Oppermann an...@freebsd.org wrote: On 11.07.2013 11:08, Steven Hartland wrote: - Original Message - From: Andre Oppermann an...@freebsd.org On 08.07.2013 16:37, Andre Oppermann wrote: On 07.07.2013 20:24, Alfred Perlstein wrote: On 7/7/13 1:34 AM, Andre Oppermann wrote: Can you help me with with testing? Yes. Please give me your proposed changes and I'll stand up a machine and give feedback. http://people.freebsd.org/~andre/mfc-autotuning-20130708.diff Any feedback from testers on this? The MFC window is closing soon. Few things I've noticed most of which look like issues against the original patch and not the MFC but worth mentioning. 1. You've introduced a new tunable kern.maxmbufmem which is autosized but doesnt seem to be exposed via a sysctl so it looks like there is no way to determine what its actually set to? Good point. I've made it global and exposed as kern.ipc.maxmbufmem (RDTUN). 2. There's a missmatch between the tuneable kern.ipc.nmbufs in tunable_mbinit and the sysctl kern.ipc.nmbuf i.e. no 's'. That's a typo, fixed. 3. Should kern.maxmbufmem be kern.ipc.maxmbufmem to sit along side all of the other sysctls? Yes, see above. 4. style issues: * @@ -178,11 +202,13 @@ ... if (newnmbjumbo9 nmbjumbo9 Thanks. All fixed in r253204. Finally out of interest what made us arrive at the various defaults for each type as it looks like the ratios have changed? Before it was an arbitrary mess. Mbufs were not limited at all and the others to some random multiple of maxusers with the net limit ending up at some 25,000 mbuf clusters by default. Now default overall limit is set at 50% of all available min(physical, kmem_map) memory to prevent mbufs from monopolizing kernel memory and leave some space for other kernel structures and buffers as well as user-space programs. It can be raised to 3/4 of available memory by the tunable. 2K and 4K (page size) mbuf clusters can each go up to 25% of this mbuf memory. The former is dominantly used on the receive path and the latter in the send path. 9K and 16K jumbo mbuf clusters can each go up to 12.5% of mbuf memory. They are only used in the receive path if large jumbo MTUs on a network interface are active. Both are special in that their memory is contiguous in KVM and physical memory. This becomes problematic due to memory fragmentation after a short amount of heavy system use. I hope to deprecate them for 10.0. Network interfaces should use 4K clusters instead and chain them together for larger packets. All modern NICs support that. Only the early and limited DMA engines without scatter-gather capabilities required contiguous physical memory. They are long gone by now. The limit for mbufs itselfs is 12.5% of mbuf memory and should be at least as many as the sum of the cluster types. Each cluster requires an mbuf to which it is attached. Two examples on the revised mbuf sizing limits: 1GB KVM: 512MB limit for mbufs 419,430 mbufs 65,536 2K mbuf clusters 32,768 4K mbuf clusters 9,709 9K mbuf clusters 5,461 16K mbuf clusters 16GB RAM: 8GB limit for mbufs 33,554,432 mbufs 1,048,576 2K mbuf clusters 524,288 4K mbuf clusters 155,344 9K mbuf clusters 87,381 16K mbuf clusters These defaults should be sufficient for even the most demanding network loads. For additional information see: http://svnweb.freebsd.org/changeset/base/243631 -- Andre ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: status of autotuning freebsd for 9.2
Please test on VMs.
I've tested -HEAD in i386 virtualbox all the way down to 128mb with no
panics. I'll test with 64mb soon. It's easy to do.
i think the i386 PAE stuff on ${LARGE} memory systems is still broken. Peter?
-adrian
On 11 July 2013 07:59, Alfred Perlstein alf...@ixsystems.com wrote:
Andre, Peter what about i386?
Ever since I touched this Peter has been worried about i386 and said we've
broken the platform.
I'm going to boot some vms but maybe we ought to get some testing from Peter
on i386?
Sent from my iPhone
On Jul 11, 2013, at 5:47 AM, Andre Oppermann an...@freebsd.org wrote:
On 11.07.2013 11:08, Steven Hartland wrote:
- Original Message - From: Andre Oppermann an...@freebsd.org
On 08.07.2013 16:37, Andre Oppermann wrote:
On 07.07.2013 20:24, Alfred Perlstein wrote:
On 7/7/13 1:34 AM, Andre Oppermann wrote:
Can you help me with with testing?
Yes. Please give me your proposed changes and I'll stand up a machine
and give feedback.
http://people.freebsd.org/~andre/mfc-autotuning-20130708.diff
Any feedback from testers on this? The MFC window is closing soon.
Few things I've noticed most of which look like issues against the original
patch and not the MFC but worth mentioning.
1. You've introduced a new tunable kern.maxmbufmem which is autosized but
doesnt seem to be exposed via a sysctl so it looks like there is no way
to determine what its actually set to?
Good point. I've made it global and exposed as kern.ipc.maxmbufmem (RDTUN).
2. There's a missmatch between the tuneable kern.ipc.nmbufs in
tunable_mbinit
and the sysctl kern.ipc.nmbuf i.e. no 's'.
That's a typo, fixed.
3. Should kern.maxmbufmem be kern.ipc.maxmbufmem to sit along side all of
the other sysctls?
Yes, see above.
4. style issues:
* @@ -178,11 +202,13 @@
...
if (newnmbjumbo9 nmbjumbo9
Thanks. All fixed in r253204.
Finally out of interest what made us arrive at the various defaults for each
type as it looks like the ratios have changed?
Before it was an arbitrary mess. Mbufs were not limited at all and the
others
to some random multiple of maxusers with the net limit ending up at some
25,000
mbuf clusters by default.
Now default overall limit is set at 50% of all available min(physical,
kmem_map)
memory to prevent mbufs from monopolizing kernel memory and leave some space
for
other kernel structures and buffers as well as user-space programs. It can
be
raised to 3/4 of available memory by the tunable.
2K and 4K (page size) mbuf clusters can each go up to 25% of this mbuf
memory.
The former is dominantly used on the receive path and the latter in the send
path.
9K and 16K jumbo mbuf clusters can each go up to 12.5% of mbuf memory. They
are
only used in the receive path if large jumbo MTUs on a network interface are
active.
Both are special in that their memory is contiguous in KVM and physical
memory.
This becomes problematic due to memory fragmentation after a short amount of
heavy
system use. I hope to deprecate them for 10.0. Network interfaces should
use 4K
clusters instead and chain them together for larger packets. All modern NICs
support that. Only the early and limited DMA engines without scatter-gather
capabilities required contiguous physical memory. They are long gone by now.
The limit for mbufs itselfs is 12.5% of mbuf memory and should be at least as
many as the sum of the cluster types. Each cluster requires an mbuf to which
it is attached.
Two examples on the revised mbuf sizing limits:
1GB KVM:
512MB limit for mbufs
419,430 mbufs
65,536 2K mbuf clusters
32,768 4K mbuf clusters
9,709 9K mbuf clusters
5,461 16K mbuf clusters
16GB RAM:
8GB limit for mbufs
33,554,432 mbufs
1,048,576 2K mbuf clusters
524,288 4K mbuf clusters
155,344 9K mbuf clusters
87,381 16K mbuf clusters
These defaults should be sufficient for even the most demanding network
loads.
For additional information see:
http://svnweb.freebsd.org/changeset/base/243631
--
Andre
___
freebsd-stable@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
___
freebsd-stable@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: status of autotuning freebsd for 9.2
- Original Message - From: Andre Oppermann an...@freebsd.org To: Steven Hartland kill...@multiplay.co.uk Cc: Alfred Perlstein alf...@ixsystems.com; sta...@freebsd.org; r...@freebsd.org; nones...@longcount.org Sent: Thursday, July 11, 2013 1:47 PM Subject: Re: status of autotuning freebsd for 9.2 On 11.07.2013 11:08, Steven Hartland wrote: - Original Message - From: Andre Oppermann an...@freebsd.org On 08.07.2013 16:37, Andre Oppermann wrote: On 07.07.2013 20:24, Alfred Perlstein wrote: On 7/7/13 1:34 AM, Andre Oppermann wrote: Can you help me with with testing? Yes. Please give me your proposed changes and I'll stand up a machine and give feedback. http://people.freebsd.org/~andre/mfc-autotuning-20130708.diff Any feedback from testers on this? The MFC window is closing soon. Few things I've noticed most of which look like issues against the original patch and not the MFC but worth mentioning. 1. You've introduced a new tunable kern.maxmbufmem which is autosized but doesnt seem to be exposed via a sysctl so it looks like there is no way to determine what its actually set to? Good point. I've made it global and exposed as kern.ipc.maxmbufmem (RDTUN). 2. There's a missmatch between the tuneable kern.ipc.nmbufs in tunable_mbinit and the sysctl kern.ipc.nmbuf i.e. no 's'. That's a typo, fixed. 3. Should kern.maxmbufmem be kern.ipc.maxmbufmem to sit along side all of the other sysctls? Yes, see above. 4. style issues: * @@ -178,11 +202,13 @@ ... if (newnmbjumbo9 nmbjumbo9 Thanks. All fixed in r253204. Finally out of interest what made us arrive at the various defaults for each type as it looks like the ratios have changed? Before it was an arbitrary mess. Mbufs were not limited at all and the others to some random multiple of maxusers with the net limit ending up at some 25,000 mbuf clusters by default. Now default overall limit is set at 50% of all available min(physical, kmem_map) memory to prevent mbufs from monopolizing kernel memory and leave some space for other kernel structures and buffers as well as user-space programs. It can be raised to 3/4 of available memory by the tunable. 2K and 4K (page size) mbuf clusters can each go up to 25% of this mbuf memory. The former is dominantly used on the receive path and the latter in the send path. 9K and 16K jumbo mbuf clusters can each go up to 12.5% of mbuf memory. They are only used in the receive path if large jumbo MTUs on a network interface are active. Both are special in that their memory is contiguous in KVM and physical memory. This becomes problematic due to memory fragmentation after a short amount of heavy system use. I hope to deprecate them for 10.0. Network interfaces should use 4K clusters instead and chain them together for larger packets. All modern NICs support that. Only the early and limited DMA engines without scatter-gather capabilities required contiguous physical memory. They are long gone by now. The limit for mbufs itselfs is 12.5% of mbuf memory and should be at least as many as the sum of the cluster types. Each cluster requires an mbuf to which it is attached. Two examples on the revised mbuf sizing limits: 1GB KVM: 512MB limit for mbufs 419,430 mbufs 65,536 2K mbuf clusters 32,768 4K mbuf clusters 9,709 9K mbuf clusters 5,461 16K mbuf clusters 16GB RAM: 8GB limit for mbufs 33,554,432 mbufs 1,048,576 2K mbuf clusters 524,288 4K mbuf clusters 155,344 9K mbuf clusters 87,381 16K mbuf clusters These defaults should be sufficient for even the most demanding network loads. For additional information see: http://svnweb.freebsd.org/changeset/base/243631 Thanks for that Andre and thanks for doing this work its something thats been sorely needed for a long time. Am I right is saying there is now the new concept of max mbufs? If so can we get this added to the output of netstat -m if it already isn't? I also think this is worth a mention in UPDATING too given the defaults are now becoming sensible ;-) Regards Steve Regards Steve This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or entity to whom it is addressed. In the event of misdirection, the recipient is prohibited from using, copying, printing or otherwise disseminating it or any information contained in it. In the event of misdirection, illegible or incomplete transmission please telephone +44 845 868 1337 or return the E.mail to postmas...@multiplay.co.uk. ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: status of autotuning freebsd for 9.2
On 07.07.2013 20:24, Alfred Perlstein wrote: On 7/7/13 1:34 AM, Andre Oppermann wrote: Can you help me with with testing? Yes. Please give me your proposed changes and I'll stand up a machine and give feedback. http://people.freebsd.org/~andre/mfc-autotuning-20130708.diff This is functional bundle MFC of these original commits: MFC r242029 (alfred): Allow autotune maxusers 384 on 64 bit machines. MFC r242847 (alfred): Allow maxusers to scale on machines with large address space. MFC r243631 (andre): Base the mbuf related limits on the available physical memory or kernel memory, whichever is lower. The overall mbuf related memory limit must be set so that mbufs (and clusters of various sizes) can't exhaust physical RAM or KVM. At the same time divorce maxfiles from maxusers and set maxfiles to physpages / 8 with a floor based on maxusers. This way busy servers can make use of the significantly increased mbuf limits with a much larger number of open sockets. MFC r243639 (andre): Complete r243631 by applying the remainder of kern_mbuf.c that got lost while merging into the commit tree. MFC r243668 (andre): Using a long is the wrong type to represent the realmem and maxmbufmem variable as they may overflow on i386/PAE and i386 with 2GB RAM. MFC r243995, r243996, r243997 (pjd): Style cleanups, Make use of the fact that uma_zone_set_max(9) already returns actual limit set. MFC r244080 (andre): Prevent long type overflow of realmem calculation on ILP32 by forcing calculation to be in quad_t space. Fix style issue with second parameter to qmin(). MFC r245469 (alfred): Do not autotune ncallout to be greater than 18508. MFC r245575 (andre): Move the mbuf memory limit calculations from init_param2() to tunable_mbinit() where it is next to where it is used later. MFC r246207 (andre): Remove unused VM_MAX_AUTOTUNE_NMBCLUSTERS define. MFC r249843 (andre): Base the calculation of maxmbufmem in part on kmem_map size instead of kernel_map size to prevent kernel memory exhaustion by mbufs and a subsequent panic on physical page allocation failure. -- Andre ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: status of autotuning freebsd for 9.2
On Mon, Jul 8, 2013 at 10:37 AM, Andre Oppermann an...@freebsd.org wrote: On 07.07.2013 20:24, Alfred Perlstein wrote: On 7/7/13 1:34 AM, Andre Oppermann wrote: Can you help me with with testing? Yes. Please give me your proposed changes and I'll stand up a machine and give feedback. http://people.freebsd.org/~**andre/mfc-autotuning-20130708.**diffhttp://people.freebsd.org/~andre/mfc-autotuning-20130708.diff This is functional bundle MFC of these original commits: MFC r242029 (alfred): Allow autotune maxusers 384 on 64 bit machines. MFC r242847 (alfred): Allow maxusers to scale on machines with large address space. MFC r243631 (andre): Base the mbuf related limits on the available physical memory or kernel memory, whichever is lower. The overall mbuf related memory limit must be set so that mbufs (and clusters of various sizes) can't exhaust physical RAM or KVM. At the same time divorce maxfiles from maxusers and set maxfiles to physpages / 8 with a floor based on maxusers. This way busy servers can make use of the significantly increased mbuf limits with a much larger number of open sockets. MFC r243639 (andre): Complete r243631 by applying the remainder of kern_mbuf.c that got lost while merging into the commit tree. MFC r243668 (andre): Using a long is the wrong type to represent the realmem and maxmbufmem variable as they may overflow on i386/PAE and i386 with 2GB RAM. MFC r243995, r243996, r243997 (pjd): Style cleanups, Make use of the fact that uma_zone_set_max(9) already returns actual limit set. MFC r244080 (andre): Prevent long type overflow of realmem calculation on ILP32 by forcing calculation to be in quad_t space. Fix style issue with second parameter to qmin(). MFC r245469 (alfred): Do not autotune ncallout to be greater than 18508. MFC r245575 (andre): Move the mbuf memory limit calculations from init_param2() to tunable_mbinit() where it is next to where it is used later. MFC r246207 (andre): Remove unused VM_MAX_AUTOTUNE_NMBCLUSTERS define. MFC r249843 (andre): Base the calculation of maxmbufmem in part on kmem_map size instead of kernel_map size to prevent kernel memory exhaustion by mbufs and a subsequent panic on physical page allocation failure. would it be safe to throw a couple of high end storage systems into this testing pool ?? each has 128G Memory, and dual quad core procs, with 10GB Intel interfaces all they are design to do it samba and nfs and ive been fighting performance with samba and nfs on these systems, Id be curious if autotuning might help, to be honest, theres so much to tweak for samba, nfs, zfs alone in different formats, im surprised anyone has it running efficiently. -- Andre __**_ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/**mailman/listinfo/freebsd-**stablehttp://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscribe@**freebsd.orgfreebsd-stable-unsubscr...@freebsd.org ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: status of autotuning freebsd for 9.2
On 08.07.2013 16:41, Outback Dingo wrote: On Mon, Jul 8, 2013 at 10:37 AM, Andre Oppermann an...@freebsd.org mailto:an...@freebsd.org wrote: On 07.07.2013 20:24, Alfred Perlstein wrote: On 7/7/13 1:34 AM, Andre Oppermann wrote: Can you help me with with testing? Yes. Please give me your proposed changes and I'll stand up a machine and give feedback. http://people.freebsd.org/~__andre/mfc-autotuning-20130708.__diff http://people.freebsd.org/%7Eandre/mfc-autotuning-20130708.diff This is functional bundle MFC of these original commits: MFC r242029 (alfred): Allow autotune maxusers 384 on 64 bit machines. MFC r242847 (alfred): Allow maxusers to scale on machines with large address space. MFC r243631 (andre): Base the mbuf related limits on the available physical memory or kernel memory, whichever is lower. The overall mbuf related memory limit must be set so that mbufs (and clusters of various sizes) can't exhaust physical RAM or KVM. At the same time divorce maxfiles from maxusers and set maxfiles to physpages / 8 with a floor based on maxusers. This way busy servers can make use of the significantly increased mbuf limits with a much larger number of open sockets. MFC r243639 (andre): Complete r243631 by applying the remainder of kern_mbuf.c that got lost while merging into the commit tree. MFC r243668 (andre): Using a long is the wrong type to represent the realmem and maxmbufmem variable as they may overflow on i386/PAE and i386 with 2GB RAM. MFC r243995, r243996, r243997 (pjd): Style cleanups, Make use of the fact that uma_zone_set_max(9) already returns actual limit set. MFC r244080 (andre): Prevent long type overflow of realmem calculation on ILP32 by forcing calculation to be in quad_t space. Fix style issue with second parameter to qmin(). MFC r245469 (alfred): Do not autotune ncallout to be greater than 18508. MFC r245575 (andre): Move the mbuf memory limit calculations from init_param2() to tunable_mbinit() where it is next to where it is used later. MFC r246207 (andre): Remove unused VM_MAX_AUTOTUNE_NMBCLUSTERS define. MFC r249843 (andre): Base the calculation of maxmbufmem in part on kmem_map size instead of kernel_map size to prevent kernel memory exhaustion by mbufs and a subsequent panic on physical page allocation failure. would it be safe to throw a couple of high end storage systems into this testing pool ?? each has 128G Memory, and dual quad core procs, with 10GB Intel interfaces all they are design to do it samba and nfs and ive been fighting performance with samba and nfs on these systems, Id be curious if autotuning might help, to be honest, theres so much to tweak for samba, nfs, zfs alone in different formats, im surprised anyone has it running efficiently. I don't know enough about the limits of Samba setups. Testing this MFC may or may not significantly improve the performance, though at least we'll know that it doesn't hurt. -- Andre ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: status of autotuning freebsd for 9.2
On 07.07.2013 08:32, Alfred Perlstein wrote: Andre, Are you going to have time to MFC things from -current for auto-tuning -stable before 9.2? I simply ran out of time on Friday and MFCing such a big change requires more testing. I fear (maybe unnecessarily?) that we are about to ship yet another release that can't do basic 10gigE when sufficient memory exists. There was some debate with myself whether such a behavior changing MFC would be appropriate for a mid-stream stable release. I guess yes, though a number of people who currently set the parameters manually would have to remove their tuning settings. If you don't have time, then let me know and I'll see what I can do. Can you help me with with testing? -- Andre ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: status of autotuning freebsd for 9.2
On 7/7/13 1:34 AM, Andre Oppermann wrote: On 07.07.2013 08:32, Alfred Perlstein wrote: Andre, Are you going to have time to MFC things from -current for auto-tuning -stable before 9.2? I simply ran out of time on Friday and MFCing such a big change requires more testing. I fear (maybe unnecessarily?) that we are about to ship yet another release that can't do basic 10gigE when sufficient memory exists. There was some debate with myself whether such a behavior changing MFC would be appropriate for a mid-stream stable release. I guess yes, though a number of people who currently set the parameters manually would have to remove their tuning settings. If you don't have time, then let me know and I'll see what I can do. Can you help me with with testing? Yes. Please give me your proposed changes and I'll stand up a machine and give feedback. -Alfred ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
