Re: this 48-core box...
The cache alignment happens because it hits a specific size threshold, and jemalloc/phkmalloc(I think!) just round everything up to be page size aligned. The underlying problem may actually be a code change to how the math is done. It just runs slower on page-aligned alignments.. adrian On 22 September 2013 05:10, Eduardo Morras emorr...@yahoo.es wrote: On Sat, 21 Sep 2013 12:53:36 -0700 Adrian Chadd adr...@freebsd.org wrote: .. just as a data point - there was a thread a while ago about numeric processing performance on linux vs bsd. It all boiled down to how jemalloc versus the linux allocator(s) allocate blocks. jemalloc will page align things after a certain size. Linux didn't. So when doing numeric processing, there was a lot of cache aliasing going on leading to inefficient cache usage and redundant memory operations. When the same workload on Linux was run on FreeBSD but with the Linux library/allocators, the performance was identical. No-one followed through. I think I may have to write a blog post about it. There's no MALLOC_OPTIONS flag to set/unset this, but adding a new flag to disable a feature is easier (or should be) than implementing new one. The only problem I see to this is if the cache align happens at sbrk/mmap level. -adrian ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org --- --- Eduardo Morras emorr...@yahoo.es ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: this 48-core box...
On Sat, 21 Sep 2013 12:53:36 -0700 Adrian Chadd adr...@freebsd.org wrote: .. just as a data point - there was a thread a while ago about numeric processing performance on linux vs bsd. It all boiled down to how jemalloc versus the linux allocator(s) allocate blocks. jemalloc will page align things after a certain size. Linux didn't. So when doing numeric processing, there was a lot of cache aliasing going on leading to inefficient cache usage and redundant memory operations. When the same workload on Linux was run on FreeBSD but with the Linux library/allocators, the performance was identical. No-one followed through. I think I may have to write a blog post about it. There's no MALLOC_OPTIONS flag to set/unset this, but adding a new flag to disable a feature is easier (or should be) than implementing new one. The only problem I see to this is if the cache align happens at sbrk/mmap level. -adrian ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org --- --- Eduardo Morras emorr...@yahoo.es ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: this 48-core box...
.. just as a data point - there was a thread a while ago about numeric processing performance on linux vs bsd. It all boiled down to how jemalloc versus the linux allocator(s) allocate blocks. jemalloc will page align things after a certain size. Linux didn't. So when doing numeric processing, there was a lot of cache aliasing going on leading to inefficient cache usage and redundant memory operations. When the same workload on Linux was run on FreeBSD but with the Linux library/allocators, the performance was identical. No-one followed through. I think I may have to write a blog post about it. -adrian ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: this 48-core box...
On Thu, 19 Sep 2013 12:05:14 -0700 (PDT) Dennis Glatting d...@pki2.com wrote: On Thu, 19 Sep 2013, Vincent Schut wrote: On Tue, 17 Sep 2013 12:08:43 -0500 Michael Chen mich...@foxbatcapital.com wrote: I'm considering bidding on this 48-core box: http://www.ebay.com/itm/Supermicro-A-Server-1042G-TF-1U-H8QG6-4-CPUS-48-cores-2-2Ghz-128GB-RAM-/151119828428?pt=COMP_EN_Servershash=item232f7195cc Does anyone have experience with it and can I use all the cores? Thanks! ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org I recently bought one like that (48 cores but 'only' 96 Gb ram). It was meant to play a double role as both zfs file server and data processing server (we do lots of satellite image processing), running FreeBSD 9.1. It connects with a SAN and we'll use it to process about 36TB of satellite data in the next months. (In a couple of weeks we will probably have budget to split those roles, and buy a dedicated file server.) After several weeks of tweaking and testing, I can say that: - the zfs/file server part runs without problems - the satellite data processing had problems scaling to all 48 cores, I got max performance when running about 18 processes in parallel, scaling up more would lower the overall performance. However, this (sorry guys) appeared to be a FreeBSD problem, and not a hardware problem. As a test I switched to linux with ZoL (ZFS on Linux), and, though zfs performance is less compared to freebsd, data processing is much much better, like a factor 12 or so. I've noticed this same scaling problem on 32+ core servers but haven't had a chance to look into the detail. From the performance graphs I am confused whether my problems are processing problems or a data I/O problem. I have done some (light) investigation as I did need the processing power. In my case the bottleneck was definitely not data I/O. Bonnie+ rates from and to the dataset were as expected, top and atop and other utils did not show any stress on the I/O system, and the algorithm which did not scale should not be IO bound, rather cpu or memory (or both). I've heard/read rumors (when I was investigating the extreme long compile time of openblas on freebsd compared to linux) about bsd being less well optimized in e.g. using the processor's L2 cache. Things like this can play an important role in the processing we do (many numerical calculations on lots of data in memory). Most of the calculations were done by quite optimized software for numerical processing (numpy/scipy using openblas (yes I did make sure openblas used only 1 thread when scaling up)). The fact that the problems disappeared when running the same under linux also point in the cpu/memory direction rather that I/O, as the ZFS on Linux performance is still behind that of ZFS on BSD. Conclusion: the hardware is alright, however when needed to do lots of heavy calculations on terabytes of data, the combination with FreeBSD appears not ideal. Of course it is you get what you pay for. Decent, OK working hardware, but none of the special handy-dandy features expensive brands will give you. If you don't need them, in my experience it is decent hardware for a good price. regards, Vincent. ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: this 48-core box...
On Tue, 17 Sep 2013 12:08:43 -0500 Michael Chen mich...@foxbatcapital.com wrote: I'm considering bidding on this 48-core box: http://www.ebay.com/itm/Supermicro-A-Server-1042G-TF-1U-H8QG6-4-CPUS-48-cores-2-2Ghz-128GB-RAM-/151119828428?pt=COMP_EN_Servershash=item232f7195cc Does anyone have experience with it and can I use all the cores? Thanks! ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org I recently bought one like that (48 cores but 'only' 96 Gb ram). It was meant to play a double role as both zfs file server and data processing server (we do lots of satellite image processing), running FreeBSD 9.1. It connects with a SAN and we'll use it to process about 36TB of satellite data in the next months. (In a couple of weeks we will probably have budget to split those roles, and buy a dedicated file server.) After several weeks of tweaking and testing, I can say that: - the zfs/file server part runs without problems - the satellite data processing had problems scaling to all 48 cores, I got max performance when running about 18 processes in parallel, scaling up more would lower the overall performance. However, this (sorry guys) appeared to be a FreeBSD problem, and not a hardware problem. As a test I switched to linux with ZoL (ZFS on Linux), and, though zfs performance is less compared to freebsd, data processing is much much better, like a factor 12 or so. Conclusion: the hardware is alright, however when needed to do lots of heavy calculations on terabytes of data, the combination with FreeBSD appears not ideal. Of course it is you get what you pay for. Decent, OK working hardware, but none of the special handy-dandy features expensive brands will give you. If you don't need them, in my experience it is decent hardware for a good price. regards, Vincent. ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: this 48-core box...
On Thu, 19 Sep 2013, Vincent Schut wrote: On Tue, 17 Sep 2013 12:08:43 -0500 Michael Chen mich...@foxbatcapital.com wrote: I'm considering bidding on this 48-core box: http://www.ebay.com/itm/Supermicro-A-Server-1042G-TF-1U-H8QG6-4-CPUS-48-cores-2-2Ghz-128GB-RAM-/151119828428?pt=COMP_EN_Servershash=item232f7195cc Does anyone have experience with it and can I use all the cores? Thanks! ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org I recently bought one like that (48 cores but 'only' 96 Gb ram). It was meant to play a double role as both zfs file server and data processing server (we do lots of satellite image processing), running FreeBSD 9.1. It connects with a SAN and we'll use it to process about 36TB of satellite data in the next months. (In a couple of weeks we will probably have budget to split those roles, and buy a dedicated file server.) After several weeks of tweaking and testing, I can say that: - the zfs/file server part runs without problems - the satellite data processing had problems scaling to all 48 cores, I got max performance when running about 18 processes in parallel, scaling up more would lower the overall performance. However, this (sorry guys) appeared to be a FreeBSD problem, and not a hardware problem. As a test I switched to linux with ZoL (ZFS on Linux), and, though zfs performance is less compared to freebsd, data processing is much much better, like a factor 12 or so. I've noticed this same scaling problem on 32+ core servers but haven't had a chance to look into the detail. From the performance graphs I am confused whether my problems are processing problems or a data I/O problem. Conclusion: the hardware is alright, however when needed to do lots of heavy calculations on terabytes of data, the combination with FreeBSD appears not ideal. Of course it is you get what you pay for. Decent, OK working hardware, but none of the special handy-dandy features expensive brands will give you. If you don't need them, in my experience it is decent hardware for a good price. regards, Vincent. ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
this 48-core box...
I'm considering bidding on this 48-core box: http://www.ebay.com/itm/Supermicro-A-Server-1042G-TF-1U-H8QG6-4-CPUS-48-cores-2-2Ghz-128GB-RAM-/151119828428?pt=COMP_EN_Servershash=item232f7195cc Does anyone have experience with it and can I use all the cores? Thanks! ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: this 48-core box...
Hi. Not sure if you can use all cores. It has been and still is my experience that SM is crap. We have several SM gpu and SM/Calxeda Arm clusters and they really lack in may ways from ipmi to chassis management to the corners they cut with the processor to memory mappings. I would consider looking at SGI UV, ultraviolet system for a fat node type system but they are not cheap. We have plenty of those and they can build it the way you want. Lead time is a couple of months due to build to order. Support from SM sucks too. :) On Sep 17, 2013 12:17 PM, Michael Chen mich...@foxbatcapital.com wrote: I'm considering bidding on this 48-core box: http://www.ebay.com/itm/**Supermicro-A-Server-1042G-TF-** 1U-H8QG6-4-CPUS-48-cores-2-**2Ghz-128GB-RAM-/151119828428?** pt=COMP_EN_Servershash=**item232f7195cchttp://www.ebay.com/itm/Supermicro-A-Server-1042G-TF-1U-H8QG6-4-CPUS-48-cores-2-2Ghz-128GB-RAM-/151119828428?pt=COMP_EN_Servershash=item232f7195cc Does anyone have experience with it and can I use all the cores? Thanks! __**_ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/**mailman/listinfo/freebsd-**questionshttp://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-** unsubscr...@freebsd.org freebsd-questions-unsubscr...@freebsd.org ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: this 48-core box...
On Tue, 17 Sep 2013, Michael Chen wrote: I'm considering bidding on this 48-core box: http://www.ebay.com/itm/Supermicro-A-Server-1042G-TF-1U-H8QG6-4-CPUS-48-cores-2-2Ghz-128GB-RAM-/151119828428?pt=COMP_EN_Servershash=item232f7195cc Does anyone have experience with it and can I use all the cores? I have one of those boards running 32 cores. You MUST run FreeBSD 9+ if you want access to more than 32 cores. Currently there is a bug in the stable/9 mfs drivers that do not allow you to boot from a RAID array. I believe a patch has been submitted. I have a copy of the patch and it works fine. I have had significant problems with ZFS under stable/9 however I haven't tried recent updates, rather I had to punt back to stable/8 (production machine). I have 22 3TB disks, 4 256GB SSDs, 256GB RAM, and 4x16 cores on my machine. I also have a 10GbE cardin my machine that runs fine. I DO NOT use the CD. Other than the issues I mentioned, runs fine. ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: this 48-core box...
Forgot to mention: 1) My board is mounted in a SC848 Chassis and I use active cooling. 2) DO NOT run a chassis like the SC848 with the top off or the disks will overheat. :) On Tue, 17 Sep 2013, Michael Chen wrote: I'm considering bidding on this 48-core box: http://www.ebay.com/itm/Supermicro-A-Server-1042G-TF-1U-H8QG6-4-CPUS-48-cores-2-2Ghz-128GB-RAM-/151119828428?pt=COMP_EN_Servershash=item232f7195cc Does anyone have experience with it and can I use all the cores? Thanks! ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: this 48-core box...
On Tue, 17 Sep 2013, iamatt wrote: Hi. Not sure if you can use all cores. It has been and still is my experience that SM is crap. We have several SM gpu and SM/Calxeda Arm clusters and they really lack in may ways from ipmi to chassis management to the corners they cut with the processor to memory mappings. I would consider looking at SGI UV, ultraviolet system for a fat node type system but they are not cheap. We have plenty of those and they can build it the way you want. Lead time is a couple of months due to build to order. Support from SM sucks too. :) On Sep 17, 2013 12:17 PM, Michael Chen mich...@foxbatcapital.com wrote: I have three personal systems and two work systems running using the H8DG6 MBs and they work fine. I'm considering bidding on this 48-core box: http://www.ebay.com/itm/**Supermicro-A-Server-1042G-TF-** 1U-H8QG6-4-CPUS-48-cores-2-**2Ghz-128GB-RAM-/151119828428?** pt=COMP_EN_Servershash=**item232f7195cchttp://www.ebay.com/itm/Supermicro-A-Server-1042G-TF-1U-H8QG6-4-CPUS-48-cores-2-2Ghz-128GB-RAM-/151119828428?pt=COMP_EN_Servershash=item232f7195cc Does anyone have experience with it and can I use all the cores? Thanks! __**_ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/**mailman/listinfo/freebsd-**questionshttp://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-** unsubscr...@freebsd.org freebsd-questions-unsubscr...@freebsd.org ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: this 48-core box...
On Tue, 17 Sep 2013, iamatt wrote: Hi. Not sure if you can use all cores. It has been and still is my experience that SM is crap. We have several SM gpu and SM/Calxeda Arm clusters and they really lack in may ways from ipmi to chassis management to the corners they cut with the processor to memory mappings. Just to clarify: My use is simply as servers and workstations. Generally I don't use IPMI on these systems. I have had trouble with the PCIe slots. Specifically, on the dual core boards some slots are serviced by one set of hardware and other slots by other sets of hardware. Consequently, if you don't have all cores populated then corresponding PCIe slots will not work. Can't say about the four core system, though. I would consider looking at SGI UV, ultraviolet system for a fat node type system but they are not cheap. We have plenty of those and they can build it the way you want. Lead time is a couple of months due to build to order. Support from SM sucks too. :) On Sep 17, 2013 12:17 PM, Michael Chen mich...@foxbatcapital.com wrote: I'm considering bidding on this 48-core box: http://www.ebay.com/itm/**Supermicro-A-Server-1042G-TF-** 1U-H8QG6-4-CPUS-48-cores-2-**2Ghz-128GB-RAM-/151119828428?** pt=COMP_EN_Servershash=**item232f7195cchttp://www.ebay.com/itm/Supermicro-A-Server-1042G-TF-1U-H8QG6-4-CPUS-48-cores-2-2Ghz-128GB-RAM-/151119828428?pt=COMP_EN_Servershash=item232f7195cc Does anyone have experience with it and can I use all the cores? Thanks! __**_ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/**mailman/listinfo/freebsd-**questionshttp://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-** unsubscr...@freebsd.org freebsd-questions-unsubscr...@freebsd.org ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: this 48-core box...
We discovered some performance issues with the the SM boards and how they are layed out. Granted these were being used with HPC clusters in a fortran development environment used in OG industry. You probably would not even notice these running your typical web servers on them. The ipmi is pretty annoying and even worse if you get their 10 blade chassis systems. Another thing they lack is the error logging abilities and tools that you get with a fully integrated system from say, ibm, sgi. Or other utilities to change bios settings on the fly .. like IBM Advanced Settings Utility. All of these may not matter as much I suppose with a small server environment. You pay what you get for. On Tue, 17 Sep 2013, iamatt wrote: Hi. Not sure if you can use all cores. It has been and still is my experience that SM is crap. We have several SM gpu and SM/Calxeda Arm clusters and they really lack in may ways from ipmi to chassis management to the corners they cut with the processor to memory mappings. Just to clarify: My use is simply as servers and workstations. Generally I don't use IPMI on these systems. I have had trouble with the PCIe slots. Specifically, on the dual core boards some slots are serviced by one set of hardware and other slots by other sets of hardware. Consequently, if you don't have all cores populated then corresponding PCIe slots will not work. Can't say about the four core system, though. I would consider looking at SGI UV, ultraviolet system for a fat node type system but they are not cheap. We have plenty of those and they can build it the way you want. Lead time is a couple of months due to build to order. Support from SM sucks too. :) On Sep 17, 2013 12:17 PM, Michael Chen mich...@foxbatcapital.com wrote: I'm considering bidding on this 48-core box: http://www.ebay.com/itm/Supermicro-A-Server-1042G-TF-http://www.ebay.com/itm/**Supermicro-A-Server-1042G-TF-** 1U-H8QG6-4-CPUS-48-cores-2-2Ghz-128GB-RAM-/151119828428? pt=COMP_EN_Servershash=item232f7195cchttp://www.** ebay.com/itm/Supermicro-A-**Server-1042G-TF-1U-H8QG6-4-** CPUS-48-cores-2-2Ghz-128GB-**RAM-/151119828428?pt=COMP_EN_** Servershash=item232f7195cchttp://www.ebay.com/itm/Supermicro-A-Server-1042G-TF-1U-H8QG6-4-CPUS-48-cores-2-2Ghz-128GB-RAM-/151119828428?pt=COMP_EN_Servershash=item232f7195cc Does anyone have experience with it and can I use all the cores? Thanks! ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questionshttp://lists.freebsd.org/**mailman/listinfo/freebsd-**questions http://lists.**freebsd.org/mailman/listinfo/**freebsd-questionshttp://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-** unsubscr...@freebsd.org freebsd-questions-**unsubscr...@freebsd.orgfreebsd-questions-unsubscr...@freebsd.org __**_ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/**mailman/listinfo/freebsd-**questionshttp://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-** unsubscr...@freebsd.org freebsd-questions-unsubscr...@freebsd.org ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
