Re: this 48-core box...

[Adrian Chadd]

The cache alignment happens because it hits a specific size threshold, and
jemalloc/phkmalloc(I think!) just round everything up to be page size
aligned.

The underlying problem may actually be a code change to how the math is
done. It just runs slower on page-aligned alignments..

adrian



On 22 September 2013 05:10, Eduardo Morras  wrote:

> On Sat, 21 Sep 2013 12:53:36 -0700
> Adrian Chadd  wrote:
>
> > .. just as a data point - there was a thread a while ago about numeric
> > processing performance on linux vs bsd.
> >
> > It all boiled down to how jemalloc versus the linux allocator(s) allocate
> > blocks. jemalloc will page align things after a certain size. Linux
> didn't.
> > So when doing numeric processing, there was a lot of cache aliasing going
> > on leading to inefficient cache usage and redundant memory operations.
> >
> > When the same workload on Linux was run on FreeBSD but with the Linux
> > library/allocators, the performance was identical.
> >
> > No-one followed through. I think I may have to write a blog post about
> it.
>
> There's no MALLOC_OPTIONS flag to set/unset this, but adding a new flag to
> disable a feature is easier (or should be) than implementing new one. The
> only problem I see to this is if the cache align happens at sbrk/mmap level.
>
>
> >
> > -adrian
> > ___
> > freebsd-questions@freebsd.org mailing list
> > http://lists.freebsd.org/mailman/listinfo/freebsd-questions
> > To unsubscribe, send any mail to "
> freebsd-questions-unsubscr...@freebsd.org"
>
>
> ---   ---
> Eduardo Morras 
> ___
> freebsd-questions@freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-questions
> To unsubscribe, send any mail to "
> freebsd-questions-unsubscr...@freebsd.org"
>
___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"

Re: this 48-core box...

[Eduardo Morras]

On Sat, 21 Sep 2013 12:53:36 -0700
Adrian Chadd  wrote:

> .. just as a data point - there was a thread a while ago about numeric
> processing performance on linux vs bsd.
> 
> It all boiled down to how jemalloc versus the linux allocator(s) allocate
> blocks. jemalloc will page align things after a certain size. Linux didn't.
> So when doing numeric processing, there was a lot of cache aliasing going
> on leading to inefficient cache usage and redundant memory operations.
> 
> When the same workload on Linux was run on FreeBSD but with the Linux
> library/allocators, the performance was identical.
> 
> No-one followed through. I think I may have to write a blog post about it.

There's no MALLOC_OPTIONS flag to set/unset this, but adding a new flag to 
disable a feature is easier (or should be) than implementing new one. The only 
problem I see to this is if the cache align happens at sbrk/mmap level.

 
> 
> -adrian
> ___
> freebsd-questions@freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-questions
> To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"


---   ---
Eduardo Morras 
___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"

Re: this 48-core box...

[Adrian Chadd]

.. just as a data point - there was a thread a while ago about numeric
processing performance on linux vs bsd.

It all boiled down to how jemalloc versus the linux allocator(s) allocate
blocks. jemalloc will page align things after a certain size. Linux didn't.
So when doing numeric processing, there was a lot of cache aliasing going
on leading to inefficient cache usage and redundant memory operations.

When the same workload on Linux was run on FreeBSD but with the Linux
library/allocators, the performance was identical.

No-one followed through. I think I may have to write a blog post about it.


-adrian
___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"

Re: this 48-core box...

[Vincent Schut]

On Thu, 19 Sep 2013 12:05:14 -0700 (PDT)
Dennis Glatting  wrote:

> 
> 
> On Thu, 19 Sep 2013, Vincent Schut wrote:
> 
> > On Tue, 17 Sep 2013 12:08:43 -0500
> > Michael Chen  wrote:
> >
> >> I'm considering bidding on this 48-core box:
> >>
> >> http://www.ebay.com/itm/Supermicro-A-Server-1042G-TF-1U-H8QG6-4-CPUS-48-cores-2-2Ghz-128GB-RAM-/151119828428?pt=COMP_EN_Servers&hash=item232f7195cc
> >>
> >> Does anyone have experience with it and can I use all the cores?
> >>
> >> Thanks!
> >> ___
> >> freebsd-questions@freebsd.org mailing list
> >> http://lists.freebsd.org/mailman/listinfo/freebsd-questions
> >> To unsubscribe, send any mail to
> >> "freebsd-questions-unsubscr...@freebsd.org"
> >>
> >
> > I recently bought one like that (48 cores but 'only' 96 Gb ram). It
> > was meant to play a double role as both zfs file server and data
> > processing server (we do lots of satellite image processing),
> > running FreeBSD 9.1. It connects with a SAN and we'll use it to
> > process about 36TB of satellite data in the next months. (In a
> > couple of weeks we will probably have budget to split those roles,
> > and buy a dedicated file server.) After several weeks of tweaking
> > and testing, I can say that:
> > - the zfs/file server part runs without problems
> > - the satellite data processing had problems scaling to all 48
> > cores, I got max performance when running about 18 processes in
> > parallel, scaling up more would lower the overall performance.
> > However, this (sorry guys) appeared to be a FreeBSD problem, and
> > not a hardware problem. As a test I switched to linux with ZoL (ZFS
> > on Linux), and, though zfs performance is less compared to freebsd,
> > data processing is much much better, like a factor 12 or so.
> >
> 
> I've noticed this same scaling problem on 32+ core servers but
> haven't had a chance to look into the detail. From the performance
> graphs I am confused whether my problems are processing problems or a
> data I/O problem.

I have done some (light) investigation as I did need the processing
power. In my case the bottleneck was definitely not data I/O. Bonnie+
rates from and to the dataset were as expected, top and atop and other
utils did not show any stress on the I/O system, and the algorithm which
did not scale should not be IO bound, rather cpu or memory (or
both). I've heard/read rumors (when I was investigating the extreme
long compile time of openblas on freebsd compared to linux) about bsd
being less well optimized in e.g. using the processor's L2 cache.
Things like this can play an important role in the processing we do
(many numerical calculations on lots of data in memory). Most of the
calculations were done by quite optimized software for numerical
processing (numpy/scipy using openblas (yes I did make sure openblas
used only 1 thread when scaling up)). The fact that the problems
disappeared when running the same under linux also point in the
cpu/memory direction rather that I/O, as the ZFS on Linux performance
is still behind that of ZFS on BSD.

> 
> 
> > Conclusion: the hardware is alright, however when needed to do lots
> > of heavy calculations on terabytes of data, the combination with
> > FreeBSD appears not ideal.
> >
> > Of course it is you get what you pay for. Decent, OK working
> > hardware, but none of the special handy-dandy features expensive
> > brands will give you. If you don't need them, in my experience it
> > is decent hardware for a good price.
> >
> > regards,
> > Vincent.
> >
> > ___
> > freebsd-questions@freebsd.org mailing list
> > http://lists.freebsd.org/mailman/listinfo/freebsd-questions
> > To unsubscribe, send any mail to
> > "freebsd-questions-unsubscr...@freebsd.org"
> >
> ___
> freebsd-questions@freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-questions
> To unsubscribe, send any mail to
> "freebsd-questions-unsubscr...@freebsd.org"
> 

___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"

Re: this 48-core box...

[Dennis Glatting]

On Thu, 19 Sep 2013, Vincent Schut wrote:


On Tue, 17 Sep 2013 12:08:43 -0500
Michael Chen  wrote:


I'm considering bidding on this 48-core box:

http://www.ebay.com/itm/Supermicro-A-Server-1042G-TF-1U-H8QG6-4-CPUS-48-cores-2-2Ghz-128GB-RAM-/151119828428?pt=COMP_EN_Servers&hash=item232f7195cc

Does anyone have experience with it and can I use all the cores?

Thanks!
___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to
"freebsd-questions-unsubscr...@freebsd.org"



I recently bought one like that (48 cores but 'only' 96 Gb ram). It was
meant to play a double role as both zfs file server and data processing
server (we do lots of satellite image processing), running FreeBSD 9.1.
It connects with a SAN and we'll use it to process about 36TB of
satellite data in the next months. (In a couple of weeks we will
probably have budget to split those roles, and buy a dedicated file
server.) After several weeks of tweaking and testing, I can say that:
- the zfs/file server part runs without problems
- the satellite data processing had problems scaling to all 48 cores, I
 got max performance when running about 18 processes in parallel,
 scaling up more would lower the overall performance. However, this
 (sorry guys) appeared to be a FreeBSD problem, and not a hardware
 problem. As a test I switched to linux with ZoL (ZFS on Linux), and,
 though zfs performance is less compared to freebsd, data processing
 is much much better, like a factor 12 or so.



I've noticed this same scaling problem on 32+ core servers but haven't had 
a chance to look into the detail. From the performance graphs I am 
confused whether my problems are processing problems or a data I/O 
problem.




Conclusion: the hardware is alright, however when needed to do lots of
heavy calculations on terabytes of data, the combination with FreeBSD
appears not ideal.

Of course it is you get what you pay for. Decent, OK working hardware,
but none of the special handy-dandy features expensive brands will give
you. If you don't need them, in my experience it is decent hardware for
a good price.

regards,
Vincent.

___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"


___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"

Re: this 48-core box...

[Vincent Schut]

On Tue, 17 Sep 2013 12:08:43 -0500
Michael Chen  wrote:

> I'm considering bidding on this 48-core box:
> 
> http://www.ebay.com/itm/Supermicro-A-Server-1042G-TF-1U-H8QG6-4-CPUS-48-cores-2-2Ghz-128GB-RAM-/151119828428?pt=COMP_EN_Servers&hash=item232f7195cc
> 
> Does anyone have experience with it and can I use all the cores?
> 
> Thanks!
> ___
> freebsd-questions@freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-questions
> To unsubscribe, send any mail to
> "freebsd-questions-unsubscr...@freebsd.org"
> 

I recently bought one like that (48 cores but 'only' 96 Gb ram). It was
meant to play a double role as both zfs file server and data processing
server (we do lots of satellite image processing), running FreeBSD 9.1.
It connects with a SAN and we'll use it to process about 36TB of
satellite data in the next months. (In a couple of weeks we will
probably have budget to split those roles, and buy a dedicated file
server.) After several weeks of tweaking and testing, I can say that:
- the zfs/file server part runs without problems
- the satellite data processing had problems scaling to all 48 cores, I
  got max performance when running about 18 processes in parallel,
  scaling up more would lower the overall performance. However, this
  (sorry guys) appeared to be a FreeBSD problem, and not a hardware
  problem. As a test I switched to linux with ZoL (ZFS on Linux), and,
  though zfs performance is less compared to freebsd, data processing
  is much much better, like a factor 12 or so.

Conclusion: the hardware is alright, however when needed to do lots of
heavy calculations on terabytes of data, the combination with FreeBSD
appears not ideal.

Of course it is you get what you pay for. Decent, OK working hardware,
but none of the special handy-dandy features expensive brands will give
you. If you don't need them, in my experience it is decent hardware for
a good price.

regards,
Vincent.

___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"

Re: this 48-core box...

[iamatt]

We discovered some performance issues with the the SM boards and how they
are layed out.  Granted these were being used  with  HPC clusters in a
fortran development environment used in O&G industry.  You probably would
not even notice these running your typical web servers on them.  The ipmi
is pretty annoying and even worse if you get their 10 blade chassis
systems.  Another thing they lack is the error logging abilities and tools
that you get with a fully integrated  system from  say,  ibm, sgi.  Or
other utilities to change bios settings on the fly ..  like IBM Advanced
Settings Utility.  All of these may not matter as much I suppose  with a
small server environment.  You pay what you get for.


On Tue, 17 Sep 2013, iamatt wrote:

 Hi.   Not sure if you can use all cores.   It has been and still is my
> experience that SM is crap.   We have several SM gpu and SM/Calxeda  Arm
> clusters and they really lack in may ways from ipmi to chassis management
> to the corners they cut with the processor to memory mappings.
>
>
Just to clarify:

My use is simply as servers and workstations. Generally I don't use IPMI on
these systems. I have had trouble with the PCIe slots. Specifically, on the
dual core boards some slots are serviced by one set of hardware and other
slots by other sets of hardware. Consequently, if you don't have all cores
populated then corresponding PCIe slots will not work.

Can't say about the four core system, though.



 I would consider looking at SGI UV,  ultraviolet system for a fat node type
> system but they are not cheap.   We have plenty of those and they can build
> it the way you want.   Lead time is a couple of months due to build  to
> order.   Support from SM sucks too. :)
> On Sep 17, 2013 12:17 PM, "Michael Chen" 
> wrote:
>
>  I'm considering bidding on this 48-core box:
>>
>> http://www.ebay.com/itm/Supermicro-A-Server-1042G-TF-
>> 1U-H8QG6-4-CPUS-48-cores-2-2Ghz-128GB-RAM-/151119828428?
>> pt=COMP_EN_Servers&hash=item232f7195cc> ebay.com/itm/Supermicro-A-**Server-1042G-TF-1U-H8QG6-4-**
>> CPUS-48-cores-2-2Ghz-128GB-**RAM-/151119828428?pt=COMP_EN_**
>> Servers&hash=item232f7195cc
>> >
>>
>> Does anyone have experience with it and can I use all the cores?
>>
>> Thanks!
>> ___
>> freebsd-questions@freebsd.org mailing list
>> http://lists.freebsd.org/mailman/listinfo/freebsd-questions
>> 
>> >
>> To unsubscribe, send any mail to "freebsd-questions-**
>> unsubscr...@freebsd.org 
>> 
>> >"
>>
>>  __**_
> freebsd-questions@freebsd.org mailing list
> http://lists.freebsd.org/**mailman/listinfo/freebsd-**questions
> To unsubscribe, send any mail to "freebsd-questions-**
> unsubscr...@freebsd.org "
>
>
___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"

Re: this 48-core box...

[Dennis Glatting]

On Tue, 17 Sep 2013, iamatt wrote:


Hi.   Not sure if you can use all cores.   It has been and still is my
experience that SM is crap.   We have several SM gpu and SM/Calxeda  Arm
clusters and they really lack in may ways from ipmi to chassis management
to the corners they cut with the processor to memory mappings.



Just to clarify:

My use is simply as servers and workstations. Generally I don't use IPMI 
on these systems. I have had trouble with the PCIe slots. Specifically, on 
the dual core boards some slots are serviced by one set of hardware and 
other slots by other sets of hardware. Consequently, if you don't have all 
cores populated then corresponding PCIe slots will not work.


Can't say about the four core system, though.




I would consider looking at SGI UV,  ultraviolet system for a fat node type
system but they are not cheap.   We have plenty of those and they can build
it the way you want.   Lead time is a couple of months due to build  to
order.   Support from SM sucks too. :)
On Sep 17, 2013 12:17 PM, "Michael Chen"  wrote:


I'm considering bidding on this 48-core box:

http://www.ebay.com/itm/**Supermicro-A-Server-1042G-TF-**
1U-H8QG6-4-CPUS-48-cores-2-**2Ghz-128GB-RAM-/151119828428?**
pt=COMP_EN_Servers&hash=**item232f7195cc

Does anyone have experience with it and can I use all the cores?

Thanks!
__**_
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/**mailman/listinfo/freebsd-**questions
To unsubscribe, send any mail to "freebsd-questions-**
unsubscr...@freebsd.org "


___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"


___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"

Re: this 48-core box...

[Dennis Glatting]

On Tue, 17 Sep 2013, iamatt wrote:


Hi.   Not sure if you can use all cores.   It has been and still is my
experience that SM is crap.   We have several SM gpu and SM/Calxeda  Arm
clusters and they really lack in may ways from ipmi to chassis management
to the corners they cut with the processor to memory mappings.

I would consider looking at SGI UV,  ultraviolet system for a fat node type
system but they are not cheap.   We have plenty of those and they can build
it the way you want.   Lead time is a couple of months due to build  to
order.   Support from SM sucks too. :)
On Sep 17, 2013 12:17 PM, "Michael Chen"  wrote:



I have three personal systems and two work systems running using the H8DG6 
MBs and they work fine.





I'm considering bidding on this 48-core box:

http://www.ebay.com/itm/**Supermicro-A-Server-1042G-TF-**
1U-H8QG6-4-CPUS-48-cores-2-**2Ghz-128GB-RAM-/151119828428?**
pt=COMP_EN_Servers&hash=**item232f7195cc

Does anyone have experience with it and can I use all the cores?

Thanks!
__**_
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/**mailman/listinfo/freebsd-**questions
To unsubscribe, send any mail to "freebsd-questions-**
unsubscr...@freebsd.org "


___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"


___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"

Re: this 48-core box...

[Dennis Glatting]

Forgot to mention:

1) My board is mounted in a SC848 Chassis and I use active cooling.

2) DO NOT run a chassis like the SC848 with the top off or the disks will 
overheat. :)




On Tue, 17 Sep 2013, Michael Chen wrote:


I'm considering bidding on this 48-core box:

http://www.ebay.com/itm/Supermicro-A-Server-1042G-TF-1U-H8QG6-4-CPUS-48-cores-2-2Ghz-128GB-RAM-/151119828428?pt=COMP_EN_Servers&hash=item232f7195cc

Does anyone have experience with it and can I use all the cores?

Thanks!
___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"


___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"

Re: this 48-core box...

[Dennis Glatting]

On Tue, 17 Sep 2013, Michael Chen wrote:


I'm considering bidding on this 48-core box:

http://www.ebay.com/itm/Supermicro-A-Server-1042G-TF-1U-H8QG6-4-CPUS-48-cores-2-2Ghz-128GB-RAM-/151119828428?pt=COMP_EN_Servers&hash=item232f7195cc

Does anyone have experience with it and can I use all the cores?




I have one of those boards running 32 cores. You MUST run FreeBSD 9+ if 
you want access to more than 32 cores.


Currently there is a bug in the stable/9 mfs drivers that do not allow you 
to boot from a RAID array. I believe a patch has been submitted. I have a 
copy of the patch and it works fine.


I have had significant problems with ZFS under stable/9 however I haven't 
tried recent updates, rather I had to punt back to stable/8 (production 
machine).


I have 22 3TB disks, 4 256GB SSDs, 256GB RAM, and 4x16 cores on my 
machine. I also have a 10GbE cardin my machine that runs fine.


I DO NOT use the CD.

Other than the issues I mentioned, runs fine.



___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"

Re: this 48-core box...

[iamatt]

Hi.   Not sure if you can use all cores.   It has been and still is my
experience that SM is crap.   We have several SM gpu and SM/Calxeda  Arm
clusters and they really lack in may ways from ipmi to chassis management
to the corners they cut with the processor to memory mappings.

I would consider looking at SGI UV,  ultraviolet system for a fat node type
system but they are not cheap.   We have plenty of those and they can build
it the way you want.   Lead time is a couple of months due to build  to
order.   Support from SM sucks too. :)
On Sep 17, 2013 12:17 PM, "Michael Chen"  wrote:

> I'm considering bidding on this 48-core box:
>
> http://www.ebay.com/itm/**Supermicro-A-Server-1042G-TF-**
> 1U-H8QG6-4-CPUS-48-cores-2-**2Ghz-128GB-RAM-/151119828428?**
> pt=COMP_EN_Servers&hash=**item232f7195cc
>
> Does anyone have experience with it and can I use all the cores?
>
> Thanks!
> __**_
> freebsd-questions@freebsd.org mailing list
> http://lists.freebsd.org/**mailman/listinfo/freebsd-**questions
> To unsubscribe, send any mail to "freebsd-questions-**
> unsubscr...@freebsd.org "
>
___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"