On 03/12/2017 12:19 AM, Kant Kodali wrote:
My response is inline.
On Sat, Mar 11, 2017 at 1:43 PM, Avi Kivity <a...@scylladb.com
<mailto:a...@scylladb.com>> wrote:
There are several issues at play here.
First, a database runs a large number of concurrent
operations, each of which only consumes a small amount of
CPU. The high concurrency is need to hide latency: disk
latency, or the latency of contacting a remote node.
*Ok so you are talking about hiding I/O latency. If all these
I/O are non-blocking system calls then a thread per core and
callback mechanism should suffice isn't it?*
Scylla uses a mix of user-level threads and callbacks. Most of
the code uses callbacks (fronted by a future/promise API).
SSTable writers (memtable flush, compaction) use a user-level
thread (internally implemented using callbacks). The important
bit is multiplexing many concurrent operations onto a single
kernel thread.
This means that the scheduler will need to switch contexts
very often. A kernel thread scheduler knows very little
about the application, so it has to switch a lot of
context. A user level scheduler is tightly bound to the
application, so it can perform the switching faster.
*sure but this applies in other direction as well. A user level
scheduler has no idea about kernel level scheduler either.
There is literally no coordination between kernel level
scheduler and user level scheduler in linux or any major OS. It
may be possible with OS's that support scheduler
activation(LWP's) and upcall mechanism. *
There is no need for coordination, because the kernel scheduler
has no scheduling decisions to make. With one thread per core,
bound to its core, the kernel scheduler can't make the wrong
decision because it has just one choice.
*Even then it is hard to say if it is all worth it (The research
shows performance may not outweigh the complexity). Golang
problem is exactly this if one creates 1000 go routines/green
threads where each of them is making a blocking system call then
it would create 1000 kernel threads underneath because it has no
way to know that the kernel thread is blocked (no upcall). *
All of the significant system calls we issue are through the main
thread, either asynchronous or non-blocking.
*And in non-blocking case I still don't even see a significant
performance when compared to few kernel threads with callback
mechanism.*
We do.
* If you are saying user level scheduling is the Future
(perhaps I would just let the researchers argue about it) As of
today that is not case else languages would have had it natively
instead of using third party frameworks or libraries.
*
User-level scheduling is great for high performance I/O intensive
applications like databases and file systems. It's not a general
solution, and it involves a lot of effort to set up the
infrastructure. However, for our use case, it was worth it.
There are also implications on the concurrency primitives in
use (locks etc.) -- they will be much faster for the
user-level scheduler, because they cooperate with the
scheduler. For example, no atomic read-modify-write
instructions need to be executed.
Second, how many (kernel) threads should you run?*This
question one will always have. If there are 10K user level
threads that maps to only one kernel thread then they cannot
exploit parallelism. so there is no right answer but a thread
per core is a reasonable/good choice.
*
Only if you can multiplex many operations on top of each of those
threads. Otherwise, the CPUs end up underutilized.
If you run too few threads, then you will not be able to
saturate the CPU resources. This is a common problem with
Cassandra -- it's very hard to get it to consume all of the
CPU power on even a moderately large machine. On the other
hand, if you have too many threads, you will see latency
rise very quickly, because kernel scheduling granularity is
on the order of milliseconds. User-level scheduling,
because it leaves control in the hand of the application,
allows you to both saturate the CPU and maintain low latency.
F*or my workload and probably others I had seen Cassandra
was always been CPU bound.*
Yes, but does it consume 100% of all of the cores on your
machine? Cassandra generally doesn't (on a larger machine), and
when you profile it, you see it spending much of its time in
atomic operations, or parking/unparking threads -- fighting with
itself. It doesn't scale within the machine. Scylla will happily
utilize all of the cores that it is assigned (all of them by
default in most configurations), and the bigger the machine you
give it, the happier it will be.
There are other factors, like NUMA-friendliness, but in the
end it all boils down to efficiency and control.
None of this is new btw, it's pretty common in the storage
world.
Avi
On 03/11/2017 11:18 PM, Kant Kodali wrote:
Here is the Java version
http://docs.paralleluniverse.co/quasar/
<http://docs.paralleluniverse.co/quasar/> but I still don't
see how user level scheduling can be beneficial (This is a
well debated problem)? How can this add to the performance?
or say why is user level scheduling necessary Given the
Thread per core design and the callback mechanism?
On Sat, Mar 11, 2017 at 12:51 PM, Avi Kivity
<a...@scylladb.com <mailto:a...@scylladb.com>> wrote:
Scylla uses a the seastar framework, which provides for
both user-level thread scheduling and simple
run-to-completion tasks.
Huge pages are limited to 2MB (and 1GB, but these
aren't available as transparent hugepages).
On 03/11/2017 10:26 PM, Kant Kodali wrote:
@Dor
1) You guys have a CPU scheduler? you mean user level
thread Scheduler that maps user level threads to
kernel level threads? I thought C++ by default creates
native kernel threads but sure nothing will stop
someone to create a user level scheduling library if
that's what you are talking about?
2) How can one create THP of size 1KB? According to
this post
<https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Performance_Tuning_Guide/s-memory-transhuge.html>
it
looks like the valid values 2MB and 1GB.
Thanks,
kant
On Sat, Mar 11, 2017 at 11:41 AM, Avi Kivity
<a...@scylladb.com <mailto:a...@scylladb.com>> wrote:
Agreed, I'd recommend to treat benchmarks as a
rough guide to see where there is potential, and
follow through with your own tests.
On 03/11/2017 09:37 PM, Edward Capriolo wrote:
Benchmarks are great for FUDly blog posts. Real
world work loads matter more. Every NoSQL vendor
wins their benchmarks.
