Re: Overriding the size of each slab page
On Jul 25, 4:38 pm, dormando wrote: > Uhhh. Can you post the output of `memcached -vvv` with your -I > adjustments? If you reduce the max page size it most certainly reduces the > number of slabs. It will increase the number of slab *pages* available. > Which doesn't affect anything. > > > > Also, I do not understand the warning, "It is not recommended to raise > > this limit above 1MB due just to performance reasons." What exactly > > are the performance issues? > > > If my default chunk size is 480 bytes and if I am storing items in 416 > > byte chunks and 448 byte chunks, then, I can store more chunks in 10 > > megabytes pages than I can in 10 kilobyte pages. So, why wouldn't I > > opt to store my chunks in 10 megabyte pages (rather than 10 kilobyte > > pages or even 1 megabyte pages)? The vast majority of my chunks are > > 448 byte chunks. So, it seems to me that I can use my memory more > > efficiently by opting for 10 megabyte slab pages. What, if anything, > > is behind the "peformance" warning? > > IF ANYTHING. So ominous. > > Using a non-brokeassed build of memcached, start it with the following > examples: > > memcached -vvv -I 128k > memcached -vvv -I 512k > memcached -vvv -I 1M > memcached -vvv -I 5M > memcached -vvv -I 10M > memcached -vvv -I 20M > > You'll see that the slab sizes get further apart. You're missunderstanding > what everything is inside the slabber. > > - slab page is set (1M by default) > - slab classes are created by starting at a minimum, multiplying by a > number (1.2 by default?) and repeating until the slab class size is equal > to the page size (1M by default) > - when you want to store an item: > - finds slab class (item size 416 bytes would go into class 8, so long > as the key isn't too long) > - try to find memory in class 8. No memory? Pull in one *page* (1M) > - divide that page into chunks of size 480 bytes (from class 8) > - hand back one page chunk for the item > - repeat > - memory between your actual item size, and the slab chunk size, is wasted > overhead > - the further apart slab classes are, the more memory you waste (perf > issue #1) > > If you give memcached a memory limit of 60 megs, and a max item size of > 10 megs, it only has enough pages to give one page to 6 slab classes. > Others will starve (tho it's actually a little more complicated than > that, but that's the idea). > > -Dormando Hi Dormando, Thank you for your detailed explanation. I am storing roughly the same size items. All of the items get written to slab class 8, which has a chuck size of 480 bytes. Most of my items are 448 bytes. Some are 416 bytes. I can store roughly 8 million items with 4 GB of total memory. My point above was that when I store the same size items, namely 480 byte chunks, it does not matter very much what value that I assign to my slab page size. If I set my slab page size to 128K, I get 273 chunks (480 bytes per chunk) in slab class 8, and I have more slab pages. If I set the slab page size to 1MB, I get 2,184 chunks (480 bytes per chunk) in slab class 8, and I have fewer slab pages. Test 1: 1M slab page size 3700 max mem 3376314121 bytes 3219.9 megabytes 8,080,800 curr_items 3762.7 MB of resident memory used Test 2: 1K slab page size 3700 max mem 3382172689 bytes 3225.5 megabytes 8,082,772 curr items 3768.4 MB of resident memory used Test 3: 10M slab page size 3690 max mem 3370641668 bytes 3214.5 megabytes 8,060,805 curr items 3754.8 MB of resident memory used With a 1K slab page size (i.e., test 2), I only get 2 chunks per page, so I have 4,041,386 total pages (8,082,772 total chunks). With 1M slab page size (i.e., test 1), I get 2,184 chunks per page, so I have 3,700 total pages (8,080,800 total chunks). I did notice that in test 2 that the system was using more virtual memory--I had 160 MB of swap spaced used in test 2, and no swap space used in test 1. So, I am assuming that a 1MB slab page size is more efficient, but I was questioning the "peformance" warning. So, if I am ONLY using slab class 8 (480 bytes), is there any advantage in setting my slab page size to 1 KB or to 10 MB? I am seeing a very slight edge to a 10 MB slab page size. In test 3 (10 MB slab page size), memcached used 8 megabytes less resident memory than in test 1 (1 MB slab page size), and test 3 stored only 10,000 less items, which is about 4.6 megabytes of data. David
Overriding the size of each slab page
On memcached version 1.4.5-1ubuntu1, there are two entries for the ‘- I’ parameter in the memcached(1) man page. -I Override the size of each slab page in bytes. In mundane words, it adjusts the maximum item size that memcached will accept. You can use the suffixes K and M to specify the size as well, so use 200 or 2000K or 2M if you want a maximum size of 2 MB per object. It is not recommended to raise this limit above 1MB due just to performance reasons. The default value is 1 MB. -I Override the default size of each slab page. Default is 1mb. Default is 1m, minimum is 1k, max is 128m. Adjusting this value changes the item size limit. Beware that this also increases the number of slabs (use -v to view), and the overal memory usage of memcached. It seems to me that the first entry is misleading. The parameter does not "adjust the maximum item size;" rather, the parameter adjusts the slab page size, and the number of items stored in each slab page. These two entries should be combined into one entry. The second entry could be further clarified by saying that reducing the page size below the 1 megabyte default page size will result in an increased number of slabs. By the way, '-I 10M' does not work. Neither does '-I 10m'. I discovered that you have to specify the byte size, i.e., '-I 10485760'. Please correct my understanding, if I am missing something. Also, I do not understand the warning, "It is not recommended to raise this limit above 1MB due just to performance reasons." What exactly are the performance issues? If my default chunk size is 480 bytes and if I am storing items in 416 byte chunks and 448 byte chunks, then, I can store more chunks in 10 megabytes pages than I can in 10 kilobyte pages. So, why wouldn't I opt to store my chunks in 10 megabyte pages (rather than 10 kilobyte pages or even 1 megabyte pages)? The vast majority of my chunks are 448 byte chunks. So, it seems to me that I can use my memory more efficiently by opting for 10 megabyte slab pages. What, if anything, is behind the "peformance" warning? Thank you for your help. David
Re: Tuning memcached on ubuntu x86_64 (2.6.35)
On Jul 21, 1:41 pm, Trond Norbye wrote: > Are you running out of virtual memory ;) > Well, the problem is that memcached will use swap, when it runs out of resident memory. When swap space fills up, memcached will crash under load. Yesterday, I had the -m option set to 3700 (or 3700 megabtyes), since I have a 4GB system. But, I started getting evictions when the dataset reached the size of 3219.9 megabytes. As I mentioned above, I started getting evictions at 3763 megabytes of RSS and 3834 megabytes of VSZ. Is the -m option the size of the dataset or is it the size of resident memory? Today, I increased the -m option to 8000 (or 8000 megabytes) to see what would happen. I only have 3954 megabytes total memory in the system. Now, memcached is filling up the swap space. I assume that I will start getting evictions when the virtual memory is full. It seems to me that I should avoid touching the swap space, since memcached can become unstable when using swap space. But, last week, I got into trouble because I set the -m option close to the total available memory on the system, and I guess that I had the value set too high, since the swap space filled up and memcached crashed. Today, I am trying to duplicate the issue that I saw last week. David
Tuning memcached on ubuntu x86_64 (2.6.35)
Is it normal to have a 16 percent virtual memory overhead in memcached on x86_64 linux? memcached STAT bytes is reporting 3219 megabytes of data, but virtual memory is 16 percent higher at 3834. Resident memory is 14 percent higher at 3763 megabytes. Is there a way to tune linux/memcached to get memcached to consume less virtual memory? At the moment, my 4GB system is full with 3219 megabtyes of data loaded in memcached. I am seeing lots of evictions when I try to load more data. Below is my configuration and stats. Ubuntu SMP x86_64 2.6.35-22-server memcached version 1.4.5-1ubuntu1 libevent-1.4-2 version 1.4.13-stable-1 I am using the default settings for memcached, namely, chunk_size of 48 and 4 threads. Stack size on my linux system is set to 8192 kilobytes. Should I reduce the stack size? I am using the default slab page size of 1MB. Should I reduce this amount? system: using free 3954 megabytes total memory 3928 megabytes used 26 megabytes free memcached process: using ps 3763 megabytes RSS 3834 megabytes VSZ 3219 megabtyes reported by STAT bytes settings: STAT maxbytes 3879731200 STAT maxconns 1024 STAT tcpport 11211 STAT udpport 11211 STAT inter NULL STAT verbosity 0 STAT oldest 0 STAT evictions on STAT domain_socket NULL STAT umask 700 STAT growth_factor 1.25 STAT chunk_size 48 STAT num_threads 4 STAT stat_key_prefix : STAT detail_enabled no STAT reqs_per_event 20 STAT cas_enabled yes STAT tcp_backlog 1024 STAT binding_protocol auto-negotiate STAT auth_enabled_sasl no STAT item_size_max 1048576 STAT pid 716 STAT uptime 10271 STAT time 1311199316 STAT version 1.4.5 STAT pointer_size 64 STAT rusage_user 175.14 STAT rusage_system 325.20 STAT curr_connections 11 STAT total_connections 73 STAT connection_structures 30 STAT cmd_get 9137902 STAT cmd_set 9067050 STAT cmd_flush 0 STAT get_hits 337524 STAT get_misses 8800378 STAT delete_misses 0 STAT delete_hits 266674 STAT incr_misses 0 STAT incr_hits 0 STAT decr_misses 0 STAT decr_hits 0 STAT cas_misses 0 STAT cas_hits 0 STAT cas_badval 0 STAT auth_cmds 0 STAT auth_errors 0 STAT bytes_read 3468460589 STAT bytes_written 244250132 STAT limit_maxbytes 3879731200 STAT accepting_conns 1 STAT listen_disabled_num 0 STAT threads 4 STAT conn_yields 0 STAT bytes 3376314121 STAT curr_items 8080800 STAT total_items 9067050 STAT evictions 719576 STAT reclaimed 0
