Sorry. Here it is.
On Wednesday, July 8, 2020 at 12:38:38 AM UTC+5:30, Dormando wrote:
>
> 'stats settings' file is empty
>
> On Tue, 7 Jul 2020, Shweta Agrawal wrote:
>
> > Hi Dormando,
> > Got the stats for production. Please find attached files for stats
> settings. stats items, stats, stats slabs. Summary for all slabs.
> >
> > Other details that might help:
> > * TTL is two days or more.
> > * Key length is in the range of 40-80 bytes.
> > Below are the parameters that we plan to change from the current
> settings:
> > 1. slab_automove : from 0 to 1
> > 2. hash_algorithm: from jenkins to murmur
> > 3. chunk_size: from 48 to 297 (as we don't have data of size less than
> that)
> > 4. growth_factor: 1.25 to 1.20 ( Can reducing this more help? Do more
> slab classes affect performance?)
> > 5. max_item_size : from 4MB to 1MB (as our data will never be more than
> 1MB large)
> > Please let me know if different values for above paramters can be more
> beneficial.
> > Are there any other parameters which we should consider to change or
> set?
> >
> > Also below are the calculations used for columns in the summary shared.
> Can you please confirm if calculations are fine.
> > 1) Total_Mem = total_pages*page_size --> total memory
> > 2) Strg_ovrHd = (mem_requested/(used_chunks*chunk_size)) * 100 -->
> storage overhead
> > 3) Free Memory = free_chunks * chunk_size ---> free memory
> > 4) To Store = mem_requested --> actual memory requested for
> storing data
> >
> > Thank you for your time and efforts in explaining concepts.
> > Shweta
> >
> > > > the rest is free memory, which should be measured
> separately.
> > > free memory for a class will be : (free_chunks *
> chunk_size)
> > > And total memory reserved by a class will be :
> (total_pages*page_size)
> > >
> > > > If you're getting evictions in class A but there's too
> much free memory in classes C, D, etc
> > > > then you have a balance issue. for example. An
> efficiency stat which just
> > > > adds up the total pages doesn't tell you what to do with
> it.
> > > I see. Got your point.Storage overhead can help in
> deciding the chunk_size and growth_factor. Let me add storage-overhead and
> > free memory as well for
> > > calculation.
> >
> > Most people don't have to worry about growth_factor very
> much. Especially
> > since the large item code was added, but it has its own
> caveats. Growth
> > factor is only typically useful if you have _very_
> statically sized
> > objects.
> >
> > > One curious question: If we have an item of 500Bytes and
> there is free memory only in class A(chunk_size: 100Bytes). Do cache
> > evict items from class with
> > > largeer chunk_size or use multiple chunks from class A?
> >
> > No, it will evict an item matching the 500 byte chunk size,
> and not touch
> > A. This is where the memory balancer comes in; it will move
> pages of
> > memory between slab classes to keep the tail age roughly the
> same between
> > classes. It does this slowly.
> >
> > > Example:
> > > In below scenario, when we try to store item with 3MB,
> even when there was memory in class with smaller chunk_size, it evicts
> > items from 512K class and
> > > other memory is blocked by smaller slabs.
> >
> > Large (> 512KB) items are an exception. It will try to evict
> from the
> > "large item" bucket, which is 512kb. It will try to do this
> up to a few
> > times, trying to free up enough memory to make space for the
> large item.
> >
> > So to make space for a 3MB item, if the tail item is 5MB in
> size or 1MB in
> > size, they will still be evicted. If the tail age is low
> compared to all
> > other classes, the memory balancer will eventually move more
> pages into
> > the 512K slab class.
> >
> > If you tend to store a lot of very large items, it works
> better if the
> > instances are larger.
> >
> > Memcached is more optimized for performance with small
> items. if you try
> > to store a small item, it will evict exactly one item to
> make space.
> > However, for very large items (1MB+), the time it takes to
> read the data
> > from the network is so large that we can afford to do extra
> processing.
> >
> > > 3Mb_items_eviction.png
> > >
> > >
> > > Thank you,
> > > Shweta
> > >
> > >
> > > On Sunday, July 5, 2020 at 1:13:19 AM UTC+5:30, Dormando
> wrote:
> > > (memory_requested / (chunk_size * chunk_used)) * 100
> > >
> > > is roughly the storage overhead of memory used in
> the system. the rest is
> > > free memory, which should be measured separately. If
> you're getting
> > > evictions in class A but there's too much free
> memory in classes C, D, etc
> > > then you have a balance issue. for example. An
> efficiency stat which just
> > > adds up the total pages doesn't tell you what to do
> with it.
> > >
> > > On Sat, 4 Jul 2020, Shweta Agrawal wrote:
> > >
> > > > > I'll need the raw output from "stats items" and
> "stats slabs". I don't
> > > > > think that efficiency column is very
> helpful. ohkay no worries. I can get by Tuesday and will share.
> > > >
> > > > Efficiency for each slab is calcuated as
> > > > (("stats slabs" -> memory_requested) / (("stats
> slabs" -> total_pages) * page_size)) * 100
> > > >
> > > >
> > > > Attaching script which has calculations for the
> same. The script is from memcahe repo with additional calculation for
> > efficiency.
> > > > Will it be possible for you to verify if the
> efficiency calculation is correct?
> > > >
> > > > Thank you,
> > > > Shweta
> > > >
> > > > On Saturday, July 4, 2020 at 1:08:23 PM UTC+5:30,
> Dormando wrote:
> > > > ah okay.
> > > >
> > > > I'll need the raw output from "stats items"
> and "stats slabs". I don't
> > > > think that efficiency column is very
> helpful.
> > > >
> > > > On Fri, 3 Jul 2020, Shweta Agrawal wrote:
> > > >
> > > > >
> > > > >
> > > > > On Saturday, July 4, 2020 at 9:41:49 AM
> UTC+5:30, Dormando wrote:
> > > > > No attachment
> > > > >
> > > > > On Fri, 3 Jul 2020, Shweta Agrawal
> wrote:
> > > > >
> > > > > >
> > > > > > Wooo...so quick. :):)
> > > > > > > Correct, close. It actually uses
> more like 3 512k chunks and then one
> > > > > > > smaller chunk from a different
> class to fit exactly 1.6MB.
> > > > > > I see.Got it.
> > > > > >
> > > > > > >Can you share snapshots from
> "stats items" and "stats slabs" for one of
> > > > > > these instances?
> > > > > >
> > > > > > Currently I have summary of it,
> sharing the same below. I can get snapshot by Tuesday as need to request
> > for it.
> > > > > >
> > > > > > pages have value from total_pages
> from stats slab for each slab
> > > > > > item_size have value
> from chunk_size from stats slab for each slab
> > > > > > Used memory is calculated as
> pages*page size ---> This has to corrected now.
> > > > > >
> > > > > >
> > > > > > prod_stats.png
> > > > > >
> > > > > >
> > > > > > > 90%+ are perfectly doable. You
> probably need to look a bit more closely
> > > > > > > into why you're not getting the
> efficiency you expect. The detailed stats
> > > > > > > output should point to why. I
> can help with that if it's confusing.
> > > > > >
> > > > > > Great. Will surely ask for your
> input whenever I have question. It is really kind of you to offer help.
> > > > > >
> > > > > > > Either the slab rebalancer isn't
> keeping up or you actually do have 39GB
> > > > > > > of data and your expecations are
> a bit off. This will also depending on
> > > > > > > the TTL's you're setting and how
> often/quickly your items change size.
> > > > > > > Also things like your
> serialization method / compression / key length vs
> > > > > > > data length / etc.
> > > > > >
> > > > > > We have much less data than 39 GB.
> As after facing evictions, it has been always kept higher than
> > expected data-size.
> > > > > > TTL is two days or more.
> > > > > > From my observation items
> size(data-length) is in the range of 300Bytes to 500K after compression.
> > > > > > Key length is in the range of
> 40-80 bytes.
> > > > > >
> > > > > > Thank you,
> > > > > > Shweta
> > > > > >
> > > > > > On Saturday, July 4, 2020 at
> 8:38:31 AM UTC+5:30, Dormando wrote:
> > > > > > Hey,
> > > > > >
> > > > > > > Putting my understanding
> to re-confirm:
> > > > > > > 1) Page size will always
> be 1MB and we cannot change it.Moreover, it's not required to be
> > changed.
> > > > > >
> > > > > > Correct.
> > > > > >
> > > > > > > 2) We can store items
> larger than 1MB and it is done by combining chunks together. (example:
> > let's say item size:
> > > ~1.6MB -->
> > > > 4 slab
> > > > > > chunks(512k slab) from
> > > > > > > 2 pages will be used)
> > > > > >
> > > > > > Correct, close. It actually
> uses more like 3 512k chunks and then one
> > > > > > smaller chunk from a
> different class to fit exactly 1.6MB.
> > > > > >
> > > > > > > We use memcache in
> production and in past we saw evictions even when free memory was present.
> > Also currently we use
> > > cluster
> > > > with
> > > > > 39GB RAM in
> > > > > > total to
> > > > > > > cache data even when data
> size we expect is ~15GB to avoid eviction of active items.
> > > > > >
> > > > > > Can you share snapshots from
> "stats items" and "stats slabs" for one of
> > > > > > these instances?
> > > > > >
> > > > > > > But as our data varies in
> size, it is possible to avoid evictions by tuning parameters:
> > chunk_size, growth_factor,
> > > > slab_automove.
> > > > > Also I
> > > > > > believe memcache
> > > > > > > is efficient and we can
> reduce cost by reducing memory size for cluster.
> > > > > > > So I am trying to find the
> best possible memory size and parameters we can have.So want to be
> > clear with my
> > > understanding
> > > > and
> > > > > calculations.
> > > > > > >
> > > > > > > So while trying different
> parameters and putting all calculations, I observed that total_pages *
> > item_size_max >
> > > physical
> > > > memory for
> > > > > a
> > > > > > machine. And from
> > > > > > > all blogs,and docs it
> didnot match my understanding. But it's clear now. Thanks to you.
> > > > > > >
> > > > > > > One last question: From my
> trials I find that we can achieve ~90% storage efficiency with
> > memcache. (i.e we need
> > > 10MB of
> > > > physical
> > > > > memory to
> > > > > > store 9MB of
> > > > > > > data. Do you recommend any
> idle memory-size interms of percentage of expected data-size?
> > > > > >
> > > > > > 90%+ are perfectly doable.
> You probably need to look a bit more closely
> > > > > > into why you're not getting
> the efficiency you expect. The detailed stats
> > > > > > output should point to why.
> I can help with that if it's confusing.
> > > > > >
> > > > > > Either the slab rebalancer
> isn't keeping up or you actually do have 39GB
> > > > > > of data and your expecations
> are a bit off. This will also depending on
> > > > > > the TTL's you're setting and
> how often/quickly your items change size.
> > > > > > Also things like your
> serialization method / compression / key length vs
> > > > > > data length / etc.
> > > > > >
> > > > > > -Dormando
> > > > > >
> > > > > > > On Saturday, July 4, 2020
> at 12:23:09 AM UTC+5:30, Dormando wrote:
> > > > > > > Hey,
> > > > > > >
> > > > > > > Looks like I never
> updated the manpage. In the past the item size max was
> > > > > > > achieved by changing
> the slab page size, but that hasn't been true for a
> > > > > > > long time.
> > > > > > >
> > > > > > > From ./memcached -h:
> > > > > > > -m,
> --memory-limit=<num> item memory in megabytes (default: 64)
> > > > > > >
> > > > > > > ... -m just means
> the memory limit in megabytes, abstract from the page
> > > > > > > size. I think that
> was always true.
> > > > > > >
> > > > > > > In any recentish
> version, any item larger than half a page size (512k) is
> > > > > > > created by stitching
> page chunks together. This prevents waste when an
> > > > > > > item would be more
> than half a page size.
> > > > > > >
> > > > > > > Is there a problem
> you're trying to track down?
> > > > > > >
> > > > > > > I'll update the
> manpage.
> > > > > > >
> > > > > > > On Fri, 3 Jul 2020,
> Shweta Agrawal wrote:
> > > > > > >
> > > > > > > > Hi,
> > > > > > > > Sorry if I am
> repeating the question, I searched the list but could not find definite
> > answer. So posting it.
> > > > > > > >
> > > > > > > > Memcache
> version: 1.5.10
> > > > > > > > I have started
> memcahce with option: -I 4m (setting maximum item size to 4MB).Verified
> > it is set by
> > > command stats
> > > > settings ,
> > > > > I can
> > > > > > see STAT
> > > > > > > item_size_max
> > > > > > > > 4194304.
> > > > > > > >
> > > > > > > > Documentation from
> git repository here stats that:
> > > > > > > >
> > > > > > > > -I,
> --max-item-size=<size>
> > > > > > > > Override the
> default size of each slab page. The default size is 1mb. Default
> > > > > > > > value for this
> parameter is 1m, minimum is 1k, max is 1G (1024 * 1024 * 1024).
> > > > > > > > Adjusting this
> value changes the item size limit.
> > > > > > > > My understanding
> from documentation is this option will allow to save items with size
> > till 4MB and the page
> > > size for
> > > > each
> > > > > slab will
> > > > > > be 4MB
> > > > > > > (as I set it as
> > > > > > > > -I 4m).
> > > > > > > >
> > > > > > > > I am able to save
> items till 4MB but the page-size is still 1MB.
> > > > > > > >
> > > > > > > > -m memory size is
> default 64MB.
> > > > > > > >
> > > > > > > > Calculation:
> > > > > > > > -> Calculated
> total pages used from stats slabs output parameter total_pages = 64 (If
> > page size is 4MB then
> > > total
> > > > pages
> > > > > should not
> > > > > > be more
> > > > > > > than 16. Also
> > > > > > > > when I store 8
> items of ~3MB it uses 25 pages but if page size is 4MB, it should use 8
> > pages right.)
> > > > > > > >
> > > > > > > > Can you please
> help me in understanding the behaviour?
> > > > > > > >
> > > > > > > > Attached files
> with details for output of command stats settings and stats slabs.
> > > > > > > > Below is the
> summarized view of the distribution.
> > > > > > > > First added items
> with variable sizes, then then added items with 3MB and above.
> > > > > > > >
> > > > > > > >
> data_distribution.png
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > Please let me know
> in case more details are required or question is not clear.
> > > > > > > >
> > > > > > > > Thank You,
> > > > > > > > Shweta
> > > > > > > >
> > > > > > > > --
> > > > > > > >
> > > > > > > > ---
> > > > > > > > You received this
> message because you are subscribed to the Google Groups "memcached"
> > group.
> > > > > > > > To unsubscribe
> from this group and stop receiving emails from it, send an email to
> > memc...@googlegroups.com.
> > > > > > > > To view this
> discussion on the web visit
> > > > > > >
> >
> https://groups.google.com/d/msgid/memcached/2b640e1f-9f59-4432-a930-d830cbe8566do%40googlegroups.com.
>
>
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > >
> > > > > > > ---
> > > > > > > You received this message
> because you are subscribed to the Google Groups "memcached" group.
> > > > > > > To unsubscribe from this
> group and stop receiving emails from it, send an email to
> > memc...@googlegroups.com.
> > > > > > > To view this discussion on
> the web visit
> > > > > >
> >
> https://groups.google.com/d/msgid/memcached/586aad58-c6fb-4ed8-89ce-6b005d59ba12o%40googlegroups.com.
>
>
> > > > > > >
> > > > > > >
> > > > > >
> > > > > > prod_stats.png
> > > > > >
> > > > > > --
> > > > > >
> > > > > > ---
> > > > > > You received this message because
> you are subscribed to the Google Groups "memcached" group.
> > > > > > To unsubscribe from this group and
> stop receiving emails from it, send an email to
> > memc...@googlegroups.com.
> > > > > > To view this discussion on the web
> visit
> > > > >
> https://groups.google.com/d/msgid/memcached/8d011c1a-deec-463f-a17e-4e9908d97bdfo%40googlegroups.com.
>
>
> > > > > >
> > > > > >
> > > > >
> > > > > --
> > > > >
> > > > > ---
> > > > > You received this message because you are
> subscribed to the Google Groups "memcached" group.
> > > > > To unsubscribe from this group and stop
> receiving emails from it, send an email to memc...@googlegroups.com.
> > > > > To view this discussion on the web visit
> > > >
> https://groups.google.com/d/msgid/memcached/f0c2bfe1-d65d-4b62-9a87-68fc42446c3do%40googlegroups.com.
>
>
> > > > >
> > > > >
> > > >
> > > > --
> > > >
> > > > ---
> > > > You received this message because you are
> subscribed to the Google Groups "memcached" group.
> > > > To unsubscribe from this group and stop receiving
> emails from it, send an email to memc...@googlegroups.com.
> > > > To view this discussion on the web visit
> > >
> https://groups.google.com/d/msgid/memcached/bcd4da5a-ae8e-470f-beb9-2705c0f0202ao%40googlegroups.com.
>
>
> > > >
> > > >
> > >
> > > --
> > >
> > > ---
> > > You received this message because you are subscribed to
> the Google Groups "memcached" group.
> > > To unsubscribe from this group and stop receiving emails
> from it, send an email to memc...@googlegroups.com.
> > > To view this discussion on the web visit
> >
> https://groups.google.com/d/msgid/memcached/5e76fa4f-7e06-468a-8b10-d99ab89d7ec2o%40googlegroups.com.
>
>
> > >
> > >
> >
> > --
> >
> > ---
> > You received this message because you are subscribed to the Google
> Groups "memcached" group.
> > To unsubscribe from this group and stop receiving emails from it, send
> an email to memc...@googlegroups.com <javascript:>.
> > To view this discussion on the web visit
> https://groups.google.com/d/msgid/memcached/71fd5680-7bd2-473b-9944-6cda8271ad5fo%40googlegroups.com.
>
>
> >
> >
--
---
You received this message because you are subscribed to the Google Groups
"memcached" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to memcached+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/memcached/372169f1-2a2e-4163-bf48-ca8176e76443o%40googlegroups.com.
stats settings
STAT maxbytes 13827571712
STAT launch_time_maxbytes 13827571712
STAT maxconns 65000
STAT tcpport 11211
STAT udpport 11211
STAT inter NULL
STAT verbosity 1
STAT oldest 0
STAT evictions on
STAT domain_socket NULL
STAT umask 700
STAT growth_factor 1.25
STAT chunk_size 48
STAT num_threads 2
STAT num_threads_per_udp 2
STAT stat_key_prefix :
STAT detail_enabled no
STAT reqs_per_event 20
STAT cas_enabled no
STAT tcp_backlog 1024
STAT binding_protocol auto-negotiate
STAT auth_enabled_sasl no
STAT item_size_max 4194304
STAT config_max 16
STAT config_size_max 65536
STAT maxconns_fast yes
STAT hashpower_init 16
STAT slab_reassign yes
STAT slab_automove 0
STAT slab_automove_ratio 0.80
STAT slab_automove_window 30
STAT slab_chunk_max 524288
STAT lru_crawler yes
STAT lru_crawler_sleep 100
STAT lru_crawler_tocrawl 0
STAT tail_repair_time 0
STAT flush_enabled yes
STAT dump_enabled yes
STAT hash_algorithm jenkins
STAT lru_maintainer_thread yes
STAT lru_segmented yes
STAT hot_lru_pct 20
STAT warm_lru_pct 40
STAT hot_max_factor 0.20
STAT warm_max_factor 2.00
STAT temp_lru no
STAT temporary_ttl 61
STAT idle_timeout 0
STAT watcher_logbuf_size 262144
STAT worker_logbuf_size 65536
STAT track_sizes no
STAT inline_ascii_response no
STAT ext_item_size 512
STAT ext_item_age 4294967295
STAT ext_low_ttl 0
STAT ext_recache_rate 2000
STAT ext_wbuf_size 4194304
STAT ext_compact_under 0
STAT ext_drop_under 0
STAT ext_max_frag 0.80
STAT slab_automove_freeratio 0.010
STAT ext_drop_unread no
END
version
VERSION 1.5.10
