Re: HELP ! How Memcached objects are stored in the good SLAB
Nice, thank you very much.
Are you ok with :
Example :
•Key : test
•Data : 90 bytes
•Flags : 0
Size : (4+1) + (90+2) + (1+2+4) + 48 + 8 = 160 Bytes => Slab 4 (192 Bytes)
Key (characters count) + 1
+ Data (characters count) + 2 bytes ( \r\n )
+ Header
+ Chunk Size
+ CAS Size
Header = Flags (characters count) + nData (characters number of DATA count
=> 90 = 2 characters)
+ 4 bytes (2 spaces and \r\n)
Chunk Size = 48 bytes (default)
CAS Size = 8 bytes (platform 64 bits)
If CAS disabled:
Size : (4+1) + (90+2) + (1+2+4) + 48 = 152 Bytes => Slab 3 (152 Bytes)
Sounds good :)
Le lundi 16 novembre 2015 23:54:34 UTC+1, Dormando a écrit :
>
> key + 1b
> val + 2b
> item struct 48b
> [optional CAS] 8b
> then the suffix trailer printf: please read the manpage for snprintf:
>
> snprintf(suffix, 40, " %d %d\r\n", flags, nbytes - 2);
>
> there are four characters in there. two spaces, and then \r and \n. the
> two %d's change into: "0" for flags, and "90" for bytes (ignore the -2 in
> there). that's 7b total.
>
> In your no-cas test: 5 + 92 + 48 + 7 == 152. that's the exact size of slab
> 3. (152b).
>
> On Mon, 16 Nov 2015, Nicolas Martinez wrote:
>
> > The test was :
> > •Key : test
> > •Data : 90 bytes
> > •Flags : 0
> >
> > :(
> >
> > Le lundi 16 novembre 2015 19:01:50 UTC+1, Nicolas Martinez a écrit :
> > Ok... i think i have understood what you say:
> >
> > Key (Characters Number) + 1 + Data (Characters Number) + Header +
> Chunk Size + CAS Size
> >
> > Header = Flags (Characters Number) + Key (Characters Numbers)
> >
> > + 2 bytes ( \r\n )
> >
> > + 4 bytes (2 spaces and 1 \r)
> >
> > Chunk Size = 48 bytes (default)
> >
> > CAS Size = 8 bytes (64 bits platform)
> >
> >
> > Seems to be good:
> >
> > 4 + 1 + 90 + (1+4+2+4) + 48 + 8 = 162 Bytes => Slab 4 (192 Bytes)
> >
> >
> > But, if i start Memcached with -C, it's wrong.
> >
> > 4 + 1 + 90 + (1+4+2+4) + 48 = 154 Bytes => Slab 3 (152 Bytes)
> >
> >
> > It must be in Slab4 (192 Bytes) No?
> >
> >
> > Le lundi 16 novembre 2015 15:37:59 UTC+1, Nicolas Martinez a
> écrit :
> > Thank you very much for yours answers.Ok for CAS... i don't
> use -C so i have to add 8 bytes
> >
> > i still don"t understand these lines :
> > >> 2b appended to data_bytes for an extra "\r\n" +
> > So, \r\n == 2b ?
> >
> > >> + 4 bytes for spaces and \r\n
> > which spaces?
> > What is this \r\n ? Isn't already counted before?
> > There is 1 "\r\n" and 1 "\"
> >
> > echo -e 'set 30 0 3600 30\r\n'$data'\r'| nc ${memcached_server}
> 11211
> >
> >
> > With your example:
> > * Key : 2c
> > * Val : 28 bytes
> > * Flg : 0 (1bytes)
> >
> > turns into:
> >* Key : 3b
> >
> > => key number characters + 1
> >* Val : 30b
> >
> > => 28 bytes + 2 bytes ("\r\n")
> >* Hdr : 4b + 3b == 7b
> >
> > => What are 4b?
> > => 3b are: flags (1b) + ??
> >* Itm : 56b
> >
> > => 48b + 8b (CAS)
> >=> 96b. which is the cap for slab 1 in a default setup.
> >
> >
> >
> > Thank you again.
> >
> > Le lundi 16 novembre 2015 00:31:07 UTC+1, Dormando a écrit :
> > Read carefully:
> >
> > item_size_ok(const size_t nkey, const int flags, const int nbytes)
> {
> >
> > passes:
> >
> > size_t ntotal = item_make_header(nkey + 1, flags, nbytes,
> >prefix, &nsuffix);
> >
> > Then conditionally:
> >
> > if (settings.use_cas) {
> > ntotal += sizeof(uint64_t);
> > }
> >
> > item_make_header is doing:
> > nsuffix = (uint8_t) snprintf(suffix, 40, " %d %d\r\n", flags,
> nbytes - 2);
> >
> > Then:
> >
> > return sizeof(item) + nkey + *nsuffix + nbytes;
> >
> > It's convoluted but shirt.
> >
> > the lengths are:
> > key +
> > 1 +
> > data_bytes +
> > 2b appended to data_bytes for an extra "\r\n" +
> > stringified rep of the flags + data length
> > + 4 bytes for spaces and \r\n (these are carriage returns, one
> byte each)
> > + 8b for CAS if enabled
> >
> > CAS can be turned off via the -C starttime arg. it takes up 8
> bytes.
> >
> > Example:
> >* Key : 2c
> >* Val : 28b
> >* Flg : 0 (1b)
> >
> > turns into:
> >* Key : 3b
> >* Val : 30b
> >* Hdr : 4b + 3b == 7b
> >* Itm : 56b
> >=> 96b. which is the cap for slab 1 in a default setup.
> >
> > It's tough to get it exact for small chunks due to the way the
> header is
> > added. You should ballpark or tune the -f value to align with your
> > observed data.
> >
> > On Sun, 15 Nov 2015, Nicolas Martinez wrote:
> >
> > > Hi,
> > > Is CAS always used?
> > > I
Re: HELP ! How Memcached objects are stored in the good SLAB
key + 1b
val + 2b
item struct 48b
[optional CAS] 8b
then the suffix trailer printf: please read the manpage for snprintf:
snprintf(suffix, 40, " %d %d\r\n", flags, nbytes - 2);
there are four characters in there. two spaces, and then \r and \n. the
two %d's change into: "0" for flags, and "90" for bytes (ignore the -2 in
there). that's 7b total.
In your no-cas test: 5 + 92 + 48 + 7 == 152. that's the exact size of slab
3. (152b).
On Mon, 16 Nov 2015, Nicolas Martinez wrote:
> The test was :
> •Key : test
> •Data : 90 bytes
> •Flags : 0
>
> :(
>
> Le lundi 16 novembre 2015 19:01:50 UTC+1, Nicolas Martinez a écrit :
> Ok... i think i have understood what you say:
>
> Key (Characters Number) + 1 + Data (Characters Number) + Header + Chunk
> Size + CAS Size
>
> Header = Flags (Characters Number) + Key (Characters Numbers)
>
> + 2 bytes ( \r\n )
>
> + 4 bytes (2 spaces and 1 \r)
>
> Chunk Size = 48 bytes (default)
>
> CAS Size = 8 bytes (64 bits platform)
>
>
> Seems to be good:
>
> 4 + 1 + 90 + (1+4+2+4) + 48 + 8 = 162 Bytes => Slab 4 (192 Bytes)
>
>
> But, if i start Memcached with -C, it's wrong.
>
> 4 + 1 + 90 + (1+4+2+4) + 48 = 154 Bytes => Slab 3 (152 Bytes)
>
>
> It must be in Slab4 (192 Bytes) No?
>
>
> Le lundi 16 novembre 2015 15:37:59 UTC+1, Nicolas Martinez a écrit :
> Thank you very much for yours answers.Ok for CAS... i don't use
> -C so i have to add 8 bytes
>
> i still don"t understand these lines :
> >> 2b appended to data_bytes for an extra "\r\n" +
> So, \r\n == 2b ?
>
> >> + 4 bytes for spaces and \r\n
> which spaces?
> What is this \r\n ? Isn't already counted before?
> There is 1 "\r\n" and 1 "\"
>
> echo -e 'set 30 0 3600 30\r\n'$data'\r'| nc ${memcached_server} 11211
>
>
> With your example:
> * Key : 2c
> * Val : 28 bytes
> * Flg : 0 (1bytes)
>
> turns into:
> * Key : 3b
>
> => key number characters + 1
> * Val : 30b
>
> => 28 bytes + 2 bytes ("\r\n")
> * Hdr : 4b + 3b == 7b
>
> => What are 4b?
> => 3b are: flags (1b) + ??
> * Itm : 56b
>
> => 48b + 8b (CAS)
> => 96b. which is the cap for slab 1 in a default setup.
>
>
>
> Thank you again.
>
> Le lundi 16 novembre 2015 00:31:07 UTC+1, Dormando a écrit :
> Read carefully:
>
> item_size_ok(const size_t nkey, const int flags, const int nbytes) {
>
> passes:
>
> size_t ntotal = item_make_header(nkey + 1, flags, nbytes,
> prefix, &nsuffix);
>
> Then conditionally:
>
> if (settings.use_cas) {
> ntotal += sizeof(uint64_t);
> }
>
> item_make_header is doing:
> nsuffix = (uint8_t) snprintf(suffix, 40, " %d %d\r\n", flags, nbytes -
> 2);
>
> Then:
>
> return sizeof(item) + nkey + *nsuffix + nbytes;
>
> It's convoluted but shirt.
>
> the lengths are:
> key +
> 1 +
> data_bytes +
> 2b appended to data_bytes for an extra "\r\n" +
> stringified rep of the flags + data length
> + 4 bytes for spaces and \r\n (these are carriage returns, one byte
> each)
> + 8b for CAS if enabled
>
> CAS can be turned off via the -C starttime arg. it takes up 8 bytes.
>
> Example:
> * Key : 2c
> * Val : 28b
> * Flg : 0 (1b)
>
> turns into:
> * Key : 3b
> * Val : 30b
> * Hdr : 4b + 3b == 7b
> * Itm : 56b
> => 96b. which is the cap for slab 1 in a default setup.
>
> It's tough to get it exact for small chunks due to the way the header is
> added. You should ballpark or tune the -f value to align with your
> observed data.
>
> On Sun, 15 Nov 2015, Nicolas Martinez wrote:
>
> > Hi,
> > Is CAS always used?
> > If yes, we have to always add 56 bytes to the KEY and VALUE ?
> > you don't count FLAGS characters??
> >
> > I've found that Flags's size (number of characters) impact the
> storage.
> >
> > Example:
> > * Key : 2 characters = 2 bytes
> > * Value : 28 characters = 28 bytes
> > * FLAGS : 1 characters = 1 bytes
> > => 31 bytes
> >
> > seems to take the same storage as
> > * Key : 1 characters = 1 bytes
> > * Value : 28 characters = 28 bytes
> > * FLAGS : 2 characters = 2 bytes
> > => 31 bytes ... wich is the limit to be stored in Slab1
> >
> > ok for the /r/n ... should take 4 bytes no?
> >
> > So, if we count 56 bytes for CAS :
> 56(cas)+31(key+value+flags)+4(/r/n)= 91
> >
> > Not good... :(
> >
> > where I'm wrong ??
> >
> > Le samedi 14 novembre 2015 23:55:12 UTC+1, Dormando a écrit :
> > The mysql docs don't speak for the main tree... that's their
> own thing.
> >
> > the "sizes" binary that comes with the sour
Re: HELP ! How Memcached objects are stored in the good SLAB
The test was :
•Key : test
•Data : 90 bytes
•Flags : 0
:(
Le lundi 16 novembre 2015 19:01:50 UTC+1, Nicolas Martinez a écrit :
>
> Ok... i think i have understood what you say:
>
> Key (Characters Number) + 1 + Data (Characters Number) + Header + Chunk
> Size + CAS Size
>
> Header = Flags (Characters Number) + Key (Characters Numbers)
>
> + 2 bytes ( \r\n )
>
> + 4 bytes (2 spaces and 1 \r)
>
> Chunk Size = 48 bytes (default)
>
> CAS Size = 8 bytes (64 bits platform)
>
> Seems to be good:
>
> 4 + 1 + 90 + (1+4+2+4) + 48 + 8 = 162 Bytes => Slab 4 (192 Bytes)
>
> But, if i start Memcached with -C, it's wrong.
>
> 4 + 1 + 90 + (1+4+2+4) + 48 = 154 Bytes => *Slab 3 (152 Bytes)*
>
> It must be in Slab4 (192 Bytes) No?
>
>
> Le lundi 16 novembre 2015 15:37:59 UTC+1, Nicolas Martinez a écrit :
>>
>> Thank you very much for yours answers.
>> Ok for CAS... i don't use -C so i have to add 8 bytes
>>
>> i still don"t understand these lines :
>> >> 2b appended to data_bytes for an extra "\r\n" +
>> So, \r\n == 2b ?
>>
>> >> + 4 bytes for spaces and \r\n
>> which spaces?
>> What is this \r\n ? Isn't already counted before?
>> There is 1 "\r\n" and 1 "\"
>>
>> echo -e 'set 30 0 3600 30\r\n'$data'\r'| nc ${memcached_server} 11211
>>
>>
>> With your example:
>> * Key : 2c
>> * Val : 28 bytes
>> * Flg : 0 (1bytes)
>>
>> turns into:
>>
>>> * Key : 3b
>>
>> => key number characters + 1
>>
>>> * Val : 30b
>>
>> => 28 bytes + 2 bytes ("\r\n")
>>
>>> * Hdr : 4b + 3b == 7b
>>
>> => What are 4b?
>> => 3b are: flags (1b) + ??
>>
>>> * Itm : 56b
>>
>> => 48b + 8b (CAS)
>>
>>> => 96b. which is the cap for slab 1 in a default setup.
>>
>>
>>
>> Thank you again.
>>
>> Le lundi 16 novembre 2015 00:31:07 UTC+1, Dormando a écrit :
>>>
>>> Read carefully:
>>>
>>> item_size_ok(const size_t nkey, const int flags, const int nbytes) {
>>>
>>> passes:
>>>
>>> size_t ntotal = item_make_header(nkey + 1, flags, nbytes,
>>> prefix, &nsuffix);
>>>
>>> Then conditionally:
>>>
>>> if (settings.use_cas) {
>>> ntotal += sizeof(uint64_t);
>>> }
>>>
>>> item_make_header is doing:
>>> nsuffix = (uint8_t) snprintf(suffix, 40, " %d %d\r\n", flags, nbytes -
>>> 2);
>>>
>>> Then:
>>>
>>> return sizeof(item) + nkey + *nsuffix + nbytes;
>>>
>>> It's convoluted but shirt.
>>>
>>> the lengths are:
>>> key +
>>> 1 +
>>> data_bytes +
>>> 2b appended to data_bytes for an extra "\r\n" +
>>> stringified rep of the flags + data length
>>> + 4 bytes for spaces and \r\n (these are carriage returns, one byte
>>> each)
>>> + 8b for CAS if enabled
>>>
>>> CAS can be turned off via the -C starttime arg. it takes up 8 bytes.
>>>
>>> Example:
>>> * Key : 2c
>>> * Val : 28b
>>> * Flg : 0 (1b)
>>>
>>> turns into:
>>> * Key : 3b
>>> * Val : 30b
>>> * Hdr : 4b + 3b == 7b
>>> * Itm : 56b
>>> => 96b. which is the cap for slab 1 in a default setup.
>>>
>>> It's tough to get it exact for small chunks due to the way the header is
>>> added. You should ballpark or tune the -f value to align with your
>>> observed data.
>>>
>>> On Sun, 15 Nov 2015, Nicolas Martinez wrote:
>>>
>>> > Hi,
>>> > Is CAS always used?
>>> > If yes, we have to always add 56 bytes to the KEY and VALUE ?
>>> > you don't count FLAGS characters??
>>> >
>>> > I've found that Flags's size (number of characters) impact the
>>> storage.
>>> >
>>> > Example:
>>> > * Key : 2 characters = 2 bytes
>>> > * Value : 28 characters = 28 bytes
>>> > * FLAGS : 1 characters = 1 bytes
>>> > => 31 bytes
>>> >
>>> > seems to take the same storage as
>>> > * Key : 1 characters = 1 bytes
>>> > * Value : 28 characters = 28 bytes
>>> > * FLAGS : 2 characters = 2 bytes
>>> > => 31 bytes ... wich is the limit to be stored in Slab1
>>> >
>>> > ok for the /r/n ... should take 4 bytes no?
>>> >
>>> > So, if we count 56 bytes for CAS :
>>> 56(cas)+31(key+value+flags)+4(/r/n)= 91
>>> >
>>> > Not good... :(
>>> >
>>> > where I'm wrong ??
>>> >
>>> > Le samedi 14 novembre 2015 23:55:12 UTC+1, Dormando a écrit :
>>> > The mysql docs don't speak for the main tree... that's their own
>>> thing.
>>> >
>>> > the "sizes" binary that comes with the source tree tells you how
>>> many
>>> > bytes an item will use (though I intend to add this output to
>>> the 'stats'
>>> > output somewhere). With CAS this is 56 bytes.
>>> >
>>> > 56 + 2 + 30 == 88. Class 1 by default (in 1.4.24) is 96 bytes,
>>> but the
>>> > item still ends up in class 2.
>>> >
>>> > Why is this? (unfortunately?) because memcached pre-renders part
>>> of the
>>> > text protocol into the item header:
>>> >
>>> > *nsuffix = (uint8_t) snprintf(suffix, 40, " %d %d\r\n", flags,
>>> nbytes -
>>> > 2);
>>> > return sizeof(item) + nkey + *nsuffix + nbytes;
>>> >
>>> > so the flags + length are getting flattened + \r\n add
Re: HELP ! How Memcached objects are stored in the good SLAB
Ok... i think i have understood what you say:
Key (Characters Number) + 1 + Data (Characters Number) + Header + Chunk
Size + CAS Size
Header = Flags (Characters Number) + Key (Characters Numbers)
+ 2 bytes ( \r\n )
+ 4 bytes (2 spaces and 1 \r)
Chunk Size = 48 bytes (default)
CAS Size = 8 bytes (64 bits platform)
Seems to be good:
4 + 1 + 90 + (1+4+2+4) + 48 + 8 = 162 Bytes => Slab 4 (192 Bytes)
But, if i start Memcached with -C, it's wrong.
4 + 1 + 90 + (1+4+2+4) + 48 = 154 Bytes => *Slab 3 (152 Bytes)*
It must be in Slab4 (192 Bytes) No?
Le lundi 16 novembre 2015 15:37:59 UTC+1, Nicolas Martinez a écrit :
>
> Thank you very much for yours answers.
> Ok for CAS... i don't use -C so i have to add 8 bytes
>
> i still don"t understand these lines :
> >> 2b appended to data_bytes for an extra "\r\n" +
> So, \r\n == 2b ?
>
> >> + 4 bytes for spaces and \r\n
> which spaces?
> What is this \r\n ? Isn't already counted before?
> There is 1 "\r\n" and 1 "\"
>
> echo -e 'set 30 0 3600 30\r\n'$data'\r'| nc ${memcached_server} 11211
>
>
> With your example:
> * Key : 2c
> * Val : 28 bytes
> * Flg : 0 (1bytes)
>
> turns into:
>
>> * Key : 3b
>
> => key number characters + 1
>
>> * Val : 30b
>
> => 28 bytes + 2 bytes ("\r\n")
>
>> * Hdr : 4b + 3b == 7b
>
> => What are 4b?
> => 3b are: flags (1b) + ??
>
>> * Itm : 56b
>
> => 48b + 8b (CAS)
>
>> => 96b. which is the cap for slab 1 in a default setup.
>
>
>
> Thank you again.
>
> Le lundi 16 novembre 2015 00:31:07 UTC+1, Dormando a écrit :
>>
>> Read carefully:
>>
>> item_size_ok(const size_t nkey, const int flags, const int nbytes) {
>>
>> passes:
>>
>> size_t ntotal = item_make_header(nkey + 1, flags, nbytes,
>> prefix, &nsuffix);
>>
>> Then conditionally:
>>
>> if (settings.use_cas) {
>> ntotal += sizeof(uint64_t);
>> }
>>
>> item_make_header is doing:
>> nsuffix = (uint8_t) snprintf(suffix, 40, " %d %d\r\n", flags, nbytes -
>> 2);
>>
>> Then:
>>
>> return sizeof(item) + nkey + *nsuffix + nbytes;
>>
>> It's convoluted but shirt.
>>
>> the lengths are:
>> key +
>> 1 +
>> data_bytes +
>> 2b appended to data_bytes for an extra "\r\n" +
>> stringified rep of the flags + data length
>> + 4 bytes for spaces and \r\n (these are carriage returns, one byte each)
>> + 8b for CAS if enabled
>>
>> CAS can be turned off via the -C starttime arg. it takes up 8 bytes.
>>
>> Example:
>> * Key : 2c
>> * Val : 28b
>> * Flg : 0 (1b)
>>
>> turns into:
>> * Key : 3b
>> * Val : 30b
>> * Hdr : 4b + 3b == 7b
>> * Itm : 56b
>> => 96b. which is the cap for slab 1 in a default setup.
>>
>> It's tough to get it exact for small chunks due to the way the header is
>> added. You should ballpark or tune the -f value to align with your
>> observed data.
>>
>> On Sun, 15 Nov 2015, Nicolas Martinez wrote:
>>
>> > Hi,
>> > Is CAS always used?
>> > If yes, we have to always add 56 bytes to the KEY and VALUE ?
>> > you don't count FLAGS characters??
>> >
>> > I've found that Flags's size (number of characters) impact the
>> storage.
>> >
>> > Example:
>> > * Key : 2 characters = 2 bytes
>> > * Value : 28 characters = 28 bytes
>> > * FLAGS : 1 characters = 1 bytes
>> > => 31 bytes
>> >
>> > seems to take the same storage as
>> > * Key : 1 characters = 1 bytes
>> > * Value : 28 characters = 28 bytes
>> > * FLAGS : 2 characters = 2 bytes
>> > => 31 bytes ... wich is the limit to be stored in Slab1
>> >
>> > ok for the /r/n ... should take 4 bytes no?
>> >
>> > So, if we count 56 bytes for CAS : 56(cas)+31(key+value+flags)+4(/r/n)=
>> 91
>> >
>> > Not good... :(
>> >
>> > where I'm wrong ??
>> >
>> > Le samedi 14 novembre 2015 23:55:12 UTC+1, Dormando a écrit :
>> > The mysql docs don't speak for the main tree... that's their own
>> thing.
>> >
>> > the "sizes" binary that comes with the source tree tells you how
>> many
>> > bytes an item will use (though I intend to add this output to the
>> 'stats'
>> > output somewhere). With CAS this is 56 bytes.
>> >
>> > 56 + 2 + 30 == 88. Class 1 by default (in 1.4.24) is 96 bytes,
>> but the
>> > item still ends up in class 2.
>> >
>> > Why is this? (unfortunately?) because memcached pre-renders part
>> of the
>> > text protocol into the item header:
>> >
>> > *nsuffix = (uint8_t) snprintf(suffix, 40, " %d %d\r\n", flags,
>> nbytes -
>> > 2);
>> > return sizeof(item) + nkey + *nsuffix + nbytes;
>> >
>> > so the flags + length are getting flattened + \r\n added to the
>> end.
>> > Together that's just enough to push it over the edge. It'd also
>> be nice to
>> > add a highly optimized numerics printf so I could twiddle options
>> to save
>> > a few bytes of memory in objects, but don't get your hopes up for
>> that
>> > happening soon :)
>> >
>> >
Re: HELP ! How Memcached objects are stored in the good SLAB
Thank you very much for yours answers.
Ok for CAS... i don't use -C so i have to add 8 bytes
i still don"t understand these lines :
>> 2b appended to data_bytes for an extra "\r\n" +
So, \r\n == 2b ?
>> + 4 bytes for spaces and \r\n
which spaces?
What is this \r\n ? Isn't already counted before?
There is 1 "\r\n" and 1 "\"
echo -e 'set 30 0 3600 30\r\n'$data'\r'| nc ${memcached_server} 11211
With your example:
* Key : 2c
* Val : 28 bytes
* Flg : 0 (1bytes)
turns into:
> * Key : 3b
=> key number characters + 1
> * Val : 30b
=> 28 bytes + 2 bytes ("\r\n")
> * Hdr : 4b + 3b == 7b
=> What are 4b?
=> 3b are: flags (1b) + ??
> * Itm : 56b
=> 48b + 8b (CAS)
> => 96b. which is the cap for slab 1 in a default setup.
Thank you again.
Le lundi 16 novembre 2015 00:31:07 UTC+1, Dormando a écrit :
>
> Read carefully:
>
> item_size_ok(const size_t nkey, const int flags, const int nbytes) {
>
> passes:
>
> size_t ntotal = item_make_header(nkey + 1, flags, nbytes,
> prefix, &nsuffix);
>
> Then conditionally:
>
> if (settings.use_cas) {
> ntotal += sizeof(uint64_t);
> }
>
> item_make_header is doing:
> nsuffix = (uint8_t) snprintf(suffix, 40, " %d %d\r\n", flags, nbytes - 2);
>
> Then:
>
> return sizeof(item) + nkey + *nsuffix + nbytes;
>
> It's convoluted but shirt.
>
> the lengths are:
> key +
> 1 +
> data_bytes +
> 2b appended to data_bytes for an extra "\r\n" +
> stringified rep of the flags + data length
> + 4 bytes for spaces and \r\n (these are carriage returns, one byte each)
> + 8b for CAS if enabled
>
> CAS can be turned off via the -C starttime arg. it takes up 8 bytes.
>
> Example:
> * Key : 2c
> * Val : 28b
> * Flg : 0 (1b)
>
> turns into:
> * Key : 3b
> * Val : 30b
> * Hdr : 4b + 3b == 7b
> * Itm : 56b
> => 96b. which is the cap for slab 1 in a default setup.
>
> It's tough to get it exact for small chunks due to the way the header is
> added. You should ballpark or tune the -f value to align with your
> observed data.
>
> On Sun, 15 Nov 2015, Nicolas Martinez wrote:
>
> > Hi,
> > Is CAS always used?
> > If yes, we have to always add 56 bytes to the KEY and VALUE ?
> > you don't count FLAGS characters??
> >
> > I've found that Flags's size (number of characters) impact the storage.
> >
> > Example:
> > * Key : 2 characters = 2 bytes
> > * Value : 28 characters = 28 bytes
> > * FLAGS : 1 characters = 1 bytes
> > => 31 bytes
> >
> > seems to take the same storage as
> > * Key : 1 characters = 1 bytes
> > * Value : 28 characters = 28 bytes
> > * FLAGS : 2 characters = 2 bytes
> > => 31 bytes ... wich is the limit to be stored in Slab1
> >
> > ok for the /r/n ... should take 4 bytes no?
> >
> > So, if we count 56 bytes for CAS : 56(cas)+31(key+value+flags)+4(/r/n)=
> 91
> >
> > Not good... :(
> >
> > where I'm wrong ??
> >
> > Le samedi 14 novembre 2015 23:55:12 UTC+1, Dormando a écrit :
> > The mysql docs don't speak for the main tree... that's their own
> thing.
> >
> > the "sizes" binary that comes with the source tree tells you how
> many
> > bytes an item will use (though I intend to add this output to the
> 'stats'
> > output somewhere). With CAS this is 56 bytes.
> >
> > 56 + 2 + 30 == 88. Class 1 by default (in 1.4.24) is 96 bytes, but
> the
> > item still ends up in class 2.
> >
> > Why is this? (unfortunately?) because memcached pre-renders part
> of the
> > text protocol into the item header:
> >
> > *nsuffix = (uint8_t) snprintf(suffix, 40, " %d %d\r\n", flags,
> nbytes -
> > 2);
> > return sizeof(item) + nkey + *nsuffix + nbytes;
> >
> > so the flags + length are getting flattened + \r\n added to the
> end.
> > Together that's just enough to push it over the edge. It'd also be
> nice to
> > add a highly optimized numerics printf so I could twiddle options
> to save
> > a few bytes of memory in objects, but don't get your hopes up for
> that
> > happening soon :)
> >
> > On Sat, 14 Nov 2015, Nicolas Martinez wrote:
> >
> > > Add: Memcached version : 1.4.4 (RedHat)
> > >
> > > Le samedi 14 novembre 2015 14:49:37 UTC+1, Nicolas Martinez a
> écrit :
> > > Hi, few days i'm reading Memcached documentation and
> blogs... and i don't understand how objects are stored.
> > >
> > > My test
> > >
> > > 3 slabs :
> > >
> > > * 96.0 Bytes
> > > * 120.0 Bytes
> > > * 240.0 Bytes
> > > Everywhere, it's told :
> > > * if data is < 96 Bytes, it will be stored in Slabs1 (96B)
> > > * if data > 96B and < 120B, it will be stored in Slabs2 (120B)
> > > * if data > 120B, it will be stored in Slabs3 (240B)
> > > * etc.
> > > BUT, for example, when i'm creat
Re: HELP ! How Memcached objects are stored in the good SLAB
Read carefully:
item_size_ok(const size_t nkey, const int flags, const int nbytes) {
passes:
size_t ntotal = item_make_header(nkey + 1, flags, nbytes,
prefix, &nsuffix);
Then conditionally:
if (settings.use_cas) {
ntotal += sizeof(uint64_t);
}
item_make_header is doing:
nsuffix = (uint8_t) snprintf(suffix, 40, " %d %d\r\n", flags, nbytes - 2);
Then:
return sizeof(item) + nkey + *nsuffix + nbytes;
It's convoluted but shirt.
the lengths are:
key +
1 +
data_bytes +
2b appended to data_bytes for an extra "\r\n" +
stringified rep of the flags + data length
+ 4 bytes for spaces and \r\n (these are carriage returns, one byte each)
+ 8b for CAS if enabled
CAS can be turned off via the -C starttime arg. it takes up 8 bytes.
Example:
* Key : 2c
* Val : 28b
* Flg : 0 (1b)
turns into:
* Key : 3b
* Val : 30b
* Hdr : 4b + 3b == 7b
* Itm : 56b
=> 96b. which is the cap for slab 1 in a default setup.
It's tough to get it exact for small chunks due to the way the header is
added. You should ballpark or tune the -f value to align with your
observed data.
On Sun, 15 Nov 2015, Nicolas Martinez wrote:
> Hi,
> Is CAS always used?
> If yes, we have to always add 56 bytes to the KEY and VALUE ?
> you don't count FLAGS characters??
>
> I've found that Flags's size (number of characters) impact the storage.
>
> Example:
> * Key : 2 characters = 2 bytes
> * Value : 28 characters = 28 bytes
> * FLAGS : 1 characters = 1 bytes
> => 31 bytes
>
> seems to take the same storage as
> * Key : 1 characters = 1 bytes
> * Value : 28 characters = 28 bytes
> * FLAGS : 2 characters = 2 bytes
> => 31 bytes ... wich is the limit to be stored in Slab1
>
> ok for the /r/n ... should take 4 bytes no?
>
> So, if we count 56 bytes for CAS : 56(cas)+31(key+value+flags)+4(/r/n)= 91
>
> Not good... :(
>
> where I'm wrong ??
>
> Le samedi 14 novembre 2015 23:55:12 UTC+1, Dormando a écrit :
> The mysql docs don't speak for the main tree... that's their own thing.
>
> the "sizes" binary that comes with the source tree tells you how many
> bytes an item will use (though I intend to add this output to the
> 'stats'
> output somewhere). With CAS this is 56 bytes.
>
> 56 + 2 + 30 == 88. Class 1 by default (in 1.4.24) is 96 bytes, but the
> item still ends up in class 2.
>
> Why is this? (unfortunately?) because memcached pre-renders part of the
> text protocol into the item header:
>
> *nsuffix = (uint8_t) snprintf(suffix, 40, " %d %d\r\n", flags, nbytes -
> 2);
> return sizeof(item) + nkey + *nsuffix + nbytes;
>
> so the flags + length are getting flattened + \r\n added to the end.
> Together that's just enough to push it over the edge. It'd also be nice
> to
> add a highly optimized numerics printf so I could twiddle options to
> save
> a few bytes of memory in objects, but don't get your hopes up for that
> happening soon :)
>
> On Sat, 14 Nov 2015, Nicolas Martinez wrote:
>
> > Add: Memcached version : 1.4.4 (RedHat)
> >
> > Le samedi 14 novembre 2015 14:49:37 UTC+1, Nicolas Martinez a écrit :
> > Hi, few days i'm reading Memcached documentation and blogs...
> and i don't understand how objects are stored.
> >
> > My test
> >
> > 3 slabs :
> >
> > * 96.0 Bytes
> > * 120.0 Bytes
> > * 240.0 Bytes
> > Everywhere, it's told :
> > * if data is < 96 Bytes, it will be stored in Slabs1 (96B)
> > * if data > 96B and < 120B, it will be stored in Slabs2 (120B)
> > * if data > 120B, it will be stored in Slabs3 (240B)
> > * etc.
> > BUT, for example, when i'm creating an 30B object, it's stored in
> Slab2 (120B), and NOT in Slab1 (96B).
> >
> > External sources:
> > For example, the default size for the smallest block is 88
> bytes (40 bytes of value, and the default 48 bytes for the key and flag
> data). If the size of the first item you store into the cache is less than 40
> bytes, then a slab with a block size of 88 bytes is created and the value
> stored.
> > =>
> https://dev.mysql.com/doc/mysql-ha-scalability/en/ha-memcached-using-memory.html
> >
> >
> > WRONG
> >
> > A slab class is a collection of pages divided into same sized
> chunks. Each slab class is referenced to by its chunk size. So we’ll have
> Slab class 80kb, Slab class 100kb and so on. When an object needs to be
> stored, its size determines where it gets stored. So if the object is larger
> than 80kb but less than 100kb, it gets stored into
> Slab
> > class 100kb.
> > =>
> http://returnfoo.com/2012/02/memcached-memory-allocation-and-optimization-2/
> >
> >
> > WRONG
> >
> > How i create an object:
> >
> > data=$(
Re: HELP ! How Memcached objects are stored in the good SLAB
Hi,
Is CAS always used?
If yes, we have to always add 56 bytes to the KEY and VALUE ?
you don't count FLAGS characters??
I've found that Flags's size (number of characters) impact the storage.
Example:
- Key : 2 characters = 2 bytes
- Value : 28 characters = 28 bytes
- FLAGS : 1 characters = 1 bytes
=> 31 bytes
seems to take the same storage as
- Key : 1 characters = 1 bytes
- Value : 28 characters = 28 bytes
- FLAGS : 2 characters = 2 bytes
=> 31 bytes ... wich is the limit to be stored in Slab1
ok for the /r/n ... should take 4 bytes no?
So, if we count 56 bytes for CAS : 56(cas)+31(key+value+flags)+4(/r/n)= 91
Not good... :(
where I'm wrong ??
Le samedi 14 novembre 2015 23:55:12 UTC+1, Dormando a écrit :
>
> The mysql docs don't speak for the main tree... that's their own thing.
>
> the "sizes" binary that comes with the source tree tells you how many
> bytes an item will use (though I intend to add this output to the 'stats'
> output somewhere). With CAS this is 56 bytes.
>
> 56 + 2 + 30 == 88. Class 1 by default (in 1.4.24) is 96 bytes, but the
> item still ends up in class 2.
>
> Why is this? (unfortunately?) because memcached pre-renders part of the
> text protocol into the item header:
>
> *nsuffix = (uint8_t) snprintf(suffix, 40, " %d %d\r\n", flags, nbytes -
> 2);
> return sizeof(item) + nkey + *nsuffix + nbytes;
>
> so the flags + length are getting flattened + \r\n added to the end.
> Together that's just enough to push it over the edge. It'd also be nice to
> add a highly optimized numerics printf so I could twiddle options to save
> a few bytes of memory in objects, but don't get your hopes up for that
> happening soon :)
>
> On Sat, 14 Nov 2015, Nicolas Martinez wrote:
>
> > Add: Memcached version : 1.4.4 (RedHat)
> >
> > Le samedi 14 novembre 2015 14:49:37 UTC+1, Nicolas Martinez a écrit :
> > Hi, few days i'm reading Memcached documentation and blogs... and
> i don't understand how objects are stored.
> >
> > My test
> >
> > 3 slabs :
> >
> > * 96.0 Bytes
> > * 120.0 Bytes
> > * 240.0 Bytes
> > Everywhere, it's told :
> > * if data is < 96 Bytes, it will be stored in Slabs1 (96B)
> > * if data > 96B and < 120B, it will be stored in Slabs2 (120B)
> > * if data > 120B, it will be stored in Slabs3 (240B)
> > * etc.
> > BUT, for example, when i'm creating an 30B object, it's stored in
> Slab2 (120B), and NOT in Slab1 (96B).
> >
> > External sources:
> > For example, the default size for the smallest block is 88 bytes
> (40 bytes of value, and the default 48 bytes for the key and flag data). If
> the size of the first item you store into the cache is less than 40 bytes,
> then a slab with a block size of 88 bytes is created and the value stored.
> > =>
> https://dev.mysql.com/doc/mysql-ha-scalability/en/ha-memcached-using-memory.html
>
> >
> >
> > WRONG
> >
> > A slab class is a collection of pages divided into same sized
> chunks. Each slab class is referenced to by its chunk size. So we’ll have
> Slab class 80kb, Slab class 100kb and so on. When an object needs to be
> stored, its size determines where it gets stored. So if the object is
> larger than 80kb but less than 100kb, it gets stored into Slab
> > class 100kb.
> > =>
> http://returnfoo.com/2012/02/memcached-memory-allocation-and-optimization-2/
> >
> >
> > WRONG
> >
> > How i create an object:
> >
> > data=$(pwgen 30 -c 1)
> > echo -e 'set 30 0 3600 30\r\n'$data'\r'| nc ${memcached_server}
> 11211
> >
> >
> > So, when 30B object is creating :
> > * key name : 30 = 2 bytes
> > * value: 30 characters = 30 bytes
> > * tags : 0 = 1 bytes
> > => All = 33 bytes
> > If i add the default 48b as explained on Mysql website : 33 + 48 = 81B
> ... so < Slab1 (91B)... but always stored in Slab2 (120B)
> >
> > So, the size used to store object in the good Slab is not:
> > * object value size
> > * sum of KEY, VALUE and TAGS in bytes
> > KEY size : 1 character = 1 B
> > VALUE size : 1 character = 1 B
> > TAGS size : 1 character = 1 B
> >
> > ... as read everywhere
> >
> > So, It seems that: (SUM of KEY+VALUE+TAGS )
> > * For slab1 96.0 Bytes, data stored if <= 31 B (SUM of 2+28+1 )
> > * For slab2 120.0 Bytes, data stored if <= 55 B (SUM of 2+52+1 )
> > * For slab3 152.0 Bytes, data stored if <= 87 B (SUM of 2+84+1 )
> > * For slab4 192.0 Bytes, data stored if <= 126 B (SUM of 3+122+1 )
> > * For slab5 240.0 Bytes, data stored if <= 174 B (SUM of 3+170+1 )
> > * etc.
> >
> > My configuration :
> > * Chunk Size : 48
> > * Chunk Growth Factore: 1,25
> > * Max Bytes: 64.0 MBytes
> >
> > So, someone could explain me how the data is stored in the right
> Slabs???
> > How calculate it???
> >
> > Thank you
> >
> >
> > --
> >
> > ---
> > You received this message because you are subscribed to the Google
> Groups "mem
Re: HELP ! How Memcached objects are stored in the good SLAB
The mysql docs don't speak for the main tree... that's their own thing.
the "sizes" binary that comes with the source tree tells you how many
bytes an item will use (though I intend to add this output to the 'stats'
output somewhere). With CAS this is 56 bytes.
56 + 2 + 30 == 88. Class 1 by default (in 1.4.24) is 96 bytes, but the
item still ends up in class 2.
Why is this? (unfortunately?) because memcached pre-renders part of the
text protocol into the item header:
*nsuffix = (uint8_t) snprintf(suffix, 40, " %d %d\r\n", flags, nbytes -
2);
return sizeof(item) + nkey + *nsuffix + nbytes;
so the flags + length are getting flattened + \r\n added to the end.
Together that's just enough to push it over the edge. It'd also be nice to
add a highly optimized numerics printf so I could twiddle options to save
a few bytes of memory in objects, but don't get your hopes up for that
happening soon :)
On Sat, 14 Nov 2015, Nicolas Martinez wrote:
> Add: Memcached version : 1.4.4 (RedHat)
>
> Le samedi 14 novembre 2015 14:49:37 UTC+1, Nicolas Martinez a écrit :
> Hi, few days i'm reading Memcached documentation and blogs... and i
> don't understand how objects are stored.
>
> My test
>
> 3 slabs :
>
> * 96.0 Bytes
> * 120.0 Bytes
> * 240.0 Bytes
> Everywhere, it's told :
> * if data is < 96 Bytes, it will be stored in Slabs1 (96B)
> * if data > 96B and < 120B, it will be stored in Slabs2 (120B)
> * if data > 120B, it will be stored in Slabs3 (240B)
> * etc.
> BUT, for example, when i'm creating an 30B object, it's stored in Slab2
> (120B), and NOT in Slab1 (96B).
>
> External sources:
> For example, the default size for the smallest block is 88 bytes (40
> bytes of value, and the default 48 bytes for the key and flag data). If the
> size of the first item you store into the cache is less than 40 bytes, then a
> slab with a block size of 88 bytes is created and the value stored.
> =>
> https://dev.mysql.com/doc/mysql-ha-scalability/en/ha-memcached-using-memory.html
>
>
> WRONG
>
> A slab class is a collection of pages divided into same sized chunks.
> Each slab class is referenced to by its chunk size. So we’ll have Slab class
> 80kb, Slab class 100kb and so on. When an object needs to be stored, its size
> determines where it gets stored. So if the object is larger than 80kb but
> less than 100kb, it gets stored into Slab
> class 100kb.
> =>
> http://returnfoo.com/2012/02/memcached-memory-allocation-and-optimization-2/
>
>
> WRONG
>
> How i create an object:
>
> data=$(pwgen 30 -c 1)
> echo -e 'set 30 0 3600 30\r\n'$data'\r'| nc ${memcached_server} 11211
>
>
> So, when 30B object is creating :
> * key name : 30 = 2 bytes
> * value: 30 characters = 30 bytes
> * tags : 0 = 1 bytes
> => All = 33 bytes
> If i add the default 48b as explained on Mysql website : 33 + 48 = 81B ... so
> < Slab1 (91B)... but always stored in Slab2 (120B)
>
> So, the size used to store object in the good Slab is not:
> * object value size
> * sum of KEY, VALUE and TAGS in bytes
> KEY size : 1 character = 1 B
> VALUE size : 1 character = 1 B
> TAGS size : 1 character = 1 B
>
> ... as read everywhere
>
> So, It seems that: (SUM of KEY+VALUE+TAGS )
> * For slab1 96.0 Bytes, data stored if <= 31 B (SUM of 2+28+1 )
> * For slab2 120.0 Bytes, data stored if <= 55 B (SUM of 2+52+1 )
> * For slab3 152.0 Bytes, data stored if <= 87 B (SUM of 2+84+1 )
> * For slab4 192.0 Bytes, data stored if <= 126 B (SUM of 3+122+1 )
> * For slab5 240.0 Bytes, data stored if <= 174 B (SUM of 3+170+1 )
> * etc.
>
> My configuration :
> * Chunk Size : 48
> * Chunk Growth Factore: 1,25
> * Max Bytes: 64.0 MBytes
>
> So, someone could explain me how the data is stored in the right Slabs???
> How calculate it???
>
> Thank you
>
>
> --
>
> ---
> You received this message because you are subscribed to the Google Groups
> "memcached" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to memcached+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>
>
Re: HELP ! How Memcached objects are stored in the good SLAB
Add: Memcached version : 1.4.4 (RedHat)
Le samedi 14 novembre 2015 14:49:37 UTC+1, Nicolas Martinez a écrit :
>
> Hi,
> few days i'm reading Memcached documentation and blogs... and i don't
> understand how objects are stored.
>
> *My test*
>
> 3 slabs :
>
>
>- 96.0 Bytes
> - 120.0 Bytes
> - 240.0 Bytes
>
> Everywhere, it's told :
>
>- if data is < 96 Bytes, it will be stored in Slabs1 (96B)
>- if data > 96B and < 120B, it will be stored in Slabs2 (120B)
>- if data > 120B, it will be stored in Slabs3 (240B)
>- etc.
>
> BUT, for example, when i'm creating an *30B* object, it's stored in Slab2
> (*120B*), and NOT in Slab1 (*96B*).
>
> External sources:
>
>> For example, the default size for the smallest block is 88 bytes (40
>> bytes of value, and the default 48 bytes for the key and flag data). If the
>> size of the first item you store into the cache is less than 40 bytes, then
>> a slab with a block size of 88 bytes is created and the value stored.
>> =>
>> https://dev.mysql.com/doc/mysql-ha-scalability/en/ha-memcached-using-memory.html
>
>
> *WRONG*
>
> A slab class is a collection of pages divided into same sized chunks. Each
>> slab class is referenced to by its chunk size. So we’ll have Slab class
>> 80kb, Slab class 100kb and so on. When an object needs to be stored, its
>> size determines where it gets stored. So if the object is larger than 80kb
>> but less than 100kb, it gets stored into Slab class 100kb.
>> =>
>> http://returnfoo.com/2012/02/memcached-memory-allocation-and-optimization-2/
>
>
> *WRONG*
>
> *How i create an object:*
>
> data=$(pwgen 30 -c 1)
>> echo -e 'set 30 0 3600 30\r\n'$data'\r'| nc ${memcached_server} 11211
>
>
> So, when 30B object is creating :
>
>- key name : 30 = 2 bytes
>- value: 30 characters = 30 bytes
>- tags : 0 = 1 bytes
>
> => All = 33 bytes
> If i add the default 48b *as explained on Mysql website* : 33 + 48 = 81B
> ... so < Slab1 (91B)... but always stored in Slab2 (120B)
>
> So, *the size used to store object* in the good Slab *is not*:
>
>- object value size
>- sum of KEY, VALUE and TAGS in bytes
>
> KEY size : 1 character = 1 B
> VALUE size : 1 character = 1 B
> TAGS size : 1 character = 1 B
>
> ... as read everywhere
>
> So, It seems that: (SUM of KEY+VALUE+TAGS )
>
>- For slab1 96.0 Bytes, data stored if <= 31 B (SUM of 2+28+1 )
>- For slab2 120.0 Bytes, data stored if <= 55 B (SUM of 2+52+1 )
>- For slab3 152.0 Bytes, data stored if <= 87 B (SUM of 2+84+1 )
>- For slab4 192.0 Bytes, data stored if <= 126 B (SUM of 3+122+1 )
>- For slab5 240.0 Bytes, data stored if <= 174 B (SUM of 3+170+1 )
>- etc.
>
>
> My configuration :
>
>- Chunk Size : 48
>- Chunk Growth Factore: 1,25
>- Max Bytes: 64.0 MBytes
>
>
> So, someone could explain me how the data is stored in the right Slabs???
> How calculate it???
>
> Thank you
>
>
>
--
---
You received this message because you are subscribed to the Google Groups
"memcached" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to memcached+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
