Re: get bytes?
Everyone, thank you very much for the dialog. At this point I need to flush out more of our use case and prototype/test some implementations. As a side note, I've been doing C# for the last 5 years or so and been away from Java. One thing I always missed about the Java space is the community, there's nothing like it. Even just the short dialog everyone has given me here has been invaluable, thanks again! Once I settle in on what works best for us I'll post a wrap-up here just for closure. Jim
RE: get bytes?
> It's not a matter of memcached being bad, it's a matter of pulling > only X items off the big list in order to perform more efficiently. > Sometimes we want the entire list .. but also sometimes we just want > to look at the last 10 and see if there are new items vs. pulling the > entire list across the network and looking at only the top few items. If you just need to know whether there's new items been added, can't you simply store two values: one with the big list and one with a "timestamp" containing the last update? So you get the timestamp value and if it's the same as last time, don't do anything. If it's different, then store the new timestamp, fetch the big list and off you go... Dean.
Re: get bytes?
On Jan 19, 12:44 pm, jim wrote: > At least initially we would use normal java serialization. Since we > are talking about using prepend to put the most recent entries in our > list, we cannot use gzip anymore since it would not gel with > prepending bytes. Are you sure? % echo hi | gzip > /tmp/blah.gz % echo hi again | gzip >> /tmp/blah.gz % gzip -dc /tmp/blah.gz hi hi again > Did you use a custom serialization scheme? Or do you mean the binary > serializer? Because at some point you HAVE to serialize something to > a byte[], right? A series of 10,000 java longs serialized with plain java serialization, that ends up being 140122 bytes of data in my quick test. A textual (space separated) representation is up to 20 bytes (though could be as low as about 20k). A naïve byte-packed representation would be 80,000 bytes. Compression on this will vary based on your actual values, of course. > Did you prepend data at all? I don't see many > people talking about utilizing the prepend/append methods within the > protocol, I'm also trying to figure out why this is. It makes me a bit uncomfortable to have limited visibility into growth, but it should work. > Even though it appears the network fetch of even 10,000 longs isn't > much data, since we have so many of these lists to process it ends up > becoming a large network hit once you've done it 200k+ times. Yeah, could take a few minutes to do them all at the same time if they were all full and there was no compression. I don't fully understand your use case. Perhaps you just need a notification mechanism to invalidate local cache when upstream changes. I've used memcached for that with a version number in my data + a key with the version number in it. I'll fetch the small value (just the version number) on request and compare it to my internal version number. If it's different or missing, I fetch the large value and memoize it. If it's the same, I know I'd get the same value again. When I update, I update the large value and then the version number. I found I had to do this with an application where the bulk of my time was spent transferring stuff out of memcached and converting the data back into my internal-memory representation.
Re: get bytes?
Did you use a custom serialization scheme? Or do you mean the binary serializer? Because at some point you HAVE to serialize something to a byte[], right? Did you prepend data at all? I don't see many people talking about utilizing the prepend/append methods within the protocol, I'm also trying to figure out why this is. Well, that is because they are relatively new compared to the life of memcached. Brian.
Re: get bytes?
That's pretty much exactly what we do in some cases. prepend/append packed bytes so the list is in order. Then we pull the data and read as much as we need without touching serialization. If the overhead is a big deal you could fiddle with it a bit. Store 100 or 1,000 items per list and use multiple keys (if you can divide the numbers by something to be able to do that easily). Or have a main 10k list and then a separate "recent" key that you expire once it gets past 100 items or so. Then you're still occasionally doing the full fetch, but not nearly as often. Updating both keys ends up being the same prepend operation. -Dormando On Tue, 19 Jan 2010, jim wrote: > At least initially we would use normal java serialization. Since we > are talking about using prepend to put the most recent entries in our > list, we cannot use gzip anymore since it would not gel with > prepending bytes. > > Did you use a custom serialization scheme? Or do you mean the binary > serializer? Because at some point you HAVE to serialize something to > a byte[], right? Did you prepend data at all? I don't see many > people talking about utilizing the prepend/append methods within the > protocol, I'm also trying to figure out why this is. > > Even though it appears the network fetch of even 10,000 longs isn't > much data, since we have so many of these lists to process it ends up > becoming a large network hit once you've done it 200k+ times. > > Jim > > On Jan 19, 3:05 pm, dormando wrote: > > How're you serializing the list? > > > > In cases where I've had to work with that, we ensure the value is a flat > > list of packed numerics, and run no serialization on them. Then you have > > the overhead of a network fetch, but testing values within the list is > > nearly free. > > > > -Dormando > > > > On Tue, 19 Jan 2010, jim wrote: > > > Hi: > > > > > I've recently started on a project that is utilizing memcached to > > > buffer lists of 'long's that represent key values for larger more > > > detailed objects to load if needed. This list will have a maximum of > > > 10,000 items. > > > > > ex: > > > > > List of longs: > > > key : actList value : maximum 10,000 longs sorted chronologically > > > based on their 'detailed' object timestamp. > > > > > Detailed entry: > > > key : act_ value : text serialized object representing detailed > > > activity object. is contained on the 'List of longs' above. > > > > > My question is there are times where we would like to load only the > > > first block of entries from the list of longs, say 10 records. We > > > would then look at those 10 records and see if they are new based on > > > what a currently displayed list shows and if so grab these new entries > > > without pulling all 10,000 across from cache only to utilize the > > > relatively small number of new entries on the top of the list. This > > > kind of goes hand in hand with the prepend (or append) operations > > > where when new activities arrive we push these new activities into the > > > front of the 'List of longs' and it's these new entries that clients > > > may be interested in if they do not yet have them. > > > > > My question is, is there a way to do this? Is there a way to grab > > > only X bytes from a value in memcache? I read over the protocol > > > document and it doesn't appear there is. Is there any interest or > > > valid use case that this seems to fill for other users? > > > > > An alternate solution I can see if to store the 'list of longs' as a > > > flat list of keys with a naming convention, such as actList_1, > > > actList_2, etc. However, this will obviously lead to an extremely > > > long key name in the multi-key 'get' as well as lots of churn in > > > managing these objects .. so it appears far less than ideal. However, > > > we have many (200,000) lists of these 10,000 item 'List of longs' > > > which leads to pulling loads of data over the wire when a large update > > > occurs that needs to be communicated with cache ... it would be much > > > more efficient to only go after a certain number of bytes in cache vs. > > > the entire cached value. > > > > > Any other thoughts? Ideas? > > > > > Thank you. > > > Jim >
Re: get bytes?
At least initially we would use normal java serialization. Since we are talking about using prepend to put the most recent entries in our list, we cannot use gzip anymore since it would not gel with prepending bytes. Did you use a custom serialization scheme? Or do you mean the binary serializer? Because at some point you HAVE to serialize something to a byte[], right? Did you prepend data at all? I don't see many people talking about utilizing the prepend/append methods within the protocol, I'm also trying to figure out why this is. Even though it appears the network fetch of even 10,000 longs isn't much data, since we have so many of these lists to process it ends up becoming a large network hit once you've done it 200k+ times. Jim On Jan 19, 3:05 pm, dormando wrote: > How're you serializing the list? > > In cases where I've had to work with that, we ensure the value is a flat > list of packed numerics, and run no serialization on them. Then you have > the overhead of a network fetch, but testing values within the list is > nearly free. > > -Dormando > > On Tue, 19 Jan 2010, jim wrote: > > Hi: > > > I've recently started on a project that is utilizing memcached to > > buffer lists of 'long's that represent key values for larger more > > detailed objects to load if needed. This list will have a maximum of > > 10,000 items. > > > ex: > > > List of longs: > > key : actList value : maximum 10,000 longs sorted chronologically > > based on their 'detailed' object timestamp. > > > Detailed entry: > > key : act_ value : text serialized object representing detailed > > activity object. is contained on the 'List of longs' above. > > > My question is there are times where we would like to load only the > > first block of entries from the list of longs, say 10 records. We > > would then look at those 10 records and see if they are new based on > > what a currently displayed list shows and if so grab these new entries > > without pulling all 10,000 across from cache only to utilize the > > relatively small number of new entries on the top of the list. This > > kind of goes hand in hand with the prepend (or append) operations > > where when new activities arrive we push these new activities into the > > front of the 'List of longs' and it's these new entries that clients > > may be interested in if they do not yet have them. > > > My question is, is there a way to do this? Is there a way to grab > > only X bytes from a value in memcache? I read over the protocol > > document and it doesn't appear there is. Is there any interest or > > valid use case that this seems to fill for other users? > > > An alternate solution I can see if to store the 'list of longs' as a > > flat list of keys with a naming convention, such as actList_1, > > actList_2, etc. However, this will obviously lead to an extremely > > long key name in the multi-key 'get' as well as lots of churn in > > managing these objects .. so it appears far less than ideal. However, > > we have many (200,000) lists of these 10,000 item 'List of longs' > > which leads to pulling loads of data over the wire when a large update > > occurs that needs to be communicated with cache ... it would be much > > more efficient to only go after a certain number of bytes in cache vs. > > the entire cached value. > > > Any other thoughts? Ideas? > > > Thank you. > > Jim
Re: get bytes?
At least initially we would use normal java serialization. Since we are talking about using prepend to put the most recent entries in our list, we cannot use gzip anymore since it would not gel with prepending bytes. Did you use a custom serialization scheme? Or do you mean the binary serializer? Because at some point you HAVE to serialize something to a byte[], right? Did you prepend data at all? I don't see many people talking about utilizing the prepend/append methods within the protocol, I'm also trying to figure out why this is. Even though it appears the network fetch of even 10,000 longs isn't much data, since we have so many of these lists to process it ends up becoming a large network hit once you've done it 200k+ times. Jim On Jan 19, 3:05 pm, dormando wrote: > How're you serializing the list? > > In cases where I've had to work with that, we ensure the value is a flat > list of packed numerics, and run no serialization on them. Then you have > the overhead of a network fetch, but testing values within the list is > nearly free. > > -Dormando > > On Tue, 19 Jan 2010, jim wrote: > > Hi: > > > I've recently started on a project that is utilizing memcached to > > buffer lists of 'long's that represent key values for larger more > > detailed objects to load if needed. This list will have a maximum of > > 10,000 items. > > > ex: > > > List of longs: > > key : actList value : maximum 10,000 longs sorted chronologically > > based on their 'detailed' object timestamp. > > > Detailed entry: > > key : act_ value : text serialized object representing detailed > > activity object. is contained on the 'List of longs' above. > > > My question is there are times where we would like to load only the > > first block of entries from the list of longs, say 10 records. We > > would then look at those 10 records and see if they are new based on > > what a currently displayed list shows and if so grab these new entries > > without pulling all 10,000 across from cache only to utilize the > > relatively small number of new entries on the top of the list. This > > kind of goes hand in hand with the prepend (or append) operations > > where when new activities arrive we push these new activities into the > > front of the 'List of longs' and it's these new entries that clients > > may be interested in if they do not yet have them. > > > My question is, is there a way to do this? Is there a way to grab > > only X bytes from a value in memcache? I read over the protocol > > document and it doesn't appear there is. Is there any interest or > > valid use case that this seems to fill for other users? > > > An alternate solution I can see if to store the 'list of longs' as a > > flat list of keys with a naming convention, such as actList_1, > > actList_2, etc. However, this will obviously lead to an extremely > > long key name in the multi-key 'get' as well as lots of churn in > > managing these objects .. so it appears far less than ideal. However, > > we have many (200,000) lists of these 10,000 item 'List of longs' > > which leads to pulling loads of data over the wire when a large update > > occurs that needs to be communicated with cache ... it would be much > > more efficient to only go after a certain number of bytes in cache vs. > > the entire cached value. > > > Any other thoughts? Ideas? > > > Thank you. > > Jim
Re: get bytes?
Ideally since this is a byte[] we would just get the X # of bytes that make up the new entries. So we would fetch only the new entries. Since this is an array of longs we are storing, this should be pretty easy to ascertain the right size to grab. Now don't get me wrong, other times we use the entire list as cached. But being able to pop off X bytes from a value and allowing the client the ability to then turn that byte[] into something useful is what I'm talking about. On Jan 19, 2:55 pm, Boris Partensky wrote: > Would you *always* come back and get the entire list if you determine that > the last 10 are new? > > > > On Tue, Jan 19, 2010 at 2:42 PM, jim wrote: > > It's not a matter of memcached being bad, it's a matter of pulling > > only X items off the big list in order to perform more efficiently. > > Sometimes we want the entire list .. but also sometimes we just want > > to look at the last 10 and see if there are new items vs. pulling the > > entire list across the network and looking at only the top few items. > > > It seems since memcached (at least for text serialization) is just > > storing serialized bytes I could say give me X byte chunks and my app > > would know how to do the deserialization. At least I THINK this > > sounds feasible. > > > On Jan 19, 12:20 pm, Boris Partensky > > wrote: > > > Can you always invalidate memcached entry when a new item is inserted > > into > > > the list? This way you can assume that what's in memcached is always > > good. > > > > Boris > > > > On Tue, Jan 19, 2010 at 9:11 AM, jim wrote: > > > > Hi: > > > > > I've recently started on a project that is utilizing memcached to > > > > buffer lists of 'long's that represent key values for larger more > > > > detailed objects to load if needed. This list will have a maximum of > > > > 10,000 items. > > > > > ex: > > > > > List of longs: > > > > key : actList value : maximum 10,000 longs sorted chronologically > > > > based on their 'detailed' object timestamp. > > > > > Detailed entry: > > > > key : act_ value : text serialized object representing detailed > > > > activity object. is contained on the 'List of longs' above. > > > > > My question is there are times where we would like to load only the > > > > first block of entries from the list of longs, say 10 records. We > > > > would then look at those 10 records and see if they are new based on > > > > what a currently displayed list shows and if so grab these new entries > > > > without pulling all 10,000 across from cache only to utilize the > > > > relatively small number of new entries on the top of the list. This > > > > kind of goes hand in hand with the prepend (or append) operations > > > > where when new activities arrive we push these new activities into the > > > > front of the 'List of longs' and it's these new entries that clients > > > > may be interested in if they do not yet have them. > > > > > My question is, is there a way to do this? Is there a way to grab > > > > only X bytes from a value in memcache? I read over the protocol > > > > document and it doesn't appear there is. Is there any interest or > > > > valid use case that this seems to fill for other users? > > > > > An alternate solution I can see if to store the 'list of longs' as a > > > > flat list of keys with a naming convention, such as actList_1, > > > > actList_2, etc. However, this will obviously lead to an extremely > > > > long key name in the multi-key 'get' as well as lots of churn in > > > > managing these objects .. so it appears far less than ideal. However, > > > > we have many (200,000) lists of these 10,000 item 'List of longs' > > > > which leads to pulling loads of data over the wire when a large update > > > > occurs that needs to be communicated with cache ... it would be much > > > > more efficient to only go after a certain number of bytes in cache vs. > > > > the entire cached value. > > > > > Any other thoughts? Ideas? > > > > > Thank you. > > > > Jim > > > > -- > > > --Boris > > -- > --Boris
Re: get bytes?
It's not a matter of memcached being bad, it's a matter of pulling only X items off the big list in order to perform more efficiently. Sometimes we want the entire list .. but also sometimes we just want to look at the last 10 and see if there are new items vs. pulling the entire list across the network and looking at only the top few items. It seems since memcached (at least for text serialization) is just storing serialized bytes I could say give me X byte chunks and my app would know how to do the deserialization. At least I THINK this sounds feasible. On Jan 19, 12:20 pm, Boris Partensky wrote: > Can you always invalidate memcached entry when a new item is inserted into > the list? This way you can assume that what's in memcached is always good. > > Boris > > > > On Tue, Jan 19, 2010 at 9:11 AM, jim wrote: > > Hi: > > > I've recently started on a project that is utilizing memcached to > > buffer lists of 'long's that represent key values for larger more > > detailed objects to load if needed. This list will have a maximum of > > 10,000 items. > > > ex: > > > List of longs: > > key : actList value : maximum 10,000 longs sorted chronologically > > based on their 'detailed' object timestamp. > > > Detailed entry: > > key : act_ value : text serialized object representing detailed > > activity object. is contained on the 'List of longs' above. > > > My question is there are times where we would like to load only the > > first block of entries from the list of longs, say 10 records. We > > would then look at those 10 records and see if they are new based on > > what a currently displayed list shows and if so grab these new entries > > without pulling all 10,000 across from cache only to utilize the > > relatively small number of new entries on the top of the list. This > > kind of goes hand in hand with the prepend (or append) operations > > where when new activities arrive we push these new activities into the > > front of the 'List of longs' and it's these new entries that clients > > may be interested in if they do not yet have them. > > > My question is, is there a way to do this? Is there a way to grab > > only X bytes from a value in memcache? I read over the protocol > > document and it doesn't appear there is. Is there any interest or > > valid use case that this seems to fill for other users? > > > An alternate solution I can see if to store the 'list of longs' as a > > flat list of keys with a naming convention, such as actList_1, > > actList_2, etc. However, this will obviously lead to an extremely > > long key name in the multi-key 'get' as well as lots of churn in > > managing these objects .. so it appears far less than ideal. However, > > we have many (200,000) lists of these 10,000 item 'List of longs' > > which leads to pulling loads of data over the wire when a large update > > occurs that needs to be communicated with cache ... it would be much > > more efficient to only go after a certain number of bytes in cache vs. > > the entire cached value. > > > Any other thoughts? Ideas? > > > Thank you. > > Jim > > -- > --Boris
Re: get bytes?
How're you serializing the list? In cases where I've had to work with that, we ensure the value is a flat list of packed numerics, and run no serialization on them. Then you have the overhead of a network fetch, but testing values within the list is nearly free. -Dormando On Tue, 19 Jan 2010, jim wrote: > Hi: > > I've recently started on a project that is utilizing memcached to > buffer lists of 'long's that represent key values for larger more > detailed objects to load if needed. This list will have a maximum of > 10,000 items. > > ex: > > List of longs: > key : actList value : maximum 10,000 longs sorted chronologically > based on their 'detailed' object timestamp. > > Detailed entry: > key : act_ value : text serialized object representing detailed > activity object. is contained on the 'List of longs' above. > > My question is there are times where we would like to load only the > first block of entries from the list of longs, say 10 records. We > would then look at those 10 records and see if they are new based on > what a currently displayed list shows and if so grab these new entries > without pulling all 10,000 across from cache only to utilize the > relatively small number of new entries on the top of the list. This > kind of goes hand in hand with the prepend (or append) operations > where when new activities arrive we push these new activities into the > front of the 'List of longs' and it's these new entries that clients > may be interested in if they do not yet have them. > > My question is, is there a way to do this? Is there a way to grab > only X bytes from a value in memcache? I read over the protocol > document and it doesn't appear there is. Is there any interest or > valid use case that this seems to fill for other users? > > An alternate solution I can see if to store the 'list of longs' as a > flat list of keys with a naming convention, such as actList_1, > actList_2, etc. However, this will obviously lead to an extremely > long key name in the multi-key 'get' as well as lots of churn in > managing these objects .. so it appears far less than ideal. However, > we have many (200,000) lists of these 10,000 item 'List of longs' > which leads to pulling loads of data over the wire when a large update > occurs that needs to be communicated with cache ... it would be much > more efficient to only go after a certain number of bytes in cache vs. > the entire cached value. > > Any other thoughts? Ideas? > > Thank you. > Jim > >
Re: get bytes?
Would you *always* come back and get the entire list if you determine that the last 10 are new? On Tue, Jan 19, 2010 at 2:42 PM, jim wrote: > It's not a matter of memcached being bad, it's a matter of pulling > only X items off the big list in order to perform more efficiently. > Sometimes we want the entire list .. but also sometimes we just want > to look at the last 10 and see if there are new items vs. pulling the > entire list across the network and looking at only the top few items. > > It seems since memcached (at least for text serialization) is just > storing serialized bytes I could say give me X byte chunks and my app > would know how to do the deserialization. At least I THINK this > sounds feasible. > > On Jan 19, 12:20 pm, Boris Partensky > wrote: > > Can you always invalidate memcached entry when a new item is inserted > into > > the list? This way you can assume that what's in memcached is always > good. > > > > Boris > > > > > > > > On Tue, Jan 19, 2010 at 9:11 AM, jim wrote: > > > Hi: > > > > > I've recently started on a project that is utilizing memcached to > > > buffer lists of 'long's that represent key values for larger more > > > detailed objects to load if needed. This list will have a maximum of > > > 10,000 items. > > > > > ex: > > > > > List of longs: > > > key : actList value : maximum 10,000 longs sorted chronologically > > > based on their 'detailed' object timestamp. > > > > > Detailed entry: > > > key : act_ value : text serialized object representing detailed > > > activity object. is contained on the 'List of longs' above. > > > > > My question is there are times where we would like to load only the > > > first block of entries from the list of longs, say 10 records. We > > > would then look at those 10 records and see if they are new based on > > > what a currently displayed list shows and if so grab these new entries > > > without pulling all 10,000 across from cache only to utilize the > > > relatively small number of new entries on the top of the list. This > > > kind of goes hand in hand with the prepend (or append) operations > > > where when new activities arrive we push these new activities into the > > > front of the 'List of longs' and it's these new entries that clients > > > may be interested in if they do not yet have them. > > > > > My question is, is there a way to do this? Is there a way to grab > > > only X bytes from a value in memcache? I read over the protocol > > > document and it doesn't appear there is. Is there any interest or > > > valid use case that this seems to fill for other users? > > > > > An alternate solution I can see if to store the 'list of longs' as a > > > flat list of keys with a naming convention, such as actList_1, > > > actList_2, etc. However, this will obviously lead to an extremely > > > long key name in the multi-key 'get' as well as lots of churn in > > > managing these objects .. so it appears far less than ideal. However, > > > we have many (200,000) lists of these 10,000 item 'List of longs' > > > which leads to pulling loads of data over the wire when a large update > > > occurs that needs to be communicated with cache ... it would be much > > > more efficient to only go after a certain number of bytes in cache vs. > > > the entire cached value. > > > > > Any other thoughts? Ideas? > > > > > Thank you. > > > Jim > > > > -- > > --Boris > -- --Boris
Re: get bytes?
It's not a matter of memcached being bad, it's a matter of pulling only X items off the big list in order to perform more efficiently. Sometimes we want the entire list .. but also sometimes we just want to look at the last 10 and see if there are new items vs. pulling the entire list across the network and looking at only the top few items. It seems since memcached (at least for text serialization) is just storing serialized bytes I could say give me X byte chunks and my app would know how to do the deserialization. At least I THINK this sounds feasible. On Jan 19, 12:20 pm, Boris Partensky wrote: > Can you always invalidate memcached entry when a new item is inserted into > the list? This way you can assume that what's in memcached is always good. > > Boris > > > > On Tue, Jan 19, 2010 at 9:11 AM, jim wrote: > > Hi: > > > I've recently started on a project that is utilizing memcached to > > buffer lists of 'long's that represent key values for larger more > > detailed objects to load if needed. This list will have a maximum of > > 10,000 items. > > > ex: > > > List of longs: > > key : actList value : maximum 10,000 longs sorted chronologically > > based on their 'detailed' object timestamp. > > > Detailed entry: > > key : act_ value : text serialized object representing detailed > > activity object. is contained on the 'List of longs' above. > > > My question is there are times where we would like to load only the > > first block of entries from the list of longs, say 10 records. We > > would then look at those 10 records and see if they are new based on > > what a currently displayed list shows and if so grab these new entries > > without pulling all 10,000 across from cache only to utilize the > > relatively small number of new entries on the top of the list. This > > kind of goes hand in hand with the prepend (or append) operations > > where when new activities arrive we push these new activities into the > > front of the 'List of longs' and it's these new entries that clients > > may be interested in if they do not yet have them. > > > My question is, is there a way to do this? Is there a way to grab > > only X bytes from a value in memcache? I read over the protocol > > document and it doesn't appear there is. Is there any interest or > > valid use case that this seems to fill for other users? > > > An alternate solution I can see if to store the 'list of longs' as a > > flat list of keys with a naming convention, such as actList_1, > > actList_2, etc. However, this will obviously lead to an extremely > > long key name in the multi-key 'get' as well as lots of churn in > > managing these objects .. so it appears far less than ideal. However, > > we have many (200,000) lists of these 10,000 item 'List of longs' > > which leads to pulling loads of data over the wire when a large update > > occurs that needs to be communicated with cache ... it would be much > > more efficient to only go after a certain number of bytes in cache vs. > > the entire cached value. > > > Any other thoughts? Ideas? > > > Thank you. > > Jim > > -- > --Boris
Re: get bytes?
Can you always invalidate memcached entry when a new item is inserted into the list? This way you can assume that what's in memcached is always good. Boris On Tue, Jan 19, 2010 at 9:11 AM, jim wrote: > Hi: > > I've recently started on a project that is utilizing memcached to > buffer lists of 'long's that represent key values for larger more > detailed objects to load if needed. This list will have a maximum of > 10,000 items. > > ex: > > List of longs: > key : actList value : maximum 10,000 longs sorted chronologically > based on their 'detailed' object timestamp. > > Detailed entry: > key : act_ value : text serialized object representing detailed > activity object. is contained on the 'List of longs' above. > > My question is there are times where we would like to load only the > first block of entries from the list of longs, say 10 records. We > would then look at those 10 records and see if they are new based on > what a currently displayed list shows and if so grab these new entries > without pulling all 10,000 across from cache only to utilize the > relatively small number of new entries on the top of the list. This > kind of goes hand in hand with the prepend (or append) operations > where when new activities arrive we push these new activities into the > front of the 'List of longs' and it's these new entries that clients > may be interested in if they do not yet have them. > > My question is, is there a way to do this? Is there a way to grab > only X bytes from a value in memcache? I read over the protocol > document and it doesn't appear there is. Is there any interest or > valid use case that this seems to fill for other users? > > An alternate solution I can see if to store the 'list of longs' as a > flat list of keys with a naming convention, such as actList_1, > actList_2, etc. However, this will obviously lead to an extremely > long key name in the multi-key 'get' as well as lots of churn in > managing these objects .. so it appears far less than ideal. However, > we have many (200,000) lists of these 10,000 item 'List of longs' > which leads to pulling loads of data over the wire when a large update > occurs that needs to be communicated with cache ... it would be much > more efficient to only go after a certain number of bytes in cache vs. > the entire cached value. > > Any other thoughts? Ideas? > > Thank you. > Jim > > -- --Boris
Re: get bytes?
No, this is not supported in memcached. Brian. http://brian.moonspot.net/ On 1/19/10 8:11 AM, jim wrote: Hi: I've recently started on a project that is utilizing memcached to buffer lists of 'long's that represent key values for larger more detailed objects to load if needed. This list will have a maximum of 10,000 items. ex: List of longs: key : actList value : maximum 10,000 longs sorted chronologically based on their 'detailed' object timestamp. Detailed entry: key : act_ value : text serialized object representing detailed activity object. is contained on the 'List of longs' above. My question is there are times where we would like to load only the first block of entries from the list of longs, say 10 records. We would then look at those 10 records and see if they are new based on what a currently displayed list shows and if so grab these new entries without pulling all 10,000 across from cache only to utilize the relatively small number of new entries on the top of the list. This kind of goes hand in hand with the prepend (or append) operations where when new activities arrive we push these new activities into the front of the 'List of longs' and it's these new entries that clients may be interested in if they do not yet have them. My question is, is there a way to do this? Is there a way to grab only X bytes from a value in memcache? I read over the protocol document and it doesn't appear there is. Is there any interest or valid use case that this seems to fill for other users? An alternate solution I can see if to store the 'list of longs' as a flat list of keys with a naming convention, such as actList_1, actList_2, etc. However, this will obviously lead to an extremely long key name in the multi-key 'get' as well as lots of churn in managing these objects .. so it appears far less than ideal. However, we have many (200,000) lists of these 10,000 item 'List of longs' which leads to pulling loads of data over the wire when a large update occurs that needs to be communicated with cache ... it would be much more efficient to only go after a certain number of bytes in cache vs. the entire cached value. Any other thoughts? Ideas? Thank you. Jim
