Re: Reviewing a script to demo memcache get-set race conditions problems if it is used for distributed locking.

[Nishant Varma]

Hi Dormando,

Normally in our application memcache is used for the purpose of caching 
database values but someone from the implementation team decided to use 
memcache for locking which lead us to this code that works 99% only. I am 
just trying to make it better especially since we know that race conditions 
exists.

In this case, I am trying to address a screen where we deal with inbound 
documents (read from a file system and show as a html checkbox) that needs 
to be moved to our database depending on the content. We want to avoid 
multiple users working on the same document as much as possible. If a 
document is in his cart, we consider that he is working on it. Its a b-2-b 
application so that is different from the normal b-2-c applications where 
cart means something else. We help other companies to use our tool to 
enhance their productivity.

A user created lock is tied up to the user session. When a user logs out or 
his session expires, his cache is always cleared removing all of his locks. 
Hence a time-out is always there for every user and by default it is 10 
minutes. He can also remove the locks if he wants. Further, on a daily 
basis the memcache is cleared as well. As example of how we track user 
locks, we not only store a key like filename: username when a user locks a 
file, we also create/append a username key which has list like 
[filename,filename] to ensure that both look-ups are possible and clearing 
out is easy when the user logs out. I was thinking if I need "cas" here, 
but I think it should be fine since we are not going to support if same 
user logs in from another machine.

Overall I felt that my scenario is some what different from the example and 
hence thought of clarifying everything before I can propose some solution 
because we already have something that works 99% of the time. I have lots 
of time on this because it is something I am doing voluntarily. It am not 
ignoring your advise but I wanted to make sure the use case is really 
clear. Sorry that I didn't mention that session time out and daily cache 
clearing which automatically takes care of removing locks if he doesn't 
voluntarily give up the lock or process it. Even if the moves out of the 
Document screen in our SPA we clear it as well. Its like a workflow.

Based on this please do let me know your thoughts. In my pseudo code I need 
to check if memcache is really up otherwise just proceed as usual without 
locks. We will get more file not found errors but that is better than 
stopping productivity. However I still am trying to find how to do this in 
Python.

I think we should use database but it was not considered maybe due to 
dynamic nature of this table. But won't the lookups costlier than hash 
maps? I am from support team so I can only suggest this to the 
implementation team.

There is no hurry. Like I said I voluntarily trying to make it a bit better 
using memcache itself. So please let me know when you have a chance.

Thanks,
Nishant

On Wednesday, June 8, 2016 at 11:58:00 PM UTC+5:30, Dormando wrote:
>
>
>
> On Tue, 7 Jun 2016, Nishant Varma wrote: 
>
> > There was too much noise removed all those posts :-) 
> >   key = filename  
> >   item = memcli:get(key)  
> >   if (! defined item) {  
> > if (memcli:add(key . "_lock", lock_timeout_time, 
> my_admin_username)) {  
> >[etc]  
> > } else {  
> ># lost the race, handle how you want  
> > }  
> >   } else if (item.value == my_admin_username) {  
> > # good to go for that future request  
> >   }  
> > 
> > 
> > In my scenario files are displayed from a certain folder (queue) in an 
> application to many users at the same time. Each users has a cart where 
> they can add files they are 
> > working upon. They are going to remove it from the cart or process it 
> from the cart. 
> > 
> > In the example pattern you have given, 
> > 
> > 1) The locking and the action are both happening together. However, we 
> want lock to move the item to the cart when the user clicks on the file. If 
> items are there in his cart, 
> > he can do anything he wants. Lock out time is also not needed, user will 
> remove it from the cart. [There is a flaw here what if someone manipulates 
> dom and add stuff to cart - 
> > but we won't worry about that scenario - your example actually takes 
> care of that since validation and action if combined] 
>
> What if a user adds something to their cart and never comes back? After 
> some amount of time you'd want to re-validate what's in their cart. ie; 
> fetch all of the values again, and do the add-dance if the gets don't say 
> they still own it. That's why the timeout is useful, even if it's very 
> long (like 8 hours, or days even). 
>
> If they lose their cart and log in again, do they lose all of their locks? 
> Is the cart stored in a session in a database somewhere? 
>
> > 2) There is a different key called filename + lock which feels a bi

Re: Reviewing a script to demo memcache get-set race conditions problems if it is used for distributed locking.

[Nishant Varma]

Hi Dormando,

Normally in our application memcache is used for the purpose of caching 
database values but someone from the implementation team decided to use 
memcache for locking which lead us to this code that works 99% only. I am 
just trying to make it better especially since we know that race conditions 
exists.

In this case, I am trying to address a screen where we deal with inbound 
documents (read from a file system) that needs to be moved to our DB 
depending on the context. We want to avoid multiple users working on the 
same document as much as possible. If a document is in his cart, we 
consider that he is working on it. Its a b-2-b application so that is 
different from the normal b-2-c applications where cart means something 
else. We help other companies to use our tool to enchance their 
productivity.

Locks are by default always tied up to the user session. So when 
administrator userlogs out, his cache is always cleared or if his session 
expires. A time-out is always implict for every user abd by default it is 
10 minutes. He can also remove the locks if he wants. Further on a daily 
basis the memcache is cleared as well. [We store a key like filename: user 
when a user locks a file which is what we have been discussing most of the 
same. We also append a user key which has a list like [filename,filename] 
to ensure that both lookups are possible and clear outs are easy when the 
user logs out etc. That reminds be that I probably need to use cas for the 
appending because of race conditions.] 

Overall I felt that the scenario is some what different here and thought of 
clarifying everything before I can propose something. I have lots of time 
on this because it is something I am doing voluntarily. It has not been my 
intention to ignore your advise but I wanted to make sure the use case is 
really clear but it seemed like we are doing it somewhat differently. It is 
my mistake that I didn't mention that session time out and daily cache 
clearing happens which automatically takes care of removing locks if he 
doesn't voluntarily give up the lock. As mentioned if he moves out of the 
Document screen, we clear it as well.

Based on this please do let me know your thoughts. Also I feel one flaw so 
far in my logic is that I need to check if memcache is really up otherwise 
just proceed as usual. We will get more errors but that is better than 
stopping productivity.

Answering some of your questions:

What if a user adds something to their cart and never comes back? It gets 
expired if he navigates out of the Documents screen in our SPA. It also 
gets expired when he times out or if he manually remove the lock. It will 
happen anyway at the end of the day if neither of these happens.

Why don't we use a database? I think we should use database but probably it 
was not considered due to dynamic nature of this table, but I can't answer 
that since it their decision so perhaps we just forward this to them.

I hope I am not taking out your time. There is no hurry on this like I said 
I voluntarily trying to make it a bit better.

Thanks,
Nishant


On Wednesday, June 8, 2016 at 11:58:00 PM UTC+5:30, Dormando wrote:
>
>
>
> On Tue, 7 Jun 2016, Nishant Varma wrote: 
>
> > There was too much noise removed all those posts :-) 
> >   key = filename  
> >   item = memcli:get(key)  
> >   if (! defined item) {  
> > if (memcli:add(key . "_lock", lock_timeout_time, 
> my_admin_username)) {  
> >[etc]  
> > } else {  
> ># lost the race, handle how you want  
> > }  
> >   } else if (item.value == my_admin_username) {  
> > # good to go for that future request  
> >   }  
> > 
> > 
> > In my scenario files are displayed from a certain folder (queue) in an 
> application to many users at the same time. Each users has a cart where 
> they can add files they are 
> > working upon. They are going to remove it from the cart or process it 
> from the cart. 
> > 
> > In the example pattern you have given, 
> > 
> > 1) The locking and the action are both happening together. However, we 
> want lock to move the item to the cart when the user clicks on the file. If 
> items are there in his cart, 
> > he can do anything he wants. Lock out time is also not needed, user will 
> remove it from the cart. [There is a flaw here what if someone manipulates 
> dom and add stuff to cart - 
> > but we won't worry about that scenario - your example actually takes 
> care of that since validation and action if combined] 
>
> What if a user adds something to their cart and never comes back? After 
> some amount of time you'd want to re-validate what's in their cart. ie; 
> fetch all of the values again, and do the add-dance if the gets don't say 
> they still own it. That's why the timeout is useful, even if it's very 
> long (like 8 hours, or days even). 
>
> If they lose their cart and log in again, do they lose all of their locks? 
> Is the cart s

Re: Reviewing a script to demo memcache get-set race conditions problems if it is used for distributed locking.

[dormando]

On Tue, 7 Jun 2016, Nishant Varma wrote:

> There was too much noise removed all those posts :-)
>   key = filename 
>   item = memcli:get(key) 
>   if (! defined item) { 
>     if (memcli:add(key . "_lock", lock_timeout_time, my_admin_username)) 
> { 
>        [etc] 
>     } else { 
>        # lost the race, handle how you want 
>     } 
>   } else if (item.value == my_admin_username) { 
>     # good to go for that future request 
>   } 
>
>
> In my scenario files are displayed from a certain folder (queue) in an 
> application to many users at the same time. Each users has a cart where they 
> can add files they are
> working upon. They are going to remove it from the cart or process it from 
> the cart.
>
> In the example pattern you have given,
>
> 1) The locking and the action are both happening together. However, we want 
> lock to move the item to the cart when the user clicks on the file. If items 
> are there in his cart,
> he can do anything he wants. Lock out time is also not needed, user will 
> remove it from the cart. [There is a flaw here what if someone manipulates 
> dom and add stuff to cart -
> but we won't worry about that scenario - your example actually takes care of 
> that since validation and action if combined]

What if a user adds something to their cart and never comes back? After
some amount of time you'd want to re-validate what's in their cart. ie;
fetch all of the values again, and do the add-dance if the gets don't say
they still own it. That's why the timeout is useful, even if it's very
long (like 8 hours, or days even).

If they lose their cart and log in again, do they lose all of their locks?
Is the cart stored in a session in a database somewhere?

> 2) There is a different key called filename + lock which feels a bit 
> redundant in my case because I can achieve everything I want by entering the 
> key as filename and like you
> said username as the value.
>
> I have loads of time to get this code in so I wanted to review this in detail 
> before suggesting something.

ok. I'm a bit confused because you've been swapping between use cases? at
first it was some kind of admin deal, now locking files, etc?

> It has one major flaw, memcache.add can also be False because server is down. 
> I will create a blocker because now users can't add anything - reducing with 
> productivity at the
> cost of stopping errors. So I also need to check if memcache client is up and 
> running, else don't worry about locks. That code is still not added.

I've said it a few times: the ghetto locking is for *advisory locks only*,
if you're gating real functionality on it you are severely, severely doing
it wrong. This must be accelerating something which is already in a
database. The system is designed to be lossy and tolerant to failure, and
it achieves that by being a layer on top of something else, conceptually.
Doing it this way guarantees pain forever: you will never want to restart
memcached, you will never want to upgrade it, you will never want to tune
it, because restarting it will break all of your active locks.

Is your database so slow that you can't note locks in it? Do you not have
a database and are using memcached as a layer in front of a file store?

It's very hard to help you. I want to, but you've been selectively
ignoring advice and it's hard to recommend using this method unless I
understand your use case extremely well: as I was asking above, what
happens when carts go away, what is the usage of a cart, how is a user
actually interacting with this thing, etc. Your own code below says it
breaks if memcached is down, does it handle a restart?

sorry. wish I could do more.

>
> def addToCart(filename, username):
>     ableToLock = memcache.add(filename, username)
>     if ableToLock:
>         # ableToLock can happen if the file is still present
>         # or if it was already processed.
>         if os.file.ispath(filename):
>             # I have a lock and file exists. Think of Cart as a
>             # JS Object from where you can pick items to "process".
>             return "Added To Cart"
>         else:
>             # I have a lock but looks like file was processed already.
>             # So removing the residual "key" created.
>             memcache.delete(filename)
>             return "Processed by another user."
>     else:
>         # "add" can also fail if remote server is down,
>         # but now we are not handling that now. It will
>         # block the user's ability to process anything.
>         user = memcache.get(filename)
>         if user and os.file.ispath(filename):
>            # I try my best to show the user processing it.
>            print "Being processed by %s" % user
>         else:
>            # But lost the race to find that.
>            print "Processed by another user."
>
>
> def process(source):
>     shutil.move(source, destination)
>     # filename => source
>     memcache.delete(

Re: Reviewing a script to demo memcache get-set race conditions problems if it is used for distributed locking.

[Nishant Varma]

There was too much noise removed all those posts :-)

key = filename 
> item = memcli:get(key) 
> if (! defined item) { 
>   if (memcli:add(key . "_lock", lock_timeout_time, my_admin_username)) { 
>  [etc] 
>   } else { 
>  # lost the race, handle how you want 
>   } 
> } else if (item.value == my_admin_username) { 
>   # good to go for that future request 
> } 
>

In my scenario files are displayed from a certain folder (queue) in an 
application to many users at the same time. Each users has a cart where 
they can add files they are working upon. They are going to remove it from 
the cart or process it from the cart.

In the example pattern you have given,

1) The locking and the action are both happening together. However, we want 
lock to move the item to the cart when the user clicks on the file. If 
items are there in his cart, he can do anything he wants. Lock out time is 
also not needed, user will remove it from the cart. [There is a flaw here 
what if someone manipulates dom and add stuff to cart - but we won't worry 
about that scenario - your example actually takes care of that since 
validation and action if combined]

2) There is a different key called filename + lock which feels a bit 
redundant in my case because I can achieve everything I want by entering 
the key as filename and like you said username as the value.

I have loads of time to get this code in so I wanted to review this in 
detail before suggesting something.

It has one major flaw, memcache.add can also be False because server is 
down. I will create a blocker because now users can't add anything - 
reducing with productivity at the cost of stopping errors. So I also need 
to check if memcache client is up and running, else don't worry about 
locks. That code is still not added.


def addToCart(filename, username):
ableToLock = memcache.add(filename, username)
if ableToLock:
# ableToLock can happen if the file is still present
# or if it was already processed.
if os.file.ispath(filename):
# I have a lock and file exists. Think of Cart as a
# JS Object from where you can pick items to "process".
return "Added To Cart"
else:
# I have a lock but looks like file was processed already.
# So removing the residual "key" created.
memcache.delete(filename)
return "Processed by another user." 
else:
# "add" can also fail if remote server is down,
# but now we are not handling that now. It will
# block the user's ability to process anything.
user = memcache.get(filename)
if user and os.file.ispath(filename):
   # I try my best to show the user processing it.
   print "Being processed by %s" % user
else:
   # But lost the race to find that.
   print "Processed by another user."


def process(source):
shutil.move(source, destination)
# filename => source 
memcache.delete(filename)




On Sunday, June 5, 2016 at 1:30:37 AM UTC+5:30, Dormando wrote:
>
> The pattern is identical between the one in the wiki and yours. Simply 
> move the delete of the key until you're done using the lock, which would 
> be in a separate request. 
>
> In your case, you would probably set the contents of the key to be the 
> name of the user who has it locked. 
>
> In the original pseudocode: 
> key  = "expensive_frontpage_item" 
> item = memcli:get(key) 
> if (! defined item) { 
> # Oh crap, we have to recache it! 
> # Give us 60 seconds to recache the item. 
> if (memcli:add(key . "_lock", 60)) { 
> item = fetch_expensive_thing_from_database 
> memcli:add(key, item, 86400) 
> memcli:delete(key . "_lock") 
> } else { 
> # Lost the race. We can do any number of things: 
> # - short sleep, then re-fetch. 
> # - try the above a few times, then slow-fetch and return the item 
> # - show the user a page without this expensive content 
> # - show some less expensive content 
> # - throw an error 
> } 
> } 
> return item 
>
> In yours: 
> key = filename 
> item = memcli:get(key) 
> if (! defined item) { 
>   if (memcli:add(key . "_lock", lock_timeout_time, my_admin_username)) { 
>  [etc] 
>   } else { 
>  # lost the race, handle how you want 
>   } 
> } else if (item.value == my_admin_username) { 
>   # good to go for that future request 
> } 
>
> Then when you're done holding the lock, delete the key. 
>
> On Sat, 4 Jun 2016, Nishant Varma wrote: 
>
> > I am reading 
> https://github.com/memcached/memcached/wiki/ProgrammingTricks#ghetto-central-locking,
>  
> it seems to deal with a slightly different lock scenario of getting some 
> > expensive item from Database to avoid "Stampeding" 
> > In my case its slightly different lock that I need. I show regular files 
> from a folder in a web application to many users. So, to "lock" a file 
> using 

Re: Reviewing a script to demo memcache get-set race conditions problems if it is used for distributed locking.

[Nishant Varma]

@dormando, The final version I can think of. It has one *fundamental flaw*, 
memcache.add can also be False because server is down. Now I am interfering 
with productivity at the cost of stopping errors. So I need to do all this 
only if memcache client is up and running, else don't worry about locks. 
That code is still not added.

I have loads of time to get a code in, so I wanted to review all the 
possibilities and also learn some stuff while at it.

Can you review it when you get some time? Maybe someone else can help too 
...

[I think the Ghetto Locking is more appropriate if I didn't have the "Cart" 
concept] but since we have it that way ... I feel, I should do it 
different. Also that example doesn't check if file still exists which is 
also important.

def addToCart(filename, username):
ableToLock = memcache.add(filename, username)
if ableToLock:
# ableToLock can happen if the file is still present
# or if it was already processed.
if os.file.ispath(filename):
# I have a lock and file exists. Think of Cart as a
# JS Object from where you can pick items to "process".
return "Added To Cart"
else:
# I have a lock but looks like file was processed already.
# So removing the residual "key" created.
memcache.delete(filename)
return "Processed by another user." 
else:
# "add" can also fail if remote server is down,
# but now we are not handling that now. It will
# block the user's ability to process anything.
user = memcache.get(filename)
if user and os.file.ispath(filename):
   # I try my best to show the user processing it.
   print "Being processed by %s" % user
else:
   # But lost the race to find that.
   print "Processed by another user."

def process(source):
shutil.move(source, destination)
# filename => source 
memcache.delete(filename)




On Monday, June 6, 2016 at 11:29:12 PM UTC+5:30, Nishant Varma wrote:
>
> One more enhancement! I am also reviewing it in 
> http://codereview.stackexchange.com/questions/131239/function-to-lock-a-file-using-memcache
>
> def addToCart(filename):
> ableToLock = memcache.add(filename, username)
> if ableToLock:
> if os.file.ispath(filename)
> # I have a lock and file exists.
> return "Added To Cart"
> else:
> # I have a lock but looks like file was processed already.
> memcache.delete(filename)
> return "Processed by another user." 
> else:
> user = memcache.get(filename)
> if user:
># I try my best to show the user processing it.
>print "Bring processed by %s" % user
> else:
># Lost the race to find that.
>print "Already processed."
>
> def process(source):
> shutil.move(source, destination)
> # filename => source 
> memcache.delete(filename)
>
>
> On Monday, June 6, 2016 at 7:07:21 PM UTC+5:30, Nishant Varma wrote:
>>
>> Enhanced it a bit to to check if the file is really present as well 
>> before I proceed ... Btw this has to be done after obtaining the lock 
>> otherwise again race condition  i.e not fileExists and ableToLock 
>> ableToLock and fileExists is the order.
>>
>> def addToCart(filename):
>> ableToLock = memcache.add(filename, username)
>> fileExists = os.file.ispath(filename)
>> if ableToLock:
>> if fileExists:
>> return "Added To Cart"
>> else:
>> memcache.delete(filename)
>> return "Processed by another user." 
>> else:
>> if fileExists:
>> return "Processed by another user."
>> else:
>> user = memcache.get(filename)
>> return "%s is Processing" % user
>>
>> I guess this should cover our scenario. Do you have any thoughts?
>>
>>
>> On Monday, June 6, 2016 at 5:20:28 PM UTC+5:30, Nishant Varma wrote:
>>>
>>> key = filename 
 item = memcli:get(key) 
 if (! defined item) { 
   if (memcli:add(key . "_lock", lock_timeout_time, my_admin_username)) 
 { 
  [etc] 
   } else { 
  # lost the race, handle how you want 
   } 
 } else if (item.value == my_admin_username) { 
   # good to go for that future request 
 } 
>>>
>>>
>>>
>>> My scenario is files are displayed from a certain folder in an 
>>> application. Each users has a "cart" where they can add files they are 
>>> working upon. They are going to remove it from the cart 
>>> (memcache.set(filename, None) or memcache.delete(filename)) or process it 
>>> from the cart (i.e move it to a DMS and archive it to another folder and 
>>> memcache.delete(filename)).
>>>
>>> In the example pattern you have given,
>>>
>>> 1) The locking and the action are both happening together. However, we 
>>> want lock to happen like picking the item a cart (i.e

Re: Reviewing a script to demo memcache get-set race conditions problems if it is used for distributed locking.

[Nishant Varma]

One more enhancement! I am also reviewing it 
in 
http://codereview.stackexchange.com/questions/131239/function-to-lock-a-file-using-memcache

def addToCart(filename):
ableToLock = memcache.add(filename, username)
if ableToLock:
if os.file.ispath(filename)
# I have a lock and file exists.
return "Added To Cart"
else:
# I have a lock but looks like file was processed already.
memcache.delete(filename)
return "Processed by another user." 
else:
user = memcache.get(filename)
if user:
   # I try my best to show the user processing it.
   print "Bring processed by %s" % user
else:
   # Lost the race to find that.
   print "Already processed."

def process(source):
shutil.move(source, destination)
# filename => source 
memcache.delete(filename)


On Monday, June 6, 2016 at 7:07:21 PM UTC+5:30, Nishant Varma wrote:
>
> Enhanced it a bit to to check if the file is really present as well before 
> I proceed ... Btw this has to be done after obtaining the lock otherwise 
> again race condition  i.e not fileExists and ableToLock ableToLock and 
> fileExists is the order.
>
> def addToCart(filename):
> ableToLock = memcache.add(filename, username)
> fileExists = os.file.ispath(filename)
> if ableToLock:
> if fileExists:
> return "Added To Cart"
> else:
> memcache.delete(filename)
> return "Processed by another user." 
> else:
> if fileExists:
> return "Processed by another user."
> else:
> user = memcache.get(filename)
> return "%s is Processing" % user
>
> I guess this should cover our scenario. Do you have any thoughts?
>
>
> On Monday, June 6, 2016 at 5:20:28 PM UTC+5:30, Nishant Varma wrote:
>>
>> key = filename 
>>> item = memcli:get(key) 
>>> if (! defined item) { 
>>>   if (memcli:add(key . "_lock", lock_timeout_time, my_admin_username)) { 
>>>  [etc] 
>>>   } else { 
>>>  # lost the race, handle how you want 
>>>   } 
>>> } else if (item.value == my_admin_username) { 
>>>   # good to go for that future request 
>>> } 
>>
>>
>>
>> My scenario is files are displayed from a certain folder in an 
>> application. Each users has a "cart" where they can add files they are 
>> working upon. They are going to remove it from the cart 
>> (memcache.set(filename, None) or memcache.delete(filename)) or process it 
>> from the cart (i.e move it to a DMS and archive it to another folder and 
>> memcache.delete(filename)).
>>
>> In the example pattern you have given,
>>
>> 1) The locking and the action are both happening together. However, we 
>> want lock to happen like picking the item a cart (i.e memcache.add, if it 
>> fails it doesn't go to his cart) when the user clicks on the file. If items 
>> are there in his cart, he can do anything he wants. So things like 
>> "lock_timeout" is not really needed for us I guess because end user decides 
>> when to remove also. [There is a flaw here what if someone manipulates DOM 
>> - but we don't worry that scenario, yours actually takes care of that since 
>> validation and action if combined]
>>
>> 2) You are creating a different key called filename + lock which feels a 
>> bit redundant in my case because I can achieve everything I want by 
>> entering the key as filename and like you said username as the value.
>>
>> 3) You are also having an additional get which is nice especially 
>> validating same user. It is something I need to think if I should support 
>> though. Usually the same user, if he logs in from other machine we don't 
>> need to support it. 
>>
>> Overall given my simplest scenario, wont this simple api just work:
>>
>> def addToCart(filename):
>> if memcache.add(filename, username):
>> return True
>> else:
>>return False
>>
>> API is as simple addToCart(filename).
>>
>> I felt that this simplest "add" works for me,  .. What are you thoughts?
>>
>> Also my get set was as simple as this, the more reason I felt the change 
>> should simple as well. Usually that happens.
>>
>> Thanks,
>> Nishant
>>
>>
>> On Sunday, June 5, 2016 at 1:30:37 AM UTC+5:30, Dormando wrote:
>>>
>>> The pattern is identical between the one in the wiki and yours. Simply 
>>> move the delete of the key until you're done using the lock, which would 
>>> be in a separate request. 
>>>
>>> In your case, you would probably set the contents of the key to be the 
>>> name of the user who has it locked. 
>>>
>>> In the original pseudocode: 
>>> key  = "expensive_frontpage_item" 
>>> item = memcli:get(key) 
>>> if (! defined item) { 
>>> # Oh crap, we have to recache it! 
>>> # Give us 60 seconds to recache the item. 
>>> if (memcli:add(key . "_lock", 60)) { 
>>> item = fetch_expensive_thing_from_database 
>>> memcli:add(key, item, 86400) 
>>> memcli:delet

Re: Reviewing a script to demo memcache get-set race conditions problems if it is used for distributed locking.

[Nishant Varma]

Enhanced it a bit to to check if the file is really present as well before 
I proceed ... Btw this has to be done after obtaining the lock otherwise 
again race condition  i.e not fileExists and ableToLock ableToLock and 
fileExists is the order.

def addToCart(filename):
ableToLock = memcache.add(filename, username)
fileExists = os.file.ispath(filename)
if ableToLock:
if fileExists:
return "Added To Cart"
else:
memcache.delete(filename)
return "Processed by another user." 
else:
if fileExists:
return "Processed by another user."
else:
user = memcache.get(filename)
return "%s is Processing" % user

I guess this should cover our scenario. Do you have any thoughts?


On Monday, June 6, 2016 at 5:20:28 PM UTC+5:30, Nishant Varma wrote:
>
> key = filename 
>> item = memcli:get(key) 
>> if (! defined item) { 
>>   if (memcli:add(key . "_lock", lock_timeout_time, my_admin_username)) { 
>>  [etc] 
>>   } else { 
>>  # lost the race, handle how you want 
>>   } 
>> } else if (item.value == my_admin_username) { 
>>   # good to go for that future request 
>> } 
>
>
>
> My scenario is files are displayed from a certain folder in an 
> application. Each users has a "cart" where they can add files they are 
> working upon. They are going to remove it from the cart 
> (memcache.set(filename, None) or memcache.delete(filename)) or process it 
> from the cart (i.e move it to a DMS and archive it to another folder and 
> memcache.delete(filename)).
>
> In the example pattern you have given,
>
> 1) The locking and the action are both happening together. However, we 
> want lock to happen like picking the item a cart (i.e memcache.add, if it 
> fails it doesn't go to his cart) when the user clicks on the file. If items 
> are there in his cart, he can do anything he wants. So things like 
> "lock_timeout" is not really needed for us I guess because end user decides 
> when to remove also. [There is a flaw here what if someone manipulates DOM 
> - but we don't worry that scenario, yours actually takes care of that since 
> validation and action if combined]
>
> 2) You are creating a different key called filename + lock which feels a 
> bit redundant in my case because I can achieve everything I want by 
> entering the key as filename and like you said username as the value.
>
> 3) You are also having an additional get which is nice especially 
> validating same user. It is something I need to think if I should support 
> though. Usually the same user, if he logs in from other machine we don't 
> need to support it. 
>
> Overall given my simplest scenario, wont this simple api just work:
>
> def addToCart(filename):
> if memcache.add(filename, username):
> return True
> else:
>return False
>
> API is as simple addToCart(filename).
>
> I felt that this simplest "add" works for me,  .. What are you thoughts?
>
> Also my get set was as simple as this, the more reason I felt the change 
> should simple as well. Usually that happens.
>
> Thanks,
> Nishant
>
>
> On Sunday, June 5, 2016 at 1:30:37 AM UTC+5:30, Dormando wrote:
>>
>> The pattern is identical between the one in the wiki and yours. Simply 
>> move the delete of the key until you're done using the lock, which would 
>> be in a separate request. 
>>
>> In your case, you would probably set the contents of the key to be the 
>> name of the user who has it locked. 
>>
>> In the original pseudocode: 
>> key  = "expensive_frontpage_item" 
>> item = memcli:get(key) 
>> if (! defined item) { 
>> # Oh crap, we have to recache it! 
>> # Give us 60 seconds to recache the item. 
>> if (memcli:add(key . "_lock", 60)) { 
>> item = fetch_expensive_thing_from_database 
>> memcli:add(key, item, 86400) 
>> memcli:delete(key . "_lock") 
>> } else { 
>> # Lost the race. We can do any number of things: 
>> # - short sleep, then re-fetch. 
>> # - try the above a few times, then slow-fetch and return the 
>> item 
>> # - show the user a page without this expensive content 
>> # - show some less expensive content 
>> # - throw an error 
>> } 
>> } 
>> return item 
>>
>> In yours: 
>> key = filename 
>> item = memcli:get(key) 
>> if (! defined item) { 
>>   if (memcli:add(key . "_lock", lock_timeout_time, my_admin_username)) { 
>>  [etc] 
>>   } else { 
>>  # lost the race, handle how you want 
>>   } 
>> } else if (item.value == my_admin_username) { 
>>   # good to go for that future request 
>> } 
>>
>> Then when you're done holding the lock, delete the key. 
>>
>> On Sat, 4 Jun 2016, Nishant Varma wrote: 
>>
>> > I am reading 
>> https://github.com/memcached/memcached/wiki/ProgrammingTricks#ghetto-central-locking,
>>  
>> it seems to deal with a slightly different lock scenario of getting some 
>> > expensive item from Database to avoid "Stam

Re: Reviewing a script to demo memcache get-set race conditions problems if it is used for distributed locking.

[Nishant Varma]

>
> key = filename 
> item = memcli:get(key) 
> if (! defined item) { 
>   if (memcli:add(key . "_lock", lock_timeout_time, my_admin_username)) { 
>  [etc] 
>   } else { 
>  # lost the race, handle how you want 
>   } 
> } else if (item.value == my_admin_username) { 
>   # good to go for that future request 
> } 



My scenario is files are displayed from a certain folder in an application. 
Each users has a "cart" where they can add files they are working upon. 
They are going to remove it from the cart (memcache.set(filename, None) or 
memcache.delete(filename)) or process it from the cart (i.e move it to a 
DMS and archive it to another folder and memcache.delete(filename)).

In the example pattern you have given,

1) The locking and the action are both happening together. However, we want 
lock to happen like picking the item a cart (i.e memcache.add, if it fails 
it doesn't go to his cart) when the user clicks on the file. If items are 
there in his cart, he can do anything he wants. So things like 
"lock_timeout" is not really needed for us I guess because end user decides 
when to remove also. [There is a flaw here what if someone manipulates DOM 
- but we don't worry that scenario, yours actually takes care of that since 
validation and action if combined]

2) You are creating a different key called filename + lock which feels a 
bit redundant in my case because I can achieve everything I want by 
entering the key as filename and like you said username as the value.

3) You are also having an additional get which is nice especially 
validating same user. It is something I need to think if I should support 
though. Usually the same user, if he logs in from other machine we don't 
need to support it. 

Overall given my simplest scenario, wont this simple api just work:

def addToCart(filename):
if memcache.add(filename, username):
return True
else:
   return False

API is as simple addToCart(filename).

I felt that this simplest "add" works for me,  .. What are you thoughts?

Also my get set was as simple as this, the more reason I felt the change 
should simple as well. Usually that happens.

Thanks,
Nishant


On Sunday, June 5, 2016 at 1:30:37 AM UTC+5:30, Dormando wrote:
>
> The pattern is identical between the one in the wiki and yours. Simply 
> move the delete of the key until you're done using the lock, which would 
> be in a separate request. 
>
> In your case, you would probably set the contents of the key to be the 
> name of the user who has it locked. 
>
> In the original pseudocode: 
> key  = "expensive_frontpage_item" 
> item = memcli:get(key) 
> if (! defined item) { 
> # Oh crap, we have to recache it! 
> # Give us 60 seconds to recache the item. 
> if (memcli:add(key . "_lock", 60)) { 
> item = fetch_expensive_thing_from_database 
> memcli:add(key, item, 86400) 
> memcli:delete(key . "_lock") 
> } else { 
> # Lost the race. We can do any number of things: 
> # - short sleep, then re-fetch. 
> # - try the above a few times, then slow-fetch and return the item 
> # - show the user a page without this expensive content 
> # - show some less expensive content 
> # - throw an error 
> } 
> } 
> return item 
>
> In yours: 
> key = filename 
> item = memcli:get(key) 
> if (! defined item) { 
>   if (memcli:add(key . "_lock", lock_timeout_time, my_admin_username)) { 
>  [etc] 
>   } else { 
>  # lost the race, handle how you want 
>   } 
> } else if (item.value == my_admin_username) { 
>   # good to go for that future request 
> } 
>
> Then when you're done holding the lock, delete the key. 
>
> On Sat, 4 Jun 2016, Nishant Varma wrote: 
>
> > I am reading 
> https://github.com/memcached/memcached/wiki/ProgrammingTricks#ghetto-central-locking,
>  
> it seems to deal with a slightly different lock scenario of getting some 
> > expensive item from Database to avoid "Stampeding" 
> > In my case its slightly different lock that I need. I show regular files 
> from a folder in a web application to many users. So, to "lock" a file 
> using memcache isn't this 
> > simple API sufficient or I still need that pattern :-)? 
> > def access(filename): 
> >  if memcache.add(filename, timestamp): 
> > return "Access Granted. Lock Obtained" # Normally this results 
> in checking HTML checkbox against the filename so User can do actions with 
> that/ 
> >  else: 
> > return "Access Denied" # Normally this leads to an alert saying 
> that someone else is working on this. 
> > 
> > Isn't this simple API using add good enough in my case? I am sorry if I 
> am repeating this, but I could not really relate the "fetching expensive 
> item from Database" to my 
> > scenario which is why I even wrote a simple script to test the validity 
> of the claim etc. 
> > 
> > Can you please let me know? 
> > 
> > 
> > On Saturday, June 4, 2016 at 6:42:35 PM UTC+5:30, Nishant

Re: Reviewing a script to demo memcache get-set race conditions problems if it is used for distributed locking.

[dormando]

The pattern is identical between the one in the wiki and yours. Simply
move the delete of the key until you're done using the lock, which would
be in a separate request.

In your case, you would probably set the contents of the key to be the
name of the user who has it locked.

In the original pseudocode:
key  = "expensive_frontpage_item"
item = memcli:get(key)
if (! defined item) {
# Oh crap, we have to recache it!
# Give us 60 seconds to recache the item.
if (memcli:add(key . "_lock", 60)) {
item = fetch_expensive_thing_from_database
memcli:add(key, item, 86400)
memcli:delete(key . "_lock")
} else {
# Lost the race. We can do any number of things:
# - short sleep, then re-fetch.
# - try the above a few times, then slow-fetch and return the item
# - show the user a page without this expensive content
# - show some less expensive content
# - throw an error
}
}
return item

In yours:
key = filename
item = memcli:get(key)
if (! defined item) {
  if (memcli:add(key . "_lock", lock_timeout_time, my_admin_username)) {
 [etc]
  } else {
 # lost the race, handle how you want
  }
} else if (item.value == my_admin_username) {
  # good to go for that future request
}

Then when you're done holding the lock, delete the key.

On Sat, 4 Jun 2016, Nishant Varma wrote:

> I am reading 
> https://github.com/memcached/memcached/wiki/ProgrammingTricks#ghetto-central-locking,
>  it seems to deal with a slightly different lock scenario of getting some
> expensive item from Database to avoid "Stampeding"
> In my case its slightly different lock that I need. I show regular files from 
> a folder in a web application to many users. So, to "lock" a file using 
> memcache isn't this
> simple API sufficient or I still need that pattern :-)?
> def access(filename):
>      if memcache.add(filename, timestamp):
>         return "Access Granted. Lock Obtained" # Normally this results in 
> checking HTML checkbox against the filename so User can do actions with that/
>      else:
>         return "Access Denied" # Normally this leads to an alert saying that 
> someone else is working on this.
>
> Isn't this simple API using add good enough in my case? I am sorry if I am 
> repeating this, but I could not really relate the "fetching expensive item 
> from Database" to my
> scenario which is why I even wrote a simple script to test the validity of 
> the claim etc.
>
> Can you please let me know?
>
>
> On Saturday, June 4, 2016 at 6:42:35 PM UTC+5:30, Nishant Varma wrote:
>   Excellent I rely on you. I guess this is the reason you say I am 
> over-engineering this problem. Makes sense :-) I will again check the link 
> you gave me. I will go
>   through the documentation this weekend.
>
>   On Saturday, June 4, 2016 at 1:33:04 PM UTC+5:30, Dormando wrote:
> Hey,
>
> You really don't need to test this: I'm telling you flatly, as an 
> author
> of this software and all of the documentation for it, that you 
> should
> absolutely not rely on that pattern. I'm trying to save you some 
> time.
>
> The pattern that is slightly better is written explicitly in 
> pseudocode in
> the link I gave you several times in the issue. Please use it.
>
> Thanks,
> -Dormando
>
> On Fri, 3 Jun 2016, Nishant Varma wrote:
>
> > Can anyone help me peer review this script 
> https://gist.github.com/varmanishant/0129286d41038cc21471652a6460a5ff that 
> demonstrate potential problems
> with get set if it is used
> > to implement distributed locking. I was suggested to modify 
> from get set to add in this thread 
> https://github.com/memcached/memcached/issues/163.
> However I wanted a small
> > simulation to demonstrate this.
> >
> > --
> >
> > ---
> > You received this message because you are subscribed to the 
> Google Groups "memcached" group.
> > To unsubscribe from this group and stop receiving emails from 
> it, send an email to memcached+...@googlegroups.com.
> > For more options, visit https://groups.google.com/d/optout.
> >
> >
>
>
> This e-mail message (including any attachments) may contain information that 
> is confidential, protected by the attorney-client or other applicable 
> privileges, or otherwise
> comprising non-public information. This message is intended to be conveyed 
> only to the designated recipient(s). If you have any reason to believe you 
> are not an intended
> recipient of this message, please notify the sender by replying to this 
> message and then deleting it from your system. Any use, dissemination, 
> distribution, or reproduction of
> this message by unintended recipients is not authorized and may be unlawful.
>
> --
>
> ---
> You received thi

Re: Reviewing a script to demo memcache get-set race conditions problems if it is used for distributed locking.

[Nishant Varma]

I did see Thread 1 
 
and Thread 2 
 as 
well. It also has some resources on this topic. Anyway, isn't the below 
pattern simple enough? Do we need any more complication in my scenario? 
Would help if you can clarify. Thanks.

Thanks,
Nishant

On Saturday, June 4, 2016 at 7:04:58 PM UTC+5:30, Nishant Varma wrote:
>
> I am reading 
> https://github.com/memcached/memcached/wiki/ProgrammingTricks#ghetto-central-locking,
>  
> it seems to deal with a slightly different lock scenario of getting some 
> expensive item from Database to avoid "Stampeding"
>
> In my case its slightly different lock that I need. I show regular files 
> from a folder in a web application to many users. So, to "lock" a file 
> using memcache isn't this simple API sufficient or I still need that 
> pattern :-)?
>
> def access(filename):
>  if memcache.add(filename, timestamp):
> return "Access Granted. Lock Obtained" # Normally this results in 
> checking HTML checkbox against the filename so User can do actions with 
> that/
>  else:
> return "Access Denied" # Normally this leads to an alert saying 
> that someone else is working on this.
>
> Isn't this simple API using add good enough in my case? I am sorry if I am 
> repeating this, but I could not really relate the "fetching expensive item 
> from Database" to my scenario which is why I even wrote a simple script to 
> test the validity of the claim etc.
>
> Can you please let me know?
>
>
> On Saturday, June 4, 2016 at 6:42:35 PM UTC+5:30, Nishant Varma wrote:
>>
>> Excellent I rely on you. I guess this is the reason you say I am 
>> over-engineering this problem. Makes sense :-) I will again check the link 
>> you gave me. I will go through the documentation this weekend.
>>
>> On Saturday, June 4, 2016 at 1:33:04 PM UTC+5:30, Dormando wrote:
>>>
>>> Hey, 
>>>
>>> You really don't need to test this: I'm telling you flatly, as an author 
>>> of this software and all of the documentation for it, that you should 
>>> absolutely not rely on that pattern. I'm trying to save you some time. 
>>>
>>> The pattern that is slightly better is written explicitly in pseudocode 
>>> in 
>>> the link I gave you several times in the issue. Please use it. 
>>>
>>> Thanks, 
>>> -Dormando 
>>>
>>> On Fri, 3 Jun 2016, Nishant Varma wrote: 
>>>
>>> > Can anyone help me peer review this script 
>>> https://gist.github.com/varmanishant/0129286d41038cc21471652a6460a5ff 
>>> that demonstrate potential problems with get set if it is used 
>>> > to implement distributed locking. I was suggested to modify from get 
>>> set to add in this thread 
>>> https://github.com/memcached/memcached/issues/163. However I wanted a 
>>> small 
>>> > simulation to demonstrate this. 
>>> > 
>>> > -- 
>>> > 
>>> > --- 
>>> > You received this message because you are subscribed to the Google 
>>> Groups "memcached" group. 
>>> > To unsubscribe from this group and stop receiving emails from it, send 
>>> an email to memcached+...@googlegroups.com. 
>>> > For more options, visit https://groups.google.com/d/optout. 
>>> > 
>>> > 
>>>
>>
-- 
This e-mail message (including any attachments) may contain information 
that is confidential, protected by the attorney-client or other applicable 
privileges, or otherwise comprising non-public information. This message is 
intended to be conveyed only to the designated recipient(s). If you have 
any reason to believe you are not an intended recipient of this message, 
please notify the sender by replying to this message and then deleting it 
from your system. Any use, dissemination, distribution, or reproduction of 
this message by unintended recipients is not authorized and may be unlawful.

-- 

--- 
You received this message because you are subscribed to the Google Groups 
"memcached" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to memcached+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Reviewing a script to demo memcache get-set race conditions problems if it is used for distributed locking.

[Nishant Varma]

I am reading 
https://github.com/memcached/memcached/wiki/ProgrammingTricks#ghetto-central-locking,
 
it seems to deal with a slightly different lock scenario of getting some 
expensive item from Database to avoid "Stampeding"

In my case its slightly different lock that I need. I show regular files 
from a folder in a web application to many users. So, to "lock" a file 
using memcache isn't this simple API sufficient or I still need that 
pattern :-)?

def access(filename):
 if memcache.add(filename, timestamp):
return "Access Granted. Lock Obtained" # Normally this results in 
checking HTML checkbox against the filename so User can do actions with 
that/
 else:
return "Access Denied" # Normally this leads to an alert saying 
that someone else is working on this.

Isn't this simple API using add good enough in my case? I am sorry if I am 
repeating this, but I could not really relate the "fetching expensive item 
from Database" to my scenario which is why I even wrote a simple script to 
test the validity of the claim etc.

Can you please let me know?


On Saturday, June 4, 2016 at 6:42:35 PM UTC+5:30, Nishant Varma wrote:
>
> Excellent I rely on you. I guess this is the reason you say I am 
> over-engineering this problem. Makes sense :-) I will again check the link 
> you gave me. I will go through the documentation this weekend.
>
> On Saturday, June 4, 2016 at 1:33:04 PM UTC+5:30, Dormando wrote:
>>
>> Hey, 
>>
>> You really don't need to test this: I'm telling you flatly, as an author 
>> of this software and all of the documentation for it, that you should 
>> absolutely not rely on that pattern. I'm trying to save you some time. 
>>
>> The pattern that is slightly better is written explicitly in pseudocode 
>> in 
>> the link I gave you several times in the issue. Please use it. 
>>
>> Thanks, 
>> -Dormando 
>>
>> On Fri, 3 Jun 2016, Nishant Varma wrote: 
>>
>> > Can anyone help me peer review this script 
>> https://gist.github.com/varmanishant/0129286d41038cc21471652a6460a5ff 
>> that demonstrate potential problems with get set if it is used 
>> > to implement distributed locking. I was suggested to modify from get 
>> set to add in this thread 
>> https://github.com/memcached/memcached/issues/163. However I wanted a 
>> small 
>> > simulation to demonstrate this. 
>> > 
>> > -- 
>> > 
>> > --- 
>> > You received this message because you are subscribed to the Google 
>> Groups "memcached" group. 
>> > To unsubscribe from this group and stop receiving emails from it, send 
>> an email to memcached+...@googlegroups.com. 
>> > For more options, visit https://groups.google.com/d/optout. 
>> > 
>> > 
>>
>
-- 
This e-mail message (including any attachments) may contain information 
that is confidential, protected by the attorney-client or other applicable 
privileges, or otherwise comprising non-public information. This message is 
intended to be conveyed only to the designated recipient(s). If you have 
any reason to believe you are not an intended recipient of this message, 
please notify the sender by replying to this message and then deleting it 
from your system. Any use, dissemination, distribution, or reproduction of 
this message by unintended recipients is not authorized and may be unlawful.

-- 

--- 
You received this message because you are subscribed to the Google Groups 
"memcached" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to memcached+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Reviewing a script to demo memcache get-set race conditions problems if it is used for distributed locking.

[Nishant Varma]

Excellent I rely on you. I guess this is the reason you say I am 
over-engineering this problem. Makes sense :-) I will again check the link 
you gave me. I will go through the documentation this weekend.

On Saturday, June 4, 2016 at 1:33:04 PM UTC+5:30, Dormando wrote:
>
> Hey, 
>
> You really don't need to test this: I'm telling you flatly, as an author 
> of this software and all of the documentation for it, that you should 
> absolutely not rely on that pattern. I'm trying to save you some time. 
>
> The pattern that is slightly better is written explicitly in pseudocode in 
> the link I gave you several times in the issue. Please use it. 
>
> Thanks, 
> -Dormando 
>
> On Fri, 3 Jun 2016, Nishant Varma wrote: 
>
> > Can anyone help me peer review this script 
> https://gist.github.com/varmanishant/0129286d41038cc21471652a6460a5ff 
> that demonstrate potential problems with get set if it is used 
> > to implement distributed locking. I was suggested to modify from get set 
> to add in this thread https://github.com/memcached/memcached/issues/163. 
> However I wanted a small 
> > simulation to demonstrate this. 
> > 
> > -- 
> > 
> > --- 
> > You received this message because you are subscribed to the Google 
> Groups "memcached" group. 
> > To unsubscribe from this group and stop receiving emails from it, send 
> an email to memcached+...@googlegroups.com . 
> > For more options, visit https://groups.google.com/d/optout. 
> > 
> > 
>

-- 
This e-mail message (including any attachments) may contain information 
that is confidential, protected by the attorney-client or other applicable 
privileges, or otherwise comprising non-public information. This message is 
intended to be conveyed only to the designated recipient(s). If you have 
any reason to believe you are not an intended recipient of this message, 
please notify the sender by replying to this message and then deleting it 
from your system. Any use, dissemination, distribution, or reproduction of 
this message by unintended recipients is not authorized and may be unlawful.

-- 

--- 
You received this message because you are subscribed to the Google Groups 
"memcached" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to memcached+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Reviewing a script to demo memcache get-set race conditions problems if it is used for distributed locking.

[dormando]

Hey,

You really don't need to test this: I'm telling you flatly, as an author
of this software and all of the documentation for it, that you should
absolutely not rely on that pattern. I'm trying to save you some time.

The pattern that is slightly better is written explicitly in pseudocode in
the link I gave you several times in the issue. Please use it.

Thanks,
-Dormando

On Fri, 3 Jun 2016, Nishant Varma wrote:

> Can anyone help me peer review this script 
> https://gist.github.com/varmanishant/0129286d41038cc21471652a6460a5ff that 
> demonstrate potential problems with get set if it is used
> to implement distributed locking. I was suggested to modify from get set to 
> add in this thread https://github.com/memcached/memcached/issues/163. However 
> I wanted a small
> simulation to demonstrate this.
>
> --
>
> ---
> You received this message because you are subscribed to the Google Groups 
> "memcached" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to memcached+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>
>

-- 

--- 
You received this message because you are subscribed to the Google Groups 
"memcached" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to memcached+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.