[web2py] Re: scheduler processes cooperation issue
Chapter 13. Concurrency Control https://www.postgresql.org/docs/9.1/static/mvcc.html -- Resources: - http://web2py.com - http://web2py.com/book (Documentation) - http://github.com/web2py/web2py (Source code) - https://code.google.com/p/web2py/issues/list (Report Issues) --- You received this message because you are subscribed to the Google Groups "web2py-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to web2py+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[web2py] Re: scheduler processes cooperation issue
sorry Dave here's the correct link : https://groups.google.com/d/msg/web2py/Z581Pk3RCYI/qdiy0INcm2oJ -- Resources: - http://web2py.com - http://web2py.com/book (Documentation) - http://github.com/web2py/web2py (Source code) - https://code.google.com/p/web2py/issues/list (Report Issues) --- You received this message because you are subscribed to the Google Groups "web2py-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to web2py+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[web2py] Re: scheduler processes cooperation issue
Dave take a look at this : https://groups.google.com/forum/?fromgroups#!topicsearchin/web2py/race$20condition;context-place=forum/web2py/web2py/Z581Pk3RCYI/web2py/Z581Pk3RCYI/web2py/Z581Pk3RCYI I wasn't aware of the "race condition issue".( concurrent Parallel Threads "collision") . I did some researches, there are some postgresql commands, config files to address this. I'll see what I can do with it. At this time I don't think the try: except fix will suffice...the book is very silent about this condition which can obviously lead to serious inconsistency. well perhaps webapps need not be more consistent than the rest ? -- Resources: - http://web2py.com - http://web2py.com/book (Documentation) - http://github.com/web2py/web2py (Source code) - https://code.google.com/p/web2py/issues/list (Report Issues) --- You received this message because you are subscribed to the Google Groups "web2py-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to web2py+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[web2py] Re: scheduler processes cooperation issue
ha: yep, the logic is totally flawed. You can't be sure your task has been correctly processed just by queueing it. The task DOESN'T obviously run in the same transaction the webpage displaying the form is. On Thursday, September 29, 2016 at 5:54:20 PM UTC+2, Pierre wrote: > > very confusing situation > problem has been addressed long time ago: > > > https://groups.google.com/forum/?fromgroups#!topicsearchin/web2py/race$20condition;context-place=forum/web2py/web2py/Z581Pk3RCYI/web2py/Z581Pk3RCYI > > what I observe is that transaction logic doesn't work as expected in the > following circonstance : > > *controller:* > > def func(): > form = SQLFORM(db.taba,...) > if form.process().accepted: > scheduler.queue_task(tska,..), > redirect(URL(...) > elif form.errors: >. >. > else: >return dict(form=form) > > if task tska raises an exception (violate unique key constraint) there's > no roll-back and still I get a new record in db.taba > > Is there a generic approach to deal with such overhead problems ? i am > starting to worry a lot...where's the tunnel exit ? > -- Resources: - http://web2py.com - http://web2py.com/book (Documentation) - http://github.com/web2py/web2py (Source code) - https://code.google.com/p/web2py/issues/list (Report Issues) --- You received this message because you are subscribed to the Google Groups "web2py-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to web2py+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[web2py] Re: scheduler processes cooperation issue
On Thursday, September 29, 2016 at 8:54:20 AM UTC-7, Pierre wrote: > > very confusing situation > problem has been addressed long time ago: > > > https://groups.google.com/forum/?fromgroups#!topicsearchin/web2py/race$20condition;context-place=forum/web2py/web2py/Z581Pk3RCYI/web2py/Z581Pk3RCYI > Are you trying to point at a specific thread? What I see is a search page. > > > what I observe is that transaction logic doesn't work as expected in the > following circonstance : > > *controller:* > > def func(): > form = SQLFORM(db.taba,...) > if form.process().accepted: > scheduler.queue_task(tska,..), > redirect(URL(...) > elif form.errors: >. >. > else: >return dict(form=form) > > if task tska raises an exception (violate unique key constraint) there's > no roll-back and still I get a new record in db.taba > > Is there a generic approach to deal with such overhead problems ? i am > starting to worry a lot...where's the tunnel exit ? > Why would tska be violating unique key constraint? Isn't the row created by the form.process()? If enforcement of uniqueness is at the DB level (see the bit you quoted in the previous post), then THAT should fail on the duplicate, and tska should never get queued. /dps -- Resources: - http://web2py.com - http://web2py.com/book (Documentation) - http://github.com/web2py/web2py (Source code) - https://code.google.com/p/web2py/issues/list (Report Issues) --- You received this message because you are subscribed to the Google Groups "web2py-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to web2py+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[web2py] Re: scheduler processes cooperation issue
very confusing situation problem has been addressed long time ago: https://groups.google.com/forum/?fromgroups#!topicsearchin/web2py/race$20condition;context-place=forum/web2py/web2py/Z581Pk3RCYI/web2py/Z581Pk3RCYI what I observe is that transaction logic doesn't work as expected in the following circonstance : *controller:* def func(): form = SQLFORM(db.taba,...) if form.process().accepted: scheduler.queue_task(tska,..), redirect(URL(...) elif form.errors: . . else: return dict(form=form) if task tska raises an exception (violate unique key constraint) there's no roll-back and still I get a new record in db.taba Is there a generic approach to deal with such overhead problems ? i am starting to worry a lot...where's the tunnel exit ? -- Resources: - http://web2py.com - http://web2py.com/book (Documentation) - http://github.com/web2py/web2py (Source code) - https://code.google.com/p/web2py/issues/list (Report Issues) --- You received this message because you are subscribed to the Google Groups "web2py-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to web2py+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[web2py] Re: scheduler processes cooperation issue
I am starting the workers/sched processes with this : python web2py.py -a 'password' -K appname,appname and the tasks are queued in a controller function then . Tasks use module functions As far as I understand ret = db.validate_and_insert(fa='something',fb='somethingelse') is not supposed to raise an exception otherwise what for to validate ?. If I run this command from a controller and there's already a key: fa='something' It fails silently as it should (no exception is raised) As said before If I run a single sched worker in the same conditions no exception occurs so if problem isn't double-workers related where does it come from ?. If I run the exact same actions from a controller (no scheduler involved) I never get an exception I'd really like to clarify this. The same instruction changing its behavior with circomstances is very confusing :( -- Resources: - http://web2py.com - http://web2py.com/book (Documentation) - http://github.com/web2py/web2py (Source code) - https://code.google.com/p/web2py/issues/list (Report Issues) --- You received this message because you are subscribed to the Google Groups "web2py-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to web2py+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[web2py] Re: scheduler processes cooperation issue
the POV of this question is wrong. Scheduler processes do what you ask them
to do, without nothing else added.
The same thing you observe would happen if you try to execute your tasks
manually. It just seems that what you are asking them to do violates your
model (i.e. either you had already a row with fa=something in the table OR
you asked more than one task to insert fa=something in the table, which
would, at the 2nd run, fail as expected)
On Wednesday, September 28, 2016 at 10:13:10 PM UTC+2, Dave S wrote:
>
>
>
> On Wednesday, September 28, 2016 at 3:58:10 AM UTC-7, Pierre wrote:
>>
>> Hi ,
>>
>>
>> I noticed an unexpected behaviour when running two scheduler processes.
>>
>>
>> The situation is this :
>>
>>
>> A db with a table like :
>>
>>
>> db.define_table('taba',
>> Field('fa', unique=True, requires = IS_NOT_IN_DB(db,'taba.fa')),
>> Field('fb'))
>>
>>
>>
>> a task (same function different args) distributed to the processes which
>> contains a :
>>
>>
>> ret = db.taba.validate_and_insert(fa='something', fb='somethingelse')
>>
>>
>> this generates an exception like so :
>>
>>
>> File "/opt/web2py/gluon/packages/dal/pydal/objects.py", line 737, in
>> validate_and_insert
>> response.id = self.insert(**new_fields)
>> File "/opt/web2py/gluon/packages/dal/pydal/objects.py", line 712, in
>> insert
>> ret = self._db._adapter.insert(self, self._listify(fields))
>> File "/opt/web2py/gluon/packages/dal/pydal/adapters/base.py", line 739,
>> in insert
>>
>> raise e
>> IntegrityError: duplicate key value violates unique constraint
>> "taba_fa_key"
>>
>> DETAIL: Key (fa)=(something) already exists.
>>
>>
>>
>> and task fails
>>
>>
>> This doesn't happen when running a single scheduler process so I guess
>> the two scheduler processes collide(….they try to write the same value to
>> the db at the same time……)
>>
>>
>> how do I fix this ? Why can't the two processes cooperate ? Am I limited
>> to running a single sched process ?
>>
>
> What do you mean by 2 sched processes? Are you calling
> schedule=Scheduler(db, migrate=False)
> twice? I don't think you're supposed to do that.
>
> There can be a variable number of workers, and ISTR that the default is 4.
>
> If you aren't instantiating twice, then the next question is how are you
> queuing your tasks? Are you queuing tasks from a controller? From a
> script?
>
> /dps
>
>
>
>
>
--
Resources:
- http://web2py.com
- http://web2py.com/book (Documentation)
- http://github.com/web2py/web2py (Source code)
- https://code.google.com/p/web2py/issues/list (Report Issues)
---
You received this message because you are subscribed to the Google Groups
"web2py-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to web2py+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
[web2py] Re: scheduler processes cooperation issue
On Wednesday, September 28, 2016 at 3:58:10 AM UTC-7, Pierre wrote:
>
> Hi ,
>
>
> I noticed an unexpected behaviour when running two scheduler processes.
>
>
> The situation is this :
>
>
> A db with a table like :
>
>
> db.define_table('taba',
> Field('fa', unique=True, requires = IS_NOT_IN_DB(db,'taba.fa')),
> Field('fb'))
>
>
>
> a task (same function different args) distributed to the processes which
> contains a :
>
>
> ret = db.taba.validate_and_insert(fa='something', fb='somethingelse')
>
>
> this generates an exception like so :
>
>
> File "/opt/web2py/gluon/packages/dal/pydal/objects.py", line 737, in
> validate_and_insert
> response.id = self.insert(**new_fields)
> File "/opt/web2py/gluon/packages/dal/pydal/objects.py", line 712, in
> insert
> ret = self._db._adapter.insert(self, self._listify(fields))
> File "/opt/web2py/gluon/packages/dal/pydal/adapters/base.py", line 739,
> in insert
>
> raise e
> IntegrityError: duplicate key value violates unique constraint
> "taba_fa_key"
>
> DETAIL: Key (fa)=(something) already exists.
>
>
>
> and task fails
>
>
> This doesn't happen when running a single scheduler process so I guess the
> two scheduler processes collide(….they try to write the same value to the
> db at the same time……)
>
>
> how do I fix this ? Why can't the two processes cooperate ? Am I limited
> to running a single sched process ?
>
What do you mean by 2 sched processes? Are you calling
schedule=Scheduler(db, migrate=False)
twice? I don't think you're supposed to do that.
There can be a variable number of workers, and ISTR that the default is 4.
If you aren't instantiating twice, then the next question is how are you
queuing your tasks? Are you queuing tasks from a controller? From a
script?
/dps
--
Resources:
- http://web2py.com
- http://web2py.com/book (Documentation)
- http://github.com/web2py/web2py (Source code)
- https://code.google.com/p/web2py/issues/list (Report Issues)
---
You received this message because you are subscribed to the Google Groups
"web2py-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to web2py+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
