Re: [web2py] Re: recognizing web2py restart

[David McKeone]

Warning: I'm still new to python and web2py so this may be wacky, but...

Could you make a cached method with side-effects and an infinite expiry? 
 Since RAM would always be cleared on a restart it would theoretically only 
ever be run on cold start.

first_time = False
def first_start_check():
global first_time; first_time = True
return True

cache.ram('startup', lambda: first_start_check(), time_expire=None)
if first_time == True:
print "It's the first time!"


On Thursday, May 31, 2012 5:28:21 AM UTC+1, Jonathan Lundell wrote:
>
> On May 30, 2012, at 7:13 PM, Massimo Di Pierro wrote:
>
> you can make a cron job in admin @reboot
>
>
> It'd have to be soft cron, yes? And it wouldn't run until after my app 
> services its first request after restart, if I remember my soft cron logic 
> aright.
>
>
> On Wednesday, 30 May 2012 21:10:13 UTC-5, Jonathan Lundell wrote:
>>
>> I've got an application that uses memcached. I'd like to recognize when 
>> web2py gets restarted (mod_wsgi, fwiw) so I can flush my cache. No doubt I 
>> can figure something out, but I'm sure I must be missing something obvious. 
>> (My motivation: in my development environment, I sometimes blow away my 
>> database when installing and starting a new copy of the app, and things get 
>> confused when the cache is still holding data from the earlier run.)
>
>
>
>

[web2py] Re: More DB issues

[David McKeone]

Bruce,

This looks like a similar error to what you received: 
StackOverflow

More generally, deadlock happens because two sessions are trying to update 
the same table but holding locks to each other's records.

Something like this:

*SESSION 1*
BEGIN;
UPDATE posts SET comment = 'Some Comment' WHERE id = 1;  -- Session 1 now 
holds a lock on post #1

*SESSION 2*
BEGIN
UPDATE posts SET comment = 'Another Comment' WHERE id = 2; -- Session 2 now 
holds a lock on post #2

*SESSION 1*
SELECT * FROM posts WHERE id = 2;  -- Session 1 now waits for Session 2 to 
finish it's transaction and release the lock

*SESSION 2*
SELECT * FROM posts WHERE id = 1;  -- Session 2 now waits for Session 1 to 
finish it's transaction and release the lock

***DEADLOCK***

Neither can complete their transactions so PostgreSQL detects this 
situation and rolls back both transactions.

Obviously that's a simple case... usually these kinds of things take you 
into cascading events with triggers, constraints or some other kind of 
order of operations.



On Friday, May 25, 2012 7:08:35 PM UTC+1, Bruce Wade wrote:
>
>  deadlock detected 
> DETAIL: Process 20969 waits for ShareLock on tuple (10,126) of relation 
> 16935 of database 16386; blocked by process 20615. Process 20615 waits for 
> ShareLock on transaction 19960600; blocked by process 20124. Process 20124 
> waits for ExclusiveLock on tuple (10,126) of relation 16935 of database 
> 16386; blocked by process 20969. HINT: See server log for query details. 
> CONTEXT: SQL statement "SELECT 1 FROM ONLY "public"."ads" x WHERE "id" 
> OPERATOR(pg_catalog.=) $1 FOR SHARE OF x" Any suggestions? 
>
> -- 
> -- 
> Regards,
> Bruce Wade
> http://ca.linkedin.com/in/brucelwade
> http://www.wadecybertech.com
> http://www.fittraineronline.com - Fitness Personal Trainers Online
> http://www.warplydesigned.com
>
>

Re: [web2py] Re: Questions for an application with a large number of features and a large database.

[David McKeone]

*Cliff* 

Thanks for the suggestion, but unfortuantely I need it to work on Windows 
as well, so I'll probably look for a code-organization based solution.

*Bruce*

I would certainly be interested to see what you've done.  I've been 
distilling Bruno's model-less 
demo<http://movu.ca/demo/article/show/11/model-less-apps-using-data-models-and-modules-in-web2py>down
 to the essential components that I need, and I came up with some that 
works like this:

*Model:*
app = MyApp()  # This is based on Bruno's concept of an app that contains 
all the database connection settings as well as member variables for db, 
mail, and crud

*Modules:*
- Base data-model module based on Bruno's basemodel.py that acts as a 
common data-model interface 
- All data models inherited from the based data-model.  Each defines its 
own fields, validators, etc...

*Controller:*
Now in the first line of each of my controllers I just need to declare the 
database variable.  It's not terribly pretty that I have to do that I 
suppose, but, since I've opted for declaring which models I need and which 
I don't, I'm going to have declare them *somewhere*. I guess it's sort of 
like an inverted decorator (below the definition instead of above).

from datamodels.user import UserModel
from datamodels.post import PostModel
from datamodels.comment import CommentModel

def user():
db = app.db([UserModel, PostModel, CommentModel])  # Connect to the 
database and define the models specified
# Do whatever processing is required for this controller

*Discussion:*
*
*
This solution could be nice for a couple of reasons:
1) MyApp holds the instance of the database so I it can be smart about not 
defining tables that have already been defined (because programmers make 
mistakes)
2) I could conditionally declare models based on request conditions and 
ramp up which tables are declared as I need them.  (Maybe that'll help for 
caching with logged-in users -- don't know yet)
3) Models are separate.  Combined with gluino it means I could possibly 
reuse the code to make a desktop app with PySide, or as the back-end for an 
iOS or Android app.

This solution isn't so nice because:
1) It's fairly verbose.

*Future:*
*
*
If I like this solution I could make it a little more clear with a closure 
that has a better name.

*Model: *
app = MyApp()
requires = app.requires()

*Controller:*
from datamodels.user import UserModel
from datamodels.post import PostModel
from datamodels.comment import CommentModel

def user():
requires([UserModel, PostModel, CommentModel])


Anyway... that's what I've done so far.  Can't wait to see yours!


On Sunday, May 27, 2012 5:52:21 PM UTC+1, Bruce Wade wrote:
>
> Yes you can soft link however when you get to the point of multiple 
> servers and upgrading code on each it is a pain.
>
> The site I`m working in is very large. I am now rewriting the code with a 
> new technique in modules and using conditional if statements in side each 
> module so you have 100% control of what loads when and where. 
>
> I should have the new code live soon and will follow up if there was a 
> huge improvement or not. Locally I have seen a huge improvement on one page 
> the loading time went from 0.128 to 0.045 seconds and a functioin call drop 
> from 85845 to 31380 calls and this example is from a page with a single 
> query to a single record in the db. You can imagine the improvement on a 
> complicated page.
> On May 27, 2012 8:56 AM, "Cliff Kachinske"  wrote:
>
>> Yes, you can soft link individual files within the file system.  You can 
>> even rename the target files so the tables load in the right order.
>>
>> It will work on Linux and OS-X and should work on all unix flavors.
>>
>> On Friday, May 25, 2012 3:25:37 PM UTC-4, David McKeone wrote:
>>>
>>> Cliff,
>>>
>>> Certainly an interesting idea.  I'm assuming that you mean soft-linking 
>>> the file system? Would that work on Windows (I have to be able to deploy to 
>>> Windows and Mac)?
>>>
>>> -David
>>>
>>> On Friday, May 25, 2012 8:05:06 PM UTC+1, Cliff Kachinske wrote:
>>>>
>>>> Maybe you could soft link the model files.
>>>>
>>>> For controller foo you would have a file models/foo/foo.py
>>>>
>>>> If controller bar needs needs data from table foo, you would create a 
>>>> soft link in you models/bar directory to models/foo/foo.py.
>>>>
>>>> Note if you link it in as foo.py, it will run after bar.py, so you 
>>>> would want to name the link according to the necessary sequence.
>>>>
>>>> Don't know what this would do for migrations on the productio

[web2py] Re: Questions for an application with a large number of features and a large database.

[David McKeone]

*Massimo and Nico:*
Thanks for looking into those things, can't wait!  

*RAM Cache and DAL?*
I've been looking into conditional models and attempting to combine them 
with the module based system just to see how far I can take it and I've run 
into a question:  

Is there any reason I shouldn't use cache.ram for a DAL instance?  I can't 
use the automatic migration tools since our data-structure wouldn't allow 
for that kind of thing (running a single column update against some of the 
bigger tables can take 30 minutes+ and we want that in a controlled 
environment, probably outside of web2py).  So, with migration out of the 
picture could I do this in the models to avoid recurring re-definition of 
tables?  

def load_models():
db = DAL('postgres://localhost:5432/Demo')
db.define_table('table', Field('field')
db.define_table('table2', Field('field')
#etc...
return db

db = cache.ram('datamodels', lambda: load_models(), time_expire=None)

My real goal is to just get the datamodel remembered between requests 
(since it'd be redundant to load it every time).  I suppose it's really 
just a process specific singleton, but it does make some difference.  Here 
are some non-scientific benchmarks I performed on my data model:

All tables defined in request: ~420ms  
All tables defined in request w/ cache hit: ~90ms

All tables defined in request (compiled app): ~350ms
All tables defined in request w/ cache hit (compiled app): ~25ms

Obviously the first request off of a cold start would be fairly slow, but 
all subsequent requests would benefit greatly.  By using caching with the 
DAL class am I potentially hurting myself in some way?


On Saturday, May 26, 2012 11:13:17 AM UTC+1, Nico de Groot wrote:
>
> Hi David,
> Got Jenkins running on mac and windows with unittests, will send you 
> details later. 
> Nico de Groot

[web2py] Re: Questions for an application with a large number of features and a large database.

[David McKeone]

Cliff,

Certainly an interesting idea.  I'm assuming that you mean soft-linking the 
file system? Would that work on Windows (I have to be able to deploy to 
Windows and Mac)?

-David

On Friday, May 25, 2012 8:05:06 PM UTC+1, Cliff Kachinske wrote:
>
> Maybe you could soft link the model files.
>
> For controller foo you would have a file models/foo/foo.py
>
> If controller bar needs needs data from table foo, you would create a soft 
> link in you models/bar directory to models/foo/foo.py.
>
> Note if you link it in as foo.py, it will run after bar.py, so you would 
> want to name the link according to the necessary sequence.
>
> Don't know what this would do for migrations on the production box, though.
>
> On Friday, May 25, 2012 11:49:51 AM UTC-4, David McKeone wrote:
>>
>> Hi Massimo,
>>
>> "You probably do not need 100 models defined for each request." and "Make 
>> sure you turn migrations off and bytecode compile your apps."
>>
>> No, I certainly don't need all 100 at all times.  That was really just a 
>> test to see where the boundaries were going to be.  It likely wasn't the 
>> optimal configuration (migrations were off, wasn't byte-compiled), but it 
>> did highlight that as the app grows that's an area I have to watch for and 
>> one that will affect the user experience.  Once I saw that a boundary 
>> existed I found Bruno's model-less design and that brought things back to 
>> great performance levels.  So I think that design will fit my needs 
>> performance wise.
>>
>> I'll investigate the conditional model system, but my understanding of 
>> that was that you would be restricted to specific controllers.  As in, I 
>> can't use a single table (model) across multiple controllers. Would that be 
>> true?
>>
>> -David
>>
>
On Friday, May 25, 2012 8:05:06 PM UTC+1, Cliff Kachinske wrote:
>
> Maybe you could soft link the model files.
>
> For controller foo you would have a file models/foo/foo.py
>
> If controller bar needs needs data from table foo, you would create a soft 
> link in you models/bar directory to models/foo/foo.py.
>
> Note if you link it in as foo.py, it will run after bar.py, so you would 
> want to name the link according to the necessary sequence.
>
> Don't know what this would do for migrations on the production box, though.
>
> On Friday, May 25, 2012 11:49:51 AM UTC-4, David McKeone wrote:
>>
>> Hi Massimo,
>>
>> "You probably do not need 100 models defined for each request." and "Make 
>> sure you turn migrations off and bytecode compile your apps."
>>
>> No, I certainly don't need all 100 at all times.  That was really just a 
>> test to see where the boundaries were going to be.  It likely wasn't the 
>> optimal configuration (migrations were off, wasn't byte-compiled), but it 
>> did highlight that as the app grows that's an area I have to watch for and 
>> one that will affect the user experience.  Once I saw that a boundary 
>> existed I found Bruno's model-less design and that brought things back to 
>> great performance levels.  So I think that design will fit my needs 
>> performance wise.
>>
>> I'll investigate the conditional model system, but my understanding of 
>> that was that you would be restricted to specific controllers.  As in, I 
>> can't use a single table (model) across multiple controllers. Would that be 
>> true?
>>
>> -David
>>
>

[web2py] Re: Questions for an application with a large number of features and a large database.

[David McKeone]

ok, so I took a look at the conditional models and I like it because it 
does seem to fit more correctly into the web2py setup.

Here is what I discovered (for those reviewing this later):
- models/[controller]/[function] is the folder structure used to determine 
which model is executed
- models/[insert_model_here].py -for- all requests
- models/user/[insert_model_here].py -for- requests to controllers/user.py
- models/user/view/[insert_model_here].py -for- requests to 
controllers/user.py -> def view():

Massimo: I wasn't able to find the patch from Mariano, do you have a link 
to that (or a discussion of that)?  I did see some of his other posts and 
it looked like he was fighting the same battle I will be, so perhaps his 
changes would be beneficial.  I certainly have a use case I can try them on 
for you.

-David


On Friday, May 25, 2012 6:46:41 PM UTC+1, Massimo Di Pierro wrote:
>
> Currently you can have models/ subfolders for controllers and for 
> functions in controllers. If two functions need the same model file, it 
> should go in the same models/controller/ subfolder or at the top level 
> (visible to all controllers).
>
> There is a proposal on the table (with a patch by Mariano) to allow 
> granular conditions models in which you can decide which model file is 
> needed by which controller or function.
>
> Is had been table because of lack of interest. I am happy to revive it if 
> there is push.
>
> massimo
>
>
>
> On Friday, 25 May 2012 10:49:51 UTC-5, David McKeone wrote:
>>
>> Hi Massimo,
>>
>> "You probably do not need 100 models defined for each request." and "Make 
>> sure you turn migrations off and bytecode compile your apps."
>>
>> No, I certainly don't need all 100 at all times.  That was really just a 
>> test to see where the boundaries were going to be.  It likely wasn't the 
>> optimal configuration (migrations were off, wasn't byte-compiled), but it 
>> did highlight that as the app grows that's an area I have to watch for and 
>> one that will affect the user experience.  Once I saw that a boundary 
>> existed I found Bruno's model-less design and that brought things back to 
>> great performance levels.  So I think that design will fit my needs 
>> performance wise.
>>
>> I'll investigate the conditional model system, but my understanding of 
>> that was that you would be restricted to specific controllers.  As in, I 
>> can't use a single table (model) across multiple controllers. Would that be 
>> true?
>>
>> -David
>>
>

[web2py] Re: Questions for an application with a large number of features and a large database.

[David McKeone]

Hi Massimo,

"You probably do not need 100 models defined for each request." and "Make 
sure you turn migrations off and bytecode compile your apps."

No, I certainly don't need all 100 at all times.  That was really just a 
test to see where the boundaries were going to be.  It likely wasn't the 
optimal configuration (migrations were off, wasn't byte-compiled), but it 
did highlight that as the app grows that's an area I have to watch for and 
one that will affect the user experience.  Once I saw that a boundary 
existed I found Bruno's model-less design and that brought things back to 
great performance levels.  So I think that design will fit my needs 
performance wise.

I'll investigate the conditional model system, but my understanding of that 
was that you would be restricted to specific controllers.  As in, I can't 
use a single table (model) across multiple controllers. Would that be true?

-David

[web2py] Questions for an application with a large number of features and a large database.

[David McKeone]

I've been reviewing Web2py over the last month so that I could evaluate it 
for use with a large-ish commercial application.  This has brought up a few 
questions, so I figured I'd see if anyone has any experience in any of the 
areas I've run into trouble.  I come from a C++ background so I'm fairly 
new to Python (and web2py by extension), but I've been a professional 
programmer for ~10 years, so I have a good general understanding how to 
handle the scale of the application, now I'm just after how to do those 
things in web2py and it's associated Python toolchain.

Hopefully this post will also be a summary of some of the knowledge I've 
found through this group.  

Here are the tools I'm going to be using (Fairly common I would think...):
- PyCharm  2.5 IDE
- Nose  for tests
- Jenkins  for continuous integration 
(CI)
- Git  for version control

I won't go into too much detail about the actual application, but to give a 
little bit of background about the scope of the problem: it is a deployed 
app and the database has just under 100 tables and can reach sizes of 
~40GB.  In this theoretical use of web2py there would be 1 copy of web2py 
and 1 copy of my application for each database.  The project would also 
have multiple developers coding against the same code base with a fairly 
quick release cycle (hence, code quality, testing, CI and Git are all 
paramount)

So, after that long intro, here are the pain points I've run into:

   1. Getting nose tests running was a bit of a challenge, and it seemed to 
   expose an area that may be missing from web2py for large-ish applications
  - Referencing Nico de Groot's excellent web2py 
sliceI
 was able to get testing going with UnitTest.  This gave me a good idea of 
  how automated testing could be done with web2py, but I really wanted to 
use 
  nose tests.
  - That led me to the Sahana Eden code 
base (the only large web2py application that I'm aware of that looks 
  comparable to the application I'd like to build).  Referencing their 
  
modules/test_utils/run.py,
 
  tests/nose.py and 
  
modules/test_utils/Web2pyNosePlugin.pyfiles
 I was able to get nose tests running (and they put out coverage and 
  xunit for Jenkins too!).  Cool.
  - Those first two bullets took some time and it really made me 
  wonder... I see that web2py itself uses nose tests (seemingly an 
admission 
  that at least one web2py developer sees value in nose) and yet us lowly 
  web2py application developers are relegated to (what I've seen to be) an 
  official stance that we should be using doctests for all our tests with 
  web2py.  Why is there not better support for automated unit testing?  
Have 
  I completely missed some documentation or recommendation somewhere?
 - Why can I do this: python web2py.py --run_system_tests
 - But I can't do this to test my application?: python web2py.py 
 --run_tests -M -S myapp   
 - or this for use with 
TDD?: 
 python web2py.py --run_tests -M -S myapp -R 
 applications/myapp/tests/controllers/user.py  (Modelled after how 
Sahana 
 Eden had their test directory structure setup)
  - This isn't really the fault of web2py, but even with my solution I 
  am still unable to use PyCharm's built-in nose testing support, since 
they 
  have their own test harness and, as far as I know, that isn't compatible 
  with web2py.py (Go vote for web2py 
supportif you haven't please!) 
   2. All models are loaded for each request
  - Loading all ~100 table models took ~3 seconds per request.
  - After reviewing this group I believe I understand why this is, but 
  I do lament the fact that the model terminology will be inaccurate.  I 
will 
  have to load all my "real" models into seperate 
modules
 (Thanks 
  for your great post about doing it though Bruno!) and the end result is 
  that in any app of significant size, models will actually mean "request 
  startup/initialization" and ../modules/datamodels will be my "real" 
models. 
   
   3. No first-class PyCharm support
  - This IDE is really quite great (love the debugger, coverage, git 
  and lint checks), did I mention that you should go vot