[web2py] Questions for an application with a large number of features and a large database.

[Nico de Groot]

Hi David,
Got Jenkins running on mac and windows with unittests, will send you details 
later. 
Nico de Groot

[web2py] Questions for an application with a large number of features and a large database.

[David McKeone]

I've been reviewing Web2py over the last month so that I could evaluate it 
for use with a large-ish commercial application.  This has brought up a few 
questions, so I figured I'd see if anyone has any experience in any of the 
areas I've run into trouble.  I come from a C++ background so I'm fairly 
new to Python (and web2py by extension), but I've been a professional 
programmer for ~10 years, so I have a good general understanding how to 
handle the scale of the application, now I'm just after how to do those 
things in web2py and it's associated Python toolchain.

Hopefully this post will also be a summary of some of the knowledge I've 
found through this group.  

Here are the tools I'm going to be using (Fairly common I would think...):
- PyCharm  2.5 IDE
- Nose  for tests
- Jenkins  for continuous integration 
(CI)
- Git  for version control

I won't go into too much detail about the actual application, but to give a 
little bit of background about the scope of the problem: it is a deployed 
app and the database has just under 100 tables and can reach sizes of 
~40GB.  In this theoretical use of web2py there would be 1 copy of web2py 
and 1 copy of my application for each database.  The project would also 
have multiple developers coding against the same code base with a fairly 
quick release cycle (hence, code quality, testing, CI and Git are all 
paramount)

So, after that long intro, here are the pain points I've run into:

   1. Getting nose tests running was a bit of a challenge, and it seemed to 
   expose an area that may be missing from web2py for large-ish applications
  - Referencing Nico de Groot's excellent web2py 
sliceI
 was able to get testing going with UnitTest.  This gave me a good idea of 
  how automated testing could be done with web2py, but I really wanted to 
use 
  nose tests.
  - That led me to the Sahana Eden code 
base (the only large web2py application that I'm aware of that looks 
  comparable to the application I'd like to build).  Referencing their 
  
modules/test_utils/run.py,
 
  tests/nose.py and 
  
modules/test_utils/Web2pyNosePlugin.pyfiles
 I was able to get nose tests running (and they put out coverage and 
  xunit for Jenkins too!).  Cool.
  - Those first two bullets took some time and it really made me 
  wonder... I see that web2py itself uses nose tests (seemingly an 
admission 
  that at least one web2py developer sees value in nose) and yet us lowly 
  web2py application developers are relegated to (what I've seen to be) an 
  official stance that we should be using doctests for all our tests with 
  web2py.  Why is there not better support for automated unit testing?  
Have 
  I completely missed some documentation or recommendation somewhere?
 - Why can I do this: python web2py.py --run_system_tests
 - But I can't do this to test my application?: python web2py.py 
 --run_tests -M -S myapp   
 - or this for use with 
TDD?: 
 python web2py.py --run_tests -M -S myapp -R 
 applications/myapp/tests/controllers/user.py  (Modelled after how 
Sahana 
 Eden had their test directory structure setup)
  - This isn't really the fault of web2py, but even with my solution I 
  am still unable to use PyCharm's built-in nose testing support, since 
they 
  have their own test harness and, as far as I know, that isn't compatible 
  with web2py.py (Go vote for web2py 
supportif you haven't please!) 
   2. All models are loaded for each request
  - Loading all ~100 table models took ~3 seconds per request.
  - After reviewing this group I believe I understand why this is, but 
  I do lament the fact that the model terminology will be inaccurate.  I 
will 
  have to load all my "real" models into seperate 
modules
 (Thanks 
  for your great post about doing it though Bruno!) and the end result is 
  that in any app of significant size, models will actually mean "request 
  startup/initialization" and ../modules/datamodels will be my "real" 
models. 
   
   3. No first-class PyCharm support
  - This IDE is really quite great (love the debugger, coverage, git 
  and lint checks), did I mention that you should go vot