Re: [web2py] Re: nginx/uwsgi problem on Debian

[Richard Vézina]

Check permissions...

Richard

On Mon, Oct 10, 2016 at 1:34 PM, Arve Fahlvik  wrote:

> What was the solution? Seems like I have a similar problem.
>
> Arve
>
> On Thursday, January 21, 2016 at 10:05:57 AM UTC+1, Johann Spies wrote:
>>
>> On 20 January 2016 at 16:35, Richard Vézina 
>> wrote:
>>
>>>
>>> Which user is regular "web" user under debian, www-data?
>>>
>>>
>> Yes.
>>
>> Thanks it is working now.
>>
>> Regards
>> Johann
>>
> --
> Resources:
> - http://web2py.com
> - http://web2py.com/book (Documentation)
> - http://github.com/web2py/web2py (Source code)
> - https://code.google.com/p/web2py/issues/list (Report Issues)
> ---
> You received this message because you are subscribed to the Google Groups
> "web2py-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to web2py+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
Resources:
- http://web2py.com
- http://web2py.com/book (Documentation)
- http://github.com/web2py/web2py (Source code)
- https://code.google.com/p/web2py/issues/list (Report Issues)
--- 
You received this message because you are subscribed to the Google Groups 
"web2py-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to web2py+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [web2py] Re: nginx/uwsgi problem on Debian

[Arve Fahlvik]

What was the solution? Seems like I have a similar problem.

Arve

On Thursday, January 21, 2016 at 10:05:57 AM UTC+1, Johann Spies wrote:
>
> On 20 January 2016 at 16:35, Richard Vézina  > wrote:
>
>>
>> Which user is regular "web" user under debian, www-data?
>>
>>
> Yes.
>  
> Thanks it is working now.
>
> Regards
> Johann 
>

-- 
Resources:
- http://web2py.com
- http://web2py.com/book (Documentation)
- http://github.com/web2py/web2py (Source code)
- https://code.google.com/p/web2py/issues/list (Report Issues)
--- 
You received this message because you are subscribed to the Google Groups 
"web2py-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to web2py+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [web2py] Re: nginx/uwsgi problem on Debian

[Richard Vézina]

:)

Richard

On Thu, Jan 21, 2016 at 4:05 AM, Johann Spies 
wrote:

> On 20 January 2016 at 16:35, Richard Vézina 
> wrote:
>
>>
>> Which user is regular "web" user under debian, www-data?
>>
>>
> Yes.
>
> Thanks it is working now.
>
> Regards
> Johann
>
> --
> Resources:
> - http://web2py.com
> - http://web2py.com/book (Documentation)
> - http://github.com/web2py/web2py (Source code)
> - https://code.google.com/p/web2py/issues/list (Report Issues)
> ---
> You received this message because you are subscribed to the Google Groups
> "web2py-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to web2py+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
Resources:
- http://web2py.com
- http://web2py.com/book (Documentation)
- http://github.com/web2py/web2py (Source code)
- https://code.google.com/p/web2py/issues/list (Report Issues)
--- 
You received this message because you are subscribed to the Google Groups 
"web2py-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to web2py+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [web2py] Re: nginx/uwsgi problem on Debian

[Johann Spies]

On 20 January 2016 at 16:35, Richard Vézina 
wrote:

>
> Which user is regular "web" user under debian, www-data?
>
>
Yes.

Thanks it is working now.

Regards
Johann

-- 
Resources:
- http://web2py.com
- http://web2py.com/book (Documentation)
- http://github.com/web2py/web2py (Source code)
- https://code.google.com/p/web2py/issues/list (Report Issues)
--- 
You received this message because you are subscribed to the Google Groups 
"web2py-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to web2py+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [web2py] Re: nginx/uwsgi problem on Debian

[Richard Vézina]

As Simone says, it seems clear that it come from permission...

Which user is regular "web" user under debian, www-data?

Even if it not the main issue, I would rather start debuging and make work
the 80 port before trying 443...

First link if I google the bind() error :
http://stackoverflow.com/questions/23886018/cant-run-uwsgi-as-root-bind-permission-denied

Good luck.

Richard

On Wed, Jan 20, 2016 at 8:57 AM, Niphlod  wrote:

> apart from being a clear "not-even-close-to-web2py" error, until you can
> get rid of
>
> uwsgi socket 0 bound to UNIX address /var/uwsgi/app/web2py/socket fd 3
> bind(): Permission denied [core/socket.c line 227]
>
> you'll never going to solve the problem.
>
> Make sure you create a valid dir with the valid permission for the user
> running uwsgi to be available. Tail the emperor.log to see if it stays up.
>
> --
> Resources:
> - http://web2py.com
> - http://web2py.com/book (Documentation)
> - http://github.com/web2py/web2py (Source code)
> - https://code.google.com/p/web2py/issues/list (Report Issues)
> ---
> You received this message because you are subscribed to the Google Groups
> "web2py-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to web2py+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
Resources:
- http://web2py.com
- http://web2py.com/book (Documentation)
- http://github.com/web2py/web2py (Source code)
- https://code.google.com/p/web2py/issues/list (Report Issues)
--- 
You received this message because you are subscribed to the Google Groups 
"web2py-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to web2py+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [web2py] Re: Nginx-uwsgi problem.

[Johann Spies]

On 26 August 2012 23:05, apps in tables  wrote:

> Hi,
>
> where can i find tree example(file system) built by web2py ?
>
>
In the Web2py directory tree?

Regards
Johann
-- 
Because experiencing your loyal love is better than life itself,
my lips will praise you.  (Psalm 63:3)

-- 

Re: [web2py] Re: Nginx-uwsgi problem.

[apps in tables]

Hi,

where can i find tree example(file system) built by web2py ?

Regards,

Ashraf

On Monday, May 14, 2012 6:33:07 PM UTC+3, Ross Peoples wrote:
>
> I'm reading through the profile log, not that I'm an expert on profiling 
> or anything, but I do have a few possible tips:
>
>1. Most of your calls are done in ~86 ms. That's pretty respectable. 
>2. Most of your time is spent on database calls, with compiling being 
>the second biggest time consumer. So compiling your code should provide 
> you 
>with a quick speed boost without having to change any code.
>3. The biggest bottleneck is almost always the queries to the 
>database, regardless of using the DAL or not. So spend more time 
> optimizing 
>your queries before optimizing your code.
>4. Your "adviewer/viewads" makes 10 calls to the database. Try to 
>optimize this either by writing fewer queries, creating a view, and/or 
> only 
>selecting fields that you need. Also make sure you get the criteria right 
>so that you (ideally) don't have any extra, unneeded rows.
>5. The "trees/index", "trees/get_binary_tree.json", 
>"trees/team_members", etc. I assume are supposed to be showing a 
>parent-child type of relationship. I've done this with folders in a 
>document management system before and trees are tough. The way I did it 
> was 
>to load only the top-level and one level below in a single database query 
>(so you would know if the folder had subfolders), and then loaded the rest 
>whenever the user "opened" or "expanded" a folder. If your application 
>resembles this type of tree, then maybe try that. Otherwise, try to load 
>two or three levels per query. Even using a sub-select will be faster than 
>repeated queries.
>6. Moving models to modules and only loading the tables you need has 
>already been mentioned, but it's worth adding to the list of tips.
>7. When you absolutely have to load a few thousand rows (or more) in a 
>query (you should avoid this whenever possible), then try using 
>"db.executesql(query)" to manually execute a hand-crafted SQL query. This 
>will always be faster than using the DAL directly.
>8. Another point about executesql: The obvious issue is reduced 
>portability, but if you are only planning on using PostgreSQL, then you 
> can 
> hand-craft a SQL query and profile it against PostgreSQL for maximum 
>performance. Once you've got it giving only the data you want, then you 
> can 
>copy and paste that query into executesql.
>
> I don't know how much, if any, of this applies to your situation, but I'd 
> like to think it's good advice in general.
>

-- 

Re: [web2py] Re: Nginx-uwsgi problem.

[Marco Tulio Cicero de M. Porto]

We've been testing NginX for a while now, but we're still not using it on
production.
We've used a script from this group to install/configure nginx/web2py/uwsgi.
This script created a web2py configuration file on sites-available with
configurations for HTTP and HTTPS.
The one error message we get is error 504.
But this don't happen often. When we have something small, it works fine.
Get something a little bigger and it starts.
In the example Ovidio told you, there's this 1 web app that has around 51
tables and it's getting a little big...
When we try to acess the application for the first time (through browser),
it'll wait for a few seconds and eventually end up with a 504 error.
After that, we need to restart uwsgi to make other apps work again .

I'll take a look on the log and conf files to share.

Cheers!

2012/6/11 pbreit 

> Probably going to need a lot more information. Have you used Nginx before?
> What does nginx.conf look like? Are you getting any error messages? Nginx
> error log? Do you have to kill nginx, uwsgi or web2py to run again?




-- 
[]'s
Marco Tulio

Re: [web2py] Re: Nginx-uwsgi problem.

[pbreit]

Probably going to need a lot more information. Have you used Nginx before? 
What does nginx.conf look like? Are you getting any error messages? Nginx 
error log? Do you have to kill nginx, uwsgi or web2py to run again?

Re: [web2py] Re: Nginx-uwsgi problem.

[Ovidio Marinho]

We have an application running under apache2 with 51 tables in
postgres and runs very well on apache, when put to run with the
postgres nginx a request timeout error appears and locks the
web-server. What might be happening? which the configuration to run
web2py + postgresql + nginx?




       Ovidio Marinho Falcao Neto
                Web Developer
             ovidio...@gmail.com
          ovidiomari...@itjp.net.br
                 ITJP - itjp.net.br
               83   8826 9088 - Oi
               83   9334 0266 - Claro
                        Brasil



2012/5/22 Bruce Wade :
> OK so here is a little bit before and after I still have a lot of work to
> do:
>
> Before:
> 
> /
> 
> Sat May 12 15:39:07 2012    profiler.log.tmp
>
>          85845 function calls (84497 primitive calls) in 0.128 seconds
>
>    Ordered by: internal time
>    List reduced from 743 to 80 due to restriction <80>
>
> After:
> 
> /
> 
> Tue May 22 16:14:27 2012    profile.log.tmp
>
>          52095 function calls (50779 primitive calls) in 0.085 seconds
>
>    Ordered by: internal time
>    List reduced from 727 to 80 due to restriction <80>
>
> On Mon, May 14, 2012 at 12:06 PM, Bruce Wade  wrote:
>>
>> Yes, I have started moving everything from models to custom api modules
>> this weekend.
>>
>> I am going to try to use most of the tips provided by everyone.
>>
>> For the adviewer I will look to see what is calling 10 query's. However I
>> can not cache this query as the returned ads most be random for every user.
>> Also I load 5 ads at first so besides rating the ad there should only be one
>> major query ever 5 * 20 (seconds).
>>
>> I will have a lot of these changes done by Weds and will provide a new
>> profile report.
>>
>> --
>> Thanks
>>
>> On Mon, May 14, 2012 at 11:07 AM, Massimo Di Pierro
>>  wrote:
>>>
>>> If one has more than 10 tables, one should use conditional models and one
>>> should move all state dependent settings into the controllers that need
>>> them,
>>>
>>> Massimo
>>>
>>>
>>> On Monday, 14 May 2012 11:36:00 UTC-5, pbreit wrote:

 Aren't 135 table definitions going to be a problem in a high volume app?
 Aren't many/most of those 85k-100k function calls going to 
 examining/parsing
 all those tabledefs?
>>
>>
>>
>>
>> --
>> --
>> Regards,
>> Bruce Wade
>> http://ca.linkedin.com/in/brucelwade
>> http://www.wadecybertech.com
>> http://www.fittraineronline.com - Fitness Personal Trainers Online
>> http://www.warplydesigned.com
>>
>
>
>
> --
> --
> Regards,
> Bruce Wade
> http://ca.linkedin.com/in/brucelwade
> http://www.wadecybertech.com
> http://www.fittraineronline.com - Fitness Personal Trainers Online
> http://www.warplydesigned.com
>

Re: [web2py] Re: Nginx-uwsgi problem.

[Bruce Wade]

OK so here is a little bit before and after I still have a lot of work to
do:

Before:

/

Sat May 12 15:39:07 2012profiler.log.tmp

 85845 function calls (84497 primitive calls) in 0.128 seconds

   Ordered by: internal time
   List reduced from 743 to 80 due to restriction <80>

After:

/

Tue May 22 16:14:27 2012profile.log.tmp

 52095 function calls (50779 primitive calls) in 0.085 seconds

   Ordered by: internal time
   List reduced from 727 to 80 due to restriction <80>

On Mon, May 14, 2012 at 12:06 PM, Bruce Wade  wrote:

> Yes, I have started moving everything from models to custom api modules
> this weekend.
>
> I am going to try to use most of the tips provided by everyone.
>
> For the adviewer I will look to see what is calling 10 query's. However I
> can not cache this query as the returned ads most be random for every user.
> Also I load 5 ads at first so besides rating the ad there should only be
> one major query ever 5 * 20 (seconds).
>
> I will have a lot of these changes done by Weds and will provide a new
> profile report.
>
> --
> Thanks
>
> On Mon, May 14, 2012 at 11:07 AM, Massimo Di Pierro <
> massimo.dipie...@gmail.com> wrote:
>
>> If one has more than 10 tables, one should use conditional models and one
>> should move all state dependent settings into the controllers that need
>> them,
>>
>> Massimo
>>
>>
>> On Monday, 14 May 2012 11:36:00 UTC-5, pbreit wrote:
>>>
>>> Aren't 135 table definitions going to be a problem in a high volume app?
>>> Aren't many/most of those 85k-100k function calls going to
>>> examining/parsing all those tabledefs?
>>
>>
>
>
> --
> --
> Regards,
> Bruce Wade
> http://ca.linkedin.com/in/brucelwade
> http://www.wadecybertech.com
> http://www.fittraineronline.com - Fitness Personal Trainers Online
> http://www.warplydesigned.com
>
>


-- 
-- 
Regards,
Bruce Wade
http://ca.linkedin.com/in/brucelwade
http://www.wadecybertech.com
http://www.fittraineronline.com - Fitness Personal Trainers Online
http://www.warplydesigned.com

Re: [web2py] Re: Nginx-uwsgi problem.

[Bruce Wade]

Yes, I have started moving everything from models to custom api modules
this weekend.

I am going to try to use most of the tips provided by everyone.

For the adviewer I will look to see what is calling 10 query's. However I
can not cache this query as the returned ads most be random for every user.
Also I load 5 ads at first so besides rating the ad there should only be
one major query ever 5 * 20 (seconds).

I will have a lot of these changes done by Weds and will provide a new
profile report.

--
Thanks

On Mon, May 14, 2012 at 11:07 AM, Massimo Di Pierro <
massimo.dipie...@gmail.com> wrote:

> If one has more than 10 tables, one should use conditional models and one
> should move all state dependent settings into the controllers that need
> them,
>
> Massimo
>
>
> On Monday, 14 May 2012 11:36:00 UTC-5, pbreit wrote:
>>
>> Aren't 135 table definitions going to be a problem in a high volume app?
>> Aren't many/most of those 85k-100k function calls going to
>> examining/parsing all those tabledefs?
>
>


-- 
-- 
Regards,
Bruce Wade
http://ca.linkedin.com/in/brucelwade
http://www.wadecybertech.com
http://www.fittraineronline.com - Fitness Personal Trainers Online
http://www.warplydesigned.com

Re: [web2py] Re: Nginx-uwsgi problem.

[Massimo Di Pierro]

If one has more than 10 tables, one should use conditional models and one 
should move all state dependent settings into the controllers that need 
them,

Massimo

On Monday, 14 May 2012 11:36:00 UTC-5, pbreit wrote:
>
> Aren't 135 table definitions going to be a problem in a high volume app? 
> Aren't many/most of those 85k-100k function calls going to 
> examining/parsing all those tabledefs?

Re: [web2py] Re: Nginx-uwsgi problem.

[pbreit]

Aren't 135 table definitions going to be a problem in a high volume app? 
Aren't many/most of those 85k-100k function calls going to 
examining/parsing all those tabledefs?

Re: [web2py] Re: Nginx-uwsgi problem.

[Anthony]

On Monday, May 14, 2012 12:06:28 PM UTC-4, Ross Peoples wrote:
>
> The problem with doing it as a C module is that it would have to be 
> compiled. I know that this is something that has been mentioned before but 
> was shot down because web2py wouldn't be easily accessible, modified, etc, 
> which is the goal of web2py. Alternatively, maybe running web2oy in PyPy 
> environment could improve the performance without taking away from the 
> flexibility of web2py.
>

C modules would also make web2py less portable (GAE, Jython). Could also be 
a problem with PyPy (even if the C modules run on PyPy, my understanding is 
that they are actually slower than pure Python).

Anthony 

Re: [web2py] Re: Nginx-uwsgi problem.

[Anthony]

>
> If you want to use db.executesql() but remain portable, you can still have 
>> the DAL generate the SQL for you by using the ._select() method:
>>
>> db.executesql(db(query)._select(...))
>>
>> Obviously in that case you don't get to hand optimize the SQL, but you 
>> still get the speed advantage of not converting the results to a Rows 
>> object (which is only significant for large results sets).
>>
>
> While true, since he is going for performance in a high-traffic 
> environment that requires low-latency, such as a site that serves ads, he 
> would definitely want to hand-craft the SQL for complex and large queries 
> that slow things down. I wouldn't recommend doing it for every query, just 
> the slow ones.
>

Sure. There are two reasons to use db.executesql: (1) save time by not 
having the DAL convert large results sets to a Rows object, (2) hand write 
SQL that the DAL either can't generate or generates less efficiently. I was 
just referring to case #1.

Anthony 

Re: [web2py] Re: Nginx-uwsgi problem.

[Richard Vézina]

Yes right, it could not be anymore included into a single zip file that you
just unpack and good to go.

About PyPy, I read long time ago from someone that has make works web2py
with it that there was not that much speed improvement.

Richard

On Mon, May 14, 2012 at 12:06 PM, Ross Peoples wrote:

> The problem with doing it as a C module is that it would have to be
> compiled. I know that this is something that has been mentioned before but
> was shot down because web2py wouldn't be easily accessible, modified, etc,
> which is the goal of web2py. Alternatively, maybe running web2oy in PyPy
> environment could improve the performance without taking away from the
> flexibility of web2py.
>
>
> On Monday, May 14, 2012 11:57:49 AM UTC-4, Richard wrote:
>>
>> Hello,
>>
>> I wonder if some of the speed problem that web2py has could be address by
>> translate some of the web2py module into C??
>>
>> There is already many walk around for speed problem, but could there is
>> some major speed improvement by rewrite some of web2py fondation into C?
>>
>> I am just curious about that possibility since I didn't see popup as
>> possible option to improve speed of app in major scalling process.
>>
>> Thanks
>>
>> Richard
>>
>> On Mon, May 14, 2012 at 11:50 AM, Anthony  wrote:
>>
>>>
1. Your "adviewer/viewads" makes 10 calls to the database. Try to
optimize this either by writing fewer queries, creating a view, and/or 
 only
selecting fields that you need. Also make sure you get the criteria 
 right
so that you (ideally) don't have any extra, unneeded rows.

 If it's feasible, also consider caching the queries for some amount of
>>> time (assuming the results don't change too frequently).
>>>

1. When you absolutely have to load a few thousand rows (or more)
in a query (you should avoid this whenever possible), then try using
"db.executesql(query)" to manually execute a hand-crafted SQL query. 
 This
will always be faster than using the DAL directly.

 Note, the difference in speed is due to the fact that the DAL won't be
>>> converting the results set to a Rows object -- so you won't have the
>>> convenience of dealing with DAL Rows and Row objects. If you do 
>>> db.executesql(query,
>>> as_dict=True), it will convert to a list of dictionaries (which is
>>> still faster than converting to a Rows object).
>>>

1. Another point about executesql: The obvious issue is reduced
portability, but if you are only planning on using PostgreSQL, then you 
 can
 hand-craft a SQL query and profile it against PostgreSQL for maximum
performance. Once you've got it giving only the data you want, then you 
 can
copy and paste that query into executesql.

 If you want to use db.executesql() but remain portable, you can still
>>> have the DAL generate the SQL for you by using the ._select() method:
>>>
>>> db.executesql(db(query)._**select(...))
>>>
>>> Obviously in that case you don't get to hand optimize the SQL, but you
>>> still get the speed advantage of not converting the results to a Rows
>>> object (which is only significant for large results sets).
>>>
>>> Anthony
>>>
>>>
>>

Re: [web2py] Re: Nginx-uwsgi problem.

[Richard Vézina]

This is cool : cache.ram.clear(key)


I didn't thought about it, I figure out that you trigger this in the
function that create new records for a given cached select... Good trade
off with caching and up to date select.

Thank for this one Ross.

Richard

On Mon, May 14, 2012 at 12:02 PM, Ross Peoples wrote:

>
>
> On Monday, May 14, 2012 11:50:06 AM UTC-4, Anthony wrote:
>>
>>
>>>1. Your "adviewer/viewads" makes 10 calls to the database. Try to
>>>optimize this either by writing fewer queries, creating a view, and/or 
>>> only
>>>selecting fields that you need. Also make sure you get the criteria right
>>>so that you (ideally) don't have any extra, unneeded rows.
>>>
>>> If it's feasible, also consider caching the queries for some amount of
>> time (assuming the results don't change too frequently).
>>
>
> This is a good point. For things that may not change a whole lot, I
> usually set the time_expires=3600 or even None. Give the key a well-defined
> name, and then when something changes, run cache.ram.clear(key) so that the
> next query will get the changes.
>
>>
>>>1. When you absolutely have to load a few thousand rows (or more) in
>>>a query (you should avoid this whenever possible), then try using
>>>"db.executesql(query)" to manually execute a hand-crafted SQL query. This
>>>will always be faster than using the DAL directly.
>>>
>>> Note, the difference in speed is due to the fact that the DAL won't be
>> converting the results set to a Rows object -- so you won't have the
>> convenience of dealing with DAL Rows and Row objects. If you do 
>> db.executesql(query,
>> as_dict=True), it will convert to a list of dictionaries (which is still
>> faster than converting to a Rows object).
>>
>>>
>>>1. Another point about executesql: The obvious issue is reduced
>>>portability, but if you are only planning on using PostgreSQL, then you 
>>> can
>>> hand-craft a SQL query and profile it against PostgreSQL for maximum
>>>performance. Once you've got it giving only the data you want, then you 
>>> can
>>>copy and paste that query into executesql.
>>>
>>> If you want to use db.executesql() but remain portable, you can still
>> have the DAL generate the SQL for you by using the ._select() method:
>>
>> db.executesql(db(query)._**select(...))
>>
>> Obviously in that case you don't get to hand optimize the SQL, but you
>> still get the speed advantage of not converting the results to a Rows
>> object (which is only significant for large results sets).
>>
>
> While true, since he is going for performance in a high-traffic
> environment that requires low-latency, such as a site that serves ads, he
> would definitely want to hand-craft the SQL for complex and large queries
> that slow things down. I wouldn't recommend doing it for every query, just
> the slow ones.
>
>
>>
>> Anthony
>>
>>

Re: [web2py] Re: Nginx-uwsgi problem.

[Ross Peoples]

The problem with doing it as a C module is that it would have to be 
compiled. I know that this is something that has been mentioned before but 
was shot down because web2py wouldn't be easily accessible, modified, etc, 
which is the goal of web2py. Alternatively, maybe running web2oy in PyPy 
environment could improve the performance without taking away from the 
flexibility of web2py.

On Monday, May 14, 2012 11:57:49 AM UTC-4, Richard wrote:
>
> Hello,
>
> I wonder if some of the speed problem that web2py has could be address by 
> translate some of the web2py module into C??
>
> There is already many walk around for speed problem, but could there is 
> some major speed improvement by rewrite some of web2py fondation into C?
>
> I am just curious about that possibility since I didn't see popup as 
> possible option to improve speed of app in major scalling process.
>
> Thanks
>
> Richard
>
> On Mon, May 14, 2012 at 11:50 AM, Anthony  wrote:
>
>>
>>>1. Your "adviewer/viewads" makes 10 calls to the database. Try to 
>>>optimize this either by writing fewer queries, creating a view, and/or 
>>> only 
>>>selecting fields that you need. Also make sure you get the criteria 
>>> right 
>>>so that you (ideally) don't have any extra, unneeded rows. 
>>>
>>> If it's feasible, also consider caching the queries for some amount of 
>> time (assuming the results don't change too frequently). 
>>
>>>
>>>1. When you absolutely have to load a few thousand rows (or more) in 
>>>a query (you should avoid this whenever possible), then try using 
>>>"db.executesql(query)" to manually execute a hand-crafted SQL query. 
>>> This 
>>>will always be faster than using the DAL directly. 
>>>
>>> Note, the difference in speed is due to the fact that the DAL won't be 
>> converting the results set to a Rows object -- so you won't have the 
>> convenience of dealing with DAL Rows and Row objects. If you do 
>> db.executesql(query, 
>> as_dict=True), it will convert to a list of dictionaries (which is still 
>> faster than converting to a Rows object). 
>>
>>>
>>>1. Another point about executesql: The obvious issue is reduced 
>>>portability, but if you are only planning on using PostgreSQL, then you 
>>> can 
>>> hand-craft a SQL query and profile it against PostgreSQL for maximum 
>>>performance. Once you've got it giving only the data you want, then you 
>>> can 
>>>copy and paste that query into executesql. 
>>>
>>> If you want to use db.executesql() but remain portable, you can still 
>> have the DAL generate the SQL for you by using the ._select() method:
>>
>> db.executesql(db(query)._select(...))
>>
>> Obviously in that case you don't get to hand optimize the SQL, but you 
>> still get the speed advantage of not converting the results to a Rows 
>> object (which is only significant for large results sets).
>>
>> Anthony
>>
>>
>

Re: [web2py] Re: Nginx-uwsgi problem.

[Ross Peoples]

On Monday, May 14, 2012 11:50:06 AM UTC-4, Anthony wrote:
>
>
>>1. Your "adviewer/viewads" makes 10 calls to the database. Try to 
>>optimize this either by writing fewer queries, creating a view, and/or 
>> only 
>>selecting fields that you need. Also make sure you get the criteria right 
>>so that you (ideally) don't have any extra, unneeded rows.
>>
>> If it's feasible, also consider caching the queries for some amount of 
> time (assuming the results don't change too frequently). 
>

This is a good point. For things that may not change a whole lot, I usually 
set the time_expires=3600 or even None. Give the key a well-defined name, 
and then when something changes, run cache.ram.clear(key) so that the next 
query will get the changes. 

>
>>1. When you absolutely have to load a few thousand rows (or more) in 
>>a query (you should avoid this whenever possible), then try using 
>>"db.executesql(query)" to manually execute a hand-crafted SQL query. This 
>>will always be faster than using the DAL directly.
>>
>> Note, the difference in speed is due to the fact that the DAL won't be 
> converting the results set to a Rows object -- so you won't have the 
> convenience of dealing with DAL Rows and Row objects. If you do 
> db.executesql(query, 
> as_dict=True), it will convert to a list of dictionaries (which is still 
> faster than converting to a Rows object). 
>
>>
>>1. Another point about executesql: The obvious issue is reduced 
>>portability, but if you are only planning on using PostgreSQL, then you 
>> can 
>> hand-craft a SQL query and profile it against PostgreSQL for maximum 
>>performance. Once you've got it giving only the data you want, then you 
>> can 
>>copy and paste that query into executesql.
>>
>> If you want to use db.executesql() but remain portable, you can still 
> have the DAL generate the SQL for you by using the ._select() method:
>
> db.executesql(db(query)._select(...))
>
> Obviously in that case you don't get to hand optimize the SQL, but you 
> still get the speed advantage of not converting the results to a Rows 
> object (which is only significant for large results sets).
>

While true, since he is going for performance in a high-traffic environment 
that requires low-latency, such as a site that serves ads, he 
would definitely want to hand-craft the SQL for complex and large queries 
that slow things down. I wouldn't recommend doing it for every query, just 
the slow ones.
 

>
> Anthony
>
>

Re: [web2py] Re: Nginx-uwsgi problem.

[Richard Vézina]

Hello,

I wonder if some of the speed problem that web2py has could be address by
translate some of the web2py module into C??

There is already many walk around for speed problem, but could there is
some major speed improvement by rewrite some of web2py fondation into C?

I am just curious about that possibility since I didn't see popup as
possible option to improve speed of app in major scalling process.

Thanks

Richard

On Mon, May 14, 2012 at 11:50 AM, Anthony  wrote:

>
>>1. Your "adviewer/viewads" makes 10 calls to the database. Try to
>>optimize this either by writing fewer queries, creating a view, and/or 
>> only
>>selecting fields that you need. Also make sure you get the criteria right
>>so that you (ideally) don't have any extra, unneeded rows.
>>
>> If it's feasible, also consider caching the queries for some amount of
> time (assuming the results don't change too frequently).
>
>>
>>1. When you absolutely have to load a few thousand rows (or more) in
>>a query (you should avoid this whenever possible), then try using
>>"db.executesql(query)" to manually execute a hand-crafted SQL query. This
>>will always be faster than using the DAL directly.
>>
>> Note, the difference in speed is due to the fact that the DAL won't be
> converting the results set to a Rows object -- so you won't have the
> convenience of dealing with DAL Rows and Row objects. If you do 
> db.executesql(query,
> as_dict=True), it will convert to a list of dictionaries (which is still
> faster than converting to a Rows object).
>
>>
>>1. Another point about executesql: The obvious issue is reduced
>>portability, but if you are only planning on using PostgreSQL, then you 
>> can
>> hand-craft a SQL query and profile it against PostgreSQL for maximum
>>performance. Once you've got it giving only the data you want, then you 
>> can
>>copy and paste that query into executesql.
>>
>> If you want to use db.executesql() but remain portable, you can still
> have the DAL generate the SQL for you by using the ._select() method:
>
> db.executesql(db(query)._select(...))
>
> Obviously in that case you don't get to hand optimize the SQL, but you
> still get the speed advantage of not converting the results to a Rows
> object (which is only significant for large results sets).
>
> Anthony
>
>

Re: [web2py] Re: Nginx-uwsgi problem.

[Anthony]

>
>
>1. Your "adviewer/viewads" makes 10 calls to the database. Try to 
>optimize this either by writing fewer queries, creating a view, and/or 
> only 
>selecting fields that you need. Also make sure you get the criteria right 
>so that you (ideally) don't have any extra, unneeded rows.
>
> If it's feasible, also consider caching the queries for some amount of 
time (assuming the results don't change too frequently). 

>
>1. When you absolutely have to load a few thousand rows (or more) in a 
>query (you should avoid this whenever possible), then try using 
>"db.executesql(query)" to manually execute a hand-crafted SQL query. This 
>will always be faster than using the DAL directly.
>
> Note, the difference in speed is due to the fact that the DAL won't be 
converting the results set to a Rows object -- so you won't have the 
convenience of dealing with DAL Rows and Row objects. If you do 
db.executesql(query, 
as_dict=True), it will convert to a list of dictionaries (which is still 
faster than converting to a Rows object). 

>
>1. Another point about executesql: The obvious issue is reduced 
>portability, but if you are only planning on using PostgreSQL, then you 
> can 
> hand-craft a SQL query and profile it against PostgreSQL for maximum 
>performance. Once you've got it giving only the data you want, then you 
> can 
>copy and paste that query into executesql.
>
> If you want to use db.executesql() but remain portable, you can still have 
the DAL generate the SQL for you by using the ._select() method:

db.executesql(db(query)._select(...))

Obviously in that case you don't get to hand optimize the SQL, but you 
still get the speed advantage of not converting the results to a Rows 
object (which is only significant for large results sets).

Anthony

Re: [web2py] Re: Nginx-uwsgi problem.

[Ross Peoples]

I'm reading through the profile log, not that I'm an expert on profiling or 
anything, but I do have a few possible tips:

   1. Most of your calls are done in ~86 ms. That's pretty respectable. 
   2. Most of your time is spent on database calls, with compiling being 
   the second biggest time consumer. So compiling your code should provide you 
   with a quick speed boost without having to change any code.
   3. The biggest bottleneck is almost always the queries to the database, 
   regardless of using the DAL or not. So spend more time optimizing your 
   queries before optimizing your code.
   4. Your "adviewer/viewads" makes 10 calls to the database. Try to 
   optimize this either by writing fewer queries, creating a view, and/or only 
   selecting fields that you need. Also make sure you get the criteria right 
   so that you (ideally) don't have any extra, unneeded rows.
   5. The "trees/index", "trees/get_binary_tree.json", 
   "trees/team_members", etc. I assume are supposed to be showing a 
   parent-child type of relationship. I've done this with folders in a 
   document management system before and trees are tough. The way I did it was 
   to load only the top-level and one level below in a single database query 
   (so you would know if the folder had subfolders), and then loaded the rest 
   whenever the user "opened" or "expanded" a folder. If your application 
   resembles this type of tree, then maybe try that. Otherwise, try to load 
   two or three levels per query. Even using a sub-select will be faster than 
   repeated queries.
   6. Moving models to modules and only loading the tables you need has 
   already been mentioned, but it's worth adding to the list of tips.
   7. When you absolutely have to load a few thousand rows (or more) in a 
   query (you should avoid this whenever possible), then try using 
   "db.executesql(query)" to manually execute a hand-crafted SQL query. This 
   will always be faster than using the DAL directly.
   8. Another point about executesql: The obvious issue is reduced 
   portability, but if you are only planning on using PostgreSQL, then you can 
hand-craft a SQL query and profile it against PostgreSQL for maximum 
   performance. Once you've got it giving only the data you want, then you can 
   copy and paste that query into executesql.

I don't know how much, if any, of this applies to your situation, but I'd 
like to think it's good advice in general.

Re: [web2py] Re: Nginx-uwsgi problem.

[Anthony]

On Sunday, May 13, 2012 8:37:17 PM UTC-4, Bruce Wade wrote:
>
> How will compiling it effect things as we are actively developing and 
> adding new features almost daily?
>

After you make a change, you can just do "Remove compiled" and then 
re-compile. You can do it via the admin interface or the command line.

Anthony 

Re: [web2py] Re: Nginx-uwsgi problem.

[Bruce Wade]

How will compiling it effect things as we are actively developing and
adding new features almost daily?

On Sun, May 13, 2012 at 5:17 PM, Anthony  wrote:

> To cut define_table execution time you could try to put migrate=False,
>> fake_migrate=False, when you call the DAL since in production the
>> model does not change (usually) at runtime.
>>
>
> Yes, definitely turn off migrations -- you can do so for the entire
> connection via:
>
> db = DAL(..., migrate_enabled=False)
>
> Also, compile the app -- I believe that particularly speeds up the views.
>
> Anthony
>



-- 
-- 
Regards,
Bruce Wade
http://ca.linkedin.com/in/brucelwade
http://www.wadecybertech.com
http://www.fittraineronline.com - Fitness Personal Trainers Online
http://www.warplydesigned.com

Re: [web2py] Re: Nginx-uwsgi problem.

[Anthony]

>
> To cut define_table execution time you could try to put migrate=False, 
> fake_migrate=False, when you call the DAL since in production the 
> model does not change (usually) at runtime. 
>

Yes, definitely turn off migrations -- you can do so for the entire 
connection via:

db = DAL(..., migrate_enabled=False)

Also, compile the app -- I believe that particularly speeds up the views.

Anthony

Re: [web2py] Re: Nginx-uwsgi problem.

[Bruce Wade]

Youadworld is HUGE some tables even have 50 columns.
On May 13, 2012 2:59 PM, "pbreit"  wrote:

> 135 tables to run YouAdWorld??

Re: [web2py] Re: Nginx-uwsgi problem.

[pbreit]

135 tables to run YouAdWorld??

Re: [web2py] Re: Nginx-uwsgi problem.

[Michele Comitini]

The first numbers to look at are times those are what affect our
lives.  The number of calls bother just the cpu ;-)

1. time to complete the action
2. cumulative time i.e. where most of the time affecting 1. is spent
3. percall time i.e. to find if there is a function that is really slow

1. is around 0.1s per call
2. most of the time is spent in db.py while it is executed
3. around 40% of the time seems to be spent in 45 define_table calls
inside the db.py; around 20% is spent in connecting to postgresql

So you need to find a way to work on define table either reducing the
number of calls, for instance splitting table definitions so that you
define only those needed in the controller actually called.
To cut define_table execution time you could try to put migrate=False,
fake_migrate=False, when you call the DAL since in production the
model does not change (usually) at runtime.
Be sure that you have connection pooling active.  If you use uwsgi in
a process only configuration (no threads) you still need to put
pool_size=1 in DAL call to activate pooling.
If you use threads... well don't.


Other considerations.

- The transaction is held for 0.1s which can be long on high
concurrent environment.  This could affect postgresql in terms of
resources and lock management.
- The time spent in the execute method of the psycopg adapter is very
low, so the query is fast.
- The number of web2py processes must be no more than twice the number
of cores on the machine and no less than that same number
- Check that you use "upstream" with keepalive in nginx server configuration.

mic


2012/5/13 Bruce Wade :
> Ok so I started it with the profile to be honest need some guidance on how
> to read this report. But like the one call there is 582,237 function calls
> that looks pretty insane.
>
> Or just loading my home page 85,545 function calls..
>
> The /ad_handler which is called by ajax and only returns None has 54,149
> function calls. See attached. If any one would like to explain how to read
> this.
>
>
> On Sat, May 12, 2012 at 2:41 PM, Bruce Wade  wrote:
>>
>> lol, yeah I guess that is my problem with being a C++ coder. But I also
>> seen that 90 tables were loaded with every request ;)
>>
>>
>> On Sat, May 12, 2012 at 1:50 PM, Michele Comitini
>>  wrote:
>>>
>>> I keep saying it but nobody listens :-)
>>>
>>> USE THE PROFILER
>>>
>>> You wasting *your* time trying to guess where the application is
>>> wasting *its* time.
>>>
>>> mic
>>>
>>> 2012/5/12 Anthony :
>>> > On Saturday, May 12, 2012 12:06:28 PM UTC-4, Bruce Wade wrote:
>>> >>
>>> >> Yeah I am going through all my controllers right now to see if I can
>>> >> use
>>> >> the folder solution. Which way would be faster? Folders or modules?
>>> >
>>> >
>>> > Well, a model file still has to be read each time it is needed, whereas
>>> > a
>>> > module remains in memory after the first time it is imported, so
>>> > perhaps the
>>> > module approach would still be a bit faster, but you might run an ab
>>> > test to
>>> > confirm (and to see if the difference is non-trivial).
>>> >
>>> > Anthony
>>
>>
>>
>>
>> --
>> --
>> Regards,
>> Bruce Wade
>> http://ca.linkedin.com/in/brucelwade
>> http://www.wadecybertech.com
>> http://www.fittraineronline.com - Fitness Personal Trainers Online
>> http://www.warplydesigned.com
>>
>
>
>
> --
> --
> Regards,
> Bruce Wade
> http://ca.linkedin.com/in/brucelwade
> http://www.wadecybertech.com
> http://www.fittraineronline.com - Fitness Personal Trainers Online
> http://www.warplydesigned.com
>

Re: [web2py] Re: Nginx-uwsgi problem.

[Bruce Wade]

Shouldn't be as I have deleted that default function.
On May 12, 2012 4:36 PM, "Bruno Rocha"  wrote:

> Do you have images being seved by web2py using the default download
> function?
>
> On Sat, May 12, 2012 at 7:51 PM, Bruce Wade  wrote:
>
>> Ok so I started it with the profile to be honest need some guidance on
>> how to read this report. But like the one call there is 582,237 function
>> calls that looks pretty insane.
>>
>> Or just loading my home page 85,545 function calls..
>>
>> The /ad_handler which is called by ajax and only returns None has 54,149
>> function calls. See attached. If any one would like to explain how to read
>> this.
>>
>>
>> On Sat, May 12, 2012 at 2:41 PM, Bruce Wade  wrote:
>>
>>> lol, yeah I guess that is my problem with being a C++ coder. But I also
>>> seen that 90 tables were loaded with every request ;)
>>>
>>>
>>> On Sat, May 12, 2012 at 1:50 PM, Michele Comitini <
>>> michele.comit...@gmail.com> wrote:
>>>
 I keep saying it but nobody listens :-)

 USE THE PROFILER

 You wasting *your* time trying to guess where the application is
 wasting *its* time.

 mic

 2012/5/12 Anthony :
 > On Saturday, May 12, 2012 12:06:28 PM UTC-4, Bruce Wade wrote:
 >>
 >> Yeah I am going through all my controllers right now to see if I can
 use
 >> the folder solution. Which way would be faster? Folders or modules?
 >
 >
 > Well, a model file still has to be read each time it is needed,
 whereas a
 > module remains in memory after the first time it is imported, so
 perhaps the
 > module approach would still be a bit faster, but you might run an ab
 test to
 > confirm (and to see if the difference is non-trivial).
 >
 > Anthony

>>>
>>>
>>>
>>> --
>>> --
>>> Regards,
>>> Bruce Wade
>>> http://ca.linkedin.com/in/brucelwade
>>> http://www.wadecybertech.com
>>> http://www.fittraineronline.com - Fitness Personal Trainers Online
>>> http://www.warplydesigned.com
>>>
>>>
>>
>>
>> --
>> --
>> Regards,
>> Bruce Wade
>> http://ca.linkedin.com/in/brucelwade
>> http://www.wadecybertech.com
>> http://www.fittraineronline.com - Fitness Personal Trainers Online
>> http://www.warplydesigned.com
>>
>>
>
>
> --
>
> Bruno Rocha
> [http://rochacbruno.com.br]
>
>

Re: [web2py] Re: Nginx-uwsgi problem.

[Bruno Rocha]

Do you have images being seved by web2py using the default download
function?

On Sat, May 12, 2012 at 7:51 PM, Bruce Wade  wrote:

> Ok so I started it with the profile to be honest need some guidance on how
> to read this report. But like the one call there is 582,237 function calls
> that looks pretty insane.
>
> Or just loading my home page 85,545 function calls..
>
> The /ad_handler which is called by ajax and only returns None has 54,149
> function calls. See attached. If any one would like to explain how to read
> this.
>
>
> On Sat, May 12, 2012 at 2:41 PM, Bruce Wade  wrote:
>
>> lol, yeah I guess that is my problem with being a C++ coder. But I also
>> seen that 90 tables were loaded with every request ;)
>>
>>
>> On Sat, May 12, 2012 at 1:50 PM, Michele Comitini <
>> michele.comit...@gmail.com> wrote:
>>
>>> I keep saying it but nobody listens :-)
>>>
>>> USE THE PROFILER
>>>
>>> You wasting *your* time trying to guess where the application is
>>> wasting *its* time.
>>>
>>> mic
>>>
>>> 2012/5/12 Anthony :
>>> > On Saturday, May 12, 2012 12:06:28 PM UTC-4, Bruce Wade wrote:
>>> >>
>>> >> Yeah I am going through all my controllers right now to see if I can
>>> use
>>> >> the folder solution. Which way would be faster? Folders or modules?
>>> >
>>> >
>>> > Well, a model file still has to be read each time it is needed,
>>> whereas a
>>> > module remains in memory after the first time it is imported, so
>>> perhaps the
>>> > module approach would still be a bit faster, but you might run an ab
>>> test to
>>> > confirm (and to see if the difference is non-trivial).
>>> >
>>> > Anthony
>>>
>>
>>
>>
>> --
>> --
>> Regards,
>> Bruce Wade
>> http://ca.linkedin.com/in/brucelwade
>> http://www.wadecybertech.com
>> http://www.fittraineronline.com - Fitness Personal Trainers Online
>> http://www.warplydesigned.com
>>
>>
>
>
> --
> --
> Regards,
> Bruce Wade
> http://ca.linkedin.com/in/brucelwade
> http://www.wadecybertech.com
> http://www.fittraineronline.com - Fitness Personal Trainers Online
> http://www.warplydesigned.com
>
>


-- 

Bruno Rocha
[http://rochacbruno.com.br]

Re: [web2py] Re: Nginx-uwsgi problem.

[Bruce Wade]

lol, yeah I guess that is my problem with being a C++ coder. But I also
seen that 90 tables were loaded with every request ;)

On Sat, May 12, 2012 at 1:50 PM, Michele Comitini <
michele.comit...@gmail.com> wrote:

> I keep saying it but nobody listens :-)
>
> USE THE PROFILER
>
> You wasting *your* time trying to guess where the application is
> wasting *its* time.
>
> mic
>
> 2012/5/12 Anthony :
> > On Saturday, May 12, 2012 12:06:28 PM UTC-4, Bruce Wade wrote:
> >>
> >> Yeah I am going through all my controllers right now to see if I can use
> >> the folder solution. Which way would be faster? Folders or modules?
> >
> >
> > Well, a model file still has to be read each time it is needed, whereas a
> > module remains in memory after the first time it is imported, so perhaps
> the
> > module approach would still be a bit faster, but you might run an ab
> test to
> > confirm (and to see if the difference is non-trivial).
> >
> > Anthony
>



-- 
-- 
Regards,
Bruce Wade
http://ca.linkedin.com/in/brucelwade
http://www.wadecybertech.com
http://www.fittraineronline.com - Fitness Personal Trainers Online
http://www.warplydesigned.com

Re: [web2py] Re: Nginx-uwsgi problem.

[Michele Comitini]

I keep saying it but nobody listens :-)

USE THE PROFILER

You wasting *your* time trying to guess where the application is
wasting *its* time.

mic

2012/5/12 Anthony :
> On Saturday, May 12, 2012 12:06:28 PM UTC-4, Bruce Wade wrote:
>>
>> Yeah I am going through all my controllers right now to see if I can use
>> the folder solution. Which way would be faster? Folders or modules?
>
>
> Well, a model file still has to be read each time it is needed, whereas a
> module remains in memory after the first time it is imported, so perhaps the
> module approach would still be a bit faster, but you might run an ab test to
> confirm (and to see if the difference is non-trivial).
>
> Anthony

Re: [web2py] Re: Nginx-uwsgi problem.

[Anthony]

On Saturday, May 12, 2012 12:06:28 PM UTC-4, Bruce Wade wrote:
>
> Yeah I am going through all my controllers right now to see if I can use 
> the folder solution. Which way would be faster? Folders or modules?
>

Well, a model file still has to be read each time it is needed, whereas a 
module remains in memory after the first time it is imported, so perhaps 
the module approach would still be a bit faster, but you might run an ab 
test to confirm (and to see if the difference is non-trivial).

Anthony

Re: [web2py] Re: Nginx-uwsgi problem.

[Bruce Wade]

OK so I am working on this technique what do you think?

API File: youadAPI.adviewer_api
from gluon import *

class AdViewerEngine:
   class tables:
AdReports = 'adreports'
Ads = 'ads'
AdveiwerSettings = 'adviewer_settings'
Keywords = 'keywords'
AdKeyword = 'ad_keyword'
AdReport = 'ad_report'
DailyAdViews = 'dailyadviews'
YearlyAdViews = 'yearlyadviews'
ViewedAds = 'viewedads'

   def __init__(self, db, mail, tables = [], migrate=True,
fake_migrate=False):
   self.db = db
   self.define_tables(tables, migrate, fake_migrate)

   def define_tables(self, tables = [], migrate=True, fake_migrate=False):
db = self.db
if (not tables or tables and AdViewerEngine.tables.AdReports in
tables) and AdViewerEngine.tables.AdReports not in db.tables:
print "Making table", AdViewerEngine.tables.AdReports
db.define_table(
AdViewerEngine.tables.AdReports,
...
migrate=migrate,
fake_migrate=fake_migrate
)

if (not tables or tables and AdViewerEngine.tables.Ads in tables)
and AdViewerEngine.tables.Ads not in db.tables:
db.define_table(
AdViewerEngine.tables.Ads,
...
migrate=migrate,
fake_migrate=fake_migrate
)

controller:
def index():
from youadAPI.adviewer_api import AdViewerEngine
adviewer_engine = AdViewerEngine(db, mail,
tables = [
AdViewerEngine.tables.Ads,
AdViewerEngine.tables.ViewedAds
],
migrate=False
)

db(db[AdViewerEngine.tables.Ads].id == 5).select().first()  # Things
like this will be moved into adviewer_api at some point

This works by only creating the tables that I define per action. Do you for
see any problems using this method?

On Sat, May 12, 2012 at 9:06 AM, Bruce Wade  wrote:

> Yeah I am going through all my controllers right now to see if I can use
> the folder solution. Which way would be faster? Folders or modules?
>
>
> On Sat, May 12, 2012 at 8:52 AM, Anthony  wrote:
>
>> On Saturday, May 12, 2012 10:51:59 AM UTC-4, Bruce Wade wrote:
>>>
>>> I am starting to think the design loading everything in models with
>>> every request, all though makes for rapid development isn't a very good
>>> design. Maybe for common settings in witch case the folder shouldn't be
>>> called models.
>>
>>
>> Note, if you have some models that are needed only for particular
>> controllers or functions, you can execute them conditionally based on the
>> controller (and optionally the function) by using sub-folders, as described
>> here: http://web2py.com/books/default/chapter/29/4#Workflow. Otherwise,
>> you can put them in modules and import them.
>>
>> Anthony
>>
>
>
>
> --
> --
> Regards,
> Bruce Wade
> http://ca.linkedin.com/in/brucelwade
> http://www.wadecybertech.com
> http://www.fittraineronline.com - Fitness Personal Trainers Online
> http://www.warplydesigned.com
>
>


-- 
-- 
Regards,
Bruce Wade
http://ca.linkedin.com/in/brucelwade
http://www.wadecybertech.com
http://www.fittraineronline.com - Fitness Personal Trainers Online
http://www.warplydesigned.com

Re: [web2py] Re: Nginx-uwsgi problem.

[Bruce Wade]

Yeah I am going through all my controllers right now to see if I can use
the folder solution. Which way would be faster? Folders or modules?

On Sat, May 12, 2012 at 8:52 AM, Anthony  wrote:

> On Saturday, May 12, 2012 10:51:59 AM UTC-4, Bruce Wade wrote:
>>
>> I am starting to think the design loading everything in models with every
>> request, all though makes for rapid development isn't a very good design.
>> Maybe for common settings in witch case the folder shouldn't be called
>> models.
>
>
> Note, if you have some models that are needed only for particular
> controllers or functions, you can execute them conditionally based on the
> controller (and optionally the function) by using sub-folders, as described
> here: http://web2py.com/books/default/chapter/29/4#Workflow. Otherwise,
> you can put them in modules and import them.
>
> Anthony
>



-- 
-- 
Regards,
Bruce Wade
http://ca.linkedin.com/in/brucelwade
http://www.wadecybertech.com
http://www.fittraineronline.com - Fitness Personal Trainers Online
http://www.warplydesigned.com

Re: [web2py] Re: Nginx-uwsgi problem.

[Anthony]

On Saturday, May 12, 2012 10:51:59 AM UTC-4, Bruce Wade wrote:
>
> I am starting to think the design loading everything in models with every 
> request, all though makes for rapid development isn't a very good design. 
> Maybe for common settings in witch case the folder shouldn't be called 
> models. 


Note, if you have some models that are needed only for particular 
controllers or functions, you can execute them conditionally based on the 
controller (and optionally the function) by using sub-folders, as described 
here: http://web2py.com/books/default/chapter/29/4#Workflow. Otherwise, you 
can put them in modules and import them.

Anthony

Re: [web2py] Re: Nginx-uwsgi problem.

[Bruce Wade]

I am starting to think the design loading everything in models with every
request, all though makes for rapid development isn't a very good design.
Maybe for common settings in witch case the folder shouldn't be called
models.

I need to go through my app completely and remove everything from models as
when ever I do any request it is getting loaded and the banner engine hits
the server every 3 times every 5 seconds. The bottle neck is definitely the
DB/layer as the old site connecting to the same db is having loading
problems now also.

On Fri, May 11, 2012 at 2:43 PM, Michele Comitini <
michele.comit...@gmail.com> wrote:

> > Connections = (pool_size) * (number of web2py processes)
> >
> > So if you have 10 threads  and pool_size = 4
> > 1 * 4 = 4 connections
> >
> > If you have 10 processes (each with 6 threads):
> > 10 * 4 = 40 connections
> >
> > As you can see the number of processes is not a term of the computation.
> > You must count the number of concurrent processes, the number of
> > threads does not count, same for the number of requests in the nginx
> > queue.
> AHEM... its too late here...
> "...the number of processes is not a term ..." should read "...the
> number of threads is not a term ..."
>
> >
> > If the db seems to be locked you can do on the db server host:
> >
> > ps ax | grep TRANSACTION
> >
> > you should get many postgres processes IDLE IN TRANSACTION.  It is a
> > symptom of web2py taking long to commit the transaction.
> > If you do not use the db in some complex view you can try to put a
> > db.rollback() at the beginning of the controller.
> > Are you using any web2py scripts (cron or the like)? check that you do
> > not keep the transaction open if the process is long.  Use alway
> > db.commit!
> >
> > mic
>



-- 
-- 
Regards,
Bruce Wade
http://ca.linkedin.com/in/brucelwade
http://www.wadecybertech.com
http://www.fittraineronline.com - Fitness Personal Trainers Online
http://www.warplydesigned.com

Re: [web2py] Re: Nginx-uwsgi problem.

[Michele Comitini]

> Connections = (pool_size) * (number of web2py processes)
>
> So if you have 10 threads  and pool_size = 4
> 1 * 4 = 4 connections
>
> If you have 10 processes (each with 6 threads):
> 10 * 4 = 40 connections
>
> As you can see the number of processes is not a term of the computation.
> You must count the number of concurrent processes, the number of
> threads does not count, same for the number of requests in the nginx
> queue.
AHEM... its too late here...
"...the number of processes is not a term ..." should read "...the
number of threads is not a term ..."

>
> If the db seems to be locked you can do on the db server host:
>
> ps ax | grep TRANSACTION
>
> you should get many postgres processes IDLE IN TRANSACTION.  It is a
> symptom of web2py taking long to commit the transaction.
> If you do not use the db in some complex view you can try to put a
> db.rollback() at the beginning of the controller.
> Are you using any web2py scripts (cron or the like)? check that you do
> not keep the transaction open if the process is long.  Use alway
> db.commit!
>
> mic

Re: [web2py] Re: Nginx-uwsgi problem.

[Michele Comitini]

2012/5/11 Bruce Wade :
> Maybe in some places of the code but not everywhere. The problem is when
> there is a large load all 3 servers get very slow on every page. I think it
> is the DB layer as we have 90 tables in one database 45 in another. I am
> also using connection pooling which I think is causing problems. Because the
> DAL is loaded with every request, wouldn't that means the pool is open for
> each request? Or should there only ever be 10 connections open even if I
> have 1000 concurrent connections?
Connections = (pool_size) * (number of web2py processes)

So if you have 10 threads  and pool_size = 4
1 * 4 = 4 connections

If you have 10 processes (each with 6 threads):
10 * 4 = 40 connections

As you can see the number of processes is not a term of the computation.
You must count the number of concurrent processes, the number of
threads does not count, same for the number of requests in the nginx
queue.

If the db seems to be locked you can do on the db server host:

ps ax | grep TRANSACTION

you should get many postgres processes IDLE IN TRANSACTION.  It is a
symptom of web2py taking long to commit the transaction.
If you do not use the db in some complex view you can try to put a
db.rollback() at the beginning of the controller.
Are you using any web2py scripts (cron or the like)? check that you do
not keep the transaction open if the process is long.  Use alway
db.commit!

mic

Re: [web2py] Re: Nginx-uwsgi problem.

[Bruce Wade]

Maybe in some places of the code but not everywhere. The problem is when
there is a large load all 3 servers get very slow on every page. I think it
is the DB layer as we have 90 tables in one database 45 in another. I am
also using connection pooling which I think is causing problems. Because
the DAL is loaded with every request, wouldn't that means the pool is open
for each request? Or should there only ever be 10 connections open even if
I have 1000 concurrent connections?

On Fri, May 11, 2012 at 6:51 AM, Richard Vézina  wrote:

> Bruce,
>
> Are you building dict with query for represent or other use?
>
> Recently I solve speed problem I had by caching dict building query.
>
> I never thought that building a dictionary could be that expensive in term
> of cpu load.
>
> Richard
>
>
> On Thu, May 10, 2012 at 5:13 PM, Bruce Wade  wrote:
>
>> Yes there are a LOT of wait state on the web2py nodes and high CPU
>>
>> I will try your suggestions.
>>
>> Thanks,
>> Bruce
>>
>>
>> On Thu, May 10, 2012 at 2:10 PM, Michele Comitini <
>> michele.comit...@gmail.com> wrote:
>>
>>> The high load on web2py nodes seems to point to code in web2py.  If it
>>> were a problem with postgres you would have a high load on postgresql
>>> and a lot of wait state and little CPU time resulting in little uptime
>>> on web2py nodes but long page rendering times.
>>> I suggest to try to convert some logic to use raw resultsets using
>>> executesql instead of DAL Row objects.  But before doing that try the
>>> query on postgres directly: you can use the _select() method to obtain
>>> the query generated by the DAL.  If postgresql answers slowly try
>>> adding indexes on columns as requested by EXPLAIN.
>>> If postgresql answers fast try the guilty query with the DAL in a
>>> python shell (i.e. python web2py.py -M -S ). If it slow than
>>> you have found the cause.
>>>
>>> Else keep using top to find if other processes are infesting the CPU
>>> maybe it is a simple problem of "ping pong" or swappiness.  Simple
>>> tuning of uWSGI could suffice.  As a rule of thumb you should not have
>>> the number of web2py processes be more than twice the number of cores.
>>>
>>> mic
>>>
>>>
>>> 2012/5/10 Bruce Wade :
>>> > Web2py is on 3 different servers/nodes, postgresql is on it's own node
>>> with
>>> > 8GB ram.
>>> >
>>> > CPU is being used by uwsgi so web2py. The slowness I think is from DB
>>> > queries as when you load a page without the DB involved much it loads
>>> > quickly
>>> >
>>> > The serving ads part is not a problem it is the other pages on the
>>> website.
>>> > At least not the adviewer the banner ads are new. The adviewer has
>>> served
>>> > over 29 million ads.
>>> >
>>> > I will try disabling the banner ads for now and set them so they are
>>> > querying from a completely different server, maybe using mongodb and
>>> node.js
>>> >
>>> >
>>> > On Thu, May 10, 2012 at 11:28 AM, pbreit 
>>> wrote:
>>> >>
>>> >> Is your traffic from serving ads or users coming to your web site?
>>> Have
>>> >> you exhausted caching opportunities?
>>> >
>>> >
>>> >
>>> >
>>> > --
>>> > --
>>> > Regards,
>>> > Bruce Wade
>>> > http://ca.linkedin.com/in/brucelwade
>>> > http://www.wadecybertech.com
>>> > http://www.fittraineronline.com - Fitness Personal Trainers Online
>>> > http://www.warplydesigned.com
>>> >
>>>
>>
>>
>>
>> --
>> --
>> Regards,
>> Bruce Wade
>> http://ca.linkedin.com/in/brucelwade
>> http://www.wadecybertech.com
>> http://www.fittraineronline.com - Fitness Personal Trainers Online
>> http://www.warplydesigned.com
>>
>>
>


-- 
-- 
Regards,
Bruce Wade
http://ca.linkedin.com/in/brucelwade
http://www.wadecybertech.com
http://www.fittraineronline.com - Fitness Personal Trainers Online
http://www.warplydesigned.com

Re: [web2py] Re: Nginx-uwsgi problem.

[Richard Vézina]

Bruce,

Are you building dict with query for represent or other use?

Recently I solve speed problem I had by caching dict building query.

I never thought that building a dictionary could be that expensive in term
of cpu load.

Richard

On Thu, May 10, 2012 at 5:13 PM, Bruce Wade  wrote:

> Yes there are a LOT of wait state on the web2py nodes and high CPU
>
> I will try your suggestions.
>
> Thanks,
> Bruce
>
>
> On Thu, May 10, 2012 at 2:10 PM, Michele Comitini <
> michele.comit...@gmail.com> wrote:
>
>> The high load on web2py nodes seems to point to code in web2py.  If it
>> were a problem with postgres you would have a high load on postgresql
>> and a lot of wait state and little CPU time resulting in little uptime
>> on web2py nodes but long page rendering times.
>> I suggest to try to convert some logic to use raw resultsets using
>> executesql instead of DAL Row objects.  But before doing that try the
>> query on postgres directly: you can use the _select() method to obtain
>> the query generated by the DAL.  If postgresql answers slowly try
>> adding indexes on columns as requested by EXPLAIN.
>> If postgresql answers fast try the guilty query with the DAL in a
>> python shell (i.e. python web2py.py -M -S ). If it slow than
>> you have found the cause.
>>
>> Else keep using top to find if other processes are infesting the CPU
>> maybe it is a simple problem of "ping pong" or swappiness.  Simple
>> tuning of uWSGI could suffice.  As a rule of thumb you should not have
>> the number of web2py processes be more than twice the number of cores.
>>
>> mic
>>
>>
>> 2012/5/10 Bruce Wade :
>> > Web2py is on 3 different servers/nodes, postgresql is on it's own node
>> with
>> > 8GB ram.
>> >
>> > CPU is being used by uwsgi so web2py. The slowness I think is from DB
>> > queries as when you load a page without the DB involved much it loads
>> > quickly
>> >
>> > The serving ads part is not a problem it is the other pages on the
>> website.
>> > At least not the adviewer the banner ads are new. The adviewer has
>> served
>> > over 29 million ads.
>> >
>> > I will try disabling the banner ads for now and set them so they are
>> > querying from a completely different server, maybe using mongodb and
>> node.js
>> >
>> >
>> > On Thu, May 10, 2012 at 11:28 AM, pbreit 
>> wrote:
>> >>
>> >> Is your traffic from serving ads or users coming to your web site? Have
>> >> you exhausted caching opportunities?
>> >
>> >
>> >
>> >
>> > --
>> > --
>> > Regards,
>> > Bruce Wade
>> > http://ca.linkedin.com/in/brucelwade
>> > http://www.wadecybertech.com
>> > http://www.fittraineronline.com - Fitness Personal Trainers Online
>> > http://www.warplydesigned.com
>> >
>>
>
>
>
> --
> --
> Regards,
> Bruce Wade
> http://ca.linkedin.com/in/brucelwade
> http://www.wadecybertech.com
> http://www.fittraineronline.com - Fitness Personal Trainers Online
> http://www.warplydesigned.com
>
>

Re: [web2py] Re: Nginx-uwsgi problem.

[Bruce Wade]

Yes there are a LOT of wait state on the web2py nodes and high CPU

I will try your suggestions.

Thanks,
Bruce

On Thu, May 10, 2012 at 2:10 PM, Michele Comitini <
michele.comit...@gmail.com> wrote:

> The high load on web2py nodes seems to point to code in web2py.  If it
> were a problem with postgres you would have a high load on postgresql
> and a lot of wait state and little CPU time resulting in little uptime
> on web2py nodes but long page rendering times.
> I suggest to try to convert some logic to use raw resultsets using
> executesql instead of DAL Row objects.  But before doing that try the
> query on postgres directly: you can use the _select() method to obtain
> the query generated by the DAL.  If postgresql answers slowly try
> adding indexes on columns as requested by EXPLAIN.
> If postgresql answers fast try the guilty query with the DAL in a
> python shell (i.e. python web2py.py -M -S ). If it slow than
> you have found the cause.
>
> Else keep using top to find if other processes are infesting the CPU
> maybe it is a simple problem of "ping pong" or swappiness.  Simple
> tuning of uWSGI could suffice.  As a rule of thumb you should not have
> the number of web2py processes be more than twice the number of cores.
>
> mic
>
>
> 2012/5/10 Bruce Wade :
> > Web2py is on 3 different servers/nodes, postgresql is on it's own node
> with
> > 8GB ram.
> >
> > CPU is being used by uwsgi so web2py. The slowness I think is from DB
> > queries as when you load a page without the DB involved much it loads
> > quickly
> >
> > The serving ads part is not a problem it is the other pages on the
> website.
> > At least not the adviewer the banner ads are new. The adviewer has served
> > over 29 million ads.
> >
> > I will try disabling the banner ads for now and set them so they are
> > querying from a completely different server, maybe using mongodb and
> node.js
> >
> >
> > On Thu, May 10, 2012 at 11:28 AM, pbreit  wrote:
> >>
> >> Is your traffic from serving ads or users coming to your web site? Have
> >> you exhausted caching opportunities?
> >
> >
> >
> >
> > --
> > --
> > Regards,
> > Bruce Wade
> > http://ca.linkedin.com/in/brucelwade
> > http://www.wadecybertech.com
> > http://www.fittraineronline.com - Fitness Personal Trainers Online
> > http://www.warplydesigned.com
> >
>



-- 
-- 
Regards,
Bruce Wade
http://ca.linkedin.com/in/brucelwade
http://www.wadecybertech.com
http://www.fittraineronline.com - Fitness Personal Trainers Online
http://www.warplydesigned.com

Re: [web2py] Re: Nginx-uwsgi problem.

[Michele Comitini]

The high load on web2py nodes seems to point to code in web2py.  If it
were a problem with postgres you would have a high load on postgresql
and a lot of wait state and little CPU time resulting in little uptime
on web2py nodes but long page rendering times.
I suggest to try to convert some logic to use raw resultsets using
executesql instead of DAL Row objects.  But before doing that try the
query on postgres directly: you can use the _select() method to obtain
the query generated by the DAL.  If postgresql answers slowly try
adding indexes on columns as requested by EXPLAIN.
If postgresql answers fast try the guilty query with the DAL in a
python shell (i.e. python web2py.py -M -S ). If it slow than
you have found the cause.

Else keep using top to find if other processes are infesting the CPU
maybe it is a simple problem of "ping pong" or swappiness.  Simple
tuning of uWSGI could suffice.  As a rule of thumb you should not have
the number of web2py processes be more than twice the number of cores.

mic


2012/5/10 Bruce Wade :
> Web2py is on 3 different servers/nodes, postgresql is on it's own node with
> 8GB ram.
>
> CPU is being used by uwsgi so web2py. The slowness I think is from DB
> queries as when you load a page without the DB involved much it loads
> quickly
>
> The serving ads part is not a problem it is the other pages on the website.
> At least not the adviewer the banner ads are new. The adviewer has served
> over 29 million ads.
>
> I will try disabling the banner ads for now and set them so they are
> querying from a completely different server, maybe using mongodb and node.js
>
>
> On Thu, May 10, 2012 at 11:28 AM, pbreit  wrote:
>>
>> Is your traffic from serving ads or users coming to your web site? Have
>> you exhausted caching opportunities?
>
>
>
>
> --
> --
> Regards,
> Bruce Wade
> http://ca.linkedin.com/in/brucelwade
> http://www.wadecybertech.com
> http://www.fittraineronline.com - Fitness Personal Trainers Online
> http://www.warplydesigned.com
>

Re: [web2py] Re: Nginx-uwsgi problem.

[Bruce Wade]

Web2py is on 3 different servers/nodes, postgresql is on it's own node with
8GB ram.

CPU is being used by uwsgi so web2py. The slowness I think is from DB
queries as when you load a page without the DB involved much it loads
quickly

The serving ads part is not a problem it is the other pages on the website.
At least not the adviewer the banner ads are new. The adviewer has served
over 29 million ads.

I will try disabling the banner ads for now and set them so they are
querying from a completely different server, maybe using mongodb and node.js


On Thu, May 10, 2012 at 11:28 AM, pbreit  wrote:

> Is your traffic from serving ads or users coming to your web site? Have
> you exhausted caching opportunities?




-- 
-- 
Regards,
Bruce Wade
http://ca.linkedin.com/in/brucelwade
http://www.wadecybertech.com
http://www.fittraineronline.com - Fitness Personal Trainers Online
http://www.warplydesigned.com

Re: [web2py] Re: Nginx-uwsgi problem.

[Michele Comitini]

Is the high load caused by web2py process or postgresql?
Is web2py on the same machine as postgresql?

Use top or htop to find which process uses the CPU.  If it is web2py
*use* the profiler. If it postgres try to find the bad query and try
to improve it by using EXPLAIN.
Could be also the swapping causing the troubles.  On VPS nodes
remember to lower the swappines (see sysctl) to n < 20 or remove swap
completely.


mic


2012/5/10 Ross Peoples :
> The DB layer is usually the bottleneck. However, moving from models to
> modules should reduce any bottleneck caused by the web server.

Re: [web2py] Re: Nginx-uwsgi problem.

[Bruno Rocha]

The verification  email has been sent!

But I am using a feature that allows first login without registration, but
you will not be able to login again

I will include a captcha...

  de rifar...@gmail.com para d...@dfgdfg.pl
 data 19 de setembro de 2011 06:27 assunto Bem-vindo, confirme seu
e-mail enviado
por gmail.com
Imagens recebidas deste remetente são sempre exibidas Não exibir de agora em
diante.
 ocultar detalhes 06:27 (8 horas atrás)
 [image: logo]  Olá, Seja bem-vindo ao Rifar.me!
Importante!Clique em
http://www.rifar.me/user/verify_email/406c7491-0689-4b47894-ad32-d2088b37ef50
para
verificar o seu e-mail.

O Rifar.me é um serviço de rifas online. Totalmente automatizado, os
compradores escolhem os números que desejam comprar (o nosso sistema
gerencia os números que estão disponíveis e os que já foram
comprados/reservados) e ainda podem, se assim desejarem, efetuar o pagamento
através do PagSeguro. Também é possível efetuar o pagamento em conta
bancária, se o criador da rifa disponibilizar uma conta.

Comece agora mesmo a criar  suas rifas
online e aproveite para comprar alguns números das
rifas que
já estão online!
Aos e cadastrar você concordou com nossos termos e
condições

Boa Sorte!
Rifar.me 

On Mon, Sep 19, 2011 at 6:36 AM, Vineet  wrote:

> (a littlebit off-topic comment)
> Bruno,
> I visited the mentioned site.
> I gave username as 'fff' & 'ddd' and a fake email; It got registered.
> Isn't it a good idea to send a verification email to prevent fake
> emails and a 'captcha' to prevent automated registrations !
>
>