[web2py] Re: Admin crashing - ticket generated but can't see it

[Dave S]

On Wednesday, November 20, 2013 7:29:46 AM UTC-8, Jim S wrote:
>
> Cleared my browser cache and now it works...
>
> -Jim
>
>
BTDT (had to be led by the hand to get out of the maze).

Glad you figured it out.

/dps


 

> On Wednesday, November 20, 2013 8:44:44 AM UTC-6, Jim S wrote:
>>
>> Just updated to latest Version 2.7.4-stable+timestamp.2013.11.19.17.37.28
>>
>> Start web2py, click on 'Administrative Interface' and a ticket is 
>> generated but can't get into the admin interface to see it.
>>
>> -Jim
>>
>

-- 
Resources:
- http://web2py.com
- http://web2py.com/book (Documentation)
- http://github.com/web2py/web2py (Source code)
- https://code.google.com/p/web2py/issues/list (Report Issues)
--- 
You received this message because you are subscribed to the Google Groups 
"web2py-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to web2py+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[web2py] Re: web2py 2.6.1 is OUT - IMPORTANT READ

[Dave S]

On Wednesday, November 20, 2013 1:33:38 AM UTC-8, Krishna Bavandlapally 
wrote:
>
> I have updated web2py today, Every thing is fine except I am unable to 
> open admin interface, not able to see the ticket also. In local machine its 
> working fine.
>
 

> Regards,
> Krishna
>  
>

If you took care of the web2py.js and appadmin.* copies mentioned in the 
top post of this thread, the next thing to try is clearing your browser 
cache.

Also, if you have tickets in your app from before the upgrade, zip them off 
somewhere out of sight, because the new ticket parser will ticket itself 
trying to read them.

/dps
 

>
> On Friday, 13 September 2013 03:57:37 UTC+5:30, Massimo Di Pierro wrote:
>>
>> Web2py 2.6.1 is finally OUT. It needs some more testing.
>>
>> READ BELOW BEFORE UPGRADING
>>
>>  Attention all users: For pre 2.6 applications to work with web2py 
>> >=2.6, you must copy static/js/web2py.js, controllers/appadmin.py, and 
>> views/appadmin.html from the welcome app to your own apps (all of them).
>>
>>  Attention production users: The updated handlers and examples are in 
>> handlers/ and examples/. The updated ones will not override the existing 
>> ones. To use the new ones it is not sufficient to upgrade web2py, you also 
>> need to copy the desired handler/example in the root web2py/ folder.
>>
>>  Attention MySQL users: The length of string fields changed from 255 
>> to 512 bytes. If you have migrations enabled this will trigger a large 
>> migration. To prevent it, first set migrate_enabled=False, upgrade, check 
>> everything is ok, then add length=255 to your string Fields, then re-enable 
>> migrations with migrate_enabled=True if needed.
>>
>> - better directory structure: handlers/ extras/ examples/
>> - better MongoDb support, thanks Alan
>> - better Admin editor interface, thanks Paolo, Roberto (codemirror 3), 
>> and Lightdot
>> - better layout.html and web2py_bootstrap.css, thanks Paolo
>> - refactored web2py.js makes code more readable, thanks Niphlod
>> - compute fields can depend on other compute(d) fields
>> - more functions in appadmin (/manage/auth), thanks Anthony
>> - support for CAST in SQL generation
>> - new API jQuery('#component').reload()
>> - new API rows.render()
>> - new API table.field.referent, table._references
>> - new API db(...).validate_and_update(...)
>> - new API Wiki(..., force_render=True) renders the page source again 
>> instead of using cached
>> - Wiki now automatically parses named component arguments 
>> @{f:a=1,b='twp',c=variable}
>> - auth.get_or_create_user(login=False)
>> - auth = Auth(crsf_protection = False) prevents creating sessions in 
>> login/register forms.
>> - enable multiple renderers in wiki, thanks Alan
>> - log messages from Auth are no longer translated (for speed and 
>> readability)
>> - update jQuery mobile to 1.3.1
>> - reduced memory footprint by conditionally loading Tk
>> - faster pbkdf2 uses OpenSSL, thanks Michele
>> - many speed improvements, thanks Michele
>> - better session logic, prevents false positive when detecting session 
>> changes.
>> - scripts/import_static.py converts a static site to a web2py app 
>> (experimental)
>> - support for new http error code 451
>> - profiler saves dump in dir, thanks Niphlod
>> - upgraded pyfpdf, thanks Mariano
>> - gluon/contrib/pdfinvoice.py for generating PDF invoices (assumes 
>> reportlab)
>> - no more double submission of forms (even without crsf protection), 
>> thanks Niphlod
>> - speedup for define_table, thanks Michele
>> - settings.cfg to admin, thanks Paolo
>> - many bugs fixed, thanks Niphlod, Michele, Roberto, Jonathan, and many 
>> others
>>
>
 

-- 
Resources:
- http://web2py.com
- http://web2py.com/book (Documentation)
- http://github.com/web2py/web2py (Source code)
- https://code.google.com/p/web2py/issues/list (Report Issues)
--- 
You received this message because you are subscribed to the Google Groups 
"web2py-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to web2py+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[web2py] Which is the best way to profile a web2py application? Any tutorial/link will be appreciated

[Jayadevan M]

Google told me about runsnakerun and Kcachegrind. Is there something 
specifically for web2py? Any helpful urls?

-- 
Resources:
- http://web2py.com
- http://web2py.com/book (Documentation)
- http://github.com/web2py/web2py (Source code)
- https://code.google.com/p/web2py/issues/list (Report Issues)
--- 
You received this message because you are subscribed to the Google Groups 
"web2py-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to web2py+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[web2py] Re: Bootstrap modal window with date widget

[webpypy]

Hi,

in version 2.7.4, , the calendar popup pops up behind the modal window.

Ashraf

-- 
Resources:
- http://web2py.com
- http://web2py.com/book (Documentation)
- http://github.com/web2py/web2py (Source code)
- https://code.google.com/p/web2py/issues/list (Report Issues)
--- 
You received this message because you are subscribed to the Google Groups 
"web2py-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to web2py+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[web2py] how to upload multiple file in uploads folder

[sonu kumar]

Hi All,

I need help in uploading multiple files from one form into upload folder.

Here are my code which I am trying but not working at the moment.

Controller:

def user():
values= ['','MMP2', 'MMP3', 'MMP8', 'MMP9', 'MMP10', 'MMP14', 'MMP15', 
'MMP16', 'MMP17', 'MMP24']
import shutil
import os

form=FORM(TABLE(TR('Select 
proteases:',SELECT(values,requires=IS_NOT_EMPTY("choose 
one"),_name='protease',_onchange="jQuery.post('%s',{'protease':jQuery(this).val()})"
 
% URL('mmp'))),
TR('Upload fasta file:',INPUT(_type='file', 
_name='fastafile',_multiple='',_id='fastafile')),
TR("",INPUT(_type="submit",_value="SUBMIT"

if form.process().accepted: 
if form.vars.fastafile <> '':
file2 = request.vars.getlist("fastafile")
file1=request.vars.file2

shutil.copyfileobj(file2,open('/opt/web-apps/web2py/applications/multiple_upload/uploads/'+file2,'wb'))
redirect(URL('upload_data'))

return dict(form=form)

View- user.html
works fine, it takes multiple file. But not seeing any files in uploads 
folder..

Help me out to correct this code

thanks

-- 
Resources:
- http://web2py.com
- http://web2py.com/book (Documentation)
- http://github.com/web2py/web2py (Source code)
- https://code.google.com/p/web2py/issues/list (Report Issues)
--- 
You received this message because you are subscribed to the Google Groups 
"web2py-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to web2py+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Re: [web2py] Re: Web2py freezing on live deployment!

[Andrew Buchan]

Hi Niphlod, what is asking for disasters? Bumping it up to 500? Can you
elaborate on why? (I'm all for avoiding disasters given what's been
happening!)
The server-status page was showing 150 worker threads active, except lots
of the seem not to be from proper requests. Here's my post on apachelounge
with more details: http://www.apachelounge.com/viewtopic.php?t=5655

Apache since became unresponsive again since, strangely I could still
access appadmin via local host, but pages in the main application weren't
responding, even from localhost... CPU and memory were both fine. Made me
wonder if I'm doing something stupid in something very top-level
application-wise (like in a model file or in layout.html), but I would
expect an error message in that case.

I'll look into clearing sessions, thanks for the tip.


On Wed, Nov 20, 2013 at 4:26 PM, Niphlod  wrote:

>
>
> On Wednesday, November 20, 2013 3:56:51 PM UTC+1, Andrew Buchan wrote:
>>
>> Further update:
>>
>> Apache did finally crash, with the following message:
>>
>> "Server ran out of threads to serve requests. Consider raising the
>> ThreadsPerChild setting"
>>
>> I've bumped that up to 500, and there are less than that number of users
>> on the system (although I understand there will be more than one connection
>> fired off per user/session).
>> But it crashed/became unresponsive again with no error message!
>>
> Asking for real disasters! Are you sure that there are more than 150
> (default) requests active ?
>
>
>>  Also, I leave sessions alone, but should I be clearing them?
>>
> Yep, you definitely should (as anyone in production)
>
>
>> Also also, I updated to the latest version of web2py and it seems someone
>> left a print statement in SQLFORM.grid() which prints every row! (Line 2373
>> in sqlhtml.py) I presume that if I were running that as a windows service
>> on rocket it would crash the moment you displayed a grid..
>>
>
> known issue. Upgrade to trunk
>
> --
> Resources:
> - http://web2py.com
> - http://web2py.com/book (Documentation)
> - http://github.com/web2py/web2py (Source code)
> - https://code.google.com/p/web2py/issues/list (Report Issues)
> ---
> You received this message because you are subscribed to a topic in the
> Google Groups "web2py-users" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/web2py/PEC1uLfzlrU/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to
> web2py+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.
>

-- 
Resources:
- http://web2py.com
- http://web2py.com/book (Documentation)
- http://github.com/web2py/web2py (Source code)
- https://code.google.com/p/web2py/issues/list (Report Issues)
--- 
You received this message because you are subscribed to the Google Groups 
"web2py-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to web2py+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[web2py] Re: Making the DAL orders-of-magnitude faster...

[Arnon Marcus]

My guess about .select(processor=..) when compared to having a separate 
"flat" function, is that the additional layer of indirection (function 
invocation) incurs a small penalty that shows-up.

I did a non-web2py-related test earlier, that showed consisant x2.2 penalty 
of adding a single-layer of indirection, by re-factoring-out a line into a 
single-liner function

Was something like this:

amount = 100


def doWorkDirectly():
List = []
for i in range(amount):
List.appent('a')




def doWork(List): List.append('a')


def doWorkIndirectly():
List = []
for i in range(amount):
doWork(List)


 
if you bench doWorkDirectly against doWorkIndirecty, you would get a very 
solid and sonsistant x2.2 penalty in favor of doWorkDirectly - regardless 
of the order-of-magnitude of 'amount' (same gap for 10, 1000 and 10)...

So this is a c-python implementation detail.

Granted, it would show-up more in very simple processing-payload, as there 
the invocation-overhead overshadows the actual computation expense - in 
more expensive procedures,  there would still be the same penalty, but it 
would be less noticeable.

So, passing a one-liner callback (or a lambda, though I havent tried 
that...) into ".select(processor=...)", for a computationally-simple 
expense such as zipping a list into a dictionary, .may fall into that 
category - you may loose more than you gain by it...

As for the parsing logic - that's beyond me, personally...
Shouldn't that move into the web2py-devs group?

-- 
Resources:
- http://web2py.com
- http://web2py.com/book (Documentation)
- http://github.com/web2py/web2py (Source code)
- https://code.google.com/p/web2py/issues/list (Report Issues)
--- 
You received this message because you are subscribed to the Google Groups 
"web2py-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to web2py+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[web2py] Model extractor for legacy Oracle databases - call for testing

[oscar . fonts]

Hello,

I needed to generate models from a couple of rather complex legacy Oracle 
databases.

Couldn't find an "extract_oracle_models.py" script, so I wrote one, derived 
from the pgsql one:
https://github.com/oscarfonts/web2py/blob/oracle-extract/scripts/extract_oracle_models.py

I'd like to contribute it back to the project. But I'm rather new to 
web2py, so before setting up a pull request, it'd better be tested 
thoroughly.

So if anyone is interested in this feature and has an Oracle instance at 
hand, please give it a try and comment back. Or, maybe you already have a 
systematic battery of tests I could run by myself. In any case, let me know 
the best way to quality check the script before merging it to trunk.

Oh, and I hope you find it useful!

Oscar Fonts.

-- 
Resources:
- http://web2py.com
- http://web2py.com/book (Documentation)
- http://github.com/web2py/web2py (Source code)
- https://code.google.com/p/web2py/issues/list (Report Issues)
--- 
You received this message because you are subscribed to the Google Groups 
"web2py-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to web2py+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[web2py] Re: web2py 2.6.1 is OUT - IMPORTANT READ

[Krishna Bavandlapally]

I have updated web2py today, Every thing is fine except I am unable to open 
admin interface, not able to see the ticket also. In local machine its 
working fine.

Regards,
Krishna
 

On Friday, 13 September 2013 03:57:37 UTC+5:30, Massimo Di Pierro wrote:
>
> Web2py 2.6.1 is finally OUT. It needs some more testing.
>
> READ BELOW BEFORE UPGRADING
>
>  Attention all users: For pre 2.6 applications to work with web2py 
> >=2.6, you must copy static/js/web2py.js, controllers/appadmin.py, and 
> views/appadmin.html from the welcome app to your own apps (all of them).
>
>  Attention production users: The updated handlers and examples are in 
> handlers/ and examples/. The updated ones will not override the existing 
> ones. To use the new ones it is not sufficient to upgrade web2py, you also 
> need to copy the desired handler/example in the root web2py/ folder.
>
>  Attention MySQL users: The length of string fields changed from 255 
> to 512 bytes. If you have migrations enabled this will trigger a large 
> migration. To prevent it, first set migrate_enabled=False, upgrade, check 
> everything is ok, then add length=255 to your string Fields, then re-enable 
> migrations with migrate_enabled=True if needed.
>
> - better directory structure: handlers/ extras/ examples/
> - better MongoDb support, thanks Alan
> - better Admin editor interface, thanks Paolo, Roberto (codemirror 3), and 
> Lightdot
> - better layout.html and web2py_bootstrap.css, thanks Paolo
> - refactored web2py.js makes code more readable, thanks Niphlod
> - compute fields can depend on other compute(d) fields
> - more functions in appadmin (/manage/auth), thanks Anthony
> - support for CAST in SQL generation
> - new API jQuery('#component').reload()
> - new API rows.render()
> - new API table.field.referent, table._references
> - new API db(...).validate_and_update(...)
> - new API Wiki(..., force_render=True) renders the page source again 
> instead of using cached
> - Wiki now automatically parses named component arguments 
> @{f:a=1,b='twp',c=variable}
> - auth.get_or_create_user(login=False)
> - auth = Auth(crsf_protection = False) prevents creating sessions in 
> login/register forms.
> - enable multiple renderers in wiki, thanks Alan
> - log messages from Auth are no longer translated (for speed and 
> readability)
> - update jQuery mobile to 1.3.1
> - reduced memory footprint by conditionally loading Tk
> - faster pbkdf2 uses OpenSSL, thanks Michele
> - many speed improvements, thanks Michele
> - better session logic, prevents false positive when detecting session 
> changes.
> - scripts/import_static.py converts a static site to a web2py app 
> (experimental)
> - support for new http error code 451
> - profiler saves dump in dir, thanks Niphlod
> - upgraded pyfpdf, thanks Mariano
> - gluon/contrib/pdfinvoice.py for generating PDF invoices (assumes 
> reportlab)
> - no more double submission of forms (even without crsf protection), 
> thanks Niphlod
> - speedup for define_table, thanks Michele
> - settings.cfg to admin, thanks Paolo
> - many bugs fixed, thanks Niphlod, Michele, Roberto, Jonathan, and many 
> others
>

-- 
Resources:
- http://web2py.com
- http://web2py.com/book (Documentation)
- http://github.com/web2py/web2py (Source code)
- https://code.google.com/p/web2py/issues/list (Report Issues)
--- 
You received this message because you are subscribed to the Google Groups 
"web2py-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to web2py+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[web2py] Uploading file through restful webservice

[Raj Chinna]

Hi,

I am new to web2py and currently built a restful webservice that accepts 
data from my client (Android Application) and stores the data into the DB. 
But, due to business need I need to send a text file from the client and 
the server needs to store it in server file system. Can someone give me a 
pointer on how to implement file handling in web2py restful service?

I did search through web2py docs and groups but couldn't find any relevant 
answer. kindly help me with a pointer.

Thanks in Advance

-- 
Resources:
- http://web2py.com
- http://web2py.com/book (Documentation)
- http://github.com/web2py/web2py (Source code)
- https://code.google.com/p/web2py/issues/list (Report Issues)
--- 
You received this message because you are subscribed to the Google Groups 
"web2py-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to web2py+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Re: [web2py] Re: Virtual Field and sqlform.grid

[Tim Richardson]

Also, my experience with virtual fields is limited to returning 
non-iterable types, not lists. 

On Thursday, November 21, 2013 9:48:57 AM UTC+11, Tim Richardson wrote:
>
> I can't see what's wrong.
> Why don't you simply pass the id
> (ie lambda row: get_category_list(row.Post.id)
>
>>  
>
> and then use breakpoints to see if that is working as you expect
>

-- 
Resources:
- http://web2py.com
- http://web2py.com/book (Documentation)
- http://github.com/web2py/web2py (Source code)
- https://code.google.com/p/web2py/issues/list (Report Issues)
--- 
You received this message because you are subscribed to the Google Groups 
"web2py-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to web2py+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Re: [web2py] Re: Virtual Field and sqlform.grid

[Tim Richardson]

I can't see what's wrong.
Why don't you simply pass the id
(ie lambda row: get_category_list(row.Post.id)

>  

and then use breakpoints to see if that is working as you expect

-- 
Resources:
- http://web2py.com
- http://web2py.com/book (Documentation)
- http://github.com/web2py/web2py (Source code)
- https://code.google.com/p/web2py/issues/list (Report Issues)
--- 
You received this message because you are subscribed to the Google Groups 
"web2py-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to web2py+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Re: [web2py] Re: Virtual Field and sqlform.grid

[Eduardo Cruz]

def posts():
grid = SQLFORM.grid(db.Post)
return dict(grid=grid)Ingresar el código aquí... 
 Sure.


El miércoles, 20 de noviembre de 2013 17:49:33 UTC-4, Tim Richardson 
escribió:
>
> can you show how you make the grid? 
>>
>> 

-- 
Resources:
- http://web2py.com
- http://web2py.com/book (Documentation)
- http://github.com/web2py/web2py (Source code)
- https://code.google.com/p/web2py/issues/list (Report Issues)
--- 
You received this message because you are subscribed to the Google Groups 
"web2py-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to web2py+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Re: [web2py] Re: avoid out of range error from googlebot

[Niphlod]

raise HTTP(404)

http://web2py.com/books/default/chapter/29/04/the-core?search=raise#HTTP-and-redirect

On Wednesday, November 20, 2013 5:41:02 PM UTC+1, BlueShadow wrote:
>
> that was the answer I was looking for. how do I direct the request to a 
> 404 page or some other function?
>
>

-- 
Resources:
- http://web2py.com
- http://web2py.com/book (Documentation)
- http://github.com/web2py/web2py (Source code)
- https://code.google.com/p/web2py/issues/list (Report Issues)
--- 
You received this message because you are subscribed to the Google Groups 
"web2py-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to web2py+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[web2py] Re: Making the DAL orders-of-magnitude faster...

[Niphlod]

re-splitting the hair in half.
I'm comparing
db(query).select(cacheable=True)
with
db(query).select(processor=myprocessor)
to find the culprit.
parse_value() is evidently the one blowing times up
Skipping back a little bit, I can shave a 1% (from 0.401 to 0.397) changing 
#L2221 
to

fields[j]['type']


Let's move "forward" to parse_value() (gets called 1k * num_of_columns 
times).
Lot's of ifs (L2016, 2021, 2023).
Commenting the first drops to 0.389
Commenting also the second drops to 0.318
Commenting also the third drops to 0.299

So, what actually I didn't expect sucking up times are those 3 ifs, that 
shaved off the 25% of the performances.
I guess those are "safety-precautions" but are they really needed ? 

On Wednesday, November 20, 2013 9:51:17 PM UTC+1, Niphlod wrote:
>
> splitting the hair in a half, 
> db(query)._select()
> takes roughly 0.0005 in my computer, compared to the "total" of 
> executesql() that has it "pregenerated", of 0.026, so we're talking about 
> the 2% of it. 
> Times to generate the query are actually a lot faster than the time to 
> parse the resultset (as expected).
> Let's see if I can break further down the parse() to see where it "drops" 
> from 0.022 to 0.396
>
>

-- 
Resources:
- http://web2py.com
- http://web2py.com/book (Documentation)
- http://github.com/web2py/web2py (Source code)
- https://code.google.com/p/web2py/issues/list (Report Issues)
--- 
You received this message because you are subscribed to the Google Groups 
"web2py-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to web2py+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Re: [web2py] Re: Virtual Field and sqlform.grid

[Tim Richardson]

can you show how you make the grid? 
>
>  

-- 
Resources:
- http://web2py.com
- http://web2py.com/book (Documentation)
- http://github.com/web2py/web2py (Source code)
- https://code.google.com/p/web2py/issues/list (Report Issues)
--- 
You received this message because you are subscribed to the Google Groups 
"web2py-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to web2py+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[web2py] Re: sqlform.grid with div instead of html tables

[User]

That's what I was thinking also but I wanted to make sure I wasn't missing 
some hidden capability of web2py

On Wednesday, November 20, 2013 6:53:21 AM UTC-5, Niphlod wrote:
>
> do it in your own views. altering all the markup of the grid and break it 
> down to do what you're asking it's just asking for disasters at any new 
> release of the grid's code (and lots of cpu wasted for nothing)
>
> On Wednesday, November 20, 2013 3:11:51 AM UTC+1, User wrote:
>>
>> This sounds like what I'm looking for.  Currently I'm manually creating a 
>> list display by having a for loop in the view to spit out  items.  
>> However I'm wondering if I can use sqlform.grid to replace the custom  
>> list.
>>
>> 1. Is this expected usage of sqlform.grid or is this more of a hack?
>>
>> 2. Can someone give a very basic example of what "myformat" function 
>> might look like?
>> mydisplay = myformat(grid.rows)
>> grid.element('.web2py_table', replace=mydisplay)
>>
>>
>>
>>
>> On Saturday, September 21, 2013 12:34:51 AM UTC-4, ssuresh wrote:
>>
>>> Hi,
>>> I want to use sqlform.grid and all its features, but do not want the 
>>> display in a tabular format. Is there a way in which I can change the 
>>> display of sqlform.grid to my custom  instead of  .
>>>
>>>  I tried doing some formatting on grid.rows
>>>
>>> mydisplay = myformat(grid.rows)
>>> grid.element('.web2py_table', replace=mydisplay)
>>>
>>> but still the resultant output is displayed within a . Is there 
>>> any alternatives?
>>>
>>> Any help would be greatly appreciated.
>>>
>>> thanx,
>>> Suresh
>>>
>>

-- 
Resources:
- http://web2py.com
- http://web2py.com/book (Documentation)
- http://github.com/web2py/web2py (Source code)
- https://code.google.com/p/web2py/issues/list (Report Issues)
--- 
You received this message because you are subscribed to the Google Groups 
"web2py-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to web2py+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[web2py] Re: Making the DAL orders-of-magnitude faster...

[Niphlod]

splitting the hair in a half, 
db(query)._select()
takes roughly 0.0005 in my computer, compared to the "total" of 
executesql() that has it "pregenerated", of 0.026, so we're talking about 
the 2% of it. 
Times to generate the query are actually a lot faster than the time to 
parse the resultset (as expected).
Let's see if I can break further down the parse() to see where it "drops" 
from 0.022 to 0.396

-- 
Resources:
- http://web2py.com
- http://web2py.com/book (Documentation)
- http://github.com/web2py/web2py (Source code)
- https://code.google.com/p/web2py/issues/list (Report Issues)
--- 
You received this message because you are subscribed to the Google Groups 
"web2py-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to web2py+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Re: [web2py] Re: Virtual Field and sqlform.grid

[Eduardo Cruz]

Web2py Version 2.7.4-stable+timestamp.2013.10.14.15.16.29



On 20 November 2013 16:26, Tim Richardson  wrote:

> What version of web2py?
>
>
> On Thursday, November 21, 2013 5:26:23 AM UTC+11, Eduardo Cruz wrote:
>>
>> Whe I use a table that has some a virtual field sqlform.grid does not
>> works
>>
>>  'Row' object has no attribute
>> 'unknown'
>>
>> and that happens only when the virtual field is uncommented
>>
>> def get_category_list(post):
>> category_id_list =  db(db.Post_in_category.post_id == post.id).select
>> (db.Post_in_category.category_id)
>> category_list = []
>> for category_id in category_id_list:
>> category_list.append(db.Category[category_id["category_id"]])
>> return category_list
>>
>> db.Post.category_list = Field.Virtual(
>> lambda row: get_category_list(row.Post)
>> )
>>
>>
>>
>> what am I doing wrong?
>>
>>  --
> Resources:
> - http://web2py.com
> - http://web2py.com/book (Documentation)
> - http://github.com/web2py/web2py (Source code)
> - https://code.google.com/p/web2py/issues/list (Report Issues)
> ---
> You received this message because you are subscribed to a topic in the
> Google Groups "web2py-users" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/web2py/tTWYuhhGr8I/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to
> web2py+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.
>



-- 
Eduardo Cruz - 829 775 4605

-- 
Resources:
- http://web2py.com
- http://web2py.com/book (Documentation)
- http://github.com/web2py/web2py (Source code)
- https://code.google.com/p/web2py/issues/list (Report Issues)
--- 
You received this message because you are subscribed to the Google Groups 
"web2py-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to web2py+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[web2py] Re: Virtual Field and sqlform.grid

[Tim Richardson]

What version of web2py?

On Thursday, November 21, 2013 5:26:23 AM UTC+11, Eduardo Cruz wrote:
>
> Whe I use a table that has some a virtual field sqlform.grid does not works
>
>  'Row' object has no attribute 'unknown'
>
> and that happens only when the virtual field is uncommented
>
> def get_category_list(post):
> category_id_list =  db(db.Post_in_category.post_id == post.id).select(
> db.Post_in_category.category_id)
> category_list = []
> for category_id in category_id_list:
> category_list.append(db.Category[category_id["category_id"]])
> return category_list
>
> db.Post.category_list = Field.Virtual(
> lambda row: get_category_list(row.Post)
> )
>
>
>
> what am I doing wrong?
>
>

-- 
Resources:
- http://web2py.com
- http://web2py.com/book (Documentation)
- http://github.com/web2py/web2py (Source code)
- https://code.google.com/p/web2py/issues/list (Report Issues)
--- 
You received this message because you are subscribed to the Google Groups 
"web2py-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to web2py+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[web2py] Re: Making the DAL orders-of-magnitude faster...

[Anthony]

Can you figure out why .select(processor=...) is slower than .executesql()? 
At least for iterating over the rows, they appear to do the same thing. Is 
there just a small fixed difference between the two that remains nearly the 
same no matter how long the overall processing takes?

Also, note that your .executesql() testing times should include the 
following code in order to be comparable with the .select() examples:

selection = [db[tableName].ALL for tableName in tableNames]
sql = db(query)._select(*selection)
sql_colnames = sql.split(' FROM')[0][7:].split(', ')

Those lines are used only by the .executesql() examples, and should 
therefore not be excluded from the timings.

But this is using just simple string-based fields... It goes way-up from 
> there when you introduce other data-types.
>

Of course, when you're selecting other data types from the database, you 
typically need them to be parsed anyway. If we implement some kind of 
faster parsing functionality, we should probably enable options to specify 
which data types to parse. A result set with zero parsing of any data types 
may be very fast, but then very inconvenient to work with (and may require 
manual parsing, eliminating the initial speed benefits).

Anthony

-- 
Resources:
- http://web2py.com
- http://web2py.com/book (Documentation)
- http://github.com/web2py/web2py (Source code)
- https://code.google.com/p/web2py/issues/list (Report Issues)
--- 
You received this message because you are subscribed to the Google Groups 
"web2py-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to web2py+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[web2py] Re: Basic routes.py question

[lesssugar]

Very useul post, Jonathan. And the router works great, thanks.

On Wednesday, November 20, 2013 3:20:22 PM UTC+1, lesssugar wrote:
>
> Just let my understand it.
>
> I created a basic routes.py file in my web2py/ folder. The whole content 
> is:
>
> default_application = 'myapp'# ordinarily set in base routes.py
> default_controller = 'default'  # ordinarily set in app-specific routes.py
> default_function = 'index'  # ordinarily set in app-specific routes.py
>
>
> When I go to mydomainname.com - default_application works fine.
>
> But when I go directly to: mydomainname.com/myapp/users/index - the URL 
> remains unchanged.
> Shouldn't it be rewritten to: mydomainname.com/users ?
>
> I mean, the default_application and default_function are set. Do I also 
> need to specify routes_in and _out for this?
>

-- 
Resources:
- http://web2py.com
- http://web2py.com/book (Documentation)
- http://github.com/web2py/web2py (Source code)
- https://code.google.com/p/web2py/issues/list (Report Issues)
--- 
You received this message because you are subscribed to the Google Groups 
"web2py-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to web2py+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[web2py] Virtual Field and sqlform.grid

[Eduardo Cruz]

Whe I use a table that has some a virtual field sqlform.grid does not works

 'Row' object has no attribute 'unknown'

and that happens only when the virtual field is uncommented

def get_category_list(post):
category_id_list =  db(db.Post_in_category.post_id == post.id).select(db
.Post_in_category.category_id)
category_list = []
for category_id in category_id_list:
category_list.append(db.Category[category_id["category_id"]])
return category_list

db.Post.category_list = Field.Virtual(
lambda row: get_category_list(row.Post)
)



what am I doing wrong?

-- 
Resources:
- http://web2py.com
- http://web2py.com/book (Documentation)
- http://github.com/web2py/web2py (Source code)
- https://code.google.com/p/web2py/issues/list (Report Issues)
--- 
You received this message because you are subscribed to the Google Groups 
"web2py-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to web2py+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Re: [web2py] How can I view the uploaded file name in grid instead of file?

[Anthony]

Doesn't look like the grid will let you do it (we should probably update 
the code to make this more flexible). I think your alternative would be to 
suppress the display of the file field and add your own additional column 
via the "links" argument.

Anthony

On Tuesday, November 19, 2013 11:47:02 PM UTC-5, Sarbjit wrote:
>
> Its not so clear from Niphold answer's on how this can be achieved.
>
> On Tuesday, November 19, 2013 9:52:56 PM UTC+5:30, Richard wrote:
>>
>> See Niphold answer here : 
>> https://groups.google.com/d/msg/web2py/-ErAU2GkTUQ/ddAV4WEYNyQJ
>>
>>
>> On Tue, Nov 19, 2013 at 5:50 AM, Sarbjit  wrote:
>>
>>> With the below models and controllers, In the view corresponding to 
>>> Mytable2, I am getting the uploaded file name as "file". How can I show the 
>>> actual file name that was being uploaded instead of "file"
>>>
>>> *Model:*
>>>
>>> db.define_table('Mytable1', Field('Mytable1_field1'),
>>> Field('Mytable1_field2','text')
>>> )
>>> 
>>> db.define_table('Mytable2', Field('Mytable2_files', 'upload'),
>>>  Field('Mytable2_ref' , 'reference 
>>> Mytable1'))
>>>
>>> db.Mytable2.Mytable2_ref.requires = IS_IN_DB(db, db.Mytable1.id, 
>>> '%(Mytable1_field1)s')
>>>
>>>
>>> *Controller:*
>>>
>>> def viewdata():
>>> grid = 
>>> SQLFORM.smartgrid(db.Mytable1,csv=False,linked_tables=['Mytable2'],)
>>> return dict(grid=grid) 
>>>
>>> -- 
>>> Resources:
>>> - http://web2py.com
>>> - http://web2py.com/book (Documentation)
>>> - http://github.com/web2py/web2py (Source code)
>>> - https://code.google.com/p/web2py/issues/list (Report Issues)
>>> --- 
>>> You received this message because you are subscribed to the Google 
>>> Groups "web2py-users" group.
>>> To unsubscribe from this group and stop receiving emails from it, send 
>>> an email to web2py+un...@googlegroups.com.
>>> For more options, visit https://groups.google.com/groups/opt_out.
>>>
>>
>>

-- 
Resources:
- http://web2py.com
- http://web2py.com/book (Documentation)
- http://github.com/web2py/web2py (Source code)
- https://code.google.com/p/web2py/issues/list (Report Issues)
--- 
You received this message because you are subscribed to the Google Groups 
"web2py-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to web2py+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[web2py] Re: Making the DAL orders-of-magnitude faster...

[Arnon Marcus]

Everyone:

You may play with the numbers at the top in this version of the code - 
everything is now customizable for generating any number of chained-joined 
tables with any number of fields and inserting any number of records.

I find that the more joins you have, the more overhead you'll pay when 
parsing to a Rows object with regular .select().
It can be anywhere from a x15 speed-bump with just 2 tables, up-to x44 with 
an extreme-case of 50 tables.
Usually, it would hover around ~x20 or so - in real-world use-cases.
But this is using just simple string-based fields... It goes way-up from 
there when you introduce other data-types.
As evident with our real-world example of almost x100 penalty for regular 
.select() in a simple 2-table join of about 15 columns.

Caching gains maybe 5% improvement...

I also found the .select(processor=<>) approach to be anywhere from 10% 
slower to x6 slower than my .quickSelect() approach for the same queries. 
The gap get's *worse *for more *simple *the use-case:
i.e:
C:\Python26\python.exe 
F:/Experimental/web2py_272/web2py/test_executesql_0.3.py
Preparing
Times are for 100 iterations, into 2 tables, of querying for 1 results, 
each with 3 values
started bench
speed_executesql() total=0.002431 avg=0.24
speed_normal_rows() total=0.028979 avg=0.000290
speed_rows_with_cacheable() total=0.029703 avg=0.000297
speed_executesql_with_colnames() total=0.002551 avg=0.26
speed_rows_with_simple_processor() total=0.015669 avg=0.000157

Process finished with exit code 0

speed_rows_with_simple_processor() is x6 times slower than 
speed_executesql_with_colnames()

This is when configured as such:
import timeit
from gluon.dal import DAL, Field


tableCount = 2
fieldCount = 1
recordCount = 1
times = 100


##prepare a large dict
...

Granted, this is hardly an issue, though...
But there is a gradient between the more simple and the more complex.

Try it out...

-- 
Resources:
- http://web2py.com
- http://web2py.com/book (Documentation)
- http://github.com/web2py/web2py (Source code)
- https://code.google.com/p/web2py/issues/list (Report Issues)
--- 
You received this message because you are subscribed to the Google Groups 
"web2py-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to web2py+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
import timeit
from gluon.dal import DAL, Field

tableCount = 5
fieldCount = 10
recordCount = 1000
times = 100

##prepare a large dict
print 'Preparing'

db = DAL('sqlite:memory')

tableNames = [('tb_'+str(i)) for i in range(tableCount)]
fieldNames = [('fl_'+str(i)) for i in range(fieldCount)]
#load 1k records into a test table

for t in range(tableCount):
fields = [Field(fieldName) for fieldName in fieldNames]
if t:
fields.append(Field(tableNames[t-1], db[tableNames[t-1]]))
db.define_table(tableNames[t], *fields)
db.commit()

for t in range(tableCount):
db(db[tableNames[t]].id>0).delete()
db.commit()

for r in range(recordCount):
ids = []
for t in range(tableCount):
tableName = tableNames[t]
values = dict([(fieldName,('%s_%s'%(tableName,fieldName))) for fieldName in fieldNames])
newId = db[tableName].insert(**values)
ids.append(newId)
for t in range(tableCount-1):
db[tableNames[t+1]][ids[t+1]].update_record(**{tableNames[t]:ids[t]})

db.commit()

query = (db[tableNames[1]][tableNames[0]]==db[tableNames[0]].id)
for t in range(1, (tableCount-1)):
query = query & (db[tableNames[t+1]][tableNames[t]]==db[tableNames[t]].id)

selection = [db[tableName].ALL for tableName in tableNames]
sql = db(query)._select(*selection)
sql_colnames = sql.split(' FROM')[0][7:].split(', ')

print("Times are for %i iterations, into %i tables, of querying for %i results, each with %i values" % (times,tableCount,recordCount,((fieldCount*tableCount)+(tableCount-1

def bench(cmd, imprt):
t = timeit.Timer(cmd, imprt)
s = t.timeit(number=times)
print("%s total=%02f avg=%02f" % (cmd, s, (s/times)))
return s

def myprocessor(rows, fields, colnames, blob_decode=True, cacheable = False):
return [dict(zip(colnames,row)) for row in rows]

def speed_executesql():
rtn = db.executesql(sql, as_dict=True)

def speed_executesql_with_colnames():
rtn = db.executesql(sql, colnames=sql_colnames, as_dict=True)

def speed_normal_rows():
rtn = db(query).select()

def speed_rows_with_cacheable():
rtn = db(query).select(cacheable=True)

def speed_rows_with_simple_processor():
rtn = db(query).select(processor=myprocessor)

def full_bench():
print 'started bench'
b1 = bench('speed_executesql()', 'from __main__ import speed_executesql')
b2 = bench('speed_normal_rows()', 'from __main__ import speed_normal_rows')
b3 = bench('speed_rows_with_cacheable()', 'from __main__ import speed_rows_with_cacheable')
b4 = bench('speed_executesql_with_colnames()', 'from __mai

[web2py] Re: Making the DAL orders-of-magnitude faster...

[Arnon Marcus]

I have elaborated the test-script - this time, with a customizable-number 
of chained-tables.
I went all the way, to test the hypothesis that it is the JOINs that are 
the culprit.
Lets see how we do with a *single *record, joining 50 tables of *1 field*each + 
*foreign-key-field 
*(except the first clumn): 

C:\Python26\python.exe F:/Experimental/web2py_272/web2py/test_executesql_0.
3.py
Preparing
Times are for 100 iterations, into 50 tables, of querying for 1 results,each 
with 99 values
started bench
speed_executesql() total=0.019787 avg=0.000198
speed_normal_rows() total=0.731611 avg=0.007316
speed_rows_with_cacheable() total=0.669495 avg=0.006695
speed_executesql_with_colnames() total=0.016620 avg=0.000166
speed_rows_with_simple_processor() total=0.364101 avg=0.003641


Process finished with exit code 0



Again - a x44 times increase in performance - quite consistent.

Now let's see a more modest and realistic example - 5 tables of 10 fields 
each - 1k records query:

C:\Python26\python.exe F:/Experimental/web2py_272/web2py/test_executesql_0.
3.py
Preparing
Times are for 100 iterations, into 5 tables, of querying for 1000 results,each 
with 54 values
started bench
speed_executesql() total=2.619404 avg=0.026194
speed_normal_rows() total=52.479113 avg=0.524791
speed_rows_with_cacheable() total=48.437764 avg=0.484378
speed_executesql_with_colnames() total=2.789842 avg=0.027898
speed_rows_with_simple_processor() total=2.865223 avg=0.028652


This is x19 times speed-improvement between speed_executesql_with_colnames
 and speed_normal_rows
This is x17 times speed-improvement between ..simple_processor and ..
rows_with_cacheable

Here is the file:

On Wednesday, November 20, 2013 1:47:47 PM UTC+2, Niphlod wrote:
>
>
>
> On Tuesday, November 19, 2013 11:12:52 PM UTC+1, Arnon Marcus wrote:
>>
>> 10x Niphlod, I'll check that tomorrow...
>>
>> I can provide any code you like - this is solid and consistent.
>>
>
> Exactly what I'm asking. a self-contained testbase to reproduce (including 
> data, models and such).
>  
>
>>
>> The interesting (and somewhat disappointing) factoid I got from your 
>> results, was that PyPy was just as "slow" in un-parsed mode... ;)
>>
>
> pypy isn't a silver bullet for everything. 
>
> BTW, lots of the time is spent actually interacting with SQLite, so the 
> fact that a pure python library on pypy catches up with the C library 
> shipped with python is quite itself the proof that pypy is a raally 
> good project.
>
> BTW2: look at the "with_simple_processor" code. It does what your patch, 
> but in one line, and it works for any select().
>

-- 
Resources:
- http://web2py.com
- http://web2py.com/book (Documentation)
- http://github.com/web2py/web2py (Source code)
- https://code.google.com/p/web2py/issues/list (Report Issues)
--- 
You received this message because you are subscribed to the Google Groups 
"web2py-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to web2py+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
import timeit
from gluon.dal import DAL, Field

tableCount = 50
fieldCount = 1
recordCount = 1
times = 100

##prepare a large dict
print 'Preparing'

db = DAL('sqlite:memory')

tableNames = [('tb_'+str(i)) for i in range(tableCount)]
fieldNames = [('fl_'+str(i)) for i in range(fieldCount)]
#load 1k records into a test table

for t in range(tableCount):
fields = [Field(fieldName) for fieldName in fieldNames]
if t:
fields.append(Field(tableNames[t-1], db[tableNames[t-1]]))
db.define_table(tableNames[t], *fields)
db.commit()

for t in range(tableCount):
db(db[tableNames[t]].id>0).delete()
db.commit()

for r in range(recordCount):
ids = []
for t in range(tableCount):
tableName = tableNames[t]
values = dict([(fieldName,('%s_%s'%(tableName,fieldName))) for fieldName in fieldNames])
newId = db[tableName].insert(**values)
ids.append(newId)
for t in range(tableCount-1):
#row = db(db[tableNames[t+1]].id==ids[t+1]).select(db[tableNames[t+1]].ALL).as_dict()
#row[tableNames[t]] = ids[t]
#row.update()
db[tableNames[t+1]][ids[t+1]].update_record(**{tableNames[t]:ids[t]})

db.commit()

query = (db[tableNames[1]][tableNames[0]]==db[tableNames[0]].id)
for t in range(1, (tableCount-1)):
query = query & (db[tableNames[t+1]][tableNames[t]]==db[tableNames[t]].id)

selection = [db[tableName].ALL for tableName in tableNames]
sql = db(query)._select(*selection)
sql_colnames = sql.split(' FROM')[0][7:].split(', ')

print("Times are for %i iterations, into %i tables, of querying for %i results, each with %i values" % (times,tableCount,recordCount,((fieldCount*tableCount)+(tableCount-1

def bench(cmd, imprt):
t = timeit.Timer(cmd, imprt)
s = t.timeit(number=times)
print("%s total=%02f avg=%02f" % (cmd, s, (s/times)))
return s

def myprocessor(rows, fields, colnames, 

Re: [web2py] Re: avoid out of range error from googlebot

[Kevin Bethke]

that was the answer I was looking for. how do I direct the request to a 404
page or some other function?


On Wed, Nov 20, 2013 at 5:21 PM, Niphlod  wrote:

> shouldn't this question be raised to google than web2py ?
> if this is unsolvable, wrap the entire thing in a try except raising a
> HTTP(404) to avoid logging the error.
>
>
>
> On Wednesday, November 20, 2013 4:23:54 PM UTC+1, BlueShadow wrote:
>>
>> Hi,
>> to increase my speed I have a fast downloadfunction for my images:
>> def fast_download():
>>session.forget(response)
>>cache.action(time_expire=604800)(lambda: 0)()
>># very basic security (only allow fast_download on
>> your_table.upload_field):
>>if not request.args(0).startswith("Images."):
>>return download()
>>
>>#-check which picture we are allowed to display
>>#get the db entry and determine whether it is a small or big
>> pic
>>rows=db(db.Images.thumb == request.args(0)).select()
>>try:
>>rows[0]
>>except:
>>small=0
>>rows=db(db.Images.Image == request.args(0)).select()
>>else:
>>small=1
>>
>>#print rows[0],small
>>row=rows[0]
>>if ...#some stuff to determine which image is shown (not relevant)
>>
>>
>>from PIL import Image
>>img=row.thumb
>>filename = os.path.join(request.folder,'uploads',img)
>>
>>return response.stream(open(filename,'rb'))
>>
>> which works quite good.
>> My Problem is that the google image bot tries quite frequently to access
>> the function directly. which gives an out of range error. (268 in the last
>> 3 weeks)
>> part of one error ticket:
>>
>> File "/home/www-data/web2py/applications/Movie_Alarm/controllers/default.py" 
>> , 
>> line 327, in fast_download
>> row=rows[0]
>>
>>
>> HTTP_FROM:
>> googlebot(at)googlebot.com
>>
>>
>> HTTP_USER_AGENT:
>> Googlebot-Image/1.0
>> REQUEST_URI:
>> /fast_download/Images.thumb.a791255ac1014c88.7466d626e61696c2e6a7067.jpg
>>
>>
>> I tried to put fast_download in the disallowed part of the robots.txt
>> but it seems to have no effect.
>> and I hate having so many errors in my error database :)
>> have you guys any ideas how I can prevent this from happening again?
>>
>  --
> Resources:
> - http://web2py.com
> - http://web2py.com/book (Documentation)
> - http://github.com/web2py/web2py (Source code)
> - https://code.google.com/p/web2py/issues/list (Report Issues)
> ---
> You received this message because you are subscribed to a topic in the
> Google Groups "web2py-users" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/web2py/fCwBUaMzfsg/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to
> web2py+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.
>

-- 
Resources:
- http://web2py.com
- http://web2py.com/book (Documentation)
- http://github.com/web2py/web2py (Source code)
- https://code.google.com/p/web2py/issues/list (Report Issues)
--- 
You received this message because you are subscribed to the Google Groups 
"web2py-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to web2py+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[web2py] Re: Static file caching on GAE

[Niphlod]

This has been already taken care of in the config examples...

https://github.com/web2py/web2py/blob/master/examples/app.example.yaml#L28

On Wednesday, November 20, 2013 1:11:58 PM UTC+1, St. Pirsch wrote:
>
> The book states that a static asset management has been introduced: 
> http://web2py.com/books/default/chapter/29/04/the-core?search=Static+asset+management#Static-asset-management
> As I understand it, static files are not cached in the browser unless a 
> dynamic versioning path is introduced. Thankfully, the book provides a 
> description to make apache and Nginx handle the versioning path. I am not 
> sure though if and how to proceed running the app on GAE. My guess is to 
> place something similar in the gaehandler-file.  I am trying to get images 
> cached Would be great to get a hint. 
> Thanks, Stephan
>

-- 
Resources:
- http://web2py.com
- http://web2py.com/book (Documentation)
- http://github.com/web2py/web2py (Source code)
- https://code.google.com/p/web2py/issues/list (Report Issues)
--- 
You received this message because you are subscribed to the Google Groups 
"web2py-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to web2py+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[web2py] Re: Web2py freezing on live deployment!

[Niphlod]

On Wednesday, November 20, 2013 3:56:51 PM UTC+1, Andrew Buchan wrote:
>
> Further update:
>  
> Apache did finally crash, with the following message:
>  
> "Server ran out of threads to serve requests. Consider raising the 
> ThreadsPerChild setting"
>  
> I've bumped that up to 500, and there are less than that number of users 
> on the system (although I understand there will be more than one connection 
> fired off per user/session).
> But it crashed/became unresponsive again with no error message!
>
Asking for real disasters! Are you sure that there are more than 150 
(default) requests active ?
 

>  Also, I leave sessions alone, but should I be clearing them?
>
Yep, you definitely should (as anyone in production)
 

> Also also, I updated to the latest version of web2py and it seems someone 
> left a print statement in SQLFORM.grid() which prints every row! (Line 2373 
> in sqlhtml.py) I presume that if I were running that as a windows service 
> on rocket it would crash the moment you displayed a grid..
>

known issue. Upgrade to trunk 

-- 
Resources:
- http://web2py.com
- http://web2py.com/book (Documentation)
- http://github.com/web2py/web2py (Source code)
- https://code.google.com/p/web2py/issues/list (Report Issues)
--- 
You received this message because you are subscribed to the Google Groups 
"web2py-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to web2py+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[web2py] Re: avoid out of range error from googlebot

[Niphlod]

shouldn't this question be raised to google than web2py ?
if this is unsolvable, wrap the entire thing in a try except raising a 
HTTP(404) to avoid logging the error.



On Wednesday, November 20, 2013 4:23:54 PM UTC+1, BlueShadow wrote:
>
> Hi,
> to increase my speed I have a fast downloadfunction for my images:
> def fast_download():
>session.forget(response)
>cache.action(time_expire=604800)(lambda: 0)()
># very basic security (only allow fast_download on 
> your_table.upload_field):
>if not request.args(0).startswith("Images."):
>return download()
>
>#-check which picture we are allowed to display
>#get the db entry and determine whether it is a small or big 
> pic
>rows=db(db.Images.thumb == request.args(0)).select()
>try:
>rows[0]
>except:
>small=0
>rows=db(db.Images.Image == request.args(0)).select()
>else:
>small=1
>
>#print rows[0],small
>row=rows[0]
>if ...#some stuff to determine which image is shown (not relevant)
>
>
>from PIL import Image
>img=row.thumb
>filename = os.path.join(request.folder,'uploads',img)
>
>return response.stream(open(filename,'rb'))
>
> which works quite good.
> My Problem is that the google image bot tries quite frequently to access 
> the function directly. which gives an out of range error. (268 in the last 
> 3 weeks)
> part of one error ticket:
>
> File "/home/www-data/web2py/applications/Movie_Alarm/controllers/default.py" 
> , 
> line 327, in fast_download
> row=rows[0]
>
>
> HTTP_FROM:
> googlebot(at)googlebot.com
>
>
> HTTP_USER_AGENT:
> Googlebot-Image/1.0
> REQUEST_URI:
> /fast_download/Images.thumb.a791255ac1014c88.7466d626e61696c2e6a7067.jpg
>
>
> I tried to put fast_download in the disallowed part of the robots.txt
> but it seems to have no effect.
> and I hate having so many errors in my error database :)
> have you guys any ideas how I can prevent this from happening again?
>

-- 
Resources:
- http://web2py.com
- http://web2py.com/book (Documentation)
- http://github.com/web2py/web2py (Source code)
- https://code.google.com/p/web2py/issues/list (Report Issues)
--- 
You received this message because you are subscribed to the Google Groups 
"web2py-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to web2py+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[web2py] Re: Saving sqlform.grid searches

[Niphlod]

look for request.vars.keywords

On Wednesday, November 20, 2013 4:42:35 PM UTC+1, Gael Princivalle wrote:
>
> Hi.
>
> I would like to save all searches made in a sqlform.grid.
> Like that I will be able to know what users search to. That will give me 
> informations for improving results quality.
>
> Is there a way to do that ?
>
> Thanks.
>

-- 
Resources:
- http://web2py.com
- http://web2py.com/book (Documentation)
- http://github.com/web2py/web2py (Source code)
- https://code.google.com/p/web2py/issues/list (Report Issues)
--- 
You received this message because you are subscribed to the Google Groups 
"web2py-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to web2py+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[web2py] Saving sqlform.grid searches

[Gael Princivalle]

Hi.

I would like to save all searches made in a sqlform.grid.
Like that I will be able to know what users search to. That will give me 
informations for improving results quality.

Is there a way to do that ?

Thanks.

-- 
Resources:
- http://web2py.com
- http://web2py.com/book (Documentation)
- http://github.com/web2py/web2py (Source code)
- https://code.google.com/p/web2py/issues/list (Report Issues)
--- 
You received this message because you are subscribed to the Google Groups 
"web2py-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to web2py+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[web2py] Re: Admin crashing - ticket generated but can't see it

[Jim S]

Cleared my browser cache and now it works...

-Jim

On Wednesday, November 20, 2013 8:44:44 AM UTC-6, Jim S wrote:
>
> Just updated to latest Version 2.7.4-stable+timestamp.2013.11.19.17.37.28
>
> Start web2py, click on 'Administrative Interface' and a ticket is 
> generated but can't get into the admin interface to see it.
>
> -Jim
>

-- 
Resources:
- http://web2py.com
- http://web2py.com/book (Documentation)
- http://github.com/web2py/web2py (Source code)
- https://code.google.com/p/web2py/issues/list (Report Issues)
--- 
You received this message because you are subscribed to the Google Groups 
"web2py-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to web2py+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[web2py] avoid out of range error from googlebot

[BlueShadow]

Hi,
to increase my speed I have a fast downloadfunction for my images:
def fast_download():
   session.forget(response)
   cache.action(time_expire=604800)(lambda: 0)()
   # very basic security (only allow fast_download on 
your_table.upload_field):
   if not request.args(0).startswith("Images."):
   return download()
   
   #-check which picture we are allowed to display
   #get the db entry and determine whether it is a small or big pic
   rows=db(db.Images.thumb == request.args(0)).select()
   try:
   rows[0]
   except:
   small=0
   rows=db(db.Images.Image == request.args(0)).select()
   else:
   small=1

   #print rows[0],small
   row=rows[0]
   if ...#some stuff to determine which image is shown (not relevant)


   from PIL import Image
   img=row.thumb
   filename = os.path.join(request.folder,'uploads',img)

   return response.stream(open(filename,'rb'))

which works quite good.
My Problem is that the google image bot tries quite frequently to access 
the function directly. which gives an out of range error. (268 in the last 
3 weeks)
part of one error ticket:

File "/home/www-data/web2py/applications/Movie_Alarm/controllers/default.py" 
, 
line 327, in fast_download
row=rows[0]


HTTP_FROM:
googlebot(at)googlebot.com


HTTP_USER_AGENT:
Googlebot-Image/1.0
REQUEST_URI:
/fast_download/Images.thumb.a791255ac1014c88.7466d626e61696c2e6a7067.jpg


I tried to put fast_download in the disallowed part of the robots.txt
but it seems to have no effect.
and I hate having so many errors in my error database :)
have you guys any ideas how I can prevent this from happening again?

-- 
Resources:
- http://web2py.com
- http://web2py.com/book (Documentation)
- http://github.com/web2py/web2py (Source code)
- https://code.google.com/p/web2py/issues/list (Report Issues)
--- 
You received this message because you are subscribed to the Google Groups 
"web2py-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to web2py+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Re: [web2py] Basic routes.py question

[Jonathan Lundell]

On 20 Nov 2013, at 6:20 AM, lesssugar  wrote:

> Just let my understand it.
> 
> I created a basic routes.py file in my web2py/ folder. The whole content is:
> 
> default_application = 'myapp'# ordinarily set in base routes.py
> default_controller = 'default'  # ordinarily set in app-specific routes.py
> default_function = 'index'  # ordinarily set in app-specific routes.py
> 
> When I go to mydomainname.com - default_application works fine.
> 
> But when I go directly to: mydomainname.com/myapp/users/index - the URL 
> remains unchanged.
> Shouldn't it be rewritten to: mydomainname.com/users ?
> 
> I mean, the default_application and default_function are set. Do I also need 
> to specify routes_in and _out for this?

I know that this is slightly confusing, but bear with me.

If there's no router specified in routes.py (or no routes.py) web2py makes 
assumptions about the default path if elements are missing. The built-in 
default is /init/default/index, and you can override that in routes.py without 
including an actual router (as in your example). When you do this, there is no 
rewriting (because there is no router).

If I understand your goal, you want something like this:

routers = dict(

# base router
BASE=dict(
default_application='myapp',
),

myapp=dict(
default_controller = 'default'
default_function = 'index'
),
)

Because you're not changing the *router* defaults for controller and function, 
you can just write this:

routers = dict(

# base router
BASE=dict(
default_application='myapp',
),
)

...but I prefer to make things explicit.


-- 
Resources:
- http://web2py.com
- http://web2py.com/book (Documentation)
- http://github.com/web2py/web2py (Source code)
- https://code.google.com/p/web2py/issues/list (Report Issues)
--- 
You received this message because you are subscribed to the Google Groups 
"web2py-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to web2py+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[web2py] Re: Web2py freezing on live deployment!

[Andrew Buchan]

Further update:
 
Apache did finally crash, with the following message:
 
"Server ran out of threads to serve requests. Consider raising the 
ThreadsPerChild setting"
 
I've bumped that up to 500, and there are less than that number of users on 
the system (although I understand there will be more than one connection 
fired off per user/session).
But it crashed/became unresponsive again with no error message!
 
I added mod_status to see what's going on, and it seems there are loads of 
requests being sent to which never exit, and I can't tell where these are 
being fired from as they don't indicate which virtualhost they came from 
(and no url mapping). I'm waiting for approval to join apachelounge to ask 
over there, but thought I'd ask here if anyone had any information on how 
web2py uses/creates requests/threads in apache, and how it could be that 
apache becomes unresponsive after a while. I'm not even sure it is running 
out of workerthreads (I thought servers queued requests till threads became 
available specifically so they wouldn't crash under heavy load).

Also, I leave sessions alone, but should I be clearing them?
 
Also also, I updated to the latest version of web2py and it seems someone 
left a print statement in SQLFORM.grid() which prints every row! (Line 2373 
in sqlhtml.py) I presume that if I were running that as a windows service 
on rocket it would crash the moment you displayed a grid...

>
>

-- 
Resources:
- http://web2py.com
- http://web2py.com/book (Documentation)
- http://github.com/web2py/web2py (Source code)
- https://code.google.com/p/web2py/issues/list (Report Issues)
--- 
You received this message because you are subscribed to the Google Groups 
"web2py-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to web2py+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[web2py] Re: Making the DAL orders-of-magnitude faster...

[Arnon Marcus]

Didn't have that bug, but you are correct - the fix makes it safer.

So, you are getting a x15 times performance-boost - interesting...

BTW: For anyone not checking the code, it is now creating 2 tables of 10 
fields each (simple string fields), and another field for the second-table, 
linking it to the first (foreign key).
It then inserts a 1000 records to each table - then queries all records 
with a JOIN query.
Pretty straight-forward.
I suppose if the fields were more varied in type, the gap would be even 
bigger.

I am doing some more tests with a more customizable version of the code:

Let's see what happens when we increase the record-count 10 fold (210k 
values):

Times are for 100 iterations, of querying for 1 results, each with 21values
Preparing
started bench
speed_executesql() total=11.081626 avg=0.110816
speed_normal_rows() total=204.889276 avg=2.048893
speed_rows_with_cacheable() total=190.929696 avg=1.909297
speed_executesql_with_colnames() total=13.822615 avg=0.138226
speed_rows_with_simple_processor() total=13.638187 avg=0.136382

Hmm... Ok, not much different - about x15.9 times faster.

Now, let's increase the column-count 10 times (same total of 210k values):

Times are for 100 iterations, of querying for 1000 results, each with 201values
Preparing
started bench
speed_executesql() total=10.222960 avg=0.102230
speed_normal_rows() total=159.059873 avg=1.590599
speed_rows_with_cacheable() total=157.991245 avg=1.579912
speed_executesql_with_colnames() total=10.30 avg=0.103222
speed_rows_with_simple_processor() total=10.510867 avg=0.105109

Same thing, x15.9 times faster...

Hmm... I'll try to customize the table-count...


On Wednesday, November 20, 2013 3:52:42 PM UTC+2, Paolo Valleri wrote:
>
> Hi, below my results:
> Times are for 100 iterations
> Preparing
> started bench
> speed_executesql() total=0.876057 avg=0.008761
> speed_normal_rows() total=15.076992 avg=0.150770
> speed_rows_with_cacheable() total=13.477355 avg=0.134774
> speed_executesql_with_colnames() total=0.989993 avg=0.009900
> speed_rows_with_simple_processor() total=1.084824 avg=0.010848
>
>
> Mind that your code doesn't work properly, I got the error: 
> "sqlite3.IntegrityError: foreign key constraint failed". Find attached an 
> updated version.
>
> Paolo
>
>
>
> On Wednesday, November 20, 2013 2:32:12 PM UTC+1, Arnon Marcus wrote:
>>
>> Here we go:
>>
>> I edited the original test-code that Niphlod posted, just enough to put 
>> my hypothesis to the test.
>> Here are the results:
>>
>> C:\Python26\python.exe F:/Experimental/web2py_272/web2py/test_executesql.
>> py
>> Times are for 100 iterations
>> Preparing
>> started bench
>> speed_executesql() total=0.920402 avg=0.009204
>> speed_normal_rows() total=20.190159 avg=0.201902
>> speed_rows_with_cacheable() total=18.954632 avg=0.189546
>> speed_executesql_with_colnames() total=1.070893 avg=0.010709
>> speed_rows_with_simple_processor() total=1.118015 avg=0.011180
>>
>>
>> Process finished with exit code 0
>>
>>
>> I left in the addition of 'colname or [...' to the 'if as_dict:' block of 
>> the executesql() in the dal.py, so it actually did the addition of the 
>> colnames that were given - otherwise, in the original test-code of Niphlod, 
>> it was disregarded... (as 'as_dict=True' is passed-in)
>> So, the 'speed_executesql_with_colnames()' is actually doing what it 
>> should in this test-run.
>>
>> Let's see:
>> speed_executesql_with_colnames is about x20 times faster than 
>> speed_normal_rows in this run...
>>
>> You can test it out yourself - here is the code:
>>
>>

-- 
Resources:
- http://web2py.com
- http://web2py.com/book (Documentation)
- http://github.com/web2py/web2py (Source code)
- https://code.google.com/p/web2py/issues/list (Report Issues)
--- 
You received this message because you are subscribed to the Google Groups 
"web2py-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to web2py+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
import timeit
from gluon.dal import DAL, Field

fieldCount = 10
recordCount = 1000
times = 100
print("Times are for %i iterations, of querying for %i results, each with %i values" % (times,recordCount,(fieldCount*2+1)))

##prepare a large dict
print 'Preparing'

db = DAL('sqlite:memory')
fieldNames = [('f_'+str(i)) for i in range(fieldCount)]
#load 1k records into a test table
db.define_table('test_table', *[Field(fieldName) for fieldName in fieldNames])
db.define_table('test_table2', Field('f_test_table', db.test_table), *[Field(fieldName) for fieldName in fieldNames])
db.commit()
db(db.test_table.id > 0).delete()
db(db.test_table2.id > 0).delete()
db.commit()
for a in range(recordCount):
values = dict([(fieldName,'testValue') for fieldName in fieldNames])
foreing_key = db.test_table.insert(**values)
values['f_test_table'] = foreing_key
db.test_table2.insert(**values)
db.commit()

query = 

[web2py] Admin crashing - ticket generated but can't see it

[Jim S]

Just updated to latest Version 2.7.4-stable+timestamp.2013.11.19.17.37.28

Start web2py, click on 'Administrative Interface' and a ticket is generated 
but can't get into the admin interface to see it.

-Jim

-- 
Resources:
- http://web2py.com
- http://web2py.com/book (Documentation)
- http://github.com/web2py/web2py (Source code)
- https://code.google.com/p/web2py/issues/list (Report Issues)
--- 
You received this message because you are subscribed to the Google Groups 
"web2py-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to web2py+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[web2py] Basic routes.py question

[lesssugar]

Just let my understand it.

I created a basic routes.py file in my web2py/ folder. The whole content is:

default_application = 'myapp'# ordinarily set in base routes.py
default_controller = 'default'  # ordinarily set in app-specific routes.py
default_function = 'index'  # ordinarily set in app-specific routes.py


When I go to mydomainname.com - default_application works fine.

But when I go directly to: mydomainname.com/myapp/users/index - the URL 
remains unchanged.
Shouldn't it be rewritten to: mydomainname.com/users ?

I mean, the default_application and default_function are set. Do I also 
need to specify routes_in and _out for this?

-- 
Resources:
- http://web2py.com
- http://web2py.com/book (Documentation)
- http://github.com/web2py/web2py (Source code)
- https://code.google.com/p/web2py/issues/list (Report Issues)
--- 
You received this message because you are subscribed to the Google Groups 
"web2py-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to web2py+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[web2py] Re: Making the DAL orders-of-magnitude faster...

[Paolo Valleri]

Hi, below my results:
Times are for 100 iterations
Preparing
started bench
speed_executesql() total=0.876057 avg=0.008761
speed_normal_rows() total=15.076992 avg=0.150770
speed_rows_with_cacheable() total=13.477355 avg=0.134774
speed_executesql_with_colnames() total=0.989993 avg=0.009900
speed_rows_with_simple_processor() total=1.084824 avg=0.010848


Mind that your code doesn't work properly, I got the error: 
"sqlite3.IntegrityError: foreign key constraint failed". Find attached an 
updated version.

Paolo



On Wednesday, November 20, 2013 2:32:12 PM UTC+1, Arnon Marcus wrote:
>
> Here we go:
>
> I edited the original test-code that Niphlod posted, just enough to put my 
> hypothesis to the test.
> Here are the results:
>
> C:\Python26\python.exe F:/Experimental/web2py_272/web2py/test_executesql.
> py
> Times are for 100 iterations
> Preparing
> started bench
> speed_executesql() total=0.920402 avg=0.009204
> speed_normal_rows() total=20.190159 avg=0.201902
> speed_rows_with_cacheable() total=18.954632 avg=0.189546
> speed_executesql_with_colnames() total=1.070893 avg=0.010709
> speed_rows_with_simple_processor() total=1.118015 avg=0.011180
>
>
> Process finished with exit code 0
>
>
> I left in the addition of 'colname or [...' to the 'if as_dict:' block of 
> the executesql() in the dal.py, so it actually did the addition of the 
> colnames that were given - otherwise, in the original test-code of Niphlod, 
> it was disregarded... (as 'as_dict=True' is passed-in)
> So, the 'speed_executesql_with_colnames()' is actually doing what it 
> should in this test-run.
>
> Let's see:
> speed_executesql_with_colnames is about x20 times faster than 
> speed_normal_rows in this run...
>
> You can test it out yourself - here is the code:
>
>

-- 
Resources:
- http://web2py.com
- http://web2py.com/book (Documentation)
- http://github.com/web2py/web2py (Source code)
- https://code.google.com/p/web2py/issues/list (Report Issues)
--- 
You received this message because you are subscribed to the Google Groups 
"web2py-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to web2py+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
import timeit
from gluon.dal import DAL, Field

times = 100
print("Times are for %i iterations" % times)

##prepare a large dict
print 'Preparing'

db = DAL('sqlite:memory')
fieldNames = [('f_'+str(i)) for i in range(10)]
#load 1k records into a test table
db.define_table('test_table', *[Field(fieldName) for fieldName in fieldNames])
db.define_table('test_table2', Field('f_test_table', db.test_table), *[Field(fieldName) for fieldName in fieldNames])
db.commit()
db(db.test_table.id > 0).delete()
db(db.test_table2.id > 0).delete()
db.commit()
for a in range(1000):
values = dict([(fieldName,'testValue') for fieldName in fieldNames])
foreing_key = db.test_table.insert(**values)
values['f_test_table'] = foreing_key
db.test_table2.insert(**values)
db.commit()

query = (db.test_table2.f_test_table==db.test_table.id)
sql = db(query)._select(db.test_table.ALL,db.test_table2.ALL)
sql_colnames = sql.split(' FROM')[0][7:].split(', ')

def bench(cmd, imprt):
t = timeit.Timer(cmd, imprt)
s = t.timeit(number=times)
print("%s total=%02f avg=%02f" % (cmd, s, (s/times)))
return s

def myprocessor(rows, fields, colnames, blob_decode=True, cacheable = False):
return [dict(zip(colnames,row)) for row in rows]

def speed_executesql():
rtn = db.executesql(sql, as_dict=True)

def speed_executesql_with_colnames():
rtn = db.executesql(sql, colnames=sql_colnames, as_dict=True)

def speed_normal_rows():
rtn = db(query).select()

def speed_rows_with_cacheable():
rtn = db(query).select(cacheable=True)

def speed_rows_with_simple_processor():
rtn = db(query).select(processor=myprocessor)

def full_bench():
print 'started bench'
b1 = bench('speed_executesql()', 'from __main__ import speed_executesql')
b2 = bench('speed_normal_rows()', 'from __main__ import speed_normal_rows')
b3 = bench('speed_rows_with_cacheable()', 'from __main__ import speed_rows_with_cacheable')
b4 = bench('speed_executesql_with_colnames()', 'from __main__ import speed_executesql_with_colnames')
b5 = bench('speed_rows_with_simple_processor()', 'from __main__ import speed_rows_with_simple_processor, myprocessor')

if __name__ == '__main__':
full_bench()

[web2py] Re: Making the DAL orders-of-magnitude faster...

[Arnon Marcus]

Here we go:

I edited the original test-code that Niphlod posted, just enough to put my 
hypothesis to the test.
Here are the results:

C:\Python26\python.exe F:/Experimental/web2py_272/web2py/test_executesql.py
Times are for 100 iterations
Preparing
started bench
speed_executesql() total=0.920402 avg=0.009204
speed_normal_rows() total=20.190159 avg=0.201902
speed_rows_with_cacheable() total=18.954632 avg=0.189546
speed_executesql_with_colnames() total=1.070893 avg=0.010709
speed_rows_with_simple_processor() total=1.118015 avg=0.011180


Process finished with exit code 0


I left in the addition of 'colname or [...' to the 'if as_dict:' block of 
the executesql() in the dal.py, so it actually did the addition of the 
colnames that were given - otherwise, in the original test-code of Niphlod, 
it was disregarded... (as 'as_dict=True' is passed-in)
So, the 'speed_executesql_with_colnames()' is actually doing what it should 
in this test-run.

Let's see:
speed_executesql_with_colnames is about x20 times faster than 
speed_normal_rows in this run...

You can test it out yourself - here is the code:

-- 
Resources:
- http://web2py.com
- http://web2py.com/book (Documentation)
- http://github.com/web2py/web2py (Source code)
- https://code.google.com/p/web2py/issues/list (Report Issues)
--- 
You received this message because you are subscribed to the Google Groups 
"web2py-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to web2py+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
import timeit
from gluon.dal import DAL, Field

times = 100
print("Times are for %i iterations" % times)

##prepare a large dict
print 'Preparing'

db = DAL('sqlite:memory')
fieldNames = [('f_'+str(i)) for i in range(10)]
#load 1k records into a test table
db.define_table('test_table', *[Field(fieldName) for fieldName in fieldNames])
db.define_table('test_table2', Field('f_test_table', db.test_table), *[Field(fieldName) for fieldName in fieldNames])
db.commit()
db(db.test_table.id > 0).delete()
db(db.test_table2.id > 0).delete()
db.commit()
for a in range(1000):
values = dict([(fieldName,'testValue') for fieldName in fieldNames])
db.test_table.insert(**values)
values['f_test_table'] = a
db.test_table2.insert(**values)
db.commit()

query = (db.test_table2.f_test_table==db.test_table.id)
sql = db(query)._select(db.test_table.ALL,db.test_table2.ALL)
sql_colnames = sql.split(' FROM')[0][7:].split(', ')

def bench(cmd, imprt):
t = timeit.Timer(cmd, imprt)
s = t.timeit(number=times)
print("%s total=%02f avg=%02f" % (cmd, s, (s/times)))
return s

def myprocessor(rows, fields, colnames, blob_decode=True, cacheable = False):
return [dict(zip(colnames,row)) for row in rows]

def speed_executesql():
rtn = db.executesql(sql, as_dict=True)

def speed_executesql_with_colnames():
rtn = db.executesql(sql, colnames=sql_colnames, as_dict=True)

def speed_normal_rows():
rtn = db(query).select()

def speed_rows_with_cacheable():
rtn = db(query).select(cacheable=True)

def speed_rows_with_simple_processor():
rtn = db(query).select(processor=myprocessor)

def full_bench():
print 'started bench'
b1 = bench('speed_executesql()', 'from __main__ import speed_executesql')
b2 = bench('speed_normal_rows()', 'from __main__ import speed_normal_rows')
b3 = bench('speed_rows_with_cacheable()', 'from __main__ import speed_rows_with_cacheable')
b4 = bench('speed_executesql_with_colnames()', 'from __main__ import speed_executesql_with_colnames')
b5 = bench('speed_rows_with_simple_processor()', 'from __main__ import speed_rows_with_simple_processor, myprocessor')

if __name__ == '__main__':
full_bench()

[web2py] Static file caching on GAE

[St. Pirsch]

The book states that a static asset management has been introduced: 
http://web2py.com/books/default/chapter/29/04/the-core?search=Static+asset+management#Static-asset-management
As I understand it, static files are not cached in the browser unless a 
dynamic versioning path is introduced. Thankfully, the book provides a 
description to make apache and Nginx handle the versioning path. I am not 
sure though if and how to proceed running the app on GAE. My guess is to 
place something similar in the gaehandler-file.  I am trying to get images 
cached Would be great to get a hint. 
Thanks, Stephan

-- 
Resources:
- http://web2py.com
- http://web2py.com/book (Documentation)
- http://github.com/web2py/web2py (Source code)
- https://code.google.com/p/web2py/issues/list (Report Issues)
--- 
You received this message because you are subscribed to the Google Groups 
"web2py-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to web2py+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Re: [web2py] Re: Installing the jqmobile plugin

[Philip Kilner]

Hi,

On 20/11/13 11:49, Philip Kilner wrote:
> I'll go open a ticket about
> it as it's a bit of a show-stopper to anyone who is a web2py newbie, or
> who is trying to evaluate web2py for mobile.

Ticket at: -

https://code.google.com/p/web2py/issues/detail?id=1788

HTH


-- 

Regards,

PhilK


e: p...@xfr.co.uk - m: 07775 796 747

'work as if you lived in the early days of a better nation'
- alasdair gray


-- 
Resources:
- http://web2py.com
- http://web2py.com/book (Documentation)
- http://github.com/web2py/web2py (Source code)
- https://code.google.com/p/web2py/issues/list (Report Issues)
--- 
You received this message because you are subscribed to the Google Groups 
"web2py-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to web2py+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[web2py] Re: sqlform.grid with div instead of html tables

[Niphlod]

do it in your own views. altering all the markup of the grid and break it 
down to do what you're asking it's just asking for disasters at any new 
release of the grid's code (and lots of cpu wasted for nothing)

On Wednesday, November 20, 2013 3:11:51 AM UTC+1, User wrote:
>
> This sounds like what I'm looking for.  Currently I'm manually creating a 
> list display by having a for loop in the view to spit out  items.  
> However I'm wondering if I can use sqlform.grid to replace the custom  
> list.
>
> 1. Is this expected usage of sqlform.grid or is this more of a hack?
>
> 2. Can someone give a very basic example of what "myformat" function might 
> look like?
> mydisplay = myformat(grid.rows)
> grid.element('.web2py_table', replace=mydisplay)
>
>
>
>
> On Saturday, September 21, 2013 12:34:51 AM UTC-4, ssuresh wrote:
>
>> Hi,
>> I want to use sqlform.grid and all its features, but do not want the 
>> display in a tabular format. Is there a way in which I can change the 
>> display of sqlform.grid to my custom  instead of  .
>>
>>  I tried doing some formatting on grid.rows
>>
>> mydisplay = myformat(grid.rows)
>> grid.element('.web2py_table', replace=mydisplay)
>>
>> but still the resultant output is displayed within a . Is there 
>> any alternatives?
>>
>> Any help would be greatly appreciated.
>>
>> thanx,
>> Suresh
>>
>

-- 
Resources:
- http://web2py.com
- http://web2py.com/book (Documentation)
- http://github.com/web2py/web2py (Source code)
- https://code.google.com/p/web2py/issues/list (Report Issues)
--- 
You received this message because you are subscribed to the Google Groups 
"web2py-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to web2py+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[web2py] Re: Script timed out before returning headers: wsgihandler.py

[Niphlod]

raising the timeout isn't gonna solve the issue that some request are 
taking more than 10s to return...
play with processes, threads and maximum-requests parameters in the apache 
config for mod_wsgi to see what's going on.

On Wednesday, November 20, 2013 8:15:46 AM UTC+1, Johann Spies wrote:
>
> I get this regularly:
>
> [error] [client 146.232.128.186] Script timed out before returning 
> headers: wsgihandler.py
>
> Yesterday I experimented with increasing the timeout from 10s to 50s in 
> /etc/apache2/mods-enabled/reqtimeout.conf and thought it made a difference 
> but this morning I had the same problem again.  
>
> In such a case the only thing that helps is restarting apache2.
>
> Why is that a problem?
>
> Regards
> Johann
> -- 
> Because experiencing your loyal love is better than life itself, 
> my lips will praise you.  (Psalm 63:3)
>  

-- 
Resources:
- http://web2py.com
- http://web2py.com/book (Documentation)
- http://github.com/web2py/web2py (Source code)
- https://code.google.com/p/web2py/issues/list (Report Issues)
--- 
You received this message because you are subscribed to the Google Groups 
"web2py-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to web2py+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[web2py] Re: Making the DAL orders-of-magnitude faster...

[Niphlod]

On Tuesday, November 19, 2013 11:12:52 PM UTC+1, Arnon Marcus wrote:
>
> 10x Niphlod, I'll check that tomorrow...
>
> I can provide any code you like - this is solid and consistent.
>

Exactly what I'm asking. a self-contained testbase to reproduce (including 
data, models and such).
 

>
> The interesting (and somewhat disappointing) factoid I got from your 
> results, was that PyPy was just as "slow" in un-parsed mode... ;)
>

pypy isn't a silver bullet for everything. 

BTW, lots of the time is spent actually interacting with SQLite, so the 
fact that a pure python library on pypy catches up with the C library 
shipped with python is quite itself the proof that pypy is a raally 
good project.

BTW2: look at the "with_simple_processor" code. It does what your patch, 
but in one line, and it works for any select().

-- 
Resources:
- http://web2py.com
- http://web2py.com/book (Documentation)
- http://github.com/web2py/web2py (Source code)
- https://code.google.com/p/web2py/issues/list (Report Issues)
--- 
You received this message because you are subscribed to the Google Groups 
"web2py-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to web2py+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[web2py] Re: Making the DAL orders-of-magnitude faster...

[Paolo Valleri]

How should I test 'your approach'? Would you share a patch?
Please, compare the two tests with the same number of iterations.

Paolo
On Wednesday, November 20, 2013 12:04:35 PM UTC+1, Arnon Marcus wrote:
>
> I get these results:
>
> C:\Python26\python.exe F:/Experimental/web2py_272/web2py/test_executesql.py
>> Times are for 100 iterations
>> Preparing
>> started bench
>> speed_executesql() total=1.132703 avg=0.011327
>> speed_normal_rows() total=5.320367 avg=0.053204
>> speed_rows_with_cacheable() total=5.349665 avg=0.053497
>> speed_executesql_with_colnames() total=1.083485 avg=0.010835
>> speed_rows_with_simple_processor() total=1.202628 avg=0.012026
>> Process finished with exit code 0
>
>
> Pretty similar results...
> Although a little slower overall, same gaps appear.
>
>
> Then I tested it with timeit using your approach:
>
> Times are for 10 iterations
>> Preparing
>> started bench
>> speed_normal_rows() total=47.711369 avg=4.771137
>> speed_executesql_with_colnames() total=0.513249 avg=0.051325
>
>
> That is a *x92.959318 times* (!) faster
> Almost full 2 orders of magnitude.
>
>
> On Tuesday, November 19, 2013 10:45:52 PM UTC+2, Niphlod wrote:
>>
>> if only those announcements came with actual reproducible tests. life 
>> would be easier for us :-P
>> I'd guess your usecase is either 1k rows with 50 columns or a really slow 
>> python server, because I can't reproduce those kind of "advertised 
>> magnitude gains". Just a 4x that is quite expected since Rows carries 
>> around lots of helpers.
>>
>> CPython 2.7.3
>> speed_executesql() total=0.846748 avg=0.008467
>> speed_normal_rows() total=4.214647 avg=0.042146
>> speed_rows_with_cacheable() total=3.705196 avg=0.037052
>> speed_executesql_with_colnames() total=0.844287 avg=0.008443
>> speed_rows_with_simple_processor() total=0.891577 avg=0.008916
>>
>> pypy 1.9.0
>> speed_executesql() total=3.344496 avg=0.033445
>> speed_normal_rows() total=3.832903 avg=0.038329
>> speed_rows_with_cacheable() total=3.472066 avg=0.034721
>> speed_executesql_with_colnames() total=2.808536 avg=0.028085
>> speed_rows_with_simple_processor() total=3.268108 avg=0.032681
>>
>> Attached the test code.
>>
>>

-- 
Resources:
- http://web2py.com
- http://web2py.com/book (Documentation)
- http://github.com/web2py/web2py (Source code)
- https://code.google.com/p/web2py/issues/list (Report Issues)
--- 
You received this message because you are subscribed to the Google Groups 
"web2py-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to web2py+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[web2py] Re: Making the DAL orders-of-magnitude faster...

[Arnon Marcus]

I get these results:

C:\Python26\python.exe F:/Experimental/web2py_272/web2py/test_executesql.py
> Times are for 100 iterations
> Preparing
> started bench
> speed_executesql() total=1.132703 avg=0.011327
> speed_normal_rows() total=5.320367 avg=0.053204
> speed_rows_with_cacheable() total=5.349665 avg=0.053497
> speed_executesql_with_colnames() total=1.083485 avg=0.010835
> speed_rows_with_simple_processor() total=1.202628 avg=0.012026
> Process finished with exit code 0


Pretty similar results...
Although a little slower overall, same gaps appear.


Then I tested it with timeit using your approach:

Times are for 10 iterations
> Preparing
> started bench
> speed_normal_rows() total=47.711369 avg=4.771137
> speed_executesql_with_colnames() total=0.513249 avg=0.051325


That is a *x92.959318 times* (!) faster
Almost full 2 orders of magnitude.


On Tuesday, November 19, 2013 10:45:52 PM UTC+2, Niphlod wrote:
>
> if only those announcements came with actual reproducible tests. life 
> would be easier for us :-P
> I'd guess your usecase is either 1k rows with 50 columns or a really slow 
> python server, because I can't reproduce those kind of "advertised 
> magnitude gains". Just a 4x that is quite expected since Rows carries 
> around lots of helpers.
>
> CPython 2.7.3
> speed_executesql() total=0.846748 avg=0.008467
> speed_normal_rows() total=4.214647 avg=0.042146
> speed_rows_with_cacheable() total=3.705196 avg=0.037052
> speed_executesql_with_colnames() total=0.844287 avg=0.008443
> speed_rows_with_simple_processor() total=0.891577 avg=0.008916
>
> pypy 1.9.0
> speed_executesql() total=3.344496 avg=0.033445
> speed_normal_rows() total=3.832903 avg=0.038329
> speed_rows_with_cacheable() total=3.472066 avg=0.034721
> speed_executesql_with_colnames() total=2.808536 avg=0.028085
> speed_rows_with_simple_processor() total=3.268108 avg=0.032681
>
> Attached the test code.
>
>

-- 
Resources:
- http://web2py.com
- http://web2py.com/book (Documentation)
- http://github.com/web2py/web2py (Source code)
- https://code.google.com/p/web2py/issues/list (Report Issues)
--- 
You received this message because you are subscribed to the Google Groups 
"web2py-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to web2py+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[web2py] Re: update database row without changing contents

[max]

thanks a lot

On Tuesday, November 19, 2013 10:42:52 PM UTC+1, Niphlod wrote:
>
> use callbacks: they'll make your life easier
>
>
> http://web2py.com/books/default/chapter/29/06/the-database-abstraction-layer#before-and-after-callbacks
>
> On Tuesday, November 19, 2013 9:54:00 PM UTC+1, max wrote:
>>
>> I have two seperate tables, which are updated using web2py forms.
>> first table has fields using calculate attbitute of the values from 
>> second table.
>> I want to update the first table, when i change something something in 
>> the second table.
>> is that possible?
>>
>>
>>
>>

-- 
Resources:
- http://web2py.com
- http://web2py.com/book (Documentation)
- http://github.com/web2py/web2py (Source code)
- https://code.google.com/p/web2py/issues/list (Report Issues)
--- 
You received this message because you are subscribed to the Google Groups 
"web2py-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to web2py+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[web2py] Re: Problem with migrate

[R. Osinga]

No I don't use migrate_enabled=False and the app isn't compiled eiter. I 
made sure the database user is a superuser. Should I add some tracing to my 
DAL to see why it fails silently? I don't know about the DAL internals 
enough to know the best place to start my hun. And even more prblematic - 
it is a production server. It doesn't happen in my test environment.

On Tuesday, November 19, 2013 2:46:47 PM UTC+1, Anthony wrote:
>
> Do you happen to have DAL(..., migrate_enabled=False)? Is your app 
> compiled?
>
> On Tuesday, November 19, 2013 7:57:39 AM UTC-5, R. Osinga wrote:
>>
>> Hi all,
>>
>> I have a series of tables declared, but one table does not get created 
>> automatically. This table is new, in the current version of my software and 
>> I expect it to be created automatically.
>> Unfotunatelly, it does not. I checked that *settings.migrate=True* and 
>> even replaced the *migrate=settings.migrate* with *migrate=True*. When I 
>> open any page that was running in the older version of my software, it runs 
>> fine. But when I open one of the new pages - one that needs the new table, 
>> I get the following exception:
>>  ('ERROR', '42P01', 
>> 'relation "t_asset_match" does not exist')
>> This is because it tries to do a select from the database.
>> I used psql, and don't see the table either. Which corresponds to the 
>> exception. But how can it be, that it does not get? Since I had to find a 
>> solution fast, I created the table by hand using SQL
>>
>> Does anyone have a clue what might have happened (or not happened)?
>>
>> Roald
>>
>

-- 
Resources:
- http://web2py.com
- http://web2py.com/book (Documentation)
- http://github.com/web2py/web2py (Source code)
- https://code.google.com/p/web2py/issues/list (Report Issues)
--- 
You received this message because you are subscribed to the Google Groups 
"web2py-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to web2py+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.