I think the problem with cache.disk is fixed. Can you check in trunk?
Massimo
On Sep 23, 8:18 pm, Paul <paulygr...@gmail.com> wrote:
> Tracing through the program execution (using pdb) I found that
> compileapp.py:build_environment was called every time a controller was
> executed. I don't know if this is the intended design (ie each
> controller runs in a fresh environment) or something I am doing/ a bug
> is causing a build_environment to be called each time.
>
> Also, there may only be one disk cache file (cache.shelve), but there
> ends up being many open file id's to the cache lock (cache.lock).
>
> On Sep 23, 7:41 pm, mdipierro <mdipie...@cs.depaul.edu> wrote:
>
> > If this is the case it is a bug. There should be only one cache file
> > per application.
> > That is the point of the cache.
>
> > Massimo
>
> > On Sep 23, 6:19 pm, Paul <paulygr...@gmail.com> wrote:
>
> > > As a temporary work around I just no longer create a CacheOnDisk
> > > object (by commenting out line cache.py:301 self.disk = CacheOnDisk
> > > (request) ) and this works fine (since I am not using any disk
> > > caching).
>
> > > I still contend there is some bug here, and most likely its that each
> > > time a controller is executed a new environment is created (and hence
> > > new cache objects) but they never get deleted, and the file handles
> > > (to cache.lock) don't get closed. This is a bigger problem on OS X
> > > where each process has a default limit of 256 open files, but may not
> > > be apparent on linux, etc which may have a much larger max open file
> > > limit.
>
> > > Paul
>
> > > On Sep 23, 12:28 pm, Yarko Tymciurak <yark...@gmail.com> wrote:
>
> > > > you may want to read through gluon/cache.py to understand how
> > > > CacheOnDisk
> > > > and portalocker work...
>
> > > > On Tue, Sep 22, 2009 at 10:47 PM, Paul <paulygr...@gmail.com> wrote:
>
> > > > > I tried your new streamer method with the same results.
>
> > > > > I am very convinced that the issue is with cache.lock opened in the
> > > > > CacheOnDisk class. I see that each time the environment is set up it
> > > > > opens 'cache/cache.lock' in the CacheOnDisk initializer but I haven't
> > > > > tracked down where it closes the file, or why it is not closing it
> > > > > after calling download.
>
> > > > > Perhaps someone can point me to where cache.lock is normally closed,
> > > > > and I can continue debugging.
>
> > > > > thanks,
>
> > > > > Paul
>
> > > > > On Sep 22, 6:40 pm, mdipierro <mdipie...@cs.depaul.edu> wrote:
> > > > > > response.stream and response.download both use streamer in gluon/
> > > > > > streamer.py and it should close the file on download.
> > > > > > Can you replace this in gluon/streamer.py and see if the problem
> > > > > > goes
> > > > > > away?
>
> > > > > > def streamer(stream, chunk_size = DEFAULT_CHUNK_SIZE, bytes = None):
> > > > > > offset = 0
> > > > > > while bytes == None or offset < bytes:
> > > > > > if bytes != None and bytes - offset < chunk_size:
> > > > > > chunk_size = bytes - offset
> > > > > > data = stream.read(chunk_size)
> > > > > > length = len(data)
> > > > > > offset += length
> > > > > > if not length:
> > > > > > break
> > > > > > else:
> > > > > > if bytes and offset>=bytes:
> > > > > > stream.close()
> > > > > > yield data
> > > > > > if length < chunk_size:
> > > > > > break
> > > > > > stream.close()
>
> > > > > > On Sep 22, 9:39 am, Paul <paulygr...@gmail.com> wrote:
>
> > > > > > > I still have the problem with the new trunk, but I did notice
> > > > > > > that I
> > > > > > > have been using a different implementation of download that the
> > > > > > > default one. mine is essentially
> > > > > > > return response.stream(open(os.path.join
> > > > > > > (request.folder,'uploads',request.args[0]),'rb'))
>
> > > > > > > this was an old implementation of download right? (because I don't
> > > > > > > remember changing this, but I may have). Does response.stream
> > > > > > > close
> > > > > > > the file when it is done? this could be my problem... can I just
> > > > > > > enclose it in a try .. finally block and close the stream in the
> > > > > > > finally block when it is done?
>
> > > > > > > Also, I tried to use the new download method ( return
> > > > > > > response.download
> > > > > > > (request,db) ) but it doesn't seem to allow access to files in the
> > > > > > > uploads directory unless they originated from an 'upload' db
> > > > > > > field. ie
> > > > > > > if I just stick a file (say img1.jpg in the uploads directory) and
> > > > > > > try /default/download/img1.jpg I get a 404 error.
> > > > > > > I can even generate internal errors if I name the file img.1.jpg
> > > > > > > (an
> > > > > > > extra . ), which causes a key error (listed below). Am I using
> > > > > > > 'download' incorrectly? because I do need a method to directly
> > > > > > > access
> > > > > > > files in the uploads directory.
>
> > > > > > > Thanks,
>
> > > > > > > Paul
>
> > > > > > > Traceback (most recent call last):
> > > > > > > File
> > > > > > > "/Users/green/work/CollaborativePhoto/Code/Server/web2py-read-
> > > > > > > only/gluon/restricted.py", line 178, in restricted
> > > > > > > exec ccode in environment
> > > > > > > File
> > > > > > > "/Users/green/work/CollaborativePhoto/Code/Server/web2py-read-
> > > > > > > only/applications/imagetest/controllers/default.py", line 61, in
> > > > > > > <module>
> > > > > > > File
> > > > > > > "/Users/green/work/CollaborativePhoto/Code/Server/web2py-read-
> > > > > > > only/gluon/globals.py", line 102, in <lambda>
> > > > > > > self._caller = lambda f: f()
> > > > > > > File
> > > > > > > "/Users/green/work/CollaborativePhoto/Code/Server/web2py-read-
> > > > > > > only/applications/imagetest/controllers/default.py", line 44, in
> > > > > > > download
> > > > > > > return response.download(request,db)
> > > > > > > File
> > > > > > > "/Users/green/work/CollaborativePhoto/Code/Server/web2py-read-
> > > > > > > only/gluon/globals.py", line 199, in download
> > > > > > > field = db[t][f]
> > > > > > > File
> > > > > > > "/Users/green/work/CollaborativePhoto/Code/Server/web2py-read-
> > > > > > > only/gluon/sql.py", line 531, in __getitem__
> > > > > > > return dict.__getitem__(self, str(key))
> > > > > > > KeyError: 'img'
>
> > > > > > > On Sep 21, 9:10 pm, mdipierro <mdipie...@cs.depaul.edu> wrote:
>
> > > > > > > > Although I cannot reproduce the problem I noticed web2py is
> > > > > > > > opening
> > > > > > > > more tempfiles files that it needs.
> > > > > > > > I fixed this and the change should also make it faster on GET
> > > > > > > > requests.
> > > > > > > > In trunk now.
>
> > > > > > > > Please check if it fixes the problem.
>
> > > > > > > > Massimo
>
> > > > > > > > On Sep 21, 4:34 pm, Paul <paulygr...@gmail.com> wrote:
>
> > > > > > > > > I don't know if you were referring to my problem or
> > > > > > > > > Iceberg's...
> > > > > > > > > regardless, I checked out the latest trunk (rev 1240) and
> > > > > > > > > tried my
> > > > > > > > > application again and was able to reproduce my bug (with
> > > > > essentially
> > > > > > > > > the same error trace as above).
>
> > > > > > > > > the view is essentially this:
>
> > > > > > > > > {{for img in imgs:}}
> > > > > > > > > {{u=URL(r=request,c='default',f='download', args=
> > > > > > > > > img.img_name)}}
> > > > > > > > > {{=A(IMG(_src=u),_href=u)}}
> > > > > > > > > {{pass}}
>
> > > > > > > > > and there are about 10 png's in the 'uploads' directory / in
> > > > > > > > > the
> > > > > > > > > images table. refresh the page ~20 times and it crashes.
>
> > > > > > > > > Paul
>
> > > > > > > > > On Sep 21, 5:06 pm, mdipierro <mdipie...@cs.depaul.edu> wrote:
>
> > > > > > > > > > I cannot reproduce this. Can you try the latest trunk and
> > > > > > > > > > let me
> > > > > know
> > > > > > > > > > if you still get before we spend time in debugging?
>
> > > > > > > > > > On Sep 21, 10:02 am, Iceberg <iceb...@21cn.com> wrote:
>
> > > > > > > > > > > Thanks for your attention, but the new experimental server
> > > > > might not
> > > > > > > > > > > be the suspect this time, because my production server is
> > > > > > > > > > > still
> > > > > using
> > > > > > > > > > > SVN rev1222, which is earlier than the experimental server
> > > > > (sneaky
> > > > > > > > > > > server) 's birth.
>
> > > > > > > > > > > On Sep21, 9:02pm, mdipierro <mdipie...@cs.depaul.edu>
> > > > > > > > > > > wrote:
>
> > > > > > > > > > > > The trunk uses a new experimental server. It still has
> > > > > > > > > > > > come
> > > > > problems.
> > > > > > > > > > > > I will revert to cherrypy for now
>
> > > > > > > > > > > > Massimo
>
> > > > > > > > > > > > On Sep 21, 4:16 am, Iceberg <iceb...@21cn.com> wrote:
>
> > > > > > > > > > > > > On Sep11, 5:13pm, zahariash <karra...@gmail.com>
> > > > > > > > > > > > > wrote:
>
> > > > > > > > > > > > > > On 11 Wrz, 07:13, mdipierro
> > > > > > > > > > > > > > <mdipie...@cs.depaul.edu>
> > > > > wrote:
>
> > > > > > > > > > > > > > > please check if it is fixed. I had to change your
> > > > > > > > > > > > > > > fix
> > > > > because it was
> > > > > > > > > > > > > > > not python 2.4 compliant.
>
> > > > > > > > > > > > > > Revision 1124 works ok. It's fixed.
>
> > > > > > > > > > > > > I never saw web2py breaks in my 17 months experience,
> > > > > > > > > > > > > but I
> > > > > just
> > > > > > > > > > > > > encountered an web2py out-of-service on my Linux
> > > > > > > > > > > > > production
> > > > > server.
> > > > > > > > > > > > > The process is still running, but every request
> > > > > > > > > > > > > results in
> > > > > an error
> > > > > > > > > > > > > ticket page, which are all
> > > > > > > > > > > > > IOError: [Errno 24] Too many open files:
> > > > > '/home/me/web2py_trunk/
> > > > > > > > > > > > > blahblah'
>
> > > > > > > > > > > > > I am using source code version from google svn
> > > > > > > > > > > > > rev1222. Is
> > > > > the leak
> > > > > > > > > > > > > found by Zahariash already fixed in this revision?
>
> > > > > > > > > > > > > After kill and restart, now everything seems ok.
> > > > > > > > > > > > > Anyway, I
> > > > > will keep
> > > > > > > > > > > > > close look to my production server, and come back to
> > > > > > > > > > > > > you if
> > > > > anything
> > > > > > > > > > > > > new happened.
>
> > > > > > > > > > > > > Iceberg.
--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"web2py-users" group.
To post to this group, send email to web2py@googlegroups.com
To unsubscribe from this group, send email to
web2py+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/web2py?hl=en
-~----------~----~----~----~------~----~------~--~---
