Tracing through the program execution (using pdb) I found that
compileapp.py:build_environment was called every time a controller was
executed. I don't know if this is the intended design (ie each
controller runs in a fresh environment) or something I am doing/ a bug
is causing a build_environment to be called each time.
Also, there may only be one disk cache file (cache.shelve), but there
ends up being many open file id's to the cache lock (cache.lock).
On Sep 23, 7:41 pm, mdipierro <mdipie...@cs.depaul.edu> wrote:
> If this is the case it is a bug. There should be only one cache file
> per application.
> That is the point of the cache.
>
> Massimo
>
> On Sep 23, 6:19 pm, Paul <paulygr...@gmail.com> wrote:
>
> > As a temporary work around I just no longer create a CacheOnDisk
> > object (by commenting out line cache.py:301 self.disk = CacheOnDisk
> > (request) ) and this works fine (since I am not using any disk
> > caching).
>
> > I still contend there is some bug here, and most likely its that each
> > time a controller is executed a new environment is created (and hence
> > new cache objects) but they never get deleted, and the file handles
> > (to cache.lock) don't get closed. This is a bigger problem on OS X
> > where each process has a default limit of 256 open files, but may not
> > be apparent on linux, etc which may have a much larger max open file
> > limit.
>
> > Paul
>
> > On Sep 23, 12:28 pm, Yarko Tymciurak <yark...@gmail.com> wrote:
>
> > > you may want to read through gluon/cache.py to understand how CacheOnDisk
> > > and portalocker work...
>
> > > On Tue, Sep 22, 2009 at 10:47 PM, Paul <paulygr...@gmail.com> wrote:
>
> > > > I tried your new streamer method with the same results.
>
> > > > I am very convinced that the issue is with cache.lock opened in the
> > > > CacheOnDisk class. I see that each time the environment is set up it
> > > > opens 'cache/cache.lock' in the CacheOnDisk initializer but I haven't
> > > > tracked down where it closes the file, or why it is not closing it
> > > > after calling download.
>
> > > > Perhaps someone can point me to where cache.lock is normally closed,
> > > > and I can continue debugging.
>
> > > > thanks,
>
> > > > Paul
>
> > > > On Sep 22, 6:40 pm, mdipierro <mdipie...@cs.depaul.edu> wrote:
> > > > > response.stream and response.download both use streamer in gluon/
> > > > > streamer.py and it should close the file on download.
> > > > > Can you replace this in gluon/streamer.py and see if the problem goes
> > > > > away?
>
> > > > > def streamer(stream, chunk_size = DEFAULT_CHUNK_SIZE, bytes = None):
> > > > > offset = 0
> > > > > while bytes == None or offset < bytes:
> > > > > if bytes != None and bytes - offset < chunk_size:
> > > > > chunk_size = bytes - offset
> > > > > data = stream.read(chunk_size)
> > > > > length = len(data)
> > > > > offset += length
> > > > > if not length:
> > > > > break
> > > > > else:
> > > > > if bytes and offset>=bytes:
> > > > > stream.close()
> > > > > yield data
> > > > > if length < chunk_size:
> > > > > break
> > > > > stream.close()
>
> > > > > On Sep 22, 9:39 am, Paul <paulygr...@gmail.com> wrote:
>
> > > > > > I still have the problem with the new trunk, but I did notice that I
> > > > > > have been using a different implementation of download that the
> > > > > > default one. mine is essentially
> > > > > > return response.stream(open(os.path.join
> > > > > > (request.folder,'uploads',request.args[0]),'rb'))
>
> > > > > > this was an old implementation of download right? (because I don't
> > > > > > remember changing this, but I may have). Does response.stream close
> > > > > > the file when it is done? this could be my problem... can I just
> > > > > > enclose it in a try .. finally block and close the stream in the
> > > > > > finally block when it is done?
>
> > > > > > Also, I tried to use the new download method ( return
> > > > > > response.download
> > > > > > (request,db) ) but it doesn't seem to allow access to files in the
> > > > > > uploads directory unless they originated from an 'upload' db field.
> > > > > > ie
> > > > > > if I just stick a file (say img1.jpg in the uploads directory) and
> > > > > > try /default/download/img1.jpg I get a 404 error.
> > > > > > I can even generate internal errors if I name the file img.1.jpg (an
> > > > > > extra . ), which causes a key error (listed below). Am I using
> > > > > > 'download' incorrectly? because I do need a method to directly
> > > > > > access
> > > > > > files in the uploads directory.
>
> > > > > > Thanks,
>
> > > > > > Paul
>
> > > > > > Traceback (most recent call last):
> > > > > > File
> > > > > > "/Users/green/work/CollaborativePhoto/Code/Server/web2py-read-
> > > > > > only/gluon/restricted.py", line 178, in restricted
> > > > > > exec ccode in environment
> > > > > > File
> > > > > > "/Users/green/work/CollaborativePhoto/Code/Server/web2py-read-
> > > > > > only/applications/imagetest/controllers/default.py", line 61, in
> > > > > > <module>
> > > > > > File
> > > > > > "/Users/green/work/CollaborativePhoto/Code/Server/web2py-read-
> > > > > > only/gluon/globals.py", line 102, in <lambda>
> > > > > > self._caller = lambda f: f()
> > > > > > File
> > > > > > "/Users/green/work/CollaborativePhoto/Code/Server/web2py-read-
> > > > > > only/applications/imagetest/controllers/default.py", line 44, in
> > > > > > download
> > > > > > return response.download(request,db)
> > > > > > File
> > > > > > "/Users/green/work/CollaborativePhoto/Code/Server/web2py-read-
> > > > > > only/gluon/globals.py", line 199, in download
> > > > > > field = db[t][f]
> > > > > > File
> > > > > > "/Users/green/work/CollaborativePhoto/Code/Server/web2py-read-
> > > > > > only/gluon/sql.py", line 531, in __getitem__
> > > > > > return dict.__getitem__(self, str(key))
> > > > > > KeyError: 'img'
>
> > > > > > On Sep 21, 9:10 pm, mdipierro <mdipie...@cs.depaul.edu> wrote:
>
> > > > > > > Although I cannot reproduce the problem I noticed web2py is
> > > > > > > opening
> > > > > > > more tempfiles files that it needs.
> > > > > > > I fixed this and the change should also make it faster on GET
> > > > > > > requests.
> > > > > > > In trunk now.
>
> > > > > > > Please check if it fixes the problem.
>
> > > > > > > Massimo
>
> > > > > > > On Sep 21, 4:34 pm, Paul <paulygr...@gmail.com> wrote:
>
> > > > > > > > I don't know if you were referring to my problem or Iceberg's...
> > > > > > > > regardless, I checked out the latest trunk (rev 1240) and tried
> > > > > > > > my
> > > > > > > > application again and was able to reproduce my bug (with
> > > > essentially
> > > > > > > > the same error trace as above).
>
> > > > > > > > the view is essentially this:
>
> > > > > > > > {{for img in imgs:}}
> > > > > > > > {{u=URL(r=request,c='default',f='download', args=
> > > > > > > > img.img_name)}}
> > > > > > > > {{=A(IMG(_src=u),_href=u)}}
> > > > > > > > {{pass}}
>
> > > > > > > > and there are about 10 png's in the 'uploads' directory / in the
> > > > > > > > images table. refresh the page ~20 times and it crashes.
>
> > > > > > > > Paul
>
> > > > > > > > On Sep 21, 5:06 pm, mdipierro <mdipie...@cs.depaul.edu> wrote:
>
> > > > > > > > > I cannot reproduce this. Can you try the latest trunk and let
> > > > > > > > > me
> > > > know
> > > > > > > > > if you still get before we spend time in debugging?
>
> > > > > > > > > On Sep 21, 10:02 am, Iceberg <iceb...@21cn.com> wrote:
>
> > > > > > > > > > Thanks for your attention, but the new experimental server
> > > > might not
> > > > > > > > > > be the suspect this time, because my production server is
> > > > > > > > > > still
> > > > using
> > > > > > > > > > SVN rev1222, which is earlier than the experimental server
> > > > (sneaky
> > > > > > > > > > server) 's birth.
>
> > > > > > > > > > On Sep21, 9:02pm, mdipierro <mdipie...@cs.depaul.edu> wrote:
>
> > > > > > > > > > > The trunk uses a new experimental server. It still has
> > > > > > > > > > > come
> > > > problems.
> > > > > > > > > > > I will revert to cherrypy for now
>
> > > > > > > > > > > Massimo
>
> > > > > > > > > > > On Sep 21, 4:16 am, Iceberg <iceb...@21cn.com> wrote:
>
> > > > > > > > > > > > On Sep11, 5:13pm, zahariash <karra...@gmail.com> wrote:
>
> > > > > > > > > > > > > On 11 Wrz, 07:13, mdipierro <mdipie...@cs.depaul.edu>
> > > > wrote:
>
> > > > > > > > > > > > > > please check if it is fixed. I had to change your
> > > > > > > > > > > > > > fix
> > > > because it was
> > > > > > > > > > > > > > not python 2.4 compliant.
>
> > > > > > > > > > > > > Revision 1124 works ok. It's fixed.
>
> > > > > > > > > > > > I never saw web2py breaks in my 17 months experience,
> > > > > > > > > > > > but I
> > > > just
> > > > > > > > > > > > encountered an web2py out-of-service on my Linux
> > > > > > > > > > > > production
> > > > server.
> > > > > > > > > > > > The process is still running, but every request results
> > > > > > > > > > > > in
> > > > an error
> > > > > > > > > > > > ticket page, which are all
> > > > > > > > > > > > IOError: [Errno 24] Too many open files:
> > > > '/home/me/web2py_trunk/
> > > > > > > > > > > > blahblah'
>
> > > > > > > > > > > > I am using source code version from google svn rev1222.
> > > > > > > > > > > > Is
> > > > the leak
> > > > > > > > > > > > found by Zahariash already fixed in this revision?
>
> > > > > > > > > > > > After kill and restart, now everything seems ok.
> > > > > > > > > > > > Anyway, I
> > > > will keep
> > > > > > > > > > > > close look to my production server, and come back to
> > > > > > > > > > > > you if
> > > > anything
> > > > > > > > > > > > new happened.
>
> > > > > > > > > > > > Iceberg.
--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"web2py-users" group.
To post to this group, send email to web2py@googlegroups.com
To unsubscribe from this group, send email to
web2py+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/web2py?hl=en
-~----------~----~----~----~------~----~------~--~---
