> > I just confirmed that the bug is even there for FreeBSD 9 RC1 !
> >
> > This is most unfortunate. Seriously.
>
> W00t, that sucks! You could migrate to another BSD (NetBSD) or Linux ... :)
No, thanks;)
> > I am running out of options, since I am willing to make my stuff
> > Python 3 compatible, but Twisted is not yet there.
> >
> > Using the backported new IO on Python 2.7 will not make open()
> automatically use the new IO, will it?
>
> No, the open() function of Python 2.7 will still use the file class which in
> return uses fopen(). You could try to monkey patch the built-in
> open() function. It's mostly API compatible with the current open()
> function:
>
> >>> import io, __builtin__
> >>> __builtin__.open = io.open
>
> It works as long as no codes checks for isinstance(obj, file). If your app
> only
> has to worry about log files, you might want to overwrite the
> _open() method of logging.FileHandler and its subclasses.
>
Thanks! This is probably the most practical option I can go.
I've just tested: the backported new IO on Python 2.7 will indeed
open >32k files on FreeBSD. It also creates the files much faster.
The old, non-monkey-patched version was getting slower and
slower as more files were opened/created ..
There seem to be slight differences though:
Non-monkey patched: I can write to the file a non-Unicode string,
even when the file was opened non-Binary.
With monkey patch: either open the file Binary-mode, or
write Unicode strings ..
I need to see if / what breaks in Twisted.
I can handle my own code .. no problem.
Thanks alot!!
import io, __builtin__
__builtin__.open = io.open
import resource
max = resource.getrlimit(resource.RLIMIT_NOFILE)
cnt = 0
print "maximum FDs", max
max = 33000
fds = []
while cnt < max:
f = open("/tmp/test1/test_%d" % cnt, "wb")
f.write("test")
fds.append(f)
cnt += 1
if cnt % 1000 == 0:
print "opened %d files" % cnt
print "ok, created %d files" % cnt
--
http://mail.python.org/mailman/listinfo/python-list
