Change by Giampaolo Rodola' :
--
nosy: -giampaolo.rodola
___
Python tracker
<https://bugs.python.org/issue26415>
___
___
Python-bugs-list mailing list
Unsub
Change by Giampaolo Rodola' :
--
assignee: -> giampaolo.rodola
resolution: -> fixed
stage: patch review -> resolved
status: open -> closed
___
Python tracker
<https://bugs.python
Giampaolo Rodola' added the comment:
New changeset 19c46a4c96553b2a8390bf8a0e138f2b23e28ed6 by Giampaolo Rodola in
branch 'master':
bpo-33695 shutil.copytree() + os.scandir() cache (#7874)
https://github.com/python/cpython/commit/19c46a4c96553b2a8390bf8a0e138f2b23e28ed6
Change by Giampaolo Rodola' :
--
nosy: +giampaolo.rodola
___
Python tracker
<https://bugs.python.org/issue34655>
___
___
Python-bugs-list mailing list
Unsub
Giampaolo Rodola' added the comment:
Do you mean raising an exception if "count" argument is passed and >
2,147,479,552? In that case I think asyncio's sendfile() should simply do the
math to transmit that many bytes by taking into account that os.sendfile() may
return le
Giampaolo Rodola' added the comment:
@Serhiy: I would like to proceed with this. Do you have further comments? Do
you prefer to bring this up on python-dev for further discussion?
--
___
Python tracker
<https://bugs.python.org/issue33
Change by Giampaolo Rodola' :
--
nosy: +giampaolo.rodola
___
Python tracker
<https://bugs.python.org/issue34934>
___
___
Python-bugs-list mailing list
Unsub
Giampaolo Rodola' added the comment:
PR merged for 3.8 branch. I don't think such a simple change deserves a
backport for previous Python versions. Closing this out as resolved.
--
resolution: -> fixed
stage: patch review -> resolved
status: open -> closed
versions: +P
Giampaolo Rodola' added the comment:
New changeset 7b313971805ca9b53f181f7b97e5376d0b89dc06 by Giampaolo Rodola
(Romuald Brunet) in branch 'master':
bpo-32680 add default "sock" on SMTP objects (#5345)
https://github.com/python/cpython/commit/7b313971805ca9b53f181f7b97e537
Change by Giampaolo Rodola' :
--
versions: -Python 2.7, Python 3.4, Python 3.5, Python 3.6, Python 3.7
___
Python tracker
<https://bugs.python.org/issue34
Giampaolo Rodola' added the comment:
Can you paste a sample output showing how it looks like pre and post patch?
--
nosy: +giampaolo.rodola
___
Python tracker
<https://bugs.python.org/issue34
Change by Giampaolo Rodola' :
--
nosy: +giampaolo.rodola
___
Python tracker
<https://bugs.python.org/issue34605>
___
___
Python-bugs-list mailing list
Unsub
Giampaolo Rodola' added the comment:
If os.name == 'nt' is True on IronPython then why not simply do:
mswindows = sys.platform == "win32" or os.name == "nt"
For the record, both variants are used in different places in cPython source
code. It would nice to figure o
Giampaolo Rodola' added the comment:
I think this feature request is reasonable for 2 reasons:
1) As it stands if dst directory exists copytree() cannot be used. The only
possible workaround is using rmtree(dst) first, but that doesn't seem to make
much sense. The change may look extremely
Change by Giampaolo Rodola' :
--
nosy: +giampaolo.rodola
___
Python tracker
<https://bugs.python.org/issue31940>
___
___
Python-bugs-list mailing list
Unsub
Giampaolo Rodola' added the comment:
It's not clear where the exception originates from. Try to use copy2() instead
of copytree().
--
___
Python tracker
<https://bugs.python.org/issue32
Giampaolo Rodola' added the comment:
What function raises ENOTSUPP exactly (traceback would be welcome)?
--
___
Python tracker
<https://bugs.python.org/issue32
Giampaolo Rodola' added the comment:
Yes, file copy (open() + read() + write()) is of course more expensive than
just "reading" a tree (os.walk(), glob()) or deleting it (rmtree()) and the
"pure file copy" time adds up to the benchmark. And indeed it's not an
coincidenc
Giampaolo Rodola' added the comment:
I agree the provided benchmark on Linux should be more refined. And I'm not
sure if "echo 3 | sudo tee /proc/sys/vm/drop_caches" before running it is
enough honestly.
The main point here is the reduction of stat() syscalls (-38%) and that
Giampaolo Rodola' added the comment:
[ https://bugs.python.org/issue24564#msg246278 ]
> Adding `EINVAL` to the ignored errnos would fix the problem, but might hide
> real failures (I'm not sure about the real failures, but it seems logical).
I think this is an acceptable comp
Change by Giampaolo Rodola' :
--
nosy: +giampaolo.rodola
___
Python tracker
<https://bugs.python.org/issue34260>
___
___
Python-bugs-list mailing list
Unsub
Giampaolo Rodola' added the comment:
Closing as duplicate of #33671.
--
resolution: -> duplicate
stage: patch review -> resolved
status: open -> closed
___
Python tracker
<https://bugs.python.or
Giampaolo Rodola' added the comment:
> PR 8287 seems short to me and it seems like psutils doesn't expose Processor
> Queue Length, so I'm not sure why we are talking about depending on psutils?
I'm not sure if you're strictly interested in getting system load or if CPU
utilization i
Change by Giampaolo Rodola' :
--
nosy: +giampaolo.rodola
___
Python tracker
<https://bugs.python.org/issue34157>
___
___
Python-bugs-list mailing list
Unsub
Giampaolo Rodola' added the comment:
psutil exposes this functionality as "psutil.cpu_percent()":
https://github.com/giampaolo/psutil/blob/ac9dccab6b038835b5d612f92cf4804ec2662c2e/psutil/_psutil_windows.c#L992
https://github.com/giampaolo/psutil/blob/ac9dccab6b038835b5d612f92cf480
Change by Giampaolo Rodola' :
--
nosy: +giampaolo.rodola
___
Python tracker
<https://bugs.python.org/issue30480>
___
___
Python-bugs-list mailing list
Unsub
Giampaolo Rodola' added the comment:
Unless somebody has complaints I think I'm gonna merge this soon.
--
___
Python tracker
<https://bugs.python.org/issue33
Giampaolo Rodola' added the comment:
I suppose there is no way to emulate os.getloadavg() on Windows because that
would necessarily imply using a thread to call the necessary routine (WMI, PDH,
whatever...) every X secs or something, correct
Giampaolo Rodola' added the comment:
> Giampaolo, on your psutil issue you specifically said, "(possibly without
> using WMI)" Is there any particular problem with using WMI?
Performance. In general WMI is (a lot) slower than the Windows API counterpart
(psutil never uses WM
Giampaolo Rodola' added the comment:
> For example, os.stat() accepts both a filename (string) or a file descriptor
> (integer).
Interesting. I never realized that. Other functions involved in file copy are:
os.lchmod
os.chmod
os.listxattr
os.getxattr
I checked and it appears th
Giampaolo Rodola' added the comment:
There is an old ticket for this in psutil with some (possible useful)
references in it:
https://github.com/giampaolo/psutil/issues/604
--
___
Python tracker
<https://bugs.python.org/issue34
Change by Giampaolo Rodola' :
--
nosy: +giampaolo.rodola
___
Python tracker
<https://bugs.python.org/issue34060>
___
___
Python-bugs-list mailing list
Unsub
Change by Giampaolo Rodola' :
--
nosy: +giampaolo.rodola
___
Python tracker
<https://bugs.python.org/issue17180>
___
___
Python-bugs-list mailing list
Unsub
Giampaolo Rodola' added the comment:
All copy* functions and move() are subjects to race conditions (the only
exception is rmtree()). You may solve the race condition in copyfile() but then
you'd have the same problem in copystat() and copymode() which are used by
copy() and copy2
Change by Giampaolo Rodola' :
--
pull_requests: -7403
___
Python tracker
<https://bugs.python.org/issue33671>
___
___
Python-bugs-list mailing list
Unsubscribe:
Change by Giampaolo Rodola' :
--
pull_requests: -7525
___
Python tracker
<https://bugs.python.org/issue33671>
___
___
Python-bugs-list mailing list
Unsubscribe:
Change by Giampaolo Rodola' :
--
pull_requests: -7482
___
Python tracker
<https://bugs.python.org/issue33671>
___
___
Python-bugs-list mailing list
Unsubscribe:
Giampaolo Rodola' added the comment:
This was implemented in https://bugs.python.org/issue30014. Closing as
duplicate.
--
resolution: -> duplicate
stage: needs patch -> resolved
status: open -> closed
versions: +Python 2.7 -Python 3.5
_
Change by Giampaolo Rodola' :
--
nosy: +giampaolo.rodola
___
Python tracker
<https://bugs.python.org/issue33935>
___
___
Python-bugs-list mailing list
Unsub
Change by Giampaolo Rodola' :
--
nosy: +giampaolo.rodola
___
Python tracker
<https://bugs.python.org/issue31861>
___
___
Python-bugs-list mailing list
Unsub
Giampaolo Rodola' added the comment:
> I re-ran benchmarks since shutil code changed after #33695.
Sorry, I meant #33671.
--
___
Python tracker
<https://bugs.python.org/issu
Giampaolo Rodola' added the comment:
PR at: https://github.com/python/cpython/pull/7874.
I re-ran benchmarks since shutil code changed after #33695. Linux went from
+13.5% to 8.8% and Windows went from +17% to 20.7%.
In the PR I explicitly avoided using a context manager around os.scandir
Change by Giampaolo Rodola' :
--
pull_requests: +7481
___
Python tracker
<https://bugs.python.org/issue33695>
___
___
Python-bugs-list mailing list
Unsubscribe:
Change by Giampaolo Rodola' :
--
nosy: +giampaolo.rodola
___
Python tracker
<https://bugs.python.org/issue32568>
___
___
Python-bugs-list mailing list
Unsub
Giampaolo Rodola' added the comment:
New changeset c7f02a965936f197354d7f4e6360f4cfc86817ed by Giampaolo Rodola in
branch 'master':
bpo-33671 / shutil.copyfile: use memoryview() with dynamic size on Windows
(#7681)
https://github.com/python/cpython/commit
Change by Giampaolo Rodola' :
--
pull_requests: +7293
___
Python tracker
<https://bugs.python.org/issue33671>
___
___
Python-bugs-list mailing list
Unsubscribe:
Giampaolo Rodola' added the comment:
For future reference, as per https://github.com/python/cpython/pull/7160
discussion, we decided not to use CopyFileEx on Windows and instead increase
read() buffer size from 16KB to 1MB (Windows only) resulting in a 40.8% speedup
(instead of 48%). Also
Giampaolo Rodola' added the comment:
New changeset 4a172ccc739065bb658c75e8929774a8e94af9e9 by Giampaolo Rodola in
branch 'master':
bpo-33671: efficient zero-copy for shutil.copy* functions (Linux, OSX and Win)
(#7160)
https://github.com/python/cpython/commit
Change by Giampaolo Rodola' :
--
nosy: +giampaolo.rodola
___
Python tracker
<https://bugs.python.org/issue23346>
___
___
Python-bugs-list mailing list
Unsub
Change by Giampaolo Rodola' :
--
nosy: +giampaolo.rodola
___
Python tracker
<https://bugs.python.org/issue27687>
___
___
Python-bugs-list mailing list
Unsub
Change by Giampaolo Rodola' :
--
nosy: +giampaolo.rodola
___
Python tracker
<https://bugs.python.org/issue15100>
___
___
Python-bugs-list mailing list
Unsub
Giampaolo Rodola' added the comment:
If such a thing goes in then copytree() should also be taken into account and
expose the same parameter. That would probably be inconsistent with
copytree()'s copy_function parameter though.
--
nosy: +giampaolo.rodola
Change by Giampaolo Rodola' :
--
nosy: +giampaolo.rodola
___
Python tracker
<https://bugs.python.org/issue10016>
___
___
Python-bugs-list mailing list
Unsub
Giampaolo Rodola' added the comment:
+1. It looks reasonable to ignore FileNotFoundError on os.rmdir(), os.unlink()
and also os.open() and os.scandir().
--
nosy: +giampaolo.rodola
___
Python tracker
<https://bugs.python.org/issue29
Change by Giampaolo Rodola' :
--
nosy: +giampaolo.rodola
___
Python tracker
<https://bugs.python.org/issue30400>
___
___
Python-bugs-list mailing list
Unsub
Giampaolo Rodola' added the comment:
This could be achieved at least on Windows with CopyFileEx [1] and on OSX with
copyfile(3) + COPYFILE_ALL which copies ACLs (but not users/groups). These are
(were, in case of CopyFileEx) exposed in
https://github.com/python/cpython/pull/7160
Change by Giampaolo Rodola' :
--
nosy: +giampaolo.rodola
___
Python tracker
<https://bugs.python.org/issue20849>
___
___
Python-bugs-list mailing list
Unsub
Change by Giampaolo Rodola' :
--
nosy: +giampaolo.rodola
___
Python tracker
<https://bugs.python.org/issue32073>
___
___
Python-bugs-list mailing list
Unsub
Giampaolo Rodola' added the comment:
+1
--
nosy: +giampaolo.rodola
___
Python tracker
<https://bugs.python.org/issue32557>
___
___
Python-bugs-list mailin
Change by Giampaolo Rodola' :
--
nosy: +giampaolo.rodola
___
Python tracker
<https://bugs.python.org/issue32347>
___
___
Python-bugs-list mailing list
Unsub
Change by Giampaolo Rodola' :
--
nosy: +giampaolo.rodola
___
Python tracker
<https://bugs.python.org/issue30235>
___
___
Python-bugs-list mailing list
Unsub
Change by Giampaolo Rodola' :
--
nosy: +giampaolo.rodola
___
Python tracker
<https://bugs.python.org/issue32453>
___
___
Python-bugs-list mailing list
Unsub
Giampaolo Rodola' added the comment:
Gotcha. Thanks for clarifying.
--
___
Python tracker
<https://bugs.python.org/issue32561>
___
___
Python-bugs-list mailin
Giampaolo Rodola' added the comment:
os.preadv() and os.pwritev() are great but to my understanding one essential
piece is still missing in order to effectively do non-blocking file IO and
avoid using a thread pool: being notified when the file fd is
readable/writable. select() and epoll
Giampaolo Rodola' added the comment:
Patch in attachment does the trick.
--
keywords: +patch
Added file: https://bugs.python.org/file47634/clinic.diff
___
Python tracker
<https://bugs.python.org/issue33
Change by Giampaolo Rodola' :
--
nosy: -giampaolo.rodola
___
Python tracker
<https://bugs.python.org/issue33743>
___
___
Python-bugs-list mailing list
Unsub
New submission from Giampaolo Rodola' :
There currently are different functions in Modules/_winapi.c that deals with
paths: CreateFile, CreateJunction, CreateNamedPipe and I'm currently working on
a patch which will add 3 more. For those functions it would be convenient to be
able to use
New submission from Giampaolo Rodola' :
If I use "Tools\clinic\clinic.py Modules\somemodule.c" on Windows argument
clinic will modify the whole file and use Windows line endings ( "\r\n"). It
would be good to use "\n" instead.
--
assignee: larry
compon
Change by Giampaolo Rodola' :
--
nosy: +benhoyt, benjamin.peterson, brett.cannon, ncoghlan, serhiy.storchaka,
stutzbach, tarek, vstinner, yselivanov
___
Python tracker
<https://bugs.python.org/issue33
Change by Giampaolo Rodola' :
--
keywords: +patch
Added file: https://bugs.python.org/file47625/bpo-33695.patch
___
Python tracker
<https://bugs.python.org/issue33
New submission from Giampaolo Rodola' :
Patch in attachment makes shutil.copytree() use os.scandir() and (differently
from #33414) DirEntry instances are passed around so that cached stat()s are
used also from within copy2() and copystat() functions. The number of times the
filesystem gets
Change by Giampaolo Rodola' :
--
nosy: +giampaolo.rodola
___
Python tracker
<https://bugs.python.org/issue33414>
___
___
Python-bugs-list mailing list
Unsub
Giampaolo Rodola' added the comment:
Yes, I tried changing benchmarks order and zero-copy variants are always
faster. As for instantaneous CoW copy, it is debatable. E.g. "cp" command does
not do it by default:
https://unix.stackexchange.com/questions/80351/why-is-cp-re
Change by Giampaolo Rodola' <g.rod...@gmail.com>:
--
nosy: +SilentGhost, StyXman, asvetlov, desbma, facundobatista, gps, josh.r,
martin.panter, ncoghlan, neologix, petr.viktorin, pitrou, python-dev,
r.david.murray, tarek, vstinner, ysel
Giampaolo Rodola' <g.rod...@gmail.com> added the comment:
Closing as duplicate of #33671.
--
stage: resolved -> patch review
___
Python tracker <rep...@bugs.python.org>
<https://bugs.pyt
Giampaolo Rodola' <g.rod...@gmail.com> added the comment:
Closing as duplicate of #33671.
--
resolution: -> duplicate
stage: patch review -> resolved
status: open -> closed
superseder: -> Efficient zero-copy for shutil.copy* functions (L
Giampaolo Rodola' <g.rod...@gmail.com> added the comment:
PR: https://github.com/python/cpython/pull/7160
--
___
Python tracker <rep...@bugs.python.org>
<https://bugs.python
Change by Giampaolo Rodola' <g.rod...@gmail.com>:
--
pull_requests: +6795
___
Python tracker <rep...@bugs.python.org>
<https://bugs.python.org/issue33671>
___
Change by Giampaolo Rodola' <g.rod...@gmail.com>:
--
title: Efficient efficient zero-copy syscalls for shutil.copy* functions
(Linux, OSX and Win) -> Efficient zero-copy for shutil.copy* functions (Linux,
OSX and Win)
___
Python tra
New submission from Giampaolo Rodola' <g.rod...@gmail.com>:
Patch in attachment uses platform specific zero-copy syscalls on Linux and
Solaris (os.sendfile(2)), Windows (CopyFileW) and OSX (fcopyfile(2)) speeding
up shutil.copyfile() and other functions using it (copy(), copy2(), co
Change by Giampaolo Rodola' <g.rod...@gmail.com>:
--
nosy: -giampaolo.rodola
___
Python tracker <rep...@bugs.python.org>
<https://bugs.python
Giampaolo Rodola' <g.rod...@gmail.com> added the comment:
My bad, should have read the doc, sorry. =)
--
resolution: -> not a bug
stage: -> resolved
status: open -> closed
___
Python tracker <rep...@bugs.python.org>
<
Giampaolo Rodola' <g.rod...@gmail.com> added the comment:
I updated the code to rely on sendfile(2) on Linux only, which apparently is
the only one supporting copy between regular files and added a check to fail
immediately in case the filesystem is full.
Can somebody review the
New submission from Giampaolo Rodola' <g.rod...@gmail.com>:
>>> import errno
>>> OSError(errno.EBADF).errno
>>> OSError(errno.EBADF, "yo").errno
9
>>> IOError(errno.EBADF).errno
>>> IOError(errno.EBADF, "yo").errno
9
>&g
Giampaolo Rodola' <g.rod...@gmail.com> added the comment:
Check for availability in configure.ac appears to be broken.
--
___
Python tracker <rep...@bugs.python.org>
<https://bugs.python
Giampaolo Rodola' <g.rod...@gmail.com> added the comment:
Oh! I got confused by the fact that #25063 was rejected due to concerns about
copyfileobj() otherwise I would have commented on your patch which I totally
missed. Yes, this overlaps with #25156 patch but it uses the same
Change by Giampaolo Rodola' <g.rod...@gmail.com>:
--
keywords: +patch
pull_requests: +6737
___
Python tracker <rep...@bugs.python.org>
<https://bugs.python
New submission from Giampaolo Rodola' <g.rod...@gmail.com>:
This is a follow up of #25063 and similar to socket.sendfile() (#17552). It
provides a 20/25% speedup when copying files with shutil.copyfile(),
shutil.copy() and shutil.copy2(). Differently from #25063 this is used for
file
Change by Giampaolo Rodola' <g.rod...@gmail.com>:
--
nosy: +giampaolo.rodola
___
Python tracker <rep...@bugs.python.org>
<https://bugs.python
Change by Giampaolo Rodola' <g.rod...@gmail.com>:
--
nosy: +giampaolo.rodola
___
Python tracker <rep...@bugs.python.org>
<https://bugs.python
Giampaolo Rodola' <g.rod...@gmail.com> added the comment:
FYI, there's psutil.Process().num_handles() which you can use to count handles
before and after subprocess invocation.
--
___
Python tracker <rep...@bugs.python.org>
<https://
Giampaolo Rodola' <g.rod...@gmail.com> added the comment:
This is a great addition. I have a working patch adding sendfile() support for
shutil.copyfileobj() which speeds it up by a factor of 1.3x on Linux. According
to this
https://lists.kernelnewbies.org/pipermail/kernelnewbies/2016
Change by Giampaolo Rodola' <g.rod...@gmail.com>:
--
nosy: +giampaolo.rodola
___
Python tracker <rep...@bugs.python.org>
<https://bugs.python
Change by Giampaolo Rodola' <g.rod...@gmail.com>:
--
nosy: +giampaolo.rodola
___
Python tracker <rep...@bugs.python.org>
<https://bugs.python
Change by Giampaolo Rodola' <g.rod...@gmail.com>:
--
nosy: +giampaolo.rodola
___
Python tracker <rep...@bugs.python.org>
<https://bugs.python
Change by Giampaolo Rodola' <g.rod...@gmail.com>:
--
nosy: +giampaolo.rodola
___
Python tracker <rep...@bugs.python.org>
<https://bugs.python
Giampaolo Rodola' <g.rod...@gmail.com> added the comment:
> For convenience, a handler that retries unlink() and rmdir() could be
> distributed with shutil. For ease of use, it could be enabled by default on
> Windows.
+1 on that. I bumped into this many times over the year
Giampaolo Rodola' <g.rod...@gmail.com> added the comment:
That makes sense to me. Thanks for deciphering this.
--
___
Python tracker <rep...@bugs.python.org>
<https://bugs.python
Giampaolo Rodola' <g.rod...@gmail.com> added the comment:
I created a psutil branch using GetLogicalProcessorInformation() to determine
both logical and physical CPUs:
https://github.com/giampaolo/psutil/pull/1257
According to https://stackoverflow.com/questions/31209256 basically all W
Giampaolo Rodola' <g.rod...@gmail.com> added the comment:
By re-reading
https://bugs.python.org/issue30581 and
https://github.com/giampaolo/psutil/issues/771#issuecomment-264457333 I now
remember why I haven't fixed this issue in psutil yet: because the whole thing
(MS APIs and doc bas
201 - 300 of 1713 matches
Mail list logo
