why are the ticker and date column labels lower than High Low Open Close
I am using this to download stock prices. But i find that the column names are not in same row. I am guessing i am missing something. import pandas as pd import numpy as np import datetime import pandas_datareader as pdr py.init_notebook_mode(connected=True) # we download the stock prices for each ticker and then we do a mapping between data and name of the ticker def get(tickers, startdate, enddate): def data(ticker): return (pdr.get_data_yahoo(ticker, start=startdate, end=enddate)) datas = map (data, tickers) return(pd.concat(datas, keys=tickers, names=['ticker', 'date'])) # Define the stocks to download. We'll download of Apple, Microsoft and the S index. tickers = ['AAPL','IBM'] # We would like all available data from 01/01/2000 until 31/12/2018. start_date = datetime.datetime(2016, 1, 1) end_date = datetime.datetime(2019, 12, 31) all_data = get(tickers, start_date, end_date) see screenshot https://i.imgur.com/Cmw20am.png -- https://mail.python.org/mailman/listinfo/python-list
Re: open, close
On Sat, Aug 31, 2019 at 3:43 PM Piet van Oostrum wrote:
>
> There is a difference here with the construct that the OP mentioned:
>
> lines = open("foo.txt").readlines()
>
> In that case the file COULD be closed, but there is no guarantee. It depends
> on garbage collection.
> In your case the file will not be closed as long as there is still a
> reference to it (as in f). When f disappears and all copies of it as well,
> the file COULD be closed similarly.
>
Yes, that is correct. I thought about mentioning the garbage
collection and the extra binding for f, but eventually it does not
change the conclusion.
The garbage collection is just too unpredictable to rely upon in any
scenario where you would deal with many open descriptors in a short
period of time e.g. when opening and processing files in a loop.
It is not easy to generalise from such simple examples. After all, if
all the program does is process one file and shut down afterwards,
this would not be an aspect to worry about.
--
https://mail.python.org/mailman/listinfo/python-list
Re: open, close
> On 1 Sep 2019, at 17:57, MRAB wrote: > > On 2019-09-01 16:46, Barry wrote: >>> On 31 Aug 2019, at 15:41, Manfred Lotz wrote: >>> When you say COULD this sounds like it is a matter of luck. My thinking >>> was that USUALLY the file will be closed after the statement because >>> then the file handle goes out of scope. >> It all depends on the way any python implementation does its garbage >> collection. The file is closed as a side effect of deleting the file object >> to reclaiming the memory of the file object. >> At the start of python 3 people where suprised when files and other >> resources where not released at the same time that python 2 released them. > Is that true? Yes. I recalling having to fix some code, but that was over 10 years ago. I have been using python 3 since the alphas of 3.0. > > I thought that it was because other implementations of Python, such as Jython > and IronPython, don't use reference counting, so files and other resources > aren't necessarily released as soon as an object loses its last reference, > and, moreover, it's not required to do so by the language definition. You can also see delayed rsource freeing with cpython. I think it was an fd leak and lsof showed the files where not closing. Adding with fixed it. But it was so long ago... > > Adding the 'with' statement added determinism. Barry > > See PEP 343 -- The "with" Statement > -- > https://mail.python.org/mailman/listinfo/python-list > -- https://mail.python.org/mailman/listinfo/python-list
Re: open, close
On Mon, Sep 2, 2019 at 3:02 AM MRAB wrote: > > On 2019-09-01 16:46, Barry wrote: > > > > > >> On 31 Aug 2019, at 15:41, Manfred Lotz wrote: > >> > >> When you say COULD this sounds like it is a matter of luck. My thinking > >> was that USUALLY the file will be closed after the statement because > >> then the file handle goes out of scope. > > > > It all depends on the way any python implementation does its garbage > > collection. The file is closed as a side effect of deleting the file object > > to reclaiming the memory of the file object. > > > > At the start of python 3 people where suprised when files and other > > resources where not released at the same time that python 2 released them. > > > Is that true? > > I thought that it was because other implementations of Python, such as > Jython and IronPython, don't use reference counting, so files and other > resources aren't necessarily released as soon as an object loses its > last reference, and, moreover, it's not required to do so by the > language definition. > Yeah. Given that the PEP introducing the 'with' statement dates back well before Python 3, I very much doubt it has anything to do with the 3.0 changes. ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: open, close
On 2019-09-01 16:46, Barry wrote: On 31 Aug 2019, at 15:41, Manfred Lotz wrote: When you say COULD this sounds like it is a matter of luck. My thinking was that USUALLY the file will be closed after the statement because then the file handle goes out of scope. It all depends on the way any python implementation does its garbage collection. The file is closed as a side effect of deleting the file object to reclaiming the memory of the file object. At the start of python 3 people where suprised when files and other resources where not released at the same time that python 2 released them. Is that true? I thought that it was because other implementations of Python, such as Jython and IronPython, don't use reference counting, so files and other resources aren't necessarily released as soon as an object loses its last reference, and, moreover, it's not required to do so by the language definition. Adding the 'with' statement added determinism. See PEP 343 -- The "with" Statement -- https://mail.python.org/mailman/listinfo/python-list
Re: open, close
On Sun, 1 Sep 2019 16:46:44 +0100 Barry wrote: > > On 31 Aug 2019, at 15:41, Manfred Lotz wrote: > > > > When you say COULD this sounds like it is a matter of luck. My > > thinking was that USUALLY the file will be closed after the > > statement because then the file handle goes out of scope. > > It all depends on the way any python implementation does its garbage > collection. The file is closed as a side effect of deleting the file > object to reclaiming the memory of the file object. > > At the start of python 3 people where suprised when files and other > resources where not released at the same time that python 2 released > them. > Thanks for this. Very interesting. -- Manfred -- https://mail.python.org/mailman/listinfo/python-list
Re: open, close
> On 31 Aug 2019, at 15:41, Manfred Lotz wrote: > > When you say COULD this sounds like it is a matter of luck. My thinking > was that USUALLY the file will be closed after the statement because > then the file handle goes out of scope. It all depends on the way any python implementation does its garbage collection. The file is closed as a side effect of deleting the file object to reclaiming the memory of the file object. At the start of python 3 people where suprised when files and other resources where not released at the same time that python 2 released them. Barry -- https://mail.python.org/mailman/listinfo/python-list
Re: open, close
On Sat, 31 Aug 2019 16:37:23 +0200
Peter Otten <__pete...@web.de> wrote:
> Manfred Lotz wrote:
>
> > Hi there,
> > This is a beginner question.
> >
> > I learned that
> >
> > with open("foo.txt") as f:
> > lines = f.readlines()
> >
> > using the with-construct is the recommended way to deal with files
> > making sure that close() always happens.
> >
> > However, I also could do:
> >
> > lines = open("foo.txt").readlines()
> >
> > I have to admit that I'm not sure if in case something bad happens a
> > close() is done implicitly as in the first example.
> >
> >
> > Could I use the latter as a substitute for the with-construct? What
> > are the recommendations of the experts?
>
> Always using
>
> with open(...) ...
>
> is a good habit to get into.
As a Python beginner I started to use with open.. in all cases but was
tempted to use the even shorter one-line way of reading a file.
So, I will continue to use with open... ALWAYS as good habits are very
helpful in life. :-)
> if you need to read all lines of a file
> very often write a helper function:
>
> def readlines(filename):
> with open(filename) as f:
> return f.readlines()
>
> That way you can write
>
> lines = readlines("foo.txt")
>
> which saves even more typing and still closes the file
> deterministically.
>
Good idea.
Thanks to all for your help.
--
Manfred
--
https://mail.python.org/mailman/listinfo/python-list
Re: open, close
On Sat, 31 Aug 2019 15:43:41 +0200
Piet van Oostrum wrote:
> Max Zettlmeißl writes:
>
> > On Sat, Aug 31, 2019 at 2:22 PM Manfred Lotz
> > wrote:
> >>
> >> Could I use the latter as a substitute for the with-construct?
> >>
> >
> > You can't use the second statement as a proper substitute for the
> > first one.
> >
> > With the context manager, it is ensured that the file is closed.
> > It's more or less equal to a "finally" clause which closes the file
> > descriptor.
> > So as long as the Python runtime environment functions properly, it
> > will be closed.
> >
> > Your second statement on the other hand, is more or less equivalent
> > to:
> >
> > f = open("foo.txt")
> > lines = f.readlines()
> >
> > Close won't be called.
>
> There is a difference here with the construct that the OP mentioned:
>
> lines = open("foo.txt").readlines()
>
> In that case the file COULD be closed, but there is no guarantee. It
> depends on garbage collection. In your case the file will not be
> closed as long as there is still a reference to it (as in f). When f
> disappears and all copies of it as well, the file COULD be closed
> similarly.
>
When you say COULD this sounds like it is a matter of luck. My thinking
was that USUALLY the file will be closed after the statement because
then the file handle goes out of scope.
However, I understand now that in contrary to `with...` there is no
guarantee.
> On the other hand, the with statement guarantees that the file will
> be closed, so it is the preferred method.
>
Thanks. So, I take that I'm better off to use 'with...'.
--
Manfred
--
https://mail.python.org/mailman/listinfo/python-list
Re: open, close
Manfred Lotz wrote:
> Hi there,
> This is a beginner question.
>
> I learned that
>
> with open("foo.txt") as f:
> lines = f.readlines()
>
> using the with-construct is the recommended way to deal with files
> making sure that close() always happens.
>
> However, I also could do:
>
> lines = open("foo.txt").readlines()
>
> I have to admit that I'm not sure if in case something bad happens a
> close() is done implicitly as in the first example.
>
>
> Could I use the latter as a substitute for the with-construct? What are
> the recommendations of the experts?
Always using
with open(...) ...
is a good habit to get into. If you need to read all lines of a file very
often write a helper function:
def readlines(filename):
with open(filename) as f:
return f.readlines()
That way you can write
lines = readlines("foo.txt")
which saves even more typing and still closes the file deterministically.
--
https://mail.python.org/mailman/listinfo/python-list
Re: open, close
On 2019-08-31, Manfred Lotz wrote:
> Hi there,
> This is a beginner question.
>
> I learned that
>
> with open("foo.txt") as f:
> lines = f.readlines()
>
> using the with-construct is the recommended way to deal with files
> making sure that close() always happens.
More importantly, it makes sure that close() always happens at a
particular point. It gets closed when the block after with [...]: exits.
> lines = open("foo.txt").readlines()
In that example, close() will very probably happen sometime soon when
the system notices that there are no remaining references to the
file-object. In the general case, it's not wise to depend on that for
large complex programs.
For small throw-away scripts it's fine if you know the program will
terminate soon and that having the file open until then isn't an
issue. For example, if the program is something trivial like this
import sys
lines = open(sys.argv[1]).readlines()
sys.stdout.write("there are %d lines\n" % len(lines))
then it's fine. The file will get closed when the program exits.
--
Grant
--
https://mail.python.org/mailman/listinfo/python-list
Re: open, close
Max Zettlmeißl writes:
> On Sat, Aug 31, 2019 at 2:22 PM Manfred Lotz wrote:
>>
>> Could I use the latter as a substitute for the with-construct?
>>
>
> You can't use the second statement as a proper substitute for the first one.
>
> With the context manager, it is ensured that the file is closed. It's
> more or less equal to a "finally" clause which closes the file
> descriptor.
> So as long as the Python runtime environment functions properly, it
> will be closed.
>
> Your second statement on the other hand, is more or less equivalent to:
>
> f = open("foo.txt")
> lines = f.readlines()
>
> Close won't be called.
There is a difference here with the construct that the OP mentioned:
lines = open("foo.txt").readlines()
In that case the file COULD be closed, but there is no guarantee. It depends on
garbage collection.
In your case the file will not be closed as long as there is still a reference
to it (as in f). When f disappears and all copies of it as well, the file COULD
be closed similarly.
On the other hand, the with statement guarantees that the file will be closed,
so it is the preferred method.
--
Piet van Oostrum
WWW: http://piet.vanoostrum.org/
PGP key: [8DAE142BE17999C4]
--
https://mail.python.org/mailman/listinfo/python-list
Re: open, close
On Sat, Aug 31, 2019 at 2:22 PM Manfred Lotz wrote:
>
> Could I use the latter as a substitute for the with-construct?
>
You can't use the second statement as a proper substitute for the first one.
With the context manager, it is ensured that the file is closed. It's
more or less equal to a "finally" clause which closes the file
descriptor.
So as long as the Python runtime environment functions properly, it
will be closed.
Your second statement on the other hand, is more or less equivalent to:
f = open("foo.txt")
lines = f.readlines()
Close won't be called.
--
https://mail.python.org/mailman/listinfo/python-list
open, close
Hi there,
This is a beginner question.
I learned that
with open("foo.txt") as f:
lines = f.readlines()
using the with-construct is the recommended way to deal with files
making sure that close() always happens.
However, I also could do:
lines = open("foo.txt").readlines()
I have to admit that I'm not sure if in case something bad happens a
close() is done implicitly as in the first example.
Could I use the latter as a substitute for the with-construct? What are
the recommendations of the experts?
--
Thanks, Manfred
--
https://mail.python.org/mailman/listinfo/python-list
Re: detect laptop open/close from within Python?
On 26/04/18 17:37, Skip Montanaro wrote: > I'm going through a bout of RSI problems with my wrists, so klst night > I refreshed an old typing watcher program I wrote a couple decades ago > (there's a comment about Python 1.4 in the code!): > > https://github.com/smontanaro/python-bits/blob/master/watch.py > > It's far from perfect and I'm sure miles behind more modern stuff, but > it does the trick for me. > > I noticed a feww minutes ago that when I opened my laptop it > immediately locked the screen. It would be nice to get notified of > open/close events on the laptop. Any idea if there is a signal I can > catch (Ubuntu 17.10) or a Tk event I can respond to? I think there's a dbus event that you should be able to listen for (but I'll let you do the googling as to what that event is, exactly) -- https://mail.python.org/mailman/listinfo/python-list
Re: detect laptop open/close from within Python?
> Thanks, I do have that. Now to figure out when it changes state... > Unfortunately, the timestamp on the file seems to update continuously, > not just on state changes. Maybe /proc/acpi/wakeup will be of some > use. It appears that I can, at least some of the time. Might need to check it more frequently than I check for human input, but that will likely be good enough for my needs. Thanks again for the pointer. Skip -- https://mail.python.org/mailman/listinfo/python-list
Re: detect laptop open/close from within Python?
> No idea if it'll work on your system, but on my laptop, there's a > pseudo-file /proc/acpi/button/lid/LID0/state that has whether the lid > is open or closed. Thanks, I do have that. Now to figure out when it changes state... Unfortunately, the timestamp on the file seems to update continuously, not just on state changes. Maybe /proc/acpi/wakeup will be of some use. Skip -- https://mail.python.org/mailman/listinfo/python-list
Re: detect laptop open/close from within Python?
On Fri, Apr 27, 2018 at 1:37 AM, Skip Montanaro <skip.montan...@gmail.com> wrote: > I'm going through a bout of RSI problems with my wrists, so klst night > I refreshed an old typing watcher program I wrote a couple decades ago > (there's a comment about Python 1.4 in the code!): > > https://github.com/smontanaro/python-bits/blob/master/watch.py > > It's far from perfect and I'm sure miles behind more modern stuff, but > it does the trick for me. > > I noticed a feww minutes ago that when I opened my laptop it > immediately locked the screen. It would be nice to get notified of > open/close events on the laptop. Any idea if there is a signal I can > catch (Ubuntu 17.10) or a Tk event I can respond to? > No idea if it'll work on your system, but on my laptop, there's a pseudo-file /proc/acpi/button/lid/LID0/state that has whether the lid is open or closed. ChrisA -- https://mail.python.org/mailman/listinfo/python-list
detect laptop open/close from within Python?
I'm going through a bout of RSI problems with my wrists, so klst night I refreshed an old typing watcher program I wrote a couple decades ago (there's a comment about Python 1.4 in the code!): https://github.com/smontanaro/python-bits/blob/master/watch.py It's far from perfect and I'm sure miles behind more modern stuff, but it does the trick for me. I noticed a feww minutes ago that when I opened my laptop it immediately locked the screen. It would be nice to get notified of open/close events on the laptop. Any idea if there is a signal I can catch (Ubuntu 17.10) or a Tk event I can respond to? Thx, Skip -- https://mail.python.org/mailman/listinfo/python-list
[issue16850] Add e mode to open(): close-and-exec (O_CLOEXEC) / O_NOINHERIT
STINNER Victor added the comment: The PEP 433 and its issue #17036 replaces this issue, they are more generic. -- resolution: - fixed status: open - closed superseder: - Implementation of the PEP 433: Easier suppression of file descriptor inheritance ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue16850 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue16850] Add e mode to open(): close-and-exec (O_CLOEXEC) / O_NOINHERIT
Changes by Arfrever Frehtes Taifersar Arahesis arfrever@gmail.com: -- nosy: +Arfrever ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue16850 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue16850] Add e mode to open(): close-and-exec (O_CLOEXEC) / O_NOINHERIT
STINNER Victor added the comment: I created the PEP 433 which proposes a more global change to set the close-on-exec flag. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue16850 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue16850] Add e mode to open(): close-and-exec (O_CLOEXEC) / O_NOINHERIT
STINNER Victor added the comment: There is another way to set close-on-exec flag on a file descriptor: ioctl(fd, FIOCLEX, 0); (and ioctl(fd, FIONCLEX, 0); to unset the flag). It is interesting because it avoids the need to get the flags before setting new flags (old | FD_CLOEXEC): 1 syscall instead of 2. ioctl(fd, FIOCLEX) is available on at least: Linux, Mac OS X, QNX, NetBSD, OpenBSD, FreeBSD. I don't know if it's available in old versions of these operating systems. (It is *not* supported by Interix.) -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue16850 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue16850] Add e mode to open(): close-and-exec (O_CLOEXEC) / O_NOINHERIT
STINNER Victor added the comment: According to the following script, FD_CLOEXEC is available on: * Solaris 8 * HP-UX 11 * Unicos 9.0 * Digital UNIX 4.0 * SunOS 4.1.1_U1 * IRIX 6.5 * Linux 2.0 and Linux 2.4 with glibc 2.2.3 * AIX 4.2 (Versions are tested versions, it doesn't mean that the flag didn't exist before) http://sam.nipl.net/ssh-fast/fsh/fshcompat.py -- Note: On Windows, it's possible to implement fcntl(fd, F_SETFD, FD_CLOEXEC) using: SetHandleInformation(fd, HANDLE_FLAG_INHERIT, 1). See for example: https://github.com/tav/pylibs/blob/master/tornado/win32_support.py#L36 I don't know if it's useful since it's possible to set the flag directly when the file is opened (using O_NOINHERIT). -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue16850 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue16850] Add x mode to open(): close-and-exec (O_CLOEXEC) / O_NOINHERIT
Changes by STINNER Victor victor.stin...@gmail.com: -- title: Atomic open + close-and-exec - Add x mode to open(): close-and-exec (O_CLOEXEC) / O_NOINHERIT ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue16850 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue16850] Add x mode to open(): close-and-exec (O_CLOEXEC) / O_NOINHERIT
STINNER Victor added the comment: Oh, my patch doesn't check fcntl() error code. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue16850 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue16850] Add e mode to open(): close-and-exec (O_CLOEXEC) / O_NOINHERIT
Changes by STINNER Victor victor.stin...@gmail.com: -- title: Add x mode to open(): close-and-exec (O_CLOEXEC) / O_NOINHERIT - Add e mode to open(): close-and-exec (O_CLOEXEC) / O_NOINHERIT ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue16850 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue16850] Add e mode to open(): close-and-exec (O_CLOEXEC) / O_NOINHERIT
STINNER Victor added the comment: Updated patch. I tested it on Linux Fedora (3.6), Linux Debian (3.0), Windows 7, FreeBSD 8.2 (don't support O_CLOEXEC), OpenIndiana (oi_148, SunOS 5.11), OpenBSD 4.9, OpenBSD 5.2, Mac OS X 10.8. test_builtin pass on all these platforms... except OpenBSD 4.9. The test fails on OpenBSD 4.9 for an unknown reason. fcntl(FD_CLOEXEC) + exec() works, but fcntl(FD_CLOSEXEC) + fork() + exec() fails. It looks like an OS bug, I don't see what Python can do for this case :-/ The e mode does correctly set FD_CLOEXEC, but FD_CLOEXEC doesn't work as expected. It's really an OS bug, because test_builtin pass on OpenBSD 5.2. I don't have other OSes to check if another platform doesn't support e mode (a platform where NotImplementedError would be raised). -- Added file: http://bugs.python.org/file28625/open_mode_e-2.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue16850 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue16850] Add e mode to open(): close-and-exec (O_CLOEXEC) / O_NOINHERIT
Changes by STINNER Victor victor.stin...@gmail.com: Removed file: http://bugs.python.org/file28561/open_mode_e.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue16850 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue16850] Atomic open + close-and-exec
Charles-François Natali added the comment: Windows provides O_NOINHERIT (_O_NOINHERIT) flag which has a similar purpose. ... and even then, many Unices don't support it. Yes, but I bet that more and more OSes will support it :-) For example, it looks like O_CLOEXEC is part of the POSIX standard 2008: Hum, OK, but many operating systems don't support it to this day. So if we expose it and the underlying operating system doesn't support it, do you want to fallback to fcntl (which wouldb't be atomic anymore, but let's pretend the GIL protection is enough). Also, I'm not sure exactly if this flag is useful enough to be exposed: there are many flags that can be passed when opening a file: O_DIRECT, O_SYNC, O_NONBLOCK, O_DSYNC... Amusingly, Java exposes some of them (but not O_CLOEXEC): http://docs.oracle.com/javase/7/docs/api/java/nio/file/StandardOpenOption.html -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue16850 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue16850] Atomic open + close-and-exec
STINNER Victor added the comment: So if we expose it and the underlying operating system doesn't support it, do you want to fallback to fcntl (which wouldb't be atomic anymore, but let's pretend the GIL protection is enough). At the beginning, I was convinced that the atomicity was important than the portability. But after having read that even fopen() uses a fallback, it is maybe much more convinient to have a fallback. For example, it would be annoying that a program works on Linux 2.6.23, but not on Linux 2.6.22 whereas the atomicity is not a must-have. Said differently: the manual fallback described in msg178957 now seems annoying to me :-) So let's say that a fallback is preferable to improve the portability, but that open(name, e) would still fail with NotImplementedError if O_CLOEXEC, O_NOINHERIT and fcntl(FD_CLOEXEC) are all missing on a platform. I guess that all platform provide at least one flag/function. You already implemented a similar fallback for subprocess: use pipe2(O_CLOEXEC) if available, or fallback to pipe()+fcntl(FD_CLOEXEC). I'm not sure exactly if this flag is useful enough to be exposed I would like to benefit of the atomicity feature of O_CLOEXEC, without having to implement myself all the tricky things to check if the platform supports it or not. O_CLOEXEC solves for example a race condition in tempfile._mkstemp_inner(): fd = _os.open(file, flags, 0o600) _set_cloexec(fd) -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue16850 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue16850] Atomic open + close-and-exec
Charles-François Natali added the comment: O_CLOEXEC solves for example a race condition in tempfile._mkstemp_inner(): fd = _os.open(file, flags, 0o600) _set_cloexec(fd) Hum... diff --git a/Lib/tempfile.py b/Lib/tempfile.py --- a/Lib/tempfile.py +++ b/Lib/tempfile.py @@ -57,6 +57,8 @@ _allocate_lock = _thread.allocate_lock _text_openflags = _os.O_RDWR | _os.O_CREAT | _os.O_EXCL +if hasattr(_os, 'O_CLOEXEC'): +_text_openflags |= _os.O_CLOEXEC if hasattr(_os, 'O_NOINHERIT'): _text_openflags |= _os.O_NOINHERIT if hasattr(_os, 'O_NOFOLLOW'): We should probably add this to 3.3 and default (IIRC O_CLOEXEC was added to the os module in 3.3). Also, I think O_NOFOLLOW is useless: if the file is created (O_EXCL|O_CREAT), then by definition it's not a symlink (juste check glibc's mkstemp() implementation, and it only passes O_CREAT|O_EXCL). -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue16850 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue16850] Atomic open + close-and-exec
STINNER Victor added the comment: We should probably add this to 3.3 and default (IIRC O_CLOEXEC was added to the os module in 3.3). I created the issue #16860. I just realized that tempfile doesn't use open() but os.open(), so this issue help to fix #16860. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue16850 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue16850] Atomic open + close-and-exec
STINNER Victor added the comment: Here is a work-in-progress patch to test my idea: add e flag to open(). Limitations/TODO: * The unit test doesn't work on Windows (it requires fcntl) * e mode and the opener argument are exclusive: if O_CLOEXEC and O_NOINHERINT are missing, I don't see how the opener can be aware of the e flag. Or should we call fcntl(F_SETFD, flags|FD_CLOEXEC) after the opener? It would be strange: the opener is supposed to be the only one to decide how the file is opened, isn't it? * NotImplementedError is raised if O_CLOEXEC, O_NOINHERIT and fcntl() are missing I only tested my patch on Linux (3.6). -- keywords: +patch Added file: http://bugs.python.org/file28561/open_mode_e.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue16850 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue16850] Atomic open + close-and-exec
New submission from STINNER Victor: Recent version on different operating systems support opening a file with close-on-exec flag set immediatly (atomic). This feature fixes a race condition when the process calls execv() between open() and fcntl() (to set the FD_CLOEXEC flag to the newly opened file). It would be nice to expose this feature in Python. The problem is the find a portable and safe way to expose the feature: neologix is against a best-effort function. For example, Linux kernel older than 2.6.22 simply ignores O_CLOEXEC flag (while the libc may expose it). The feature looks to be supported by at least: * Linux kernel = 2.6.23 * FreeBSD 8+ * Windows: _open(filename, _O_NOINHERIT). Is it supported by Windows XP and older versions? http://msdn.microsoft.com/en-us/library/z0kc8e3z.aspx See also: * Issue #12760 (closed): This issue added an x mode to open() to create a file in exclusive mode * Issue #12103: Document how to use open with os.O_CLOEXEC * Issue #12105: It was proposed to add an e mode to open() for O_CLOEXEC -- components: Interpreter Core messages: 178949 nosy: alexey-smirnov, amaury.forgeotdarc, haypo, neologix, sbt priority: normal severity: normal status: open title: Atomic open + close-and-exec versions: Python 3.4 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue16850 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue16850] Atomic open + close-and-exec
STINNER Victor added the comment: The problem is the find a portable and safe way to expose the feature A solution is to add a e mode to open() which would raise a NotImplementedError if the platform is not known to support this feature. For example, if the OS is linux, we would check if the kernel version is at least 2.6.23, otherwise an exception would be raised. The check (on the OS/version) would be done at the first call the function (if the e mode if used). We already have such behaviour on other functions. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue16850 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue16850] Atomic open + close-and-exec
Christian Heimes added the comment: You could do both: use the O_CLOEXEC flag and do a fcntl() call on POSIX. In my opinion it's enough to document that the x flag may be affected by a race condition issue on some operation systems. -- nosy: +christian.heimes ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue16850 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue16850] Atomic open + close-and-exec
STINNER Victor added the comment: You could do both: use the O_CLOEXEC flag and do a fcntl() call on POSIX This is the best-effort option. It was already discussed and rejected in the issue #12760. We had a similar discussion for the PEP 418 on monotonic clock. The final decision is not only provide time.monotonic() if the OS supports monotonic clock. Using an exception, a developer can develop its own best-effort function: try: fileobj = open(filename, e) except NotImplementedError: fileobj = open(filename, r) # may also fail here if the fcntl module is missing import fcntl flags = fcntl.fcntl(self.socket, fcntl.F_GETFD) fcntl.fcntl(fileobj, fcntl.F_SETFD, flags | fcntl.FD_CLOEXEC) We may expose the best-effort function somewhere else, but not in open() which must be atomic in my opinion. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue16850 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue16850] Atomic open + close-and-exec
Christian Heimes added the comment: Do you mean #12105? I didn't know about the ticket before. How about two options, like 'e' for guaranteed atomic CLOEXEC and 'E' for CLOEXEC with or without atomic ops? It's not much additional work and lowers the burden on the user. It's going to be hard to get a list of platforms and versions where O_CLOEXEC is supported. We should go for a white list approach and only enable it on platforms that are either documented to support it or have been tested. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue16850 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue16850] Atomic open + close-and-exec
Changes by Ross Lagerwall rosslagerw...@gmail.com: -- nosy: +rosslagerwall ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue16850 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue16850] Atomic open + close-and-exec
Antoine Pitrou added the comment: How about two options, like 'e' for guaranteed atomic CLOEXEC and 'E' for CLOEXEC with or without atomic ops? Why would you want that? Best effort is sufficient. Also, I'm not sure why e. -- nosy: +pitrou ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue16850 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue16850] Atomic open + close-and-exec
Charles-François Natali added the comment: I don't comfortable exposing this. The main reason is that this flag is really non-portable. Having open() fail at runtime because the platform doesn't support it looks really wrong to me. And silently ignore it is even worse. The 'x' flag was added because it is useful, on available on all platforms (I mean, even Windows has it). Here's, it's by definition Unix-specific, and even then, many Unices don't support it. So I'm -1. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue16850 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue16850] Atomic open + close-and-exec
STINNER Victor added the comment: The feature looks to be supported by at least: * FreeBSD 8+ Hum, it looks like it was only added to FreeBSD 8.3: http://www.freebsd.org/cgi/man.cgi?query=openapropos=0sektion=0manpath=FreeBSD+8.3-RELEASEarch=defaultformat=html (O_CLOEXEC doesn't appear in FreeBSD 8.2 manual page) -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue16850 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue16850] Atomic open + close-and-exec
STINNER Victor added the comment: Also, I'm not sure why e. The choice of the e letter comes from the GNU version of fopen(). Extract of fopen manual page on Linux: e (since glibc 2.7) Open the file with the O_CLOEXEC flag. See open(2) for more information. Oh, by the way: e mode is a best-effort approach. It uses fcntl() if O_CLOEXEC is not supported by the running kernel. fopen() doesn't check the kernel version, but pass O_CLOEXEC and check (once) if FD_CLOEXEC is set: http://sourceware.org/git/?p=glibc.git;a=blob;f=libio/fileops.c;h=61b61b30d88d0fca9e6b20a2fe62a2406cc027cf;hb=HEAD#l319 It's interesting to know that the glibc chose the best-effort approach. The 'x' flag was added because it is useful, on available on all platforms (I mean, even Windows has it). Here's, it's by definition Unix-specific, ... Windows provides O_NOINHERIT (_O_NOINHERIT) flag which has a similar purpose. ... and even then, many Unices don't support it. Yes, but I bet that more and more OSes will support it :-) For example, it looks like O_CLOEXEC is part of the POSIX standard 2008: http://pubs.opengroup.org/onlinepubs/9699919799/functions/open.html (O_CLOEXEC: If set, the FD_CLOEXEC flag for the new file descriptor shall be set.) The O_CLOEXEC flag of Linux comes from QNX and BeOS which support it since many years (I don't know when, at least since 2004). So O_CLOEXEC is not really specific to Linux. -- For more information of the Linux support, this article of Ulrich Drepper contains many useful information: http://udrepper.livejournal.com/20407.html -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue16850 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
fil open close very slow
hiya all, I have a large script written in python. There is in that big script at a certain point a loop in which I create a lots of simple small text files. Each text file just contain I think 4 or 5 short lines. When I simply run it in IDLE, it runs fast. But when I compile it to binary files, it becomes very slow. I turned on the profiler and discorvered it was slow at the open() and close() calls. I removed those calls to confirm this, and yes my program runs very fast then. Does anyone has an idea or suggestion why this can be possible. I only compile them to .pyc files, which as what I see, when I normally run in IDLe these files are created as well, so I really got no clue. rh. -- http://mail.python.org/mailman/listinfo/python-list
Tix Tree open/close issue
It works. Thanks Rob. Sorin __ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com -- http://mail.python.org/mailman/listinfo/python-list
Tix Tree open/close issue
Hi,
I tried for the first time a Tix Tree, so, if my
question is naive, I apologize upfront.
The following code:
code
from Tix import *
r=Tk()
tr=Tree(r)
tr.subwidget('hlist').add('br1',text='branch 1')
tr.subwidget('hlist').add('br1.b1',text='branch 1-1')
tr.subwidget('hlist').add('br1.b1.b1',text='branch
1-1-1')
tr.subwidget('hlist').add('br1.b1.b2',text='branch
1-1-2')
tr.subwidget('hlist').add('br1.b2',text='branch 1-2')
tr.subwidget('hlist').add('br1.b2.b1',text='branch
1-2-1')
tr.subwidget('hlist').add('br1.b2.b2',text='branch
1-2-2')
r.tk.call(tr,'setmode','br1','close')
r.tk.call(tr,'setmode','br1.b1','open')
r.tk.call(tr,'setmode','br1.b2','open')
#tr.setmode('br1','close')
#tr.setmode('br1.b1','open')
#tr.setmode('br1.b2','open')
tr.grid()
r.mainloop()
/code
will show the whole tree, but will set correctly the
(+) and (-) boxes. (The commented out statements where
in place before direct calls to Tk - same result).
I was hoping to have some branches hidden (the leaves,
in this case).
How should I proceed?
Thanks,
Sorin
__
Do You Yahoo!?
Tired of spam? Yahoo! Mail has the best spam protection around
http://mail.yahoo.com
--
http://mail.python.org/mailman/listinfo/python-list
Re: Tix Tree open/close issue
Sorin Schwimmer wrote in news:mailman.8142.1152816058.27775.python-
[EMAIL PROTECTED] in comp.lang.python:
The following code:
code
from Tix import *
r=Tk()
tr=Tree(r)
tr.subwidget('hlist').add('br1',text='branch 1')
tr.subwidget('hlist').add('br1.b1',text='branch 1-1')
tr.subwidget('hlist').add('br1.b1.b1',text='branch
1-1-1')
tr.subwidget('hlist').add('br1.b1.b2',text='branch
1-1-2')
tr.subwidget('hlist').add('br1.b2',text='branch 1-2')
tr.subwidget('hlist').add('br1.b2.b1',text='branch
1-2-1')
tr.subwidget('hlist').add('br1.b2.b2',text='branch
1-2-2')
r.tk.call(tr,'setmode','br1','close')
r.tk.call(tr,'setmode','br1.b1','open')
r.tk.call(tr,'setmode','br1.b2','open')
#tr.setmode('br1','close')
#tr.setmode('br1.b1','open')
#tr.setmode('br1.b2','open')
Try instead:
tr.setmode('br1.b1','close')
tr.setmode('br1.b2','close')
tr.setmode('br1','close')
tr.close('br1.b2')
tr.close('br1.b1')
tr.grid()
r.mainloop()
/code
will show the whole tree, but will set correctly the
(+) and (-) boxes. (The commented out statements where
in place before direct calls to Tk - same result).
I was hoping to have some branches hidden (the leaves,
in this case).
My solution above make any sense till I read the docs,
http://epydoc.sourceforge.net/stdlib/public/Tix.Tree-class.html
quote
say:
setmode(self, entrypath, mode='none')
[snip]
... If mode is set to close, a (-)
indicator is drawn next the the entry.
... The
open mode indicates the entry has hidden children and this entry can be
opened by the user. The close mode indicates that all the children of the
entry are now visible and the entry can be closed by the user.
/quote
Rob.
--
http://www.victim-prime.dsl.pipex.com/
--
http://mail.python.org/mailman/listinfo/python-list
