Re: [Python-ideas] Rewriting file - pythonic way

[Alexey Shrub]

https://pypi.python.org/pypi/in-place
> * Instead of hijacking sys.stdout, a new filehandle is returned for 
writing.
> * The filehandle supports all of the standard I/O methods, not just 
readline().


why fileinput did not support this things?

___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Rewriting file - pythonic way

[Alexey Shrub]

В Понедельник, 16 апр. 2018 в 2:48 , Alexey Shrub 
 написал:

https://pypi.python.org/pypi/in-place


I like in_place module
https://github.com/worldmind/scripts/blob/master/filerewrite/inplacetest.py
it fix some strange features of fileinput module.
Maybe in_place must be in standard library instead fileinput?

___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Rewriting file - pythonic way

[Alexey Shrub]

В Воскресенье, 15 апр. 2018 в 10:47 , George Fischhof 
 написал:

https://docs.python.org/3/library/fileinput.html


https://pypi.python.org/pypi/in-place
looks not bad too
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Rewriting file - pythonic way

[Alexey Shrub]

В Воскресенье, 15 апр. 2018 в 1:12 , Serhiy Storchaka 
 написал:
Actually the reliable code should write into a separate file and 
replace

the original file by the new file only if writing is successful. Or
backup the old file and restore it if writing is failed. Or do both. 
And
handle hard and soft links if necessary. And use file locks if needed 
to
prevent race condition when read/write by different processes. 
Depending

on the specific of the application you may need different code. Your
three lines are enough for a one-time script if the risk of a powerful
blackout or disk space exhaustion is insignificant or if the data is 
not

critical.


I not sure that solving described problems is a task of this level, 
maybe it problem for higher level


___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Rewriting file - pythonic way

[Alexey Shrub]

В Воскресенье, 15 апр. 2018 в 10:47 , George Fischhof 
 написал:

https://docs.python.org/3/library/fileinput.html


Thanks, it works
https://github.com/worldmind/scripts/blob/master/filerewrite/fileinputtest.py
but looks like that way only for line by line processing

___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Rewriting file - pythonic way

[Alexey Shrub]

В Воскресенье, 15 апр. 2018 в 6:19 , Oleg Broytman 
 написал:

Can I recommend to catch exceptions in `backuper.backup()`,
cleanup backuper and unlock locker?


Yes, thanks, I move .backup() to try, about other exception I think 
that it must be catched outside, because this module don't know that to 
do with such problems


___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Rewriting file - pythonic way

[George Fischhof]

Hi,

some similar thing already exist in standard:
https://docs.python.org/3/library/fileinput.html

fileinput(... inplace=True...)

BR,
George

2018-04-15 10:57 GMT+02:00 Alexey Shrub :

> Hi all,
>
> I am new in python (i am moving from Perl world), but I always love Python
> for hight level, beatuful and clean syntax.
> Now I have question/idea about working with files.
> On mine opinion it very popular use case:
> 1. Open file (for read and write)
> 2. Read data from file
> 3. Modify data.
> 4. Rewrite file by modified data.
>
> But now it is looks not so pythonic:
>
> with open(filename, 'r+') as file:
>data = file.read()
>data = data.replace('old', 'new')
>file.seek(0)
>file.write(data)
>file.truncate()
>
> or something like this
>
> with open(filename) as file:
>data = file.read()
> data = data.replace('old', 'new')
> with open(filename) as file:
>file.write(data)
>
> I think best way is something like this
>
> with open(filename, 'r+') as file:
>data = file.read()
>data = data.replace('old', 'new')
>file.rewrite(data)
>
> but for this io.BufferedIOBase must contain rewrite method
>
> what you think about this?
>
>
>
> ___
> Python-ideas mailing list
> Python-ideas@python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Rewriting file - pythonic way

[Oleg Broytman]

On Sun, Apr 15, 2018 at 09:10:57AM -0700, Mahmoud Hashemi  
wrote:
> Depending on how firm your requirements around locking are, you may find
> this code useful:
> https://github.com/mahmoud/boltons/blob/6b0721b6aeda6d3ec6f5d31be7c741bc7fcc4635/boltons/fileutils.py#L303
> 
> (docs here:
> http://boltons.readthedocs.io/en/latest/fileutils.html#atomic-file-saving )
> 
> Basically every operating system has _some_ way of doing an atomic file
> replacement, letting us guarantee that a file at a given location is always
> valid. atomic_save provides a unified interface to that cross-platform
> behavior.
> 
> The code does not do locking, as neither I nor its other users have wanted
> it, but I'd be happy to extend it if there's a sensible default.

   I don't like it renames the file at the end. Renaming could lead to
changed file ownership and permissions; restoring permissions is not
always possible, restoring ownership is almost never possible. Renaming
is also not always possible due to restricted directory permissions.

> On Sun, Apr 15, 2018 at 8:19 AM, Oleg Broytman  wrote:
> 
> > On Sun, Apr 15, 2018 at 05:15:55PM +0300, Alexey Shrub 
> > wrote:
> > > В Воскресенье, 15 апр. 2018 в 2:40 , Nick Coghlan 
> > > написал:
> > > > https://bugs.python.org/issue8604#msg174104 is the relevant tracker
> > > > discussion
> > >
> > > Thanks all, I agree that universal and absolutly safe solution is very
> > > difficult, but for experiment I made some draft
> > > https://github.com/worldmind/scripts/tree/master/filerewrite
> >
> >Good!
> >
> > > main code here
> > > https://github.com/worldmind/scripts/blob/master/
> > filerewrite/filerewrite.py#L46
> >
> >Can I recommend to catch exceptions in `backuper.backup()`,
> > cleanup backuper and unlock locker?


Oleg.
-- 
 Oleg Broytmanhttp://phdru.name/p...@phdru.name
   Programmers don't die, they just GOSUB without RETURN.
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Rewriting file - pythonic way

[Mahmoud Hashemi]

Depending on how firm your requirements around locking are, you may find
this code useful:
https://github.com/mahmoud/boltons/blob/6b0721b6aeda6d3ec6f5d31be7c741bc7fcc4635/boltons/fileutils.py#L303

(docs here:
http://boltons.readthedocs.io/en/latest/fileutils.html#atomic-file-saving )

Basically every operating system has _some_ way of doing an atomic file
replacement, letting us guarantee that a file at a given location is always
valid. atomic_save provides a unified interface to that cross-platform
behavior.

The code does not do locking, as neither I nor its other users have wanted
it, but I'd be happy to extend it if there's a sensible default.

On Sun, Apr 15, 2018 at 8:19 AM, Oleg Broytman  wrote:

> On Sun, Apr 15, 2018 at 05:15:55PM +0300, Alexey Shrub 
> wrote:
> > В Воскресенье, 15 апр. 2018 в 2:40 , Nick Coghlan 
> > написал:
> > > https://bugs.python.org/issue8604#msg174104 is the relevant tracker
> > > discussion
> >
> > Thanks all, I agree that universal and absolutly safe solution is very
> > difficult, but for experiment I made some draft
> > https://github.com/worldmind/scripts/tree/master/filerewrite
>
>Good!
>
> > main code here
> > https://github.com/worldmind/scripts/blob/master/
> filerewrite/filerewrite.py#L46
>
>Can I recommend to catch exceptions in `backuper.backup()`,
> cleanup backuper and unlock locker?
>
> Oleg.
> --
>  Oleg Broytmanhttp://phdru.name/p...@phdru.name
>Programmers don't die, they just GOSUB without RETURN.
> ___
> Python-ideas mailing list
> Python-ideas@python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Rewriting file - pythonic way

[Oleg Broytman]

On Sun, Apr 15, 2018 at 05:15:55PM +0300, Alexey Shrub  wrote:
> В Воскресенье, 15 апр. 2018 в 2:40 , Nick Coghlan 
> написал:
> > https://bugs.python.org/issue8604#msg174104 is the relevant tracker
> > discussion
> 
> Thanks all, I agree that universal and absolutly safe solution is very
> difficult, but for experiment I made some draft
> https://github.com/worldmind/scripts/tree/master/filerewrite

   Good!

> main code here
> https://github.com/worldmind/scripts/blob/master/filerewrite/filerewrite.py#L46

   Can I recommend to catch exceptions in `backuper.backup()`,
cleanup backuper and unlock locker?

Oleg.
-- 
 Oleg Broytmanhttp://phdru.name/p...@phdru.name
   Programmers don't die, they just GOSUB without RETURN.
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Rewriting file - pythonic way

[Alexey Shrub]

В Воскресенье, 15 апр. 2018 в 2:40 , Nick Coghlan 
 написал:

https://bugs.python.org/issue8604#msg174104 is the relevant tracker
discussion


Thanks all, I agree that universal and absolutly safe solution is very 
difficult, but for experiment I made some draft

https://github.com/worldmind/scripts/tree/master/filerewrite
main code here
https://github.com/worldmind/scripts/blob/master/filerewrite/filerewrite.py#L46

___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Rewriting file - pythonic way

[Nick Coghlan]

On 15 April 2018 at 20:47, Paul Moore  wrote:
> On 15 April 2018 at 11:22, Elazar  wrote:
>> בתאריך יום א׳, 15 באפר׳ 2018, 13:13, מאת Serhiy Storchaka
>> ‏:
>>> Actually the reliable code should write into a separate file and replace
>>> the original file by the new file only if writing is successful. Or
>>> backup the old file and restore it if writing is failed. Or do both. And
>>> handle hard and soft links if necessary. And use file locks if needed to
>>> prevent race condition when read/write by different processes. Depending
>>> on the specific of the application you may need different code. Your
>>> three lines are enough for a one-time script if the risk of a powerful
>>> blackout or disk space exhaustion is insignificant or if the data is not
>>> critical.
>>
>> This pitfall sounds like a good reason to have such a function in the
>> standard library.
>
> It certainly sounds like a good reason for someone to write a "safe
> file rewrite" library function. But I don't think that it's such a
> common need that it needs to be a stdlib function. It may well even be
> the case that there's such a function already available on PyPI - has
> anyone actually checked?

There wasn't last time I checked (which admittedly was several years ago now).

The issue is that it's painfully difficult to write a robust
cross-platform "atomic rewrite" operation that can cleanly handle a
wide range of arbitrary use cases - instead, folks are more likely to
write simpler alternatives that work well enough given whichever
simplifying assumptions are applicable to their use case (which may
even include "I don't care about atomicity, and am quite happy to let
a poorly timed Ctrl-C or unexpected system shutdown corrupt the file
I'm rewriting").

https://bugs.python.org/issue8604#msg174104 is the relevant tracker
discussion (deliberately linking into the middle of it, since the
early part is akin to this thread: reactions mostly along the lines of
"that's easy, and doesn't need to be in the standard library". It
definitely *isn't* easy, but it's also challenging to publish on PyPI,
since it's a quagmire of platform specific complexity and edge cases,
if you mess it up you can cause significant data loss, and anyone that
already knows they need atomic rewrites is likely to be able to come
up with their own purpose specific implementation in less time than it
would take them to assess the suitability of 3rd party alternatives).

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Rewriting file - pythonic way

[Paul Moore]

On 15 April 2018 at 11:22, Elazar  wrote:
> בתאריך יום א׳, 15 באפר׳ 2018, 13:13, מאת Serhiy Storchaka
> ‏:
>> Actually the reliable code should write into a separate file and replace
>> the original file by the new file only if writing is successful. Or
>> backup the old file and restore it if writing is failed. Or do both. And
>> handle hard and soft links if necessary. And use file locks if needed to
>> prevent race condition when read/write by different processes. Depending
>> on the specific of the application you may need different code. Your
>> three lines are enough for a one-time script if the risk of a powerful
>> blackout or disk space exhaustion is insignificant or if the data is not
>> critical.
>
> This pitfall sounds like a good reason to have such a function in the
> standard library.

It certainly sounds like a good reason for someone to write a "safe
file rewrite" library function. But I don't think that it's such a
common need that it needs to be a stdlib function. It may well even be
the case that there's such a function already available on PyPI - has
anyone actually checked? And if there isn't, then writing module and
publishing it there would seem like a *very* good starting point - as
well as allowing the developer to thrash out the best API, it would
also provide for lots of testing in unusual scenarios that the
developer may not have thought about (Windows file locking is very
different from Unix, what is an atomic operation differs between
platforms, error handling and retries may be something to consider,
etc).

The result would be a useful package, and the download and activity
stats for it would be a great indication of whether it's a frequent
enough need to justify including in core Python.

IMO, it probably isn't. I suspect that most uses would be fine with
the quoted 3-liner, but very few people would need the sort of
robustness that Serhiy is describing (and that level of robustness
*would* be needed for a stdlib implementation). So PyPI is likely a
better home for the "bulletproof" version, and 3 lines of code is a
perfectly acceptable and Pythonic solution for people with simpler
needs.

Paul
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Rewriting file - pythonic way

[Paul Moore]

On 15 April 2018 at 10:49, Alexey Shrub  wrote:
> В Воскресенье, 15 апр. 2018 в 12:40 , Serhiy Storchaka 
> написал:
>>
>> If the problem is that you want to use a single line instead of three
>> line, you can add a function
>
>
> Yes, I think that single line with word 'rewrite' is much more readable than
> those three lines.
> And yes, I can make my own function, but it is typical task - maybe it must
> be in standard library?

I don't think it's *that* typical. I don't recall even having wanted
to do this in all the time I've been using Python...
Paul
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Rewriting file - pythonic way

[Elazar]

This pitfall sounds like a good reason to have such a function in the
standard library.

Elazar

בתאריך יום א׳, 15 באפר׳ 2018, 13:13, מאת Serhiy Storchaka ‏<
storch...@gmail.com>:

> 15.04.18 12:49, Alexey Shrub пише:
> > В Воскресенье, 15 апр. 2018 в 12:40 , Serhiy Storchaka
> >  написал:
> >> If the problem is that you want to use a single line instead of three
> >> line, you can add a function
> >
> > Yes, I think that single line with word 'rewrite' is much more readable
> > than those three lines.
> > And yes, I can make my own function, but it is typical task - maybe it
> > must be in standard library?
>
> Not every three lines of code must be a function in standard library.
> And these three lines don't look enough common.
>
> Actually the reliable code should write into a separate file and replace
> the original file by the new file only if writing is successful. Or
> backup the old file and restore it if writing is failed. Or do both. And
> handle hard and soft links if necessary. And use file locks if needed to
> prevent race condition when read/write by different processes. Depending
> on the specific of the application you may need different code. Your
> three lines are enough for a one-time script if the risk of a powerful
> blackout or disk space exhaustion is insignificant or if the data is not
> critical.
>
> ___
> Python-ideas mailing list
> Python-ideas@python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Rewriting file - pythonic way

[Serhiy Storchaka]

15.04.18 12:49, Alexey Shrub пише:
В Воскресенье, 15 апр. 2018 в 12:40 , Serhiy Storchaka 
 написал:

If the problem is that you want to use a single line instead of three
line, you can add a function


Yes, I think that single line with word 'rewrite' is much more readable 
than those three lines.
And yes, I can make my own function, but it is typical task - maybe it 
must be in standard library?


Not every three lines of code must be a function in standard library. 
And these three lines don't look enough common.


Actually the reliable code should write into a separate file and replace 
the original file by the new file only if writing is successful. Or 
backup the old file and restore it if writing is failed. Or do both. And 
handle hard and soft links if necessary. And use file locks if needed to 
prevent race condition when read/write by different processes. Depending 
on the specific of the application you may need different code. Your 
three lines are enough for a one-time script if the risk of a powerful 
blackout or disk space exhaustion is insignificant or if the data is not 
critical.


___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Rewriting file - pythonic way

[Alexey Shrub]

В Воскресенье, 15 апр. 2018 в 12:40 , Serhiy Storchaka 
 написал:

If the problem is that you want to use a single line instead of three
line, you can add a function


Yes, I think that single line with word 'rewrite' is much more readable 
than those three lines.
And yes, I can make my own function, but it is typical task - maybe it 
must be in standard library?


___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Rewriting file - pythonic way

[Serhiy Storchaka]

15.04.18 11:57, Alexey Shrub пише:
I am new in python (i am moving from Perl world), but I always love 
Python for hight level, beatuful and clean syntax.

Now I have question/idea about working with files.
On mine opinion it very popular use case:
1. Open file (for read and write)
2. Read data from file
3. Modify data.
4. Rewrite file by modified data.

But now it is looks not so pythonic:

with open(filename, 'r+') as file:
    data = file.read()
    data = data.replace('old', 'new')
    file.seek(0)
    file.write(data)
    file.truncate()


What do you mean by calling this not pythonic?


I think best way is something like this

with open(filename, 'r+') as file:
    data = file.read()
    data = data.replace('old', 'new')
    file.rewrite(data)

but for this io.BufferedIOBase must contain rewrite method


If the problem is that you want to use a single line instead of three 
line, you can add a function:


def file_rewrite(file, data):
 file.seek(0)
 file.write(data)
 file.truncate()

and use it. This looks pretty pythonic to me.

___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Rewriting file - pythonic way

[Alexey Shrub]

Hi all,

I am new in python (i am moving from Perl world), but I always love 
Python for hight level, beatuful and clean syntax.

Now I have question/idea about working with files.
On mine opinion it very popular use case:
1. Open file (for read and write)
2. Read data from file
3. Modify data.
4. Rewrite file by modified data.

But now it is looks not so pythonic:

with open(filename, 'r+') as file:
   data = file.read()
   data = data.replace('old', 'new')
   file.seek(0)
   file.write(data)
   file.truncate()

or something like this

with open(filename) as file:
   data = file.read()
data = data.replace('old', 'new')
with open(filename) as file:
   file.write(data)

I think best way is something like this

with open(filename, 'r+') as file:
   data = file.read()
   data = data.replace('old', 'new')
   file.rewrite(data)

but for this io.BufferedIOBase must contain rewrite method

what you think about this?



___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/