Re: [Python-ideas] tarfile.extractall progress

[Serhiy Storchaka]

01.09.17 14:50, Tarek Ziadé пише:

For large archives, I want to display a progress bar while the archive
is being extracted with:

https://docs.python.org/3/library/tarfile.html#tarfile.TarFile.extractall

I could write my own version of extractall() to do this, or maybe we
could introduce a callback option that gets called
everytime .extract() is called in extractall()

The callback can receive the tarinfo object and where it's being
extracted. This is enough to plug a progress bar
and avoid reinventing .extractall()


This is not enough if extract large files. In that case you perhaps want 
to update the progress bar more often.


If add this feature to tarfile, it may be worth to add it to zipfile and 
shutil functions (copytree, rmtree). And if call a callback for every 
extracted/copied entity, it may be worth to use its result for filtering.


___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] tarfile.extractall progress

[Tarek Ziadé]

On Fri, Sep 1, 2017, at 02:04 PM, Oleg Broytman wrote:
> Hi!
> 
> On Fri, Sep 01, 2017 at 01:50:13PM +0200, Tarek Ziad?? 
> wrote:
> > Hey,
> > 
> > For large archives, I want to display a progress bar while the archive
> > is being extracted with:
> > 
> > https://docs.python.org/3/library/tarfile.html#tarfile.TarFile.extractall
> > 
> > I could write my own version of extractall() to do this, or maybe we
> > could introduce a callback option that gets called
> > everytime .extract() is called in extractall()
> > 
> > The callback can receive the tarinfo object and where it's being
> > extracted. This is enough to plug a progress bar
> > and avoid reinventing .extractall()
> 
>What is "where" here? I think it should be 2 parameters -- position
> in the file (in bytes) and total file size; the total could be None if
> the size is unknown (the tar is piped from network or a (g/bz)zip
> subprocess).

Interesting. In my mind, I was thinking about a high level callable that
would just let me count the files and directory that are being
extracted,

my hackish implementation with clint:

with tarfile.open(file, "r:gz") as tar:
size = len(list(tar))
with progress.Bar(expected_size=size) as bar:
def _extract(self, *args, **kw):
bar.show(bar.last_progress + 1)
return self.old(*args, **kw)

tar.old = tar.extract
tar.extract = functools.partial(_extract, tar)
tar.extractall(profile_dir)

What I would expect to be able to do with the new option, something
like:

with tarfile.open(file, "r:gz") as tar:
size = len(list(tar))
with progress.Bar(expected_size=size) as bar:
def _progress(tarinfo):
bar.show(bar.last_progress + 1)
tar.extractall(profile_dir, onextracted=_progress)



> 
> > I can add a ticket and maybe a patch if people think this is a good
> > little enhancement
> 
>Definitely a good idea!
> 
> > Cheers
> > Tarek
> > 
> > -- 
> > 
> > Tarek Ziad?? | coding: https://ziade.org | running: https://foule.es |
> > twitter: @tarek_ziade
> 
> Oleg.
> -- 
>  Oleg Broytmanhttp://phdru.name/   
>  p...@phdru.name
>Programmers don't die, they just GOSUB without RETURN.
> ___
> Python-ideas mailing list
> Python-ideas@python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] tarfile.extractall progress

[Tarek Ziadé]

On Fri, Sep 1, 2017, at 02:18 PM, Paul Moore wrote:
[..]
> 
> Sounds like a reasonable enhancement, but for your particular use
> couldn't you just subclass TarFile and call your progress callback at
> the end of the extract method after the base class extract?

Yes that's what I ended up doing.  But a callable in extractall() sounds
like a simpler way to do it.

___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] tarfile.extractall progress

[Paul Moore]

On 1 September 2017 at 12:50, Tarek Ziadé  wrote:
> Hey,
>
> For large archives, I want to display a progress bar while the archive
> is being extracted with:
>
> https://docs.python.org/3/library/tarfile.html#tarfile.TarFile.extractall
>
> I could write my own version of extractall() to do this, or maybe we
> could introduce a callback option that gets called
> everytime .extract() is called in extractall()
>
> The callback can receive the tarinfo object and where it's being
> extracted. This is enough to plug a progress bar
> and avoid reinventing .extractall()
>
> I can add a ticket and maybe a patch if people think this is a good
> little enhancement

Sounds like a reasonable enhancement, but for your particular use
couldn't you just subclass TarFile and call your progress callback at
the end of the extract method after the base class extract?

Paul
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] tarfile.extractall progress

[Oleg Broytman]

Hi!

On Fri, Sep 01, 2017 at 01:50:13PM +0200, Tarek Ziad??  wrote:
> Hey,
> 
> For large archives, I want to display a progress bar while the archive
> is being extracted with:
> 
> https://docs.python.org/3/library/tarfile.html#tarfile.TarFile.extractall
> 
> I could write my own version of extractall() to do this, or maybe we
> could introduce a callback option that gets called
> everytime .extract() is called in extractall()
> 
> The callback can receive the tarinfo object and where it's being
> extracted. This is enough to plug a progress bar
> and avoid reinventing .extractall()

   What is "where" here? I think it should be 2 parameters -- position
in the file (in bytes) and total file size; the total could be None if
the size is unknown (the tar is piped from network or a (g/bz)zip
subprocess).

> I can add a ticket and maybe a patch if people think this is a good
> little enhancement

   Definitely a good idea!

> Cheers
> Tarek
> 
> -- 
> 
> Tarek Ziad?? | coding: https://ziade.org | running: https://foule.es |
> twitter: @tarek_ziade

Oleg.
-- 
 Oleg Broytmanhttp://phdru.name/p...@phdru.name
   Programmers don't die, they just GOSUB without RETURN.
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] tarfile.extractall progress

[Tarek Ziadé]

Hey,

For large archives, I want to display a progress bar while the archive
is being extracted with:

https://docs.python.org/3/library/tarfile.html#tarfile.TarFile.extractall

I could write my own version of extractall() to do this, or maybe we
could introduce a callback option that gets called
everytime .extract() is called in extractall()

The callback can receive the tarinfo object and where it's being
extracted. This is enough to plug a progress bar
and avoid reinventing .extractall()

I can add a ticket and maybe a patch if people think this is a good
little enhancement

Cheers
Tarek

-- 

Tarek Ziadé | coding: https://ziade.org | running: https://foule.es |
twitter: @tarek_ziade
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/