[issue22208] tarfile can't add in memory files (reopened)

[Roundup Robot]

Roundup Robot added the comment:

New changeset 94a94deaf06a by Martin Panter in branch '3.5':
Issues #22468, #21996, #22208: Clarify gettarinfo() and TarInfo usage
https://hg.python.org/cpython/rev/94a94deaf06a

New changeset 9d5217aaea13 by Martin Panter in branch '2.7':
Issues #22468, #21996, #22208: Clarify gettarinfo() and TarInfo usage
https://hg.python.org/cpython/rev/9d5217aaea13

--
nosy: +python-dev

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22208] tarfile can't add in memory files (reopened)

[Martin Panter]

Martin Panter added the comment:

In Issue 22468, I posted a patch which encourages using TarInfo directly, and 
hopefully clarifies that gettarinfo() is only for OS files. I think that should 
cover the documentation aspect of this bug, although an enhancement to 
synthesize TarInfo objects for regular files etc might still be a good idea.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22208
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22208] tarfile can't add in memory files (reopened)

[Martin Panter]

Martin Panter added the comment:

If you read the documentation it is clear that gettarfile() requires an OS 
file, so won’t work with an internal Python-only file object. Maybe the 
documentation could be tweaked, but I don’t think the gettarfile() 
implementation should be changed. To me the whole point of it is to call 
fstat() on the file and fill in the TarInfo attributes appropriately.

Instead, perhaps an enhancement could be made that allowed something like this:

metadata = TarInfo.make_file('helloworld.txt', len(b))
tarFileObj.addfile(metadata, io.BytesIO(b))

The corresponding TarInfo class could grow new presets looking something like:

class TarInfo:
@classmethod
def make_file(cls, name, size):  # Name and size are mandatory
self = cls(name)
self.type = REGTYPE
self.size = size
self.mtime = None  # Force addfile() to set it to some default 
time.time() value
self.mode = 0o644
return self

@classmethod
def make_executable(cls, name, size):
...
self.mode = 0o755
...

@classmethod
def make_directory(cls, name):
...
self.type = DIRTYPE
...

def make_hard_link(cls, name, target)
def make_symlink(cls, name, target)
def make_block_device(cls, name, major, minor)  # Set undocumented 
attributes
def make_char_device(cls, name, major, minor)

--
nosy: +vadmium

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22208
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22208] tarfile can't add in memory files (reopened)

[Lars Gustäbel]

Lars Gustäbel added the comment:

Please provide a patch which allows easy addition of file-like objects (not 
only io.BytesIO) and directories, preferably hard and symbolic links, too. It 
would be nice to still be able to change attributes of a TarInfo before 
addition. Please also add tests.

--
stage:  - needs patch
type: behavior - enhancement

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22208
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22208] tarfile can't add in memory files (reopened)

[Lars Gustäbel]

Lars Gustäbel added the comment:

I don't have an idea how to make it easier and still meet all/most requirements 
and without cluttering up the api. The way it currently works allows the 
programmer to control every tiny aspect of a tar member. Maybe it's best to 
simply add a new entry to the Examples section of the tarfile documentation.


import tarfile, io

with tarfile.open(sample.tar, mode=w) as tar:
t = tarfile.TarInfo(foo)
t.type = tarfile.DIRTYPE
tar.addfile(t)

b = Hello world!.encode(ascii)

t = tarfile.TarInfo(foo/bar)
t.size = len(b)
tar.addfile(t, io.BytesIO(b))

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22208
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22208] tarfile can't add in memory files (reopened)

[Mark Grandi]

Mark Grandi added the comment:

 I don't have an idea how to make it easier and still meet all/most 
 requirements and without cluttering up the api.

That is what i mentioned in my original post, a lot of the time users just 
_don't care_ about a lot of the stuff that a tar archive can store (permission 
bits, uid/gid, etc).

Say i'm on my mac. I can select a bunch of files and then right click - 
compress. Pretending that it saves the resulting archive as a .tar.gz rather 
then a .zip, that's really it. The user doesn't care about the permission bits, 
uid/gid or any of that, they just want a compressed archive.

While the api does do a good job of exposing the low level parts of the api 
with TarInfo, being able to set all the stuff manually or have it figured out 
through gettarinfo() calling os.stat()

My original reasoning for this bug report is that its way too hard to do it for 
in-memory files, as those don't have file descriptors so os.stat() fails. But 
why can't we just make it so:

gettarinfo() is called
* if it's a regular file, it continues as it does not
* if it is NOT a regular file (no file descriptor), then it returns a 
TarInfo object with the 'name' and 'size' set, and the rest of the fields set 
to default values (the current user's uid and gid, acceptable permission bits 
and the correct type for a regular file (REGFILE?)
* if gettarinfo() is called with a non regular file and it's name has a 
slash, then its assumed to be a folder structure, so then it will add the 
correct TarInfo with type = DIRTYPE and then insert the file underneath that 
folder, sorta how zipfile works. I looked at the tarfile.py code and it seems 
it does this already. 


This just adds the needed easy use case for the tarfile module, as the 
complicated low level api is there, we just need something that users just want 
to create an archive without worrying too much about the low level stuff. So 
then they can just:

import tarfile, io

fileToAdd = io.BytesIO(hello world!.encode(utf-8))
with tarfile.open(sample.tar, mode=w) as tar:

# this TarInfo object has:
#name = 'somefile.txt'
#type = REGTYPE (or whatever is 'just a regular file')
#uid = 501, gid = 20, gname=staff, uname=markgrandi, mode=644
tmpTarInfo = tar.gettarinfo(somefile.txt, fileToAdd)
tar.addfile()



So basically its just having defaults for the TarInfo object when gettarinfo() 
is given a file that doesn't have a file descriptor.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22208
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22208] tarfile can't add in memory files (reopened)

[R. David Murray]

Changes by R. David Murray rdmur...@bitdance.com:


--
nosy: +r.david.murray

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22208
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22208] tarfile can't add in memory files (reopened)

[Mark Grandi]

Mark Grandi added the comment:

I still don't see why TarFile.add() can't be changed to accept a file like 
object so users don't have to fumble around with a TarInfo object when they 
just want add a file like object, and don't care about the permission bits and 
whatnot. It also says un the TarInfo section that you get those objects from 
gettarinfo(), and its not clear (at least to me) that you can just manually 
construct one, and that you only need the size property set.

I still the easiest way to solve this is to have gettarinfo() or add() accept 
file like objects that don't have file descriptors, as it would go a long way 
to simplifying the use of this module as well. (in zip file you don't ever have 
to worry about ZipInfo objects unless you need them...)

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22208
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22208] tarfile can't add in memory files (reopened)

[Lars Gustäbel]

Lars Gustäbel added the comment:

tarfile needs to know the size of a file object beforehand because the tar 
header is written first followed by the file object's data. If the file object 
is not based on a real file descriptor, tarfile cannot simply use os.fstat() 
but the user has to pass the size somehow. And I doubt that it's a good idea to 
add size arguments to TarFile.add() and .addfile() because it might lead to 
confusion.

I think tarfile is rather good at exposing the important parts of its low-level 
api to the programmer, in a way that still leaves some work for him to do but 
without getting in his way.  I don't see why manually creating TarInfo objects 
is such a big deal. It is the far superior way because it offers the maximum 
freedom for the programmer - admittedly at the cost of a slightly steeper 
learning curve. And we have to account for many different use cases that people 
have. For example, you don't mention what you think creating directories from 
scratch should be like in your opinion.

With regard to the usage of the size attribute the documentation for 
TarFile.addfile() says clearly:

Add the TarInfo object tarinfo to the archive. If fileobj is given, 
tarinfo.size bytes are read from it and added to the archive. You can create 
TarInfo objects using gettarinfo().

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22208
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22208] tarfile can't add in memory files (reopened)

[Mark Grandi]

Mark Grandi added the comment:

I was just thinking that if os.stat fails, then you try getting the size by 
just calling len() on it, as stuff like io.BytesIO and io.StringIO will respond 
to that.

But if we are not changing the behavior of the API, at the very least there 
needs to be documentation changes / an example of adding a file that does not 
have a file descriptor. I can do that if needed.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22208
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22208] tarfile can't add in memory files (reopened)

[Antoine Pitrou]

Antoine Pitrou added the comment:

The example given by Lars shows that it's not that easy to come up with the 
right code. Why not make it easier?

--
nosy: +pitrou

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22208
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22208] tarfile can't add in memory files (reopened)

[Lars Gustäbel]

Lars Gustäbel added the comment:

Why overcomplicate things?

import io, tarfile

with tarfile.open(foo.tar, mode=w) as tar:
b = hello world!.encode(utf-8)

t = tarfile.TarInfo(helloworld.txt)
t.size = len(b) # this is crucial
tar.addfile(t, io.BytesIO(b))


My answer to issue10369 was never supposed to be used as a reference on how to 
add file-like objects to a TarFile. I posted it as a simpler but equivalent 
version of the code of the original poster, which is why it looks hackish.

I think the documentation on TarFile.gettarinfo() is rather clear on how to use 
it (i.e. that it needs a file object with a valid file descriptor). Also, I 
think that the code above is intuitive and simple.

--
assignee:  - lars.gustaebel
priority: normal - low
type:  - behavior

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22208
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22208] tarfile can't add in memory files (reopened)

[Mark Grandi]

New submission from Mark Grandi:

So I ran into this problem today, where near impossible to create a 
tarfile.TarFile object, then add files to the archive, when the files are in 
memory file-like objects (like io.BytesIO, io.StringIO, etc)

code example:

###
import tarfile, io

tarFileIo = io.BytesIO()

tarFileObj = tarfile.open(fileobj=tarFileIo, mode=w:xz)

fileToAdd = io.BytesIO(hello world!.encode(utf-8))

# fixes AttributeError: '_io.BytesIO' object has no attribute 'name'
fileToAdd.name=helloworld.txt

# fails with 'io.UnsupportedOperation: fileno'
tarInfo = tarFileObj.gettarinfo(arcname=helloworld.txt, fileobj=fileToAdd)

# never runs
tarFileObj.addfile(tarInfo, fileobj=fileToAdd)
###

this was previously reported as this bug: http://bugs.python.org/issue10369 but 
I am unhappy with the resolution of its not a bug, and the 'hack' that Lars 
posted as a solution. My reasons:

1: The zipfile module supports writing in memory files / bytes , using the 
following code (which is weird but it works)

tmp = zipfile.ZipFile(tmp.zip, mode=w)
import io
x = io.BytesIO(hello world!.encode(utf-8))
tmp.writestr(helloworld.txt, x.getbuffer())
tmp.close()

2: the 'hack' that Lars posted, while it works, this is unintuitive and 
confusing, and isn't the intended behavior. What happens if your script is 
cross platform, what file do you open to give to os.stat()? In the code posted 
it uses open('/etc/passwd/') for the fileobj parameter to gettarinfo(), but 
that file doesn't exist on windows, now not only are you doing this silly hack, 
you have to have code that checks platform.system() to get a valid file that is 
known to exist for every system, or use sys.executable, except the 
documentation for that says it can return None or an empty string.

3: it is easy to fix (at least to me), in tarfile.gettarinfo(), if fileobj is 
passed in, and it doesn't have a fileno, then to create the TarInfo object, you 
set 'name' to be the arcname parameter, size = len(fileobj), then have default 
(maybe overridden by keyword args to gettarinfo()) values for 
uid/gid/uname/gname.

On a random tar.gz file that I downloaded from sourceforge, the uid/gid are 
'500' (when my gid is 20 and uid is 501), and the gname/uname are just empty 
strings. So its obvious that those don't matter most of the time, and when they 
do matter, you can modify the TarInfo object after creation or pass in values 
for them in a theoretical keywords argument to gettarinfo().

If no one wants to code this I can provide a patch, I just want the previous 
bug report's status of not a bug to be reconsidered.

--
components: Library (Lib)
messages: 225374
nosy: markgrandi
priority: normal
severity: normal
status: open
title: tarfile can't add in memory files (reopened)
versions: Python 3.4

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22208
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22208] tarfile can't add in memory files (reopened)

[Ned Deily]

Changes by Ned Deily n...@acm.org:


--
nosy: +lars.gustaebel

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22208
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com