New submission from Yaroslav Halchenko <yarikop...@gmail.com>:
originally detected on python 2.7, but replicated with python 3.5.3 --
apparently io.FileIO, if given a bytestring of 2GB or more, cannot write it all
at once -- saves (and returns that size) only 2GB - 4096.
I found no indication for such behavior anywhere in the documentation. And it
is surprising to me especially since regular file.write does it just fine!
attached is the code snippet which I list below and which demonstrates it
$> python3 --version; python3 longwrite.py
Python 3.5.3
Written 2147479552 out of 2147483648
4096 bytes were not written
Traceback (most recent call last):
File "longwrite.py", line 28, in <module>
assert in_digest == out_digest, "Digests do not match"
AssertionError: Digests do not match
python3 longwrite.py 7.03s user 5.80s system 99% cpu 12.848 total
1 11365 ->1.....................................:Sat 30 Sep 2017 04:56:26 PM
EDT:.
smaug:/mnt/btrfs/scrap/tmp
$> cat longwrite.py
# -*- coding: utf-8 -*-
import io
import os
import hashlib
s = u' '*(256**4//2) #+ u"перфекто"
s=s.encode('utf-8')
#s=' '*(10)
in_digest = hashlib.md5(s).hexdigest()
fname = 'outlong.dat'
if os.path.exists(fname):
os.unlink(fname)
with io.FileIO(fname, 'wb') as f:
#with open(fname, 'wb') as f:
n = f.write(s)
#n = os.stat(fname).st_size
print("Written %d out of %d" % (n, len(s)))
if n != len(s):
print("%d bytes were not written" % (len(s) - n))
# checksum
with open(fname, 'rb') as f:
out_digest = hashlib.md5(f.read()).hexdigest()
assert in_digest == out_digest, "Digests do not match"
print("all ok")
----------
components: IO
files: longwrite.py
messages: 303429
nosy: Yaroslav.Halchenko
priority: normal
severity: normal
status: open
title: io.FileIO cannot write more than 2GB (-4096) bytes??? must be documented
(if not fixed)
type: behavior
versions: Python 2.7, Python 3.5
Added file: https://bugs.python.org/file47182/longwrite.py
_______________________________________
Python tracker <rep...@bugs.python.org>
<https://bugs.python.org/issue31651>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
