[issue34990] year 2038 problem in compileall.py

[STINNER Victor]

Change by STINNER Victor :


--
nosy:  -vstinner

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34990] year 2038 problem in compileall.py

[Jakub Wilk]

Change by Jakub Wilk :


--
nosy: +jwilk

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34990] year 2038 problem in compileall.py

[Ammar Askar]

Change by Ammar Askar :


--
resolution:  -> fixed
stage: patch review -> resolved
status: open -> closed
versions: +Python 3.10, Python 3.11 -Python 3.5, Python 3.6, Python 3.7, Python 
3.8

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34990] year 2038 problem in compileall.py

[Ammar Askar]

Ammar Askar  added the comment:


New changeset 0af681b652c43f0ba90988400ecc1e7934fbfc5d by Miss Islington (bot) 
in branch '3.10':
[3.10] bpo-34990: Treat the pyc header's mtime in compileall as an unsigned int 
(GH-19708)
https://github.com/python/cpython/commit/0af681b652c43f0ba90988400ecc1e7934fbfc5d


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34990] year 2038 problem in compileall.py

[Ammar Askar]

Ammar Askar  added the comment:


New changeset 9d3b6b2472f7c7ef841e652825de652bc8af85d7 by Miss Islington (bot) 
in branch '3.9':
[3.9] bpo-34990: Treat the pyc header's mtime in compileall as an unsigned int 
(GH-19708)
https://github.com/python/cpython/commit/9d3b6b2472f7c7ef841e652825de652bc8af85d7


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34990] year 2038 problem in compileall.py

[miss-islington]

Change by miss-islington :


--
pull_requests: +26379
pull_request: https://github.com/python/cpython/pull/27929

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34990] year 2038 problem in compileall.py

[miss-islington]

Change by miss-islington :


--
nosy: +miss-islington
nosy_count: 6.0 -> 7.0
pull_requests: +26378
pull_request: https://github.com/python/cpython/pull/27928

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34990] year 2038 problem in compileall.py

[Serhiy Storchaka]

Serhiy Storchaka  added the comment:


New changeset bb21e28fd08f894ceff2405544a2f257d42b1354 by Ammar Askar in branch 
'main':
bpo-34990: Treat the pyc header's mtime in compileall as an unsigned int 
(GH-19708)
https://github.com/python/cpython/commit/bb21e28fd08f894ceff2405544a2f257d42b1354


--
nosy: +serhiy.storchaka

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34990] year 2038 problem in compileall.py

[STINNER Victor]

STINNER Victor  added the comment:

I would prefer to mimick importlib._bootstrap_external which uses:

def _pack_uint32(x):
"""Convert a 32-bit integer to little-endian."""
return (int(x) & 0x).to_bytes(4, 'little')

Using 64-bit timestamp (PR 19651), treat timestamp as unsigned (PR 9892 and PR 
19708) have drawback:

* 64-bit timestamp make .pyc files larger
* unsigned timestamp no longer support timestamp before 1969 which can cause 
practical issues

"& 0x" looks dead simple, uses a fixed size of 4 bytes and doesn't have 
any limitation of year 2038.

The timestamp doesn't have to be exact. In practice, it sounds very unlikely 
that two timestamps are equal when compared using (ts1 & 0x) == (ts2 & 
0x). I expect file modification times to be close by a few days, not 
separated by 2**32 seconds (136 years).

Use hash based .pyc to avoid any issuse with file modification time: it should 
make Python more deterministic (more "reproducible").
https://docs.python.org/dev/reference/import.html#pyc-invalidation

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34990] year 2038 problem in compileall.py

[Ammar Askar]

Change by Ammar Askar :


--
pull_requests: +19029
pull_request: https://github.com/python/cpython/pull/19708

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34990] year 2038 problem in compileall.py

[Ammar Askar]

Change by Ammar Askar :


--
nosy: +ammar2
nosy_count: 4.0 -> 5.0
pull_requests: +18977
pull_request: https://github.com/python/cpython/pull/19651

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34990] year 2038 problem in compileall.py

[Bernhard M. Wiedemann]

Bernhard M. Wiedemann  added the comment:

ping.
Another 19th of January passed.

I'd still like to see progress on this, because this hinders my other y2038 bug 
discovery work.

--
versions: +Python 3.5, Python 3.8, Python 3.9

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34990] year 2038 problem in compileall.py

[Karthikeyan Singaravelan]

Karthikeyan Singaravelan  added the comment:

Victor seems there was some discussion about 2038 problem in the original PR 
but I don't know if it's related to this. Reference : 
https://github.com/python/cpython/pull/4575#discussion_r153376173

Thanks

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34990] year 2038 problem in compileall.py

[Stéphane Wirtel]

Change by Stéphane Wirtel :


--
keywords: +patch
pull_requests: +9256
stage:  -> patch review

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34990] year 2038 problem in compileall.py

[Karthikeyan Singaravelan]

Karthikeyan Singaravelan  added the comment:

A reproducer in Python that can be added to test_compileall if needed : 

def test_compile_all_2038(self):
with open(self.source_path, 'r') as f:
os.utime(f.name, (2147558400, 2147558400)) # Jan 20, 2038 as touch
self.assertTrue(compileall.compile_file(pathlib.Path(self.source_path)))


./python.exe -m unittest -v 
test.test_compileall.CompileallTestsWithSourceEpoch.test_compile_all_2038
test_compile_all_2038 (test.test_compileall.CompileallTestsWithSourceEpoch) ... 
ERROR

==
ERROR: test_compile_all_2038 
(test.test_compileall.CompileallTestsWithSourceEpoch)
--
Traceback (most recent call last):
  File 
"/Users/karthikeyansingaravelan/stuff/python/cpython/Lib/test/test_py_compile.py",
 line 30, in wrapper
return fxn(*args, **kwargs)
  File 
"/Users/karthikeyansingaravelan/stuff/python/cpython/Lib/test/test_compileall.py",
 line 114, in test_compile_all_2038
self.assertTrue(compileall.compile_file(pathlib.Path(self.source_path)))
  File "/Users/karthikeyansingaravelan/stuff/python/cpython/Lib/compileall.py", 
line 142, in compile_file
expect = struct.pack('<4sll', importlib.util.MAGIC_NUMBER,
struct.error: 'l' format requires -2147483648 <= number <= 2147483647

--


Thanks

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34990] year 2038 problem in compileall.py

[Stéphane Wirtel]

Stéphane Wirtel  added the comment:

So we need to fix compileall.py.

maybe we could add the label 'easy' to this issue.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34990] year 2038 problem in compileall.py

[STINNER Victor]

STINNER Victor  added the comment:

Timestamp with year >= 2038 are accepted: 
importlib._bootstrap_external._code_to_timestamp_pyc() uses (int(x) & 
0x). It's not a bug, but by design. compileall should just do the same. 
Sorry, I don't know if it's specified somewhere, but I know that it's done on 
purpose.

--
nosy: +vstinner

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34990] year 2038 problem in compileall.py

[Bernhard M. Wiedemann]

Bernhard M. Wiedemann  added the comment:

It does not need to be fixed tomorrow, but 2037 is too late, because by then 
there will be a lot of legacy systems around.
(Un)fortunately many systems live 10+ years now

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34990] year 2038 problem in compileall.py

[Stéphane Wirtel]

Stéphane Wirtel  added the comment:

But until 2038, maybe there will be a new format for the .pyc file.

We should keep this issue and try to fix it for 3.8 or 3.9?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34990] year 2038 problem in compileall.py

[Stéphane Wirtel]

Stéphane Wirtel  added the comment:

With 3.8a

Traceback (most recent call last):
  File "/home/stephane/src/github.com/python/cpython/Lib/runpy.py", line 192, 
in _run_module_as_main
return _run_code(code, main_globals, None,
  File "/home/stephane/src/github.com/python/cpython/Lib/runpy.py", line 85, in 
_run_code
exec(code, run_globals)
  File "/home/stephane/src/github.com/python/cpython/Lib/compileall.py", line 
326, in 
exit_status = int(not main())
  File "/home/stephane/src/github.com/python/cpython/Lib/compileall.py", line 
303, in main
if not compile_file(dest, args.ddir, args.force, args.rx,
  File "/home/stephane/src/github.com/python/cpython/Lib/compileall.py", line 
142, in compile_file
expect = struct.pack('<4sll', importlib.util.MAGIC_NUMBER,
struct.error: 'l' format requires -2147483648 <= number <= 2147483647

--
nosy: +matrixise

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34990] year 2038 problem in compileall.py

[Karthikeyan Singaravelan]

Change by Karthikeyan Singaravelan :


--
nosy: +xtreak

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34990] year 2038 problem in compileall.py

[Bernhard M. Wiedemann]

New submission from Bernhard M. Wiedemann :

To reproduce:
touch -d 2038-01-20 /usr/lib/python3.6/site-packages/six.py
python3 /usr/lib64/python3.6/compileall.py


  File "/usr/lib64/python3.6/compileall.py", line 198, in compile_path
legacy=legacy, optimize=optimize)
  File "/usr/lib64/python3.6/compileall.py", line 90, in compile_dir
legacy, optimize):
  File "/usr/lib64/python3.6/compileall.py", line 138, in compile_file
mtime)
struct.error: 'l' format requires -2147483648 <= number <= 2147483647

It could use either 
64 bit int (requires new .pyc format with different magic number) or
unsigned 32 bit int (gives us only another 68 years)

--
components: Build
messages: 327743
nosy: bmwiedemann
priority: normal
severity: normal
status: open
title: year 2038 problem in compileall.py
type: compile error
versions: Python 3.6, Python 3.7

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com