Re: Y2038 - best way forward in Debian?

2020-03-06 Thread Michael Stone

On Fri, Mar 06, 2020 at 07:46:26AM +0100, Eduard Bloch wrote:

So, wouldn't a restart of the i386 architecture under a different name
give an excelent opportunity to get rid of many of such workarounds?


In theory, sure...but I don't see that there's any actual demand for a 
new/clean i386 architecture in 2038.




Re: Y2038 - best way forward in Debian?

2020-03-06 Thread Florian Weimer
* Eduard Bloch:

> I vaguelly remember that glibc keeps collecting workarounds for replaced
> APIs all the time, adjusting binary compatibility with manually
> redirected symbols. Glibc folks might correct me, though.
>
> So, wouldn't a restart of the i386 architecture under a different name
> give an excelent opportunity to get rid of many of such workarounds?

Yes, that's what happens automatically once you start a new
architecture for glibc, with a new ABI baseline.  All the compat
symbols before that baseline are automatically removed.

But for one of the largest pieces with unwanted ABI exposure (the
stdio implementation, also known as libio), we do not have a clean
replacement.  Part of the challenge is that even current gnulib pokes
at stdio internals, so it's not just about maintaining compatibility
with legacy binaries back when programmers didn't know you shouldn't
do that (or simply had no way to influence development of the system
libc).



Re: Y2038 - best way forward in Debian?

2020-03-05 Thread Eduard Bloch
Hallo,
* Steve McIntyre [Tue, Feb 04 2020, 01:14:10PM]:

>  * 32-bit ABIs/arches *can* be told to use 64-bit time_t from glibc
>upwards, but this will of course affect the ABI. Embedded uses of
>time_t in libraries will change size, etc. This *will* be safe for
>2038.
>
> So, we're all fine? Not so much: for our 32-bit Debian arches, we will
> need to basically rebuild the world to be 2038-safe. When we had to do
> something like this in the past, to deal with the libc5->libc6
> transition, we had an SONAME change in libc to work with. We decided

Just thinking:

I vaguelly remember that glibc keeps collecting workarounds for replaced
APIs all the time, adjusting binary compatibility with manually
redirected symbols. Glibc folks might correct me, though.

So, wouldn't a restart of the i386 architecture under a different name
give an excelent opportunity to get rid of many of such workarounds?

Also, if the download statistics reveal the then-legacy i386
architecture becoming useless, it could be retired sooner without being
sorry.

Best regards,
Eduard.

--
 Lambda-Kalkuel ist fuer Hacker so was wie das, was fuer Jedis
"Die Macht" darstellt. Das, worauf man zurueckgreift, wenn man
ganz abartig schwierige Dinge erledigen muss.



Re: Y2038 - best way forward in Debian?

2020-02-25 Thread Thorsten Glaser
Arnd Bergmann wrote:

>is clearly needed anyway. Once there is a working armhf version with
>full time64 user space, there can be a separate discussion about what
>to do with the i386 port (phase out i386 before y2038, migrate all of
>i386 to time64 quickly, have two separate i386 ports, or something

I’d greatly appreciate that. I don’t own a̲n̲y̲ 64-bit machines (of any
architecture)¹ and would like to proceed using Debian on some, and,
perhaps even now already, be able to calculate things less than two
decades in the future.

If there’s need for manpower to do the 64-bit switch for i386 count me
in, I already did part of the work on MirBSD and on Debian/x32 already
anyway.

Thanks in advance,
//mirabilos
① I occasionally work on a Thinkpad X61, which belongs to my employer
  though, not me, as does the x32 desktop and the virtualisation hosts
  at $dayjob
-- 
 den AGP stecker anfeilen, damit er in den slot aufm 440BX board passt…
oder netzteile, an die man auch den monitor angeschlossen hat und die dann für
ein elektrisch aufgeladenes gehäuse gesorgt haben […] für lacher gut auf jeder
LAN party │  damals, als der pizzateig noch auf dem monior "gegangen" ist



Re: Y2038 - best way forward in Debian?

2020-02-18 Thread Ben Hutchings
On Tue, 2020-02-18 at 12:25 +0100, Marco d'Itri wrote:
> On Feb 17, Wouter Verhelst  wrote:
> 
> > - Figure out a way for 32-bit binary-only programs to keep working when
> >   they touch a time_t beyond 2038.
> I am sure that around 2035 "time namespaces" or something similar will 
> be implemented to solve this.

Linux 5.6 introduces time namespaces - though currently they only allow
offsets to monotonic clocks (so containers can have monotonic time when
migrated), not CLOCK_REALTIME.

Ben.

-- 
Ben Hutchings
Tomorrow will be cancelled due to lack of interest.




signature.asc
Description: This is a digitally signed message part


Re: Y2038 - best way forward in Debian?

2020-02-18 Thread Russ Allbery
Simon Richter  writes:
> On Tue, Feb 18, 2020 at 12:25:48PM +0100, Marco d'Itri wrote:

>> I am sure that around 2035 "time namespaces" or something similar will 
>> be implemented to solve this.

> Hopefully a bit earlier. I have a few pieces of software that only build in
> virtual machines with a timezone offset to UTC of -10 years, and I'd really
> like to move them to containers or light them on fire.

Time namespaces were merged for Linux 5.6.

https://lwn.net/Articles/766089/
https://lwn.net/Articles/810780/

-- 
Russ Allbery (r...@debian.org)  



Re: Y2038 - best way forward in Debian?

2020-02-18 Thread Simon Richter
Hi,

On Tue, Feb 18, 2020 at 12:25:48PM +0100, Marco d'Itri wrote:

> > - Figure out a way for 32-bit binary-only programs to keep working when
> >   they touch a time_t beyond 2038.

> I am sure that around 2035 "time namespaces" or something similar will 
> be implemented to solve this.

Hopefully a bit earlier. I have a few pieces of software that only build in
virtual machines with a timezone offset to UTC of -10 years, and I'd really
like to move them to containers or light them on fire.

   Simon



Re: Y2038 - best way forward in Debian?

2020-02-18 Thread Marco d'Itri
On Feb 17, Wouter Verhelst  wrote:

> - Figure out a way for 32-bit binary-only programs to keep working when
>   they touch a time_t beyond 2038.
I am sure that around 2035 "time namespaces" or something similar will 
be implemented to solve this.

-- 
ciao,
Marco


signature.asc
Description: PGP signature


Re: Y2038 - best way forward in Debian?

2020-02-17 Thread Steve McIntyre
On Mon, Feb 17, 2020 at 09:03:22AM +0100, Wouter Verhelst wrote:
>On Sat, Feb 15, 2020 at 11:41:32PM +, Steve McIntyre wrote:
>> Hey John,
>> 
>> John Goarzen wrote:
>> >On Tue, Feb 04 2020, Steve McIntyre wrote:
>> >
>> >The thing that we have to remember is that an operating system is a
>> >platform for running software.  This problem is rather thorny, because:
>> >
>> >1) Some software is provided in only binary form and cannot be
>> >recompiled
>> 
>> Oh, absolutely. In that situation there's not a lot we can sensibly
>> do, modulo telling people to run such things in a time-shifted VM. I'm
>> more worried about making *our* software work in the future.
>
>This feels like a waste of effort, then. The only reason why people want
>to run i386 is "multiarch, because I have this old binary that won't go
>away". For all other stuff, there's amd64. Especially since RHEL doesn't
>even do i386 anymore these days, so ISVs will have to compile for amd64
>if they want it to work on whatever their customers run.
>
>In my opinion, there are really only two viable options:
>
>- Throw away the i386 port, and tell people that we no longer support
>  it;
>- Figure out a way for 32-bit binary-only programs to keep working when
>  they touch a time_t beyond 2038.

Right, that's it for our existing i386 port then. But we do have other
32-bit ports with different ecosystems and requirements - hence why I
started this thread.

-- 
Steve McIntyre, Cambridge, UK.st...@einval.com
"War does not determine who is right - only who is left."
   -- Bertrand Russell



Re: Y2038 - best way forward in Debian?

2020-02-17 Thread Wouter Verhelst
On Sat, Feb 15, 2020 at 11:41:32PM +, Steve McIntyre wrote:
> Hey John,
> 
> John Goarzen wrote:
> >On Tue, Feb 04 2020, Steve McIntyre wrote:
> >
> >The thing that we have to remember is that an operating system is a
> >platform for running software.  This problem is rather thorny, because:
> >
> >1) Some software is provided in only binary form and cannot be
> >recompiled
> 
> Oh, absolutely. In that situation there's not a lot we can sensibly
> do, modulo telling people to run such things in a time-shifted VM. I'm
> more worried about making *our* software work in the future.

This feels like a waste of effort, then. The only reason why people want
to run i386 is "multiarch, because I have this old binary that won't go
away". For all other stuff, there's amd64. Especially since RHEL doesn't
even do i386 anymore these days, so ISVs will have to compile for amd64
if they want it to work on whatever their customers run.

In my opinion, there are really only two viable options:

- Throw away the i386 port, and tell people that we no longer support
  it;
- Figure out a way for 32-bit binary-only programs to keep working when
  they touch a time_t beyond 2038.

Everything else feels like a waste of effort.

-- 
 Home is where you have to wash the dishes.
  -- #debian-devel, Freenode, 2004-09-22



Re: Y2038 - best way forward in Debian?

2020-02-16 Thread Adrian Bunk
On Fri, Feb 14, 2020 at 10:56:47AM -0600, John Goerzen wrote:
> 
> The thing that we have to remember is that an operating system is a
> platform for running software.  This problem is rather thorny, because:
> 
> 1) Some software is provided in only binary form and cannot be
> recompiled
> 
> 2) Some software can be recompiled but makes assumptions about the size
> of variables, may use int instead of time_t, and other assorted
> messiness

Cleanest would be to run such legacy programs inside some virtualization 
that also provides a virtual time long before 2038, using the last 
Debian release with the old architecture.

> 3) Some software is going to break now, due to forward-looking time
> calculations, and for others, it may be fine (or nearly so) even past
> 2038 due to timekeeping being only ancillary to its purpose.  For
> instance, I have some old games that are binary-only and really don't
> care what time it is.
>...

How confident are you that they work fine with negative time_t values?
Functions like time(2) return (time_t)-1 on error.

> John

cu
Adrian



Re: Y2038 - best way forward in Debian?

2020-02-15 Thread Steve McIntyre
anth...@derobert.net wrote:

...

>Lastly, 64-bit time_t has been tested widely (e.g., on amd64). That's not 
>perfect of
>course since 32-bit archs have smaller basic integer types, but likely a lot 
>less work
>fixing the stray "int"s than adding epochs all over the place. 
>
>PS: Debian-devel is likely the wrong place to redesign C/POSIX functions. 

+1000

-- 
Steve McIntyre, Cambridge, UK.st...@einval.com
  Armed with "Valor": "Centurion" represents quality of Discipline,
  Honor, Integrity and Loyalty. Now you don't have to be a Caesar to
  concord the digital world while feeling safe and proud.



Re: Y2038 - best way forward in Debian?

2020-02-15 Thread Steve McIntyre
Hey John,

John Goarzen wrote:
>On Tue, Feb 04 2020, Steve McIntyre wrote:
>
>The thing that we have to remember is that an operating system is a
>platform for running software.  This problem is rather thorny, because:
>
>1) Some software is provided in only binary form and cannot be
>recompiled

Oh, absolutely. In that situation there's not a lot we can sensibly
do, modulo telling people to run such things in a time-shifted VM. I'm
more worried about making *our* software work in the future.

>2) Some software can be recompiled but makes assumptions about the size
>of variables, may use int instead of time_t, and other assorted
>messiness

Nod. I'm sure we'll find quite a bit of that, and need to file (and
maybe fix) those bugs. Hell, we're also likely to find some places
where we won't be able to sensibly fix things. But it's better to know
and document those places than not.

>3) Some software is going to break now, due to forward-looking time
>calculations, and for others, it may be fine (or nearly so) even past
>2038 due to timekeeping being only ancillary to its purpose.  For
>instance, I have some old games that are binary-only and really don't
>care what time it is.

Yup.

>This option #1 sounds like a significant effort (because not only would
>we need two versions of libraries, but also of include files).  But it
>certainly passes the "correctness" test better than your option #2.

Sure. It comes down to how long we think we can / want to spend on
this. I'd *hope* that a lot of software *will* work correctly with a
rebootstrap, but of course we know it won't be 100%. I'm concerned
that we don't leave it too long to get on with fixing things - that
will continue to build an ever-growing corpus of broken systems that
people will need to deal with later.

-- 
Steve McIntyre, Cambridge, UK.st...@einval.com
  Armed with "Valor": "Centurion" represents quality of Discipline,
  Honor, Integrity and Loyalty. Now you don't have to be a Caesar to
  concord the digital world while feeling safe and proud.



Crossgrading support (was Re: Y2038 - best way forward in Debian?)

2020-02-15 Thread Steve McIntyre
Adam Borowski wrote:
>On Wed, Feb 12, 2020 at 06:10:53PM +, Steve McIntyre wrote:
>> 
>> /me ponders - this could be a fun task for a GSoC/outreachy student to
>> hack on, maybe?
>
>Prior art:
>
>* my unfinished https://github.com/kilobyte/crossgrade -- does a lot of
>  error checking and continue-after-problem, but currently stops after
>  crossgrading the essential set
>
>* stuff linked from https://wiki.debian.org/CrossGrading

Ah, cool!

>Problems:
>
>* crossgrades fail _badly_ in a hard to recover way if packages for the
>  two architectures come in different versions (including binNMUs).  This
>  hurts especially if -ports archs are involved.

Yup. Probably best done on top of a stable release, by default, to
avoid the worst of this,

>* arch-specific or local packages are not as bad but 1. crossgrade must
>  know what's safe to remove, 2. what should be left unchanged (and their
>  recursive deps unremoved), 3. there may be non-multiarchable packages
>  within 1. or 2.'s dependency chain

Yup. So I think it would be good to have a tool which can work through
the existing state of a system up-front and analyse what's likely to
work or not.

>* systemd can't handle being crossgraded (I'm a systemd hater, but same was
>  noticed by eg. anarcat and Simon Richter).  On a minimal system it may
>  survive but usually it starts killing daemons (including X), unmounting
>  stuff, choking and forcing an unclean reboot, etc.  This could be worked
>  around by:
>  • switching to sysvinit
>  • booting from a different media then chrooting to crossgrade (not for
>ordinary users unless it's eg. scripted from d-i)
>  • having a package bring a grub entry that boots with
>init=/usr/sbin/crossgrade to do the thing?

ACK. Booting in a *simple* state is likely to be key.

-- 
Steve McIntyre, Cambridge, UK.st...@einval.com
  Armed with "Valor": "Centurion" represents quality of Discipline,
  Honor, Integrity and Loyalty. Now you don't have to be a Caesar to
  concord the digital world while feeling safe and proud.



Re: Y2038 - best way forward in Debian?

2020-02-14 Thread Anthony DeRobertis



On February 14, 2020 7:45:30 PM UTC, Alan Corey  wrote:
>What if we define an epoch to be 50 years and the epoch number becomes
>part of how the computer keeps track of the date.  Something similar
>is done in astronomy I think, star charts always have an epoch.  So
>epoch 0 was 1970, epoch 1 is 2000, epoch 2 is 2050.   Then we can keep
>a time_t at 32 bits.  Things like strptime() and strftime() and
>ctime() would need changing but since they were valid in epoch 0
>they'd only need to change for epoch 1 and later. 

Everything that handles time would need to change. Just think of the fun of 
"what's one year from today?" when you cross an epic. Far easier to just make 
time_t 64-bit.

And file formats and protocols that currently use time_t would need to be 
adjusted to add a second field for an epoch. Same incompatibility concerns as 
64-bit time_t. 

Not to mention that the Gregorian calendar leap year cycle is 400 years long, 
so if you ignore the epic, you'll have no idea if it's a leap year. Which also 
means that you have no idea how long your 50-year epic is, since they vary, 
they're not all the same number of seconds, or even days.

Ultimately, you'd wind up with a presumably int-sized epoch plus an int sized 
time_t. That's 64 bits still...

Lastly, 64-bit time_t has been tested widely (e.g., on amd64). That's not 
perfect of course since 32-bit archs have smaller basic integer types, but 
likely a lot less work fixing the stray "int"s than adding epochs all over the 
place. 

PS: Debian-devel is likely the wrong place to redesign C/POSIX functions. 



Re: Y2038 - best way forward in Debian?

2020-02-14 Thread Alan Corey
What if we define an epoch to be 50 years and the epoch number becomes
part of how the computer keeps track of the date.  Something similar
is done in astronomy I think, star charts always have an epoch.  So
epoch 0 was 1970, epoch 1 is 2000, epoch 2 is 2050.   Then we can keep
a time_t at 32 bits.  Things like strptime() and strftime() and
ctime() would need changing but since they were valid in epoch 0
they'd only need to change for epoch 1 and later.  A day will continue
to be 86400 seconds of a 32-bit number but every 50 years the epoch
would change.  Dates can overlap epochs so 2020 could be part of epoch
0 and epoch 1.  The year part of a time_t and struct tm ceases to be
absolute like an hour isn't absolute on a 12-hour clock.  It's
probably been thought of.

On 2/14/20, John Goerzen  wrote:
> On Tue, Feb 04 2020, Steve McIntyre wrote:
>
>> Arnd scanned the library packages in the Debian archive and identified
>> that about one third of our library packages would need rebuilding
>> (and tracking) to make a (recursive) transition. We can see two
>> different possible routes to follow:
>>
>>  A Follow a similar path to last time (rename library packages). This
>>will allow us to do partial upgrades, but the cost is that a vast
>>number of packages will need work to make this happen,
>>*potentially* building library packages twice to allow us to
>>continue both 32-bit and 64-bit time_t support forwards for a
>>while. This effort will be *needed* only for the sake of our 32-bit
>>ports, but would affect *everybody*.
>
> The thing that we have to remember is that an operating system is a
> platform for running software.  This problem is rather thorny, because:
>
> 1) Some software is provided in only binary form and cannot be
> recompiled
>
> 2) Some software can be recompiled but makes assumptions about the size
> of variables, may use int instead of time_t, and other assorted
> messiness
>
> 3) Some software is going to break now, due to forward-looking time
> calculations, and for others, it may be fine (or nearly so) even past
> 2038 due to timekeeping being only ancillary to its purpose.  For
> instance, I have some old games that are binary-only and really don't
> care what time it is.
>
> This option #1 sounds like a significant effort (because not only would
> we need two versions of libraries, but also of include files).  But it
> certainly passes the "correctness" test better than your option #2.
>
> John
>
>


-- 
-
No, I won't  call it "climate change", do you have a "reality problem"? - AB1JX
Cities are cages built to contain excess people and keep them from
cluttering up nature.
Impeach  Impeach  Impeach  Impeach  Impeach  Impeach  Impeach  Impeach



Re: Y2038 - best way forward in Debian?

2020-02-14 Thread John Goerzen
On Tue, Feb 04 2020, Steve McIntyre wrote:

> Arnd scanned the library packages in the Debian archive and identified
> that about one third of our library packages would need rebuilding
> (and tracking) to make a (recursive) transition. We can see two
> different possible routes to follow:
>
>  A Follow a similar path to last time (rename library packages). This
>will allow us to do partial upgrades, but the cost is that a vast
>number of packages will need work to make this happen,
>*potentially* building library packages twice to allow us to
>continue both 32-bit and 64-bit time_t support forwards for a
>while. This effort will be *needed* only for the sake of our 32-bit
>ports, but would affect *everybody*.

The thing that we have to remember is that an operating system is a
platform for running software.  This problem is rather thorny, because:

1) Some software is provided in only binary form and cannot be
recompiled

2) Some software can be recompiled but makes assumptions about the size
of variables, may use int instead of time_t, and other assorted
messiness

3) Some software is going to break now, due to forward-looking time
calculations, and for others, it may be fine (or nearly so) even past
2038 due to timekeeping being only ancillary to its purpose.  For
instance, I have some old games that are binary-only and really don't
care what time it is.

This option #1 sounds like a significant effort (because not only would
we need two versions of libraries, but also of include files).  But it
certainly passes the "correctness" test better than your option #2.

John



Re: Y2038 - best way forward in Debian?

2020-02-13 Thread YunQiang Su
Steve McIntyre  于2020年2月14日周五 上午12:20写道:
>
> YunQiang Su wrote:
> >Ansgar  于2020年2月13日周四 下午5:29写道:
> >>
> >> For arm* and mips*, we mostly seem to be talking about special-purpose
> >> systems where just switching to a new architecture/port doesn't seem to
> >> be that much as a problem as for i386.  I think rebuilding the world and
> >> breaking ABI might thus be acceptable there.
> >>
> >> i386 seems different.  I think option C above would be the only
> >> realistic proposal so far to fix the time_t problem for (parts of) i386,
> >> but if glibc upstream doesn't want to expose two interfaces then i386
> >> will probably just break.
> >>
> >
> >just redefine time_t to 64bit may also cause a problem:
> >   a bad designed and old network protocol which aims  only target 32bit 
> > system,
> >   a binary data packet, may contain time_t:
> >struct {
> >int a;
> >time_t b;
> >}
> >just define time_t to 64 will break this protocol, although it is bad 
> >designed.
>
> Oh, sure. We'll find bugs like this, guaranteed.
>
> >Currently, the major task of 32bit ports is to keep compatible with
> >old system/binary.
> >Should we really want to break them?
>
> Well, that's the question. AIUI people seem to be wanting to keep i386
> as-is, due to the existing ecosystem of binaries (both free and
> proprietary), and I've not seen anybody really saying that i386 needs
> to live beyond 2038.
>
> armhf is different, and we want to fix it (/replace it with
> armhf_) with a 64-bit clean ABI. Where do the other existing
> 32-bit ports sit?
>
>  * armel? anybody want to chime in?
>  * mipsel?

For mipsel, I prefer keeping align with i386.

>
> I'd like to start making decisions *soon* on what we want to do, so we
> can start work. I'm *hoping* that we might be able to get a new armhf
> port done and released with bullseye, but that's clearly up to the
> release team to make a call on. The longer we leave things, the harder
> that target will be.
>
> --
> Steve McIntyre, Cambridge, UK.st...@einval.com
>   Armed with "Valor": "Centurion" represents quality of Discipline,
>   Honor, Integrity and Loyalty. Now you don't have to be a Caesar to
>   concord the digital world while feeling safe and proud.
>


-- 
YunQiang Su



Re: Y2038 - best way forward in Debian?

2020-02-13 Thread Michael Stone

On Thu, Feb 13, 2020 at 04:08:22PM +0100, Andrej Shadura wrote:

On Thu, 13 Feb 2020, 10:40 YunQiang Su,  wrote:
   just redefine time_t to 64bit may also cause a problem:
      a bad designed and old network protocol which aims  only target 32bit
   system,
      a binary data packet, may contain time_t:
           struct {
               int a;
               time_t b;
           }
   just define time_t to 64 will break this protocol, although it is bad
   designed.

   Currently, the major task of 32bit ports is to keep compatible with
   old system/binary.
   Should we really want to break them?


Aren't such things already broken on amd64?


If they're trying to talk, then yes. On-disk structures are probably 
much more common than network protocols and not necessarily already 
addressed. Even stuff like last(1) or who(1) would need changes if you 
redefine the size of a time_t. And these aren't simple changes--files 
which weren't intended to be portable don't generally store anything 
about how big their time_t is, so you need heuristics to figure that out 
and then make some tough decisions about how to proceed. (E.g., in the 
case of wtmp you'd probably not want to just start writing larger 
records halfway through a file, but you also have a very large number of 
programs that can potentially write those records and upgrading them 
simultaneously is impossible. A file with mixed record sizes is probably 
not 100% recoverable.) 



Re: Y2038 - best way forward in Debian?

2020-02-13 Thread Michael Stone

On Thu, Feb 13, 2020 at 10:29:35AM +0100, Ansgar wrote:

For arm* and mips*, we mostly seem to be talking about special-purpose
systems where just switching to a new architecture/port doesn't seem to
be that much as a problem as for i386.  I think rebuilding the world and
breaking ABI might thus be acceptable there.


Historically those systems have had major issues with upgrades due to 
things like proprietary kernel patches that aren't available for newer 
releases. In practical terms, would trying to avoid a new architecture 
provide much benefit (especially in terms of the amount of work it would 
take)? I suspect that if you have a 25 year old embedded system in 2038, 
the availability of a 64 bit time interface in the C library of a 
general purpose linux distribution that's compiled for your CPU's 
instruction set is going to be the least of your maintenance problems.



i386 seems different.


i386 should just stay a relic. I don't see any use case for i386 in 2038 
that doesn't involve trying to run a binary that can't be recompiled, 
and the only solution for that involves virtualization of the old ABI in 
a time-shifted environment.




Re: Y2038 - best way forward in Debian?

2020-02-13 Thread Adam Borowski
On Wed, Feb 12, 2020 at 06:10:53PM +, Steve McIntyre wrote:
> Russ Allbery wrote:
> >If we go down this path, can we make cross-grading a supported feature for
> >the next stable release?  I'm sure I'm not the only one who is stuck with
> >continuously-upgraded i386 hosts who has been wanting to switch but has
> >been waiting until cross-grading is a little bit less scary.
> 
> ACK - I've heard quite a few people asking for this over the last few
> years. I've personally cross-graded a couple of systems (and my home
> server has moved twice, i386->amd64->arm64), but it's not for the
> faint-hearted.
> 
> /me ponders - this could be a fun task for a GSoC/outreachy student to
> hack on, maybe?

Prior art:

* my unfinished https://github.com/kilobyte/crossgrade -- does a lot of
  error checking and continue-after-problem, but currently stops after
  crossgrading the essential set

* stuff linked from https://wiki.debian.org/CrossGrading


Problems:

* crossgrades fail _badly_ in a hard to recover way if packages for the
  two architectures come in different versions (including binNMUs).  This
  hurts especially if -ports archs are involved.

* arch-specific or local packages are not as bad but 1. crossgrade must
  know what's safe to remove, 2. what should be left unchanged (and their
  recursive deps unremoved), 3. there may be non-multiarchable packages
  within 1. or 2.'s dependency chain

* systemd can't handle being crossgraded (I'm a systemd hater, but same was
  noticed by eg. anarcat and Simon Richter).  On a minimal system it may
  survive but usually it starts killing daemons (including X), unmounting
  stuff, choking and forcing an unclean reboot, etc.  This could be worked
  around by:
  • switching to sysvinit
  • booting from a different media then chrooting to crossgrade (not for
ordinary users unless it's eg. scripted from d-i)
  • having a package bring a grub entry that boots with
init=/usr/sbin/crossgrade to do the thing?


Meow!
-- 
⢀⣴⠾⠻⢶⣦⠀ .globl _start↵.data↵rc: .ascii "/etc/init.d/rcS\0"↵.text↵_start
⣾⠁⢰⠒⠀⣿⡁ mov $57,%rax↵syscall↵cmp $0,%rax↵jne child↵parent:↵mov $61,%rax
⢿⡄⠘⠷⠚⠋⠀ mov $-1,%rdi↵xor %rsi,%rsi↵xor %rdx,%rdx↵syscall↵jmp parent↵child:
⠈⠳⣄ mov $59,%rax↵mov $rc,%rdi↵xor %rsi,%rsi↵xor %rdx,%rdx↵syscall



Re: Y2038 - best way forward in Debian?

2020-02-13 Thread Adam Borowski
On Thu, Feb 13, 2020 at 05:40:29PM +0800, YunQiang Su wrote:
> just redefine time_t to 64bit may also cause a problem:
>a bad designed and old network protocol which aims  only target 32bit 
> system,
>a binary data packet, may contain time_t:
> struct {
> int a;
> time_t b;
> }
> just define time_t to 64 will break this protocol, although it is bad 
> designed.
> 
> Currently, the major task of 32bit ports is to keep compatible with
> old system/binary.
> Should we really want to break them?

Proposal D:
on 32-bit (except x32 which already has 64-bit time_t):
typedef time_t unsigned long;

That'll keep the ABI intact yet will kick the problem 68 years into the
future.  By year 2106, folks will be used to using qemu/equivalent for
running 20th century stuff...

Some programs may still break (if they cast time_t to int or signed long),
but most should either work ok or get only display wrong.

I sometimes still play Master of Orion 2, and seeing saves dated 19120 is
not that bad. :p


Meow!
-- 
⢀⣴⠾⠻⢶⣦⠀
⣾⠁⢠⠒⠀⣿⡁ 
⢿⡄⠘⠷⠚⠋⠀ A white dwarf seeks a red giant for a binary relationship.
⠈⠳⣄



Re: Y2038 - best way forward in Debian?

2020-02-13 Thread Steve McIntyre
Andrej Shadura wrote:
>On Thu, 13 Feb 2020, 10:40 YunQiang Su,  wrote:
>>
>> just redefine time_t to 64bit may also cause a problem:
>>a bad designed and old network protocol which aims  only target 32bit
>> system,
>>a binary data packet, may contain time_t:
>> struct {
>> int a;
>> time_t b;
>> }
>> just define time_t to 64 will break this protocol, although it is bad
>> designed.
>
>Aren't such things already broken on amd64?

Of course they are, yes. We may have cases where nobody has tested
interop between 32 and 64 bit machines, though. :-/

-- 
Steve McIntyre, Cambridge, UK.st...@einval.com
  Armed with "Valor": "Centurion" represents quality of Discipline,
  Honor, Integrity and Loyalty. Now you don't have to be a Caesar to
  concord the digital world while feeling safe and proud.



Re: Y2038 - best way forward in Debian?

2020-02-13 Thread Steve McIntyre
YunQiang Su wrote:
>Ansgar  于2020年2月13日周四 下午5:29写道:
>>
>> For arm* and mips*, we mostly seem to be talking about special-purpose
>> systems where just switching to a new architecture/port doesn't seem to
>> be that much as a problem as for i386.  I think rebuilding the world and
>> breaking ABI might thus be acceptable there.
>>
>> i386 seems different.  I think option C above would be the only
>> realistic proposal so far to fix the time_t problem for (parts of) i386,
>> but if glibc upstream doesn't want to expose two interfaces then i386
>> will probably just break.
>>
>
>just redefine time_t to 64bit may also cause a problem:
>   a bad designed and old network protocol which aims  only target 32bit 
> system,
>   a binary data packet, may contain time_t:
>struct {
>int a;
>time_t b;
>}
>just define time_t to 64 will break this protocol, although it is bad designed.

Oh, sure. We'll find bugs like this, guaranteed.

>Currently, the major task of 32bit ports is to keep compatible with
>old system/binary.
>Should we really want to break them?

Well, that's the question. AIUI people seem to be wanting to keep i386
as-is, due to the existing ecosystem of binaries (both free and
proprietary), and I've not seen anybody really saying that i386 needs
to live beyond 2038.

armhf is different, and we want to fix it (/replace it with
armhf_) with a 64-bit clean ABI. Where do the other existing
32-bit ports sit?

 * armel? anybody want to chime in?
 * mipsel?

I'd like to start making decisions *soon* on what we want to do, so we
can start work. I'm *hoping* that we might be able to get a new armhf
port done and released with bullseye, but that's clearly up to the
release team to make a call on. The longer we leave things, the harder
that target will be.

-- 
Steve McIntyre, Cambridge, UK.st...@einval.com
  Armed with "Valor": "Centurion" represents quality of Discipline,
  Honor, Integrity and Loyalty. Now you don't have to be a Caesar to
  concord the digital world while feeling safe and proud.



Re: Y2038 - best way forward in Debian?

2020-02-13 Thread Steve McIntyre
Florian Weimer wrote:
>* Steve McIntyre:
>
>>>In addition if we are using a new multiarch triplet, and need to
>>>rebuild the world, are going to be ABI incompatible anyway, we might
>>>as well use a proper multiarch-qualified ld.so pathname that does
>>>not collide with anything.
>>
>> Hmmm. Moving ld.so is *hard* - we were already bitten by stuff here
>> when we bootstrapped armhf initially. What we didn't know then (but
>> know now!) is that the final element of the path (i.e. the filename)
>> must be globally unique for glibc's code to work. We can't (for
>> example) just move ld-linux-armhf.so.3 to a new directory, we'd have
>> to rename the file itself. (Apologies if this is stuff you already
>> know - I think it's worth explaining for others too!)
>
>These changes also require updates to the ABI manual and upstream
>patches to the toolchain.

Nod. (and "yay" - been there, done that before...)

>I don't think Debian's multi-arch path changes have been properly
>upstreamed yet, either.  Of course it does not help that key parts of
>the GNU toolchain are more or less dead upstream; the lack of an
>autoconf release is a real problem.

ACK. :-(

-- 
Steve McIntyre, Cambridge, UK.st...@einval.com
  Armed with "Valor": "Centurion" represents quality of Discipline,
  Honor, Integrity and Loyalty. Now you don't have to be a Caesar to
  concord the digital world while feeling safe and proud.



Re: Y2038 - best way forward in Debian?

2020-02-13 Thread Florian Weimer
* Steve McIntyre:

>>In addition if we are using a new multiarch triplet, and need to
>>rebuild the world, are going to be ABI incompatible anyway, we might
>>as well use a proper multiarch-qualified ld.so pathname that does
>>not collide with anything.
>
> Hmmm. Moving ld.so is *hard* - we were already bitten by stuff here
> when we bootstrapped armhf initially. What we didn't know then (but
> know now!) is that the final element of the path (i.e. the filename)
> must be globally unique for glibc's code to work. We can't (for
> example) just move ld-linux-armhf.so.3 to a new directory, we'd have
> to rename the file itself. (Apologies if this is stuff you already
> know - I think it's worth explaining for others too!)

These changes also require updates to the ABI manual and upstream
patches to the toolchain.

I don't think Debian's multi-arch path changes have been properly
upstreamed yet, either.  Of course it does not help that key parts of
the GNU toolchain are more or less dead upstream; the lack of an
autoconf release is a real problem.



Re: Y2038 - best way forward in Debian?

2020-02-13 Thread YunQiang Su
Ansgar  于2020年2月13日周四 下午5:29写道:
>
> Steve McIntyre writes:
> > wzss...@gmail.com wrote:
> >>Is there the option C?
> >>
> >>Draft:
> >>  1. define time64_t
> >>  2. for data_struct which act as a part of ABI, define a new data_struct64
> >>  3. for function which act as part of ABI, define a new version func64.
> >>  4. patch all packages to use time64_t instead of time_t step by step.
> >>  5. set a time as deadline (2030?), and then treat all packages
> >>haven't finished
> >>  the migration as rc-buggy.
> >>
> >> Since we have enough time, we can patch all packages in that period.
> >
> > It's possible, but...
> >
> > The problem here is that we have many thousands of packages to work
> > on, glibc up through other libs to all the applications that use
> > them. It's invasive work, and likely to take a very long time. Since
> > it's work that would *not* be needed for 64-bit architectures, we're
> > also likely to see push-back from some upstreams.
> >
> > 2030 is also too late to fix the problem - people are already starting
> > to see issues in some applications. We want to make a difference soon:
> > the earlier that we have fixes available, the fewer broken systems
> > will have to be fixed / removed / replaced later.
>
> For arm* and mips*, we mostly seem to be talking about special-purpose
> systems where just switching to a new architecture/port doesn't seem to
> be that much as a problem as for i386.  I think rebuilding the world and
> breaking ABI might thus be acceptable there.
>
> i386 seems different.  I think option C above would be the only
> realistic proposal so far to fix the time_t problem for (parts of) i386,
> but if glibc upstream doesn't want to expose two interfaces then i386
> will probably just break.
>

just redefine time_t to 64bit may also cause a problem:
   a bad designed and old network protocol which aims  only target 32bit system,
   a binary data packet, may contain time_t:
struct {
int a;
time_t b;
}
just define time_t to 64 will break this protocol, although it is bad designed.

Currently, the major task of 32bit ports is to keep compatible with
old system/binary.
Should we really want to break them?

> Ansgar
>


-- 
YunQiang Su



Re: Y2038 - best way forward in Debian?

2020-02-13 Thread Ansgar
Steve McIntyre writes:
> wzss...@gmail.com wrote:
>>Is there the option C?
>>
>>Draft:
>>  1. define time64_t
>>  2. for data_struct which act as a part of ABI, define a new data_struct64
>>  3. for function which act as part of ABI, define a new version func64.
>>  4. patch all packages to use time64_t instead of time_t step by step.
>>  5. set a time as deadline (2030?), and then treat all packages
>>haven't finished
>>  the migration as rc-buggy.
>>
>> Since we have enough time, we can patch all packages in that period.
>
> It's possible, but...
>
> The problem here is that we have many thousands of packages to work
> on, glibc up through other libs to all the applications that use
> them. It's invasive work, and likely to take a very long time. Since
> it's work that would *not* be needed for 64-bit architectures, we're
> also likely to see push-back from some upstreams.
>
> 2030 is also too late to fix the problem - people are already starting
> to see issues in some applications. We want to make a difference soon:
> the earlier that we have fixes available, the fewer broken systems
> will have to be fixed / removed / replaced later.

For arm* and mips*, we mostly seem to be talking about special-purpose
systems where just switching to a new architecture/port doesn't seem to
be that much as a problem as for i386.  I think rebuilding the world and
breaking ABI might thus be acceptable there.

i386 seems different.  I think option C above would be the only
realistic proposal so far to fix the time_t problem for (parts of) i386,
but if glibc upstream doesn't want to expose two interfaces then i386
will probably just break.

Ansgar



Re: Y2038 - best way forward in Debian?

2020-02-13 Thread Arnd Bergmann
On Wed, Feb 12, 2020 at 11:15 PM Steve McIntyre  wrote:
> Simon McVittie wrote:
> >On Tue, 04 Feb 2020 at 13:14:10 +, Steve McIntyre wrote:
> >> Arnd scanned the library packages in the Debian archive and identified
> >> that about one third of our library packages would need rebuilding
> >> (and tracking) to make a (recursive) transition.
> >
> >Is a list of affected/unaffected packages available? (I know they'll
> >be long lists.) Depending on how high up the stack they are, the right
> >approach might change.
>
> I don't have the list, I'm afraid. Arnd?

I lost both the script and its output after a crash unfortunately, but some
of it was captured in a mailing list post at

https://www.openwall.com/lists/musl/2019/07/02/2

It took me about a day to download all the -dev packages and scan through
the header files for known issues (time_t, struct timeval, struct timespec,
struct stat, struct timex), but it should not be hard to reproduce what
I did then.

The list obviously has both false positives (struct names in the headers
but not actually part of the ABI), and false negatives (indirect ABI changes
resulting from other types), but it's a good first approximation.

> >Thinking about i386 specifically, since the tradeoffs for that architecture
> >are likely to be a little different due to the prevalence of proprietary
> >binaries, the use-cases I am aware of are:
> >
> >- Old hardware that doesn't implement amd64.
> >  Breaking ABI would help here, but equally, this use-case seems likely
> >  to go away "naturally" in the next 10? years (and for example Ubuntu is
> >  already relegating i386 to a second-class-citizen status as a multiarch
> >  library stack on amd64 systems, with no kernel or full bootable OS).
> >
> >- amd64-capable machines that have inherited a legacy i386 installation
> >  because their owner doesn't want to reinstall or sidegrade to amd64.
> >  Breaking the i386 ABI seems like it would be counterproductive here:
> >  they'll still need to reinstall or sidegrade, and sidegrading from
> >  i386 to i386t would probably be very crash-prone, since those will
> >  presumably both have to be represented by ELF32 x86 binaries?
> >
> >- 32-bit Wine (to run 32-bit Windows programs).
> >  On the Unix side, breaking ABI would maybe help. On the Windows side,
> >  Wine needs to match whatever Microsoft does to solve this problem
> >  (apparently time_t already defaults to 64-bit on Windows, but older
> >  binaries will presumably use 32-bit time_t forever).
>
> Hmmm. I didn't think Microsoft used *time_t* as such, but had their
> own 64-bit format based on an epoch in ~1600?

Both the C standard and POSIX require time_t, so they do define one
with the POSIX semantics, but it's not that common and they changed
it to 64 bit a long time ago. Note that 'long' is also 32 bit on Win64,
so this was probably more urgent for them.

The 1600 (1604 IIRC) based epoch is used in NTFS, not sure where else.

> >- Running old proprietary or otherwise sourceless binaries (e.g. games).
> >  Breaking ABI is counterproductive: we can't recompile the games anyway.
> >  Some solutions for running these (e.g. the Steam Runtime used by Steam)
> >  rely on host libraries being approximately ABI-compatible with the ones
> >  the binaries were compiled against, because they need to fetch graphics
> >  drivers and their dependencies from the host system, otherwise they
> >  could not work on recent GPUs (although by 2038, with faster CPUs, we'll
> >  hopefully be able to run 2010s games at acceptable performance with
> >  software rendering like llvmpipe, which would sidestep this issue).
> >  (See the slides/video from my recent FOSDEM talk for more details)
>
> ACK. I'm in agreement here. I also personally don't see the same
> benefit to doing a 64-bit time_t transition for i386, using any of the
> models we've suggested. It's a legacy ABI that we should explicitly
> warn people will break. Have the Steam folks considered this issue at
> all yet, OOC?

I suspect for any particular software package, it should be fairly
straightforward to address the time64 conversion by inspecting
the libraries that are actually linked in, presumably steam is dealing
with dozens of dependencies, but not thousands of them that you
would see for a complete Debian port.

Similarly, Ubuntu and Redhat only ship a limited number of 32-bit
libraries these days, and that number may be shrinking further,
to the point where an exhaustive review of the interfaces makes
sense. For any particular library, there is probably a way to use
symbol versioning or some macro based abstraction to have it support
both time32 and time64 interfaces in the same binary.

   Arnd



Re: Y2038 - best way forward in Debian?

2020-02-12 Thread Steve McIntyre
Hey Simon,

Simon McVittie wrote:
>On Tue, 04 Feb 2020 at 13:14:10 +, Steve McIntyre wrote:
>> Arnd scanned the library packages in the Debian archive and identified
>> that about one third of our library packages would need rebuilding
>> (and tracking) to make a (recursive) transition.
>
>Is a list of affected/unaffected packages available? (I know they'll
>be long lists.) Depending on how high up the stack they are, the right
>approach might change.

I don't have the list, I'm afraid. Arnd?

>Thinking about i386 specifically, since the tradeoffs for that architecture
>are likely to be a little different due to the prevalence of proprietary
>binaries, the use-cases I am aware of are:
>
>- Old hardware that doesn't implement amd64.
>  Breaking ABI would help here, but equally, this use-case seems likely
>  to go away "naturally" in the next 10? years (and for example Ubuntu is
>  already relegating i386 to a second-class-citizen status as a multiarch
>  library stack on amd64 systems, with no kernel or full bootable OS).
>
>- amd64-capable machines that have inherited a legacy i386 installation
>  because their owner doesn't want to reinstall or sidegrade to amd64.
>  Breaking the i386 ABI seems like it would be counterproductive here:
>  they'll still need to reinstall or sidegrade, and sidegrading from
>  i386 to i386t would probably be very crash-prone, since those will
>  presumably both have to be represented by ELF32 x86 binaries?
>
>- 32-bit Wine (to run 32-bit Windows programs).
>  On the Unix side, breaking ABI would maybe help. On the Windows side,
>  Wine needs to match whatever Microsoft does to solve this problem
>  (apparently time_t already defaults to 64-bit on Windows, but older
>  binaries will presumably use 32-bit time_t forever).

Hmmm. I didn't think Microsoft used *time_t* as such, but had their
own 64-bit format based on an epoch in ~1600?

>- Running old proprietary or otherwise sourceless binaries (e.g. games).
>  Breaking ABI is counterproductive: we can't recompile the games anyway.
>  Some solutions for running these (e.g. the Steam Runtime used by Steam)
>  rely on host libraries being approximately ABI-compatible with the ones
>  the binaries were compiled against, because they need to fetch graphics
>  drivers and their dependencies from the host system, otherwise they
>  could not work on recent GPUs (although by 2038, with faster CPUs, we'll
>  hopefully be able to run 2010s games at acceptable performance with
>  software rendering like llvmpipe, which would sidestep this issue).
>  (See the slides/video from my recent FOSDEM talk for more details)

ACK. I'm in agreement here. I also personally don't see the same
benefit to doing a 64-bit time_t transition for i386, using any of the
models we've suggested. It's a legacy ABI that we should explicitly
warn people will break. Have the Steam folks considered this issue at
all yet, OOC?

armhf is a different kettle of fish, without the same set of legacy
binaries. There are *many* legacy proprietary binaries for the
architecture, but the only ones that really matter are on other
platforms (Android or iOS) and really not our problem here. :-)

-- 
Steve McIntyre, Cambridge, UK.st...@einval.com
  Armed with "Valor": "Centurion" represents quality of Discipline,
  Honor, Integrity and Loyalty. Now you don't have to be a Caesar to
  concord the digital world while feeling safe and proud.



Re: Y2038 - best way forward in Debian?

2020-02-12 Thread Steve McIntyre
[ Skimming through this - Arnd already responded ... ]

Guillem Jover wrote:
>On Tue, 2020-02-04 at 13:14:10 +, Steve McIntyre wrote:

...

>> So, we're all fine? Not so much: for our 32-bit Debian arches, we will
>> need to basically rebuild the world to be 2038-safe. When we had to do
>> something like this in the past, to deal with the libc5->libc6
>> transition, we had an SONAME change in libc to work with. We decided
>> to append the "g" tag to the names of our library binary packages to
>> signal that a library was built against libc6. We can still see some
>> of the effects of this in the archive, many years later
>> (e.g. zlib1g). The problem now is that we will *not* have an soname
>> change here to help identify code compatibility on the 32-bit front.
>
>Well, I guess such a new (conditinally selectable) name could be
>coordinated with glibc upstream? Say bump 32-bit ports to use libc6.1?
>(We already have precedent in some ports that do not use the same
>prevalent SONAME, such as libc6.1, libc0.1 and libc0.1.) But it would
>indeed involve lots of work, with massive amounts of package renames. :/

Nod. :-(

>>* users would *not* be able to simply upgrade from one to the
>>  other, due to lack of binary compatibility
>
>I think this conclusion stems from an incorrect premise. See below.
>
>>We'd need to decide exactly which of our 32-bit ports we would want
>>to do this path with (probably armhf->arhmft?, maybe
>>armel->armelt?, i386->i386t?. mipsel???). Due to the libc6 soname
>>continuity, we will most likely *not* end up with a different
>>visible ABI via the triplet and the runtime linker, so old/new
>>multi-arch will be impossible.
>
>A new dpkg architecture must use a different triplet, as these are
>required to be bijective. See “lpia” for a previous example of this.
>(This would then make it possible to cross-grade.)

ACK, that's feasible of course. It's not a *simple* upgrade, though.

>In addition if we are using a new multiarch triplet, and need to
>rebuild the world, are going to be ABI incompatible anyway, we might
>as well use a proper multiarch-qualified ld.so pathname that does
>not collide with anything.

Hmmm. Moving ld.so is *hard* - we were already bitten by stuff here
when we bootstrapped armhf initially. What we didn't know then (but
know now!) is that the final element of the path (i.e. the filename)
must be globally unique for glibc's code to work. We can't (for
example) just move ld-linux-armhf.so.3 to a new directory, we'd have
to rename the file itself. (Apologies if this is stuff you already
know - I think it's worth explaining for others too!)

>I also think we have a C option, which would be something like:
>
>  C Do a transition equivalent to the LFS one, by either switching
>libraries opportunistically to 64-bit time_t on their next SONAME
>bumps, or by updating them to provide 64-bit variants of their
>interfaces along their 32-bit ones, as done in glibc. Of course
>the main problem here is that the LFS "transition" is not one we
>should be very proud of, as it's been dragging on for a very long
>time, and it's not even close to be finished…
>
>
> 
>(Hmm there seems to be something borked with lintian.d.o, as the
>general tag numbers seem extremely low. :)

I don't think this helps very much though - it's the embedded copies
of "time_t" all the way up the library stack that will break...

-- 
Steve McIntyre, Cambridge, UK.st...@einval.com
  Armed with "Valor": "Centurion" represents quality of Discipline,
  Honor, Integrity and Loyalty. Now you don't have to be a Caesar to
  concord the digital world while feeling safe and proud.



Re: Y2038 - best way forward in Debian?

2020-02-12 Thread Steve McIntyre
Florian Weimer wrote:
>* Steve McIntyre:
>
>> The kernel is *basically* fixed now. Internally, data structures
>> should now be safe. There are a small number places where 32-bit time
>> is still a thing, but it's in hand. A number of syscalls, ioctls,
>> etc. have needed updates for the user-kernel interface level.
>
>XFS is the elephant in the room, though.

Oh, sure. There are a few other places that will break too, I'm sure.

>> glibc is the place that needs to care about most of this.
>
>I'm not so sure.  I really do not want glibc to parse and rewrite
>ioctl arguments and return values, or ancillary socket data.

Sorry - to be clear, I'm talking about in terms of the definitions of
various structures and library routines that are used further up the
stack. If we set our new 64-bit glibc to *always* expose 64-bit time_t
and use the right 64-bit syscalls here, then the vast majority of the
libraries and applications above it will just inherit the right
answers by default.

There will (of course!) be some other packages that will need work -
we'll need to look for those too but they're much smaller targets.

>> The glibc folks have taken an interesting approach.
>>
>>  * 64-bit ABIs/architectures are already using 64-bit time_t
>>throughout. The design is sane and so we should already be mostly
>>safe here, modulo silly code bugs (we'll always have those!)
>
>With the exception of utmp/utmpx, see __WORDSIZE_TIME64_COMPAT32 in
>the headers.  Unfortuantely, this also affects on-disk data
>structures.

ACK. :-(

>>  * 32-bit ABIs/arches are more awkward. glibc will continue *by
>>default* to use 32-bit time_t to keep compatibility with existing
>>code. This will *not* be safe as we approach 2038.
>>
>>  * 32-bit ABIs/arches *can* be told to use 64-bit time_t from glibc
>>upwards, but this will of course affect the ABI. Embedded uses of
>>time_t in libraries will change size, etc. This *will* be safe for
>>2038.
>>
>> So, we're all fine? Not so much: for our 32-bit Debian arches, we will
>> need to basically rebuild the world to be 2038-safe.
>
>The question is whether anyone wants an i386 port where this has
>happened.
>
>My opinion (professional in this case, even) is that i386 users want
>compatibility with their binaries from 1998.  Otherwise they would
>have rebuilt them for x86-64 by now.  Under this worldview, i386 is
>for backwards compatibility with existing software.  Users will want
>to run these old programs in a time namespace with shifted time, too.

Agreed!

>There is also substantial commercial interest in those old binaries
>(which becomes apparent when you break glibc compatibility with them
>8-/).
>
>>  B Decide which 32-bit ports we expect to care about by 2038, and
>>re-bootstrap new versions of those ports *with different
>>names*. This would allow most of our developers to ignore the
>>problem here (as 64-bit arches are not affected) and let a smaller
>>number of people re-bootstrap with new ABIs with 64-bit time_t
>>embedded. There are some downsides here:
>>
>>* we would end up with two versions of some ports for a short time
>>  - probably one release of overlap
>>
>>* users would *not* be able to simply upgrade from one to the
>>  other, due to lack of binary compatibility
>>
>>We *would* be able to run old and new ABIs on top of the same (new
>>enough) kernel (e.g. for buildds), so a lump of this would just be
>>simply building the new world and filing bugs where needed.
>>
>>We'd need to decide exactly which of our 32-bit ports we would want
>>to do this path with (probably armhf->arhmft?, maybe
>>armel->armelt?, i386->i386t?. mipsel???). Due to the libc6 soname
>>continuity, we will most likely *not* end up with a different
>>visible ABI via the triplet and the runtime linker, so old/new
>>multi-arch will be impossible.
>
>Yes, it really has to be B for i386 at least.
>
>The other issue is that the Y2038 work has tentacles everywhere.
>Porting to new architectures (whether they are 32-bit or 64-bit) will
>require additional changes to those applications which make direct
>system calls, in some cases even if they do not even use time_t or
>struct timespec with those system calls at all.  For example, new
>ports will not define __NR_futex, only __NR_futex_time64.

Nod.

-- 
Steve McIntyre, Cambridge, UK.st...@einval.com
  Armed with "Valor": "Centurion" represents quality of Discipline,
  Honor, Integrity and Loyalty. Now you don't have to be a Caesar to
  concord the digital world while feeling safe and proud.



Re: Y2038 - best way forward in Debian?

2020-02-12 Thread Steve McIntyre
Simon McVittie wrote:
>On Fri, 07 Feb 2020 at 09:28:24 +0200, Wouter Verhelst wrote:
>> Why not? This seems like the type of problem that SONAMEs are made for.
>> What am I missing?
>
>SONAMEs are set by the upstream developer in their build system (building
>the same source code produces the same SONAME), and are cross-distro
>comparable/compatible. This makes them good for representing ABI breaks
>that result from changes to the upstream source code (like deleting
>a deprecated function or changing a struct's members), but bad for
>representing ABI breaks that result from external factors like a toolchain
>or compilation-option change.
>
>We didn't/couldn't change SONAMEs for the C++11 std::string transition
>(g++ 5, "v5" suffix), which was a similar situation.

Nod, exactly. This involves tracking all the way up the library stack,
which makes it hard.

We *can* go the suffix route if preferred, but that's a *lot* of
effort - see proposal A...

-- 
Steve McIntyre, Cambridge, UK.st...@einval.com
  Armed with "Valor": "Centurion" represents quality of Discipline,
  Honor, Integrity and Loyalty. Now you don't have to be a Caesar to
  concord the digital world while feeling safe and proud.



Re: Y2038 - best way forward in Debian?

2020-02-12 Thread Steve McIntyre
Sam Hartman wrote:
>Steve, you're presuming that we would not create a new soname for libc6
>on architectures where we want a new time ABI.
>
>That's not at all obvious to me.
>It seems in the same level of drastic as the other options you are
>considering.
>So taking it off the table without discussion or consideration doesn't
>seem like a good call.

As Arnd has mentioned, it seems the (upstream) glibc maintainers are
not keen to change soname. The last thing we want is to diverge from
them (and hence other distros), so it's a discussion we would need to
have.

-- 
Steve McIntyre, Cambridge, UK.st...@einval.com
  Armed with "Valor": "Centurion" represents quality of Discipline,
  Honor, Integrity and Loyalty. Now you don't have to be a Caesar to
  concord the digital world while feeling safe and proud.



Re: Y2038 - best way forward in Debian?

2020-02-12 Thread Steve McIntyre
wzss...@gmail.com wrote:
>Steve McIntyre  于2020年2月4日周二 下午9:15写道:

...

>>  A Follow a similar path to last time (rename library packages). This
>>will allow us to do partial upgrades, but the cost is that a vast
>>number of packages will need work to make this happen,
>>*potentially* building library packages twice to allow us to
>>continue both 32-bit and 64-bit time_t support forwards for a
>>while. This effort will be *needed* only for the sake of our 32-bit
>>ports, but would affect *everybody*.
>>
>>  *** OR ***
>>
>>  B Decide which 32-bit ports we expect to care about by 2038, and
>>re-bootstrap new versions of those ports *with different
>>names*. This would allow most of our developers to ignore the
>>problem here (as 64-bit arches are not affected) and let a smaller
>>number of people re-bootstrap with new ABIs with 64-bit time_t
>>embedded. There are some downsides here:

...

>> So, which way should we go? My own personal gut feel matches Arnd's,
>> which would be route B. Some of us already have fair experience with
>> bootstrapping new arches, and we could almost just crank the handle
>> for most of this work.
>
>Is there the option C?
>
>Draft:
>  1. define time64_t
>  2. for data_struct which act as a part of ABI, define a new data_struct64
>  3. for function which act as part of ABI, define a new version func64.
>  4. patch all packages to use time64_t instead of time_t step by step.
>  5. set a time as deadline (2030?), and then treat all packages
>haven't finished
>  the migration as rc-buggy.
>
> Since we have enough time, we can patch all packages in that period.

It's possible, but...

The problem here is that we have many thousands of packages to work
on, glibc up through other libs to all the applications that use
them. It's invasive work, and likely to take a very long time. Since
it's work that would *not* be needed for 64-bit architectures, we're
also likely to see push-back from some upstreams.

2030 is also too late to fix the problem - people are already starting
to see issues in some applications. We want to make a difference soon:
the earlier that we have fixes available, the fewer broken systems
will have to be fixed / removed / replaced later.

-- 
Steve McIntyre, Cambridge, UK.st...@einval.com
  Armed with "Valor": "Centurion" represents quality of Discipline,
  Honor, Integrity and Loyalty. Now you don't have to be a Caesar to
  concord the digital world while feeling safe and proud.



Re: Y2038 - best way forward in Debian?

2020-02-12 Thread Steve McIntyre
Steve McIntyre wrote:
>Russ Allbery wrote:
>>
>>If we go down this path, can we make cross-grading a supported feature for
>>the next stable release?  I'm sure I'm not the only one who is stuck with
>>continuously-upgraded i386 hosts who has been wanting to switch but has
>>been waiting until cross-grading is a little bit less scary.
>
>ACK - I've heard quite a few people asking for this over the last few
>years. I've personally cross-graded a couple of systems (and my home
>server has moved twice, i386->amd64->arm64), but it's not for the
>faint-hearted.
>
>/me ponders - this could be a fun task for a GSoC/outreachy student to
>hack on, maybe?

https://wiki.debian.org/SummerOfCode2020/UnApprovedProjects/ArchitectureCrossGrade

-- 
Steve McIntyre, Cambridge, UK.st...@einval.com
  Armed with "Valor": "Centurion" represents quality of Discipline,
  Honor, Integrity and Loyalty. Now you don't have to be a Caesar to
  concord the digital world while feeling safe and proud.



Re: Y2038 - best way forward in Debian?

2020-02-12 Thread Steve McIntyre
Russ Allbery wrote:
>Ansgar  writes:
>
>> So maybe just recommend people to move to 64-bit architectures and put
>> 32-bit applications in a time namespace so they believe they are still
>> in 2001 ;-) 32-bit architectures will probably still be useful in
>> embedded contexts for a long time and there it might be easier to just
>> change the ABI, but for a general-purpose distribution we start seeing
>> more and more problems and I don't really see us supporting them as a
>> full architecture in 10+ years.
>
>If we go down this path, can we make cross-grading a supported feature for
>the next stable release?  I'm sure I'm not the only one who is stuck with
>continuously-upgraded i386 hosts who has been wanting to switch but has
>been waiting until cross-grading is a little bit less scary.

ACK - I've heard quite a few people asking for this over the last few
years. I've personally cross-graded a couple of systems (and my home
server has moved twice, i386->amd64->arm64), but it's not for the
faint-hearted.

/me ponders - this could be a fun task for a GSoC/outreachy student to
hack on, maybe?

-- 
Steve McIntyre, Cambridge, UK.st...@einval.com
  Armed with "Valor": "Centurion" represents quality of Discipline,
  Honor, Integrity and Loyalty. Now you don't have to be a Caesar to
  concord the digital world while feeling safe and proud.



Re: Y2038 - best way forward in Debian?

2020-02-11 Thread Arnd Bergmann
On Tue, Feb 11, 2020 at 9:10 PM Florian Weimer  wrote:
> * Ansgar:
> > Arnd Bergmann writes:
> >> On Mon, Feb 10, 2020 at 11:16 PM Florian Weimer  wrote:
> >>> There's going to be a _TIME_BITS selector, similar to
> >>> _FILE_OFFSET_BITS.
> >>>
> >>> There was a proposal to have only one API before, but I think the
> >>> agreement was that it wouldn't save much complexity.
> >>
> >> It should be easy to force the _TIME_BITS selection by patching the
> >> glibc headers in Debian though if we want.
> >>
> >> The problem with setting _TIME_BITS=64 only using dpkg-buildflags
> >> but leaving the libc default at 32 bits is that anything that users build
> >> themselves or that ignores the buildflags ends up with a broken ABI
> >> when linking against one of the many libraries that expose time_t
> >> in their ABI.
> >
> > It breaks the other way too:
> >
> > If you have an old library with a time_t in some public ABI, but an
> > application using it sets _TIME_BITS=64, the headers suddenly define a
> > different ABI from the one implemented by the compiled library.
> >
> > If you rebuild the library with _TIME_BITS=64, it changes ABI too and
> > breaks reverse dependencies (ignoring special handling like libc does
> > with versioned symbols). So you cannot just change it by default.
> >
> > I don't understand how this can work unless time_t is *never* exposed by
> > any library other than libc.

I think Steve explained this already in the initial email: in approach A,
rebuilding any library that exposes a modified time_t implies an incompatible
library update and updating anything that depends on it to be built with
the time64 interface as well, which quickly gets out of hand when you do this
recursively.

With approach B, there is no attempt at binary compatibility, instead
a new armhf_time64 port would use the same package names and
versions as the existing armhf port, but have an incompatible ABI
for anything exposing time_t based interfaces. Changing the target
triplet then allows a multiarch installation to coexist with armhf, the
same way that you can have armel and armhf coexisting today, with
an incompatible ABI.

> Correct.  We have this problem with off_t today, which is used in some
> header files.  The problem is particularly thorny because like
> off64_t, time64_t will never be a standard type, so some projects
> don't want to use it in headers.  And using time64_t would still be an
> ABI break on i386.

AFAIU time64_t is not even going to be exposed in the glibc headers
in the first place, you just pick which ABI you get when compiling an
application and hope it matches the other libraries.

Arnd



Re: Y2038 - best way forward in Debian?

2020-02-11 Thread Florian Weimer
* Ansgar:

> Arnd Bergmann writes:
>> On Mon, Feb 10, 2020 at 11:16 PM Florian Weimer  wrote:
>>> There's going to be a _TIME_BITS selector, similar to
>>> _FILE_OFFSET_BITS.
>>>
>>> There was a proposal to have only one API before, but I think the
>>> agreement was that it wouldn't save much complexity.
>>
>> It should be easy to force the _TIME_BITS selection by patching the
>> glibc headers in Debian though if we want.
>>
>> The problem with setting _TIME_BITS=64 only using dpkg-buildflags
>> but leaving the libc default at 32 bits is that anything that users build
>> themselves or that ignores the buildflags ends up with a broken ABI
>> when linking against one of the many libraries that expose time_t
>> in their ABI.
>
> It breaks the other way too:
>
> If you have an old library with a time_t in some public ABI, but an
> application using it sets _TIME_BITS=64, the headers suddenly define a
> different ABI from the one implemented by the compiled library.
>
> If you rebuild the library with _TIME_BITS=64, it changes ABI too and
> breaks reverse dependencies (ignoring special handling like libc does
> with versioned symbols). So you cannot just change it by default.
>
> I don't understand how this can work unless time_t is *never* exposed by
> any library other than libc.

Correct.  We have this problem with off_t today, which is used in some
header files.  The problem is particularly thorny because like
off64_t, time64_t will never be a standard type, so some projects
don't want to use it in headers.  And using time64_t would still be an
ABI break on i386.



Re: Y2038 - best way forward in Debian?

2020-02-11 Thread Arnd Bergmann
On Tue, Feb 11, 2020 at 12:07 PM Marco d'Itri  wrote:
>
> On Feb 11, Arnd Bergmann  wrote:
>
> > I agree that changing the i386 port is probably a bad idea at the moment,
> > let's see how the armhf port turns out and fix all the bugs first, as this
> > is clearly needed anyway. Once there is a working armhf version with
> > full time64 user space, there can be a separate discussion about what
> > to do with the i386 port (phase out i386 before y2038, migrate all of
> > i386 to time64 quickly, have two separate i386 ports, or something
> > else).
>
> I have still not seen any explanation about why we expect that 32 bit
> ARM systems will still be popular (as non-embedded) ~15 years from now.

I think most installations of Debian on 32-bit ARM are already for embedded
systems, but that doesn't make them less important IMHO. In some deeply
embedded systems, you'd be looking at installing a current version of Debian
(because why not) and then running it for decades beyond the end of support
without updates. While this is often no problem in the absence of attack
vectors, the time32 problem means that a piece of industrial equipment
may be created for a 40 year lifetime today and work flawlessly for the first
18 years before suddenly breaking. The sooner time64 gets supported in
Debian, the more of them have a  chance of surviving.

I did some research last year to see how long these systems tend to
live before they are obsolete, here is a rough summary for each architecture
version:

* ARMv4 based chips were released between ~1997 and 2007, the last
  supported released was Debian Lenny, originally in 2009, with support
  ending in 2012.
* ARMv4T based chips were more popular and came out during the
  same timeframe (1997 to 2007), last supported in Debian Stretch,
  released in 2017 and LTS support ending in 2022.
* ARMv5 was introduced in ~2002 and is still supported, with a few
  new chips still being taped out (Microchip SAM9X60, Allwinner
  F-Series) and being actively deployed in large numbers for several
  years to come. I expect the last Debian release with ARMv5 support
  between 2025 and 2030, with the last users relying on Debian LTS
  powering the hardware off well before 2038.
* ARMv6 was less popular, most chips (i.MX3, S3C64xx, OMAP2)
  from 2005 to 2014 are obsolete or never ran Debian in the first
  place (e.g. AST2500). The main exception is probably Raspbian
  on BCM2835, which surely will be used as long as you can
  fit Debian into 512MB of RAM, plus several years after support
  ends, probably beyond 2038.
* ARMv7 chips came out between 2005 and 2015 but were more
  popular than the above combined. Since Cortex-A5 is now part of
  Arm's DesignStart program, we will see new tapeouts for years to
  come, and some of them are likely to run Debian for longer than the
  existing chips.
* ARMv7VE (first introduced in 2012) is still the most common
  architecture for embedded designs today, we'll probably see new
  Cortex-A7 based chips for another 5 to 10 years, followed by several
  years of new boards based on those chips and a very long time
  before these finally get powered off.
* ARMv8 is surely taking over the lead in the embedded space from
  ARMv7/ARMv7VE over the next few years. Almost all of these
  can run 64-bit code, but when shipping enough of them, the cost
  of the on-board RAM means you are stuck with a single 4Gbit DDR3
  chip (512MB), which in turn means you want to run 32-bit user
  space to lower the memory consumption for the application.
  The next sensible step up from 512MB DDR3 (on a single chip) is
  probably 2GB of LP-DDR4 with 64-bit user space, but it will likely
  take many years before that is actually cheap enough completely
  obsolete DDR3 memory and 32-bit mode with it.

  Arnd



Re: Y2038 - best way forward in Debian?

2020-02-11 Thread Steve McIntyre
On Tue, Feb 11, 2020 at 12:01:28PM +0100, Marco d'Itri wrote:
>On Feb 11, Arnd Bergmann  wrote:
>
>> I agree that changing the i386 port is probably a bad idea at the moment,
>> let's see how the armhf port turns out and fix all the bugs first, as this
>> is clearly needed anyway. Once there is a working armhf version with
>> full time64 user space, there can be a separate discussion about what
>> to do with the i386 port (phase out i386 before y2038, migrate all of
>> i386 to time64 quickly, have two separate i386 ports, or something
>> else).
>I have still not seen any explanation about why we expect that 32 bit 
>ARM systems will still be popular (as non-embedded) ~15 years from now.

That's a fair question to ask!

We're already seeing quite a number of people embedding small-ish
ARMv7 boards in infrastructure, and AIUI a lot of them are using
Debian as a base already - see the Civil Infrastructure Platform as an
example: https://www.cip-project.org/

As these people are already engaging in ELTS work, I don't expect that
this setup is going to go away any time soon.

-- 
Steve McIntyre, Cambridge, UK.st...@einval.com
You raise the blade, you make the change... You re-arrange me 'til I'm sane...



Re: Y2038 - best way forward in Debian?

2020-02-11 Thread Marco d'Itri
On Feb 11, Arnd Bergmann  wrote:

> I agree that changing the i386 port is probably a bad idea at the moment,
> let's see how the armhf port turns out and fix all the bugs first, as this
> is clearly needed anyway. Once there is a working armhf version with
> full time64 user space, there can be a separate discussion about what
> to do with the i386 port (phase out i386 before y2038, migrate all of
> i386 to time64 quickly, have two separate i386 ports, or something
> else).
I have still not seen any explanation about why we expect that 32 bit 
ARM systems will still be popular (as non-embedded) ~15 years from now.

-- 
ciao,
Marco


signature.asc
Description: PGP signature


Re: Y2038 - best way forward in Debian?

2020-02-11 Thread Ansgar
Arnd Bergmann writes:
> On Mon, Feb 10, 2020 at 11:16 PM Florian Weimer  wrote:
>> There's going to be a _TIME_BITS selector, similar to
>> _FILE_OFFSET_BITS.
>>
>> There was a proposal to have only one API before, but I think the
>> agreement was that it wouldn't save much complexity.
>
> It should be easy to force the _TIME_BITS selection by patching the
> glibc headers in Debian though if we want.
>
> The problem with setting _TIME_BITS=64 only using dpkg-buildflags
> but leaving the libc default at 32 bits is that anything that users build
> themselves or that ignores the buildflags ends up with a broken ABI
> when linking against one of the many libraries that expose time_t
> in their ABI.

It breaks the other way too:

If you have an old library with a time_t in some public ABI, but an
application using it sets _TIME_BITS=64, the headers suddenly define a
different ABI from the one implemented by the compiled library.

If you rebuild the library with _TIME_BITS=64, it changes ABI too and
breaks reverse dependencies (ignoring special handling like libc does
with versioned symbols). So you cannot just change it by default.

I don't understand how this can work unless time_t is *never* exposed by
any library other than libc.

Ansgar



Re: Y2038 - best way forward in Debian?

2020-02-11 Thread Arnd Bergmann
On Mon, Feb 10, 2020 at 11:16 PM Florian Weimer  wrote:
>
> * Ben Hutchings:
>
> > On Sun, 2020-02-09 at 11:57 +0100, Florian Weimer wrote:
> >> * Ben Hutchings:
> >>
> >> > If I recall correctly, glibc *will* provide both entry points, so there
> >> > is no ABI break.  But the size of time_t (etc.) exposed through libc-
> >> > dev is fixed at glibc build time.
> >>
> >> Is this a Debian-specific decision?
> >
> > I though that was the *upstream* decision, but perhaps that's still not
> > decided after all?
>
> There's going to be a _TIME_BITS selector, similar to
> _FILE_OFFSET_BITS.
>
> There was a proposal to have only one API before, but I think the
> agreement was that it wouldn't save much complexity.

It should be easy to force the _TIME_BITS selection by patching the
glibc headers in Debian though if we want.

The problem with setting _TIME_BITS=64 only using dpkg-buildflags
but leaving the libc default at 32 bits is that anything that users build
themselves or that ignores the buildflags ends up with a broken ABI
when linking against one of the many libraries that expose time_t
in their ABI.

 Arnd



Re: Y2038 - best way forward in Debian?

2020-02-11 Thread Arnd Bergmann
On Fri, Feb 7, 2020 at 10:21 PM Florian Weimer  wrote:
>
> * Steve McIntyre:
>
> > The kernel is *basically* fixed now. Internally, data structures
> > should now be safe. There are a small number places where 32-bit time
> > is still a thing, but it's in hand. A number of syscalls, ioctls,
> > etc. have needed updates for the user-kernel interface level.
>
> XFS is the elephant in the room, though.

XFS is one of the main concerns for 64-bit kernels, but the patches
are likely to get merged in one of the next releases.

For 32-bit, all the ABI issues need to be solved first to have
a working system at all, so let's get that sorted. Obviously all
problems that 64-bit has also need to be solved on 32-bit.

> > glibc is the place that needs to care about most of this.
>
> I'm not so sure.  I really do not want glibc to parse and rewrite
> ioctl arguments and return values, or ancillary socket data.

For a new distro target, it's easy to mandate a minimum kernel
version that has all the ioctls. Users that are stuck on old kernels
for a particular hardware can keep using the existing armhf user
space until that is discontinued.

> >  * 32-bit ABIs/arches are more awkward. glibc will continue *by
> >default* to use 32-bit time_t to keep compatibility with existing
> >code. This will *not* be safe as we approach 2038.
> >
> >  * 32-bit ABIs/arches *can* be told to use 64-bit time_t from glibc
> >upwards, but this will of course affect the ABI. Embedded uses of
> >time_t in libraries will change size, etc. This *will* be safe for
> >2038.
> >
> > So, we're all fine? Not so much: for our 32-bit Debian arches, we will
> > need to basically rebuild the world to be 2038-safe.
>
> The question is whether anyone wants an i386 port where this has
> happened.
>
> My opinion (professional in this case, even) is that i386 users want
> compatibility with their binaries from 1998.  Otherwise they would
> have rebuilt them for x86-64 by now.  Under this worldview, i386 is
> for backwards compatibility with existing software.  Users will want
> to run these old programs in a time namespace with shifted time, too.

I agree that changing the i386 port is probably a bad idea at the moment,
let's see how the armhf port turns out and fix all the bugs first, as this
is clearly needed anyway. Once there is a working armhf version with
full time64 user space, there can be a separate discussion about what
to do with the i386 port (phase out i386 before y2038, migrate all of
i386 to time64 quickly, have two separate i386 ports, or something
else).

  Arnd



Re: Y2038 - best way forward in Debian?

2020-02-10 Thread Florian Weimer
* Ben Hutchings:

> On Sun, 2020-02-09 at 11:57 +0100, Florian Weimer wrote:
>> * Ben Hutchings:
>> 
>> > If I recall correctly, glibc *will* provide both entry points, so there
>> > is no ABI break.  But the size of time_t (etc.) exposed through libc-
>> > dev is fixed at glibc build time.
>> 
>> Is this a Debian-specific decision?
>
> I though that was the *upstream* decision, but perhaps that's still not
> decided after all?

There's going to be a _TIME_BITS selector, similar to
_FILE_OFFSET_BITS.

There was a proposal to have only one API before, but I think the
agreement was that it wouldn't save much complexity.



Re: Y2038 - best way forward in Debian?

2020-02-09 Thread Ben Hutchings
On Sun, 2020-02-09 at 11:57 +0100, Florian Weimer wrote:
> * Ben Hutchings:
> 
> > If I recall correctly, glibc *will* provide both entry points, so there
> > is no ABI break.  But the size of time_t (etc.) exposed through libc-
> > dev is fixed at glibc build time.
> 
> Is this a Debian-specific decision?

I though that was the *upstream* decision, but perhaps that's still not
decided after all?

Ben.

> There has been a proposal upstream not to support 32-bit time_t for
> new applications at all, but I don't think we will go in that
> direction.
-- 
Ben Hutchings
The world is coming to an end.  Please log off.




signature.asc
Description: This is a digitally signed message part


Re: Y2038 - best way forward in Debian?

2020-02-09 Thread Florian Weimer
* Ben Hutchings:

> If I recall correctly, glibc *will* provide both entry points, so there
> is no ABI break.  But the size of time_t (etc.) exposed through libc-
> dev is fixed at glibc build time.

Is this a Debian-specific decision?

There has been a proposal upstream not to support 32-bit time_t for
new applications at all, but I don't think we will go in that
direction.



Re: Y2038 - best way forward in Debian?

2020-02-07 Thread Florian Weimer
* Michael Stone:

> On Fri, Feb 07, 2020 at 02:46:19PM +0200, Wouter Verhelst wrote:
>>On Fri, Feb 07, 2020 at 10:31:16AM +, Simon McVittie wrote:
>>> On Fri, 07 Feb 2020 at 09:28:24 +0200, Wouter Verhelst wrote:
>>> > Why not? This seems like the type of problem that SONAMEs are made for.
>>> > What am I missing?
>>>
>>> SONAMEs are set by the upstream developer in their build system
>>
>>Yes, I am aware of that, thank you. I meant to say that this feels like
>>something upstream could do a SONAME bump for.
>>
>>I'm wondering why they chose not to.
>
> You'd have to bump the soname for almost every library on the system.

And make sure that upstreams introduce per-architecture sonames,
otherwise we have soname bumps all across the board on 64-bit
architectures, too.



Re: Y2038 - best way forward in Debian?

2020-02-07 Thread Florian Weimer
* Sam Hartman:

> Steve, you're presuming that we would not create a new soname for libc6
> on architectures where we want a new time ABI.

That seems to be a reasonable assumption because Debian would have to
use a different soname from upstream.  glibc upstream does not seem
likely to change soname for this at this point.



Re: Y2038 - best way forward in Debian?

2020-02-07 Thread Florian Weimer
* Steve McIntyre:

> The kernel is *basically* fixed now. Internally, data structures
> should now be safe. There are a small number places where 32-bit time
> is still a thing, but it's in hand. A number of syscalls, ioctls,
> etc. have needed updates for the user-kernel interface level.

XFS is the elephant in the room, though.

> glibc is the place that needs to care about most of this.

I'm not so sure.  I really do not want glibc to parse and rewrite
ioctl arguments and return values, or ancillary socket data.

> The glibc folks have taken an interesting approach.
>
>  * 64-bit ABIs/architectures are already using 64-bit time_t
>throughout. The design is sane and so we should already be mostly
>safe here, modulo silly code bugs (we'll always have those!)

With the exception of utmp/utmpx, see __WORDSIZE_TIME64_COMPAT32 in
the headers.  Unfortuantely, this also affects on-disk data
structures.

>  * 32-bit ABIs/arches are more awkward. glibc will continue *by
>default* to use 32-bit time_t to keep compatibility with existing
>code. This will *not* be safe as we approach 2038.
>
>  * 32-bit ABIs/arches *can* be told to use 64-bit time_t from glibc
>upwards, but this will of course affect the ABI. Embedded uses of
>time_t in libraries will change size, etc. This *will* be safe for
>2038.
>
> So, we're all fine? Not so much: for our 32-bit Debian arches, we will
> need to basically rebuild the world to be 2038-safe.

The question is whether anyone wants an i386 port where this has
happened.

My opinion (professional in this case, even) is that i386 users want
compatibility with their binaries from 1998.  Otherwise they would
have rebuilt them for x86-64 by now.  Under this worldview, i386 is
for backwards compatibility with existing software.  Users will want
to run these old programs in a time namespace with shifted time, too.

There is also substantial commercial interest in those old binaries
(which becomes apparent when you break glibc compatibility with them
8-/).

>  B Decide which 32-bit ports we expect to care about by 2038, and
>re-bootstrap new versions of those ports *with different
>names*. This would allow most of our developers to ignore the
>problem here (as 64-bit arches are not affected) and let a smaller
>number of people re-bootstrap with new ABIs with 64-bit time_t
>embedded. There are some downsides here:
>
>* we would end up with two versions of some ports for a short time
>  - probably one release of overlap
>
>* users would *not* be able to simply upgrade from one to the
>  other, due to lack of binary compatibility
>
>We *would* be able to run old and new ABIs on top of the same (new
>enough) kernel (e.g. for buildds), so a lump of this would just be
>simply building the new world and filing bugs where needed.
>
>We'd need to decide exactly which of our 32-bit ports we would want
>to do this path with (probably armhf->arhmft?, maybe
>armel->armelt?, i386->i386t?. mipsel???). Due to the libc6 soname
>continuity, we will most likely *not* end up with a different
>visible ABI via the triplet and the runtime linker, so old/new
>multi-arch will be impossible.

Yes, it really has to be B for i386 at least.

The other issue is that the Y2038 work has tentacles everywhere.
Porting to new architectures (whether they are 32-bit or 64-bit) will
require additional changes to those applications which make direct
system calls, in some cases even if they do not even use time_t or
struct timespec with those system calls at all.  For example, new
ports will not define __NR_futex, only __NR_futex_time64.



Re: Y2038 - best way forward in Debian?

2020-02-07 Thread Samuel Thibault
Wouter Verhelst, le ven. 07 févr. 2020 14:46:19 +0200, a ecrit:
> On Fri, Feb 07, 2020 at 10:31:16AM +, Simon McVittie wrote:
> > On Fri, 07 Feb 2020 at 09:28:24 +0200, Wouter Verhelst wrote:
> > > Why not? This seems like the type of problem that SONAMEs are made for.
> > > What am I missing?
> > 
> > SONAMEs are set by the upstream developer in their build system
> 
> Yes, I am aware of that, thank you. I meant to say that this feels like
> something upstream could do a SONAME bump for.
> 
> I'm wondering why they chose not to.

Because the ABI change is voluntary on the application build side. The
existing libc.so is ABI-compatible with previously-built applications.

Samuel



Re: Y2038 - best way forward in Debian?

2020-02-07 Thread Michael Stone

On Fri, Feb 07, 2020 at 02:46:19PM +0200, Wouter Verhelst wrote:

On Fri, Feb 07, 2020 at 10:31:16AM +, Simon McVittie wrote:

On Fri, 07 Feb 2020 at 09:28:24 +0200, Wouter Verhelst wrote:
> Why not? This seems like the type of problem that SONAMEs are made for.
> What am I missing?

SONAMEs are set by the upstream developer in their build system


Yes, I am aware of that, thank you. I meant to say that this feels like
something upstream could do a SONAME bump for.

I'm wondering why they chose not to.


You'd have to bump the soname for almost every library on the system.



Re: Y2038 - best way forward in Debian?

2020-02-07 Thread Wouter Verhelst
On Fri, Feb 07, 2020 at 10:31:16AM +, Simon McVittie wrote:
> On Fri, 07 Feb 2020 at 09:28:24 +0200, Wouter Verhelst wrote:
> > Why not? This seems like the type of problem that SONAMEs are made for.
> > What am I missing?
> 
> SONAMEs are set by the upstream developer in their build system

Yes, I am aware of that, thank you. I meant to say that this feels like
something upstream could do a SONAME bump for.

I'm wondering why they chose not to.

-- 
To the thief who stole my anti-depressants: I hope you're happy

  -- seen somewhere on the Internet on a photo of a billboard



Re: Y2038 - best way forward in Debian?

2020-02-07 Thread Simon McVittie
On Tue, 04 Feb 2020 at 13:14:10 +, Steve McIntyre wrote:
> Arnd scanned the library packages in the Debian archive and identified
> that about one third of our library packages would need rebuilding
> (and tracking) to make a (recursive) transition.

Is a list of affected/unaffected packages available? (I know they'll
be long lists.) Depending on how high up the stack they are, the right
approach might change.

Thinking about i386 specifically, since the tradeoffs for that architecture
are likely to be a little different due to the prevalence of proprietary
binaries, the use-cases I am aware of are:

- Old hardware that doesn't implement amd64.
  Breaking ABI would help here, but equally, this use-case seems likely
  to go away "naturally" in the next 10? years (and for example Ubuntu is
  already relegating i386 to a second-class-citizen status as a multiarch
  library stack on amd64 systems, with no kernel or full bootable OS).

- amd64-capable machines that have inherited a legacy i386 installation
  because their owner doesn't want to reinstall or sidegrade to amd64.
  Breaking the i386 ABI seems like it would be counterproductive here:
  they'll still need to reinstall or sidegrade, and sidegrading from
  i386 to i386t would probably be very crash-prone, since those will
  presumably both have to be represented by ELF32 x86 binaries?

- 32-bit Wine (to run 32-bit Windows programs).
  On the Unix side, breaking ABI would maybe help. On the Windows side,
  Wine needs to match whatever Microsoft does to solve this problem
  (apparently time_t already defaults to 64-bit on Windows, but older
  binaries will presumably use 32-bit time_t forever).

- Running old proprietary or otherwise sourceless binaries (e.g. games).
  Breaking ABI is counterproductive: we can't recompile the games anyway.
  Some solutions for running these (e.g. the Steam Runtime used by Steam)
  rely on host libraries being approximately ABI-compatible with the ones
  the binaries were compiled against, because they need to fetch graphics
  drivers and their dependencies from the host system, otherwise they
  could not work on recent GPUs (although by 2038, with faster CPUs, we'll
  hopefully be able to run 2010s games at acceptable performance with
  software rendering like llvmpipe, which would sidestep this issue).
  (See the slides/video from my recent FOSDEM talk for more details)

smcv



Re: Y2038 - best way forward in Debian?

2020-02-07 Thread Simon McVittie
On Fri, 07 Feb 2020 at 09:28:24 +0200, Wouter Verhelst wrote:
> Why not? This seems like the type of problem that SONAMEs are made for.
> What am I missing?

SONAMEs are set by the upstream developer in their build system (building
the same source code produces the same SONAME), and are cross-distro
comparable/compatible. This makes them good for representing ABI breaks
that result from changes to the upstream source code (like deleting
a deprecated function or changing a struct's members), but bad for
representing ABI breaks that result from external factors like a toolchain
or compilation-option change.

We didn't/couldn't change SONAMEs for the C++11 std::string transition
(g++ 5, "v5" suffix), which was a similar situation.

smcv



Re: Y2038 - best way forward in Debian?

2020-02-06 Thread Wouter Verhelst
On Tue, Feb 04, 2020 at 01:14:10PM +, Steve McIntyre wrote:
> So, we're all fine? Not so much: for our 32-bit Debian arches, we will
> need to basically rebuild the world to be 2038-safe. When we had to do
> something like this in the past, to deal with the libc5->libc6
> transition, we had an SONAME change in libc to work with. We decided
> to append the "g" tag to the names of our library binary packages to
> signal that a library was built against libc6. We can still see some
> of the effects of this in the archive, many years later
> (e.g. zlib1g). The problem now is that we will *not* have an soname
> change here to help identify code compatibility on the 32-bit front.

Why not? This seems like the type of problem that SONAMEs are made for.
What am I missing?

-- 
To the thief who stole my anti-depressants: I hope you're happy

  -- seen somewhere on the Internet on a photo of a billboard



Re: Y2038 - best way forward in Debian?

2020-02-06 Thread Arnd Bergmann
On Tue, Feb 4, 2020 at 4:03 PM Guillem Jover  wrote:
>
> On Tue, 2020-02-04 at 13:14:10 +, Steve McIntyre wrote:
> > The glibc folks have taken an interesting approach.
> >
> >  * 64-bit ABIs/architectures are already using 64-bit time_t
> >throughout. The design is sane and so we should already be mostly
> >safe here, modulo silly code bugs (we'll always have those!)
> >
> >  * 32-bit ABIs/arches are more awkward. glibc will continue *by
> >default* to use 32-bit time_t to keep compatibility with existing
> >code. This will *not* be safe as we approach 2038.
> >
> >  * 32-bit ABIs/arches *can* be told to use 64-bit time_t from glibc
> >upwards, but this will of course affect the ABI. Embedded uses of
> >time_t in libraries will change size, etc. This *will* be safe for
> >2038.
>
> I think there is still work pending in glibc to expose both 32-bit and
> 64-bit interfaces at the same time? (Currently it seems to be either-or.)

Correct. I was hoping this could be part of the next glibc release
(2.32), but if it takes longer than that, it may be necessary to start
the bootstrap with an inofficial patched version to find the bugs
and the rebuild everything against a proper release.

A new development that Steve did not mention is that musl
already has working time64 support in its git snapshots (release
will be soon), and Adelie Linux has a first release candidate
based on that. There is also a list of known issues at
https://wiki.adelielinux.org/wiki/Project:Time64 with things that
were either discover during the time64 bootstrap of Adelie
or that I found using Debian Codesearch. Most of these are
for packages that use low-level system calls directly rather than
going through glibc, either for syscalls that don't have an
abstraction (seccomp, futex, ...) or for implementing a runtime
environment for a language other than C.

> I'd like to use this for example in libbsd, to make a smooth
> transition similar to the one made in glibc, w/o needing to bump the
> SONAME.

This can clearly work for specific libraries, but I fear doing this across
all affected library packages adds so much extra work and time before
completion that there won't be many people left to use it by the time
the work is done.;-)

The problem is that new deployments of 32-bit systems only
become rarer over time, even though armhf is still the most common
platform in embedded deployments today, ahead of arm64.

I expect that out of the armhf systems still running in 2038 (or a
few years earlier, while the problems of future dated timestamps
start to build up), most of them will run software that is already
deployed (and broken) or being build in the next few years with a
chance of being fixed.

> > So, we're all fine? Not so much: for our 32-bit Debian arches, we will
> > need to basically rebuild the world to be 2038-safe. When we had to do
> > something like this in the past, to deal with the libc5->libc6
> > transition, we had an SONAME change in libc to work with. We decided
> > to append the "g" tag to the names of our library binary packages to
> > signal that a library was built against libc6. We can still see some
> > of the effects of this in the archive, many years later
> > (e.g. zlib1g). The problem now is that we will *not* have an soname
> > change here to help identify code compatibility on the 32-bit front.
>
> Well, I guess such a new (conditinally selectable) name could be
> coordinated with glibc upstream? Say bump 32-bit ports to use libc6.1?
> (We already have precedent in some ports that do not use the same
> prevalent SONAME, such as libc6.1, libc0.1 and libc0.1.) But it would
> indeed involve lots of work, with massive amounts of package renames. :/

The glibc developers have expressed in the past that they do not
consider this an ABI break and would continue to provide an extended
but compatible ABI with the same soname.

On the distro scale, I fear the opposite is true as Steve explained.

> >  B Decide which 32-bit ports we expect to care about by 2038, and
> >re-bootstrap new versions of those ports *with different
> >names*. This would allow most of our developers to ignore the
> >problem here (as 64-bit arches are not affected) and let a smaller
> >number of people re-bootstrap with new ABIs with 64-bit time_t
> >embedded. There are some downsides here:
>
> (If we are going for this, I'd say it might make sense to consider
> also enabling LFS, although that might be a dangerous change, see
> the lintian tag for the rationale. :)

Absolutely, this is already a done deal, as both musl and glibc do not
allow the combination of 32-bit off_t with 64-bit time_t. In musl,
both are now 64-bit for new binaries anyway, and in glibc setting
a 64-bit time_t implies LFS.

> >We'd need to decide exactly which of our 32-bit ports we would want
> >to do this path with (probably armhf->arhmft?, maybe
> >armel->armelt?, i386->i386t?. mipsel???). Due to the libc6 so

Re: Y2038 - best way forward in Debian?

2020-02-05 Thread Guillem Jover
On Wed, 2020-02-05 at 15:35:28 +0100, Ansgar wrote:
> On Wed, 2020-02-05 at 08:33 -0500, Sam Hartman wrote:
> > Steve, you're presuming that we would not create a new soname for
> > libc6 on architectures where we want a new time ABI.
> 
> Isn't the libc ABI for some reason part of Debian's architecture name? 
> uclibc-linux-amd64, musl-linux-i386, i386 are distict architectures
> after all.  So an incompatible newglibc-linux-i386 would be different
> from i386 as well?

Each libc needs its own architecture as that requires its own matching
C compiler, for the libc CRT, ld.so linker, some parts of the calling
convention, data types, or executable format ABI, etc. This is one
side of the architecture ABI, the other is coming from the CPU and
kernel ABIs, for example.

Given our current usage and definition of Debian architectures and
mapping to GNU triplets, the libc4 to libc5 (+ a.out to ELF migration),
or the libc5 to libc6 might have indeed demanded a new arch, as these
required new compilers (or a least different spec files) AFAIR and some
kind of hacked triplet-qualified hierarchies, but back then our
requirements for architecture definitions seems to have been more
primitive (first mostly mapping between CPU names, then CPU + kernel
names), we didn't have multiarch, and I guess a new arch (if it was even
considered, probably not) would have been deemed a cost too high, and
direct upgradability was probably way more important at the time.

Also the ABI tracked by the libc SONAME is a different part, and that
should not necessarily require a different compiler, otherwise you
could not build binaries against different SOVERSIONs for the same
libc.

I suppose the problem here is that the C compiler and C library are both
in charge of defining part of the architecture ABI, but at the same time
the C compiler and C library both also provide a set of shared libraries
with their own specific ABIs, so this all gets muddled into the same
basket.

Thanks,
Guillem



Re: Y2038 - best way forward in Debian?

2020-02-05 Thread Ansgar
On Wed, 2020-02-05 at 09:55 -0500, Sam Hartman wrote:
> > > > > > "Ansgar" == Ansgar   writes:
> 
> Ansgar> On Wed, 2020-02-05 at 08:33 -0500, Sam Hartman wrote:
> >> Steve, you're presuming that we would not create a new soname
> for
> >> libc6 on architectures where we want a new time ABI.
> 
> Ansgar> Isn't the libc ABI for some reason part of Debian's
> Ansgar> architecture name?  uclibc-linux-amd64, musl-linux-i386,
> Ansgar> i386 are distict architectures after all.  So an
> Ansgar> incompatible newglibc-linux-i386 would be different from
> Ansgar> i386 as well?
> 
> Not if they are coinstallable.
> As an example libc5 and libc6 were both on the i386 architecture.

Why wouldn't musl-linux-i386 be coinstallable with glibc-linux-i386?
Debian's musl package (on amd64) is coinstallable with libc6; we just
don't build anything[1] against alternative libraries.

One practical problem is that you don't want to end up linking two
different C libraries (or really: any core libraries).

I don't really see much difference between libc6 vs libc7 or libc6 vs
libmusl (besides applications that require glibc and might not work
with musl, but that is a separate issue).

This might also mean that musl-linux-i386 shouldn't be a different
architecture than i386.  If they are different, then maybe libstdc++6-
libc6-linux-i386 and libc++-libc6-linux-i386 would need to be different
architectures too?

Ansgar

  [1]: Yes, very few exceptions might exist.



Re: Y2038 - best way forward in Debian?

2020-02-05 Thread Sam Hartman
> "Ansgar" == Ansgar   writes:

Ansgar> On Wed, 2020-02-05 at 08:33 -0500, Sam Hartman wrote:
>> Steve, you're presuming that we would not create a new soname for
>> libc6 on architectures where we want a new time ABI.

Ansgar> Isn't the libc ABI for some reason part of Debian's
Ansgar> architecture name?  uclibc-linux-amd64, musl-linux-i386,
Ansgar> i386 are distict architectures after all.  So an
Ansgar> incompatible newglibc-linux-i386 would be different from
Ansgar> i386 as well?

Not if they are coinstallable.
As an example libc5 and libc6 were both on the i386 architecture.



Re: Y2038 - best way forward in Debian?

2020-02-05 Thread Ansgar
On Wed, 2020-02-05 at 08:33 -0500, Sam Hartman wrote:
> Steve, you're presuming that we would not create a new soname for
> libc6 on architectures where we want a new time ABI.

Isn't the libc ABI for some reason part of Debian's architecture name? 
uclibc-linux-amd64, musl-linux-i386, i386 are distict architectures
after all.  So an incompatible newglibc-linux-i386 would be different
from i386 as well?

Ansgar



Re: Y2038 - best way forward in Debian?

2020-02-05 Thread Sam Hartman
Steve, you're presuming that we would not create a new soname for libc6
on architectures where we want a new time ABI.

That s not at all obvious to me.
It seems in the same level of drastic as the other options you are
considering.
So taking it off the table without discussion or consideration doesn't
seem like a good call.



Re: Y2038 - best way forward in Debian?

2020-02-05 Thread YunQiang Su
Steve McIntyre  于2020年2月4日周二 下午9:15写道:
>
> Hey folks,
>
> Apologies - I should have sent this mail quite a while back. :-/
>
> Arnd Bergmann (in CC) has been helping to drive a lot of work to solve
> the 2038 problem in the Linux world. I've spoken about this before,
> e.g. at DebConf17. He's been pushing a lot of updates throughout the
> Linux kernel, and has also been analysing code elsewhere. He's very
> interested in how to update complete systems like Debian too, and we
> spoke about this some time ago.
>
> The kernel is *basically* fixed now. Internally, data structures
> should now be safe. There are a small number places where 32-bit time
> is still a thing, but it's in hand. A number of syscalls, ioctls,
> etc. have needed updates for the user-kernel interface level. glibc is
> the place that needs to care about most of this.
>
> The glibc folks have taken an interesting approach.
>
>  * 64-bit ABIs/architectures are already using 64-bit time_t
>throughout. The design is sane and so we should already be mostly
>safe here, modulo silly code bugs (we'll always have those!)
>
>  * 32-bit ABIs/arches are more awkward. glibc will continue *by
>default* to use 32-bit time_t to keep compatibility with existing
>code. This will *not* be safe as we approach 2038.
>
>  * 32-bit ABIs/arches *can* be told to use 64-bit time_t from glibc
>upwards, but this will of course affect the ABI. Embedded uses of
>time_t in libraries will change size, etc. This *will* be safe for
>2038.
>
> So, we're all fine? Not so much: for our 32-bit Debian arches, we will
> need to basically rebuild the world to be 2038-safe. When we had to do
> something like this in the past, to deal with the libc5->libc6
> transition, we had an SONAME change in libc to work with. We decided
> to append the "g" tag to the names of our library binary packages to
> signal that a library was built against libc6. We can still see some
> of the effects of this in the archive, many years later
> (e.g. zlib1g). The problem now is that we will *not* have an soname
> change here to help identify code compatibility on the 32-bit front.
>
> Arnd scanned the library packages in the Debian archive and identified
> that about one third of our library packages would need rebuilding
> (and tracking) to make a (recursive) transition. We can see two
> different possible routes to follow:
>
>  A Follow a similar path to last time (rename library packages). This
>will allow us to do partial upgrades, but the cost is that a vast
>number of packages will need work to make this happen,
>*potentially* building library packages twice to allow us to
>continue both 32-bit and 64-bit time_t support forwards for a
>while. This effort will be *needed* only for the sake of our 32-bit
>ports, but would affect *everybody*.
>
>  *** OR ***
>
>  B Decide which 32-bit ports we expect to care about by 2038, and
>re-bootstrap new versions of those ports *with different
>names*. This would allow most of our developers to ignore the
>problem here (as 64-bit arches are not affected) and let a smaller
>number of people re-bootstrap with new ABIs with 64-bit time_t
>embedded. There are some downsides here:
>
>* we would end up with two versions of some ports for a short time
>  - probably one release of overlap
>
>* users would *not* be able to simply upgrade from one to the
>  other, due to lack of binary compatibility
>
>We *would* be able to run old and new ABIs on top of the same (new
>enough) kernel (e.g. for buildds), so a lump of this would just be
>simply building the new world and filing bugs where needed.
>
>We'd need to decide exactly which of our 32-bit ports we would want
>to do this path with (probably armhf->arhmft?, maybe
>armel->armelt?, i386->i386t?. mipsel???). Due to the libc6 soname
>continuity, we will most likely *not* end up with a different
>visible ABI via the triplet and the runtime linker, so old/new
>multi-arch will be impossible.
>
> So, which way should we go? My own personal gut feel matches Arnd's,
> which would be route B. Some of us already have fair experience with
> bootstrapping new arches, and we could almost just crank the handle
> for most of this work.
>

Is there the option C?

Draft:
  1. define time64_t
  2. for data_struct which act as a part of ABI, define a new data_struct64
  3. for function which act as part of ABI, define a new version func64.
  4. patch all packages to use time64_t instead of time_t step by step.
  5. set a time as deadline (2030?), and then treat all packages
haven't finished
  the migration as rc-buggy.

 Since we have enough time, we can patch all packages in that period.

> What do people think here? Which do you think is the better path? Feel
> free to point out things you think I may have missed. We should get
> started on this soon - the longer we leave it, the more likely it is
> that we

Re: Y2038 - best way forward in Debian?

2020-02-05 Thread Ansgar
Russ Allbery writes:
> Ansgar  writes:
>
>> So maybe just recommend people to move to 64-bit architectures and put
>> 32-bit applications in a time namespace so they believe they are still
>> in 2001 ;-) 32-bit architectures will probably still be useful in
>> embedded contexts for a long time and there it might be easier to just
>> change the ABI, but for a general-purpose distribution we start seeing
>> more and more problems and I don't really see us supporting them as a
>> full architecture in 10+ years.
>
> If we go down this path, can we make cross-grading a supported feature for
> the next stable release?  I'm sure I'm not the only one who is stuck with
> continuously-upgraded i386 hosts who has been wanting to switch but has
> been waiting until cross-grading is a little bit less scary.

I think the window of opportunity to support this is already over: most
people have already migrated to amd64, there is no real incentive for
these to spend effort on adding support for cross-grading (besides
academic interest, but I for example find adding support for one-time
migrations of legacy systems not that rewarding).

So you would need to recruit people still running i386 hosts, but for
each individual it might be easier to just wait for still existing i386
systems to be decommissioned or just reinstall. (I would likely
recommend reinstalling ancient systems for various reasons anyway.)

For just avoiding the Y2038 problem, i386 might sort itself out already
without additional intervention. From recent popcon reports I get:

| Release|  i386 |  amd64 | i386/amd64 (%) |
|+---++|
| 1.41 (etch)|   323 | 66 |   4.89 |
| 1.46 (lenny)   |   958 |440 |   2.18 |
| 1.49 (squeeze) |  1753 |   2553 |   0.69 |
| 1.56 (wheezy)  |  3573 |   9455 |   0.38 |
| 1.61 (jessie)  |  4687 |  26977 |   0.17 |
| 1.64 (stretch) |  4932 |  62619 |   0.08 |
| 1.67 (buster)  |  3978 |  66169 |   0.06 |
| 1.69 (t/u) |   360 |  10445 |   0.03 |
|+---++|
| Overall| 20728 | 180079 |   0.12 |
#+TBLFM: $4=$2/$3;%.2f

I'll assume the composition of installations within a release didn't
change over time.

This shows exponential decay for i386 installations (which is expected)
with a factor of ~0.5 for each release.  If we consider 10 years in the
future or 5 releases, one would expect i386 usage to have gone down to
about 0.03 of the current value, that is about 0.002 of installations.

Ansgar



Re: Y2038 - best way forward in Debian?

2020-02-04 Thread Steve Langasek
On Tue, Feb 04, 2020 at 11:03:03AM -0500, Lennart Sorensen wrote:
> On Tue, Feb 04, 2020 at 09:38:46AM -0500, Stefan Monnier wrote:
> > >  * 32-bit ABIs/arches are more awkward. glibc will continue *by
> > >default* to use 32-bit time_t to keep compatibility with existing
> > >code. This will *not* be safe as we approach 2038.
> > >
> > >  * 32-bit ABIs/arches *can* be told to use 64-bit time_t from glibc
> > >upwards, but this will of course affect the ABI. Embedded uses of
> > >time_t in libraries will change size, etc. This *will* be safe for
> > >2038.

> > And that's chosen at build time (i.e. when compiling glibc)?
> > Why not provide different entry points for time-manipulating functions,
> > so a single build can support both applications using 32bit time and
> > applications using 64bit time?

> > E.g.

> > struct time32_t ...
> > struct time64_t ...

> > double difftime32 (time32_t time1, time32_t time0);
> > double difftime64 (time64_t time1, time64_t time0);

> > and in the time.h have

> > #if TOO_OLD_TO_LIVE_PAST_2038
> > typedef time32_t time_t;
> > ...
> > #else
> > typedef time64_t time_t;
> > ...
> > #endif

> I agree.  Why should this be any different than 64 bit file support?
> Transparent on 64 bit architectures, and 32bit code gets to pick the
> one it wants to support at compile time while glibc supports both,
> and eventually just about everything switches.  You can even eventually
> add warnings when any program calls the 32 bit version of the functions.

In practice, despite LFS_CFLAGS having been a priority for Debian as far
back as 2003, I think it was no more than 5 years ago that I was still
finding libraries in the archive that were incompatible with LFS because
they were leaking 32-bit types into their own ABIs.  I think the lesson to
be learned from 64-bit file support is that this was NOT managed effectively
as a transition.

-- 
Steve Langasek   Give me a lever long enough and a Free OS
Debian Developer   to set it on, and I can move the world.
Ubuntu Developer   https://www.debian.org/
slanga...@ubuntu.com vor...@debian.org


signature.asc
Description: PGP signature


Re: Y2038 - best way forward in Debian?

2020-02-04 Thread Russ Allbery
Ansgar  writes:

> So maybe just recommend people to move to 64-bit architectures and put
> 32-bit applications in a time namespace so they believe they are still
> in 2001 ;-) 32-bit architectures will probably still be useful in
> embedded contexts for a long time and there it might be easier to just
> change the ABI, but for a general-purpose distribution we start seeing
> more and more problems and I don't really see us supporting them as a
> full architecture in 10+ years.

If we go down this path, can we make cross-grading a supported feature for
the next stable release?  I'm sure I'm not the only one who is stuck with
continuously-upgraded i386 hosts who has been wanting to switch but has
been waiting until cross-grading is a little bit less scary.

-- 
Russ Allbery (r...@debian.org)  



Re: Y2038 - best way forward in Debian?

2020-02-04 Thread Simon Richter
Hi Colin,

On Tue, Feb 04, 2020 at 06:48:43PM +, Colin Watson wrote:

> > This may be a good time to mention that SONAMEs like libc6.1 are not 
> > supported by libtool, so in libxcrypt I had to conditionally patch the 
> > generated libtool executable for the architectures that did this.

> What exactly went wrong?  libpipeline uses libtool and I've never had to
> patch it for libc6.1.

Libtool computes the version number in the SONAME of the newly generated
library, so it must be a single integer. There is no way to force it to use
anything else.

That is not a problem for using libtool to link against a library that by
some other method got a SONAME that doesn't follow the conventions.

   Simon



Re: Y2038 - best way forward in Debian?

2020-02-04 Thread Ben Hutchings
On Tue, 2020-02-04 at 11:03 -0500, Lennart Sorensen wrote:
> On Tue, Feb 04, 2020 at 09:38:46AM -0500, Stefan Monnier wrote:
> > >  * 32-bit ABIs/arches are more awkward. glibc will continue *by
> > >default* to use 32-bit time_t to keep compatibility with existing
> > >code. This will *not* be safe as we approach 2038.
> > > 
> > >  * 32-bit ABIs/arches *can* be told to use 64-bit time_t from glibc
> > >upwards, but this will of course affect the ABI. Embedded uses of
> > >time_t in libraries will change size, etc. This *will* be safe for
> > >2038.
> > 
> > And that's chosen at build time (i.e. when compiling glibc)?
> > Why not provide different entry points for time-manipulating functions,
> > so a single build can support both applications using 32bit time and
> > applications using 64bit time?

If I recall correctly, glibc *will* provide both entry points, so there
is no ABI break.  But the size of time_t (etc.) exposed through libc-
dev is fixed at glibc build time.

You should expect that all proposals have already been made and
discussed on the glibc-alpha mailing list over the past few years.

> > E.g.
> > 
> > struct time32_t ...
> > struct time64_t ...
> > 
> > double difftime32 (time32_t time1, time32_t time0);
> > double difftime64 (time64_t time1, time64_t time0);
> > 
> > and in the time.h have
> > 
> > #if TOO_OLD_TO_LIVE_PAST_2038
> > typedef time32_t time_t;
> > ...
> > #else
> > typedef time64_t time_t;
> > ...
> > #endif
> 
> I agree.  Why should this be any different than 64 bit file support?
> Transparent on 64 bit architectures, and 32bit code gets to pick the
> one it wants to support at compile time while glibc supports both,
> and eventually just about everything switches.  You can even eventually
> add warnings when any program calls the 32 bit version of the functions.

LFS is a great example of how *not* to do it.  23 years on, we still
have open bugs for programs that should opt in but didn't.  Not every
program needs to handle > 2 GiB files, but there are now filesystems
with 64-bit inode numbers and they break every non-LFS program that
calls stat().

Similarly, every program that uses wall-clock time will fail as we
approach 2038, and the failure mode is likely to be even worse than
with LFS as few programs will check for errors from time APIs.

Ben.

-- 
Ben Hutchings
Life is what happens to you while you're busy making other plans.
  - John Lennon



signature.asc
Description: This is a digitally signed message part


Re: Y2038 - best way forward in Debian?

2020-02-04 Thread Colin Watson
On Tue, Feb 04, 2020 at 06:13:12PM +0100, Marco d'Itri wrote:
> On Feb 04, Guillem Jover  wrote:
> > Well, I guess such a new (conditinally selectable) name could be
> > coordinated with glibc upstream? Say bump 32-bit ports to use libc6.1?
> > (We already have precedent in some ports that do not use the same
> > prevalent SONAME, such as libc6.1, libc0.1 and libc0.1.) But it would
> > indeed involve lots of work, with massive amounts of package renames. :/
> 
> This may be a good time to mention that SONAMEs like libc6.1 are not 
> supported by libtool, so in libxcrypt I had to conditionally patch the 
> generated libtool executable for the architectures that did this.

What exactly went wrong?  libpipeline uses libtool and I've never had to
patch it for libc6.1.

-- 
Colin Watson   [cjwat...@debian.org]



Re: Y2038 - best way forward in Debian?

2020-02-04 Thread Marco d'Itri
On Feb 04, Guillem Jover  wrote:

> Well, I guess such a new (conditinally selectable) name could be
> coordinated with glibc upstream? Say bump 32-bit ports to use libc6.1?
> (We already have precedent in some ports that do not use the same
> prevalent SONAME, such as libc6.1, libc0.1 and libc0.1.) But it would
> indeed involve lots of work, with massive amounts of package renames. :/
This may be a good time to mention that SONAMEs like libc6.1 are not 
supported by libtool, so in libxcrypt I had to conditionally patch the 
generated libtool executable for the architectures that did this.

-- 
ciao,
Marco


signature.asc
Description: PGP signature


Re: Y2038 - best way forward in Debian?

2020-02-04 Thread Lennart Sorensen
On Tue, Feb 04, 2020 at 09:38:46AM -0500, Stefan Monnier wrote:
> >  * 32-bit ABIs/arches are more awkward. glibc will continue *by
> >default* to use 32-bit time_t to keep compatibility with existing
> >code. This will *not* be safe as we approach 2038.
> >
> >  * 32-bit ABIs/arches *can* be told to use 64-bit time_t from glibc
> >upwards, but this will of course affect the ABI. Embedded uses of
> >time_t in libraries will change size, etc. This *will* be safe for
> >2038.
> 
> And that's chosen at build time (i.e. when compiling glibc)?
> Why not provide different entry points for time-manipulating functions,
> so a single build can support both applications using 32bit time and
> applications using 64bit time?
> 
> E.g.
> 
> struct time32_t ...
> struct time64_t ...
> 
> double difftime32 (time32_t time1, time32_t time0);
> double difftime64 (time64_t time1, time64_t time0);
> 
> and in the time.h have
> 
> #if TOO_OLD_TO_LIVE_PAST_2038
> typedef time32_t time_t;
> ...
> #else
> typedef time64_t time_t;
> ...
> #endif

I agree.  Why should this be any different than 64 bit file support?
Transparent on 64 bit architectures, and 32bit code gets to pick the
one it wants to support at compile time while glibc supports both,
and eventually just about everything switches.  You can even eventually
add warnings when any program calls the 32 bit version of the functions.

-- 
Len Sorensen



Re: Y2038 - best way forward in Debian?

2020-02-04 Thread Guillem Jover
Hi!

On Tue, 2020-02-04 at 13:14:10 +, Steve McIntyre wrote:
> The glibc folks have taken an interesting approach.
> 
>  * 64-bit ABIs/architectures are already using 64-bit time_t
>throughout. The design is sane and so we should already be mostly
>safe here, modulo silly code bugs (we'll always have those!)
> 
>  * 32-bit ABIs/arches are more awkward. glibc will continue *by
>default* to use 32-bit time_t to keep compatibility with existing
>code. This will *not* be safe as we approach 2038.
> 
>  * 32-bit ABIs/arches *can* be told to use 64-bit time_t from glibc
>upwards, but this will of course affect the ABI. Embedded uses of
>time_t in libraries will change size, etc. This *will* be safe for
>2038.

I think there is still work pending in glibc to expose both 32-bit and
64-bit interfaces at the same time? (Currently it seems to be either-or.)

I'd like to use this for example in libbsd, to make a smooth
transition similar to the one made in glibc, w/o needing to bump the
SONAME.

> So, we're all fine? Not so much: for our 32-bit Debian arches, we will
> need to basically rebuild the world to be 2038-safe. When we had to do
> something like this in the past, to deal with the libc5->libc6
> transition, we had an SONAME change in libc to work with. We decided
> to append the "g" tag to the names of our library binary packages to
> signal that a library was built against libc6. We can still see some
> of the effects of this in the archive, many years later
> (e.g. zlib1g). The problem now is that we will *not* have an soname
> change here to help identify code compatibility on the 32-bit front.

Well, I guess such a new (conditinally selectable) name could be
coordinated with glibc upstream? Say bump 32-bit ports to use libc6.1?
(We already have precedent in some ports that do not use the same
prevalent SONAME, such as libc6.1, libc0.1 and libc0.1.) But it would
indeed involve lots of work, with massive amounts of package renames. :/

> Arnd scanned the library packages in the Debian archive and identified
> that about one third of our library packages would need rebuilding
> (and tracking) to make a (recursive) transition. We can see two
> different possible routes to follow:
> 
>  A Follow a similar path to last time (rename library packages).

>  *** OR ***

>  B Decide which 32-bit ports we expect to care about by 2038, and
>re-bootstrap new versions of those ports *with different
>names*. This would allow most of our developers to ignore the
>problem here (as 64-bit arches are not affected) and let a smaller
>number of people re-bootstrap with new ABIs with 64-bit time_t
>embedded. There are some downsides here:

(If we are going for this, I'd say it might make sense to consider
also enabling LFS, although that might be a dangerous change, see
the lintian tag for the rationale. :)

>* users would *not* be able to simply upgrade from one to the
>  other, due to lack of binary compatibility

I think this conclusion stems from an incorrect premise. See below.

>We'd need to decide exactly which of our 32-bit ports we would want
>to do this path with (probably armhf->arhmft?, maybe
>armel->armelt?, i386->i386t?. mipsel???). Due to the libc6 soname
>continuity, we will most likely *not* end up with a different
>visible ABI via the triplet and the runtime linker, so old/new
>multi-arch will be impossible.

A new dpkg architecture must use a different triplet, as these are
required to be bijective. See “lpia” for a previous example of this.
(This would then make it possible to cross-grade.)

In addition if we are using a new multiarch triplet, and need to
rebuild the world, are going to be ABI incompatible anyway, we might
as well use a proper multiarch-qualified ld.so pathname that does
not collide with anything.

I also think we have a C option, which would be something like:

  C Do a transition equivalent to the LFS one, by either switching
libraries opportunistically to 64-bit time_t on their next SONAME
bumps, or by updating them to provide 64-bit variants of their
interfaces along their 32-bit ones, as done in glibc. Of course
the main problem here is that the LFS "transition" is not one we
should be very proud of, as it's been dragging on for a very long
time, and it's not even close to be finished…


(Hmm there seems to be something borked with lintian.d.o, as the
general tag numbers seem extremely low. :)

Thanks,
Guillem



Re: Y2038 - best way forward in Debian?

2020-02-04 Thread Marco d'Itri
On Feb 04, Steve McIntyre  wrote:

>We'd need to decide exactly which of our 32-bit ports we would want
>to do this path with (probably armhf->arhmft?, maybe
>armel->armelt?, i386->i386t?. mipsel???). Due to the libc6 soname
I agree with Ansgar here: there is no point in rebuilding i386.
The porters for the other architectures who believe that there is value 
in such a port should explain why they believe that in ~10 years from 
now there will still be NEW 32 bit hardware that will run Debian.

The real questions should be: what is the latest reasonable year to 
eventually start these new ports? 2038 is far away, and maybe enough 
procrastination will solve this...

-- 
ciao,
Marco


signature.asc
Description: PGP signature


Re: Y2038 - best way forward in Debian?

2020-02-04 Thread Holger Levsen
Hi Steve,

many thanks for this heads-up report!

On Tue, Feb 04, 2020 at 01:14:10PM +, Steve McIntyre wrote:
> So, we're all fine? Not so much: for our 32-bit Debian arches, we will
> need to basically rebuild the world to be 2038-safe. 

for being able to reproducibly rebuild more Debian packages in 
bullseye, this would be good anyway and on all archs.

(everything built before December 2016 for example)


-- 
cheers,
Holger

---
   holger@(debian|reproducible-builds|layer-acht).org
   PGP fingerprint: B8BF 5413 7B09 D35C F026 FE9D 091A B856 069A AA1C



signature.asc
Description: PGP signature


Re: Y2038 - best way forward in Debian?

2020-02-04 Thread Michael Stone

On Tue, Feb 04, 2020 at 03:17:50PM +0100, Ansgar wrote:

At least for i386, I expect it to be used mostly for legacy
applications (and legacy installations).  So breaking ABI by switching
to a "new" architecture or by just changing major libraries like libc6
probably diminishes its value so much that there would no longer be any
use for it: one could just switch to amd64 instead of i386t.


Yes, for x86 in particular I don't see any reason to try to rebuild 
x86-2038 vs x32 for whatever niche might be filled by a 32 bit ILP.




Re: Y2038 - best way forward in Debian?

2020-02-04 Thread Ansgar
On Tue, 2020-02-04 at 13:14 +, Steve McIntyre wrote:
> What do people think here? Which do you think is the better path? Feel
> free to point out things you think I may have missed. We should get
> started on this soon - the longer we leave it, the more likely it is
> that we'll see 2038 bugs biting people.

At least for i386, I expect it to be used mostly for legacy
applications (and legacy installations).  So breaking ABI by switching
to a "new" architecture or by just changing major libraries like libc6
probably diminishes its value so much that there would no longer be any
use for it: one could just switch to amd64 instead of i386t.

So maybe just recommend people to move to 64-bit architectures and put
32-bit applications in a time namespace so they believe they are still
in 2001 ;-)  32-bit architectures will probably still be useful in
embedded contexts for a long time and there it might be easier to just
change the ABI, but for a general-purpose distribution we start seeing
more and more problems and I don't really see us supporting them as a
full architecture in 10+ years.

Ansgar



Re: Y2038 - best way forward in Debian?

2020-02-04 Thread Andrey Rahmatullin
On Tue, Feb 04, 2020 at 01:14:10PM +, Steve McIntyre wrote:
>  B Decide which 32-bit ports we expect to care about by 2038, and
... and consider options taking into account the size of the result list.

-- 
WBR, wRAR


signature.asc
Description: PGP signature


Y2038 - best way forward in Debian?

2020-02-04 Thread Steve McIntyre
Hey folks,

Apologies - I should have sent this mail quite a while back. :-/

Arnd Bergmann (in CC) has been helping to drive a lot of work to solve
the 2038 problem in the Linux world. I've spoken about this before,
e.g. at DebConf17. He's been pushing a lot of updates throughout the
Linux kernel, and has also been analysing code elsewhere. He's very
interested in how to update complete systems like Debian too, and we
spoke about this some time ago.

The kernel is *basically* fixed now. Internally, data structures
should now be safe. There are a small number places where 32-bit time
is still a thing, but it's in hand. A number of syscalls, ioctls,
etc. have needed updates for the user-kernel interface level. glibc is
the place that needs to care about most of this.

The glibc folks have taken an interesting approach.

 * 64-bit ABIs/architectures are already using 64-bit time_t
   throughout. The design is sane and so we should already be mostly
   safe here, modulo silly code bugs (we'll always have those!)

 * 32-bit ABIs/arches are more awkward. glibc will continue *by
   default* to use 32-bit time_t to keep compatibility with existing
   code. This will *not* be safe as we approach 2038.

 * 32-bit ABIs/arches *can* be told to use 64-bit time_t from glibc
   upwards, but this will of course affect the ABI. Embedded uses of
   time_t in libraries will change size, etc. This *will* be safe for
   2038.

So, we're all fine? Not so much: for our 32-bit Debian arches, we will
need to basically rebuild the world to be 2038-safe. When we had to do
something like this in the past, to deal with the libc5->libc6
transition, we had an SONAME change in libc to work with. We decided
to append the "g" tag to the names of our library binary packages to
signal that a library was built against libc6. We can still see some
of the effects of this in the archive, many years later
(e.g. zlib1g). The problem now is that we will *not* have an soname
change here to help identify code compatibility on the 32-bit front.

Arnd scanned the library packages in the Debian archive and identified
that about one third of our library packages would need rebuilding
(and tracking) to make a (recursive) transition. We can see two
different possible routes to follow:

 A Follow a similar path to last time (rename library packages). This
   will allow us to do partial upgrades, but the cost is that a vast
   number of packages will need work to make this happen,
   *potentially* building library packages twice to allow us to
   continue both 32-bit and 64-bit time_t support forwards for a
   while. This effort will be *needed* only for the sake of our 32-bit
   ports, but would affect *everybody*.

 *** OR ***

 B Decide which 32-bit ports we expect to care about by 2038, and
   re-bootstrap new versions of those ports *with different
   names*. This would allow most of our developers to ignore the
   problem here (as 64-bit arches are not affected) and let a smaller
   number of people re-bootstrap with new ABIs with 64-bit time_t
   embedded. There are some downsides here:

   * we would end up with two versions of some ports for a short time
 - probably one release of overlap

   * users would *not* be able to simply upgrade from one to the
 other, due to lack of binary compatibility

   We *would* be able to run old and new ABIs on top of the same (new
   enough) kernel (e.g. for buildds), so a lump of this would just be
   simply building the new world and filing bugs where needed.

   We'd need to decide exactly which of our 32-bit ports we would want
   to do this path with (probably armhf->arhmft?, maybe
   armel->armelt?, i386->i386t?. mipsel???). Due to the libc6 soname
   continuity, we will most likely *not* end up with a different
   visible ABI via the triplet and the runtime linker, so old/new
   multi-arch will be impossible.

So, which way should we go? My own personal gut feel matches Arnd's,
which would be route B. Some of us already have fair experience with
bootstrapping new arches, and we could almost just crank the handle
for most of this work.

What do people think here? Which do you think is the better path? Feel
free to point out things you think I may have missed. We should get
started on this soon - the longer we leave it, the more likely it is
that we'll see 2038 bugs biting people.

-- 
Steve McIntyre, Cambridge, UK.st...@einval.com
"Yes, of course duct tape works in a near-vacuum. Duct tape works
 anywhere. Duct tape is magic and should be worshipped."
   -― Andy Weir, "The Martian"