Re: Compiler runs out of memory when building a package

[Johannes Schauer Marin Rodrigues]

Hi,

Quoting Dima Kogan (2023-08-13 22:54:48)
> I'm looking for a suggestion to fix a problem.
> 
> I uploaded a package, and it cleared NEW a few days ago. I now see that
> it fails to build on most 32-bit arches becaues the compiler runs out of
> memory.

I ran into the same problem with my package vcmi. The fix I've been using was
to add this to my d/rules:

export DEB_CXXFLAGS_MAINT_APPEND+=--param ggc-min-expand=5

https://sources.debian.org/src/vcmi/1.1.0%2Bdfsg-1/debian/rules/#L20

Thanks!

cheers, josch

signature.asc
Description: signature

Re: Potential MBF: packages failing to build twice in a row

[Johannes Schauer Marin Rodrigues]

Hi,

Quoting John Goerzen (2023-08-13 23:32:03)
> On Sat, Aug 05 2023, Lucas Nussbaum wrote:
> > I wonder what we should do, because 5000+ failing packages is a lot...
> Let's think about the level of trouble we cause trying to tackle something
> that has clearly not bothered anyone for years.

this is not the first time that is has been said in this thread that "this
hasn't bothered anybody for years". I wanted to come out and say that it has
bothered me. It just hasn't bothered me enough to investigate what the proper
way to solve it is. It hasn't bothered me enough to bother other people with
this issue. After all, I can just run "git clean -fdx" to "solve" the problem
whenever it happens.

Lucas filed bugs against three of my packages. Since I didn't know how to
properly solve this issue and since this thread exists I finally felt it was
time to ask my fellow DDs how to fix this and I'll be happy to upload packages
with this problem fixed.

Thank you Lucas for doing this work!

> Personally, I'm a volunteer.  I have X amount of time to devote to Debian.
> I work on Debian because I enjoy it.  It is satisfying!

I'm also just a volunteer doing this in my free time as my hobby because I find
it fun. I can totally understand how fixing this can be anything but fun for
others and I do not want to invalidate your opinion on this. But as this thread
has a number of people saying how they don't find this useful and how they
don't find fixing this fun, I wanted to write a message with my very different
view on the subject. I cannot tell you exactly why but my *feeling* is that I'm
very happy that Lucas filed bugs against three of my packages and I will
personally find it fun to upload a version that fixes them. I understand that
there are many who do not think like that and I understand that it's worth
discussing the practical benefit of it all. But I for one am happy that these
bugs were filed and I will have fun closing them with a fix.

Thanks!

cheers, josch

signature.asc
Description: signature

Re: Compiler runs out of memory when building a package

[Chow Loong Jin]

On Sun, Aug 13, 2023 at 01:54:48PM -0700, Dima Kogan wrote:
> Hi.
> 
> I'm looking for a suggestion to fix a problem.
> 
> I uploaded a package, and it cleared NEW a few days ago. I now see that
> it fails to build on most 32-bit arches becaues the compiler runs out of
> memory. Logs:
> 
>   https://buildd.debian.org/status/package.php?p=gtsam
> 
> Arbitrary 32-bit log (armhf):
> 
>   
> https://buildd.debian.org/status/fetch.php?pkg=gtsam&arch=armhf&ver=4.2%7E9%2Bdfsg-2&stamp=1691926953&raw=0
>

Hi,

We had a similar build failure in slic3r-prusa, and the way we fixed it
was to reduce debug symbols on 32-bit architectures, and disable it on
mips + mipsel.

See [1], [2] and [3] for what was done. Maybe you can try adding those
build flags to gtsam.

[1] 
https://salsa.debian.org/3dprinting-team/slic3r-prusa/-/blob/51a729e5a8bf45285d6e0dab93617c544eec885f/debian/rules#L5-10
[2] https://bugs.debian.org/961107
[3] https://bugs.debian.org/1041870

-- 
Kind regards,
Loong Jin


signature.asc
Description: PGP signature

Re: Potential MBF: packages failing to build twice in a row

[Vincent Lefevre]

On 2023-08-13 16:32:03 -0500, John Goerzen wrote:
> - Upstream wants to ship things that may get modified during build.  Ie,
>   autoconf/automake replaces files they ship because they want it to
>   work "out of the box" in some fashion.  Another example is
>   documentation; upstream may ship built docs even though we rebuild it
>   for completeness.

Yes, the GNU Coding Standards even say:

  GNU distributions usually contain some files which are not
  source files—for example, Info files, and the output from
  Autoconf, Automake, Bison or Flex.

For mpfr4, the test is failing just because the makeinfo version
is now newer in Debian:

--- mpfr4-4.2.0.orig/doc/mpfr.info
+++ mpfr4-4.2.0/doc/mpfr.info
@@ -1,4 +1,4 @@
-This is mpfr.info, produced by makeinfo version 6.8 from mpfr.texi.
+This is mpfr.info, produced by makeinfo version 7.0.3 from mpfr.texi.
 This manual documents how to install and use the Multiple Precision
 Floating-Point Reliable Library, version 4.2.0.

-- 
Vincent Lefèvre  - Web: 
100% accessible validated (X)HTML - Blog: 
Work: CR INRIA - computer arithmetic / AriC project (LIP, ENS-Lyon)

Re: Potential MBF: packages failing to build twice in a row

[John Goerzen]

On Sat, Aug 05 2023, Lucas Nussbaum wrote:

> I wonder what we should do, because 5000+ failing packages is a lot...

Let's think about the level of trouble we cause trying to tackle
something that has clearly not bothered anyone for years.

>From the packaging side, there are many reasons that proper clean
targets can be difficult:

- Upstream wants to ship things that may get modified during build.  Ie,
  autoconf/automake replaces files they ship because they want it to
  work "out of the box" in some fashion.  Another example is
  documentation; upstream may ship built docs even though we rebuild it
  for completeness.

- Upstream's clean target is insufficient.

- Our own build processes make modifications.  (eg, quilt; yes I know it
  is supposed to be cleaned up but this is not always perfect)

- It is difficult to prepare a proper clean target, because build
  artifacts may lack predictable names (eg, have the architecture
  embedded in them)

Now, what are the possible options for dealing with this?

- Invest a lot of time writing bespoke scripts to handle it

- Copy the upstream source to a temp directory and build from there.
  (Some packages already do this; if we are going to hack in this
  direction all over, why not just do it by default everywhere?)

- Repack the upstream source to exclude files we generate.  But this has
  a ton of downfalls, including breaking the trust chain from upstream
  to us.  It should be a last resort (eg, making the tarball DFSG-free).

- Just tell people to run "git reset --hard HEAD; git clean -xfd"

Personally, I'm a volunteer.  I have X amount of time to devote to
Debian.   I work on Debian because I enjoy it.  It is satisfying!

I maintain packages written in at least 6 different languages.  Some
packages are sleek and modern tools.  One I am about to take over traces
its codebase to 1981 and the upstream author has an explicit goal that
it still builds on operating systems that were discontinued in the
mid-80s.

I have an alias (not just for Debian) that does "git reset --hard HEAD;
git clean -xfd".

Spending hours to make a clean target I have no need for, already have
an equivalent of, and has no real purpose, drains my enthusiasm and
steals time that I could otherwise be using doing more high-value work
like fixing bugs.

Let's focus our energies on things that matter more.

> Should we give up on requiring a 'clean' target that works? After all,
> when 17% of packages are failing, it means that many maintainers don't
> depend on it in their workflow.

Yes.

- John

Re: Potential MBF: packages failing to build twice in a row

[Timo Röhling]

* Jeremy Stanley  [2023-08-13 21:18]:

Similarly, I got one for __pycache__/*.cpython-311.pyc file
overwrites... is that something dh_python should clean?


As a matter of fact, I also have one like that, ironically it is not
even a Python package, it just happens to run Sphinx for its
documentation. Thus, I'm seriously wondering if dh_clean should
start removing __pycache__ folders, as it already cleans up some
other random stuff like autom4te.cache folders, *.orig and *.rej
patch backups, and even DEADJOE files.


Cheers
Timo

--
⢀⣴⠾⠻⢶⣦⠀   ╭╮
⣾⠁⢠⠒⠀⣿⡁   │ Timo Röhling   │
⢿⡄⠘⠷⠚⠋⠀   │ 9B03 EBB9 8300 DF97 C2B1  23BF CC8C 6BDD 1403 F4CA │
⠈⠳⣄   ╰╯


signature.asc
Description: PGP signature

Re: Potential MBF: packages failing to build twice in a row

[John Goerzen]

On Sat, Aug 05 2023, Andrey Rakhmatullin wrote:

> On Sat, Aug 05, 2023 at 08:10:35PM +0300, Adrian Bunk wrote:
>> Debian maintainers with proper git workflows are already exporting all
>> their changes from git to debian/patches/ as one file - currently the
>> preferred form of modification of a Debian package has to be in salsa
>> and not in our archive when the changes cannot be represented as quilt
>> patches against tarballs.
> Is the gbp-pq workflow improper?


Improper?  I don't know.  But bad, yes.

If we ignore the decades of history -- yes I know we don't live in that
vacuum, but humor me here -- it is a weird process.  We (typically) get
upstream code in git, and (typically) maintain it in git in Debian.  git
already has features built in to do all this tracking, but we do this
really weird thing where we use git to store text files consisting of
diffs that, in that case, are more properly maintained in git anyhow.

I actually prefer the old source format to quilt, because I can just
maintain things in git the proper way with it, rather than have to do
all this weirdness.  But that's just me.

Not all of my packages have their upstream in git, but I maintain 100%
of them in git at the Debian level.

Here's the key.  By using a lot of nonstandard things, Debian is doing
two things:

1) Raising the barrier to newcomers to participate

2) Assuring that we will lag behind what others are doing

For #2, the reason is that however much we work on our bespoke tooling,
we cannot hope to match what the rest of the world combined is doing.

That doesn't mean we give up on .deb and adopt RPM or something.  But,
in an ideal world, would gbp-pq need to exist?  I don't think so.  A
world in which it doesn't should be our target.

- John

Re: Potential MBF: packages failing to build twice in a row

[Jeremy Stanley]

On 2023-08-13 22:59:53 +0200 (+0200), Timo Röhling wrote:
[...]
> There's talk on #d-python if pybuild could/should deal with it; I'd
> give it a few days and see if that pans out.
> 
> If not, extend-diff-ignore as Scott suggested or simply removing the
> egg-info folders with a *.egg-info/ line in d/clean should both
> work. (Personally, I'd use extend-diff-ignore if the egg-info is
> also shipped in the source tarball and d/clean if not)

Similarly, I got one for __pycache__/*.cpython-311.pyc file
overwrites... is that something dh_python should clean?
-- 
Jeremy Stanley


signature.asc
Description: PGP signature

Compiler runs out of memory when building a package

[Dima Kogan]

Hi.

I'm looking for a suggestion to fix a problem.

I uploaded a package, and it cleared NEW a few days ago. I now see that
it fails to build on most 32-bit arches becaues the compiler runs out of
memory. Logs:

  https://buildd.debian.org/status/package.php?p=gtsam

Arbitrary 32-bit log (armhf):

  
https://buildd.debian.org/status/fetch.php?pkg=gtsam&arch=armhf&ver=4.2%7E9%2Bdfsg-2&stamp=1691926953&raw=0

I thought it maybe is due to doing too many things in parallel, but
building with -j1 doesn't fix it. I have boiled down the problem to
compiling a single source file. Unsuprisingly, this is a big generated
C++ source file used to define the Python interface. I can reproduce
this with a single compile command on abel.d.o (the
armhf porterbox):

  apt source gtsam

  cd gtsam-4.2~9+dfsg

  PYTHONPATH=wrap \
  python3 "wrap/scripts/pybind_wrap.py" \
--src "gtsam/geometry/geometry.i" \
--out geometry.cpp \
--module_name gtsam \
--top_module_namespaces gtsam \
--ignore gtsam::Point2 gtsam::Point3 gtsam::ISAM2ThresholdMapValue 
gtsam::FactorIndices gtsam::FactorIndexSet gtsam::IndexPairSetMap 
gtsam::IndexPairVector gtsam::BetweenFactorPose2s gtsam::BetweenFactorPose3s 
gtsam::Point2Vector gtsam::Point2Pairs gtsam::Point3Pairs gtsam::Pose3Pairs 
gtsam::Pose3Vector gtsam::Rot3Vector gtsam::KeyVector 
gtsam::BinaryMeasurementsPoint3 gtsam::BinaryMeasurementsUnit3 
gtsam::BinaryMeasurementsRot3 gtsam::DiscreteKey gtsam::KeyPairDoubleMap 
gtsam::gtsfm::MatchIndicesMap gtsam::gtsfm::KeypointsVector 
gtsam::gtsfm::SfmTrack2dVector \
--template "python/gtsam/gtsam.tpl" \
--is_submodule \
--use-boost

  < cmake/dllexport.h.in sed \
  ' s/#cmakedefine/#define/g;
s/@library_name@/GTSAM/g;
  ' > gtsam/dllexport.h

  < gtsam/config.h.in sed \
  ' s/#cmakedefine/#define/g;
s/@GTSAM_VERSION_MAJOR@/4/g;
s/@GTSAM_VERSION_MINOR@/2/g;
s/@GTSAM_VERSION_PATCH@/0/g;
s/@GTSAM_VERSION_NUMERIC@/40200/g;
s/@GTSAM_VERSION_STRING@/4.2a9/g;
s/@GTSAM_EIGEN_VERSION_WORLD@/3/g;
s/@GTSAM_EIGEN_VERSION_MAJOR@/4/g;
s/.*define .*_USE.*_MKL.*//g;
s/.*define GTSAM_EIGEN_VERSION_MINOR.*//g;
s/.*define GTSAM_ALLOCATOR_BOOSTPOOL.*//g;
s/.*define GTSAM_ALLOCATOR_STL.*//g;
s/.*define GTSAM_SLOW_BUT_CORRECT_BETWEENFACTOR.*//g;
s/.*define GTSAM_USE_QUATERNIONS.*//g;
  ' > gtsam/config.h

  /usr/bin/c++ \
-DBOOST_ALL_NO_LIB \
-DBOOST_ATOMIC_DYN_LINK \
-DBOOST_CHRONO_DYN_LINK \
-DBOOST_DATE_TIME_DYN_LINK \
-DBOOST_FILESYSTEM_DYN_LINK \
-DBOOST_PROGRAM_OPTIONS_DYN_LINK \
-DBOOST_REGEX_DYN_LINK \
-DBOOST_SERIALIZATION_DYN_LINK \
-DBOOST_SYSTEM_DYN_LINK \
-DBOOST_THREAD_DYN_LINK \
-DBOOST_TIMER_DYN_LINK \
-Dgtsam_py_EXPORTS \
-I"." \
-I"CppUnitLite" \
-isystem /usr/include/python3.11 \
-isystem /usr/include/eigen3 \
-g \
-O2 \
-fstack-protector-strong \
-Wformat \
-Werror=format-security \
-Wdate-time \
-D_FORTIFY_SOURCE=2 \
-g \
-DNDEBUG \
-fPIC \
-fvisibility=hidden \
-o /tmp/tst.o \
-c geometry.cpp

On abel.d.o this crunches for a while, and then says

  cc1plus: out of memory allocating 152612 bytes after a total of 59252736 bytes

I will report this upstream, but I don't yet know what to tell them. Any
suggestions here for debian and/or for upstream?

If I was upstream, I'd do a lot of this differently, but I'm not going
to ask them to majorly rearchitect their thing.

Thanks.

Re: Potential MBF: packages failing to build twice in a row

[Timo Röhling]

Hi,

* Johannes Schauer Marin Rodrigues  [2023-08-13 22:28]:

since this issue seems to be affecting a few more packages than plakativ, I
wanted to ask here what the canonical way is to fix this issue?

There's talk on #d-python if pybuild could/should deal with it; I'd
give it a few days and see if that pans out.

If not, extend-diff-ignore as Scott suggested or simply removing the
egg-info folders with a *.egg-info/ line in d/clean should both
work. (Personally, I'd use extend-diff-ignore if the egg-info is
also shipped in the source tarball and d/clean if not)


Cheers
Timo

--
⢀⣴⠾⠻⢶⣦⠀   ╭╮
⣾⠁⢠⠒⠀⣿⡁   │ Timo Röhling   │
⢿⡄⠘⠷⠚⠋⠀   │ 9B03 EBB9 8300 DF97 C2B1  23BF CC8C 6BDD 1403 F4CA │
⠈⠳⣄   ╰╯


signature.asc
Description: PGP signature

Re: Potential MBF: packages failing to build twice in a row

[Vincent Bernat]

On 2023-08-10 14:38, Lucas Nussbaum wrote:

On 08/08/23 at 10:26 +0200, Helmut Grohne wrote:

Are we ready to call for consensus on dropping the requirement that
`debian/rules clean; dpkg-source -b` shall work or is anyone interested
in sending lots of patches for this?


My reading of the discussion is that there's sufficient interest for
ensuring that building-source-after-successful-binary-build works.


There is a bias asking d-devel@. You'll get people with enough time on 
their hands to care about this. Nobody ever complained about not being 
able to build twice in a row for years. We are asking a lot of people to 
fix problems that don't really exist.


I don't have time to deal with the amount of bug reported against my own 
packages (most of them with Python) and just having the social pressure 
to spend time on them make me wonder if I really want to invest time at 
all. If these bugs become RC, so be it.

Re: Potential MBF: packages failing to build twice in a row

[Scott Kitterman]

On August 13, 2023 8:28:08 PM UTC, Johannes Schauer Marin Rodrigues 
 wrote:
>Hi,
>
>Quoting Simon McVittie (2023-08-06 12:27:04)
>> On Sat, 05 Aug 2023 at 21:29:08 +0200, Andrey Rakhmatullin wrote:
>> > I expect all Python packages that ship
>> > $name.egg-info and don't remove it in clean and don't exclude it via
>> > extend-diff-ignore (all of which is unneeded busywork even if recommended)
>> > to behave the same.
>> 
>> Python packages that *don't* ship $name.egg-info in their upstream source,
>> don't remove it in clean and don't exclude it via extend-diff-ignore will
>> also fail Lucas' test if they are 3.0 (quilt) format (or presumably will
>> have unintended diff instead if they are format 1.0). That's the only reason
>> bmap-tools_3.6-2 was on Lucas' list, for example.
>
>I just had #1045290 filed against a Python package of mine:
>
>> dpkg-source: info: local changes detected, the modified files are:
>>  plakativ-0.5.1/plakativ.egg-info/SOURCES.txt
>
>since this issue seems to be affecting a few more packages than plakativ, I
>wanted to ask here what the canonical way is to fix this issue?

Generally something like this (although depending on the Python build tool, it 
may need some adjustment):

https://salsa.debian.org/python-team/packages/dkimpy/-/commit/bf39be7563b680bdc73a67a4db7dc1af05af

Scott K

Re: Potential MBF: packages failing to build twice in a row

[Johannes Schauer Marin Rodrigues]

Hi,

Quoting Simon McVittie (2023-08-06 12:27:04)
> On Sat, 05 Aug 2023 at 21:29:08 +0200, Andrey Rakhmatullin wrote:
> > I expect all Python packages that ship
> > $name.egg-info and don't remove it in clean and don't exclude it via
> > extend-diff-ignore (all of which is unneeded busywork even if recommended)
> > to behave the same.
> 
> Python packages that *don't* ship $name.egg-info in their upstream source,
> don't remove it in clean and don't exclude it via extend-diff-ignore will
> also fail Lucas' test if they are 3.0 (quilt) format (or presumably will
> have unintended diff instead if they are format 1.0). That's the only reason
> bmap-tools_3.6-2 was on Lucas' list, for example.

I just had #1045290 filed against a Python package of mine:

> dpkg-source: info: local changes detected, the modified files are:
>  plakativ-0.5.1/plakativ.egg-info/SOURCES.txt

since this issue seems to be affecting a few more packages than plakativ, I
wanted to ask here what the canonical way is to fix this issue?

Thanks!

cheers, josch

signature.asc
Description: signature

Strange Error After Update

[John Darrah]

Hi,

I receive the following error after logging into the Gnome desktop
following and update.

Detailed errors from the package manager follow:

failed to build result for 71b677ca0f1bc2c5b804fa1d59e52064ce589293

This is reported in a pop-up from the Gnome Software app I believe, and
today is the second time I have seen this in the last week.

I'm tracking Testing and always update in a Linux console so I can view
the update progress as it happens. I have seen no errors or failed
packages. The logs also don't indicate any issues and everything works
fine.

Thanks

-- john

Re: Request for review of debootstrap change [was: Re: Second take at DEP17 - consensus call on /usr-merge matters]

[Holger Levsen]

On Fri, Aug 11, 2023 at 10:56:03PM +0200, Helmut Grohne wrote:
> > what about cdebootstrap?
> cdebootstrap (and mmdebstrap) never implemented a merging step[1] and to
> this date rely on the usrmerge package doing it at postinst time. Once
> base-files ships the aliasing symlinks, both will produce /usr-merged
> trees without any modifications. The reason that we need a change to
> debootstrap is that its current merging implementation breaks when
> base-files ships aliasing symlinks.
> 
> So the main reason for doing this change to debootstrap is that it
> enables us to continue supporting cdebootstrap and mmdebstrap without
> any changes there.

ah, thank you!


-- 
cheers,
Holger

 ⢀⣴⠾⠻⢶⣦⠀
 ⣾⠁⢠⠒⠀⣿⡁  holger@(debian|reproducible-builds|layer-acht).org
 ⢿⡄⠘⠷⠚⠋⠀  OpenPGP: B8BF54137B09D35CF026FE9D 091AB856069AAA1C
 ⠈⠳⣄

Just because other people are also responsible, does not mean you are not
responsible.


signature.asc
Description: PGP signature