Re: F35 Change: Debuginfod By Default (Self-Contained Change proposal)

[Mark Wielaard]

Hi Panu,

On Wed, Jun 05, 2024 at 09:26:14AM +0300, Panu Matilainen wrote:
> On 6/4/24 21:43, Frank Ch. Eigler wrote:
> >dvlasenk wrote:
> >
> >>[...] Do you understand how many fetches of debuginfo will be
> >>attempted by e.g.  a kernel build tooling when it runs readelf on 8000
> >>freshly built modules _for every kernel build_? How slow it is?
> >
> >If remote debuginfo is not needed for these particular readelf
> >invocations, then the tools should not be making debuginfod calls.
> >Can you help identify examples?
> >
> >>Now various tools need to forcibly unset the variable to stop this
> >>madness.
> >
> >The defaults are set with normal developers in mind.
> 
> The problem is that nobody expects a tool this low-level to do
> internet lookups, and silently at that. It's like 'ls' suddenly
> started looking up stuff from the net *by default*.
> 
> And, not just interactive use but scripts too, scripts that existed
> years or even decades before this thing and cannot possibly expect
> such activity. Like the case of
> https://bugzilla.redhat.com/show_bug.cgi?id=2079600

This surprises me. You are right that nobody would expect readelf -d
(to get the dynamic table entries) would use debuginfod. And it indeed
doesn't. Might this just have been an old bug in binutils since fixed?

> And, it's made much worse by the packaging which means you cannot
> just remove the package and be done with it, because so much
> important tooling links to it. If the library and the configuration
> to actually enable it were split between different packages, it'd be
> less offensive at least.

I am sorry you are so offended by the packaging, but nobody has
suggested this before. Please do file a bug with this suggestion if
you think it makes sense to split the default /etc files and the
actual debuginfod-client library.

Cheers,

Mark
--
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam, report it: 
https://pagure.io/fedora-infrastructure/new_issue

Re: OCaml binary stripping (was: Re: OCaml 5 again)

[Mark Wielaard]

Hi,

On Wed, Jul 12, 2023 at 03:38:35PM -0600, Jerry James wrote:
> On Wed, Jul 12, 2023 at 3:34 PM Richard W.M. Jones  wrote:
> > I looked through the section names with objdump -h and none of them
> > looked unusual.  However I think in the past when we analyzed this we
> > found it was doing something really crazy, like actually appending the
> > bytecode to the file (ie. not using the ELF format at all).
> 
> That was the conclusion I had just come to.  In the unstripped binary,
> the ELF structure accounts for about 700K, but the file size is over
> 3M.  It looks like they really do just append the bytecode to the
> file.

Aha, yeah, then any tool that tries to process the ELF file will
likely just overwrite that or throw it away because it has no way of
knowing that data is even part of the file.

Cheers,

Mark
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam, report it: 
https://pagure.io/fedora-infrastructure/new_issue

Re: OCaml binary stripping (was: Re: OCaml 5 again)

[Mark Wielaard]

Hi,

On Wed, Jul 12, 2023 at 09:54:49AM -0600, Jerry James wrote:
> On Wed, Jul 12, 2023 at 4:38 AM Richard W.M. Jones  wrote:
> > During the OCaml 5 rebuild I found there's a generic problem on s390x
> > & ppc64le (ie. the bytecode-only architectures) involving stripping of
> > OCaml binaries. [...]
> >
> > The binary is correct when built, but gets corrupted after one of
> > these steps is applied (not sure which):
> >
> >   + /usr/lib/rpm/redhat/brp-strip-lto /usr/bin/strip
> >   + /usr/lib/rpm/brp-strip-static-archive /usr/bin/strip

OK, that is later than I suspect.  I believe these steps run after the
debuginfo packages are created.  And I don't know much about these
steps, so if that is the case my advise might not be relevant.

> > Anyway this could potentially affect any s390x / ppc64le OCaml package
> > that contains a binary so it's something to look out for ...  For some
> > reason dune doesn't seem to be affected.
> [...]
> I encountered that with a couple of packages, ocaml-findlib and
> coccinelle.  Sorry I missed the others.  Yes, stripping those binaries
> removes the bytecode payload, leaving only the bytecode interpreter.
> I also noticed that dune isn't affected, which is interesting.  I
> wonder if it is the difference between -output-complete-exe and
> -custom.  Dune now uses the former instead of the latter.

It would be interesting to know what the difference between these
methods is. Specifically if they create different names (or
alloced/unalloced) ELF sections.

If the issue is with creating the debuginfo packages, then it likely
is because there is an (unallocated) ELF section which is moved from
the main executable to the .debug file (because debuginfo-find
believes it isn't needed at runtime.

In that case, and you can find out how such sections are named, then
you might be able to workaround it by adding someting like:

%define _find_debuginfo_opts --keep-section .special_ocaml_bytecode

Cheers,

Mark
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam, report it: 
https://pagure.io/fedora-infrastructure/new_issue

Re: Conflicting build-ids in nekovm and haxe

[Mark Wielaard]

Hi Andy,

(Please do put me on CC, I do try to read all of devel list, but it is easy to 
miss something)

On Tue, May 02, 2023 at 11:42:30AM -, Andy Li wrote:
> Sorry for the late reply. I just got back to this issue.
> 
> > In general this (should) only happen if the binaries are really
> > identical (e.g. when one of the packages build requires the other, but
> > instead of linking/rebuilding some sources it simply copies a binary
> > directly).
> > 
> > The build-id is (also) derived from the full nvra. So really cannot be
> > identical even if the sources are. But this relies on the binaries
> > being build with debuginfo. So maybe some code isn't build with -g?
> 
> I *think* both neko and haxelib were built with debuginfo, as I can find `-g` 
> in the logs.

Yes, that doesn't seem to be the issue.

The fetching the latest x86_64 packages
haxe-4.2.5-5.fc38.x86_64.rpm
nekovm-2.3.0-11.fc38.x86_64.rpm
you can see that the conflicting build-id is
 /usr/lib/.build-id/b8/acf4f8271b78cd6dca636cb5b877c5c64f47f4
Which points to /usr/bin/haxelib in the first package
and to /usr/bin/neko in the second.

And if you compare them then they are indeed the same ELF file:
$ eu-elfcmp usr/bin/neko usr/bin/haxelib
$ echo $?
0

But they aren't really the same:
$ cmp usr/bin/neko usr/bin/haxelib
cmp: EOF on usr/bin/neko after byte 24632, in line 29

So after the ELF file data there is something else.

What I suspect is happening is that haxelib is really just a direct
copy of the neko binary with some bytecode added.

Cheers,

Mark
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam, report it: 
https://pagure.io/fedora-infrastructure/new_issue

Re: It’s time to transform the Fedora devel list into something new

[Mark Wielaard]

Hi,

On Thu, Apr 20, 2023 at 07:21:54PM -0400, Simo Sorce wrote:
> The mailing list make messages land in my client, on which I am very
> efficient, therefore I can check all messages once a day, and respond
> if I find a worthy topic.
> 
> Unless this discourse has some great mail bridge (it doesn't) or maybe
> an rss feed (I do not use those at work, but I guess I could ?) So that
> I can skim messages on my terms, I think I (and those like me) will be
> the next "missing people".

I think I would fall in the same category. The fedora-devel mailinglist
keeps me connected to the project even though I rarely reply to messages.

> Your own post communicates to me (whether you intended it or not) that
> in the end the thread that will be generated by this post won't matter,
> because this is just a courtesy post and you already think that the
> opinion of the "minority of self selected mailing list lovers and
> dinosaurs" does not matter much.

Agreed. It was a very long post with several important and interesting
observations about communication in a large groups of people. But it
felt like it wasn't meant as actually discussing how we can
communicate better, but to work towards a conclusion that a specific
forum technology should be adopted and get rid of people who use the
mailinglist to participate in the project.

Cheers,

Mark
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam, report it: 
https://pagure.io/fedora-infrastructure/new_issue

Re: Yet another unwinding approach

[Mark Wielaard]

Hi Florian,

On Wed, 2023-01-18 at 15:01 +0100, Florian Weimer wrote:
> We won't have unwind data for JIT-compiled code, including libffi
> trampolines.  We could stop backtracing there (what does the ABI say
> about frames without unwinding information?), but I'm not sure if
> that's
> going to be useful for the desktop.

In this model, such generated code would register their own unwinder
(or .eh_frame contribution).

> > But it is something to keep in mind, accidents happen. I assume
> > that
> > any (seg) fault caused by the unwind signal thread will simply end
> > that
> > contribution of user space to the backtrace (instead of really
> > generating a SEGV)
> 
> That smells like SEH. 8-/

Sorry, I don't know what SEH means in this context and whether that is
good or bad thing (8-/ isn't part
of https://en.wikipedia.org/wiki/List_of_emoticons)

> > But it is tricky to handle that without kernel support. As a fallback
> > you could install a SIGSEGV handler that handles the fault and somehow
> > makes the original SIGPROF based handler (if you use that) sigreturn.
> 
> Signal handlers are process-wide, and SIGPROF does not suspend the
> entire process.  So I really don't think this can be made to work
> reliably without kernel support.

I don't think any sampling profile based backtrace support is going to
be perfect, but I don't think that is the goal. Even gdb won't be able
to create perfect backtraces all the time. We just need something that
works out of the box most of the time. kernel support would be nice,
but I don't think we shouldn't try some experimentation first before we
settle on the perfect kernel interface. Whatever we can do in user
space we should first try out in user space imho.

Cheers,

Mark
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam, report it: 
https://pagure.io/fedora-infrastructure/new_issue

Re: Yet another unwinding approach

[Mark Wielaard]

Hi Florian,

On Wed, 2023-01-18 at 07:22 +0100, Florian Weimer wrote:
> * Daniel Colascione:
> 
> > See, both pro-FP and anti-FP camps think that it's the kernel that has
> > to do the unwinding unless we copy whole stacks into traces.
> 
> Well, I think we should explore hardware-assisted backtraces (shadow
> stacks), which hopefully are going to get merged in Linux 6.2.

Yes, that is also a good alternative if available. The shadow stack
seems to be ideal because it is just the function return addresses,
which basically is just the backtrace you are looking for.

> If the unwind information is incomplete, this …
> 
> > 7) signal handler unwinds the calling thread however it wants (and can
> > sleep and take page faults if needed)
> 
> … might encounter segmentation faults and terminate the process.  So
> far, incorrect unwind information (whether it's a clobbered frame
> pointer, or missing DWARF information about clobbered registers) is not
> a problem, but it's kind of hard to validate this information from
> within the process itself.  Maybe we'd have to add a magic memcpy first
> to the vDSO, which the kernel recognizes based on the code addresses,
> and suppresses sending the signal for it.

Yeah, I am not too afraid of that happening with an .eh_frame based
unwinder unless someone deliberately produced a bad table (or dlcloses
a library which still has code on the call stack). You still have to be
careful about the stack bounds of course.

But it is something to keep in mind, accidents happen. I assume that
any (seg) fault caused by the unwind signal thread will simply end that
contribution of user space to the backtrace (instead of really
generating a SEGV)

But it is tricky to handle that without kernel support. As a fallback
you could install a SIGSEGV handler that handles the fault and somehow
makes the original SIGPROF based handler (if you use that) sigreturn.

Cheers,

Mark
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam, report it: 
https://pagure.io/fedora-infrastructure/new_issue

Re: Yet another unwinding approach

[Mark Wielaard]

Hi Daniel,

On Mon, Jan 16, 2023 at 08:30:21PM -, Daniel Colascione wrote:
> As mentioned in [1], instead of finding a way to have the kernel
> unwind user programs, we can create a protocol through which the
> kernel can ask usermode to unwind itself.

I like this idea and was discussing and thinking along similar lines.
It would be great if we can implement this using existing mechanisms
to prototype something to show it is feasible and fast enough.  And
for existing linux/glibc installs that don't yet have the new
interfaces.

> It could work like this:
> 
> 1) backtrace requested in the kernel (e.g. to a perf counter
> overflow)
> 
> 2) kernel unwinds itself to the userspace boundary the usual way
> 
> 3) kernel forms a nonce (e.g. by incrementing a 64-bit counter)
> 
> 4) kernel logs a stack trace the usual way (e.g. to the ftrace ring
> buffer), but with the final frame referring to the nonce created in
> the previous step
> 
> 5) kernel queues a signal (one userspace has explicitly opted into
> via a new prctl()); the siginfo_t structure encodes (e.g. via
> si_status and si_value) the nonce

So before it does this prctl the process needs to setup all
datastructures it needs to safely handle the unwinding during signal
handling. That does mean that early process setup won't be able to be
profiled with user backtraces.

> 6) kernel eventually returns to userspace; queued signal handler
> gains control

So at this point, if the event triggered while that user space thread
was running the event logged by the kernel is basically just that
nonce?

> 7) signal handler unwinds the calling thread however it wants (and
> can sleep and take page faults if needed)

So in theory this can do anything that is async signal safe. But what
if it takes too long and another event gets triggered? Or it does a
syscall that produces an event? Another signal arrives? Or if it
causes a SEGV?

> 8) signal handler logs the result of its unwind, along with the
> nonce, to the system log (e.g. via a new system call, a sysfs write,
> an io_uring submission, etc.)

A (shared) memory region seems simplest, whatever has been put into it
when the signal handler returns is the user space backtrace
contribution. Maybe the prctl call can set this up? Or maybe the
kernel can provide it through one of the siginfo_t fields?

> Post-processing tools can associate kernel stacks with user stacks
> tagged with the corresponding nonces and reconstitute the full
> stacks in effect at the time of each logged event.
>
> We can avoid duplicating unwindgs too: at step #3, if the kernel
> finds that the current thread already has an unwind pending, it can
> uses the already-pending nonce instead of making a new one and
> queuing a signal: many kernel stacks can end with the same user
> stack "tail".

This is probably a generic optimization for most backtraces, most will
have a similar tail.
 
> One nice property of this scheme is that the userspace unwinding
> isn't limited to native code. Libc could arbitrate unwinding across
> an arbitrary number of managed runtime environments in the context
> of a single process: the system could be smart enough to know that
> instead of unwinding through, e.g. Python interpreter frames, the
> unwinder (which is normal userspace code, pluggable via DSO!) could
> traverse and log *Python* stack frames instead, with meaningful
> function names. And if you happened to have, say, a JavaScript
> runtime in the same process, both JavaScript and Python could
> participate in the semantic unwinding process.
>
> A pluggable userspace unwind mechanism would have zero cost in the
> case that we're not recording stack frames. On top of that, a
> pluggable userspace unwinder *could* be written to traverse frame
> pointers just as the kernel unwinder does today, if userspace thinks
> that's the best option. Without breaking kernel ABI, that userspace
> unwinder could use DWARF, or ORC, or any other userspace unwinding
> approach. It's future-proof.

This is nice, but does need some coordination for handing off the
unwinding context between different unwinders. You could register an
unwinder by memory region (if address is in this range, it is in the
lua interpreter). And then you can start generating a backtrace using
the dedicated unwinder for that memory region. That original unwinder
can start with the full ucontext_t. But what is the contract between
unwinders? If we start with e.g. a frame-pointer unwinder then when we
get to a part that needs to use the python unwinder we only have a pc,
sp and fp left. Is that enough context for the python unwinder to
continue?

What would we need to prototype this idea and show that we can produce
quick backtraces using fast eh_frame unwinding before we convinced the
kernel to provide this interface and have it in glibc (or the vdso,
without any fancy caching, it might fit the vdso)?

We could use LD_PRELOAD or some ptrace parasite code like criu uses to
insert the unwinder code a

Re: FESCo revote on "Add -fno-omit-frame-pointer" Change proposal [was Re: Schedule for Tuesday's FESCo Meeting (2023-01-03)]

[Mark Wielaard]

Hi Matthew,

On Mon, Jan 09, 2023 at 11:47:54AM -0500, Matthew Miller wrote:
> On Sun, Jan 08, 2023 at 06:06:47PM +0100, Kevin Kofler via devel wrote:
> > Therefore, I hereby request that the vote be annulled as having happened
> > in violation of the Change policy.
> 
> So, from a purely "what are the rules?" view, the Change process says:
> 
>   FESCo will vote to approve or deny a change proposal in accordance with
>   the FESCo ticket policy.
> 
> ... and I won't quote all of that, but looking at
> https://docs.fedoraproject.org/en-US/fesco/#_ticket_policy...
> I don't see any violations, either in the letter or the spirit of what is
> written.
> 
> And, from a practical point of view, since this passed with six +1 votes,
> I'm not sure what benefit canceling and re-voting would really add.

It does feel to me as being against the spirit, and only not against
the letter because it is presented as a "revote" on an existing change
proposal instead of proposing a new change proposal (which this really
is IMHO).

Practically people have started preparing for the mass rebuild at the
end of last year since that was when the change checkpoint for change
proposals requiring a mass rebuild was. And at that point the Change
Proposal was already decided to not be included. So it was never
expected that the build flags would suddenly change so drastically
(and some flags are still wrong). Cancelling and backing out these
last minute changes will cause a lot less stress.

Socially I think it will be better for all involved if the policies on
revoting and/or reintroducing a change proposal are first clarified
before allowing a revote. At the moment everybody involved seems hurt
because of the unclear policy. Not having clear rules on the needed
visibility and time needed to discuss this revote/resubmission of the
change proposal caused people to assume the worst about others. Lets
reset and take the time to heal first, so people start actually
talking about real solutions again.

Cheers,

Mark
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam, report it: 
https://pagure.io/fedora-infrastructure/new_issue

Re: Schedule for Tuesday's FESCo Meeting (2023-01-03)

[Mark Wielaard]

Hi Neal,

On Wed, 2023-01-04 at 08:44 -0500, Neal Gompa wrote:
> On Wed, Jan 4, 2023 at 8:30 AM Vitaly Zaitsev via devel
>  wrote:
> > 
> > Already rejected proposal was submitted because big corporations
> > weren't
> > happy with the results. This is a VERY BAD precedent for Fedora.
> > 
> 
> Actually, the Change owners were prepared to give up. I was the one
> that pushed for it to be reconsidered

I must say I find this rather odd. As you say there was an agreement on
moving forward without frame pointers. And as far as I could see there
was even an healthy discussion about alternative ways to get faster and
more accurate unwinding/backtracing between the profiling and
compiler/tools hackers.

I don't mind if you would re-try to get this change in for f39 or f40
if it turns out those discussions about alternative unwinders didn't
result in faster/better profilers.

But trying to do it while multiple stackholders were away and unaware
of this because it wasn't really announced doesn't feel good.

Cheers,

Mark
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam, report it: 
https://pagure.io/fedora-infrastructure/new_issue

Re: Uninitialized variables and F37

[Mark Wielaard]

Hi Steve,

On Wed, 2022-05-11 at 22:35 -0400, Steve Grubb wrote:
> On Monday, May 9, 2022 5:10:07 AM EDT Daniel P. Berrangé wrote:
> > Are you going to take this idea forward and make a formal change proposal
> > for Fedora to set -ftrivial-auto-var-init=zero by default in its default
> > RPM build flags ?
> 
> I've connected with the gcc folks and we will file a proposal in the near 
> future.

I am not a fan, because I think this mainly hides bugs. But also
because the original change proposal made it sound like we don't have
any other way to find and fix these kind of bugs. While a little
analysis of your examples showed we can find and fix 100% of these
issues with the existing gcc and analysis tools.

So my counter proposal would probably be to enforce -Werror and running
all package test-suites under valgrind. But maybe others won't like
that "solution".

If you do propose this again could you at least make clear it's another
tool in the toolbox, not a replacement, and that the other tools do
work, and are trusted (if you pay attention to them). Then at least we
could have a honest discussion why (and in which circumstances) each of
the tools might or might not work/catch an issue. And if we
may/can/should require packagers to pay more attention to compiler
warnings and/or running analysis tools over the sources they package.

Thanks,

Mark
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: Uninitialized variables and F37

[Mark Wielaard]

Hi,

On Thu, Jan 27, 2022 at 10:41:36AM +0100, Roberto Ragusa wrote:
> On 1/22/22 10:05 PM, Mark Wielaard wrote:
> 
> > So I would give valgrind a 6/6 (100%) score :)
> 
> But if the compiler starts copying zeros on uninitialized memory,
> valgrind loses any ability to detect the defect in the code.

Yes. So that is the compromise. You'll always get initialized zeros
for local variables, so any usage is defined (though probably buggy).
But some of the tools, like valgrind memcheck, will be unable to
detect the buggy code.

If you believe the tools we have are pretty bad to begin with and/or
not actually used by people to find such bugs then this is a good
compromise. If you believe the tools are pretty good for detecting
these issues (and I believe they are, the example given was just
unfortunate because some of the issues weren't actually bad code and
some others were rightfully optimized out, so would never trigger),
then it is a bad compromise. But we definitely need to encourage
people to use the tools more.

Cheers,

Mark
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: Uninitialized variables and F37

[Mark Wielaard]

Hi Steve,

On Fri, 2022-01-21 at 13:04 -0500, Steve Grubb wrote:
> This is a continuation of the discussion from F36 Change: GNU
> Toolchain Update.
> 
> Uninitialized variables are a big problem. They can be sources of information 
> exposure if parts of a buffer are not initialized. They can also cause 
> unexpected execution paths if the attacker can groom the memory to a value of 
> their choosing. If the variable is a pointer to heap, this can cause free to 
> corrupt memory under certain circumstances. If the uninitialized memory is 
> part of user input, this can lead to improper input validation. This is not 
> hypothetical. All of these come from a paper doing an emprical study of 
> android flaws. [1]  The data used in the paper is here. [2]
> 
> Part of the problem is that compilers and static analysis tools can't always 
> find them. I created a test program that has 8 uses of unintialized 
> variables. 
> Gcc 11 misses all of them. Gcc 12 finds 2. Clang 13 finds 1. cppcheck finds 2 
> or 
> 3 - but does so much complaining you'd think it found all. Valgrind finds 2. 
> Flexelint, a commercial linter, finds 1.
> 
> Since tools can't always find them, the only option we have right now is 
> force 
> initialization to something the attacker cannot control.

Although I see how having a deterministic bug (use of know bad zero
value) is better than having an undeterministic bug (use of undefined
value), it is still a bug. And I think you don't give the tools we have
much credit.

In my experience almost all such bugs will be found with a combination
of gcc -fsanitize=undefined and valgrind memcheck. Your test program
didn't trigger most because gcc, when optimizing, is smart enough to
simply not emit unused code.

Of course gcc -fsanitize=undefined cannot be used on production code.
And while valgrind memcheck can be used on production code, it is often
fairly slow (and arguably now it is too late, because already in
production).

Although I am not against trying to turn nondeterministic bugs into
deterministic ones and getting rid off more undefined code, I am
slightly worried it means those bugs will be harder to find in
production. Also I really hope we do also encourage people to use the
various tools to find those bugs before they get into production. They
really aren't as bad at finding these issues as you make them out to
be.

Cheers,

Mark
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: Uninitialized variables and F37

[Mark Wielaard]

On Sat, 2022-01-22 at 20:49 +, Richard W.M. Jones wrote:
> On Sat, Jan 22, 2022 at 03:00:14PM -0500, Steve Grubb wrote:
> > On Saturday, January 22, 2022 6:36:01 AM EST Vitaly Zaitsev via
> > devel wrote:
> > > On 21/01/2022 19:04, Steve Grubb wrote:
> > > > Uninitialized variables are a big problem.
> > > 
> > > Yes, but as a package maintainer, I don't want to deal with
> > > dozens of
> > > crashes after this change.
> > 
> > As much as I don't want this to cause unnecessary work for anyone,
> > I also 
> > don't want to see preventable exploits happen. Nearly all major
> > software 
> > vendors are doing this.
> > 
> > I mentioned in the original proposal that I have a test program
> > with 8 test 
> > cases. This is it in case anyone wants to try it out:
> > 
> > #include 
> > 
> > struct test {
> > int one;
> > int two;
> > };
> > 
> > void func2(const struct test *t)
> > {
> > if (t->one == 0)
> > printf("init func2\n");
> > 
> > if (t->two == 0)  // Uninitialized 1
> > printf("uninit func2\n");
> > }
> > 
> > void func1(struct test *t)
> > {
> > t->one = 1; // two is uninitialized
> > func2(t);
> > }
> > 
> > int func3(int num)
> > {
> > if (num) // Uninitialized 2
> > return num;
> > else
> > return 0;
> > }
> > 
> > void func4(int *a, int max)
> > {
> > int i;
> > // skip the first
> > for (i=1; i > a[i] = 0;
> > }
> > 
> > void func5(const int *a, int max)
> > {
> > int i;
> > for (i=0; i > if (a[i]) // Uninitialized 3
> > printf("func5: %d\n", i);
> > }
> > }
> > 
> > int func6(const int *num)
> > {
> > if (*num) // Uninitialized 4
> > return *num;
> > else
> > return 0;
> > }
> > 
> > int j;
> > int func7(void)
> > {
> > return j;  // Uninitialized 5
> > }
> > 
> > void func8(const int *a, int max)
> > {
> > int i;
> > for (i=0; i > if (a[i]) // Uninitialized 6
> > printf("func8: %d\n", i);
> > }
> > }
> > 
> > enum {RED, AMBER, GREEN, BLACK};
> > 
> > int main(void)
> > {
> > struct test t;
> > int num;
> > int arry[10];
> > int go;
> > int color = BLACK;
> > 
> > func1(&t);
> > func3(num);
> > func4(arry, 10);
> > func5(arry, 10);
> > func6(&num);
> > 
> > printf("num: %d\n", num); // Uninitialized 7
> > printf("func7: %d\n", func7());
> > arry[0] = func7(); // copy uninitiliazed j into arry[0]
> > func8(arry, 10);
> > 
> > switch (color) {
> > case RED:
> > case AMBER:
> > go = 0;
> > break;
> > case GREEN:
> > go = 1;
> > break;
> > }
> > 
> > printf("go :%d\n", go); // Uninitialized 8
> > 
> > return 0;
> > }
> > 
> > 
> > Detection results:
> > gcc11 : 0
> > gcc11+fanalyzer: 0
> > gcc12: 2
> > gcc12+fanalyzer: 3
> > cppcheck: 2   but describes different aspects of the same problems
> > gcc11+asan: 0
> > gcc11:+ubsan: 0
> > clang13: 1
> > valgrind+clang: 0
> > valgrind+gcc: 2
> > Flexelint: 1
> > splint: 2
> > 
> > The one surprising result is that valgrind's results differ by the
> > compiler 
> > choice.
> 
> valgrind seems to do better for me than described.  It spots errors
> in
> these places:
> 
>   test.c:13 (Uninitialised 1)
>   test.c:25 (Uninitialised 2)
>   test.c:43 (Uninitialised 3)
>   test.c:50 (Uninitialised 4)
>   test.c:87 (Uninitialized 7)
>   test.c:102 (Uninitialized 8)
> 
> So I'd say it's getting 6/8.
> 
> I did:
> 
>   $ gcc -g test.c -o test
>   $ valgrind ./test

Right. That is because:

return j;  // Uninitialized 5

Isn't actually use (it just returns) and j is an global variable, which
are already initialized to zero by default.

And so also this:

arry[0] = func7(); // copy uninitiliazed j into arry[0]

Is actually assigning an initialized (zero) value.

So I would give valgrind a 6/6 (100%) score :)

If you compile with -O2 gcc will only leave 2 actual uses of
uninitialized values in the code, so valgrind can also only spot 2. The
compiler (correctly) determines all other code isn't actually used and
so optimizes it out.

Cheers,

Mark
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: F36 Change: GNU Toolchain Update (gcc 12, glibc 2.35) (late System-Wide Change proposal)

[Mark Wielaard]

Hi,

On Thu, 2022-01-20 at 17:56 -0500, Marek Polacek wrote:
> On Tue, Jan 11, 2022 at 05:00:57PM -0500, Carlos O'Donell wrote:
> > On 1/11/22 13:00, Steve Grubb wrote:
> > > Reading through the GCC 12 changes, there is a significant new feature to 
> > > GCC 
> > > that would appear to be useful for security. There is a new:
> > > 
> > > -ftrivial-auto-var-init=zero
> > > 
> > > flag that initializes all stack variables to zero. Zero being a nice safe 
> > > value that makes programs crash instead of being exploitable.
> > > 
> > > Are there plans to enable this flag so that all applications, but more 
> > > importantly the kernel, are hardened against uninitialized stack 
> > > variables? 
> > > This is one of the major classes of security bugs that could potentially 
> > > be 
> > > eliminated during this mass rebuild.
> > 
> > There are currently no plans that I am aware of that involve turning on
> > '-ftrivial-auto-var-init=zero' in the short term for Fedora. CC'ing Jakub
> > and Marek to comment.
> 
> Also not aware of any plans to always enable it.
>  
> > It is something that should be discussed, turned on in Rawhide first,
> > and likely via redhat-rpm-config default flags first, and then we should
> > fix any fallout.
> > 
> > I'd only be comfortable if we did it early and worked through the 
> > consequences.
> > So it could be something to discuss for F37.
> 
> Right.  It reminds me of MALLOC_PERTURB_, but for automatic variables.
> 
> Obviously it's always important to measure its slowdown (maybe run a SPEC
> benchmark) / compile time / stack usage.  Some of it has been done:
> https://gcc.gnu.org/pipermail/gcc-patches/2021-January/562872.html
> but that was an early version of the patch.  Still, it seems like it'd be
> acceptable.
> 
> It's a new feature, only present in GCC 12 (which hasn't been released as of
> now), so I think it needs more testing before it could be (considered to be)
> enabled by default.
> 
> A good thing is that it doesn't suppress the -Wuninitialized warning so
> you still get a chance to fix your bugs.  It also comes with an attribute
> to keep variables uninitialized even when the options is turned on.

Note that it does make it impossible for valgrind memcheck to track use
of uninitialized (stack) values because it will believe they have been
initialized with zero in any code that is build with this flag, and it
will assume that zero is a valid value.

Obviously as a valgrind hacker I am biased, believing lots of people
run valgrind on production code. So I do think that makes it harder to
find real security issues. Now the code will just appear to work using
a possibly bogus value of zero instead of valgrind memcheck pointing
out where and why you are using an uninitialized value.

Cheers,

Mark
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: F36 Change: Package information on ELF objects (System-Wide Change proposal)

[Mark Wielaard]

Hi Lennart,

On Thu, 2021-10-28 at 09:56 +0200, Lennart Poettering wrote:
> But as someone who's at the upstream receiving end of bug
> reports
> of one major project I can tell you that MiniDebugInfo is literally
> the best thing since sliced bread: in systemd upstream the bug reports
> we get from Fedora are *ridiculously* more useful than bug reports
> from any other distro, since the default way how things are reported
> already carry backtraces that are quite useful. It's a complete mess
> with other distros, since we have to ask people to come back with
> proper backtraces with debuginfo installed, and only a subset of
> people is willing to bother with that. Bug reports from ArchLinux,
> Debian, Gentoo and so on are total crap by default compared to
> Fedora.

This is good to hear (the Fedora part). Note that we have moved all the
symbol/debuginfo manipulation done in Fedora into upstream rpm first
and have now spun out that part into a separate project "debugedit"
that rpm (and flatpak) now uses and that we hope other
distros/packaging tools will adopt: https://sourceware.org/debugedit/

  debugedit provides programs and scripts for creating debuginfo and
  source file distributions, collect build-ids and rewrite source paths
  in DWARF data for debugging, tracing and profiling.

So please point other distros/packagers our way and we'll try to work
with them to make these tools as useful to them as they are to Fedora.

Thanks,

Mark
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: F36 Change: Package information on ELF objects (System-Wide Change proposal)

[Mark Wielaard]

Hi Daniel,

On Wed, 2021-10-27 at 10:01 +0100, Daniel P. Berrangé wrote:
> Getting RPM NEVRs directly from the coredumps is something that
> will be incredibly helpful for people dealing with support
> requests after crashes. build-ids have always been very tedious
> to deal with and as you say RPM database may be newer than what
> is in memory.  This benefit alone will make it all worthwhile
> from my POV.
> [...]
> Furthermore as someone dealing with bug reports I don't have access
> to the RPM database. That is on the end user's machine. Often all I
> get is a core dump attached to a bug report, and if I'm lucky they
> manually typed a couple of RPM NEVRs into the bug description. On
> many occassions I've found the NEVRs the user supplied in the
> description to be wrong due to mistakes on the bug reporter's side
> collecting the data.

Note that we have been working on integrating debuginfod in more
distros (it will be enable by default in Fedora 35):
https://debuginfod.fedoraproject.org/

There is also an "global" debuginfod server that aggregates all other
distros debuginfod servers: https://debuginfod.elfutils.org/
Which should delegate to the fedora, opensuse, debian, altlinux, etc.
debuginfod servers to query their build-id databases.

Which means it should be a lot easier to get, given a build-id (in a
core file, or simply in memory even if the original ELF file on disk is
gone) the executable, debuginfo and associated source files.

For the next version of debuginfod we are trying to also give you
information about the original archive (including the nevr in the name)
that the information came from.

This is not to say we don't need the Package information in ELF files.
Having that (especially if it includes the corresponding debuginfod
URL) will make it even easier to get the corresponding debuginfo and
sources for a given ELF binary/build-id.

Cheers,

Mark
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: F36 Change: Package information on ELF objects (System-Wide Change proposal)

[Mark Wielaard]

Hi,

On Mon, 2021-10-25 at 15:09 -0400, Ben Cotton wrote:
> https://fedoraproject.org/wiki/Changes/Package_information_on_ELF_objects

I like this idea. It will be really useful when multiple distros adopt
it.

> === New system: `.note.package` ===
> 
> The new note is created and propagated similarly to
> `.note.gnu.build-id`. The difference is that we inject the
> information
> about package ''nevra'' from the build system.
> 
> The implementation is very simple: `%{build_ldflags}` are extended
> with a command to insert a custom note as a separate section in an
> ELF
> object. See [
> https://github.com/systemd/package-notes/blob/main/hello.spec
> hello.spec] for an example. This is done in the default macros, so
> all
> packages that use the prescribed link flags will be affected.
> 
> The note is a compact json string. This allows the format to be
> trivially extensible (new fields can be added at will), easy to
> process (json is extremely popular and parsers are widely available).
> Using a single field rather than a set of separated notes is more
> space-efficient. With multiple fields the padding and alignment
> requirements cause unnecessary overhead.
> 
> The system was designed with cross-distro collaboration and is
> flexible enough to identify binaries from different packaging formats
> and build systems (rpms, debs, custom binaries).
> 
> See https://systemd.io/COREDUMP_PACKAGE_METADATA/ for detailed
> description of the format.

It isn't immediately clear to me which of the key's will be included.
The format describes 6 standard ones: type, os, osVersion, name,
version, osCpe and debugInfoUrl. But the example given later only shows
type, name, version and osCpe.

To me the debugInfoUrl is the most interesting because combined with
the already existing build-id note it allows automagic fetching of the
debuginfo and sources (and the debuginfod server could tell you the
type, name and version of the archive, so in theory you don't have to
include those fields, but they might still be useful if you cannot
access the debuginfod url).

One ELF technicality I don't see described is the alignment and padding
of the ELF NOTE. Normally on GNU systems ELF NOTEs are aligned on 4
bytes (for both 32/64 bit ELF files). But the recent .note.gnu.property
NOTEs are aligned on 4 bytes for 32bit systems, but 8 bytes on 64bit
systems (this is causing some interesting issues trying to combine
those NOTEs...). What alignment/padding will this new ELF note have?

Thanks,

Mark
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: libcurl-minimal

[Mark Wielaard]

Hi,

On Thu, 2021-10-14 at 16:49 +0200, Kamil Dudka wrote:
> As I understand it, Zbyszek is now proposing to make changes to other 
> packages 
> and/or distribution metadata in order to make (lib)curl-minimal actually used 
> on some Fedora installations by default.

This sounds interesting. For elfutils-debuginfod-client we really only
support http(s):// and file:// so replacing our dependency from full
libcurl to libcurl-minimal seems a good idea (assuming those 3
protocols are in libcurl-minimal). Please let us know what we should
do/test to see if this works as intended.

Thanks,

Mark
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: Triggering fedora-ci tests

[Mark Wielaard]

On Tue, 2021-07-13 at 14:52 +0200, Petr Pisar wrote:
> V Tue, Jul 13, 2021 at 02:27:57PM +0200, Mark Wielaard napsal(a):
> > Or How do you trigger a fedora-ci.koji-build.tier0.functional run for a
> > package?
> > 
> 
> $ bodhi updates trigger-tests UPDATE_ID

Thanks! That worked, the test were run, CI passed and the package was
added to stable.

Is there are place to see which tests are pending? Or do you just have
to run trigger-tests after some time if they haven't yet?

Cheers,

Mark
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Triggering fedora-ci tests

[Mark Wielaard]

Hi,

For valgrind we added some integration tests, and they seem useful (for
example they caught a change in gdb breakpoint location when set after
attaching to the valgrind gdbserver). But sometimes they don't trigger
and then the package is just stuck waiting till the CI tests are ran.

For example the last rawhide build just says that fedora-ci.koji-
build.tier0.functional didn't have any results:
https://bodhi.fedoraproject.org/updates/FEDORA-2021-5d732932ff

Is there a place where I can inspect why a run seems to be stuck?
Or How do you trigger a fedora-ci.koji-build.tier0.functional run for a
package?

Thanks,

Mark
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: valgrinded executable seem to run far more slowly in Rawhide

[Mark Wielaard]

Hi Rich,

On Tue, Jun 08, 2021 at 04:09:33PM +0100, Richard W.M. Jones wrote:
> I haven't filed a bug yet because I'm not sure what component to
> assign this too.  I upgraded to Rawhide this morning and noticed that
> some tests that previously ran OK are now timing out.  It turned out
> that the tests were failing because a valgrinded executable runs much
> more slowly in Rawhide today.
> 
> Here's my test:
> 
> Upgrade to Rawhide and install these additional packages:
> 
>   # dnf install libnbd nbdkit valgrind
> 
> Unpack the certificates from
> http://oirase.annexia.org/tmp/rawhide-valgrind-bug/pki.tar.gz into /tmp
> 
> Run this command as NON-root (it's self-contained and just connects
> the two commands together over localhost port 10809):
> 
>   $ time nbdkit --tls=require --tls-certificates=/tmp/pki null --run 
> 'valgrind nbdinfo nbds://localhost'
> 
> For me this takes over 40 seconds in Rawhide, versus a couple of
> seconds in Fedora 33/34.  Also if you remove "valgrind" it runs
> instantaneously even in Rawhide.  If you run it as root, it runs
> quickly again (WTF!)
> 
> The perf flamegraph seems to indicate it's doing a lot of stuff with
> DWARF 3 symbols.
> 
>   http://oirase.annexia.org/tmp/rawhide-valgrind-bug/libnbd.svg
> 
> Uninstalling all relevant debuginfo packages didn't seem to help.

Thanks for using valgrind and sorry we made it even slower.  I am
working on it. As you already figured out, it has to do with
debuginfo. Basically valgrind should only really need to read the line
tables, but it does a lot more. This is the fedora bug:
https://bugzilla.redhat.com/show_bug.cgi?id=1662656

This has become more urgent now that debuginfod has become the default
in rawhide (valgrind supports debuginfod since 3.17.0):
https://fedoraproject.org/wiki/Changes/DebuginfodByDefault

That also explains your observations, it depends on whether or not
DEBUGINFOD_URLS is set in the environment (and whether
elfutils-debuginfod-client is installed).

I'll work on making valgrind fast again, as far as valgrind is fast to
begin with :)

Cheers,

Mark
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: Storing package metadata in ELF objects

[Mark Wielaard]

Hi Guillem,

On Wed, 2021-05-19 at 02:19 +0200, Guillem Jover wrote:
> So this is where I guess I'm missing something. To be able to make
> sense of the coredumps there are two things that might end up being
> relevant, backtraces and source code. systemd-coredump might already
> emit a backtrace, and depending on the information provided it might
> be more or less useful. If one needs the actual debug symbols there's
> already some external querying/fetching required, and if distribution
> specific source code is required because many distributions patch
> upstream source, then even more querying/fetching will be required.
> 
> Which is why I'm not seeing why this standalone and isolated metadata
> would be of much help by itself. As in, the way I see it, either the
> information from systemd (w/o the extra metadata) is sufficient to
> track down bugs, or that querying/fetching would be needed anyway, at
> which point the metadata can be inferred too then?

Because without that metadata you cannot easily figure out where/how to
get the files needed to properly track down the bugs. But using that
metadata you can figure out where the debuginfod server is that can
provide all that information for the binaries/core files (or a distro
specific method if the distributor doesn't have a debuginfod server
yet).

> Oh, I was thinking about those mixed environments, full chroots or
> stripped down containers, from different vendors, but affecting Debian
> installations. What I'm also probably missing here is how does the
> metadata help for a third-party that is not expected to track/keep/upload
> debug symbols nor source packages into some repository, because otherwise
> I'm not seeing how they'd make use of the cores (if they are insufficient
> by themselves) to debug issues? I guess my question back would be what
> would they do with the metadata if they do not have the debug symbols
> nor the sources readily available? Also assuming of course they exercise
> good practices such as never reusing the same package-version-arch tuple
> for different builds, and similar. :)

Different builds will have different build-ids, so they can be kept
apart. But even if without the distributor having added debuginfod
meta-data and providing a server to fetch the extra symbols, debuginfo,
sources, the extra meta-data is useful to administrators. Just seeing
that crashes are associated with specific distro/version/packages helps
narrow down instabilities.

Cheers,

Mark
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: Storing package metadata in ELF objects

[Mark Wielaard]

Hi Luca,

On Tue, 2021-05-04 at 14:43 +0100, Luca Boccassi wrote:
> On Fri, 2021-04-30 at 19:57 +0200, Mark Wielaard wrote:
> > Is there a list of default keys (and their canonical spelling, upper-
> > lower-Camel_Case, etc.)? If there is, could we have a "debuginfod" key
> > with as value an URL pointing to the debuginfod server URL where the
> > embedded build-id executable, debuginfo and sources can be found?
> > https://sourceware.org/elfutils/Debuginfod.html
> 
> The "Implementation" section of the spec lists the "main" fields:
> 
> https://systemd.io/COREDUMP_PACKAGE_METADATA/
> 
> (source for that is 
> https://github.com/systemd/systemd/blob/main/docs/COREDUMP_PACKAGE_METADATA.md
>  )
> 
> Would you like to send a PR to update it and add that field?

Sorry, I don't have a github account. But attached is a patch for to
document it and one for the package-notes generator to add an --
debuginfod argument (maybe the distro should set a default value for
that?) Hopefully those patches could be applied somehow.

Thanks,

Mark
From 176dde09fcfeff9bff97fbf65d47a641dbd2fa3a Mon Sep 17 00:00:00 2001
From: Mark Wielaard 
Date: Thu, 6 May 2021 03:05:02 +0200
Subject: [PATCH] docs/COREDUMP_PACKAGE_METADATA.md: Add debuginfod key

Signed-off-by: Mark Wielaard 
---
 docs/COREDUMP_PACKAGE_METADATA.md | 5 +
 1 file changed, 5 insertions(+)

diff --git a/docs/COREDUMP_PACKAGE_METADATA.md b/docs/COREDUMP_PACKAGE_METADATA.md
index ab7c4596de..b7be9ca797 100644
--- a/docs/COREDUMP_PACKAGE_METADATA.md
+++ b/docs/COREDUMP_PACKAGE_METADATA.md
@@ -26,6 +26,10 @@ Thus it becomes desirable to add additional metadata to a binary at build time,
 `systemd-coredump` and other services analyzing core files are able to extract said
 metadata simply from the core file itself, without external dependencies.
 
+And in case external dependencies and network are available the core file
+should also include the debuginfod server URL that can provide the original
+executable, debuginfo and sources to facilitate debugging.
+
 ## Implementation
 
 This document will attempt to define a common metadata format specification, so that
@@ -58,6 +62,7 @@ Value: a JSON string with the structure described below
  "name":"coreutils",
  "version": "4711.0815.fc13.arm32",
  "osCpe":   # A CPE name for the operating system, `CPE_NAME` from os-release is a good default
+ "debuginfod": "https://debuginfod.fedoraproject.org";
 }
 ```
 
-- 
2.18.4

From 6264605699cbac02ed3060cc1ff6497b32e90382 Mon Sep 17 00:00:00 2001
From: Mark Wielaard 
Date: Thu, 6 May 2021 03:13:57 +0200
Subject: [PATCH] generate-package-notes.py: Add --debuginfod argument

Signed-off-by: Mark Wielaard 
---
 generate-package-notes.py | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/generate-package-notes.py b/generate-package-notes.py
index c0b5d98..92126f7 100755
--- a/generate-package-notes.py
+++ b/generate-package-notes.py
@@ -71,6 +71,7 @@ def parse_args():
 p.add_argument('--package-version')
 p.add_argument('--cpe')
 p.add_argument('--rpm', metavar='NEVRA')
+p.add_argument('--debuginfod')
 
 opts = p.parse_args()
 
@@ -143,6 +144,8 @@ def generate_section(opts):
 else:
 data['os'] = read_os_release('ID')
 data['osVersion'] = read_os_release('VERSION_ID')
+if opts.debuginfod:
+data['debuginfod'] = opts.debuginfod
 
 json = json_serialize(data)
 
-- 
2.18.4

___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: Storing package metadata in ELF objects

[Mark Wielaard]

Hi,

On Sat, 2021-04-10 at 18:44 +, Zbigniew Jędrzejewski-Szmek wrote:
> [I'm forwarding the mail from Luca who is not subscribed to fedora-
> devel]
> On Sat, Apr 10, 2021 at 01:38:31PM +0100, Luca Boccassi wrote:
> Cross-posting to the mailing lists of a few relevant projects.

Note that in this version of the email the [N] references in your email
don't seem to point anywhere. I found an older variant of the same
email which contained:

[0] https://github.com/systemd/systemd/issues/18433
[1] https://systemd.io/COREDUMP_PACKAGE_METADATA/
[2] https://fedoraproject.org/wiki/Changes/Package_information_on_ELF_objects
[3] https://github.com/systemd/package-notes

> After an initial discussion [0], recently we have been working on a new
> specification [0] to encode rich package-level metadata inside ELF
> objects, so that it can be included automatically in generated coredump
> files. The prototype to parse this in systemd-coredump and store the
> information in systemd-journal is ready for testing and merged
> upstream. We are now seeking further comments/opinions/suggestions, as
> we have a few months before the next release and thus there's plenty of
> time to make incompatible changes to the format and implementation, if
> required.
> 
> A proposal to use this by default for all packages built in Fedora 35
> has been submitted [1].
> 
> The Fedora Wiki and the systemd.io document have more details, but to
> make a long story short, a new .notes.package section with a JSON
> payload will be included in ELF objects, encoding various package-
> build-time information like distro name&version, package
> name&version,
> etc.

Is there a list of default keys (and their canonical spelling, upper-
lower-Camel_Case, etc.)? If there is, could we have a "debuginfod" key
with as value an URL pointing to the debuginfod server URL where the
embedded build-id executable, debuginfo and sources can be found?
https://sourceware.org/elfutils/Debuginfod.html

> To summarize from the discussion, the main reasons why we believe this
> is useful are as following:
> 
> 1) minimal containers: the rpm database is not installed in the
> containers. The information about build-ids needs to be stored
> externally, so package name information is not available immediately,
> but only after offline processing. The new note doesn't depend on the
> rpm db in any way.
> 
> 2) handling of a core from a container, where the container and host
> have different distros
> 
> 3) self-built and external packages: unless a lot of care is taken to
> keep access to the debuginfo packages, this information may be lost.
> The new note is available even if the repository metadata gets lost.
> Users can easily provide equivalent information in a format that makes
> sense in their own environment. It should work even when rpms and debs
> and other formats are mixed, e.g. during container image creation.
> 
> Other than in Fedora, we are already making the required code changes
> at Microsoft to use the same format&specification for internally-built
> binaries, and for tools that parse core files and logs.
> 
> Tools for RPM and DEB (debhelper) integration are also available [3].
> 
> > -- 
> > Kind regards,
> > Luca Boccassi
> 
> 
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: Rewriting RPM macros using Lua

[Mark Wielaard]

Hi Florian,

On Thu, 2021-04-29 at 16:33 +0200, Florian Weimer wrote:
> I need to wrap find-debuginfo.sh because there is a single shared
> object
> in my package that must not be stripped.  find-debuginfo.sh does not
> support that, and probably never will, at least not in its current
> shell-based incarnation.

I happened to have a patch that was never submitted upstream that might
(or might not) help:
https://code.wildebeest.org/git/user/mjw/rpm/commit/?h=find-debuginfo-exclude

Would that be useful? find-debuginfo.sh is currently maintained in the
debugedit project https://sourceware.org/debugedit which will hopefully
soon enter Fedora so rpm can use it:
https://bugzilla.redhat.com/show_bug.cgi?id=1953633

Cheers,

Mark
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: F35 Change: CompilerPolicy Change (System-Wide Change proposal)

[Mark Wielaard]

Hi,

On Fri, 2021-04-23 at 20:01 +0200, Jakub Jelinek wrote:
> On Fri, Apr 23, 2021 at 11:18:59AM -0400, Ben Cotton wrote:
> > The Red Hat tools team believes that LLVM/Clang and GCC should be
> > considered equals from
> > a Fedora policy standpoint.  Selection of one toolchain over the other 
> > should be
> > driven by the packager's preferences not by Fedora specific policy.
> > The only policy restriction should be that the compiler must exist in
> > Fedora.
> 
> I'll be probably repeating myself, but the two compilers are known to
> be
> ABI incompatible in very important ways, none of the
> https://bugs.llvm.org/buglist.cgi?quicksearch=42439%2C19909%2C44228%2C12207
> bugs have been touched since last time this was discussed and I'm not
> aware
> of any ABI compatibility testing between the two compilers.
> So if some library packages switch compilers and programs using those
> libraries don't or vice versa, this proposal has quite high risks of
> introducing hard to debug debugs.

So should the policy at least say that this is only allowed for
packages that don't provide libraries or other code that might be
linked against?

> I agree that for open source software and generally for users it is useful
> if there are multiple competing implementations, in this case for the
> toolchain, and the competition has been IMHO quite fruitful for both GCC and
> LLVM over the past years.  It is also very useful if open source packages
> are made portable or kept portable, so that one can use multiple compilers
> to compile them, one can benefit from different diagnostics, instrumentation
> etc.  But I'm not sure this proposal is a step in the right direction
> though, e.g. in the Firefox case that is being used as an example for this
> proposal that leads in a completely different direction, a package that has
> been formerly portable and supported multiple compilers will turn into a
> single compiler only application and will lose its portability.
> Google, Apple, BSDs don't really care about toolchain choice and they push
> a single toolchain only.

I cannot say I like this proposed policy very much. For the packages I
work on, elfutils, valgrind, debugedit, etc. we have enough trouble
keeping up with gcc versions, new flags, optimizations, etc. I don't
think we will have time to debug issues because packages are now also
build with another compiler. Leaving the system as a whole in a worse
state because of it.

I understand there are upstreams which are more aligned with other
toolchains, distros or a distribution model where a specific compiler
version is mandated. But even if a packages switches compilers it will
be the version as packaged in Fedora, and so some porting, testing,
etc. (because the version in Fedora will be different anyway) will be
necessary. It seem better and more consistent for packages to use the
default system compiler (gcc) unless it really cannot be build with it.

Cheers,

Mark
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: How to handle pull request with git?

[Mark Wielaard]

On Wed, Apr 14, 2021 at 04:04:21PM +0200, Robert-André Mauchin wrote:
> On 4/14/21 3:28 PM, Mark Wielaard wrote:
> > I got a "pull request" for one of my packages and wanted to make some
> > changes to discuss with the submitter and see if we could merge it
> > back with those changes to the rawhide branch. But somehow I did
> > something wrong and I am not sure what or how to fix it.
> > 
> > So I saw this webpage with the suggested change:
> > https://src.fedoraproject.org/rpms/valgrind/pull-request/4
> > 
> > I added the following line to my .git/config at the end of the [remote
> > "origin"] section to be able to pull it:
> > 
> > fetch = +refs/pull/*/head:refs/remotes/origin/pr/*
> > 
> > Then git pulled and checkout pr/4, made the changes, committed them
> > and pushed them back, hoping that would update the pr.
> > 
> > But it didn't, apparently I created a new origin/pr/4 branch, which I
> > am now unable to delete because:
> > 
> > $ git push origin --delete pr/4
> > remote: Branch deletion is not allowed
> > remote: Denied push for ref 'refs/heads/pr/4' for user 'mjw'
> > remote: All changes have been rejected
> > To ssh://pkgs.fedoraproject.org/rpms/valgrind
> >   ! [remote rejected] pr/4 (pre-receive hook declined)
> > error: failed to push some refs to 
> > 'ssh://pkgs.fedoraproject.org/rpms/valgrind'
> > 
> > What did I do wrong and how do I fix this?
> > 
> 
> 1. You clone the forked repository
> 2. You checkout the branch where the modification has been made
> 3. You edit the files you want to change
> 4. You commit (or amend) the new changes
> 5. You push (or force push) the commit
> 6. Your commit will appear in the Pull Request ar "Rebased blah blah"
> 7. Merge your changes

Thanks. So I believed I was doing the above, but apparently not.
Would you mind explaining steps 1, 2 and 5 a bit more.

So what I did to clone the prs was to add that extra fetch line in my
.git/config. Which seemed to allow me to pull and work on the branch
remote pull/4 as pr/4 locally. Then I though I pushed it back, but
apparently that created a new branch in the origin called pr/4 instead
of getting my changes back into the remote pull/4.

Apparently the fetch line doesn't work like I expected.

Thanks,

Mark
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: How to handle pull request with git?

[Mark Wielaard]

On Wed, Apr 14, 2021 at 02:38:40PM +0100, Tom Hughes via devel wrote:
> On 14/04/2021 14:28, Mark Wielaard wrote:
> 
> > I added the following line to my .git/config at the end of the [remote
> > "origin"] section to be able to pull it:
> > 
> > fetch = +refs/pull/*/head:refs/remotes/origin/pr/*
> > 
> > Then git pulled and checkout pr/4, made the changes, committed them
> > and pushed them back, hoping that would update the pr.
> 
> Is there anywhere that works?
> 
> I don't think even github allows you to push back to the
> generated head for the PR like that. If the person who opened
> the PR allowed it then you can push to their original branch
> to update the PR - no idea if pagure supports that.

I have no experience with github, but it looked like a normal remote
branch to me, so I expected to be able to push to it. How else are you
supposed to coordinate on a PR/patch like that?

> Possibly src.fpo should reject pushes to the PR heads to
> avoid this kind of accident though.

Yes, I did certainly mess up here, don't completely understand how,
but it would have been nice if the system would have stopped me.

> > But it didn't, apparently I created a new origin/pr/4 branch, which I
> > am now unable to delete because:
> > 
> > $ git push origin --delete pr/4
> > remote: Branch deletion is not allowed
> > remote: Denied push for ref 'refs/heads/pr/4' for user 'mjw'
> > remote: All changes have been rejected
> > To ssh://pkgs.fedoraproject.org/rpms/valgrind
> >   ! [remote rejected] pr/4 (pre-receive hook declined)
> > error: failed to push some refs to 
> > 'ssh://pkgs.fedoraproject.org/rpms/valgrind'
> > 
> > What did I do wrong and how do I fix this?
> 
> You pushed a branch into source git which is bad as hey
> can't be deleted.
> 
> I believe in principle you can ask an admin to delete it
> so long as no builds have been done from it.

That would be nice. How do I contact an admin?

Thanks,

Mark
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

How to handle pull request with git?

[Mark Wielaard]

Hi,

I got a "pull request" for one of my packages and wanted to make some
changes to discuss with the submitter and see if we could merge it
back with those changes to the rawhide branch. But somehow I did
something wrong and I am not sure what or how to fix it.

So I saw this webpage with the suggested change:
https://src.fedoraproject.org/rpms/valgrind/pull-request/4

I added the following line to my .git/config at the end of the [remote
"origin"] section to be able to pull it:

fetch = +refs/pull/*/head:refs/remotes/origin/pr/*

Then git pulled and checkout pr/4, made the changes, committed them
and pushed them back, hoping that would update the pr.

But it didn't, apparently I created a new origin/pr/4 branch, which I
am now unable to delete because:

$ git push origin --delete pr/4
remote: Branch deletion is not allowed
remote: Denied push for ref 'refs/heads/pr/4' for user 'mjw'
remote: All changes have been rejected
To ssh://pkgs.fedoraproject.org/rpms/valgrind
 ! [remote rejected] pr/4 (pre-receive hook declined)
error: failed to push some refs to 'ssh://pkgs.fedoraproject.org/rpms/valgrind'

What did I do wrong and how do I fix this?

Thanks,

Mark
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: New dwz silently segfaulting

[Mark Wielaard]

Hi Mamoru,

On Fri, 2021-01-22 at 22:27 +0900, Mamoru TASAKA wrote:
> It looks like new dwz-0.13-6.fc34 is silently segfaulting when
> rebuilding some srpm,
> but it does not prevent whole rpmbuild process from stopping.
> 
> Some random example:
> * vorbis-tools-1.4.2-1.fc34
>https://koji.fedoraproject.org/koji/buildinfo?buildID=1672461
>x86_64 build log:

I just noticed that too and filed:
https://bugzilla.redhat.com/show_bug.cgi?id=1919243

> Since rpmbuild process seems to finish anyway, this can be ignored??
> (I guess not)

It doesn't break the build, but it does make the debuginfo packages
bigger than necessary. So please do report such issues in bugzilla.

I just came up with a patch and build dwz-0.13-7.fc34 which should fix
this problem.
https://bodhi.fedoraproject.org/updates/FEDORA-2021-7198012a4d

Cheers,

Mark
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Mass spec file change: Adding BuildRequires: make

[Mark Wielaard]

Hi Tom,

On Mon, 2020-11-30 at 14:06 -0800, Tom Stellard wrote:
> As part of the f34 change request[1] for removing make from the 
> buildroot, I will be doing a mass update of packages[2] to add 
> BuildRequires: make where it is needed.

As part of a previous change request [1] various packages were
transformed to use make macros (%make_build, %make_install) do these
macros automagically pull in the make build requires (or can they be
made to do so)?

Thanks,

Mark

[1] https://fedoraproject.org/wiki/Changes/UseMakeBuildInstallMacro
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Conflicting build-ids in nekovm and haxe

[Mark Wielaard]

Hi Andy,

On Fri, 2020-11-13 at 02:16 +, Andy Li wrote:
> Re. https://bugzilla.redhat.com/show_bug.cgi?id=1896901
> 
> Since haxe-4.1.3-4 and nekovm-2.3.0-4, both nekovm and haxe packages
> contains "/usr/lib/.build-
> id/b0/aed4ddf2d45372bcc79d5e95d2834f5045c09c".
> The nekovm one is a symlink to "/usr/bin/neko". The haxe one to
> "/usr/bin/haxelib".
> Both the neko and haxelib binaries are built with libneko, with a
> nearly identical main.c with the only difference of the present of
> neko bytecode embedded as a byte array (neko: the byte array is null;
> haxelib: the byte array is the haxelib neko bytecode).
> 
> I'm not sure how to resolve it.
> Please advice.

In general this (should) only happen if the binaries are really
identical (e.g. when one of the packages build requires the other, but
instead of linking/rebuilding some sources it simply copies a binary
directly).

The build-id is (also) derived from the full nvra. So really cannot be
identical even if the sources are. But this relies on the binaries
being build with debuginfo. So maybe some code isn't build with -g?

Cheers,

Mark
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: F34 Change proposal: Debug Info Standardization (from DWZ to -fdebug-types-section) (System-Wide Change proposal)

[Mark Wielaard]

On Tue, Oct 06, 2020 at 04:46:24PM -0600, Jeff Law wrote:
> 
> On 9/30/20 3:50 AM, Jan Kratochvil wrote:
> > On Wed, 30 Sep 2020 01:31:29 +0200, Jeff Law wrote:
> >> But the GCC community
> >> doesn't really test that option and it's known to be broken with LTO.
> > I haven't seen any GCC PR for -fdebug-types-section being broken with LTO.
> 
>  I'm not aware of one either.  But as Jakub has previously pointed out
> debug-types-section is disabled when LTO is enabled.  I don't know the
> details of why that is done.
>
> > Could you suggest what is wrong on -fdebug-types-section?
> 
> Your best bet is to discuss with Jakub and perhaps Jason.   They're far
> more familiar with the debuginfo generation than I am.

It didn't used to work with LTO, but some patches have been backported
so it at least is passed through now without crashing. There are
however still various bugs in the implementation:

Excess debug info -fdebug-types-section 
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=78320

-fdebug-types-section drops DW_AT_object_pointer
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=94875

gcc drops top-level dies with -fdebug-types-section
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=90232

Fission + type units + compression are suboptimal
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=78321

Cheers,

Mark
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

ELF section/file compression

[Mark Wielaard]

Hi,

Changing subject because this has nothing to do with that Change
Proposal anymore.

On Tue, Oct 06, 2020 at 01:49:13PM -0600, Jeff Law wrote:
> However, I think it's perfectly valid to discuss zstd if folks wanted to
> change the compression scheme for debug sections.  In fact, I'd claim
> sticking with zlib, gzip,  bzip2, xz, 7z, etc is unwise.  The world has
> moved and zstd seems like the place we should be.  In fact, we use it
> for various things within GCC already.

Personally I must admit that I am not really a fan of using ELF
section/file compression. It makes it impossible to simply mmap the
data in or to quickly read just a tiny bit because you first have to
decompress (and allocate new memory) for it. IMHO .debug files are no
different from other ELF files for which we would also not do this. We
can just use rpm package compression to reduce the distro distribution
size, but we should not (re)compress the install/on-disk files. That
will just mean programs will create an extra cache of uncompressed
files they need to consult frequently.

But if you are going to do it, then it does make sense to use the most
efficient algorithm there is. If you are going to experiment with this
there are two ways to go about it.

First you can use full file compression. That is actually already
(almost transparently) supported for tools based on elfutils libdw
when using the dwfl functions. For example eu-readelf (and some other
eu- tools) can be run on any compressed file directly (the version in
rawhide also supports zstd because the vmlinuz file now uses
that). You would have to convince other DWARF consumers to do the
same. And decide for those that use .gnu_debuglink instead of build-id
lookups (or when build-id lookup fails) whether the filenames should
include the compression extension like .zst or if a consumer is
responsible for trying all (?) known compression extensions when
resolving the .debug file.

Secondly you can use ELF section compression. The ELF spec leaves room
for adding new compression algorithms. The Chdr struct(s) contain a
ch_type which describes the algorithm. Currently only one is
specified, but there is a lot of room for expansion:

/* Legal values for ch_type (compression algorithm).  */
#define ELFCOMPRESS_ZLIB1  /* ZLIB/DEFLATE algorithm.  */
#define ELFCOMPRESS_LOOS0x6000 /* Start of OS-specific.  */
#define ELFCOMPRESS_HIOS0x6fff /* End of OS-specific.  */
#define ELFCOMPRESS_LOPROC  0x7000 /* Start of processor-specific.  */
#define ELFCOMPRESS_HIPROC  0x7fff /* End of processor-specific.  */

So you could propose something on gnu-g...@sourceware.org for a GNU
extension or at generic-...@googlegroups.com for a generic ELF one.
And then get the ELF processing tools to adopt the new compression
type.

Cheers,

Mark
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Switching to DWARF5 default for GCC11 (and the default Fedora 34 toolchain)

[Mark Wielaard]

Hi Neal,

On Tue, 2020-09-29 at 19:59 -0400, Neal Gompa wrote:
> For the record, Mark has started implementing DWARF-5 support in dwz:
> https://sourceware.org/git/?p=dwz.git;a=log
> 
> I think I would rather like to see a Change proposal to switch to
> DWARF-5 for Fedora 34, especially since it looks like dwz will be
> ready for it.

That is indeed my goal, but I wasn't planning on filing a specific
Change Proposal for it.

First because as you observed in the past we did some of these
debuginfo things Fedora first and then it took years (!) for some of
the default settings we had changed and helper scripts to make it
upstream. So I am concentrating on getting everything ready upstream
first before making and proposing any changes for Fedora. 

Secondly I am hoping that because of the first point the GCC11 for
Fedora 34 Change Proposal will simply say "-gdwarf-5 is now the
default".

Lastly, and sadly, I find the whole Fedora change proposal debates
extremely hostile. They often seem to quickly result in people
attacking you because you made a choice to spend time to work on A and
not their favorite feature B, and if they cannot have feature B then
you should also not spend any more time on A.

So I am happy to describe the work I am doing to try to get DWARF5 the
default for GCC11 and by extension for the Fedora 34 default toolchain,
but I will mainly do that work upstream and then see whether it is all
ready on time to enable it for Fedora 34. But I am not interested in a
heated debate on how I should prioritize my time and energy.

= Why DWARF5 for GCC?

- A couple of new tags and attributes make it easier/more accurate to
  describe some of the newer language features (although most were
  already covered by various GNU extensions)

- A lot of GNU extensions to DWARF4 have been standardized in DWARF5.
  By adopting the standardized variant alternative toolchains will
  hopefully find it easier to support these features.

- The representation of various data structures in DWARF5 is much more
  efficient causing a 25% on-disk size reduction (before any other
  compression method) for the .debug sections:
  https://gcc.gnu.org/pipermail/gcc-patches/2020-September/553527.html

= DWARF5 for the (extended) GNU toolchain

- binutils (gas) is responsible for turning part of the assembly
  produced by the compiler into a line table (.debug_line) and the
  linker sometimes reads parts of the DWARF (for example
  when producing warnings about where a symbol was defined). The just
  released binutils 2.35.1 should have all fixes necessary to support
  DWARF5.

- gcc needs to use the new gas features. Jakub has a patch (not
  committed yet):
  https://gcc.gnu.org/pipermail/gcc-patches/2020-September/553992.html

- gdb seems ready except for one corner case with C++ static member
  variables in classes. This is because in DWARF5 these are represented
  not as variables, which might be optimized away when not used. In
  this case gcc probably shouldn't optimize out the unused variables
  (or gdb should not depend on being able to show optimized out static
  member variables). Ongoing debate how to resolve this:
  https://sourceware.org/bugzilla/show_bug.cgi?id=26525
  https://gcc.gnu.org/pipermail/gcc-patches/2020-September/553102.html

- The just released elfutils 0.181 seems to have all needed support,
  which should cover systemtap, dwarves, perf, systemd, libabigail.
  More testing ongoing.

- For valgrind I initially wanted to switch the DWARF reader to an
  external helper program based on elfutils libdw. But to get a
  solution faster I will tweak the internal reader to deal with just
  the minimal DWARF5 as generated by gcc for now. I haven't started on
  this yet.

= DWARF5 for the (Fedora) packaging tools

- rpm debugedit has patches to support DWARF5 but we have to make sure
  they have testcases to work with gcc:
  http://lists.rpm.org/pipermail/rpm-maint/2020-August/014833.html

- dwz is seeing active work towards supporting DWARF5:
  https://sourceware.org/pipermail/dwz/2020q3/000668.html
  I am hoping that by end of next week we have generic support.
  That might not do optimal compression yet and will probably need lots
  of testing (and bug fixing).

= What I am NOT working on

- We'll keep using .gdb_index for now, moving to .debug_names only
  when that is ready in gdb:
  https://sourceware.org/pipermail/gdb/2020-September/048879.html

- Optional DWARF5 features like debug-types, forms, operations or index
  tables only used for split-dwarf by GCC (e.g. DW_FORM_strx,
  DW_FORM_addrx, DW_FORM_loclistx, DW_FORM_rnglistx, DW_OP_addrx,
  DW_OP_constx).

- Any other tool, project not mentioned above or other
  native toolchains like golang, rust, clang/llvm or ocaml.
  I expect those to simply keep producing DWARF4.

That doesn't mean I won't help with any of the above if others propose
to do that work for the various pieces of the toolchain and packaging,
but I currently don't have time for it and I d

Re: F34 Change proposal: Debug Info Standardization (from DWZ to -fdebug-types-section) (System-Wide Change proposal)

[Mark Wielaard]

Hi Jan,

On Mon, Sep 28, 2020 at 04:50:59PM +0200, Jan Kratochvil wrote:
> To make DWZ better consumable it needs to have the partial units separately
> parseable. That way they can be shared at IR level and not just at DWARF
> level
> That means:
>  * DW_TAG_partial_unit should have DW_AT_language.
>  * DW_TAG_partial_unit must contain only types (struct/class).
>Currently they contain for example also static constant variables but when
>you parse such independent DW_TAG_partial_unit into which dictionary you
>will register such variable? That makes no sense.

You might want to look at the experiments to do something like that
from Tom de Vries:
https://sourceware.org/pipermail/dwz/2020q1/000579.html
https://sourceware.org/pipermail/dwz/2020q1/000568.html

> It is all about engineering effort. I agree if the support of DWZ
> was trivial (or there were unlimited engineering resources) then DWZ
> is really better than -fdebug-types-section (except it would need a
> DWZ tool with less bugs and better coding practices). But reality
> shows the DWZ support is not trivial engineering resources for
> Fedora are very limited and so we have to decide whether the serious
> effort to support DWZ is better spent on DWZ or on making the
> debuggers better/really usable.

Hacking on dwz and supporting partial units and DWARF supplemential
files in debugger like tools isn't trivial. But it is IMHO also not
such a big effort that we have to drop everything else. Lets see if we
can work together on this.

Cheers,

Mark
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: F34 Change proposal: Debug Info Standardization (from DWZ to -fdebug-types-section) (System-Wide Change proposal)

[Mark Wielaard]

On Mon, Sep 28, 2020 at 05:46:08PM +0200, Jan Kratochvil wrote:
> On Mon, 28 Sep 2020 17:35:26 +0200, Jakub Jelinek wrote:
> > So, was this compiled by GCC or clang?
> 
> Fedora Koji package:
>   lldb-debuginfo-11.0.0-0.2.rc3.fc34.x86_64
> 
> GNU GIMPLE 10.2.1 20200916 (Red Hat 10.2.1-4) -m64 -mtune=generic 
> -march=x86-64 -g -g -g -O2 -O2 -O2 -O2 -fno-openmp -fno-openacc 
> -fcf-protection=none -ffat-lto-objects -fexceptions -fstack-protector-strong 
> -fasynchronous-unwind-tables -fstack-clash-protection -fPIC 
> -ffunction-sections -fdata-sections -fltrans -fplugin=annobin

Note that you are using -ffunction-sections together with -flto.
With -flto you don't need -ffunction-sections.

-ffunction sections might cause functions to be dropped by the linker
without updating the DWARF DIEs, causing things like a zero
DW_AT_low_pc.

Just using -flto should not cause such issues.

Cheers,

Mark
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: F34 Change proposal: Debug Info Standardization (from DWZ to -fdebug-types-section) (System-Wide Change proposal)

[Mark Wielaard]

Hi Jan,

On Mon, Sep 28, 2020 at 05:28:57PM +0200, Jan Kratochvil wrote:
> On Mon, 28 Sep 2020 12:31:59 +0200, Mark Wielaard wrote:
> > I do find your statistics per package useful because they show dwz is
> > in general effective by producing at least 20% (more) on-disk size
> > reduction,
> 
> I am ignoring the on-disk size, I always measure just *-debuginfo.rpm size.
> 
> If anyone is concerned about on-disk size then Fedora should have already
> enabled zlib section compression which would reduce the on-disk *.debug size
> by 52.84% for the whole distro. Or even implement zstd section compression
> probably with even bigger size decrease (and even lower performance hit
> already low enough).

I was just discussing that recently with the Hotspot Perf GUI
maintainer. And we concluded that if .debug files would be compressed
then we would need an uncompressed cache somewhere. The issue with
having the on-disk debuginfo files compressed is that for
debugger/tracing/profiling tools it incurs an significant
decompression time delay and extra memory usage. Especially for a
profiling tool that only needs to quickly get a little information it
is much more convenient to be able to simply mmap the .debug file,
check the aranges and directly jump to the correct DIE offset. See
e.g. https://github.com/KDAB/hotspot/issues/115

> So why do you mention the on-disk size now?

Because I believe it is the most important benchmark. The on-disk size
is not just the installed file size, but it is also the in memory size
of the data structures that need to be stored and parsed. So a 20%
smaller on-disk size also (roughly) means 20% less DIEs and abbrevs
that need to be parsed or held into memory.

Cheers,

Mark
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: F34 Change proposal: Debug Info Standardization (from DWZ to -fdebug-types-section) (System-Wide Change proposal)

[Mark Wielaard]

Hi,

On Fri, 2020-09-25 at 17:18 +0200, Florian Weimer wrote:
> * Robbie Harwood:
> > Jan Kratochvil  writes:
> > > So why is Google using it for everything?
> > 
> > If I could eliminate one bad thought pattern in software design it would
> > probably be this one.
> > 
> > In brief: you are not Google, nor are you Facebook, nor Amazon.  Your
> > problems are not their problems.  Your use case is not their use case.
> > Plenty of things work great for them that will work terribly for you.
> > 
> > So saying "Google does it" (or similar) is *not* a good argument.
> 
> Agreed, especially since we know that e.g. Google's use of C++ does not
> align well with how many other programmers use the language.

The Google engineers responsible for their internal build system don't
make it a secret. They use debug-types combined with [out of .o file]
split-dwarf[=split]. But they also admit that it is for a specialized
use case that might only makes sense if you have a central build system
that farms out different parts of the build/compile/link steps to
different machines.

   (for Google) - a distributed build system that is trying to avoid
   moving more bytes than it must to one machine to run the link step.
   So not having to ship all the DWARF bytes to one machine for
   interactive debugging (pulling down from a distributed file system
   only the needed .dwo files during debugging - not all of them) - or
   at least being able to ship all the .dwo files to one machine to
   make a .dwp, and ship all the .o files to another machine for the
   link.

It is certainly a clever setup and makes sense if your build bottleneck
is sending files around between different machines. But I don't think
this is the generic Fedora packager or developer use case.

Cheers,

Mark
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: F34 Change proposal: Debug Info Standardization (from DWZ to -fdebug-types-section) (System-Wide Change proposal)

[Mark Wielaard]

Hi Jan,

On Fri, 2020-09-25 at 11:43 +0200, Jan Kratochvil wrote:
> On Fri, 25 Sep 2020 01:35:43 +0200, Mark Wielaard wrote:
> > Replying since I am mentioned by name in this proposal and it seems to
> > argue for removing a feature I am currently working on to make sure it
> > works correctly with GCC11 if it switches to producing DWARF5 by
> > default. The proposal seems based on some misunderstandings.
> 
> The problem is you are not working on DWARF-5 features produced by LLVM so
> even your planned DWZ is still not usable for LLVM-built binaries.
> [...] It is also unfair to restrict LLVM due to a deficiency of DWZ.
> [...] Fedora has been waiting for DWARF-5 for 3.5 years due to DWZ.
> [...] From my point of view the DWZ technology makes no sense and so
> I find a better fix to drop DWZ. [...] GCC also does not support most
> of DWARF-5 after it is standardized for 3.5 years (only these days
> you started implementing better DWARF-5 support). [...] It is easy
> for primitive tools like GDB [...] I understand DWZ is not a problem
> for trivial utilities, that is not the case of LLDB. [...]
> I have switched to clang in 2013 and I have never looked back.

You aren't really making sure to win people over if you tell others 
they aren't working fast enough, they aren't prioritizing your bugs,
disparaging others work because they do have implemented support for
newer DWARF constructs, just not in a way you would have done it, and
not being willing to help out with fixing things because you aren't
personally interested in the Fedora default GNU toolchain.

DWZ works totally fine with llvm produced binaries, there is even a
whole distro build with clang that uses DWZ (it just doesn't use DWARF5
by default). Your comments about GCC are also wrong. GCC does support
and produce almost any construct DWARF-5 specifies (after all, they
used to be GNU extensions before), it just doesn't use certain forms or
index tables unless you are producing split-dwarf (which like debug-
types also isn't enabled by default).

If you want to make -fdebug-types-sections the default you really
should work with the upstream GCC developers to figure out why they
don't want that. Trying to override upstream defaults in Fedora without
understanding why upstream decided on the current defaults isn't a good
idea IMHO.

I totally get that it is frustrating if you worked for a long time on a
new feature to support some DWARF constructs for lldb and your aren't
able to get the patches in shape to be accepted upstream. But I am
happy people now know about your patches and seem to find them useful.

You say it is difficult to support DWARF partial units as generated by
dwz in lldb, but dwz doesn't really do anything non-standard (and GCC
with LTO also generates partial units). If your complaint is that
partial unit DIEs are missing some for your use case essential
attributes, then we can look at adding them (note that Tom de Vries has
experimented with --devel-gen-cu and --devel-uni-lang in dwz git, which
might provide you with something you can use, see the dwz commit
messages for some more background).

I do find your statistics per package useful because they show dwz is
in general effective by producing at least 20% (more) on-disk size
reduction, even though there are some packages where dwz doesn't seem
as effective as it could be. We definitely should investigate those
issues. When I looked at the build.logs in the past it did show some
issues with either the DWARF producers, rpm debugedit or dwz. Please do
report bugs if they aren't known yet to the upstream projects.

But I don't really understand why you then focus on the zstd compressed
rpms (even if even those favor dwz). That simply shows that zstd
compression seems to work pretty well. But it doesn't show the on-disk
(and in-memory) size reductions. Or why for the debuginfod use case you
seem to do the opposite, not take into account that the http debuginfod
server will compress the files before sending over the network. I don't
think either of those later statistics are really relevant with respect
to your proposal.

Finally I am interested in your proposal to implement a different way
to reduce the size of DIE trees by eliminating "unused" DIEs. It is
hard to predict what effect that would have without seeing an
implementation (in theory GCC with LTO would not actually generate
debuginfo for unused functions). But I think that can be done separate
from your proposal and combined with other size reduction techniques.

Cheers,

Mark
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: F34 Change proposal: Debug Info Standardization (from DWZ to -fdebug-types-section) (System-Wide Change proposal)

[Mark Wielaard]

On Thu, Sep 24, 2020 at 08:34:48PM +0200, Jan Kratochvil wrote:
> > The tool is not easily maintained,
> 
> I did not mention it there but it is true there are some longterm unfixed
> bugs in DWZ so that it gives up on optimization of many builds:
>   error: Allocatable section after non-allocatable ones
> https://sourceware.org/bugzilla/show_bug.cgi?id=24251#c10

That isn't a longterm unfixed bug. It is is a comment by you from a
couple of days ago that you could still reproduce an already fixed
bug. So we reopened the bug and asked you for a reproducer. Which you
didn't provide. When we then tried to replicate the issue ourselves we
found that dwz correctly seems to detect an issue with the golang
toolchain.

>   error: Couldn't find DIE referenced by DW_AT_abstract_origin
>   (maybe DWZ error or maybe compiler error: Unknown DWARF DW_OP_255)

We cannot fix bugs if you don't report them. Please file bugs for the
above issue (including how to reproduce them) and we'll take a look.

Cheers,

Mark
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: F34 Change proposal: Debug Info Standardization (from DWZ to -fdebug-types-section) (System-Wide Change proposal)

[Mark Wielaard]

Hi,

Replying since I am mentioned by name in this proposal and it seems to
argue for removing a feature I am currently working on to make sure it
works correctly with GCC11 if it switches to producing DWARF5 by
default. The proposal seems based on some misunderstandings.

On Thu, Sep 24, 2020 at 11:59:44AM -0400, Ben Cotton wrote:
> https://fedoraproject.org/wiki/Changes/DebugInfoStandardization
> 
> == Summary ==
> Fedora 18 implemented [[Features/DwarfCompressor]]. As the format did
> not get widespread and the tool is not much maintained

As others pointed out dwz is widely used. It is used by almost every
distro in some form and even freedesktop.org flatpaks use dwz for
their debuginfo.

The upstream project is actively maintained. Even though there are
just 3 committers (including me) the project is still seeing ~2.5
commits a week (about 130 in a year).

> There exist several methods
> how to make the *-debuginfo.rpms at least a bit smaller. Fedora 18
> started using DWZ tool (from [[Features/DwarfCompressor]]) while
> [https://gcc.gnu.org/pipermail/gcc-patches/2008-August/246281.html
> Google implemented] the same goal in a different way called
> -fdebug-types-section.

Note that these methods are not in conflict. Both started out as GNU
extensions but have been standardized since.

> so its support is missing in tools like
> [https://lldb.llvm.org/ LLDB],

But you have been maintaining an out of tree patch for several years
to support partial units and supplemental files (both of which dwz
produces and are now standard DWARF). It would be good if you
upstreamed those patches.

> [[llvm-dwarfdumphttps://llvm.org/docs/CommandGuide/llvm-dwarfdump.html|llvm-dwarfdump]]

Note that normal dwarfdump as shipped with libdwarf-tools does support
both partial units and supplemental files (you do need to provide the
alt file with --file-tied=/path/to/alt-file which probably should be
done automatically).

> or binutils readelf.

binutils readelf is used as the main tool in the dwz testsuite. It
certainly supports both partial units and supplemental files. Also
note the the -wK (=follow-links) option [it isn't on by default, maybe
it should] that resolves alt links.

I don't know of any other tool which doesn't support either partial
units or supplemental files, since both are (now) standard DWARF.

> * DWZ disadvantage: DWZ requires 8x times more complicated (LoC count)
> support in consumers than -fdebug-types-section.

I have worked on support for both in various consumers and didn't find
one method more difficult to support than the other. Maybe debug-types
was actually more difficult. Because the representation changed from
separate section in DWARF4, to intermingled with other .debug_info
units in DWARF5. Making supporting objects that contained mixed
versions a bit of a pain.

> * DWZ disadvantage: DWZ cannot update LLVM .debug_names index which
> can be generated only by clang (it cannot be regenerated later for
> DWZ-compressed file)

dwz does support .gdb_index, the pre-standardized DWARF5 .debug_names
variant. gdb hasn't switched to supporting .debug_names yet and
.gdb_index does work with DWARF5. I don't think the gdb maintainers
would mind you adding .debug_names support if you believe it to be
better than .gdb_index.

> * DWZ disadvantage: DWZ DWARF-5 support is a work-in-progress. DWZ has
> been blocking DWARF-5 for Fedora for 3.5 years and only after I have
> now proposed to drop DWZ Mark Wielaard has started porting DWZ to
> DWARF-5.

I have worked on DWARF5 support for various projects for the last
couple of years. Since it looks like GCC11 might switch to DWARF5 by
default I started added DWARF5 support to dwz. I started that earlier
this month after we had a discussion on DWARF5 at the virtual GNU
Tools Cauldron, I didn't know you were proposing to drop DWZ in
Fedora. You can follow the progress on the dwz mailinglist.
https://sourceware.org/pipermail/dwz/current/

> This proposed DWARF format was originally submitted already for Fedora
> 18 as [[Features/DebugTypesSections]].

Note that it isn't a different format, just an additional way to
represent some parts of DWARF. I have also proposed gcc would emit
debug-types by default and even discussed it two times at the GNU
Tools Cauldron. But I couldn't get buy-in from the GCC hackers that it
is a good idea. And I do think they have a point. It is a not very
flexible design with a somewhat high overhead (somewhat reduced in
DWARF5 by not making it part of a separate section). If you think it
really is a good idea to use them more broadly and by default then you
should probably start by getting consensus from the upstream gcc
developers that it should be enabled by default. I don't think it is
in conflict with also using dwz. But it will probably require some
extra work.

Cheers,

Mark

In which order does ELN build packages, what build root is it using?

[Mark Wielaard]

Hi,

I couldn't build elfutils because of an annobin bug that showed up on
ppc64le. Nick was nice enough to fix it and push a new annobin version:
https://bodhi.fedoraproject.org/updates/FEDORA-2020-c78fce452e
So I could build elfutils again:
https://bodhi.fedoraproject.org/updates/FEDORA-2020-06dafb46c1
But now I am getting notices about the ELN build failures:
https://koji.fedoraproject.org/koji/buildinfo?buildID=1613859
Which is clearly because it doesn't have the new annobin package.

Is this something to worry about? Will ELN resync and re-try the build
automatically? Or do I somehow need to take action to get ELN buildroot
synced with fedora/rawhide again?

Thanks,

Mark
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Conflicting build-ids in chromium and chromium-freeworld

[Mark Wielaard]

Hi,

On Mon, 2020-09-21 at 08:55 +0300, Panu Matilainen wrote:
> On 9/21/20 12:25 AM, Marcin Zajączkowski wrote:
> > Hi. There is an ongoing problem with conflicting build-ids in chromium
> > and chromium-freeworld [1][2]:
> > > Error: Transaction test error:
> > >file /usr/lib/.build-id/61/91aba223f60784c4a2fb95cdedcedc97217e5b from 
> > > install of chromium-freeworld-85.0.4183.102-1.fc32.x86_64 conflicts with 
> > > file from package chromium-85.0.4183.102-1.fc32.x86_64
> > >file /usr/lib/.build-id/82/5827dc3adff19282b7b337b044b381e2f226ee from 
> > > install of chromium-freeworld-85.0.4183.102-1.fc32.x86_64 conflicts with 
> > > file from package chromium-85.0.4183.102-1.fc32.x86_64
> > >file /usr/lib/.build-id/c1/a0a2a071572e46f4b3800bbb0f7ba217b42df7 from 
> > > install of chromium-freeworld-85.0.4183.102-1.fc32.x86_64 conflicts with 
> > > file from package chromium-85.0.4183.102-1.fc32.x86_64
> > >file /usr/lib/.build-id/cc/2a382a1ab1ec74354adf012ad958a17f880f88 from 
> > > install of chromium-freeworld-85.0.4183.102-1.fc32.x86_64 conflicts with 
> > > file from package chromium-85.0.4183.102-1.fc32.x86_64
> > >file /usr/lib/.build-id/d9/1c0feec7526201e135457e3a7f4ff4e4aebfea from 
> > > install of chromium-freeworld-85.0.4183.102-1.fc32.x86_64 conflicts with 
> > > file from package chromium-85.0.4183.102-1.fc32.x86_64
> > 
> > There is no clear idea why it happens, but my assumption is that due to
> > the fact of building with the same source code (with similar patches),
> > some generated shared libraries are the same which generates conflict(s).

The error message could probably be improved.
You might want to look at where the /usr/lib/.build-id/xx/ symlinks
are pointing at to get a better idea which binaries are identical
between the packages.

> > The quick look at the algorithm for build-id generation [3] doesn't
> > answer my question what exactly is taken into account for their
> > generation and more precisely is there is way to add some "fuzz" (having
> > different buildroots doesn't help) to distinguish those libraries.

The build-id is calculated over all relevant build environment bits (by
the linker, not rpm). This includes the debuginfo in the original (not
split) ELF file. The debuginfo contains the build paths (which should
be different for different NVRs and include the compiler version and
flags). This really should prevent identical build-ids whenever
something is really build from source. Normally you only get identical
build-ids when building the same source code in the same package with
the same compiler flags. Identical build-ids between packages are
really very rare and are often caused by some binary blob simply being
copied between packages (is there a blob in the upstream tar ball that
isn't build from source?) or if debuginfo (-g) is missing.

> > To wrap up:
> > 1. Is there a better way to deal with the aforementioned build-id
> > conflicts than disable their generation on one side (with "%global
> > _build_id_links none")?
> 
> Instead of disabling entirely, you could tell rpm to put it all into 
> -debuginfo packages (ie the original debuginfo layout). Which would 
> still conflict, but fewer people are affected:
> 
> %global _build_id_links alldebug

Yes, that is a much better workaround than using none.

> > 2. Maybe my assumption about the same generated shared libraries is
> > wrong and there is something else what can be done to fix the root problem?
> 
> That's exactly the cause, build-id's are engineered to reproducably 
> identify identical built objects, regardless of their location. Which 
> causes conflicts when the house rules of not shipping multiple idental 
> copies is broken (one might call this a feature).

Yes, I would call this a feature :)

> Short of unbundling the shared libraries, I guess a reasonable 
> workaround would be patching them to include some identifier string 
> based on the containing package name, which would cause them to have 
> different build_ids.

If build from source and building with debuginfo that should already be
the case because the build-id is calculated over the original debuginfo
which contains the original (pre-debugedit) build paths, which should
contain the package nvr in their name. So double check that things are
build with -g.

Cheers,

Mark
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Still seeing "DWARF version 0 unhandled" (was: Re: No debugsource generated, weird DWARF errors)

[Mark Wielaard]

Hi Richard,

On Tue, Aug 04, 2020 at 09:55:45AM +0100, Richard W.M. Jones wrote:
> In https://kojipkgs.fedoraproject.org//work/tasks/5877/48605877/build.log
> (from https://koji.fedoraproject.org/koji/taskinfo?taskID=48605877)
> I'm still seeing errors like:
> 
>   /usr/lib/rpm/debugedit: 
> /builddir/build/BUILDROOT/ocaml-ppx-tools-versioned-5.4.0-5.fc33.x86_64/usr/lib64/ocaml/ppx_tools_versioned/metaquot_409/ppx.exe:
>  DWARF version 0 unhandled
> 
> This is with binutils 2.35-10.fc33 which AFAIK should fix this.
> [...]
>   + gcc -O2 -fno-strict-aliasing -fwrapv -O2 -g -pipe -Wall 
> -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS 
> -fexceptions -fstack-protector-strong -grecord-gcc-switches 
> -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 
> -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -mtune=generic 
> -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -Wall 
> -Wdeclaration-after-statement -Werror -fno-common -fexcess-precision=standard 
> -fno-tree-vrp -ffunction-sections -D_FILE_OFFSET_BITS=64 -D_REENTRANT 
> -DCAML_NAME_SPACE  -Wl,-E -o '.ppx/344604af5ef56e11f45ae295381bcce7/ppx.exe'  
> '-L/usr/lib64/ocaml/compiler-libs' 
> '-L/usr/lib64/ocaml/ocaml-migrate-parsetree' 
> '-L/usr/lib64/ocaml/ppx_derivers' '-L/usr/lib64/ocaml/result' '-L.' 
> '-L.ppx_tools_versioned.objs/byte' '-L.ppx_tools_versioned.objs/native' 
> '-L.ppx_tools_versioned_metaquot_409.objs/byte' 
> '-L.ppx_tools_versioned_metaquot_409.objs/native' '-L/usr/lib64/ocaml'  
> '/tmp/camlstartup93ae0f.o' '/usr/lib64/ocaml/std_exit.o' 
> '.ppx/344604af5ef56e11f45ae295381bcce7/_ppx.o' 
> 'ppx_tools_versioned_metaquot_409.a' 'ppx_tools_versioned.a' 
> '/usr/lib64/ocaml/ocaml-migrate-parsetree/migrate_parsetree.a' 
> '/usr/lib64/ocaml/ppx_derivers/ppx_derivers.a' 
> '/usr/lib64/ocaml/result/result.a' 
> '/usr/lib64/ocaml/compiler-libs/ocamlcommon.a' '/usr/lib64/ocaml/stdlib.a' 
> '/usr/lib64/ocaml/libasmrun.a' -Wl,-z,relro -Wl,--as-needed  -Wl,-z,now 
> -specs=/usr/lib/rpm/redhat/redhat-hardened-ld -lm -ldl 
> 
> Not sure how to diagnose this further.  None of this is reproducible
> for me locally even with all the same Rawhide packages installed.

If I had to guess, then I would guess that one of the static libraries
under /usr/lib/64/ocaml/*.a contain bad DWARF. Might it be that one of
those was build with a bad binutils gas?

Cheers,

Mark
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: No debugsource generated, weird DWARF errors

[Mark Wielaard]

On Fri, 2020-07-31 at 12:44 +0100, Richard W.M. Jones wrote:
> On Fri, Jul 31, 2020 at 10:40:57AM +0100, Nick Clifton wrote:
> > That binutils was rebuilt (thanks to Richard Jones) and I have now
> > built an even newer one which contains a fix for AArch64 PLT problems.
> > Both of these are built with LTO disabled, so the bug should not affect 
> > them.
> 
> guestfish on x86-64 was failing for me with something that looks a lot
> like the same PLT problem.  But it's not aarch64.  Very easy to
> reproduce, see:
> 
> https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/thread/ULGH5JYL7MHKDKTINJLOEN2QG6LOHWH7/

Just for the record, I don't think that is the aarch64 plt thing.
Which is https://bugzilla.redhat.com/show_bug.cgi?id=1862110
That would show up as "Failed to update file: invalid section entry
size". I think this is a real LTO issue (both the aarch64 plt and zero
DWARF version weren't, just "normal" bugs in binutils as and binutils
ld, unrelated to LTO).

Cheers,

Mark
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Can we use emulation of other architectures to run integration tests?

[Mark Wielaard]

Hi Aleksandra,

On Thu, 2020-07-30 at 12:24 +0200, Aleksandra Fedorova wrote:
> I'd like to get some understanding on the current state of emulation
> of other architectures.
> 
> In the current CI infra we have infinite(*) access to x86_64 compute
> resources, but we haven't yet got our hands on any non x86_64
> hardware.
> 
> As COPR has recently got support for s390 builds, the question is: if
> emulation is good enough for building packages, can we use it for
> testing? What are the limitations there? Is it worth it?
> 
> If yes, can you suggest some links, tips on the topic?

I tried the copr emulation of arm32 and s390x with the valgrind
package. But although it seems to be able to produce binaries it was
unable to run the tests. Which makes it hard to know whether the
produced binaries are OK. And they make the build fail because we do
require at least some tests to succeed in %check.

Cheers,

Mark
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: We have to talk about annobin... again

[Mark Wielaard]

On Thu, 2020-07-30 at 10:09 +0200, Kevin Kofler wrote:
> Jeff Law wrote:
> > :(  I'll raise it again with Nick and Jakub, it's a sore point for
> > everyone I think and it causes far more friction than just what we see
> > here in Fedora.
> 
> IMHO, we should just drop annobin from Fedora (or at least disable it). It 
> provides no tangible benefit to end users and the regular sync issues with 
> the toolchain are a constant source of pain for packagers. Would it not be 
> enough to do periodic rebuilds with annobin in a side tag to catch packages 
> that are built with insecure or otherwise suboptimal flags, instead of 
> burdening the production repository with this debugging tool?

Although the sync issues are annoying I do think we, as developers of
and developers on the Fedora platform benefit from having the annobin
notes in the binaries. It is like making sure there is unwind
information or debug packages for each binary. If things are perfect,
they are just there for assurance. But if there is an issue you want to
look into, or you get a crash, want to do some profiling to see what
your machine is doing, writing a new program, combine two libraries,
etc. you are glad the information is there.

Cheers,

Mark
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: No debugsource generated, weird DWARF errors

[Mark Wielaard]

Hi Richard,

On Thu, 2020-07-30 at 12:01 +0100, Richard W.M. Jones wrote:
> On Thu, Jul 30, 2020 at 08:37:05AM +0100, Nick Clifton wrote:
> > I will apply this patch to the rawhide binutils (and the upstream
> > binutils sources).
> 
> I don't know how worried we should be about this, but it seems every
> test in the x86-64 build failed (although the build was successful):
> 
> https://kojipkgs.fedoraproject.org//work/tasks/6878/48186878/build.log

Nick is on it:
https://sourceware.org/pipermail/binutils/2020-July/112637.html

Looks like most are simply because the default DWARF version changed to
4 and the test expected another version even though it didn't specify
one.

Cheers,

Mark
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: No debugsource generated, weird DWARF errors

[Mark Wielaard]

Hi Jakub,

On Wed, 2020-07-29 at 19:10 +0200, Jakub Jelinek wrote:
> On Wed, Jul 29, 2020 at 06:33:45PM +0200, Mark Wielaard wrote:
> > Yes, I think we have a winner!
> > gas generates a tiny bit of debuginfo even if you don't supply -g.
> > 
> > But... in 2.35 you can give the DWARF level you want.
> > The problem with not supplying -g (or --gdwarf-[VERSION]) is that the
> > dwarf_level is never set! And then gas will happily create an
> > .debug_info section with DWARF CUs that have a version of zero (!).
> > 
> > This, defaulting to version 4. Fixes it for me:
> 
> Older as versions emitted DWARF 2, so wouldn't that be a better default
> and let gcc pass --gdwarf-? to as if it detects support at compile time
> depending on gcc's default dwarf version or -gdwarf-N option?

Defaulting to dwarf_level = 2 in the patch should also work. But I
think it is high time we defaulted all tools to DWARF version 4 these
days.

Cheers,

Mark
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: No debugsource generated, weird DWARF errors

[Mark Wielaard]

Hi Richard,

On Wed, 2020-07-29 at 17:55 +0100, Richard W.M. Jones wrote:
> On Wed, Jul 29, 2020 at 06:33:45PM +0200, Mark Wielaard wrote:
> > This, defaulting to version 4. Fixes it for me:
> > 
> > diff --git a/gas/as.c b/gas/as.c
> > index 4c5881abd88..c2da78870ef 100644
> > --- a/gas/as.c
> > +++ b/gas/as.c
> > @@ -103,7 +103,7 @@ int verbose = 0;
> >  int flag_dwarf_cie_version = -1;
> >  
> >  /* The maximum level of DWARF DEBUG information we should manufacture.  */
> > -unsigned int dwarf_level = 0;
> > +unsigned int dwarf_level = 4;
> >  
> >  #if defined OBJ_ELF || defined OBJ_MAYBE_ELF
> >  int flag_use_elf_stt_common = DEFAULT_GENERATE_ELF_STT_COMMON;
> 
> So if I understand the proposed fix correctly, we _don't_ need to pass
> the -g option to gas?  Or should we do that also?

Currently passing -g to gas is a workaround. But IMHO it shouldn't be
necessary. It isn't necessary with the patch above.

Cheers,

Mark
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: No debugsource generated, weird DWARF errors

[Mark Wielaard]

Hi Xavier,

On Wed, 2020-07-29 at 18:50 +0200, Xavier Leroy wrote:
> If we need to add a directive to the generated assembly, or pass a `-g`
> option to the assembler, or use `gcc -c -g` as the assembler, let us know
> and we'll see what we can do.
> 
> However, please keep backward compatibility in mind: from the Info manual
> it looks like gas version 2.34 has no `-g` option and no directives to
> control the generation of DWARF information.  So there should be reasonable
> defaults for gas 2.35 to behave roughly like gas 2.34.

I think the change in binutils/gas 2.35 was supposed to be backward
compatible. I am pretty sure it was just an oversight that the DWARF
version (dwarf_level) was not set in the none -g case.

Lets see what Nick (the binutils maintainer) says. My patch is just the
first thing I came up with. I am sure there are other ways to fix this.

Cheers,

Mark

> > But... in 2.35 you can give the DWARF level you want.
> > The problem with not supplying -g (or --gdwarf-[VERSION]) is that the
> > dwarf_level is never set! And then gas will happily create an
> > .debug_info section with DWARF CUs that have a version of zero (!).
> > 
> > This, defaulting to version 4. Fixes it for me:
> > 
> > diff --git a/gas/as.c b/gas/as.c
> > index 4c5881abd88..c2da78870ef 100644
> > --- a/gas/as.c
> > +++ b/gas/as.c
> > @@ -103,7 +103,7 @@ int verbose = 0;
> >  int flag_dwarf_cie_version = -1;
> > 
> >  /* The maximum level of DWARF DEBUG information we should
> > manufacture.  */
> > -unsigned int dwarf_level = 0;
> > +unsigned int dwarf_level = 4;
> > 
> >  #if defined OBJ_ELF || defined OBJ_MAYBE_ELF
> >  int flag_use_elf_stt_common = DEFAULT_GENERATE_ELF_STT_COMMON;
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: No debugsource generated, weird DWARF errors

[Mark Wielaard]

Hi,

On Wed, 2020-07-29 at 17:18 +0100, Richard W.M. Jones wrote:
> (Adding OCaml author)

(Adding binutils/gas maintainer)

> On Wed, Jul 29, 2020 at 05:54:52PM +0200, Mark Wielaard wrote:
> > On Wed, 2020-07-29 at 16:07 +0100, Richard W.M. Jones wrote:
> I should probably add that I'm building these on my local machine
> which isn't completely updated to Rawhide.  I'm not sure whether or
> not that will make a difference - I'm assuming _not_ for what the
> OCaml compiler generates (since I'm running the latest of that), but
> not sure about the rest of the toolchain.  I'm using
> binutils-2.34-3.fc32.x86_64.

Right. Same here. Dunno why mockbuild didn't pick up the 2.35 version,
but I think the bug is in that one. See below.

> > > I also saved an asm file from one of them which may be helpful:
> > > 
> > > http://oirase.annexia.org/tmp/hacha.s
> > 
> > So ml depends on binutils gas to generate the actual debuginfo.
> > I assume it gets called with: as -g
> > Is there a way to see how exactly gas is called (with which arguments)?
> 
> In fact it *isn't* passing -g to as:
> 
> $ /usr/bin/ocamlopt.opt -c -w +a-3-4-9-41-45-67 -g -annot -safe-string -o 
> hacha.cmx hacha.ml -S -verbose
> + as  -o 'hacha.o' 'hacha.s'
> 
> Is this a problem?  I sort of assumed that as would have nothing to do
> with generating debug information, beyond what is contained explicitly
> in the .s file itself.

Yes, I think we have a winner!
gas generates a tiny bit of debuginfo even if you don't supply -g.

But... in 2.35 you can give the DWARF level you want.
The problem with not supplying -g (or --gdwarf-[VERSION]) is that the
dwarf_level is never set! And then gas will happily create an
.debug_info section with DWARF CUs that have a version of zero (!).

This, defaulting to version 4. Fixes it for me:

diff --git a/gas/as.c b/gas/as.c
index 4c5881abd88..c2da78870ef 100644
--- a/gas/as.c
+++ b/gas/as.c
@@ -103,7 +103,7 @@ int verbose = 0;
 int flag_dwarf_cie_version = -1;
 
 /* The maximum level of DWARF DEBUG information we should manufacture.  */
-unsigned int dwarf_level = 0;
+unsigned int dwarf_level = 4;
 
 #if defined OBJ_ELF || defined OBJ_MAYBE_ELF
 int flag_use_elf_stt_common = DEFAULT_GENERATE_ELF_STT_COMMON;
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: No debugsource generated, weird DWARF errors

[Mark Wielaard]

Hi Richard,

On Wed, 2020-07-29 at 16:07 +0100, Richard W.M. Jones wrote:
> On Wed, Jul 29, 2020 at 04:11:56PM +0200, Mark Wielaard wrote:
> > Given these are .ml files I suspect it is not gcc, but some other
> > code/DWARF generator issue. Maybe it does use the default
> > (binutils)
> > liker though?
> 
> It uses its own DWARF generator but everything is linked together
> using standard binutils (via GCC).
> 
> > Is there a way to extract the /usr/bin/hacha from the BUILDROOT so
> > we
> > can inspect it?
> 
> I've uploaded the binary which I built on my own machine here:
> 
> http://oirase.annexia.org/tmp/hacha.native

This one looks OK.

> plus some of the *.o files which went into it:
> 
> http://oirase.annexia.org/tmp/myLexing.o
> http://oirase.annexia.org/tmp/myStack.o
> http://oirase.annexia.org/tmp/hacha.o

And so do these.

I also tried to do a mockbuild locally, and that one succeeded.
Don't know what is different from the koji buildroot :{

> I also saved an asm file from one of them which may be helpful:
> 
> http://oirase.annexia.org/tmp/hacha.s

So ml depends on binutils gas to generate the actual debuginfo.
I assume it gets called with: as -g
Is there a way to see how exactly gas is called (with which arguments)?

One of the gas 2.35 features is:

* Add --gdwarf-5 option to the assembler to generate DWARF 5 debug output
  (if such output is being generated).  Added the ability to generate
  version 5 .debug_line sections.

Which is the version that just hit fedora rawhide. Maybe that changed
something about the gas -g output as well?

___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: No debugsource generated, weird DWARF errors

[Mark Wielaard]

Hi,

On Wed, 2020-07-29 at 08:05 -0600, Jeff Law wrote:
> On Wed, 2020-07-29 at 13:15 +0100, Richard W.M. Jones wrote:
> > https://koji.fedoraproject.org/koji/taskinfo?taskID=48101965 fails
> > with:
> > 
> >   error: Empty %files file 
> > /builddir/build/BUILD/hevea-2.34/debugsourcefiles.list
> > 
> > However it works when I build locally:
> > 
> >   $ rpm -qlp 
> > /home/rjones/d/fedora/hevea/master/x86_64/hevea-debugsource-2.34-7.fc33.x86_64.rpm
> >   /usr/src/debug/hevea-2.34-7.fc33.x86_64
> >   /usr/src/debug/hevea-2.34-7.fc33.x86_64/_build
> >   /usr/src/debug/hevea-2.34-7.fc33.x86_64/_build/auxx.ml
> >   /usr/src/debug/hevea-2.34-7.fc33.x86_64/_build/bibhva.ml
> >   /usr/src/debug/hevea-2.34-7.fc33.x86_64/_build/color.ml
> >   [etc]
> > 
> > There's an earlier error which occurs in the Koji build which doesn't
> > occur for me locally but looks related:
> > 
> >   Dwarf Error: wrong version in compilation unit header (is 0, should be 2, 
> > 3, 4 or 5) [in module 
> > /builddir/build/BUILDROOT/hevea-2.34-7.fc33.x86_64/usr/bin/hacha]
> >   gdb-add-index: No index was created for 
> > /builddir/build/BUILDROOT/hevea-2.34-7.fc33.x86_64/usr/bin/hacha
> >   gdb-add-index: [Was there no debuginfo? Was there already an index?]
> > 
> > Did something change with DWARF or gdb in Rawhide that we should be
> > aware of?  Obviously these are not C code and OCaml has its own DWARF
> > generator.
> 
> I think mjw is tracking something like this yesterday, but it was aarch64.  
> I'm
> going to be offline for a couple hours, but you might want to reach out to him
> and see if there's any useful state (on cc)

This looks unrelated to what I was looking at, which was indeed aarch64
specific (but doesn't actually seem to be impacting the mass rebuild):
https://bugzilla.redhat.com/show_bug.cgi?id=1861423

The above happens if none of the debuginfo could be read at all, so no
source files could be extracted.

Given these are .ml files I suspect it is not gcc, but some other
code/DWARF generator issue. Maybe it does use the default (binutils)
liker though?

Is there a way to extract the /usr/bin/hacha from the BUILDROOT so we
can inspect it?

Cheers,

Mark
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Dropping elfutils-libelf-devel-static and elfutils-devel-static subpackages

[Mark Wielaard]

Hi,

On Thu, 2020-07-23 at 09:59 +0200, Fabio Valentini wrote:
> On Thu, Jul 23, 2020 at 1:25 AM Mark Wielaard 
> > I committed the removal of the two static subpackages and added an
> > Obsoletes: elfutils-devel-static%{depsuffix} < 0.180-5 to elfutils-
> > devel and an Obsoletes: elfutils-libelf-devel-static%{depsuffix} <
> > 0.180-5 to elfutils-devel.
> > 
> > But I haven't done a build yet to give people a chance to yell and
> > scream this is the wrong way to do it. If nobody complains I'll do
> > a build of elfutils-0.180-5 for rawhide on Friday.
> 
> The subpackage removal and addition of appropriate Obsoletes tags
> looks correct to me.

Thanks. I had to make one small adjustment, rpmbuild in rawhide doesn't
accept the depsuffix on an Obsoletes. It is unnecessary anyway (it is
required on the Requires, because the libraries are multilib).

I have now build elfutils-0.180-5 which drops the -static packages in
rawhide. This should only affect the kexec-tools (the maintainers are
aware of the BuildRequires change necessary). But if there is any other
fallout please yell and scream.

BTW. Can Obsoletes ever be removed? We have an Obsoletes: libelf <=
0.8.2-2 on elfutils-libelf since the original cvsdist import of 2004
because there used to be a different libelf implementation (with a dead
upstream these days). Can I remove that? Or is it better to keep it
"just in case"?

Thanks,

Mrk
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Dropping elfutils-libelf-devel-static and elfutils-devel-static subpackages

[Mark Wielaard]

Hi,

On Tue, 2020-07-21 at 15:21 -0700, Josh Stone wrote:
> On 7/21/20 3:12 PM, Mark Wielaard wrote:
> > > > Is there a procedure to follow for dropping these sub-packages, or can
> > > > I simply remove them from the spec file?
> > > 
> > > Maybe add Obsoletes to the devel subpackages, but not Provides.
> > > 
> > > https://docs.fedoraproject.org/en-US/packaging-guidelines/#renaming-or-replacing-existing-packages
> > 
> > You mean add an Obsoletes: elfutils-devel-static to the elfutils-devel
> > subpackage? Does that really work with packages that are normally build
> > requires of other packages?
> 
> Yes, that's what I mean, and make sure to version that like the link
> describes. AFAIK the Obsoletes alone will just solve the upgrade path,
> but you would need Provides to satisfy the build requires. That doesn't
> make sense to do in this case, since you're not going to provide static
> functionality in the "replacement".

OK. Thanks. I had a chat with the kexec-tools maintainers and it turns
out that they already link dynamically (since a couple of years) but
the elfutils-devel-static BuildRequires was left in (and since it
Requires elfutils-devel, it all build fine anyway). They agreed to
remove the BuildRequires. So nothing should block the removal anymore.

I committed the removal of the two static subpackages and added an
Obsoletes: elfutils-devel-static%{depsuffix} < 0.180-5 to elfutils-
devel and an Obsoletes: elfutils-libelf-devel-static%{depsuffix} <
0.180-5 to elfutils-devel.

But I haven't done a build yet to give people a chance to yell and
scream this is the wrong way to do it. If nobody complains I'll do a
build of elfutils-0.180-5 for rawhide on Friday.

Thanks,

Mark
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: pagure pull-request email workflow

[Mark Wielaard]

Hi,

On Wed, 2020-07-22 at 09:32 -0400, Neal Gompa wrote:
> On Wed, Jul 22, 2020 at 9:19 AM Tom Hughes via devel
>  wrote:
> > 
> > On 22/07/2020 13:19, Mark Wielaard wrote:
> > 
> > > As you say, the web api is even more resourceful and we can integrate
> > > some of those requests into the library: https://pagure.io/api/
> > > 
> > > The only thing that is not very nice are those pagure_tokens. I was
> > > hoping you could get a temporary one through simple fedora kerberos
> > > authentication. But the only way to create one seems to be through a
> > > webbrowser: https://src.fedoraproject.org/settings#nav-api-tab
> > > You can create one that is valid for 2 years, which is helpful, but
> > > doesn't feel very secure. How do people manage those tokens?
> > > 
> > > And I really would like to see a mail backend (maybe simply accepting
> > > gpg signed emails) so you don't have to be online just to deal with
> > > these kind of pull requests.
> > 
> > Given it's all getting replaced by gitlab anyway it's probably
> > not worth spending a lot of time on it...
> > 
> 
> I'd rather it didn't get replaced with GitLab, and I wish that process
> wasn't done so poorly so that it wasn't a foregone conclusion like it
> apparently was. The worst part of that is CPE leadership just paid lip
> service to us and just did it anyway.

That is not how I understand the current situation. CPE has a proposal
to provide a forge for use by the fedora project possibly based on
gitlab, but pagure wouldn't go away. They are currently investigating
if they could provide such a gitlab forge in a way that would be
acceptable to the fedora project, which means self-hosted and based on
the free software edition. But that project seemed to have stalled
since it is unclear that is even possible. Meanwhile, if they would
offer such a forge and it would replace our use of pagure then they
would be on the hook for providing integration with the tools fedora
uses for packaging workflows. Given that the pagure api is public and
open it seems fine to use it for more integration right now.

Anyway, I have handled a couple of request for some packages now
through some quick libpagure python scripting. It is a little rough
around the edges, but seems useful.

Cheers,

Mark
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: pagure pull-request email workflow

[Mark Wielaard]

Hi Tom,

On Tue, 2020-07-21 at 13:53 +0100, Tom Hughes via devel wrote:
> On 21/07/2020 13:12, Mark Wielaard wrote:
> > > I normally just edit .git/config and add to the origin remote
> > > an extra fetch:
> > > 
> > > fetch = +refs/pull/*/head:refs/remotes/origin/pull/*
> > > 
> > > then after fetching you can merge origin/pull/NNN.
> > 
> > But this is very helpful! Thanks.
> > 
> > So with that I can easily do checks, adjustments, create my own
> > commits, do cherry-picks or merges, etc. But how does it interact with
> > the website? How do I sent comments? I see it comes with a mini-CI koji
> > run, does it add a tag to the commit as tested when it succeeds/fails?
> > How do I indicate which changes I made/pushed or which changes I would
> > like the submitter to make? How do I discard a pull if I determine it
> > isn't useful? etc.
> 
> You use the web site as far as I know.
> 
> Unlike github it doesn't even notice when you push a merge so
> you still have to close the PR on the web site as well.
> 
> > And if I wish to create a pagure pull request myself, do I simply push
> > to pull/NNN? How do I determine which NNN to use?
> 
> Well you push to a branch on your fork but then you have to
> use the web site to open the PR I think.
> 
> I mean there is a pagure API through which you can likely do
> all these things but AFAIK there is no CLI interface for it so
> you have to hit it with curl or something ;-)

I found libpagure, which makes it possible to write some command line
tooling. The library is a little clunky imho, but that might be because
my python is a little rusty. But I was able to comment on the pull
request and actually merge it. Although I wish I hadn't because it
doesn't do a rebase first. But combined with the .git/config trick to
add:

fetch = +refs/pull/*/head:refs/remotes/origin/pull/*

libpagure has the potential to create a useful workflow. For others
that want to try, you'll initialize through something like:

>>> from libpagure import Pagure
>>> pg = Pagure(pagure_token="xxx",instance_url="
https://src.fedoraproject.org/",pagure_repository="rpms/valgrind";)
>>> pg.comment_request(1, "Trying out libpagure commenting. This
request looks reasonable.")

As you say, the web api is even more resourceful and we can integrate
some of those requests into the library: https://pagure.io/api/

The only thing that is not very nice are those pagure_tokens. I was
hoping you could get a temporary one through simple fedora kerberos
authentication. But the only way to create one seems to be through a
webbrowser: https://src.fedoraproject.org/settings#nav-api-tab
You can create one that is valid for 2 years, which is helpful, but
doesn't feel very secure. How do people manage those tokens?

And I really would like to see a mail backend (maybe simply accepting
gpg signed emails) so you don't have to be online just to deal with
these kind of pull requests.

Thanks,

Mark
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Dropping elfutils-libelf-devel-static and elfutils-devel-static subpackages

[Mark Wielaard]

Hi Josh,

On Tue, 2020-07-21 at 14:47 -0700, Josh Stone wrote:
> On 7/21/20 2:24 PM, Mark Wielaard wrote:
> > Nothing seems to require these packages, but that might be simply be
> > because they are static libraries, so there aren't any runtime
> > requirements. Is there some way to determine if anything would start
> > failing to build if I simply remove them? Apart from simply waiting for
> > the bug reports to pop up :)
> 
> I found one:
> 
> $ dnf repoquery --quiet --repo rawhide-source --arch src \
> --whatrequires 'elfutils-*-static'
> kexec-tools-0:2.0.20-14.fc33.src

Thanks, nice trick. I'll contact the maintainers to ask if they can
rebuild against the normal -devel packages before I drop the -static-
devel ones.

> > Is there a procedure to follow for dropping these sub-packages, or can
> > I simply remove them from the spec file?
> 
> Maybe add Obsoletes to the devel subpackages, but not Provides.
> 
> https://docs.fedoraproject.org/en-US/packaging-guidelines/#renaming-or-replacing-existing-packages

You mean add an Obsoletes: elfutils-devel-static to the elfutils-devel
subpackage? Does that really work with packages that are normally build
requires of other packages?

Thanks,

Mark
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Dropping elfutils-libelf-devel-static and elfutils-devel-static subpackages

[Mark Wielaard]

Hi,

I would like to drop the elfutils-libelf-devel-static and elfutils-
devel-static subpackages which provide static libraries for libelf and
libdw/libasm. They seem to have been provided a long time ago for some
core binaries, like rpm, to provide static binaries. But it looks like
nothing is using this mechanism anymore today. The static libdw library
probably never really worked correctly till 0.180 (released just a few
weeks ago), because it depended on dynamically open some other non-
static libraries... oops. I don't believe there is any good reason to
not simply link against the normal shared libraries (they have been
binary compatible for years).

Nothing seems to require these packages, but that might be simply be
because they are static libraries, so there aren't any runtime
requirements. Is there some way to determine if anything would start
failing to build if I simply remove them? Apart from simply waiting for
the bug reports to pop up :)

Is there a procedure to follow for dropping these sub-packages, or can
I simply remove them from the spec file?

Thanks,

Mark
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: pagure pull-request email workflow

[Mark Wielaard]

On Tue, 2020-07-21 at 12:25 +0100, Tom Hughes via devel wrote:
> On 21/07/2020 11:56, Mark Wielaard wrote:
> 
> > Do you have to handle them on that pagure website? Is it possible
> > to
> > handle these pull-request through email? Or is there a normal (git)
> > command line interface for these?
> 
> Pagure supports the same pull heads are things like github

Never having used github, that doesn't immediately help me.

> so yes you can just fetch them and merge them in your local
> fepdkg checkout if you want.
> 
> I normally just edit .git/config and add to the origin remote
> an extra fetch:
> 
> fetch = +refs/pull/*/head:refs/remotes/origin/pull/*
> 
> then after fetching you can merge origin/pull/NNN.

But this is very helpful! Thanks.

So with that I can easily do checks, adjustments, create my own
commits, do cherry-picks or merges, etc. But how does it interact with
the website? How do I sent comments? I see it comes with a mini-CI koji
run, does it add a tag to the commit as tested when it succeeds/fails?
How do I indicate which changes I made/pushed or which changes I would
like the submitter to make? How do I discard a pull if I determine it
isn't useful? etc.

And if I wish to create a pagure pull request myself, do I simply push
to pull/NNN? How do I determine which NNN to use?

Thanks,

Mark
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

pagure pull-request email workflow

[Mark Wielaard]

Hi,

I got a pagure pull-request for my package (a first!).
But I am slightly confused how to handle this.

The email that pagure sents is not very helpful since it doesn't
include the actual patch to try out. Also when I replied to the email
it seems to have not gone to the actual submitter (and just seems to
have disappeared completely). So I have no idea how to interact with
them (actually I figured out there is a patch link on the website,
which luckily does include the submitters email, so I could email them
with some questions, but it wasn't very intuitive or easily
discoverable).

Do you have to handle them on that pagure website? Is it possible to
handle these pull-request through email? Or is there a normal (git)
command line interface for these?

Thanks,

Mark
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Fedora 33 System-Wide Change proposal: CompilerPolicy Change

[Mark Wielaard]

Hi Jeff,

On Fri, 2020-06-05 at 10:07 -0600, Jeff Law wrote:
> On Fri, 2020-06-05 at 15:56 +, devel-requ...@lists.fedoraproject.org 
> wrote:
> > One issue I am concerned about here is debuginfo quality. GCC produced
> > pretty bad debuginfo with LTO in older versions. The EarlyDebug work
> > did make it usable. And we needed some work on the DARF consumers to
> > correctly process GCC 10 LTO produced DWARF (I actually have two small
> > patches for elfutils pending so it correctly parses the cross CU-
> > references GCC 10 LTO now produces).
> > 
> > I don't know if anybody did any analysis on llvm LTO produced
> > debuginfo. In the past it was observed that llvm doesn't do VTA (Var
> > Tracking Assignments) that is default in GCC. And VTA is really
> > important for debugging and performance/tracing tools like systemtap.
> > So even without LTO I am concerned about the observability of llvm
> > generated binaries.
> 
> And these are reasonable concerns (even more so in an LTO world and I expect 
> I'll
> be spending most of my summer poking at debuginfo stuff for GCC LTO :-).
> 
> IMHO this is one of the issues an upstream project needs to evaluate for their
> toolchain choice -- no different than performance, diagnostics, sanitizer
> support, plug-in support, etc etc.  The level of observability needed by any
> particular upstream project can vary.  ie, the kernel may well have different
> needs than the Ruby interpreter.

I think you place too much faith in upstream and give Fedora and Fedora
packagers not enough credit for creating a well integrated
distribution. For Fedora it is important that what we ship is
observable by our users. We, together with our users, sometimes need to
debug, trace or do performance analysis on the binaries as we ship
them. Some upstream projects don't care about that at all. They will
happily tell their users to just rebuild with some other
tools/optimizations/etc. Or just rebuild their whole world with some
sanitizers enabled. So then an upstream developer might prefer a
different toolchain, simply because they personally would rebuild the
whole world with it. In such cases Fedora should not simply switch to a
non-default system compiler.

> I have not done any evaluation of LLVM's debug info.  I would not be 
> surprised if
> GCC is generally better, particularly with Alex's work over the last couple
> years.  But again, I think the upstream project is in the best position to
> determine how to balance this against the other factors in toolchain 
> selection.

GCC's is certainly better, it has much better coverage. That doesn't
mean that LLVM's debuginfo is invalid, but it is less useful. At some
point the kernel accidentially disabled VTA and it made the kernel a
lot less observable by systemtap for example.

And even if it is correct it might still not be fully supported by all
our tools. For example clang used to default to not generate
.debug_aranges, which is valid, but means a DWARF consumer needs to
construct its own aranges table by reading the all the CUs. elfutils
doesn't support that, and so some address/line lookups simply fail with
clang generated binaries. I could cerainly add support for missing
.debug_aranges, but there is so much more to do (luckily I could
convince them to switch the default to generate .debug_aranges, at
least for rust code).

In general when someone reports an issue with an elfutils or valgrind
based tool and it is against code not compiled with the default gcc
toolchain I simply have to tell people to please use the system
compiler. There are just not enough time to also try to resolve any
debuginfo issues because people use an alternative toolchain.

So I like this proposal to be a bit more strict in when it is OK to not
use the system toolchain. Just deferring to upstream preference seems
too weak. If upstream supports building with gcc, but some upstream
developers have a preference for using clang, I would prefer to make it
clear that the Fedora package should keep using gcc. We simply don't
have the resources for all other tools to have to deal with another
toolchain.

Thanks,

Mark
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Fedora 33 System-Wide Change proposal: CompilerPolicy Change

[Mark Wielaard]

Hi,

On Fri, 2020-06-05 at 09:25 -0600, Jeff Law wrote:
> The LTO bytecode streams do not survive past any given package build.  ie, 
> they
> are used within the build, then discarded.  They're not supposed to show up in
> any installed libraries.
> 
> Thus the fact that the two compilers use totally different LTO representations
> (and always will) is a non-issue here.

One issue I am concerned about here is debuginfo quality. GCC produced
pretty bad debuginfo with LTO in older versions. The EarlyDebug work
did make it usable. And we needed some work on the DARF consumers to
correctly process GCC 10 LTO produced DWARF (I actually have two small
patches for elfutils pending so it correctly parses the cross CU-
references GCC 10 LTO now produces).

I don't know if anybody did any analysis on llvm LTO produced
debuginfo. In the past it was observed that llvm doesn't do VTA (Var
Tracking Assignments) that is default in GCC. And VTA is really
important for debugging and performance/tracing tools like systemtap.
So even without LTO I am concerned about the observability of llvm
generated binaries.

Cheers,

Mark
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Fedora 33 System-Wide Change proposal: CompilerPolicy Change

[Mark Wielaard]

On Fri, 2020-06-05 at 11:19 +0200, Jakub Jelinek wrote:
> On Fri, Jun 05, 2020 at 09:52:09AM +0200, Kevin Kofler wrote:
> > I do not see why we should allow yet another special case for Firefox, nor 
> > why we should let random packages make their own choice of compiler and 
> > risk 
> > running into hidden binary incompatibilities. We have a system compiler for 
> > a reason.
> 
> I'll note that there are some even not really hidden binary
> incompatibilities, where LLVM diverges from the psABI for years, it has been
> reported and nothing has been changed.
> So if a library is built with clang/LLVM and used by GCC built package or
> vice versa, one might very well run into those (this is e.g. about passing
> std::byte or other scoped enums with char/short underlying type by value, or
> in some cases even about passing char/short arguments).
> And of course unknown ABI bugs on both sides.

So the new policy is only for non-library packages then?
Packages which provide libraries that can be linked to by other
packages still need to use the default system compiler to make sure
there is no breakage. At least to the other packages. A package that is
build with llvm might subtly break when linked against system
libraries.

Thanks,

Mark
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: CPE Weekly: 2020-04-04

[Mark Wielaard]

Hi,

On Mon, 2020-04-06 at 09:49 -0400, Ben Rosser wrote:
> > Not everyone is inclined to loudly argue their positions on the
> > mailing
> > list. There have only been 12 unique participants to this thread and 57
> > to the other thread.
> > 
> > That isn't indicative of the entire Fedora packager ecosystem. A lot of
> > people are staying silent.
> > [...]
> 
> I'm a packager who has been staying silent, but I generally strongly
> agree with the points that Adam, Miro, Neal, and others have been
> making with a few caveats:

This is also why I am staying silent. I believe the points I would make
have already been made, and probably much better than I could have
formulated them. Including what Ben just said.

> * I don't _really_ mind if we wind up using Gitlab over Pagure, but if
> we do, I do feel pretty strongly that we should use Gitlab CE and
> self-host it-- I don't think it would be right for Fedora to use an
> externally hosted solution and I don't think we should use the
> enterprise edition.
> 
> * I don't like how this process has been conducted, and I think that
> official responses from CPE thus far haven't really made things
> better-- if anything, the "we apologize, but this is the decision
> we've made" attitude is making things worse.
> 
> * I fear that, once again, we haven't adequately understood the
> consequences of replacing pagure and some of the features that were
> recently-- finally!-- added to it in order to replace missing pkgdb2
> functionality will again be lost for a long period of time... and
> nothing I've read in any of these threads so far has helped reassure
> me that's not the case.

It seems to me that nobody is happy with the process that was followed.
Reading back it is clear that people feel that what they perceive as
the Fedora goals and mission were not taken into account. Or at least
not formulated correctly, or strongly enough, during this process. And
that the process wasn't transparent, because various steps were not
visible enough. So a large part of the community was surprised by the
decision and believe that their input wasn't heard or simply ignored.
Including the input of some of who will have to do part of the
(integration) work.

I am sure that wasn't what was intended. And that there were just a few
communication accidents that caused things to break down. But
recognizing that some mistakes were made and trying to correct them so
that a real community can be build up around these kind of decisions is
important.

Cheers,

Mark
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

elfutils.spec typo broke rawhide koji builds

[Mark Wielaard]

Hi,

I made a typo in the elfutils.spec for elfutils-0.178-3.fc32 which
broke the provide/requires dependency for elfutils-libelf. I fixed it
in elfutils-0.178-4. But I cannot get it to build in koji because
rpmbuild depends on elfutils and sees the broken requires.

Is there a way to remove elfutils-0.178-3 and build elfutils-0.178-4
against a non-broken earlier version to unbreak koji?

Thanks,

Mark
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

debuginfod non-standard-uid and cache permissions

[Mark Wielaard]

Hi fedora devel list,

The new elfutils upstream comes with a debuginfod server which we want
packaged (as a sub-package) for fedora. Testing looks good and
everything seems to work, but rpmlint flags a couple of issues that I
don't think should be real issues. Could someone help me understand why
rpmlint seems unhappy with:

   elfutils-debuginfod.x86_64: W: non-standard-uid /var/cache/debuginfod 
debuginfod
   elfutils-debuginfod.x86_64: W: non-standard-gid /var/cache/debuginfod 
debuginfod
   elfutils-debuginfod.x86_64: E: non-standard-dir-perm /var/cache/debuginfod 
700
   elfutils-debuginfod.x86_64: W: non-standard-uid 
/var/cache/debuginfod/debuginfod.sqlite debuginfod
   elfutils-debuginfod.x86_64: W: non-standard-gid 
/var/cache/debuginfod/debuginfod.sqlite debuginfod
   elfutils-debuginfod.x86_64: E: non-readable 
/var/cache/debuginfod/debuginfod.sqlite 600
   elfutils-debuginfod.x86_64: E: zero-length 
/var/cache/debuginfod/debuginfod.sqlite

These are all done deliberately through the following constructs in the spec 
file:

In %pre to create the debuginfod user:

   getent group debuginfod >/dev/null || groupadd -r debuginfod
   getent passwd debuginfod >/dev/null || \
   useradd -r -g debuginfod -d /var/cache/debuginfod -s /sbin/nologin \
   -c "elfutils debuginfo server" debuginfod
   exit 0

In %install to create the dir/file artifacts:

   mkdir -p ${RPM_BUILD_ROOT}%{_localstatedir}/cache/debuginfod
   touch ${RPM_BUILD_ROOT}%{_localstatedir}/cache/debuginfod/debuginfod.sqlite

And in %files to install them with the right permissions:

   %dir %attr(0700,debuginfod,debuginfod) %{_localstatedir}/cache/debuginfod
   %verify(not md5 size mtime) %attr(0600,debuginfod,debuginfod) 
%{_localstatedir}/cache/debuginfod/debuginfod.sqlite

Should anything be done differently or does any of that violate
(rpmlint) policy somehow?

Thanks,

Mark
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: systemtap doesn't run

[Mark Wielaard]

Hi Jan,

CC systemtap upstream list, because I think this is not a great error
message.

On Fri, 2019-09-06 at 09:53 +0200, Jan Synacek wrote:
> I'm trying to run systemtap on F29 and I'm getting the following
> error:
> 
> $ sudo stap -v journal.stap
> Pass 1: parsed user script and 491 library scripts using
> 355824virt/129076res/9628shr/119256data kb, in 290usr/40sys/334real
> ms.
> semantic error: while resolving probe point: identifier 'process' at
> journal.stap:1:7
> source: probe
> process("/usr/lib/systemd/systemd-
> journald").function("dispatch_message_real")
> {
>   ^
> 
> semantic error: no match (similar functions: read, free, getenv,
> page_size,
> safe_atoi)
> 
> So, 'process' is not a valid identifier? There seems to be something
> wrong
> with the basic systemtap installation. I do have matching debuginfo
> for
> both kernel and systemd installed. Running stap-prep only wants to
> install
> kernel-debuginfo.
> 
> How do I make this basic use-case work?

It is a bit hard to say, because you didn't include journal.stap.
But I can replicate what you get with:

stap -v -e 'probe process("/usr/lib/systemd/systemd-
journald").function("dispatch_message_real") { log ("hit"); }'

You get that error message if stap cannot find that function symbol.
So first that ^ carrot should really not be at "process", but at
"function" (or really "dispatch_message_real").

stap really should tell you how to get that symbol. By installing the
matching debuginfo package.

You also get that message if the debuginfo and main package don't match
up. I had:

$ rpm -q systemd-debuginfo systemd
systemd-debuginfo-241-12.git1e19bcd.fc30.x86_64
  systemd-241-10.git511646b.fc30.x86_64

The full version-release number should be identical.

After updating the systemd package, so it matched the systemd-
debuginfo 
package, things finally started to work.

[mark@f30 ~]$ stap -e 'probe process("/usr/lib/systemd/systemd-
journald").function("dispatch_message_real") { log ("hit"); }'
hit
hit
hit

Cheers,

Mark
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: systemtap doesn't run

[Mark Wielaard]

Hi Jan,

CC systemtap upstream list, because I think this is not a great error
message.

On Fri, 2019-09-06 at 09:53 +0200, Jan Synacek wrote:
> I'm trying to run systemtap on F29 and I'm getting the following
> error:
> 
> $ sudo stap -v journal.stap
> Pass 1: parsed user script and 491 library scripts using
> 355824virt/129076res/9628shr/119256data kb, in 290usr/40sys/334real ms.
> semantic error: while resolving probe point: identifier 'process' at
> journal.stap:1:7
> source: probe
> process("/usr/lib/systemd/systemd-journald").function("dispatch_message_real")
> {
>   ^
> 
> semantic error: no match (similar functions: read, free, getenv, page_size,
> safe_atoi)
> 
> So, 'process' is not a valid identifier? There seems to be something wrong
> with the basic systemtap installation. I do have matching debuginfo for
> both kernel and systemd installed. Running stap-prep only wants to install
> kernel-debuginfo.
> 
> How do I make this basic use-case work?

It is a bit hard to say, because you didn't include journal.stap.
But I can replicate what you get with:

stap -v -e 'probe 
process("/usr/lib/systemd/systemd-journald").function("dispatch_message_real") 
{ log ("hit"); }'

You get that error message if stap cannot find that function symbol.
So first that ^ carrot should really not be at "process", but at
"function" (or really "dispatch_message_real").

stap really should tell you how to get that symbol. By installing the
matching debuginfo package.

You also get that message if the debuginfo and main package don't match
up. I had:

$ rpm -q systemd-debuginfo systemd
systemd-debuginfo-241-12.git1e19bcd.fc30.x86_64
  systemd-241-10.git511646b.fc30.x86_64

The full version-release number should be identical.

After updating the systemd package, so it matched the systemd-debuginfo 
package, things finally started to work.

[mark@f30 ~]$ stap -e 'probe process("/usr/lib/systemd/systemd-
journald").function("dispatch_message_real") { log ("hit"); }'
hit
hit
hit

Cheers,

Mark
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Fedora 31 System-Wide Change proposal (late): No i686 Repositories

[Mark Wielaard]

Hi Kevin,

On Sun, 2019-07-14 at 15:50 -0700, Kevin Fenzi wrote:
> On 7/14/19 2:35 PM, John Reiser wrote:
> > Kevin Fenzi wrote:
> > > Neal Gompa wrote:
> > 
> > [[snip]]
> > 
> > > > This will also make it impossible for people to locally do multilib
> > > > build/installs. It will remove COPR’s ability to do the same. For that
> > > > reason alone, I don’t particularly want this change to happen.
> > > Can you expand on what you mean by 'locally do' ?
> > 
> > I want to run "gcc -m32 -o my_app-i686 *.o ..." locally on my own box
> > to build executables that run as 32-bit apps on multilib x86_64.
> > For some apps 2GB of malloc() arena is plenty, and they run faster
> > in 32-bit mode because a 64-byte cache line contains 16 pointers
> > instead of only 8.
> 
> This should still work, unless there are libraries you use that are not
> multilib.
> 
> > [[snip]]
> > 
> > > Finally, if you would prefer this not happen now, is there a time when
> > > you would further down the road? Whats the critera/goalpost/cutoff?
> > 
> > One year after Red Hat Enterprise Linux version 7 reaches end-of-support.
> > It would be handy for Fedora to have 32-bit *-devel packages until then.
> 
> We will still have 32bit devel packages in the x86_64 repos after this
> change. It doesn't affect multilib at all. It only stops building and
> publishing the 'pure' i386 repos on the mirror network.
> 
> I don't think we can drop multilib until at least steam/wine are ready
> for it at least.

Being able to build 32bit is useful during development too. It is much
easier to expose some memory issues. For example fuzzing some
applications is best done against the 32bit variant to reduce the
memory search space. So imho making sure Fedora is a good developer
distro makes sure multilib gcc -m32 keeps working. So you need at least
the foobar-devel.i686 libraries in your x86_64 repository.

Could you explain a bit more how this (keeps) working? I think my
mental model of how Fedora repositories work in the case of multilib
devel packages is a bit flawed. At first I assumed that this suggestion
would kill that. Because I was under the impression that it worked by
having the i386 repository be part of the x86_64 package repository. So
removing the i686 repositories would mean removing all 32bit packages
from x86_64. But from your explanation above it seems that is not how
it works. So if there are no i686 repositories, then where/how do the
32bit multilib packages come from?

I think I am mixing up buildroots, koji targets, repositories in my
mind.

For example I maintain two packages that provide 32bit multilib
binaries. They are produced by the i686 koji target. Then elfutils-
devel.i686 (to create 32bit multilib binaries) and valgrind.i686 (to
run 32bit multilib binaries) are provided to the user because the i686
repository is part of the x86_64 repository (if I understand things
correctly). Will this just keep working, or will I have to make changes
to my packages to keep this working after the disappearance of the i686
repository?

Thanks,

Mark
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: strange failures with gcc-9.0.1-0.11.fc31.x86_64

[Mark Wielaard]

Hi,

On Thu, 2019-03-28 at 14:28 +, Zbigniew Jędrzejewski-Szmek wrote:
> On Thu, Mar 28, 2019 at 02:14:31PM +0100, Jakub Jelinek wrote:
> > On Thu, Mar 28, 2019 at 08:52:18AM +, Zbigniew Jędrzejewski-Szmek wrote:
> > > On Wed, Mar 27, 2019 at 01:55:44PM +, Zbigniew Jędrzejewski-Szmek 
> > > wrote:
> > > > I'm trying to compile systemd in koji and mock, and I'm getting 
> > > > suspicious
> > > > crashes...
> > > > 
> > > > $ valgrind x86_64-redhat-linux-gnu/test-terminal-util
> > > > /* test_default_term_for_tty */
> > > > ...
> > > > /* test_read_one_char */
> > > > ==21== Invalid read of size 4
> > > > ==21==at 0x48C09EC: fputs (in /usr/lib64/libc-2.29.9000.so)
> > > > ==21==by 0x109301: UnknownInlinedFun (test-terminal-util.c:43)
> > > > ==21==by 0x109301: main (test-terminal-util.c:80)
> > > > ==21==  Address 0x0 is not stack'd, malloc'd or (recently) free'd
> > > > ==21== 
> > > > ==21== 
> > > > ==21== Process terminating with default action of signal 11 (SIGSEGV)
> > > > 
> > > > The problem is at this line, there is just a call to (a function which
> > > > transitively calls) mkostemp(). It seems like the inlining is somehow
> > > > going wrong.
> > > 
> > > It turns out that our test case was wrong. I was confused because the
> > > inlining causes the backtrace to report an unrelated spot.
> > 
> > So do you still need anything from me to debug?
> 
> Thanks. I need some advice mostly. There's still the question of bogus
> backtrace returned by valgrind. Is this a valgrind issue or the debug
> data produced by gdb or something else? If we cannot rely on
> backtraces with LTO, this would be a big drawback.

The above backtrace is produced by valgrind. The addresses should be
correct, but as "UnknownInlinedFun" shows it has some trouble resolving
the associated function/symbol names.

I don't know if LTO makes that valgrind bug worse.

If gdb works then you can also use gdb and valgrind together:
https://tromey.com/blog/?p=731

http://valgrind.org/docs/manual/manual-core-adv.html#manual-core-adv.gdbserver

gdb probably can produce a better backtrace than valgrind.
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Packaging HTML/Javascript "application" for visualizing some json data

[Mark Wielaard]

On Sat, Mar 23, 2019 at 07:07:53AM -0700, John Reiser wrote:
> The output  dhat.out.3759  should be created in the current directory
> or at the destination specified by valgrind's usual  --xml-*=
> command-line parameters.  This is no different from any other output
> generated by valgrind.

Yes, that makes sense.

> The user will enter the URL to the browser
> app's "load" command as  file://  or as the result of a
> "browse" dialog, much like File > Open.

Right, where we would like to print out file:///dh_view.html at
the end of the dhat run to tell users how to load the viewer.

> The html/css/js "application" (4 files dh_view.html, dh_view.css,
> dh_view.js and dh_test.js) belongs in a directory such as
>  /usr/libexec/dhat .

OK, so given upstream uses autotools just install in something like
$(libexecdir)/valgrind/dhat/ ?

Thanks,

Mark
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Packaging HTML/Javascript "application" for visualizing some json data

[Mark Wielaard]

Hi,

The next release of valgrind (3.15.0) will have an updated dhat tool
which creates a json output file. To make it easier to use the data it
comes with a small html/css/js "application" that makes it easy to
sort/visualize the data.

This html/css/js application is self-contained, it doesn't use any
other javascript code/libraries. It consists of 4 files:
dh_view.html, dh_view.css, dh_view.js and dh_test.js.

You can see all files at:
https://gnu.wildebeest.org/~mark/dh_view/

You would use it something like:
$ valgrind --tool=dhat /bin/sort < /usr/share/dict/words > /dev/null

Which generates a json file dhat.out.3759 (as attached).
Which you can then "load" into the dh_view.html app to see various
different representations of the data collected.

You can try it out on the attached file at:
https://gnu.wildebeest.org/~mark/dh_view/dh_view.html
By clicking "Load" and giving it a copy of the attache dhat.out.3759
file.

sort isn't a very interesting example since it doesn't have a very
complicated allocation pattern, but it should give you an idea of what
the visualization does.

But it isn't really meant as "web app", the idea is that the user runs
this locally in their own browser on their locally generated dhat.out
json files.

I am not sure how to properly package this little app/viewer thingy as
part of valgrind. There are the Java Script and Web Assets guidelines,
but those are really for web applications, which this isn't.

Is there an example that I could follow?

How would users discover where/how to load the html/js application?

We could make the valgrind --tool=dhat output a location of the
html/css/js to load (and the generated json file), but which file
system location is proper for it?

Thanks,

Mark
{"dhatFileVersion":1
,"cmd":"/bin/sort"
,"pid":3759
,"mi":167361679,"ei":6481435849
,"aps":
 [{"tb":5,"tbk":1,"tli":129
  ,"mb":5,"mbk":1
  ,"gb":0,"gbk":0
  ,"fb":0,"fbk":0
  ,"rb":9,"wb":5
  ,"acc":[-4,2,3]
  ,"fs":[1,2,3,4,5]
  }
 ,{"tb":120,"tbk":1,"tli":6481261872
  ,"mb":120,"mbk":1
  ,"gb":120,"gbk":1
  ,"fb":0,"fbk":0
  ,"rb":440,"wb":112
  ,"acc":[-8,2,-8,28,-8,5,-8,3,-8,4,-24,3,-8,0,-48,3]
  ,"fs":[6,3,4,5]
  }
 ,{"tb":11,"tbk":1,"tli":6481261077
  ,"mb":11,"mbk":1
  ,"gb":11,"gbk":1
  ,"fb":0,"fbk":0
  ,"rb":803,"wb":11
  ,"acc":[66,-9,54,67]
  ,"fs":[7,8,3,4,5]
  }
 ,{"tb":3344,"tbk":12,"tli":5089283
  ,"mb":3344,"mbk":12
  ,"gb":3344,"gbk":12
  ,"fb":0,"fbk":0
  ,"rb":197299980,"wb":3192
  ,"fs":[9,10,3,4,5]
  }
 ,{"tb":132,"tbk":12,"tli":64812497572
  ,"mb":132,"mbk":12
  ,"gb":110,"gbk":10
  ,"fb":0,"fbk":0
  ,"rb":1342,"wb":132
  ,"acc":[101,-9,93,136]
  ,"fs":[7,11,5]
  }
 ,{"tb":11,"tbk":1,"tli":8795
  ,"mb":11,"mbk":1
  ,"gb":0,"gbk":0
  ,"fb":0,"fbk":0
  ,"rb":0,"wb":11
  ,"acc":[-11,1]
  ,"fs":[12,13,5]
  }
 ,{"tb":34,"tbk":1,"tli":6481235478
  ,"mb":34,"mbk":1
  ,"gb":34,"gbk":1
  ,"fb":0,"fbk":0
  ,"rb":24,"wb":34
  ,"acc":[-24,2,-10,1]
  ,"fs":[14,15,16]
  }
 ,{"tb":10,"tbk":1,"tli":6481234116
  ,"mb":10,"mbk":1
  ,"gb":10,"gbk":1
  ,"fb":0,"fbk":0
  ,"rb":0,"wb":10
  ,"acc":[-10,1]
  ,"fs":[7,17,18]
  }
 ,{"tb":11,"tbk":1,"tli":6481227879
  ,"mb":11,"mbk":1
  ,"gb":11,"gbk":1
  ,"fb":0,"fbk":0
  ,"rb":309,"wb":11
  ,"acc":[22,-6,20,21,-2,20,32]
  ,"fs":[7,19,20,21]
  }
 ,{"tb":11,"tbk":1,"tli":5498
  ,"mb":11,"mbk":1
  ,"gb":0,"gbk":0
  ,"fb":0,"fbk":0
  ,"rb":0,"wb":11
  ,"acc":[-11,1]
  ,"fs":[12,22,20,21]
  }
 ,{"tb":11,"tbk":1,"tli":35688
  ,"mb":11,"mbk":1
  ,"gb":0,"gbk":0
  ,"fb":0,"fbk":0
  ,"rb":17,"wb":12
  ,"acc":[-7,2,3,-2,2,3]
  ,"fs":[23,24,25,21]
  }
 ,{"tb":11,"tbk":1,"tli":6481222428
  ,"mb":11,"mbk":1
  ,"gb":11,"gbk":1
  ,"fb":0,"fbk":0
  ,"rb":310,"wb":11
  ,"acc":[22,-6,20,21,-2,20,33]
  ,"fs":[7,19,26,21]
  }
 ,{"tb":11,"tbk":1,"tli":6481222191
  ,"mb":11,"mbk":1
  ,"gb":11,"gbk":1
  ,"fb":0,"fbk":0
  ,"rb":0,"wb":11
  ,"acc":[-11,1]
  ,"fs":[12,22,26,21]
  }
 ,{"tb":11,"tbk":1,"tli":31314
  ,"mb":11,"mbk":1
  ,"gb":0,"gbk":0
  ,"fb":0,"fbk":0
  ,"rb":17,"wb":12
  ,"acc":[-7,2,3,-2,2,3]
  ,"fs":[23,24,27,21]
  }
 ,{"tb":48,"tbk":12,"tli":4572508
  ,"mb":48,"mbk":12
  ,"gb":48,"gbk":12
  ,"fb":48,"fbk":12
  ,"rb":1020,"wb":48
  ,"acc":[122,78,76,72]
  ,"fs":[23,28,21]
  }
 ,{"tb":32,"tbk":1,"tli":6481219674
  ,"mb":32,"mbk":1
  ,"gb":32,"gbk":1
  ,"fb":0,"fbk":0
  ,"rb":1184,"wb":32
  ,"acc":[-8,98,-8,2,-8,50,-8,2]
  ,"fs":[29,30,31,32,33,21]
  }
 ,{"tb":208,"tbk":1,"tli":6481218337
  ,"mb":208,"mbk":1
  ,"gb":208,"gbk":1
  ,"fb":0,"fbk":0
  ,"rb":776,"wb":96
  ,"acc":[-8,50,-12,1,-4,0,-16,1,-8,49,-44,1,-4,0,-8,1,-104,0]
  ,"fs":[34,35,36,37,30,31,32,33,21]
  }
 ,{"tb":208,"tbk":1,"tli":6481217501
  ,"mb":208,"mbk":1
  ,"gb":208,"gbk":1
  ,"fb":0,"fbk":0
  ,"rb":392,"wb":96
  ,"acc":[-8,26,-12,1,-4,0,-16,1,-8,25,-44,1,-4,0,-8,1,-104,0]
  ,"fs":[34,35,38,39,30,31,32,33,21]
  }
 ,{"tb":8,"tbk":1,"tli":3956
  ,"mb":8,"mbk":1
  ,"gb":0,"gbk":0
  ,"fb":0,"fbk":0
  ,"rb":0,"wb":0
  ,"acc":[-8,0]
  ,"fs":[23,40,41]
  }
 ,{"tb":128,"tbk":1,"tli":6

Re: rpmbuild: File listed twice (but only build-ids)

[Mark Wielaard]

On Tue, 2019-01-15 at 10:40 -0600, Richard Shaw wrote:
> Ooooaaa...
> 
> So this has been around since at least 2017 and there's no fix?

Much longer than that. At least since 2012, probably earlier.
%exclude is discouraged, which is why it doesn't seem to have higher
priority. See also:
https://bugzilla.redhat.com/show_bug.cgi?id=878863
(Although that bug is a little confusing since it seems to mix up two
issues with exclude files and build-id symlinks.)

It is a variant of:
https://github.com/rpm-software-management/rpm/issues/284
The fix for that excludes debug files for files which were excluded in
the non-debug package.

A similar fix should be done for generating the build-ids only for
binaries not in the pkg->fileExcludeList. Currently the
generateBuildIDs () function in build/files.c only gets the full
package file list. It probably should also get the fileExcludeList and
only generate the build-id links for files not on that list.

The interaction between the different package lists is a little
confusing though.

It is probably best to ask for guidance on the upstream mailinglist
rpm-ma...@lists.rpm.org (CCed).

> Is there an option to make it a warning again instead of an error?

I don't think without disabling build-ids completely.

Cheers,

Mark
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Random *** stack smashing detected *** message

[Mark Wielaard]

On Wed, 2018-09-05 at 18:04 +0100, Richard W.M. Jones wrote:
> On Wed, Sep 05, 2018 at 06:40:47PM +0200, Mark Wielaard wrote:
> > We don't know the exact release version, but given the build-id
> > [0aea4b30d53d7cc6386f1773a8dc8972793def1a] we should be able to
> > match it against an older glibc package.
> 
> Here are all the versions of glibc installed on that machine as far
> back as the DNF logs go (which is only a couple of months
> unfortunately):
> 
> 2.26.9000-41.fc28
> 2.26.9000-48.fc28
> 2.27.9000-28.fc29
> 2.27.9000-35.fc29
> 2.28-6.fc29
> 2.28.9000-4.fc30
> 2.28-9.fc29
> 
> Of those only 2.27.9000-35.fc29 contains the build ID.
> 
> That's the glibc-debuginfo that I already have installed (I realize
> now why my previous email was wrong - the file is called
> /usr/lib/debug/.build-id/0a/ea4b30d53d7cc6386f1773a8dc8972793def1a).
> 
> However it still can't find a symbol matching the address, so I guess
> we're out of luck.

It might just be that the stack got corrupted, so the addresses don't
really make sense. Note however that libc.so is a shared library that
may be mapped everywhere. If you look at the partial backtrace that eu-
stack generated you see:

#0  0x7f13c71ce46f - /usr/lib64/libc-2.27.9000.so (deleted)
[0aea4b30d53d7cc6386f1773a8dc8972793def1a]@0x7f13c7197000+0x3746f
#1  0x7f13c72a4435 - 1 - /usr/lib64/libc-2.27.9000.so (deleted)
[0aea4b30d53d7cc6386f1773a8dc8972793def1a]@0x7f13c7197000+0x10d434

Which says the file with build-id 0aea... was mapped at 0x7f13c7197000
so the offsets from the start of the mapping are 0x3746f and 0x10d434.

Normally you would then try to match these against the .symtab in the
.debug file. But glibc is special and keeps the symtab in the main
file...

So if you really want to hunt this down you might have to get the main
glibc 2.27.9000-35.fc29 rpm and rpm2cpio it somewhere. Then try to
extract the symbol table from that .../usr/lib64/libc-2.27.9000.so (eu-
readelf -s).

Or since you have the debugfile already try searching the .debuginfo
with eu-readelf --debug-dump=info and look for a low_pc with 03746[a-f] 
to see if there is a construct that matches.

Sorry this is all a bit messy (and possibly unresolvable if the stack
was just corrupted).

Cheers,

Mark
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Random *** stack smashing detected *** message

[Mark Wielaard]

On Wed, Sep 05, 2018 at 04:30:59PM +0100, Richard W.M. Jones wrote:
> On Wed, Sep 05, 2018 at 05:23:45PM +0200, Mark Wielaard wrote:
> > On Wed, 2018-09-05 at 13:59 +0100, Richard W.M. Jones wrote:
> > > On Wed, Sep 05, 2018 at 02:27:44PM +0200, Florian Weimer wrote:
> > > > But which one?  For 2.28-9.fc29 or 2.27.9000-35.fc29?  If GDB can't
> > > > find the build ID, I'd suggest try the other version as well.
> > > 
> > > Oh I see, good point.  I only had debuginfo installed for the
> > > new version.
> > > 
> > > I've now installed every debuginfo related to the coredump,
> > > and the old glibc debuginfo, but still:
> > > 
> > > Thread 1 (Thread 0x7f13c7b7d740 (LWP 31373)):
> > > #0  0x7f13c71ce46f in ?? ()
> > > #1  0x in ?? ()
> > > 
> > > So, I don't know ...
> > 
> > You could try eu-stack, with -l it should print out a memory map which
> > might give you a hint about the modules/build-ids.
> > $ eu-stack -v -l -b -m --core core.file
> 
> It gives an odd error (see attached).  Is that a bug in eu-stack?

It just says it also cannot unwind the stack, the stack might
just be corrupt, or because of the missing libraries it might not
have enough information or a wrong view of what is mapped where.

But we do now know that the process is bash-4.4.23-2 running against
the old, no longer installed, libc-2.27.9000.so.

We don't know the exact release version, but given the build-id
[0aea4b30d53d7cc6386f1773a8dc8972793def1a] we should be able to
match it against an older glibc package.

Cheers,

Mark
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Random *** stack smashing detected *** message

[Mark Wielaard]

On Wed, 2018-09-05 at 13:59 +0100, Richard W.M. Jones wrote:
> On Wed, Sep 05, 2018 at 02:27:44PM +0200, Florian Weimer wrote:
> > But which one?  For 2.28-9.fc29 or 2.27.9000-35.fc29?  If GDB can't
> > find the build ID, I'd suggest try the other version as well.
> 
> Oh I see, good point.  I only had debuginfo installed for the
> new version.
> 
> I've now installed every debuginfo related to the coredump,
> and the old glibc debuginfo, but still:
> 
> Thread 1 (Thread 0x7f13c7b7d740 (LWP 31373)):
> #0  0x7f13c71ce46f in ?? ()
> #1  0x in ?? ()
> 
> So, I don't know ...

You could try eu-stack, with -l it should print out a memory map which
might give you a hint about the modules/build-ids.
$ eu-stack -v -l -b -m --core core.file

Or try eu-unstrip -n --core core.file

Which should do the same and list what it knows about the
files/modules/build-ids in the core file.

Cheers,

Mark
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: %{valgrind_arches}

[Mark Wielaard]

On Sun, 2018-08-05 at 15:55 -0700, Samuel Sieb wrote:
> On 08/05/2018 01:13 PM, Florian Weimer wrote:
> > There already is such a macro, %{valgrind_arches}, but it may not 
> > accurately reflect the suitability of the run-time behavior of valgrind 
> > on a particular architecture.  For example, the undefinedness tracking 
> > might not be sufficiently accurate for the testsuite of a specific 
> > package, so running the testsuite under valgrind gives false positives.
> 
> So the original post is only to be used for specific exceptional cases?

Yes, Florian was just being very thorough.
We noticed packages disabled valgrind support on different
architectures for 2 different reasons.

It was disabled based on which architectures the package thought
valgrind was available for. If you just need to enable/disable valgrind
support because of this then please just use the %{valgrind_arches}
macro instead of hardcoding an architecture list in your package. The
macro will be updated whenever valgrind gets support for a new
architecture (or if an architecture is not longer completely supported,
it will be removed from this macro).

Secondly there might be a bug in valgrind on a particular architecture
that is triggered by something specific in the package. In that case
please use the construct Florian suggested to enable support based on
the %{valgrind_arches} macro with some package specific exception for
that particular architecture. And please file a bug against valgrind,
so it might get fixed.

Cheers,

Mark
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/message/QLI3DS7ACNRL4ZB5XTA5KO4MICKISTPB/

Re: %{valgrind_arches}

[Mark Wielaard]

On Sun, 2018-08-05 at 22:13 +0200, Florian Weimer wrote:
> On 08/05/2018 09:48 PM, Samuel Sieb wrote:
> > What am I missing here?  Why can't this be put in RPM macros?  Then when 
> > the situation changes in the future, there's only one place to change.
> 
> There already is such a macro, %{valgrind_arches}, but it may not 
> accurately reflect the suitability of the run-time behavior of valgrind 
> on a particular architecture.  For example, the undefinedness tracking 
> might not be sufficiently accurate for the testsuite of a specific 
> package, so running the testsuite under valgrind gives false positives.

Right. %{valgrind_arches} is the set of arches that valgrind supports.
So you should be able to use it to select whether or not to add
valgrind support to your package or run your package check testsuite
under valgrind. But there can always be bugs that are specific to a
specific architecture, package or testsuite. If there is such a bug and
you have to workaround running your package testsuite against valgrind
on some specific architecture, then please do report a bug against
valgrind. No guarantees it can be fixed quickly. But we can only fix
issues we know about.

Thanks,

Mark
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/message/W6WXAMWASFOM5Y2BEPKHMERJA4QLHHVU/

Re: What to BuildRequire for libstdc++.so __cxa_demangle?

[Mark Wielaard]

On Sat, 2018-07-21 at 18:52 +0200, Mark Wielaard wrote:
> The elfutils tools can demangle C++ symbols through the standard
> _cxa_demangle interface. The elfutils tools are written in C and
> so simply link with -lstdc++ to get access to __cxa_demangle.
> 
> There is a BuildRequires: libstdc++-devel in the elfutils.spec.
> But it looks like that isn't enough anymore to pull in libstdc++.so
> to build against.
> 
> It looks like that is provided through a symlink in gcc-c++.
> Should the elfutils.spec just BuildRequires: gcc-c++ instead of
> libstdc++-devel? Or is this a packaging/requires issue somewhere
> else?

Just in case anybody else is wondering what the correct answer is.
Whether libstdc++.so is provided by by gcc-c++ or libstdc++-devel is
architecture dependent. libstdc++-devel is there for multilib reasons.
You should just depend on gcc-c++ if you need to link against
libstdc++.so. Even if you don't use g++ to link against it. gcc-c++
requires libstdc++-devel.

Cheers,

Mark
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/message/ZK47BQOLXNLPNRYU6YTTOB246FXWEQK6/

Re: Installed (but unpackaged) file(s) found on s390x and armv7hl: .s390x.debug.#dwz#

[Mark Wielaard]

On Wed, 2018-07-25 at 21:04 +0300, Pavel Alexeev wrote:
> On 07/23/2018 12:36 PM, Dan Horák wrote:
> > On Mon, 23 Jul 2018 10:43:43 +0200
> > Mark Wielaard  wrote:
> > 
> > > On Sun, Jul 22, 2018 at 10:52:38PM +0300, Pavel Alexeev wrote:
> > > > Hello.
> > > > 
> > > > I try build new version of perdition package.
> > > > 
> > > > It build fine
> > > > (https://koji.fedoraproject.org/koji/taskinfo?taskID=28526416)
> > > > on
> > > > all architectures except armv7hl and s390x. On that I got
> > > > (https://kojipkgs.fedoraproject.org//work/tasks/6424/28526424/b
> > > > uild.log):
> > > > 
> > > > error: Installed (but unpackaged) file(s) found:
> > > > /usr/lib/debug/usr/sbin/perdition.imap4-2.2-
> > > > 1.fc29.s390x.debug.#dwz#.sWwnyG
> > > > /usr/lib/debug/usr/sbin/perdition.imap4s-2.2-
> > > > 1.fc29.s390x.debug.#dwz#.eE9BPY
> > > > /usr/lib/debug/usr/sbin/perdition.imaps-2.2-
> > > > 1.fc29.s390x.debug.#dwz#.WRTN7g
> > > > /usr/lib/debug/usr/sbin/perdition.managesieve-2.2-
> > > > 1.fc29.s390x.debug.#dwz#.GWCloz
> > > > /usr/lib/debug/usr/sbin/perdition.pop3-2.2-
> > > > 1.fc29.s390x.debug.#dwz#.
> > > > 2Sm2W5 /usr/lib/debug/usr/sbin/perdition.pop3s-2.2-
> > > > 1.fc29.s390x.debug.#dwz#.kvArfo
> > > > 
> > > > Could someone please help me solve that problem?
> > > 
> > > It looks like dwz crashed and left those temporary files behind.
> > > Strangely there are no indication in the log files that dwz
> > > crashed.
> > > But there is an rm -f statement in the log right before the
> > > find-debuginfo.sh/dwz invocation that does seem to touch those
> > > files.
> > > I cannot explain where that comes from. It must be somewhere at
> > > the
> > > end of the %install phase, but there is nothing in the .spec file
> > > that hints at where it is coming from.
> > > 
> > > It might be necessary to run on a real s390x or armv7vhl machine
> > > to track down what is going on.
> > 
> > so I can reproduce that locally on my rawhide s390x guest
> > 
> > Mark, I'll give you the machine info thru other channels.
> 
> Sorry, is there any progress?

Sorry, I did sent an update, but it apparently didn't go to the list
for some reason. See attached.

Unfortunately some other things came up, so I couldn't immediately try
to look deeper. And I managed to loose the files that helped me
replicate the issue.

Now trying to rebuild the package I suddenly get these errors:

ssl.c: In function '__perdition_verify_callback':
ssl.c:243:35: error: dereferencing pointer to incomplete type
'X509_STORE_CTX' {aka 'struct x509_store_ctx_st'}
  if (__perdition_verify_result(ctx->error, cert)
   ^~
ssl.c: In function '__perdition_ssl_check_common_name':
ssl.c:714:42: error: dereferencing pointer to incomplete type
'X509_NAME_ENTRY' {aka 'struct X509_name_entry_st'}
   if (!__perdition_ssl_compare_key(key, e->value->data,
  ^~
make[3]: *** [Makefile:643: ssl.o] Error 1

>  Should I fill bug for that? Against what component?

It really looks like a bug in dwz, so please file a bug against that.
I think a workaround for now would be to add %undefine
_find_debuginfo_dwz_opts to your spec. But I haven't been able to test
because of the above error.

Cheers,

Mark--- Begin Message ---
On Mon, Jul 23, 2018 at 11:36:14AM +0200, Dan Horák wrote:
> On Mon, 23 Jul 2018 10:43:43 +0200
> Mark Wielaard  wrote:
> > On Sun, Jul 22, 2018 at 10:52:38PM +0300, Pavel Alexeev wrote:
> > > all architectures except armv7hl and s390x. On that I got
> > > (https://kojipkgs.fedoraproject.org//work/tasks/6424/28526424/build.log):
> > > 
> > > error: Installed (but unpackaged) file(s) found:
> > > /usr/lib/debug/usr/sbin/perdition.imap4-2.2-1.fc29.s390x.debug.#dwz#.sWwnyG
> > > /usr/lib/debug/usr/sbin/perdition.imap4s-2.2-1.fc29.s390x.debug.#dwz#.eE9BPY
> > > /usr/lib/debug/usr/sbin/perdition.imaps-2.2-1.fc29.s390x.debug.#dwz#.WRTN7g
> > > /usr/lib/debug/usr/sbin/perdition.managesieve-2.2-1.fc29.s390x.debug.#dwz#.GWCloz
> > > /usr/lib/debug/usr/sbin/perdition.pop3-2.2-1.fc29.s390x.debug.#dwz#.
> > > 2Sm2W5 
> > > /usr/lib/debug/usr/sbin/perdition.pop3s-2.2-1.fc29.s390x.debug.#dwz#.kvArfo
> > > 
> > > Could someone please help me solve that problem?
> > 
> > It looks like dwz

Re: Installed (but unpackaged) file(s) found on s390x and armv7hl: .s390x.debug.#dwz#

[Mark Wielaard]

On Sun, Jul 22, 2018 at 10:52:38PM +0300, Pavel Alexeev wrote:
> Hello.
> 
> I try build new version of perdition package.
> 
> It build fine (https://koji.fedoraproject.org/koji/taskinfo?taskID=28526416)
> on all architectures except armv7hl and s390x. On that I got
> (https://kojipkgs.fedoraproject.org//work/tasks/6424/28526424/build.log):
> 
> error: Installed (but unpackaged) file(s) found:
> /usr/lib/debug/usr/sbin/perdition.imap4-2.2-1.fc29.s390x.debug.#dwz#.sWwnyG
> /usr/lib/debug/usr/sbin/perdition.imap4s-2.2-1.fc29.s390x.debug.#dwz#.eE9BPY
> /usr/lib/debug/usr/sbin/perdition.imaps-2.2-1.fc29.s390x.debug.#dwz#.WRTN7g
> /usr/lib/debug/usr/sbin/perdition.managesieve-2.2-1.fc29.s390x.debug.#dwz#.GWCloz
> /usr/lib/debug/usr/sbin/perdition.pop3-2.2-1.fc29.s390x.debug.#dwz#.2Sm2W5
> /usr/lib/debug/usr/sbin/perdition.pop3s-2.2-1.fc29.s390x.debug.#dwz#.kvArfo
> 
> Could someone please help me solve that problem?

It looks like dwz crashed and left those temporary files behind.
Strangely there are no indication in the log files that dwz crashed.
But there is an rm -f statement in the log right before the
find-debuginfo.sh/dwz invocation that does seem to touch those files.
I cannot explain where that comes from. It must be somewhere at the end
of the %install phase, but there is nothing in the .spec file that hints
at where it is coming from.

It might be necessary to run on a real s390x or armv7vhl machine
to track down what is going on.

Cheers,

Mark
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/message/EPPEPSVKREOK6IUZNGXRKAU523API6KE/

What to BuildRequire for libstdc++.so __cxa_demangle?

[Mark Wielaard]

Hi,

The elfutils tools can demangle C++ symbols through the standard
_cxa_demangle interface. The elfutils tools are written in C and
so simply link with -lstdc++ to get access to __cxa_demangle.

There is a BuildRequires: libstdc++-devel in the elfutils.spec.
But it looks like that isn't enough anymore to pull in libstdc++.so
to build against.

It looks like that is provided through a symlink in gcc-c++.
Should the elfutils.spec just BuildRequires: gcc-c++ instead of
libstdc++-devel? Or is this a packaging/requires issue somewhere
else?

Thanks,

Mark
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/message/25AHUHKLSC3PQYZJWLPKGKN7NIGD3MDS/

Re: Improving the glibc32 situation

[Mark Wielaard]

Hi,

On Tue, Mar 13, 2018 at 03:31:40PM +0100, Florian Weimer wrote:
> On 03/13/2018 03:03 PM, Jakub Jelinek wrote:
> > The right fix would be to make koji deal with multilibs properly,
> > mock can handle that fine already, then glibc32 wouldn't be needed at all.
> 
> I think we requested these feature fifteen years ago, even before there was
> Koji.  I recently (two months ago) tried to bump the priority, but haven't
> heard anything back.  It's a pretty fringe use case, and probably needs
> major complications within Koji.
> 
> So we need to assume that we'll still need glibc32 going forward.
> 
> The big question, at this point, is what we as packages have to do to
> prevent leaking a rebuild into the compose.  As far as I understand it,
> every package rebuild will trigger this.  Build with --skip-tag and file a
> releng ticket to have it tagged correctly?

I sadly have no idea.

> > If that doesn't happen, we need glibc32, but only 0.01% of packages actually
> > need it, so e.g. forcing it into every buildroot would be a vaste of time.
> 
> It's actually a bit worse because since it's buildroot-only, we need to
> express build requires in such a way that they can be fulfilled by
> glibc-devel.i686.  Otherwise, people would not be able to rebuild these
> packages using mock.

Yeah, the current way to pull things in that works everywhere seems to
be to do use file based requires like /usr/lib/libc.so because otherwise
things break in koji.

I always assumed koji was just a mock running in "the cloud", but I assume
it is slightly more involved than that.

Cheers,

Mark
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org

Re: Improving the glibc32 situation

[Mark Wielaard]

On Fri, 2018-03-09 at 14:07 +0100, Florian Weimer wrote:
> Some x86_64 packages need a 32-bit glibc during build time.  Koji
> does not provide it.
> [...]
> Comments?  Suggestions?

Why don't we just make koji provide it?
That is how a normal 64bit Fedora install looks like. Those have the
32bit packages available. So it makes sense to me that the koji build
repo also provides it for consistency. Why isn't the koji buildroot
just like a normal install?

Cheers,

Mark
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org

Re: [HEADS UP] Replacing %post/%postun -p /sbin/ldconfig

[Mark Wielaard]

Hi Igor,

On Wed, 2018-02-14 at 21:59 +0100, Igor Gnatenko wrote:
> Your options:
> 
> * Speak up and tell package names I should not touch because … (you should
> complete this sentence).
> * Fix up packages and tell package names I should not touch because you did
> that already.
> * Tell package names you want to remove ldconfig scriptlets entirely instead
> of replacing them with %ldconfig_scriptlets and get fix **for free**.
> * Ignore this message and get fix **for free**.
> [...]
> elfutils aoliva fche jakub jankratochvil mjw pmachata roland

I saw you already fixed this one up, thanks.

Although for some reason I didn't get any notification about this
change.

I had to tweak it a little though so the spec could still be build
older RHEL or Fedora (I reuse the spec to build on RHEL and with SCL).
Maybe something like the following is better for people who have a spec
file they might reuse on systems that might not have this macro:

# Only the latest Fedora and EPEL have these scriptlets,
# older, or not up to date, Fedora and plain RHEL don't.
%if 0%{?ldconfig_scriptlets:1}
%ldconfig_scriptlets libs
%ldconfig_scriptlets libelf
%else
%post libs -p /sbin/ldconfig
%postun libs -p /sbin/ldconfig
%post libelf -p /sbin/ldconfig
%postun libelf -p /sbin/ldconfig
%endif

Cheers,

Mark
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org

Split valgrind-tools-devel from valgrind-devel package

[Mark Wielaard]

Hi,

With valgrind-3.13.0-15.fc28 the valgrind-devel package  only contains
the development headers needed for building valgrind aware applications.
So it only contains the stand alone headers valgrind.h, callgrind.h,
drd.h, helgrind.h and memcheck.h that have the client request macros
that give hints to the valgrind tools.

I build various packages locally that BuildRequire valgrind-devel
to make sure they only required these headers.

valgrind-tools-devel contains all development files, headers and
static libraries, to build valgrind tools. Currently there is no
package in Fedora that needs those files. Building "out of tree"
valgrind tools isn't really supported because the tools need to
be linked staticly and upstream doesn't guarantee API. So such
tools would have to be rebuild for every new valgrind release.

The new setup means that valgrind-devel is only 80K now with
valgrind-tools-devel containing all the big static libraries.

Please let me know if you use the valgrind-devel package and the
new setup gives you any trouble.

Thanks,

Mark
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org

Re: EPEL support in "master" branch (aka speeding up Fedora development)

[Mark Wielaard]

On Mon, 2018-01-22 at 08:38 +, Tomasz Kłoczko wrote:
> Effectively Fedora has not to much from relation with RH *between*
> major EL releases in form of straight contribution to Fedora constant
> changes. They may be sending back some fixes done for RHEL customers
> but that is all. However as we are talking about CVEs and other
> critical changes it is IMO more likely that after submitting such
> fixes to original source code maintainers most of those changes are
> coming to Fedora in form of next version source tar balls than
> patches.

I think this depends on the team(s) that are maintaining the Fedora and
RHEL packages, and whether they are the same or different people. For
the packages I maintain personally in both Fedora and RHEL I make sure
any update to the RHEL (or DTS) package, first goes upstream and then
into the Fedora package. Only after that will it hit the RHEL or DTS
package as a bug fix (through a simple merge, because the spec files
are kept in sync).

Cheers,

Mark
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org

Re: EPEL support in "master" branch (aka speeding up Fedora development)

[Mark Wielaard]

On Sat, Jan 20, 2018 at 12:27:38PM +0100, Igor Gnatenko wrote:
> Why I'm writing this? I want to hear from you if you think it would be
> good to prohibit (or advise, or whatever mechanism would work) usage if
> conditionals in (at least) master branch to allow us to develop features
> faster. Thoughts?  Suggestions?

Personally I try to keep my package spec files buildable as is on
Fedora, CentOS and for SCL (DTS devtoolset). To me this is really
useful because it means I can easily provide the latest and greatest
features of my package once fixes hit Fedora rawhide. You can then just
automatically provide them to users through copr for older Fedora/CentOS.
By having some small SCL macros you can then also provide them as
part of a software collection. If newer rpm spec features would be
backported (or maybe provided as SCL?) on CentOS 6 and 7 then I would
certainly use them.

Cheers,

Mark
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org

Re: Empty debugsourcefiles.list when building out of source

[Mark Wielaard]

On Thu, 2018-01-11 at 11:21 +0100, Patrick Monnerat wrote:
> I also encountered this problem before and I fixed it by respecting
> the 
> RPM_OPT_FLAGS. Before cmake, use:
> 
> export CFLAGS="${RPM_OPT_FLAGS}"
> export CXXFLAGS="${RPM_OPT_FLAGS}"
> 
> This did the trick.
> 
> Obviously the problem depends on a compiler flag, but I can't tell
> you which.

It means -g was missing. -g tells gcc to generate debuginfo. If there
isn't any debuginfo at all, then it is impossible to generate a sources
list.

Cheers,

Mark
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org

Re: F28 System Wide Change: Hardening Flags Updates for Fedora 28

[Mark Wielaard]

Hi Richard,

On Sat, 2018-01-06 at 11:27 +, Richard W.M. Jones wrote:
> I noticed as a side effect of compiling GCC for riscv64 that RISC-V's
> GCC doesn't support -fstack-clash-protection.  Do you know what is
> involved to add it?  From a naive point of view I don't understand
> why this feature depends on architecture at all.

Sorry, I don't know the precise details on the gcc side. I believe it
is architecture dependent because it depends on the exact calling
conventions and how specific call instructions that manipulate the
stack. gcc wants the stack probes to be as efficient as possible. So it
only explicitly lowers the stack and inserts probes if absolutely
necessary. That is also why there were originally subtle issues around
no-return functions (because their calling conventions are different
and so gcc couldn't rely on the stack pointer already being lowered
and/or arguments having been pushed on it already).

Cheers,

Mark
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org

Re: F28 System Wide Change: Hardening Flags Updates for Fedora 28

[Mark Wielaard]

On Thu, Jan 04, 2018 at 09:36:27PM -0800, John Reiser wrote:
> 2) The explicit write by the stack probe can mask a memcheck(valgrind)
>violation, at least until memcheck groks the probe.

That should not be true. The probe is done after the stack pointer is
lowered, so memcheck/valgrind knows that memory is used. If the probe
is done below the actual stack pointer that would be a bug in the gcc
-fstack-protector implementation. Please do report such issues when you
encounter them. There were indeed such bugs in the past on some
architectures (especially in combination with no-return functions),
but I believe they have all been fixed now.

Thanks,

Mark
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org

Re: gdb: No symbol table info available

[Mark Wielaard]

On Sun, 2017-12-10 at 14:10 +0100, Jan Kratochvil wrote:
> the debuginfo-install command installs only *-debuginfo and not the
> *-debugsource counterpart which is then missing for GDB.  That is now
> a DNF bug I guess.

It is indeed: https://bugzilla.redhat.com/show_bug.cgi?id=1494628

There is a patch, but the dnf maintainer doesn't seem to like it. The
next version of rpmbuild will generate a soft depends Recommends
between debuginfo and debugsource packages, which should pull in
debugsources automatically. But of course that needs a rebuild of every
package and won't help you in Fedora 27.

Cheers,

Mark
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org

Re: debug facilities

[Mark Wielaard]

On Fri, 2017-12-08 at 10:17 -0500, Przemek Klosowski wrote:
> I still don't quite understand why only some packages have the 
> debuginfo/debugsource split (I counted ~10k debugsources to ~30k 
> debuginfo). Is that an automatic process that just needs to happen
> over time, or do the maintainers need to hand-polish individual
> packages?

No, that is expected.

There is one debugsources package per source rpm.
And there is one debuginfo package per binary subpackage.

This is because the subpackages are build from the same sources, so
they cannot easily be split.

See also:
https://fedoraproject.org/wiki/Changes/SubpackageAndSourceDebuginfo
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org

Re: debug facilities

[Mark Wielaard]

On Mon, 2017-12-04 at 15:47 -0500, Przemek Klosowski wrote:
> The bottom line is that it's pretty tricky to figure this out, which is 
> a pity because easy debuggability is one of the important cultural 
> features of Fedora and FOSS. It's a regression: GDB used to point to 
> integrated .debuginfo packages, which was sufficient. Now, maybe GDB 
> should suggest installing the source RPM?

Sorry I only saw this now. I think this is simply a bug in dnf.
dnf debuginfo-install really should do the right thing by default:
https://bugzilla.redhat.com/show_bug.cgi?id=1494628

There is a patch, but the dnf maintainer doesn't seem to like it. The
next version of rpmbuild will generate a soft depends Recommends
between debuginfo and debugsource packages, which should pull in
debugsources automatically. But of course that needs a rebuild of every
package and won't help you in Fedora 27.

Cheers,

Mark
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org

Re: Debuginfo exclusion list?

[Mark Wielaard]

Hi David,

On Tue, 2017-11-07 at 11:23 +, David Howells wrote:
> Is it possible to specify from a specfile an list of files and/or
> directories
> in the installation output that should be ignored for the purposes of
> debuginfo extraction?  If not, can this be added?

Debuginfo extraction is only done for executable files. So you can use
something like the following trick (this is from the valgrind.spec):

%install
[...]
# We don't want debuginfo generated for the vgpreload libraries.
# Turn off execute bit so they aren't included in the debuginfo.list.
# We'll turn the execute bit on again in %%files.
chmod 644 $RPM_BUILD_ROOT%{_libdir}/valgrind/vgpreload*-%{valarch}-*so

%files
[...]
# Turn on executable bit again for vgpreload libraries.
# Was disabled in %%install to prevent debuginfo stripping.
%attr(0755,root,root) %{_libdir}/valgrind/vgpreload*-%{valarch}-*so

Hope that helps.

Cheers,

Mark
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org

Re: 'Unknown DWARF' messages (bz#1506949)

[Mark Wielaard]

Hi Antonio,

On Fri, 2017-10-27 at 21:03 +0200, Antonio Trande wrote:
> [...]
> dwz: ./usr/lib64/libpsb_base.so.3.5.0.debug: Unknown DWARF DW_OP_253
> dwz: ./usr/lib64/libpsb_util.so.3.5.0.debug: Unknown DWARF DW_OP_253
> dwz: Too few files for multifile optimization
> 
> What does mean? And must i do something about?

It means that dwz cannot handle the DWARF and that it doesn't compress
any of it.  It isn't fatal. But it makes your debuginfo packages much
bigger than they need to be. Please file a bug report against the dwz
that it needs to handle DW_OP_GNU_variable_value (253/0xfd) which gcc
emits since GCC7.

Thanks,

Mark 
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org