On 10/3/12 5:22 PM, Honza Bambas wrote:
I just hit
some compilation error caused by some recent patch in conjunction with
my build config and mach has just failed with a python trace back:
There was no warning log as well. I was a bit hoping that mach would
help with this (tracking build error in parallel builds), but it
didn't. Would it possible to let mach track and show the actual error
in the log somehow to make it simpler to find?
There are two problems here:
1) Build errors are hard to identify with make. Parallel execution can
make them even harder to track down. Poor output from invoked processes
is also a problem.
2) mach prints an unnecessary trace back when processes exit with non-0
exit code.
As Ehsan said, #2 is being tracked in bug 795427.
#1 is a larger problem with no easy solution. Without significantly
altering how make/pymake work, about the best we can do is parse the
process output of the build and try to print something intelligent if
the build failed. Unfortunately, this is far from robust because
parallel make sometimes interleaves output from multiple processes on
stdout.
Anyway, we already have something similar that does log parsing: TBPL.
If someone were to port TBPL's build output parsing logic to mach
(actually the build logic mach uses is in python/mozbuild/), I think
that would be an excellent addition to the tree!
FWIW, mach uses a structured logging system under the hood [1]. So, once
you parse build output into a data structure, that's capable of being
written to a log file without losing its structured aspect. In theory,
once mach is used on the buildbot machines, instead of recording whole
log blobs and parsing every time they are requested, we could record the
individual structured events. This opens up new avenues for indexing and
querying build logs. Additionally, we could hook the log stream up to
metlog [2] to enable interesting scenarios such as streaming build logs
off of buildbot hosts in real-time. A brokering agent could convert
streamed build logs to websocket streams in real-time (using something
like Node.js perhaps) and now you have real-time viewing of logs via
TBPL. Cool! Of course, all this can be done for test output too.
In summary, mach and structured logging enable the build system and test
runners to produce a more rich data stream (one that doesn't need to be
parsed) and allow it to be consumed in new and novel ways. If you are in
position to enact change, I'd love to talk to you about realizing some
of the scenarios above.
[1]
https://github.com/mozilla/mozilla-central/blob/master/python/mozbuild/README.rst
[2] https://github.com/mozilla-services/metlog-py
_______________________________________________
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform
