[jira] [Commented] (YETUS-286) The native build randomly fails with "failed to map segment from shared object"
[
https://issues.apache.org/jira/browse/YETUS-286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15102418#comment-15102418
]
Colin Patrick McCabe commented on YETUS-286:
I did some Googling and most of the people complaining about "failed to map
segment from shared object: Permission denied" are complaining about selinux.
Is selinux enabled in any of our Docker images? Of course, that begs the
question of why the build failure would be intermittent, rather than repeatable.
Another random theory I had is that some process is updating {{libxml2.so.2}},
etc. etc. while the build is in progress, potentially deleting or renaming the
old version and causing this failure. However, I thought this had been
addressed during Dockerization.
Examples of this failure:
https://builds.apache.org/job/PreCommit-HADOOP-Build/8414/artifact/patchprocess/patch-compile-root-jdk1.8.0_66.txt
https://builds.apache.org/job/PreCommit-HADOOP-Build/8414/artifact/patchprocess/patch-compile-root-jdk1.7.0_91.txt
Obviously this is really frustrating and until we can fix this, we're going to
burn a lot of jenkins time re-running native patches... :(
> The native build randomly fails with "failed to map segment from shared
> object"
> ---
>
> Key: YETUS-286
> URL: https://issues.apache.org/jira/browse/YETUS-286
> Project: Yetus
> Issue Type: Bug
> Components: Test Patch
>Affects Versions: 0.2.0
>Reporter: Colin Patrick McCabe
>Assignee: Sean Busbey
>Priority: Critical
>
> On Jenkins, the native build seems to randomly fail with "failed to map
> segment from shared object." For example:
> {code}
> [WARNING] /usr/bin/cmake: error while loading shared libraries: libxml2.so.2:
> failed to map segment from shared object: Permission denied
> {code}
> We have also seen this happen with {{librtmp.so.0}}, {{libcurl.so.4}}, and
> other native libraries which are dependencies of the thing being built. When
> this happens, the native build fails and returns a -1 even if there was no
> actual problem. This requires us to re-run Jenkins to get a correct result.
> The problem does not seem to reproduce locally.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YETUS-286) The native build randomly fails with "failed to map segment from shared object"
[
https://issues.apache.org/jira/browse/YETUS-286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15102433#comment-15102433
]
Colin Patrick McCabe commented on YETUS-286:
We now we have some C++ code in the tree whose build time is drastically
reduced by using a higher concurrency level. Typical maven builds don't peg
the CPU because maven spends a lot of its time downloading things from the
internet, writing files, running unit tests, and so on. gcc processes also
don't use a lot of memory when compared to maven (most of the time). So I'm
skeptical that running make -j 16 will be that harmful in practice. We could
certainly look into tuning this, but I don't see why it would be the root cause
of "failed to map segment from shared object" failures.
> The native build randomly fails with "failed to map segment from shared
> object"
> ---
>
> Key: YETUS-286
> URL: https://issues.apache.org/jira/browse/YETUS-286
> Project: Yetus
> Issue Type: Bug
> Components: Test Patch
>Affects Versions: 0.2.0
>Reporter: Colin Patrick McCabe
>Assignee: Sean Busbey
>Priority: Critical
>
> On Jenkins, the native build seems to intermittently fail with "failed to map
> segment from shared object." For example:
> {code}
> [WARNING] /usr/bin/cmake: error while loading shared libraries: libxml2.so.2:
> failed to map segment from shared object: Permission denied
> {code}
> We have also seen this happen with {{librtmp.so.0}}, {{libcurl.so.4}}, and
> other native libraries which are dependencies of the thing being built.
> Basically, there is some random chance any particular native dependency will
> get "failed to map segment." When this happens, the native build fails and
> returns a -1 even if there was no actual problem. This requires us to re-run
> Jenkins to get a correct result. The problem does not seem to reproduce
> locally.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YETUS-286) The native build randomly fails with "failed to map segment from shared object"
[
https://issues.apache.org/jira/browse/YETUS-286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15102425#comment-15102425
]
Allen Wittenauer commented on YETUS-286:
Hmm:
{code}
[INFO] Running make -j 16 VERBOSE=1
{code}
Why is it suddenly running so many make's at one time? That's going to pretty
much burn all of the nodes resources to the ground, especially when two builds
are running at once
> The native build randomly fails with "failed to map segment from shared
> object"
> ---
>
> Key: YETUS-286
> URL: https://issues.apache.org/jira/browse/YETUS-286
> Project: Yetus
> Issue Type: Bug
> Components: Test Patch
>Affects Versions: 0.2.0
>Reporter: Colin Patrick McCabe
>Assignee: Sean Busbey
>Priority: Critical
>
> On Jenkins, the native build seems to randomly fail with "failed to map
> segment from shared object." For example:
> {code}
> [WARNING] /usr/bin/cmake: error while loading shared libraries: libxml2.so.2:
> failed to map segment from shared object: Permission denied
> {code}
> We have also seen this happen with {{librtmp.so.0}}, {{libcurl.so.4}}, and
> other native libraries which are dependencies of the thing being built. When
> this happens, the native build fails and returns a -1 even if there was no
> actual problem. This requires us to re-run Jenkins to get a correct result.
> The problem does not seem to reproduce locally.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YETUS-286) The native build randomly fails with "failed to map segment from shared object"
[
https://issues.apache.org/jira/browse/YETUS-286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15102503#comment-15102503
]
Colin Patrick McCabe commented on YETUS-286:
We can definitely tune the concurrency level of make... or use cgroups as
[~andrew.wang] has suggested.
However, even assuming that we ran out of memory because of running too many
{{make}} processes at once, I don't understand how that would lead to a
{{failed to map segment from shared object: Permission denied}} error. This
corresponds to the POSIX error code {{EACESS}}. If we couldn't allocate
memory, I would expect to see {{Cannot allocate memory}}, which corresponds to
{{ENOMEM}}. Or I would expect to see the OOM killer send a SIGKILL, which
would lead to "Terminated" messages. Perhaps I am missing something, but I am
having a hard time seeing how running out of memory or CPU would cause
"Permission denied" errors.
> The native build randomly fails with "failed to map segment from shared
> object"
> ---
>
> Key: YETUS-286
> URL: https://issues.apache.org/jira/browse/YETUS-286
> Project: Yetus
> Issue Type: Bug
> Components: Test Patch
>Affects Versions: 0.2.0
>Reporter: Colin Patrick McCabe
>Assignee: Sean Busbey
>Priority: Critical
>
> On Jenkins, the native build seems to intermittently fail with "failed to map
> segment from shared object." For example:
> {code}
> [WARNING] /usr/bin/cmake: error while loading shared libraries: libxml2.so.2:
> failed to map segment from shared object: Permission denied
> {code}
> We have also seen this happen with {{librtmp.so.0}}, {{libcurl.so.4}}, and
> other native libraries which are dependencies of the thing being built.
> Basically, there is some random chance any particular native dependency will
> get "failed to map segment." When this happens, the native build fails and
> returns a -1 even if there was no actual problem. This requires us to re-run
> Jenkins to get a correct result. The problem does not seem to reproduce
> locally.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YETUS-286) The native build randomly fails with "failed to map segment from shared object"
[
https://issues.apache.org/jira/browse/YETUS-286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15102509#comment-15102509
]
Sean Busbey commented on YETUS-286:
---
the few complaints I can find about this online all appear to be based on
trying to make use of libraries that are in a place with noexec.
> The native build randomly fails with "failed to map segment from shared
> object"
> ---
>
> Key: YETUS-286
> URL: https://issues.apache.org/jira/browse/YETUS-286
> Project: Yetus
> Issue Type: Bug
> Components: Test Patch
>Affects Versions: 0.2.0
>Reporter: Colin Patrick McCabe
>Assignee: Sean Busbey
>Priority: Critical
>
> On Jenkins, the native build seems to intermittently fail with "failed to map
> segment from shared object." For example:
> {code}
> [WARNING] /usr/bin/cmake: error while loading shared libraries: libxml2.so.2:
> failed to map segment from shared object: Permission denied
> {code}
> We have also seen this happen with {{librtmp.so.0}}, {{libcurl.so.4}}, and
> other native libraries which are dependencies of the thing being built.
> Basically, there is some random chance any particular native dependency will
> get "failed to map segment." When this happens, the native build fails and
> returns a -1 even if there was no actual problem. This requires us to re-run
> Jenkins to get a correct result. The problem does not seem to reproduce
> locally.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YETUS-286) The native build randomly fails with "failed to map segment from shared object"
[
https://issues.apache.org/jira/browse/YETUS-286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15102507#comment-15102507
]
Colin Patrick McCabe commented on YETUS-286:
I'd also add that we've seen this in the past before we increased the
concurrency of {{make}} past 1. I don't think anyone ever definitively
identified the problem. I always assumed that it had something to do with
{{apt-get}} running in the background, but that was a pure guess, I admit.
> The native build randomly fails with "failed to map segment from shared
> object"
> ---
>
> Key: YETUS-286
> URL: https://issues.apache.org/jira/browse/YETUS-286
> Project: Yetus
> Issue Type: Bug
> Components: Test Patch
>Affects Versions: 0.2.0
>Reporter: Colin Patrick McCabe
>Assignee: Sean Busbey
>Priority: Critical
>
> On Jenkins, the native build seems to intermittently fail with "failed to map
> segment from shared object." For example:
> {code}
> [WARNING] /usr/bin/cmake: error while loading shared libraries: libxml2.so.2:
> failed to map segment from shared object: Permission denied
> {code}
> We have also seen this happen with {{librtmp.so.0}}, {{libcurl.so.4}}, and
> other native libraries which are dependencies of the thing being built.
> Basically, there is some random chance any particular native dependency will
> get "failed to map segment." When this happens, the native build fails and
> returns a -1 even if there was no actual problem. This requires us to re-run
> Jenkins to get a correct result. The problem does not seem to reproduce
> locally.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YETUS-286) The native build randomly fails with "failed to map segment from shared object"
[
https://issues.apache.org/jira/browse/YETUS-286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15102517#comment-15102517
]
Sean Busbey commented on YETUS-286:
---
looks like Mesos hit the same issue using docker on asf infra in MESOS-2233.
let's try their fix.
> The native build randomly fails with "failed to map segment from shared
> object"
> ---
>
> Key: YETUS-286
> URL: https://issues.apache.org/jira/browse/YETUS-286
> Project: Yetus
> Issue Type: Bug
> Components: Test Patch
>Affects Versions: 0.2.0
>Reporter: Colin Patrick McCabe
>Assignee: Sean Busbey
>Priority: Critical
>
> On Jenkins, the native build seems to intermittently fail with "failed to map
> segment from shared object." For example:
> {code}
> [WARNING] /usr/bin/cmake: error while loading shared libraries: libxml2.so.2:
> failed to map segment from shared object: Permission denied
> {code}
> We have also seen this happen with {{librtmp.so.0}}, {{libcurl.so.4}}, and
> other native libraries which are dependencies of the thing being built.
> Basically, there is some random chance any particular native dependency will
> get "failed to map segment." When this happens, the native build fails and
> returns a -1 even if there was no actual problem. This requires us to re-run
> Jenkins to get a correct result. The problem does not seem to reproduce
> locally.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YETUS-286) The native build randomly fails with "failed to map segment from shared object"
[
https://issues.apache.org/jira/browse/YETUS-286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15102529#comment-15102529
]
Sean Busbey commented on YETUS-286:
---
yes, exactly! how serendipitous!
> The native build randomly fails with "failed to map segment from shared
> object"
> ---
>
> Key: YETUS-286
> URL: https://issues.apache.org/jira/browse/YETUS-286
> Project: Yetus
> Issue Type: Bug
> Components: Test Patch
>Affects Versions: 0.2.0
>Reporter: Colin Patrick McCabe
>Assignee: Sean Busbey
>Priority: Critical
>
> On Jenkins, the native build seems to intermittently fail with "failed to map
> segment from shared object." For example:
> {code}
> [WARNING] /usr/bin/cmake: error while loading shared libraries: libxml2.so.2:
> failed to map segment from shared object: Permission denied
> {code}
> We have also seen this happen with {{librtmp.so.0}}, {{libcurl.so.4}}, and
> other native libraries which are dependencies of the thing being built.
> Basically, there is some random chance any particular native dependency will
> get "failed to map segment." When this happens, the native build fails and
> returns a -1 even if there was no actual problem. This requires us to re-run
> Jenkins to get a correct result. The problem does not seem to reproduce
> locally.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YETUS-286) The native build randomly fails with "failed to map segment from shared object"
[
https://issues.apache.org/jira/browse/YETUS-286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15102796#comment-15102796
]
Xiao Chen commented on YETUS-286:
-
FYI - the above build succeeded!
> The native build randomly fails with "failed to map segment from shared
> object"
> ---
>
> Key: YETUS-286
> URL: https://issues.apache.org/jira/browse/YETUS-286
> Project: Yetus
> Issue Type: Bug
> Components: Test Patch
>Affects Versions: 0.2.0
>Reporter: Colin Patrick McCabe
>Assignee: Sean Busbey
>Priority: Critical
>
> On Jenkins, the native build seems to intermittently fail with "failed to map
> segment from shared object." For example:
> {code}
> [WARNING] /usr/bin/cmake: error while loading shared libraries: libxml2.so.2:
> failed to map segment from shared object: Permission denied
> {code}
> We have also seen this happen with {{librtmp.so.0}}, {{libcurl.so.4}}, and
> other native libraries which are dependencies of the thing being built.
> Basically, there is some random chance any particular native dependency will
> get "failed to map segment." When this happens, the native build fails and
> returns a -1 even if there was no actual problem. This requires us to re-run
> Jenkins to get a correct result. The problem does not seem to reproduce
> locally.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YETUS-286) The native build randomly fails with "failed to map segment from shared object"
[
https://issues.apache.org/jira/browse/YETUS-286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15102637#comment-15102637
]
Xiao Chen commented on YETUS-286:
-
Thanks everyone for the work here!
I've been running into this in HADOOP-12715 twice consecutively this today. I
just kicked off another build with that same patch at
https://builds.apache.org/job/PreCommit-HADOOP-Build/8424/ . Hopefully this run
can help confirm.
> The native build randomly fails with "failed to map segment from shared
> object"
> ---
>
> Key: YETUS-286
> URL: https://issues.apache.org/jira/browse/YETUS-286
> Project: Yetus
> Issue Type: Bug
> Components: Test Patch
>Affects Versions: 0.2.0
>Reporter: Colin Patrick McCabe
>Assignee: Sean Busbey
>Priority: Critical
>
> On Jenkins, the native build seems to intermittently fail with "failed to map
> segment from shared object." For example:
> {code}
> [WARNING] /usr/bin/cmake: error while loading shared libraries: libxml2.so.2:
> failed to map segment from shared object: Permission denied
> {code}
> We have also seen this happen with {{librtmp.so.0}}, {{libcurl.so.4}}, and
> other native libraries which are dependencies of the thing being built.
> Basically, there is some random chance any particular native dependency will
> get "failed to map segment." When this happens, the native build fails and
> returns a -1 even if there was no actual problem. This requires us to re-run
> Jenkins to get a correct result. The problem does not seem to reproduce
> locally.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YETUS-286) The native build randomly fails with "failed to map segment from shared object"
[
https://issues.apache.org/jira/browse/YETUS-286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15102856#comment-15102856
]
Allen Wittenauer commented on YETUS-286:
I think the Linux kernel's ability to find new and inventive ways to be bad is
probably it's most interesting feature.
> The native build randomly fails with "failed to map segment from shared
> object"
> ---
>
> Key: YETUS-286
> URL: https://issues.apache.org/jira/browse/YETUS-286
> Project: Yetus
> Issue Type: Bug
> Components: Test Patch
>Affects Versions: 0.2.0
>Reporter: Colin Patrick McCabe
>Assignee: Sean Busbey
>Priority: Critical
>
> On Jenkins, the native build seems to intermittently fail with "failed to map
> segment from shared object." For example:
> {code}
> [WARNING] /usr/bin/cmake: error while loading shared libraries: libxml2.so.2:
> failed to map segment from shared object: Permission denied
> {code}
> We have also seen this happen with {{librtmp.so.0}}, {{libcurl.so.4}}, and
> other native libraries which are dependencies of the thing being built.
> Basically, there is some random chance any particular native dependency will
> get "failed to map segment." When this happens, the native build fails and
> returns a -1 even if there was no actual problem. This requires us to re-run
> Jenkins to get a correct result. The problem does not seem to reproduce
> locally.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YETUS-286) The native build randomly fails with "failed to map segment from shared object"
[
https://issues.apache.org/jira/browse/YETUS-286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15102896#comment-15102896
]
Sean Busbey commented on YETUS-286:
---
good to hear it worked. I'll leave this open until later next week, presuming
y'all are actively working on the native code?
> The native build randomly fails with "failed to map segment from shared
> object"
> ---
>
> Key: YETUS-286
> URL: https://issues.apache.org/jira/browse/YETUS-286
> Project: Yetus
> Issue Type: Bug
> Components: Test Patch
>Affects Versions: 0.2.0
>Reporter: Colin Patrick McCabe
>Assignee: Sean Busbey
>Priority: Critical
>
> On Jenkins, the native build seems to intermittently fail with "failed to map
> segment from shared object." For example:
> {code}
> [WARNING] /usr/bin/cmake: error while loading shared libraries: libxml2.so.2:
> failed to map segment from shared object: Permission denied
> {code}
> We have also seen this happen with {{librtmp.so.0}}, {{libcurl.so.4}}, and
> other native libraries which are dependencies of the thing being built.
> Basically, there is some random chance any particular native dependency will
> get "failed to map segment." When this happens, the native build fails and
> returns a -1 even if there was no actual problem. This requires us to re-run
> Jenkins to get a correct result. The problem does not seem to reproduce
> locally.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YETUS-286) The native build randomly fails with "failed to map segment from shared object"
[
https://issues.apache.org/jira/browse/YETUS-286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15102783#comment-15102783
]
Colin Patrick McCabe commented on YETUS-286:
Thanks, all. We'll keep an eye out... hopefully this is the last we'll see
this.
> The native build randomly fails with "failed to map segment from shared
> object"
> ---
>
> Key: YETUS-286
> URL: https://issues.apache.org/jira/browse/YETUS-286
> Project: Yetus
> Issue Type: Bug
> Components: Test Patch
>Affects Versions: 0.2.0
>Reporter: Colin Patrick McCabe
>Assignee: Sean Busbey
>Priority: Critical
>
> On Jenkins, the native build seems to intermittently fail with "failed to map
> segment from shared object." For example:
> {code}
> [WARNING] /usr/bin/cmake: error while loading shared libraries: libxml2.so.2:
> failed to map segment from shared object: Permission denied
> {code}
> We have also seen this happen with {{librtmp.so.0}}, {{libcurl.so.4}}, and
> other native libraries which are dependencies of the thing being built.
> Basically, there is some random chance any particular native dependency will
> get "failed to map segment." When this happens, the native build fails and
> returns a -1 even if there was no actual problem. This requires us to re-run
> Jenkins to get a correct result. The problem does not seem to reproduce
> locally.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
