Re: [vpp-dev] opensuse expert needed

[Damjan Marion via Lists.Fd.Io]

> On 31 Jan 2019, at 14:12, Marco Varlese  wrote:
> 
> Damjan,
> 
> On 1/31/19 12:41 PM, Damjan Marion wrote:
>> 
>> 
>>> On 31 Jan 2019, at 08:57, Marco Varlese >> > wrote:
>>> 
>>> Damjan,
>>> 
>>> On 1/30/19 3:29 PM, Damjan Marion via Lists.Fd.Io wrote:
 
 Folks,
 
 Anybody can help with understanding why opensuse jobs are failing with:
 
 *13:54:57*
 /w/workspace/vpp-verify-master-osleap15/build-root/rpmbuild/BUILD/vpp-19.04/src/plugins/nsim/nsim.c:620:1:
 fatal error: error writing to /tmp/ccJaw6Cw.s: Cannot allocate memory
 *13:54:57*  };
 *13:54:57*  ^
 *13:54:57* compilation terminated.
 
 
 *13:54:57* gcc-7: internal compiler error: Killed (program cc1)
 *13:54:57* Please submit a full bug report,
 *13:54:57* with preprocessed source if appropriate.
 *13:54:57* See  for instructions.
>>> This to me looks like an infrastructure issue.
>>> If it was a problem with the distro / setup it would have happened all
>>> the time whilst things were pretty stable over the past months...
>> 
>> Well, that line clearly asks for filling bug with opensuse.org
>> ...
> Sure... that's pretty standard I'd say.
> 
> Anyw... I did ask around to some compiler guys and that
> looks like an out-of-memory issue (e.g. no more memory left on the node
> where the compilation was running)...
> 
>> 
>>> 
 
 
 
 Also, how we can avoid in future that people need to deal with this kind
 of issues?
>>> I am not sure how to answer this question.
>>> 
 
 Personally I don't see lot of value in compiling VPP agains every single
 distro on the planet in each verify job,
 but i'm fine with that as long as things are stable and not PITA for
 people trying to get patch verified
>>> Are you suggesting to remove openSUSE builds from Jenkins for VPP?
>> 
>> That is not what I said. I just don't see lot of value of running each
>> verify job against 3 distros,
>> I believe one is enough, and I'm not suggesting which one is that.
>> 
>>> Beside, I see verify jobs are passing now. Has anything been done in the
>>> meantime or all started to work auto-magically?
>> 
>> No, centos job was failing after that. but then everything passed.
>> it is lottery, whole infra is fragile and more jobs we run makes it
>> harder to have winning lottery ticket.
> I think it's rather a matter of sizing the running jobs queue to the
> actual capacity we have in the back-end. I remember something similar
> happened during summer / end-of summer time where jobs were randomly
> failing due to infra issues.
> Perhaps, another approach (instead of reducing the number of distros)
> could be to have only couple of patches building at a time. I would
> imagine that could be driven by Jenkins jobs/queues/etc.
> In that way, we could try addressing the capacity issue without reducing
> distros support.

Fine with me if we have volunteers to drive that. At the moment I see many
people expecting that their jobs are part of infra, but I don't see any 
initiative
to make infra less fragile (with Paul who offered to look at this 2 days ago
being an exception).

-- 
Damjan

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#12095): https://lists.fd.io/g/vpp-dev/message/12095
Mute This Topic: https://lists.fd.io/mt/29594604/21656
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub  [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [vpp-dev] opensuse expert needed

[Marco Varlese]

Damjan,

On 1/31/19 12:41 PM, Damjan Marion wrote:
> 
> 
>> On 31 Jan 2019, at 08:57, Marco Varlese > > wrote:
>>
>> Damjan,
>>
>> On 1/30/19 3:29 PM, Damjan Marion via Lists.Fd.Io wrote:
>>>
>>> Folks,
>>>
>>> Anybody can help with understanding why opensuse jobs are failing with:
>>>
>>> *13:54:57*
>>> /w/workspace/vpp-verify-master-osleap15/build-root/rpmbuild/BUILD/vpp-19.04/src/plugins/nsim/nsim.c:620:1:
>>> fatal error: error writing to /tmp/ccJaw6Cw.s: Cannot allocate memory
>>> *13:54:57*  };
>>> *13:54:57*  ^
>>> *13:54:57* compilation terminated.
>>>
>>>
>>> *13:54:57* gcc-7: internal compiler error: Killed (program cc1)
>>> *13:54:57* Please submit a full bug report,
>>> *13:54:57* with preprocessed source if appropriate.
>>> *13:54:57* See  for instructions.
>> This to me looks like an infrastructure issue.
>> If it was a problem with the distro / setup it would have happened all
>> the time whilst things were pretty stable over the past months...
> 
> Well, that line clearly asks for filling bug with opensuse.org
> ...
Sure... that's pretty standard I'd say.

Anyw... I did ask around to some compiler guys and that
looks like an out-of-memory issue (e.g. no more memory left on the node
where the compilation was running)...

> 
>>
>>>
>>>
>>>
>>> Also, how we can avoid in future that people need to deal with this kind
>>> of issues?
>> I am not sure how to answer this question.
>>
>>>
>>> Personally I don't see lot of value in compiling VPP agains every single
>>> distro on the planet in each verify job,
>>> but i'm fine with that as long as things are stable and not PITA for
>>> people trying to get patch verified
>> Are you suggesting to remove openSUSE builds from Jenkins for VPP?
> 
> That is not what I said. I just don't see lot of value of running each
> verify job against 3 distros,
> I believe one is enough, and I'm not suggesting which one is that.
> 
>> Beside, I see verify jobs are passing now. Has anything been done in the
>> meantime or all started to work auto-magically?
> 
> No, centos job was failing after that. but then everything passed.
> it is lottery, whole infra is fragile and more jobs we run makes it
> harder to have winning lottery ticket.
I think it's rather a matter of sizing the running jobs queue to the
actual capacity we have in the back-end. I remember something similar
happened during summer / end-of summer time where jobs were randomly
failing due to infra issues.
Perhaps, another approach (instead of reducing the number of distros)
could be to have only couple of patches building at a time. I would
imagine that could be driven by Jenkins jobs/queues/etc.
In that way, we could try addressing the capacity issue without reducing
distros support.
> 
> -- 
> Damjan
> 
- Marco



signature.asc
Description: OpenPGP digital signature
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#12091): https://lists.fd.io/g/vpp-dev/message/12091
Mute This Topic: https://lists.fd.io/mt/29594604/21656
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub  [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [vpp-dev] opensuse expert needed

[Damjan Marion via Lists.Fd.Io]

> On 31 Jan 2019, at 08:57, Marco Varlese  wrote:
> 
> Damjan,
> 
> On 1/30/19 3:29 PM, Damjan Marion via Lists.Fd.Io wrote:
>> 
>> Folks,
>> 
>> Anybody can help with understanding why opensuse jobs are failing with:
>> 
>> *13:54:57* 
>> /w/workspace/vpp-verify-master-osleap15/build-root/rpmbuild/BUILD/vpp-19.04/src/plugins/nsim/nsim.c:620:1:
>>  fatal error: error writing to /tmp/ccJaw6Cw.s: Cannot allocate memory
>> *13:54:57*  };
>> *13:54:57*  ^
>> *13:54:57* compilation terminated.
>> 
>> 
>> *13:54:57* gcc-7: internal compiler error: Killed (program cc1)
>> *13:54:57* Please submit a full bug report,
>> *13:54:57* with preprocessed source if appropriate.
>> *13:54:57* See  for instructions.
> This to me looks like an infrastructure issue.
> If it was a problem with the distro / setup it would have happened all
> the time whilst things were pretty stable over the past months...

Well, that line clearly asks for filling bug with opensuse.org...

> 
>> 
>> 
>> 
>> Also, how we can avoid in future that people need to deal with this kind
>> of issues?
> I am not sure how to answer this question.
> 
>> 
>> Personally I don't see lot of value in compiling VPP agains every single
>> distro on the planet in each verify job,
>> but i'm fine with that as long as things are stable and not PITA for
>> people trying to get patch verified
> Are you suggesting to remove openSUSE builds from Jenkins for VPP?

That is not what I said. I just don't see lot of value of running each verify 
job against 3 distros,
I believe one is enough, and I'm not suggesting which one is that.

> Beside, I see verify jobs are passing now. Has anything been done in the
> meantime or all started to work auto-magically?

No, centos job was failing after that. but then everything passed.
it is lottery, whole infra is fragile and more jobs we run makes it harder to 
have winning lottery ticket.

-- 
Damjan

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#12089): https://lists.fd.io/g/vpp-dev/message/12089
Mute This Topic: https://lists.fd.io/mt/29594604/21656
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub  [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [vpp-dev] opensuse expert needed

[Marco Varlese]

Damjan,

On 1/30/19 3:29 PM, Damjan Marion via Lists.Fd.Io wrote:
> 
> Folks,
> 
> Anybody can help with understanding why opensuse jobs are failing with:
> 
> *13:54:57* 
> /w/workspace/vpp-verify-master-osleap15/build-root/rpmbuild/BUILD/vpp-19.04/src/plugins/nsim/nsim.c:620:1:
>  fatal error: error writing to /tmp/ccJaw6Cw.s: Cannot allocate memory
> *13:54:57*  };
> *13:54:57*  ^
> *13:54:57* compilation terminated.
> 
> 
> *13:54:57* gcc-7: internal compiler error: Killed (program cc1)
> *13:54:57* Please submit a full bug report,
> *13:54:57* with preprocessed source if appropriate.
> *13:54:57* See  for instructions.
This to me looks like an infrastructure issue.
If it was a problem with the distro / setup it would have happened all
the time whilst things were pretty stable over the past months...

> 
> 
> 
> Also, how we can avoid in future that people need to deal with this kind
> of issues?
I am not sure how to answer this question.

> 
> Personally I don't see lot of value in compiling VPP agains every single
> distro on the planet in each verify job,
> but i'm fine with that as long as things are stable and not PITA for
> people trying to get patch verified
Are you suggesting to remove openSUSE builds from Jenkins for VPP?

Beside, I see verify jobs are passing now. Has anything been done in the
meantime or all started to work auto-magically?

> 
> -- 
> Damjan
> 
> 
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> 
> View/Reply Online (#12062): https://lists.fd.io/g/vpp-dev/message/12062
> Mute This Topic: https://lists.fd.io/mt/29594604/675056
> Group Owner: vpp-dev+ow...@lists.fd.io
> Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub  [mvarl...@suse.de]
> -=-=-=-=-=-=-=-=-=-=-=-
> 

-- 
Marco Varlese, Architect Developer Technologies, SUSE Labs
SUSE LINUX GmbH | GF: Felix Imendörffer, Jane Smithard, Graham Norton
HRB 21284 (AG Nürnberg) Maxfeldstr. 5, D-90409, Nürnberg



signature.asc
Description: OpenPGP digital signature
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#12087): https://lists.fd.io/g/vpp-dev/message/12087
Mute This Topic: https://lists.fd.io/mt/29594604/21656
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub  [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [vpp-dev] opensuse expert needed

[Damjan Marion via Lists.Fd.Io]

> On 30 Jan 2019, at 16:26, Paul Vinciguerra  wrote:
> 
> I would agree that the error seems to indicate that the build-host has 
> insufficient /tmp space.
> 
> There is no doubt, that the build system, like the test system,  could use 
> some love.
> What is your feeling on moving the CI logic out of the makefile?

How is that going to help with not enough space in /tmp?

> 
> Do you care about the error, or do you care about the time you had to wait to 
> get the feedback?  For me, it's usually the latter.

later, i spent whole day trying to push one patch over the finish line

latest news: centos job is failing blindly and suse is passing after recheck. 
At least suse was polite enough to throw few lines out 

-- 
Damjan

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#12067): https://lists.fd.io/g/vpp-dev/message/12067
Mute This Topic: https://lists.fd.io/mt/29594604/21656
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub  [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [vpp-dev] opensuse expert needed

[Benoit Ganne (bganne) via Lists.Fd.Io]

> Anybody can help with understanding why opensuse jobs are failing with:
> 13:54:57 /w/workspace/vpp-verify-master-osleap15/build-
> root/rpmbuild/BUILD/vpp-19.04/src/plugins/nsim/nsim.c:620:1: fatal error:
> error writing to /tmp/ccJaw6Cw.s: Cannot allocate memory

Could it be that /tmp is mounted as tmpfs and is too small?

Ben
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#12063): https://lists.fd.io/g/vpp-dev/message/12063
Mute This Topic: https://lists.fd.io/mt/29594604/21656
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub  [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-