Re: [OE-core] Yocto Project Status WW47’17
On 11/28/2017 03:49 PM, Robert Yang wrote: On 11/23/2017 06:20 PM, Richard Purdie wrote: On Mon, 2017-11-20 at 20:22 -0500, Randy MacLeod wrote: o Issues with 4.13.10 host kernels booting kvm x86 guests on Tumbleweed (Suse) and Fedora 26 (attempting to see if 4.13.12 helps) Robert, can you test Fedora 26. It would help to have a defect open with steps to reproduce or something about the typical workflow/ build time/ day of the week/ phase of the moon. We have some further data: a) The issue occurs on 4.13.12 rpurdie@tumbleweed:/tmp> cat /proc/version Linux version 4.13.12-1-vanilla (geeko@buildhost) (gcc version 7.2.1 20171020 [gcc-7-branch revision 253932] (SUSE Linux)) #1 SMP PREEMPT Wed Nov 8 11:21:09 UTC 2017 (9151c66) b) The hang usually occurs at the TIMER line in the kernel logs but can occur after booting into userspace around the udevd line, or occasionally later in the boot process. c) The similarity between this and the ppc bug I worked on make me strongly suspect qemu's timers are stopping firing and the guest is sitting in the idle loop. d) I do now have a way to brute force the hangs at will. The attached "runqemu-parallel.py" script runs 50 qemus in parallel. In around 45s I had 10 hung on the autobuilder. I can provide more info on using that script if its not obvious. It does assume my recent master changes to the qemuconf files so we don't need to run bitbake -e to run runqemu. This could well be the same kind of locking issue we saw on ppc. I'll continue to look into that. Hopefully this extra information will put us on a good track to resolving it now. It is continuing to break builds and stop patch merging. I modified the script to run on my host (I used qemux86, not qemux86-64, the later one is still in building) (Up to date Fedora 26, the kernel is 4.13.13): $ cat /proc/version Linux version 4.13.13-200.fc26.x86_64 (mockbu...@bkernel01.phx2.fedoraproject.org) (gcc version 7.2.1 20170915 (Red Hat 7.2.1-2) (GCC)) #1 SMP Wed Nov 15 15:46:36 UTC 2017 But didn't see any hangs after 10 minutes, all the qemus reached login. So maybe it had been fixed on 4.13.13 ? I tested qemux86-64, the boot became very slow (just like hang), no one booted after about 2 hours (still in booting). And it only happened when use qemux86-64 + kvm: # Works $ runqemu tmp/deploy/images/qemux86-64/ nographic snapshot # Doesn't work, like hang: $ runqemu tmp/deploy/images/qemux86-64/ nographic snapshot kvm // Robert // Robert Cheers, Richard -- ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
Re: [OE-core] Yocto Project Status WW47’17
On 11/23/2017 06:20 PM, Richard Purdie wrote: On Mon, 2017-11-20 at 20:22 -0500, Randy MacLeod wrote: o Issues with 4.13.10 host kernels booting kvm x86 guests on Tumbleweed (Suse) and Fedora 26 (attempting to see if 4.13.12 helps) Robert, can you test Fedora 26. It would help to have a defect open with steps to reproduce or something about the typical workflow/ build time/ day of the week/ phase of the moon. We have some further data: a) The issue occurs on 4.13.12 rpurdie@tumbleweed:/tmp> cat /proc/version Linux version 4.13.12-1-vanilla (geeko@buildhost) (gcc version 7.2.1 20171020 [gcc-7-branch revision 253932] (SUSE Linux)) #1 SMP PREEMPT Wed Nov 8 11:21:09 UTC 2017 (9151c66) b) The hang usually occurs at the TIMER line in the kernel logs but can occur after booting into userspace around the udevd line, or occasionally later in the boot process. c) The similarity between this and the ppc bug I worked on make me strongly suspect qemu's timers are stopping firing and the guest is sitting in the idle loop. d) I do now have a way to brute force the hangs at will. The attached "runqemu-parallel.py" script runs 50 qemus in parallel. In around 45s I had 10 hung on the autobuilder. I can provide more info on using that script if its not obvious. It does assume my recent master changes to the qemuconf files so we don't need to run bitbake -e to run runqemu. This could well be the same kind of locking issue we saw on ppc. I'll continue to look into that. Hopefully this extra information will put us on a good track to resolving it now. It is continuing to break builds and stop patch merging. I modified the script to run on my host (I used qemux86, not qemux86-64, the later one is still in building) (Up to date Fedora 26, the kernel is 4.13.13): $ cat /proc/version Linux version 4.13.13-200.fc26.x86_64 (mockbu...@bkernel01.phx2.fedoraproject.org) (gcc version 7.2.1 20170915 (Red Hat 7.2.1-2) (GCC)) #1 SMP Wed Nov 15 15:46:36 UTC 2017 But didn't see any hangs after 10 minutes, all the qemus reached login. So maybe it had been fixed on 4.13.13 ? // Robert Cheers, Richard -- ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
Re: [OE-core] Yocto Project Status WW47’17
On Mon, 2017-11-20 at 20:22 -0500, Randy MacLeod wrote:
> > o Issues with 4.13.10 host kernels booting kvm x86 guests on
> > Tumbleweed (Suse) and Fedora 26 (attempting to see if 4.13.12
> > helps)
>
> Robert, can you test Fedora 26. It would help to have a defect open
> with steps to reproduce or
> something about the typical workflow/ build time/ day of the week/
> phase of the moon.
We have some further data:
a) The issue occurs on 4.13.12
rpurdie@tumbleweed:/tmp> cat /proc/version
Linux version 4.13.12-1-vanilla (geeko@buildhost) (gcc version 7.2.1 20171020
[gcc-7-branch revision 253932] (SUSE Linux)) #1 SMP PREEMPT Wed Nov 8 11:21:09
UTC 2017 (9151c66)
b) The hang usually occurs at the TIMER line in the kernel logs but can
occur after booting into userspace around the udevd line, or
occasionally later in the boot process.
c) The similarity between this and the ppc bug I worked on make me
strongly suspect qemu's timers are stopping firing and the guest is
sitting in the idle loop.
d) I do now have a way to brute force the hangs at will. The attached
"runqemu-parallel.py" script runs 50 qemus in parallel. In around 45s I
had 10 hung on the autobuilder. I can provide more info on using that
script if its not obvious. It does assume my recent master changes to
the qemuconf files so we don't need to run bitbake -e to run runqemu.
This could well be the same kind of locking issue we saw on ppc. I'll
continue to look into that.
Hopefully this extra information will put us on a good track to
resolving it now. It is continuing to break builds and stop patch
merging.
Cheers,
Richard
#!/usr/bin/env python3
import shutil
import time
import subprocess
import os
procs = list(range(1, 50))
logs = "/tmp/runqemu-log"
errlogs = "/tmp/zrunqemu-log"
logfds = {}
errlogfds = {}
processes = {}
image = "/media/build1/poky/build/tmp-musl/deploy/images/qemuppc/core-image-sato-qemuppc.qemuboot.conf"
kernel = "tmp-musl/deploy/images/qemuppc/vmlinux-qemuppc.bin"
machine = "qemuppc"
image = "/media/build1/poky/build/tmp-musl/deploy/images/qemux86-64/core-image-sato-qemux86-64.qemuboot.conf"
kernel = "tmp-musl/deploy/images/qemux86-64/bzImage-qemux86-64.bin"
machine = "qemux86-64"
env = os.environ.copy()
env['MACHINE'] = machine
def start(p):
logfds[p] = open(logs + ".%d" % p, "w")
errlogfds[p] = open(errlogs + ".%d" % p, "w")
#debugopts = "-d unimp,guest_errors,int,out_asm,op,op_opt,op_ind,mmu,pcall,cpu_reset,nochain"
#exec,cpu,in_asm,
debugopts = "-d unimp,guest_errors,int"
debugopts += " -D /tmp/qemu.%d -monitor pty" % p
processes[p] = subprocess.Popen(["runqemu",machine,"nographic", "snapshot", "kvm", kernel, image, "qemuparams=-pidfile /tmp/zzqemu.%d.pid %s" % (p, debugopts)], stdout=logfds[p], stderr=errlogfds[p], env=env)
print("Started %d" % p)
for p in procs:
start(p)
lastsize = {}
while procs:
sizes = {}
time.sleep(0.5)
for p in procs:
logfile = logs + ".%d" % p
pidfn = "/tmp/zzqemu.%d.pid" % p
with open(logfile, "r") as lf:
for l in lf:
if "login:" in l:
with open(pidfn) as pidfile:
pid = int(pidfile.readline().strip())
print("Killing %d with pid %d" % (p, pid))
os.kill(pid, 9)
procs.remove(p)
if p in lastsize:
del lastsize[p]
time.sleep(0.1)
logfds[p].close()
errlogfds[p].close()
start(p)
procs.append(p)
sizes[p] = os.stat(logfile).st_size
if p in lastsize and sizes[p]:
if not sizes[p] > lastsize[p]:
print("%d stalled?" % p)
lastsize = sizes
--
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core
Re: [OE-core] Yocto Project Status WW47’17
On 11/21/2017 07:09 PM, Richard Purdie wrote: On Mon, 2017-11-20 at 20:22 -0500, Randy MacLeod wrote: On 2017-11-20 10:36 AM, Jolley, Stephen K wrote: Current Dev Position: YP 2.5 Planning and M1 development Next Deadline: YP 2.5 M1 cut off of 12/4/17 SWAT team rotation: Juro -> Paul on Nov.17, 2017. SWAT team rotation: Paul -> Todor on Nov. 24, 2017. https://wiki.yoctoproject.org/wiki/Yocto_Build_Failure_Swat_Team Key Status/Updates: · There is no real change to the status from last week. We continue to suffer intermittent build failures and are continuing to attempt to debug these. · Currently open issues are: Some US-based people may be on holiday later this week so I'm offering help from the frozen Northland and more importantly from the team in Beijing. ;-) o qemuppc continues to demonstrate random hangs in boot in userspace Is we can create a defect for this and point / copy the wiki notes into it, that would help. https://wiki.yoctoproject.org/wiki/Qemuppc_Boot_Hangs I think I had asked Chi to see if he could reproduce this a week or two ago. When the lack of entropy problem was identified and fix, many people thought this hang went away as well. Chi can you read the wiki report and see if you can add anything to them? Good news is that the qemuppc issue has been identified as a bug in qemu ppc locking which breaks timer interrupt handling. I've posted on the qemu mailing list asking for help in verifying what I think is happening. I have a patch ready to merge which should address this one, just cleaning up my environment and doing some further stress testing. [There is a defect somewhere for this btw, I created the wiki as it was a better place to dump and update information as we learnt what it is/is not without having to follow a train of thought updating in the bugzilla]. o Issues with 4.13.10 host kernels booting kvm x86 guests on Tumbleweed (Suse) and Fedora 26 (attempting to see if 4.13.12 helps) Robert, can you test Fedora 26. It would help to have a defect open with steps to reproduce or something about the typical workflow/ build time/ day of the week/ phase of the moon. FWIW we have noticed that the choice of kernel timers seems to vary in the x86_64 boots but not with a pattern that matches the hangs. o nfs inode count for the sstate/dldir share appears to break periodically causing the disk monitor to halt the builds (bug 12267) Likely specific to the AB server so no plans to do anything for this bug. Agreed, this one is our infrastructure somehow :(. We have a workaround in -next for this at least. o a perf build race (bug 12302) I'll take a look to: - see if I can duplicate the bug on a fast build host - check upstream to see if the bug is known/fixed - see if I can spot a race in the build rules. Sounds good, thanks! o An ext filesystem creation/sizing issue (bug 12304) Saul, Are you around this week? Do you have any additional information before leaving for Thanksgiving? Jackie, Can you look at the code around the image creation and try to reproduce this one? Saul hasn't been able to reproduce. I've asked at the minimum we add better logging so if/when this happens again, we can debug it properly next time. I did also wonder about fuzzing the image size code, writing some code which puts in all possible input values and checks the sanity of the resulting image size. Its the kind of problem a computer can probably brute force. Anyone interested in trying that? I have interests on it, but I don't quite understand what did you mean. Currently, the image size is from "du -sh " * factor, about the bug 12304, it got 8192 which is the default value of core-image-minimal, so maybe something is wrong with "du -sh", the normal value is larger than 8192. // Robert Cheers, Richard -- ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
Re: [OE-core] Yocto Project Status WW47’17
On Tue, 2017-11-21 at 11:09 +, Richard Purdie wrote: > On Mon, 2017-11-20 at 20:22 -0500, Randy MacLeod wrote: > > On 2017-11-20 10:36 AM, Jolley, Stephen K wrote: > > > Current Dev Position: YP 2.5 Planning and M1 development > > > Next Deadline: YP 2.5 M1 cut off of 12/4/17 > > > > > > SWAT team rotation: Juro -> Paul on Nov.17, 2017. > > > SWAT team rotation: Paul -> Todor on Nov. 24, 2017. > > > https://wiki.yoctoproject.org/wiki/Yocto_Build_Failure_Swat_Team > > > > > > Key Status/Updates: > > > ·There is no real change to the status from last week. We > > > continue to suffer intermittent build failures and are continuing > > > to attempt to debug these. > > > ·Currently open issues are: > > > > > > Some US-based people may be on holiday later this week so I'm > > offering > > help from the frozen Northland and more importantly from the team > > in > > Beijing. ;-) > > > > > o qemuppc continues to demonstrate random hangs in boot in > > > userspace > > > > > > Is we can create a defect for this and point / copy the wiki notes > > into it, that > > would help. > >https://wiki.yoctoproject.org/wiki/Qemuppc_Boot_Hangs > > > > I think I had asked Chi to see if he could reproduce this a week or > > two ago. > > When the lack of entropy problem was identified and fix, many > > people > > thought > > this hang went away as well. Chi can you read the wiki report and > > see > > if you > > can add anything to them? > > Good news is that the qemuppc issue has been identified as a bug in > qemu ppc locking which breaks timer interrupt handling. I've posted > on > the qemu mailing list asking for help in verifying what I think is > happening. > > I have a patch ready to merge which should address this one, just > cleaning up my environment and doing some further stress testing. > This is great news hopefully you will hear back from the qemu ML verifying your patch. > [There is a defect somewhere for this btw, I created the wiki as it > was > a better place to dump and update information as we learnt what it > is/is not without having to follow a train of thought updating in the > bugzilla]. > > > > o Issues with 4.13.10 host kernels booting kvm x86 guests on > > > Tumbleweed (Suse) and Fedora 26 (attempting to see if 4.13.12 > > > helps) > > > > > > Robert, can you test Fedora 26. It would help to have a defect open > > with steps to reproduce or > > something about the typical workflow/ build time/ day of the week/ > > phase of the moon. > > FWIW we have noticed that the choice of kernel timers seems to vary > in > the x86_64 boots but not with a pattern that matches the hangs. > > > > o nfs inode count for the sstate/dldir share appears to break > > > periodically causing the disk monitor to halt the builds (bug > > > 12267) > > > > > > Likely specific to the AB server so no plans to do anything for > > this > > bug. > > Agreed, this one is our infrastructure somehow :(. We have a > workaround > in -next for this at least. > > > > o a perf build race (bug 12302) > > > > I'll take a look to: > > - see if I can duplicate the bug on a fast build host > > - check upstream to see if the bug is known/fixed > > - see if I can spot a race in the build rules. > > Sounds good, thanks! > > > > o An ext filesystem creation/sizing issue (bug 12304) > > > > > > Saul, Are you around this week? Do you have any additional > > information before > > leaving for Thanksgiving? > > > > Jackie, > > Can you look at the code around the image creation and try to > > reproduce this one? > > Saul hasn't been able to reproduce. I've asked at the minimum we add > better logging so if/when this happens again, we can debug it > properly Patch sent which provides some additional debugging information to the log files, ideally, this will be saved with the bug next time this issue occurs. Sau! > next time. I did also wonder about fuzzing the image size code, > writing > some code which puts in all possible input values and checks the > sanity > of the resulting image size. Its the kind of problem a computer can > probably brute force. Anyone interested in trying that? > > Cheers, > > Richard > > -- ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
Re: [OE-core] Yocto Project Status WW47’17
On 21 November 2017 at 11:09, Richard Purdie < richard.pur...@linuxfoundation.org> wrote: > > > o Issues with 4.13.10 host kernels booting kvm x86 guests on > > > Tumbleweed (Suse) and Fedora 26 (attempting to see if 4.13.12 > > > helps) > > > > Robert, can you test Fedora 26. It would help to have a defect open > > with steps to reproduce or > > something about the typical workflow/ build time/ day of the week/ > > phase of the moon. > > FWIW we have noticed that the choice of kernel timers seems to vary in > the x86_64 boots but not with a pattern that matches the hangs. Also of note is that last week the cluster was hitting this once in every nightly run, but after a complete reboot and kernel upgrade we haven't seen it so far. So either it's gone with the latest kernels, or it's a slow accumulation of non-movable memory that eventually breaks kvm (my money is on the latter). > > > o a perf build race (bug 12302) > > I'll take a look to: > > - see if I can duplicate the bug on a fast build host > > - check upstream to see if the bug is known/fixed > > - see if I can spot a race in the build rules. > > Sounds good, thanks! It looks and sounds like a classic makefile dependency bug, should be reproducible on demand with careful explicit target steps. The perf makefiles are not exactly easy to hack though... Ross -- ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
Re: [OE-core] Yocto Project Status WW47’17
On Mon, 2017-11-20 at 20:22 -0500, Randy MacLeod wrote: > On 2017-11-20 10:36 AM, Jolley, Stephen K wrote: > > Current Dev Position: YP 2.5 Planning and M1 development > > Next Deadline: YP 2.5 M1 cut off of 12/4/17 > > > > SWAT team rotation: Juro -> Paul on Nov.17, 2017. > > SWAT team rotation: Paul -> Todor on Nov. 24, 2017. > > https://wiki.yoctoproject.org/wiki/Yocto_Build_Failure_Swat_Team > > > > Key Status/Updates: > > · There is no real change to the status from last week. We > > continue to suffer intermittent build failures and are continuing > > to attempt to debug these. > > · Currently open issues are: > > Some US-based people may be on holiday later this week so I'm > offering > help from the frozen Northland and more importantly from the team in > Beijing. ;-) > > > o qemuppc continues to demonstrate random hangs in boot in > > userspace > > Is we can create a defect for this and point / copy the wiki notes > into it, that > would help. > https://wiki.yoctoproject.org/wiki/Qemuppc_Boot_Hangs > > I think I had asked Chi to see if he could reproduce this a week or > two ago. > When the lack of entropy problem was identified and fix, many people > thought > this hang went away as well. Chi can you read the wiki report and see > if you > can add anything to them? Good news is that the qemuppc issue has been identified as a bug in qemu ppc locking which breaks timer interrupt handling. I've posted on the qemu mailing list asking for help in verifying what I think is happening. I have a patch ready to merge which should address this one, just cleaning up my environment and doing some further stress testing. [There is a defect somewhere for this btw, I created the wiki as it was a better place to dump and update information as we learnt what it is/is not without having to follow a train of thought updating in the bugzilla]. > > o Issues with 4.13.10 host kernels booting kvm x86 guests on > > Tumbleweed (Suse) and Fedora 26 (attempting to see if 4.13.12 > > helps) > > Robert, can you test Fedora 26. It would help to have a defect open > with steps to reproduce or > something about the typical workflow/ build time/ day of the week/ > phase of the moon. FWIW we have noticed that the choice of kernel timers seems to vary in the x86_64 boots but not with a pattern that matches the hangs. > > o nfs inode count for the sstate/dldir share appears to break > > periodically causing the disk monitor to halt the builds (bug > > 12267) > > Likely specific to the AB server so no plans to do anything for this > bug. Agreed, this one is our infrastructure somehow :(. We have a workaround in -next for this at least. > > o a perf build race (bug 12302) > I'll take a look to: > - see if I can duplicate the bug on a fast build host > - check upstream to see if the bug is known/fixed > - see if I can spot a race in the build rules. Sounds good, thanks! > > o An ext filesystem creation/sizing issue (bug 12304) > > Saul, Are you around this week? Do you have any additional > information before > leaving for Thanksgiving? > > Jackie, > Can you look at the code around the image creation and try to > reproduce this one? Saul hasn't been able to reproduce. I've asked at the minimum we add better logging so if/when this happens again, we can debug it properly next time. I did also wonder about fuzzing the image size code, writing some code which puts in all possible input values and checks the sanity of the resulting image size. Its the kind of problem a computer can probably brute force. Anyone interested in trying that? Cheers, Richard -- ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
Re: [OE-core] Yocto Project Status WW47’17
> > > From: MacLeod, Randy > Sent: Tuesday, November 21, 2017 09:22 > To: openembedded-core@lists.openembedded.org; Armin Kuster; Huang, Jie > (Jackie); Xu, Chi; Yang, Liezhi; WOLD, SAUL > Subject: Re: [OE-core] Yocto Project Status WW47’17 > > On 2017-11-20 10:36 AM, Jolley, Stephen K wrote: > Current Dev Position: YP 2.5 Planning and M1 development > Next Deadline: YP 2.5 M1 cut off of 12/4/17 > > SWAT team rotation: Juro -> Paul on Nov.17, 2017. > SWAT team rotation: Paul -> Todor on Nov. 24, 2017. > https://wiki.yoctoproject.org/wiki/Yocto_Build_Failure_Swat_Team > > Key Status/Updates: > ?There is no real change to the status from last week. We continue to > suffer intermittent build failures and are continuing to attempt to debug > these. > ?Currently open issues are: > > Some US-based people may be on holiday later this week so I'm offering > help from the frozen Northland and more importantly from the team in Beijing. > ;-) > > > o qemuppc continues to demonstrate random hangs in boot in userspace > > Is we can create a defect for this and point / copy the wiki notes into it, > that > would help. >https://wiki.yoctoproject.org/wiki/Qemuppc_Boot_Hangs > > I think I had asked Chi to see if he could reproduce this a week or two ago. > When the lack of entropy problem was identified and fix, many people thought > this hang went away as well. Chi can you read the wiki report and see if you > can add anything to them? > > > o Issues with 4.13.10 host kernels booting kvm x86 guests on Tumbleweed > (Suse) and Fedora 26 (attempting to see if 4.13.12 helps) > > Robert, can you test Fedora 26. It would help to have a defect open with > steps to reproduce or > something about the typical workflow/ build time/ day of the week/ phase of > the moon. > > > o nfs inode count for the sstate/dldir share appears to break periodically > causing the disk monitor to halt the builds (bug 12267) > > Likely specific to the AB server so no plans to do anything for this bug. > > > o a perf build race (bug 12302) > I'll take a look to: > - see if I can duplicate the bug on a fast build host > - check upstream to see if the bug is known/fixed > - see if I can spot a race in the build rules. > > > o An ext filesystem creation/sizing issue (bug 12304) > > Saul, Are you around this week? Do you have any additional information before > leaving for Thanksgiving? > > Jackie, > Can you look at the code around the image creation and try to reproduce this > one? There is no steps to reproduce in this bug and it seems to be a rare issue, but yes, I can take a look at the code and info from yocto's autobuilder and try to reproduce it. Thanks, Jackie > > ../Randy > > > ?Until we resolve these issues patches will continue to be slow to > merge, if at all. This also blocks several core developers from doing any > feature work at this point in time (e.g. layer setup tool is on hold, again). > ?We can only continue to stress that unless others step up and help > to try and root cause these issues, things will stall with the project. > > Planned upcoming dot releases: > YP 2.2.3 is planned and should be out in the next few weeks. > YP 2.3.3 is planned, but date TBD during YP 2.5 planning. > YP 2.4.1 is planned, but date TBD during YP 2.5 planning. > > Key YP 2.5 Dates are: > YP 2.5 M1 cut off of 12/4/17 > YP 2.5 M1 release of 12/15/17 > YP 2.5 M2 cut off of 1/15/18 > YP 2.5 M2 release of 1/26/18 > YP 2.5 M3 cut off of 2/19/18 > YP 2.5 M3 release of 3/2/18 > YP 2.5 M4 cut off of 4/2/18 > YP 2.5 M4 release of 4/27/18 > > Tracking Metrics: > WDD 2655 (last week 2640) >(https://wiki.yoctoproject.org/charts/combo.html) > > Key Status Links for YP: > https://wiki.yoctoproject.org/wiki/Yocto_Project_v2.5_Status > https://wiki.yoctoproject.org/wiki/Yocto_2.5_Schedule > https://wiki.yoctoproject.org/wiki/Yocto_2.5_Features > [If anyone has suggestions for other information you’d like to see on this > weekly status update, let us know!] > > Thanks, > > Stephen K. Jolley > Yocto Project Program Manager > INTEL, MS JF1-255, 2111 N.E. 25th Avenue, Hillsboro, OR 97124 > Work Telephone:(503) 712-0534 > 4Cell: (208) 244-4460 > 00Email:stephen.k.jol...@intel.com > > > > > -- > # Randy MacLeod. WR Linux > # Wind River an Intel Company > -- ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
Re: [OE-core] Yocto Project Status WW47’17
On 2017-11-20 10:36 AM, Jolley, Stephen K wrote: Current Dev Position: YP 2.5 Planning and M1 development Next Deadline: YP 2.5 M1 cut off of 12/4/17 SWAT team rotation: Juro -> Paul on Nov.17, 2017. SWAT team rotation: Paul -> Todor on Nov. 24, 2017. https://wiki.yoctoproject.org/wiki/Yocto_Build_Failure_Swat_Team Key Status/Updates: ·There is no real change to the status from last week. We continue to suffer intermittent build failures and are continuing to attempt to debug these. ·Currently open issues are: Some US-based people may be on holiday later this week so I'm offering help from the frozen Northland and more importantly from the team in Beijing. ;-) oqemuppc continues to demonstrate random hangs in boot in userspace Is we can create a defect for this and point / copy the wiki notes into it, that would help. https://wiki.yoctoproject.org/wiki/Qemuppc_Boot_Hangs I think I had asked Chi to see if he could reproduce this a week or two ago. When the lack of entropy problem was identified and fix, many people thought this hang went away as well. Chi can you read the wiki report and see if you can add anything to them? oIssues with 4.13.10 host kernels booting kvm x86 guests on Tumbleweed (Suse) and Fedora 26 (attempting to see if 4.13.12 helps) Robert, can you test Fedora 26. It would help to have a defect open with steps to reproduce or something about the typical workflow/ build time/ day of the week/ phase of the moon. onfs inode count for the sstate/dldir share appears to break periodically causing the disk monitor to halt the builds (bug 12267) Likely specific to the AB server so no plans to do anything for this bug. oa perf build race (bug 12302) I'll take a look to: - see if I can duplicate the bug on a fast build host - check upstream to see if the bug is known/fixed - see if I can spot a race in the build rules. oAn ext filesystem creation/sizing issue (bug 12304) Saul, Are you around this week? Do you have any additional information before leaving for Thanksgiving? Jackie, Can you look at the code around the image creation and try to reproduce this one? ../Randy ·Until we resolve these issues patches will continue to be slow to merge, if at all. This also blocks several core developers from doing any feature work at this point in time (e.g. layer setup tool is on hold, again). ·We can only continue to stress that unless others step up and help to try and root cause these issues, things will stall with the project. Planned upcoming dot releases: YP 2.2.3 is planned and should be out in the next few weeks. YP 2.3.3 is planned, but date TBD during YP 2.5 planning. YP 2.4.1 is planned, but date TBD during YP 2.5 planning. Key YP 2.5 Dates are: YP 2.5 M1 cut off of 12/4/17 YP 2.5 M1 release of 12/15/17 YP 2.5 M2 cut off of 1/15/18 YP 2.5 M2 release of 1/26/18 YP 2.5 M3 cut off of 2/19/18 YP 2.5 M3 release of 3/2/18 YP 2.5 M4 cut off of 4/2/18 YP 2.5 M4 release of 4/27/18 Tracking Metrics: WDD 2655 (last week 2640) (https://wiki.yoctoproject.org/charts/combo.html) Key Status Links for YP: https://wiki.yoctoproject.org/wiki/Yocto_Project_v2.5_Status https://wiki.yoctoproject.org/wiki/Yocto_2.5_Schedule https://wiki.yoctoproject.org/wiki/Yocto_2.5_Features [If anyone has suggestions for other information you’d like to see on this weekly status update, let us know!] Thanks, */Stephen K. Jolley/**//* *Yocto Project Program Manager* *INTEL, MS JF1-255, 2111 N.E. 25th Avenue, Hillsboro, OR 97124 * (*Work Telephone*: (503) 712-0534 (*Cell*: (208) 244-4460 * *Email*:_stephen.k.jolley@intel.com_ -- # Randy MacLeod. WR Linux # Wind River an Intel Company -- ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
[OE-core] Yocto Project Status WW47’17
Current Dev Position: YP 2.5 Planning and M1 development Next Deadline: YP 2.5 M1 cut off of 12/4/17 SWAT team rotation: Juro -> Paul on Nov.17, 2017. SWAT team rotation: Paul -> Todor on Nov. 24, 2017. https://wiki.yoctoproject.org/wiki/Yocto_Build_Failure_Swat_Team Key Status/Updates: ·There is no real change to the status from last week. We continue to suffer intermittent build failures and are continuing to attempt to debug these. ·Currently open issues are: o qemuppc continues to demonstrate random hangs in boot in userspace o Issues with 4.13.10 host kernels booting kvm x86 guests on Tumbleweed (Suse) and Fedora 26 (attempting to see if 4.13.12 helps) o nfs inode count for the sstate/dldir share appears to break periodically causing the disk monitor to halt the builds (bug 12267) o a perf build race (bug 12302) o An ext filesystem creation/sizing issue (bug 12304) ·Until we resolve these issues patches will continue to be slow to merge, if at all. This also blocks several core developers from doing any feature work at this point in time (e.g. layer setup tool is on hold, again). ·We can only continue to stress that unless others step up and help to try and root cause these issues, things will stall with the project. Planned upcoming dot releases: YP 2.2.3 is planned and should be out in the next few weeks. YP 2.3.3 is planned, but date TBD during YP 2.5 planning. YP 2.4.1 is planned, but date TBD during YP 2.5 planning. Key YP 2.5 Dates are: YP 2.5 M1 cut off of 12/4/17 YP 2.5 M1 release of 12/15/17 YP 2.5 M2 cut off of 1/15/18 YP 2.5 M2 release of 1/26/18 YP 2.5 M3 cut off of 2/19/18 YP 2.5 M3 release of 3/2/18 YP 2.5 M4 cut off of 4/2/18 YP 2.5 M4 release of 4/27/18 Tracking Metrics: WDD 2655 (last week 2640) (https://wiki.yoctoproject.org/charts/combo.html) Key Status Links for YP: https://wiki.yoctoproject.org/wiki/Yocto_Project_v2.5_Status https://wiki.yoctoproject.org/wiki/Yocto_2.5_Schedule https://wiki.yoctoproject.org/wiki/Yocto_2.5_Features [If anyone has suggestions for other information you’d like to see on this weekly status update, let us know!] Thanks, Stephen K. Jolley Yocto Project Program Manager INTEL, MS JF1-255, 2111 N.E. 25th Avenue, Hillsboro, OR 97124 • Work Telephone:(503) 712-0534 •Cell: (208) 244-4460 • Email:stephen.k.jol...@intel.com -- ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
