Re: [Xen-devel] [OSSTEST PATCH] README.hardware-acquisition [and 1 more messages] [and 2 more messages] [and 2 more messages]
Hi Ian,
Given that the Debian bug is filed so either way we have a path forward,
should I get started with the discussion about shipping the hardware
(get various approvals, prepare boxes for shipping, etc.)? That's going
to take a few weeks for sure. Or would you like to wait for the Debian
bug to be resolved / alternative code to be written ?
Cheers,
Stefano
On Mon, 5 Nov 2018, Ian Jackson wrote:
> Julien Grall writes ("Re: [OSSTEST PATCH] README.hardware-acquisition [and 1
> more messages] [and 2 more messages] [and 2 more messages]"):
> > AFAICT, Ian's main concern was adding yet another dependency on external
> > entity.
>
> We could put the .deb repository on xenbits.
> There remains the question of who will do the manual rebuilds.
>
> Another thing that occurred to me is that not all kernel .debs are
> created equal. Do ones that come from the kernel source tree, without
> any of the Debian packaging code, interact properly with Debian's
> initramfs and bootloader update machinery ?
>
> > However, OSStest already provides scripts to build kernel. So why would you
> > want
> > to use Dockerfiles/ViryaOS?
>
> That seems to me to be a very salient point. Earlier I wrote:
>
>Ie the biggest work here of all kinds is is glue. Adding more
>entities to the problem will increase rather than reduce the amount of
>glue code that needs to be written.
>
> The amount of new glue that is needed depends also on how much of the
> existing systems can be reused.
>
>
> I am starting to think that it may have been a mistake of me to
> explain in clear and precise detail what would be needed, to have
> osstest directly use its own-built kernels for baremetal install.
>
> If you look at the length of my description it looks like a lot of
> work. But the competing proposals are being described by some
> handwaving. Obviously they look simpler then!
>
> No-one has written a comparabily detailed plan for exactly what work
> would need to be done, and what risks there are, for example in this
> scheme to use docker and ViryaOS and an apt repository. It's true
> that such a plan would have fewer changes to osstest; but more of it
> would be manual steps. In the case of manual steps IMO a comparably
> detailed plan would include a rough set of proposed command line
> runes, etc. I'm not even sure that this scheme has been thought
> through enough for us to write such a plan with confidence that it
> will work.
>
> Almost certainly such a detailed plan, if we can write it, would be
> considerably longer and more complex than the plan I posted earlier.
>
> And of course it has a higher overall maintenance burden because
> updates would be manual forever more - not only manual, but also
> dependent on the dockerfile continuing to work, which means the manual
> steps depend on the availability (and integrity!) of whatever external
> entities the dockerfile uses.
___
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [Xen-devel] [OSSTEST PATCH] README.hardware-acquisition [and 1 more messages] [and 2 more messages] [and 2 more messages]
Julien Grall writes ("Re: [OSSTEST PATCH] README.hardware-acquisition [and 1
more messages] [and 2 more messages] [and 2 more messages]"):
> AFAICT, Ian's main concern was adding yet another dependency on external
> entity.
We could put the .deb repository on xenbits.
There remains the question of who will do the manual rebuilds.
Another thing that occurred to me is that not all kernel .debs are
created equal. Do ones that come from the kernel source tree, without
any of the Debian packaging code, interact properly with Debian's
initramfs and bootloader update machinery ?
> However, OSStest already provides scripts to build kernel. So why would you
> want
> to use Dockerfiles/ViryaOS?
That seems to me to be a very salient point. Earlier I wrote:
Ie the biggest work here of all kinds is is glue. Adding more
entities to the problem will increase rather than reduce the amount of
glue code that needs to be written.
The amount of new glue that is needed depends also on how much of the
existing systems can be reused.
I am starting to think that it may have been a mistake of me to
explain in clear and precise detail what would be needed, to have
osstest directly use its own-built kernels for baremetal install.
If you look at the length of my description it looks like a lot of
work. But the competing proposals are being described by some
handwaving. Obviously they look simpler then!
No-one has written a comparabily detailed plan for exactly what work
would need to be done, and what risks there are, for example in this
scheme to use docker and ViryaOS and an apt repository. It's true
that such a plan would have fewer changes to osstest; but more of it
would be manual steps. In the case of manual steps IMO a comparably
detailed plan would include a rough set of proposed command line
runes, etc. I'm not even sure that this scheme has been thought
through enough for us to write such a plan with confidence that it
will work.
Almost certainly such a detailed plan, if we can write it, would be
considerably longer and more complex than the plan I posted earlier.
And of course it has a higher overall maintenance burden because
updates would be manual forever more - not only manual, but also
dependent on the dockerfile continuing to work, which means the manual
steps depend on the availability (and integrity!) of whatever external
entities the dockerfile uses.
Thanks,
Ian.
___
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [Xen-devel] [OSSTEST PATCH] README.hardware-acquisition [and 1 more messages] [and 2 more messages] [and 2 more messages]
Hi Stefano, On 02/11/2018 23:44, Stefano Stabellini wrote: On Fri, 2 Nov 2018, Julien Grall wrote: On 02/11/2018 17:56, Stefano Stabellini wrote: On Fri, 2 Nov 2018, Ian Jackson wrote: Ie the biggest work here of all kinds is is glue. Adding more entities to the problem will increase rather than reduce the amount of glue code that needs to be written. Basically, you are saying that even if had a well maintained deb repository of kernel packages for OSSTest to use, doesn't matter how we achieve this goal, it would still require a non trivial amount of work to do the integration with OSSTest. This is not how I understood the thread. I believe this was related to use an external CI loop. In our case, once we have a deb package. Then this is not much different than using a backport kernel. We already have such support in Osstest, so it should not be too difficult to adapt it for a different repo. OK, I misunderstood Ian, thanks for stepping in. If that is the case, then: 1) We already have a Xen ARM Linux tree which we have to maintain for Dom0 The maintenance model and testing of this tree doesn't change regardless. 2) We already have Dockerfiles and scripts by Doug and ViryaOS to build kernels on gitlab We could improve them to build a full deb repository. Which does not work on Arm today... (see the previous discussions). Lars and Wei are working towards addressing this. What's stopping us from using (2) to setup up a Xen Project deb repo? Does it really matter who executes scripts (2) and whether they are run on your laptop or in the cloud as long as the build is fully reproducible? We are not talking about hacking config files around and using who knows what compiler to build something. docker build will give you the same output no matter the host. AFAICT, Ian's main concern was adding yet another dependency on external entity. However, OSStest already provides scripts to build kernel. So why would you want to use Dockerfiles/ViryaOS? Cheers, -- Julien Grall ___ Xen-devel mailing list Xen-devel@lists.xenproject.org https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [Xen-devel] [OSSTEST PATCH] README.hardware-acquisition [and 1 more messages] [and 2 more messages] [and 2 more messages]
Stefano Stabellini writes ("Re: [OSSTEST PATCH] README.hardware-acquisition
[and 1 more messages] [and 2 more messages] [and 2 more messages]"):
> Basically, you are saying that even if had a well maintained deb
> repository of kernel packages for OSSTest to use, doesn't matter how we
> achieve this goal, it would still require a non trivial amount of work
> to do the integration with OSSTest.
No, sorry. The one thing that osstest can easily consume is a
well-maintained deb repository.
Ian.
___
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [Xen-devel] [OSSTEST PATCH] README.hardware-acquisition [and 1 more messages] [and 2 more messages] [and 2 more messages]
On Fri, 2 Nov 2018, Julien Grall wrote: > On 02/11/2018 17:56, Stefano Stabellini wrote: > > On Fri, 2 Nov 2018, Ian Jackson wrote: > > > Ie the biggest work here of all kinds is is glue. Adding more > > > entities to the problem will increase rather than reduce the amount of > > > glue code that needs to be written. > > > > Basically, you are saying that even if had a well maintained deb > > repository of kernel packages for OSSTest to use, doesn't matter how we > > achieve this goal, it would still require a non trivial amount of work > > to do the integration with OSSTest. > > This is not how I understood the thread. I believe this was related to use an > external CI loop. > > In our case, once we have a deb package. Then this is not much different than > using a backport kernel. We already have such support in Osstest, so it should > not be too difficult to adapt it for a different repo. OK, I misunderstood Ian, thanks for stepping in. If that is the case, then: 1) We already have a Xen ARM Linux tree which we have to maintain for Dom0 The maintenance model and testing of this tree doesn't change regardless. 2) We already have Dockerfiles and scripts by Doug and ViryaOS to build kernels on gitlab We could improve them to build a full deb repository. What's stopping us from using (2) to setup up a Xen Project deb repo? Does it really matter who executes scripts (2) and whether they are run on your laptop or in the cloud as long as the build is fully reproducible? We are not talking about hacking config files around and using who knows what compiler to build something. docker build will give you the same output no matter the host. ___ Xen-devel mailing list Xen-devel@lists.xenproject.org https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [Xen-devel] [OSSTEST PATCH] README.hardware-acquisition [and 1 more messages] [and 2 more messages] [and 2 more messages]
On 02/11/2018 17:56, Stefano Stabellini wrote: On Fri, 2 Nov 2018, Ian Jackson wrote: Ie the biggest work here of all kinds is is glue. Adding more entities to the problem will increase rather than reduce the amount of glue code that needs to be written. Basically, you are saying that even if had a well maintained deb repository of kernel packages for OSSTest to use, doesn't matter how we achieve this goal, it would still require a non trivial amount of work to do the integration with OSSTest. This is not how I understood the thread. I believe this was related to use an external CI loop. In our case, once we have a deb package. Then this is not much different than using a backport kernel. We already have such support in Osstest, so it should not be too difficult to adapt it for a different repo. Cheers, -- Julien Grall ___ Xen-devel mailing list Xen-devel@lists.xenproject.org https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [Xen-devel] [OSSTEST PATCH] README.hardware-acquisition [and 1 more messages] [and 2 more messages] [and 2 more messages]
On Fri, 2 Nov 2018, Ian Jackson wrote: > > > If the Debian cross compilers are OK, then I think the necessary > > > changes to osstest are: > ... > > I thought that we could have provided a deb repository with alternative > > kernels for OSSTests to use. We would have scripts to generate those deb > > packages from the Xen ARM Linux tree in a repository on xenbits, but we > > wouldn't necessarily have OSSTest run the script. Initially, we could > > run the scripts by hand, then, we could run them automatically in > > OSSTest or elsewhere. Is that a possibility? I already have Dockerfiles > > (AKA bash scripts) to build an ARM kernel on a few distros, that's > > something I could make available. > > Everything that osstest consumes should either be either: > (i) snapshotted and push gated, so that osstest always uses a version >that it itself has previously tested > (ii) maintained to a very high standard of quality and availability. > > For things for which we take approach (ii), regressions of any kind, > or unavailability of the download server, cause what is effectively a > whole-system outage for osstest: every test run ends up plagued by > whatever lossage is going on. Implying no disrespect, but doing (ii) > for a kernel apt repository is very hard. > > So I think (i) would be more appropriate. That would mean generating, > periodically, a whole new apt repository, alongside the old one. > Presumably they would have the generation date in the filename, like > our Debian installer images do. Updating would involve a commit to > the osstest config file, which is push-gate-controlled. > > Overall I think, though, that this is probably not the best approach. > > What you are saying is that you do not have the effort to automate the > building of kernel binaries, and instead you propose to do it > manually. That seems like a false economy. > > The task of automating the building of kernel binaries is points 1-3 > of my plan to cross build the kernels and use the result for baremetal > builds; that's not the hardest part. [...] > Ie the biggest work here of all kinds is is glue. Adding more > entities to the problem will increase rather than reduce the amount of > glue code that needs to be written. Basically, you are saying that even if had a well maintained deb repository of kernel packages for OSSTest to use, doesn't matter how we achieve this goal, it would still require a non trivial amount of work to do the integration with OSSTest. I would have thought that it would have made OSSTest work (almost) out of the box. Too bad. In that case, I'll leave this thread and the implementation choices to the people that are actually going to do work on it. ___ Xen-devel mailing list Xen-devel@lists.xenproject.org https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [Xen-devel] [OSSTEST PATCH] README.hardware-acquisition [and 1 more messages] [and 2 more messages] [and 2 more messages]
Julien Grall writes ("Re: [OSSTEST PATCH] README.hardware-acquisition [and 1
more messages] [and 2 more messages] [and 2 more messages]"):
> On 02/11/2018 15:44, Ian Jackson wrote:
> > We have more x86 capacity and are deploying some dedicated build
> > hosts. But cross-compilation is more complex (particularly the
> > compiler conflict...)
>
> Do you mean the multilib problem? This issue is only when cross-compiling
> tools
> (which I would not recommend anyway...). For things like kernel and
> hypervisor,
> you don't need multilib.
Yes, that problem.
I think by `cross-compiling' here you mean to include running gcc -m32
on an amd64 host or vice versa, which is not quite real
cross-compiling in my view. But it does require gcc-multilib.
But, as it turns out, you are right that this is not a problem for
osstest. When osstest does builds for an i386 dom0, it does the
hypervisor build on an amd64 host and the tools build separately on an
i386 host.
Ian.
___
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [Xen-devel] [OSSTEST PATCH] README.hardware-acquisition [and 1 more messages] [and 2 more messages] [and 2 more messages]
Hi,
On 02/11/2018 15:44, Ian Jackson wrote:
Julien Grall writes ("Re: [OSSTEST PATCH] README.hardware-acquisition [and 1 more
messages] [and 2 more messages] [and 2 more messages]"):
On 02/11/2018 15:05, Ian Jackson wrote:
I guess that's probably tolerable. We'd have to think about whether
we should snapshot them, or install them directly from upstream. The
latter is less coding effort in osstest (just add an apt source and
run the install rune I guess) but it would imply tracking these in an
uncontrolled way, so we would want to be confident that they are
maintained to a high standard, since problems with the compiler would
break everything for us. Also I would ask: how reliable is their apt
repository hosting ? If it goes down, likewise, everything would
break for us.
Linaro hosting is pretty reliable. But AFAIK, they don't have an apt repo for
the toolchains. We would need to download a tarball and unpack it.
That would be doable. If the tarballs have versions we could put the
version number in the osstest config and then it would be push gated.
Perhaps Julien has time to do that.
Arm64 cross-compiler in Debian should be suitable enough for building the
kernel. If there are any issue with them, then a bug report should be filled.
OK, that's definitely going to be easier then.
My preference is to avoid cross-compiling if we can. This would avoid to use x86
resource for building kernel that could be re-used for testing Xen itself.
We have more x86 capacity and are deploying some dedicated build
hosts. But cross-compilation is more complex (particularly the
compiler conflict...)
Do you mean the multilib problem? This issue is only when cross-compiling tools
(which I would not recommend anyway...). For things like kernel and hypervisor,
you don't need multilib.
Cheers,
--
Julien Grall
___
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [Xen-devel] [OSSTEST PATCH] README.hardware-acquisition [and 1 more messages] [and 2 more messages] [and 2 more messages]
Julien Grall writes ("Re: [OSSTEST PATCH] README.hardware-acquisition [and 1
more messages] [and 2 more messages] [and 2 more messages]"):
> On 02/11/2018 15:05, Ian Jackson wrote:
> > I guess that's probably tolerable. We'd have to think about whether
> > we should snapshot them, or install them directly from upstream. The
> > latter is less coding effort in osstest (just add an apt source and
> > run the install rune I guess) but it would imply tracking these in an
> > uncontrolled way, so we would want to be confident that they are
> > maintained to a high standard, since problems with the compiler would
> > break everything for us. Also I would ask: how reliable is their apt
> > repository hosting ? If it goes down, likewise, everything would
> > break for us.
>
> Linaro hosting is pretty reliable. But AFAIK, they don't have an apt repo for
> the toolchains. We would need to download a tarball and unpack it.
That would be doable. If the tarballs have versions we could put the
version number in the osstest config and then it would be push gated.
> > Perhaps Julien has time to do that.
>
> Arm64 cross-compiler in Debian should be suitable enough for building the
> kernel. If there are any issue with them, then a bug report should be filled.
OK, that's definitely going to be easier then.
> My preference is to avoid cross-compiling if we can. This would avoid to use
> x86
> resource for building kernel that could be re-used for testing Xen itself.
We have more x86 capacity and are deploying some dedicated build
hosts. But cross-compilation is more complex (particularly the
compiler conflict...)
Ian.
___
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [Xen-devel] [OSSTEST PATCH] README.hardware-acquisition [and 1 more messages] [and 2 more messages] [and 2 more messages]
Hi,
On 02/11/2018 15:05, Ian Jackson wrote:
Stefano Stabellini writes ("Re: [OSSTEST PATCH] README.hardware-acquisition [and 1
more messages] [and 2 more messages]"):
Thank you for the detailed answer and the willingness to see osstest
changed in this respect.
Sure. It's not fixed in stone, and a variety of approachs will fit
into it I think.
Let me premise that as much as I would like this to be done, I had a
look at my schedule, and, realistically, I can only volunteer very
little time on this. In regards to the two Xilinx boards, it looks like
we'll just have to wait for Debian.
OK. That's perhaps less work overall at our end anyway. Let me know
if you think any aspect of this is getting stuck somehow; I have
contacts in Debian which I could use to enquire or whatever.
On Thu, 1 Nov 2018, Ian Jackson wrote:
The prerequisite is obviously an appropriate cross-compiler. Will the
Debian cross-compilers do ?
Probably it would work, but I don't know for sure. Most people use the
Linaro compiler and toolchain:
https://releases.linaro.org/components/toolchain/binaries/latest-7/aarch64-linux-gnu/
https://releases.linaro.org/components/toolchain/gcc-linaro/latest-7/
I guess that's probably tolerable. We'd have to think about whether
we should snapshot them, or install them directly from upstream. The
latter is less coding effort in osstest (just add an apt source and
run the install rune I guess) but it would imply tracking these in an
uncontrolled way, so we would want to be confident that they are
maintained to a high standard, since problems with the compiler would
break everything for us. Also I would ask: how reliable is their apt
repository hosting ? If it goes down, likewise, everything would
break for us.
Linaro hosting is pretty reliable. But AFAIK, they don't have an apt repo for
the toolchains. We would need to download a tarball and unpack it.
Testing the Debian cross-compiler would be very easy.
Perhaps Julien has time to do that.
Arm64 cross-compiler in Debian should be suitable enough for building the
kernel. If there are any issue with them, then a bug report should be filled.
[...]
Wei Liu writes ("Re: [OSSTEST PATCH] README.hardware-acquisition [and 1 more
messages] [and 2 more messages]"):
Debian's cross-compiler package conflicts with its native compiler,
Specifically it conflicts with the multilib style native compiler used
for Xen. This is a bug in Debian but it is intractable because of the
approach of the Debiabn GCC maintainer, so we will have to live with
it.
However, this is not a blocker because osstest could use a dedicated
x86 setup for the ARM cross builds. That's not ideal because it
shares resources less well, but it's certainly workable.
But maybe we just want to build on ThunderX. That leaves steps 4-6 of
my plan, which almost any approach (other than fixing the kernels in
Debian) require.
My preference is to avoid cross-compiling if we can. This would avoid to use x86
resource for building kernel that could be re-used for testing Xen itself.
Cheers,
--
Julien Grall
___
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [Xen-devel] [OSSTEST PATCH] README.hardware-acquisition [and 1 more messages] [and 2 more messages] [and 2 more messages]
Stefano Stabellini writes ("Re: [OSSTEST PATCH] README.hardware-acquisition
[and 1 more messages] [and 2 more messages]"):
> Thank you for the detailed answer and the willingness to see osstest
> changed in this respect.
Sure. It's not fixed in stone, and a variety of approachs will fit
into it I think.
> Let me premise that as much as I would like this to be done, I had a
> look at my schedule, and, realistically, I can only volunteer very
> little time on this. In regards to the two Xilinx boards, it looks like
> we'll just have to wait for Debian.
OK. That's perhaps less work overall at our end anyway. Let me know
if you think any aspect of this is getting stuck somehow; I have
contacts in Debian which I could use to enquire or whatever.
> On Thu, 1 Nov 2018, Ian Jackson wrote:
> > The prerequisite is obviously an appropriate cross-compiler. Will the
> > Debian cross-compilers do ?
>
> Probably it would work, but I don't know for sure. Most people use the
> Linaro compiler and toolchain:
>
> https://releases.linaro.org/components/toolchain/binaries/latest-7/aarch64-linux-gnu/
> https://releases.linaro.org/components/toolchain/gcc-linaro/latest-7/
I guess that's probably tolerable. We'd have to think about whether
we should snapshot them, or install them directly from upstream. The
latter is less coding effort in osstest (just add an apt source and
run the install rune I guess) but it would imply tracking these in an
uncontrolled way, so we would want to be confident that they are
maintained to a high standard, since problems with the compiler would
break everything for us. Also I would ask: how reliable is their apt
repository hosting ? If it goes down, likewise, everything would
break for us.
> Testing the Debian cross-compiler would be very easy.
Perhaps Julien has time to do that.
> > If the Debian cross compilers are OK, then I think the necessary
> > changes to osstest are:
...
> I thought that we could have provided a deb repository with alternative
> kernels for OSSTests to use. We would have scripts to generate those deb
> packages from the Xen ARM Linux tree in a repository on xenbits, but we
> wouldn't necessarily have OSSTest run the script. Initially, we could
> run the scripts by hand, then, we could run them automatically in
> OSSTest or elsewhere. Is that a possibility? I already have Dockerfiles
> (AKA bash scripts) to build an ARM kernel on a few distros, that's
> something I could make available.
Everything that osstest consumes should either be either:
(i) snapshotted and push gated, so that osstest always uses a version
that it itself has previously tested
(ii) maintained to a very high standard of quality and availability.
For things for which we take approach (ii), regressions of any kind,
or unavailability of the download server, cause what is effectively a
whole-system outage for osstest: every test run ends up plagued by
whatever lossage is going on. Implying no disrespect, but doing (ii)
for a kernel apt repository is very hard.
So I think (i) would be more appropriate. That would mean generating,
periodically, a whole new apt repository, alongside the old one.
Presumably they would have the generation date in the filename, like
our Debian installer images do. Updating would involve a commit to
the osstest config file, which is push-gate-controlled.
Overall I think, though, that this is probably not the best approach.
What you are saying is that you do not have the effort to automate the
building of kernel binaries, and instead you propose to do it
manually. That seems like a false economy.
The task of automating the building of kernel binaries is points 1-3
of my plan to cross build the kernels and use the result for baremetal
builds; that's not the hardest part.
> This morning Julien had one more different suggestion: building the
> kernel with OSSTest on SoftIron, that we know it works, it would be a
> native compilation. Then we could use the built kernel together with the
> Debian installer on the other boards (Xilinx, Renesas, etc.)
Yes, that is also a possibility. But it still involves steps 4-6 from
my plan.
> Either way, the kernel to be used with the embedded boards doesn't need
> to be rebuilt often, only once a month or so.
Of course we shouldn't waste it, but computer time is much cheaper
than human time. osstest already has mechanisms to optimise by
reusing builds where appropriate. The easiest thing is to tell
osstest `we want a kernel built like this' and it will DTRT.
Lars Kurth writes ("Re: [OSSTEST PATCH] README.hardware-acquisition [and 1 more
messages] [and 2 more messages]"):
> Either way, the kernel to be used with the embedded boards doesn't need
> to be rebuilt often, only once a month or so.
>
> That would fit with the https://gitlab.com/xen-project/xen/container_registry
> model
> where we store Dom0 baselines as containers for builds via the Gitlab CI
>
> This may be a stupid idea, but I wante
