[Xen-devel] [stage1-xen PATCH v2 06/11] build/fedora: Add `xen-4.9.0-runit/*` scripts

2017-09-19 Thread Rajiv Ranganath 
From: Rajiv M Ranganath <rajiv.rangan...@atihita.com>

Signed-off-by: Rajiv Ranganath <rajiv.rangan...@atihita.com>
---
 build/fedora/xen-4.9.0-runit/setup.sh  |   18 
 build/fedora/xen-4.9.0-runit/teardown.sh   |   18 
 .../xen-4.9.0-runit/xen-init-dom0-disk-backend/run |   11 ++
 build/fedora/xen-4.9.0-runit/xen-init-dom0/run |9 
 build/fedora/xen-4.9.0-runit/xenconsoled/run   |   13 +++
 build/fedora/xen-4.9.0-runit/xenstored/run |   23 
 6 files changed, 92 insertions(+)
 create mode 100755 build/fedora/xen-4.9.0-runit/setup.sh
 create mode 100755 build/fedora/xen-4.9.0-runit/teardown.sh
 create mode 100755 build/fedora/xen-4.9.0-runit/xen-init-dom0-disk-backend/run
 create mode 100755 build/fedora/xen-4.9.0-runit/xen-init-dom0/run
 create mode 100755 build/fedora/xen-4.9.0-runit/xenconsoled/run
 create mode 100755 build/fedora/xen-4.9.0-runit/xenstored/run

diff --git a/build/fedora/xen-4.9.0-runit/setup.sh 
b/build/fedora/xen-4.9.0-runit/setup.sh
new file mode 100755
index 000..935d332
--- /dev/null
+++ b/build/fedora/xen-4.9.0-runit/setup.sh
@@ -0,0 +1,18 @@
+#!/bin/bash
+
+set -e
+
+# runit RPM creates `/etc/service` directory
+if [ ! -d "/etc/service" ]; then
+echo "/etc/service directory not found. Please install runit RPM."
+exit 1
+fi
+
+runit_services="xenconsoled xen-init-dom0 xen-init-dom0-disk-backend xenstored"
+
+for service in $runit_services; do
+ln -sf /opt/xen-4.9.0-runit/$service /etc/service/$service
+done
+
+echo "Successfully created symlinks in /etc/service directory."
+exit 0
diff --git a/build/fedora/xen-4.9.0-runit/teardown.sh 
b/build/fedora/xen-4.9.0-runit/teardown.sh
new file mode 100755
index 000..d333807
--- /dev/null
+++ b/build/fedora/xen-4.9.0-runit/teardown.sh
@@ -0,0 +1,18 @@
+#!/bin/bash
+
+set -e
+
+# runit RPM creates `/etc/service` directory
+if [ ! -d "/etc/service" ]; then
+echo "/etc/service directory not found."
+exit 1
+fi
+
+runit_services="xenconsoled xen-init-dom0 xen-init-dom0-disk-backend xenstored"
+
+for service in $runit_services; do
+rm -f /etc/service/$service
+done
+
+echo "Successfully deleted symlinks in /etc/service directory."
+exit 0
diff --git a/build/fedora/xen-4.9.0-runit/xen-init-dom0-disk-backend/run 
b/build/fedora/xen-4.9.0-runit/xen-init-dom0-disk-backend/run
new file mode 100755
index 000..a59660c
--- /dev/null
+++ b/build/fedora/xen-4.9.0-runit/xen-init-dom0-disk-backend/run
@@ -0,0 +1,11 @@
+#!/bin/bash
+
+set -e
+
+sv check xenstored >/dev/null || exit 1
+sv check xenconsoled >/dev/null || exit 1
+
+# In case of failure, allow user to run teardown script
+sleep 5s
+
+exec /opt/xen-4.9.0/lib/xen/bin/qemu-system-i386 -xen-domid 0 -xen-attach 
-name dom0 -nographic -M xenpv -monitor /dev/null -serial /dev/null -parallel 
/dev/null -nodefaults -no-user-config
diff --git a/build/fedora/xen-4.9.0-runit/xen-init-dom0/run 
b/build/fedora/xen-4.9.0-runit/xen-init-dom0/run
new file mode 100755
index 000..3b98091
--- /dev/null
+++ b/build/fedora/xen-4.9.0-runit/xen-init-dom0/run
@@ -0,0 +1,9 @@
+#!/bin/bash
+
+set -e
+
+sv check xenstored >/dev/null || exit 1
+
+/opt/xen-4.9.0/lib/xen/bin/xen-init-dom0
+
+exec chpst -b xen-init-dom0 runit-pause
diff --git a/build/fedora/xen-4.9.0-runit/xenconsoled/run 
b/build/fedora/xen-4.9.0-runit/xenconsoled/run
new file mode 100755
index 000..a1c9a2c
--- /dev/null
+++ b/build/fedora/xen-4.9.0-runit/xenconsoled/run
@@ -0,0 +1,13 @@
+#!/bin/bash
+
+set -e
+
+sv check xen-init-dom0 >/dev/null || exit 1
+
+[ ! -d /var/log/xen/console ] && mkdir -p /var/log/xen/console
+
+# In case of failure, allow user to run teardown script
+sleep 5s
+
+# --log=[none|guest|hv|all]
+exec /opt/xen-4.9.0/sbin/xenconsoled -i --log=none
diff --git a/build/fedora/xen-4.9.0-runit/xenstored/run 
b/build/fedora/xen-4.9.0-runit/xenstored/run
new file mode 100755
index 000..e49eb64
--- /dev/null
+++ b/build/fedora/xen-4.9.0-runit/xenstored/run
@@ -0,0 +1,23 @@
+#!/bin/bash
+
+set -e
+
+[ ! -d /var/run/xen ] && mkdir -p /var/run/xen
+[ ! -d /var/run/xenstored ] && mkdir -p /var/run/xenstored
+[ ! -d /var/log/xen ] && mkdir -p /var/log/xen
+[ ! -d /var/lib/xen ] && mkdir -p /var/lib/xen
+[ ! -d /var/lib/xen/dump ] && mkdir -p /var/lib/xen/dump
+[ ! -d /var/lib/xen/xenpaging ] && mkdir -p /var/lib/xen/paging
+[ ! -d /var/lib/xenstored ] && mkdir -p /var/lib/xenstored
+modprobe -q xen-evtchn || exit 1
+modprobe -q xen-gntdev || exit 1
+modprobe -q xen-gntalloc || exit 1
+mountpoint -q /proc/xen || mount -t xenfs xenfs /proc/xen
+mountpoint -q /var/lib/xenstored || mount -t tmpfs xenstored /var/lib/xenstored
+grep -q "control_d" /proc/xen/capabilities || exit 1
+

[Xen-devel] [stage1-xen PATCH v2 08/11] README.md: Add CircleCI badge

2017-09-19 Thread Rajiv Ranganath 
From: Rajiv M Ranganath <rajiv.rangan...@atihita.com>

Signed-off-by: Rajiv Ranganath <rajiv.rangan...@atihita.com>
Reviewed-by: Stefano Stabellini <sstabell...@kernel.org>
---
 README.md |2 ++
 1 file changed, 2 insertions(+)

diff --git a/README.md b/README.md
index 9ea6adf..e1cd40c 100644
--- a/README.md
+++ b/README.md
@@ -1,5 +1,7 @@
 # stage1-xen - A Xen based stage1 for CoreOS rkt
 
+[![Build 
Status](https://circleci.com/gh/rkt/stage1-xen/tree/master.svg?style=shield=:circle-token)](https://circleci.com/gh/rkt/stage1-xen/tree/master)
+
 ## Goal
 
 CoreOS rkt is a modular container engine with [three stages of 
execution](https://coreos.com/rkt/docs/latest/devel/stage1-implementors-guide.html).
 Stage1 is responsible for creating the execution environment for the contained 
applications.


___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

[Xen-devel] [stage1-xen PATCH v2 10/11] BUILDING.md: Add Fedora instructions

2017-09-19 Thread Rajiv Ranganath 
Signed-off-by: Rajiv Ranganath <rajiv.rangan...@atihita.com>
Reviewed-by: Stefano Stabellini <sstabell...@kernel.org>
---
 BUILDING.md |   96 ---
 1 file changed, 91 insertions(+), 5 deletions(-)

diff --git a/BUILDING.md b/BUILDING.md
index 3ef5311..4775b65 100644
--- a/BUILDING.md
+++ b/BUILDING.md
@@ -1,7 +1,13 @@
 # Build
-stage1-xen requires new Xen and QEMU versions at the time of writing. You are 
unlikely to find them already packaged with your distro. This document 
describes how to build and install the latest Xen and QEMU from scratch. In 
addition, given that CoreOS rkt is also missing from reasonably new distros 
such as Ubuntu Xenial Xerus, I added instructions on how to build that too. The 
document includes the dependencies needed for the build based on Ubuntu Xenial 
Xerus.
+stage1-xen requires new Xen and QEMU versions at the time of writing. You are 
unlikely to find them already packaged with your distro. This document 
describes how to build and install the latest Xen, QEMU and rkt from scratch 
for Ubuntu Xenial Xerus and Fedora. Differently from documentation for Ubuntu, 
the documentation for Fedora uses a Docker container for the build. There is 
also support for building on host on Fedora.
 
-## Building Xen
+ * [Ubuntu Xenial Xerus](#build_ubuntu)
+ * [Fedora](#build_fedora)
+
+
+## Ubuntu Xenial Xerus
+
+### Building Xen
 ```
 apt-get install git build-essential python-dev gettext uuid-dev 
libncurses5-dev libyajl-dev libaio-dev pkg-config libglib2.0-dev libssl-dev 
libpixman-1-dev bridge-utils wget libfdt-dev bin86 bcc liblzma-dev iasl 
libc6-dev-i386
 
@@ -17,7 +23,7 @@ reboot
 Make sure to select Xen at boot, or edit /boot/grub/grub.cfg to make it the 
default, changing "set default="0" to point to the appropriate entry below (the 
one booting xen.gz), which could be entry number "4" for example.
 
 
-## Building QEMU
+### Building QEMU
 ```
 apt-get install libglib2.0-dev libpixman-1-dev libcap-dev libattr1-dev
 
@@ -54,7 +60,7 @@ make install
 cp i386-softmmu/qemu-system-i386 /usr/lib/xen/bin/
 ```
 
-## Building CoreOS rkt
+### Building CoreOS rkt
 ```
 apt-get install golang automake libacl1-dev libsystemd-dev
 ./configure --disable-tpm --with-stage1-flavors=coreos
@@ -62,7 +68,7 @@ make
 cp build-rkt-1.26.0+git/target/bin/rkt /usr/sbin
 ```
 
-## Building stage1-xen
+### Building stage1-xen
 ```
 apt-get install busybox-static jq
 
@@ -72,3 +78,83 @@ export GOPATH=/path/to/gopath
 bash build.sh
 cp stage1-xen.aci /home/username
 ```
+
+
+## Fedora
+
+On Fedora there are two ways to build stage1-xen artifacts.
+
+ * [Container Build](#build_fedora_container_build)
+ * [Manual Build](#build_fedora_manual_build)
+
+
+### Container Build
+
+We can build stage1-xen artifacts (Xen, QEMU and rkt) automatically in a 
docker container as follows 
+
+```
+cd stage1-xen
+
+docker pull lambdalinuxfedora/stage1-xen-fedora-buildroot
+
+docker run --rm \
+  -v `pwd`:/root/gopath/src/github.com/rkt/stage1-xen \
+  -v /tmp:/tmp \
+  -t -i lambdalinuxfedora/stage1-xen-fedora-buildroot \
+  /sbin/my_init -- /root/bin/run
+```
+
+Once `docker run` completes, the build artifact `stage1-xen-build.tar.gz` is 
generated in `/tmp` directory. Please see 
[RUNNING_STAGE1_XEN.md](build/fedora/RUNNING_STAGE1_XEN.md) for details on how 
to setup Fedora for running stage1-xen.
+
+
+### Manual Build
+
+It is also possible to manually build stage1-xen components on a Fedora host.
+
+Please ensure that you have all the dependencies installed. The dependencies 
for Xen, QEMU, rkt and stage1-xen is documented in 
[buildroot-Dockerfile](build/fedora/buildroot-Dockerfile). You will also need 
to install [`binutils`](https://github.com/lambda-linux-fedora/binutils) 
package that is compiled with `i386pe` support. You can download the pre-built 
RPMs from [here](https://drive.google.com/open?id=0B_tTbuxmuRzIR05wQ3E1eWVyaGs).
+
+Install `binutils` package.
+
+```
+tar xvf binutils-2.26.1-1.1.fc25.tar
+
+dnf install -y 
./binutils/2.26.1/1.1.fc25/x86_64/binutils-2.26.1-1.1.fc25.x86_64.rpm
+```
+
+You can verify `i386pe` support in `binutils` by doing the following.
+
+```
+[root@localhost]# ld -V
+GNU ld version 2.26.1-1.1.fc25  Supported emulations:
+   elf_x86_64
+   elf32_x86_64
+   elf_i386
+   elf_iamcu
+   i386linux
+   elf_l1om
+   elf_k1om
+   i386pep
+   i386pe
+```
+
+You should see the lines `i386pep` and `i386pe` in the output.
+
+Next you can build Xen, Qemu and rkt using the following scripts 
+
+ * [`build/fedora/components/xen`](build/fedora/components/xen)
+ * [`build/fedora/components/qemu`](build/fedora/components/qemu)
+ * [`build/fedora/components/rkt`](build/fedora/components/rkt)
+
+Please review the scripts and adjust the paths according to your requirements.
+
+Once the dependencies are installed, you can build stage1-xen
+
+```
+git clone https://github.com/rkt/stage1-xen.git
+cd stage1-xen

[Xen-devel] [stage1-xen PATCH v2 11/11] README.md: Mention preview status

2017-09-19 Thread Rajiv Ranganath 
From: Rajiv M Ranganath <rajiv.rangan...@atihita.com>

Signed-off-by: Rajiv Ranganath <rajiv.rangan...@atihita.com>
---
 README.md |1 +
 1 file changed, 1 insertion(+)

diff --git a/README.md b/README.md
index e1cd40c..d1ae6b9 100644
--- a/README.md
+++ b/README.md
@@ -12,6 +12,7 @@ Stage1s come in the form of 
[ACI](https://github.com/appc/spec) images, and they
 ```
 This project aims at providing a new stage1 based on the Xen hypervisor. Each 
[pod](https://coreos.com/rkt/docs/latest/subcommands/run.html#run-multiple-applications-in-the-same-pod)
 (a small set of contained applications) is run in a separated Xen virtual 
machine. On x86 PV and PVH virtual machines are used, depending on the 
availability of hardware virtualization support.
 
+**Note:** stage1-xen is under active development. We encourage you to try 
stage1-xen and give us your feedback. However unlike upstream [Xen 
Project](https://wiki.xenproject.org/wiki/Xen_Project_Release_Features), we are 
still in _preview_ phase. So please do not expect _long term support_ or 
_backward compatibility_ as yet.
 
 ## Build and Output
 


___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

[Xen-devel] [stage1-xen PATCH v2 09/11] build/fedora: Add `RUNNING_STAGE1_XEN.md`

2017-09-19 Thread Rajiv Ranganath 
We are installing xen in `/opt/` directory. By keeping builds isolated in
`/opt/` we can test multiple versions of xen during development. In
production a simliar approach can be used to support multiple versions of
xen along with a higher level tool that can do rolling updates and
rollbacks.

Signed-off-by: Rajiv Ranganath <rajiv.rangan...@atihita.com>
---
 build/fedora/RUNNING_STAGE1_XEN.md |  391 
 1 file changed, 391 insertions(+)
 create mode 100644 build/fedora/RUNNING_STAGE1_XEN.md

diff --git a/build/fedora/RUNNING_STAGE1_XEN.md 
b/build/fedora/RUNNING_STAGE1_XEN.md
new file mode 100644
index 000..f87f51e
--- /dev/null
+++ b/build/fedora/RUNNING_STAGE1_XEN.md
@@ -0,0 +1,391 @@
+# Running Stage1 Xen on Fedora
+
+This document outlines the steps to get started with stage1-xen on Fedora. 
They are 
+
+ * [Preparing your machine and installing minimal 
Fedora](#preparing_your_machine_and_installing_minimal_fedora)
+ * [Booting into Xen](#booting_into_xen)
+ * [Launching Xen services](#launching_xen_services)
+ * [Setting up Xen networking](#setting_up_xen_networking)
+ * [Running stage1-xen](#running_stage1-xen)
+
+
+## Preparing your machine and installing minimal Fedora
+
+On x86 platform there are two ways to start an operating system or a 
hypervisor. They are 
+
+ * Legacy BIOS Mode
+ * EFI Mode
+
+Latest operating systems and hypervisors including Fedora and Xen has support 
for EFI mode. If you are unfamiliar with EFI we recommend checking out this 
[article](http://www.rodsbooks.com/efi-bootloaders/principles.html).
+
+By default, most BIOS now boot using EFI Mode. In your BIOS menu, there might 
be an option to toggle _Legacy BIOS Mode_. Do not toggle that option.
+
+### Enable VT-x and VT-d
+
+Please ensure that you have enabled VT-x and if available VT-d.
+
+### Disable Secure Boot
+
+As we will be booting a custom build of Xen, we need to disable secure boot. 
You will find an option in your BIOS menu to disable secure boot.
+
+### Installing minimal Fedora
+
+The default Fedora installation installs packages that we do not require when 
running Xen. We recommend doing a minimal Fedora as follows.
+
+ 1. Download Net Install image
+
+ 2. Prepare a USB drive
+
+ 3. Do a minimal Fedora Install
+
+You can download the Fedora net install image 
[here](https://alt.fedoraproject.org/). You can select either the Fedora Server 
or Fedora Workstation image, it doesn't really matter.
+
+After downloading the net install images, please copy the raw image onto a USB 
drive. Please see 
[this](https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/html/Installation_Guide/sect-making-usb-media.html)
 link on how to prepare USB drive.
+
+EFI BIOS comes with a _BIOS Boot Menu_ using which you can select the device 
to boot from. Insert the USB drive, then go into your BIOS Boot Menu and boot 
using the USB drive. This should start the Fedora Network Installer.
+
+In the Fedora Installer, there is a section for under _SOFTWARE_ called 
_SOFTWARE SELECTION_. In this section please **select** either _Minimal 
Install_ or _Basic Desktop_, **without** any add-ons.
+
+**Note:** If there is existing data on the hard disk, please ensure that 
_INSTALLATION DESTINATION_ under _SYSTEM_ section is appropriately configured.
+
+Then click on _Begin Installation_ to complete the installation.
+
+Once the installation is complete, please disable SELinux by editing 
`/etc/selinux/config`.
+
+You now have a minimal Fedora Installation, which is good for working with Xen.
+
+
+## Booting into Xen
+
+Build and install Xen and stage1-xen. Please see 
[BUILDING.md](/BUILDING.md#build_fedora).
+
+If you followed the container build with Docker, then copy over 
`stage1-xen-build.tar.gz`. Extract `stage1-xen-build.tar.gz` into `/opt` 
directory.
+
+```shell
+[root@localhost ~]# tar zxvf stage1-xen-build.tar.gz -C /opt
+
+[root@localhost ~]# ls /opt
+qemu-2.10.0  stage1-xen  xen-4.9.0  xen-4.9.0-runit
+```
+
+This will extract all the build artifacts into `/opt` directory.
+
+Next we will create a BIOS Boot Menu entry to boot `xen-4.9.0.efi`. This will 
start Xen hypervisor. Xen will then start Fedora as Dom-0 guest.
+
+On Fedora, EFI system partition (ESP) is usually mounted at `/boot/efi`. This 
is a `vfat` partition. You can check if EFI system partition is mounted as 
follows 
+
+```shell
+[root@localhost ~]# mount | grep '\/boot\/efi'
+/dev/sda1 on /boot/efi type vfat 
(rw,relatime,fmask=0077,dmask=0077,codepage=437,iocharset=ascii,shortname=winnt,errors=remount-ro)
+```
+
+Create a directory for Xen under `/boot/efi/EFI` and copy over `xen-4.9.0.efi`.
+
+```shell
+[root@localhost ~]# mkdir -p /boot/efi/EFI/xen
+[root@localhost ~]# cp /opt/xen-4.9.0/boot/efi/EFI/xen/xen-4.9.0.efi 
/boot/efi/EFI/xen/
+```
+
+Inspect `/boot/efi/EFI/fedora/grub.cfg`. Under section `### BEGIN 
/etc/grub.d/10_linux ###` you will find `menuentry` for Fedora kernel and 
initrd

[Xen-devel] [stage1-xen PATCH v2 07/11] .circleci/config.yml: Add

2017-09-19 Thread Rajiv Ranganath 
From: Rajiv M Ranganath <rajiv.rangan...@atihita.com>

Signed-off-by: Rajiv Ranganath <rajiv.rangan...@atihita.com>
---
 .circleci/config.yml |   21 +
 1 file changed, 21 insertions(+)
 create mode 100644 .circleci/config.yml

diff --git a/.circleci/config.yml b/.circleci/config.yml
new file mode 100644
index 000..ba2e284
--- /dev/null
+++ b/.circleci/config.yml
@@ -0,0 +1,21 @@
+version: 2
+jobs:
+  build:
+working_directory: /root
+docker:
+  - image: lambdalinuxfedora/stage1-xen-fedora-buildroot:1709190549
+command: /sbin/my_init
+steps:
+  - run:
+  # We create `stage1-xen` directory in Dockerfile for local dev
+  # environment. Removing it here so CircleCI checkout step can work
+  # correctly
+  name: Removing stage1-xen directory from GOPATH...
+  command: |
+rm -rf /root/gopath/src/github.com/rkt/stage1-xen
+  - checkout:
+  path: /root/gopath/src/github.com/rkt/stage1-xen
+  - run:
+  name: Starting run...
+  command: |
+/root/bin/run


___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

[Xen-devel] [stage1-xen PATCH v2 05/11] build/fedora: Add `source_path.sh`

2017-09-19 Thread Rajiv Ranganath 
From: Rajiv M Ranganath <rajiv.rangan...@atihita.com>

Signed-off-by: Rajiv Ranganath <rajiv.rangan...@atihita.com>
---
 build/fedora/source_path.sh |3 +++
 1 file changed, 3 insertions(+)
 create mode 100755 build/fedora/source_path.sh

diff --git a/build/fedora/source_path.sh b/build/fedora/source_path.sh
new file mode 100755
index 000..8076af3
--- /dev/null
+++ b/build/fedora/source_path.sh
@@ -0,0 +1,3 @@
+#!/bin/sh
+
+export 
PATH=/opt/xen-4.9.0/sbin:/opt/xen-4.9.0/bin:/opt/stage1-xen/bin:/opt/qemu-2.10.0/bin:$PATH


___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

[Xen-devel] [stage1-xen PATCH v2 03/11] build/fedora: Add `buildroot-Dockerfile`

2017-09-19 Thread Rajiv Ranganath 
From: Rajiv M Ranganath <rajiv.rangan...@atihita.com>

Signed-off-by: Rajiv Ranganath <rajiv.rangan...@atihita.com>
Reviewed-by: Stefano Stabellini <sstabell...@kernel.org>
---
 build/fedora/buildroot-Dockerfile |  113 +
 1 file changed, 113 insertions(+)
 create mode 100644 build/fedora/buildroot-Dockerfile

diff --git a/build/fedora/buildroot-Dockerfile 
b/build/fedora/buildroot-Dockerfile
new file mode 100644
index 000..971560e
--- /dev/null
+++ b/build/fedora/buildroot-Dockerfile
@@ -0,0 +1,113 @@
+# tarballs checksum
+# -
+# 974b3091232d781c4fc410ccca98fb62ba9febe9e6a988e348804483c4f66742  
binutils-2.26.1-1.1.fc25.tar
+
+FROM lambdalinuxfedora/baseimage-fedora
+
+CMD ["/sbin/my_init"]
+
+COPY [ \
+  "./binutils-2.26.1-1.1.fc25.tar", \
+  \
+  "./components/*", \
+  "./run", \
+  "/tmp/docker-build/" \
+]
+
+RUN \
+  # dnf
+  echo "Running dnf update..." && \
+  dnf update -y && \
+  dnf install -y less && \
+  dnf install -y sudo && \
+  \
+  # circleci container requirements
+  # 
https://circleci.com/docs/2.0/custom-images/#adding-required-and-custom-tools-or-files
+  dnf install -y git && \
+  dnf install -y openssh-clients && \
+  dnf install -y tar && \
+  dnf install -y gzip && \
+  dnf install -y ca-certificates && \
+  \
+  # install `binutils`
+  pushd /tmp/docker-build && \
+# verify checksum
+echo "974b3091232d781c4fc410ccca98fb62ba9febe9e6a988e348804483c4f66742  
binutils-2.26.1-1.1.fc25.tar" | sha256sum -c - && \
+tar xvf binutils-2.26.1-1.1.fc25.tar && \
+dnf install -y 
./binutils/2.26.1/1.1.fc25/x86_64/binutils-2.26.1-1.1.fc25.x86_64.rpm && \
+  popd && \
+  \
+  dnf install -y @buildsys-build && \
+  \
+  # Having `ipdb` around is useful when debugging `run` script. Uncomment this
+  # section as required
+  # dnf install -y python2-devel && \
+  # dnf install -y python-pip && \
+  # su -l root -c "pip2 install --user ipdb==0.8 ipython==5.3.0" && \
+  \
+  # Note: xen and qemu has some duplicate package dependencies. We are
+  # explicitly calling out dependencies for xen and qemu
+  #
+  # xen build dependencies
+  dnf install -y bridge-utils && \
+  dnf install -y gettext && \
+  dnf install -y glib2-devel && \
+  dnf install -y glibc-devel.i686 && \
+  dnf install -y grub2 && \
+  dnf install -y iasl && \
+  dnf install -y libaio-devel && \
+  dnf install -y libuuid-devel && \
+  dnf install -y ncurses-devel && \
+  dnf install -y openssl-devel && \
+  dnf install -y pixman-devel && \
+  dnf install -y python2-devel && \
+  dnf install -y wget && \
+  dnf install -y yajl-devel && \
+  \
+  # qemu build dependencies
+  dnf install -y glib2-devel && \
+  dnf install -y libaio-devel && \
+  dnf install -y libattr-devel && \
+  dnf install -y libcap-devel && \
+  dnf install -y libcap-ng-devel && \
+  dnf install -y pixman-devel && \
+  dnf install -y zlib-devel && \
+  \
+  # rkt build dependencies
+  dnf install -y autoconf && \
+  dnf install -y automake && \
+  dnf install -y git && \
+  dnf install -y glibc-static && \
+  dnf install -y gnupg && \
+  dnf install -y golang && \
+  dnf install -y libacl-devel && \
+  dnf install -y squashfs-tools && \
+  dnf install -y systemd-devel && \
+  dnf install -y wget && \
+  \
+  # stage1-xen build dependencies
+  dnf install -y bc && \
+  dnf install -y busybox && \
+  dnf install -y glide && \
+  dnf install -y golang && \
+  dnf install -y jq && \
+  dnf install -y libacl-devel && \
+  dnf install -y wget && \
+  \
+  # copy `run` file and `components/{qemu,rkt,xen}`
+  su -l root -c "mkdir /root/bin" && \
+  su -l root -c "cp /tmp/docker-build/run /root/bin" && \
+  su -l root -c "mkdir /root/bin/components" && \
+  su -l root -c "cp /tmp/docker-build/qemu /root/bin/components" && \
+  su -l root -c "cp /tmp/docker-build/rkt /root/bin/components" && \
+  su -l root -c "cp /tmp/docker-build/xen /root/bin/components" && \
+  \
+  # create `stage1-xen` directory
+  mkdir -p /root/gopath/src/github.com/rkt/stage1-xen && \
+  \
+  # cleanup
+  rm -rf /tmp/docker-build && \
+  dnf clean all && \
+  rm -rf /var/cache/dnf/* && \
+  rm -rf /tmp/* && \
+  rm -rf /var/tmp/*


___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

[Xen-devel] [stage1-xen PATCH v2 01/11] .gitignore: Add

2017-09-19 Thread Rajiv Ranganath 
From: Rajiv M Ranganath <rajiv.rangan...@atihita.com>

Signed-off-by: Rajiv Ranganath <rajiv.rangan...@atihita.com>
Reviewed-by: Stefano Stabellini <sstabell...@kernel.org>
---
 .gitignore |2 ++
 1 file changed, 2 insertions(+)
 create mode 100644 .gitignore

diff --git a/.gitignore b/.gitignore
new file mode 100644
index 000..873f8f6
--- /dev/null
+++ b/.gitignore
@@ -0,0 +1,2 @@
+# build/fedora
+build/fedora/binutils-*.tar


___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

[Xen-devel] [stage1-xen PATCH v2 02/11] build/fedora: Add `buildroot-README.md`

2017-09-19 Thread Rajiv Ranganath 
From: Rajiv M Ranganath <rajiv.rangan...@atihita.com>

Signed-off-by: Rajiv Ranganath <rajiv.rangan...@atihita.com>
Reviewed-by: Stefano Stabellini <sstabell...@kernel.org>
---
 build/fedora/buildroot-README.md |   50 ++
 1 file changed, 50 insertions(+)
 create mode 100644 build/fedora/buildroot-README.md

diff --git a/build/fedora/buildroot-README.md b/build/fedora/buildroot-README.md
new file mode 100644
index 000..0efb150
--- /dev/null
+++ b/build/fedora/buildroot-README.md
@@ -0,0 +1,50 @@
+## stage1-xen Fedora Buildroot
+
+stage1-xen build artifacts for Fedora is built in two phases. In the first 
phase
+a docker container is prepared with all the build dependencies. We refer to it
+as `stage1-xen-fedora-buildroot`. In the next phase we execute the `run` script
+that uses `stage1-xen-fedora-buildroot` and to produce the build artifacts.
+
+### Building `stage1-xen-fedora-buildroot`
+
+`stage1-xen-fedora-buildroot` has a external dependency
+on [`binutils`](https://github.com/lambda-linux-fedora/binutils) package that 
is
+compiled with `i386pe` support. You can download the pre-built RPMs
+from [here](https://drive.google.com/open?id=0B_tTbuxmuRzIR05wQ3E1eWVyaGs).
+Please download `binutils-2.26.1-1.1.fc25.tar`.
+
+To build docker image
+
+```
+cd stage1-xen/build/fedora
+
+docker build -f buildroot-Dockerfile -t stage1-xen-fedora-buildroot .
+```
+
+### Running `stage1-xen-fedora-buildroot`
+
+```
+cd stage1-xen
+
+docker run --rm \
+  -v `pwd`:/root/gopath/src/github.com/rkt/stage1-xen \
+  -v /tmp:/tmp \
+  -t -i stage1-xen-fedora-buildroot \
+  /sbin/my_init -- /root/bin/run
+```
+
+The generated build artifacts are in `/tmp` directory.
+
+To debug build issues -
+
+```
+cd stage1-xen
+
+docker run --rm \
+  -v `pwd`:/root/gopath/src/github.com/rkt/stage1-xen \
+  -v /tmp:/tmp \
+  -t -i stage1-xen-fedora-buildroot \
+  /sbin/my_init -- /bin/bash
+```
+
+Also see section on `ipdb` in `buildroot-Dockerfile`.


___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

[Xen-devel] [stage1-xen PATCH v2 04/11] build/fedora: Add `run` and `components/*` scripts

2017-09-19 Thread Rajiv Ranganath 
From: Rajiv M Ranganath <rajiv.rangan...@atihita.com>

In order to build stage1-xen we require three components -

1. Xen
2. Qemu
3. Rkt

These components are built using scripts in `build/fedora/components/`
directory. These scripts have constants (for example `XEN_PREFIX`) that can
be used to customize the build either on the host or while building inside
a container.

`run` script makes use of scripts in `components/` directory to build
stage1-xen artifacts in a container.

Signed-off-by: Rajiv Ranganath <rajiv.rangan...@atihita.com>
---
 build/fedora/components/qemu |   52 +
 build/fedora/components/rkt  |   59 ++
 build/fedora/components/xen  |   47 +
 build/fedora/run |   56 
 4 files changed, 214 insertions(+)
 create mode 100755 build/fedora/components/qemu
 create mode 100755 build/fedora/components/rkt
 create mode 100755 build/fedora/components/xen
 create mode 100755 build/fedora/run

diff --git a/build/fedora/components/qemu b/build/fedora/components/qemu
new file mode 100755
index 000..06538d8
--- /dev/null
+++ b/build/fedora/components/qemu
@@ -0,0 +1,52 @@
+#!/usr/bin/python2
+
+import shlex
+import subprocess
+import sys
+import os
+
+# Modify this if you would like to install Qemu elsewhere on your filesystem or
+# a different version of Qemu
+QEMU_PREFIX = '/opt/qemu-2.10.0'
+# This can be a git tag or branch
+QEMU_BRANCH = 'v2.10.0'
+
+# This should correspond to your Xen install prefix
+XEN_PREFIX = '/opt/xen-4.9.0'
+
+
+# helper function to capture stdout from a long running process
+def subprocess_stdout(cmd, cwd, env):
+p = subprocess.Popen(
+shlex.split(cmd), cwd=cwd, env=env, stdout=subprocess.PIPE)
+while p.poll() is None:
+l = p.stdout.readline()
+sys.stdout.write(l)
+if p.returncode != 0:
+sys.exit(1)
+
+
+env = os.environ.copy()
+
+# build and install qemu
+print "Cloning qemu..."
+cmd = "git clone --branch %(branch)s git://git.qemu.org/qemu.git" % {
+'branch': QEMU_BRANCH
+}
+subprocess.check_output(shlex.split(cmd), cwd='/root')
+
+steps = [
+"./configure --prefix=%(qemu_prefix)s --enable-xen 
--target-list=i386-softmmu --extra-cflags=\"-I%(xen_prefix)s/include\" 
--extra-ldflags=\"-L%(xen_prefix)s/lib -Wl,-rpath,%(xen_prefix)s/lib\" 
--disable-kvm --enable-virtfs --enable-linux-aio"
+% {
+'qemu_prefix': QEMU_PREFIX,
+'xen_prefix': XEN_PREFIX
+}, 'make', 'make install'
+]
+for cmd in steps:
+cwd = '/root/qemu'
+subprocess_stdout(cmd, cwd, env)
+
+cmd = "cp i386-softmmu/qemu-system-i386 
%(xen_prefix)s/lib/xen/bin/qemu-system-i386" % {
+'xen_prefix': XEN_PREFIX
+}
+subprocess.check_output(shlex.split(cmd), cwd='/root/qemu')
diff --git a/build/fedora/components/rkt b/build/fedora/components/rkt
new file mode 100755
index 000..a8d6541
--- /dev/null
+++ b/build/fedora/components/rkt
@@ -0,0 +1,59 @@
+#!/usr/bin/python2
+
+import shlex
+import subprocess
+import sys
+import os
+
+# `rkt` is installed in the same prefix as `stage1-xen`. Modify this if you
+# would like to install rkt elsewhere on your filesystem.
+STAGE1_XEN_PREFIX = '/opt/stage1-xen'
+RKT_PREFIX = STAGE1_XEN_PREFIX
+# This can be a git tag or branch
+RKT_BRANCH = 'v1.28.1'
+
+# Adjust this according to what RKT_BRANCH generates (see configure.ac)
+RKT_BUILD_VER = 'rkt-1.28.1'
+
+
+# helper function to capture stdout from a long running process
+def subprocess_stdout(cmd, cwd, env):
+p = subprocess.Popen(
+shlex.split(cmd), cwd=cwd, env=env, stdout=subprocess.PIPE)
+while p.poll() is None:
+l = p.stdout.readline()
+sys.stdout.write(l)
+if p.returncode != 0:
+sys.exi(1)
+
+
+env = os.environ.copy()
+
+# build rkt
+print "Cloning rkt..."
+cmd = "git clone --branch %(branch)s https://github.com/rkt/rkt.git; % {
+'branch': RKT_BRANCH
+}
+subprocess.check_output(shlex.split(cmd), cwd='/root')
+
+steps = [
+'./autogen.sh', './configure --disable-tpm --with-stage1-flavors=coreos',
+'make'
+]
+for cmd in steps:
+cwd = '/root/rkt'
+subprocess_stdout(cmd, cwd, env)
+
+# install rkt build artifacts to RKT_PREFIX
+steps = [
+"mkdir -p %(prefix)s/bin" % {
+'prefix': RKT_PREFIX
+},
+"cp /root/rkt/build-%(build_ver)s/target/bin/rkt %(prefix)s/bin/rkt" % {
+'build_ver': RKT_BUILD_VER,
+'prefix': RKT_PREFIX
+}
+]
+for cmd in steps:
+cwd = '/root/rkt'
+subprocess_stdout(cmd, cwd, env)
diff --git a/build/fedora/components/xen b/build/fedora/components/xen
new file mode 100755
index 000..45d11f2
--- /dev/null
+++ b/build/fedora/components/xen
@@ -0,0 +1,47 @@
+#!/usr/bin/python2
+
+import shlex
+import subprocess
+import sys
+import os
+
+# Modify this if you

[Xen-devel] [stage1-xen PATCH v2 00/11] Add Fedora support

2017-09-19 Thread Rajiv Ranganath 
Hi Stefano,

I've made the following changes based on v1 series feedback.

Kindly please review.

build/fedora: Add `run` and `components/*` scripts
- Updated commit message
- Switched to Xen 4.9.0, Qemu 2.10.0, Rkt 1.28.1

build/fedora: Add `source_path.sh`
- Updated paths to qemu and xen

build/fedora: Add `xen-4.9.0-runit/*` scripts
- Changed from `xen-unstable-runit` to `xen-4.9.0-runit`

.circleci/config.yml: Add
- Updated CI docker image

build/fedora: Add `RUNNING_STAGE1_XEN.md`
- Updated documentation to refer to xen-4.9.0 instead of
  xen-4.10-unstable
- Updated commit message to describe the rationale for installing in
  `/opt/`

README.md: Mention preview status
- New

Here is a temporary repo [1] with v2 series.

Best,
Rajiv

[1]: https://github.com/lambda-linux-fedora/stage1-xen/tree/wip2

___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] [stage1-xen PATCH v1 04/10] build/fedora: Add `run` and `components/*` scripts

2017-09-12 Thread Rajiv Ranganath 

On Tue, Sep 12 2017 at 01:36:04 AM, Stefano Stabellini  
wrote:

[...]

> Fortunately, from the stage1-xen code point of view, there is very
> little difference between PVHv2 and PV. Switching from one to the
> other should be a matter of adding one line to the xl config file.

There is a related use-case here that I think will be important to
users.

In stage1-xen we are packaging a Dom-U kernel. When this kernel crashes
we would want to capture its crash log. Depending on the nature of the
issue, users can then work with their own kernel team, vendor (who is
open to supporting LTS kernels) or upstream.

We might also want to consider supporting two LTS kernel versions on a
rolling basis. Users can then use something like labels [1] or
annotations [2] to toggle the kernel version. That way if their
containers start crashing under a newer Dom-U kernel, they can roll back
to a working kernel.

[...]

>> 3. Multiboot2 - One of the reasons why I documented using EFI is because
>> I could not get multiboot2 to work. It looks like the fix for it is on
>> its way. I anticipate using multiboot2 would be easier for users.
>
> That's for the host right? I didn't have that problem, but maybe because
> I am not using Fedora.

That's correct! I ran into this issue on Fedora host.

[...]

> You have a good point. I think we should be clear about the stability
> of the project and the backward compatibility in the README. We should
> openly say that it is still a "preview" and there is no "support" or
> "compatibility" yet.

Sounds good. I'll update README to reflect this.

> Choosing Xen 4.9 should not be seen as a statement of support. I think
> we should choose the Xen version based only on the technical merits.
>
> In the long term it would be great to support multiple stable versions
> and a development version of Xen. As of now, I think it makes sense to
> have an "add-hoc approach": I would use Xen 4.9 just because it is the
> best choice at the moment. Then, I would update to other versions when
> it makes sense, manually. I don't think that building against a changing
> target ("master") is a good idea, because we might end up stumbling
> across confusing and time-consuming bugs that have nothing to do with
> stage1-xen. However, we could pick a random commit on the Xen tree if
> that's convenient for us, because at this stage there is no support
> really. For example, PVCalls will require some tools changes in Xen.
> Once they are upstream, we'll want to update the Xen version to the
> latest with PVCalls support.
>
> Does it make sense?

Yes, it does. I'll switch to xen-4.9, qemu-2.10 and rkt-1.28 in the next
version of the patchset.

Best,
Rajiv

[1] https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/
[2] 
https://kubernetes.io/docs/concepts/overview/working-with-objects/annotations/

___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] [stage1-xen PATCH v1 06/10] build/fedora: Add `xen-unstable-runit/*` scripts

2017-09-12 Thread Rajiv Ranganath 

On Tue, Sep 12 2017 at 04:38:19 AM, Andrew Cooper  
wrote:
> On 11/09/2017 21:20, Stefano Stabellini wrote:

[...]

>> My only concern is about diverging from the upstream Xen codebase. I
>> think the runit scripts should call xencommons underneath. If xencommons
>> cannot cope with being called from runit, we could make changes to
>> xencommon in xen.git to make it so.
>>
>> Otherwise, we will end up in a situation such as:
>> - xen.git changes xencommons
>> - we don't notice
>> - we upgrade Xen version
>> - stage1-xen doesn't work anymore
>>
>> If we used xencommons underneath we would avoid this, and it looks like
>> xencommons could be made to work well with runit.
>
> If possible, upstream Xen should be made to be compatible with runit
> (this would be the ideal case). If not, upstream Xen should contain
> different styles of these files, which are selected between by a
> ./configure option (this is suboptimal, but better than locally
> forking). This offers the greatest chance that updates to one don't
> cause the other to be stale.

I agree that it would be beneficial to have upstream Xen support for
runit. However, runit is packaged differently in every distro.

We work around this issue by packaging our own version of runit [1].
Fedora does not include runit in its repositories. That helps because we
don't have to worry about conflicting with distro packaged runit.

One option to consider is for xen project to package its own version of
runit for major distros (we will have one for Fedora in stage1-xen), and
use that as the basis for runit support.

Since stage1-xen is still under development, maybe we can use runit in
stage1-xen as a testing ground. If things work out well, we can then see
how best to integrate with xencommons or add a configure option.

By then we will also know if there is broader community interest in
having runit support in xen.

As to changes to xencommons breaking stage1-xen, as we get closer to
stable release, we probably will have integration tests to catch this
and many other things! :-)

Best,
Rajiv

[1]: https://github.com/lambda-linux-fedora/runit/tree/ver-2.1.2-1.1.fc25

___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] [stage1-xen PATCH v1 09/10] build/fedora: Add `RUNNING_STAGE1_XEN.md`

2017-09-08 Thread Rajiv Ranganath 

On Sat, Sep 09 2017 at 07:40:17 AM, Rajiv Ranganath 
<rajiv.rangan...@atihita.com> wrote:

[...]

>>
>> and do the same for the other components.
>
> Yes, we can do that, but I feel its a good idea. :-)
 

I meant to say _not_ a good idea! Sorry for the typo!

Best,
Rajiv

___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] [stage1-xen PATCH v1 09/10] build/fedora: Add `RUNNING_STAGE1_XEN.md`

2017-09-08 Thread Rajiv Ranganath 

On Thu, Sep 07 2017 at 12:44:16 AM, Stefano Stabellini  
wrote:

[...]

>> +[root@localhost ~]# ls /opt
>> +qemu-unstable  stage1-xen  xen-unstable  xen-unstable-runit
>> +```
>> +
>> +This will extract all the build artifacts into `/opt` directory.
>
> Is there a reason to keep all the binaries under /opt? I mean, at this
> point, we could do something like
>
>   cp -ar /opt/xen-unstable/* /
>
> and do the same for the other components.

Yes, we can do that, but I feel its a good idea. :-)

Outside of specific paths (such as /var or /etc), its better to let RPM
manage files in the / hierarchy. That way rpm -qf can return sensible
results when we need to login and debug issues.

> Do we keep them under /opt for ease of management, so that the next time
> we do a build, we can easily test with a different Xen version? Or is
> there another reason?

That's correct. Keeping things isolated in /opt lets us test different
versions of xen during development. In production we can use the same
approach to support multiple versions of xen and do rolling updates or
rollbacks.

Btw, I should point out that this is not something new. NixOS has been
using the approach of building packages in separate filesystem hierarchy
for a while now.

We are just selectively adopting their ideas.

Best,
Rajiv

___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] [stage1-xen PATCH v1 04/10] build/fedora: Add `run` and `components/*` scripts

2017-09-08 Thread Rajiv Ranganath 

On Thu, Sep 07 2017 at 12:29:54 AM, Stefano Stabellini  
wrote:

[...]

>> +QEMU_BRANCH = 'master'
>
> I am not sure we want to checkout always the latest QEMU. It is a
> running target. It makes sense to use one of the latest releases
> instead, such as v2.10.0?
>

[...]

I feel once we have an understanding around what stable xen container
experience for our users should be, it makes a lot of sense to support
two stable versions (on a rolling basis) along with unstable/devel
versions of xen, qemu and rkt.

I am hoping we can include the following before adding support for
stable version.

1. Kernel - PV Calls backend support will be in 4.14, which is few
months away.

2. PVHv2 - xl and PVHv2 support is inflight for 4.10. I would like to
see xen container users start off with PVHv2 and using PV Calls
networking. Therefore I am a bit hesitant adding support for Xen 4.9.

3. Multiboot2 - One of the reasons why I documented using EFI is because
I could not get multiboot2 to work. It looks like the fix for it is on
its way. I anticipate using multiboot2 would be easier for users.

4. Rkt - Support for Kubernetes CRI and OCI image format will be of
importance to our users. Rkt is working on it but I'm not sure of their
progress. There are other projects that are also incubating in CNCF -
cri-o and cri-containerd.

PV Calls networking is new to me, and I wanted to do some prototyping to
understand how it would integrate with the rest of the container
ecosystem it after landing this series.

By adding support for xen-4.9, qemu-2.10 or rkt-1.28.1 I feel we should
not set some kind stability or backward compatibility expectations
around stage1-xen as yet.

My preference would be to keep things on master (albeit deliberately)
till we can figure out a good xen container experience for our users.

Please let me know what you think.

>> +if p.returncode != 0:
>> +sys.exit(1)
>
> Is this the same as
>   #!/bin/bash
>   set -e
> ?

That's right. 

> Please add a few words in the commit message about the benefit of this
> approach of writing scripts.
>

I'll update the commit message in the next version of the series.

Best,
Rajiv


___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] [stage1-xen PATCH v1 06/10] build/fedora: Add `xen-unstable-runit/*` scripts

2017-09-08 Thread Rajiv Ranganath 

On Thu, Sep 07 2017 at 12:10:21 AM, Stefano Stabellini  
wrote:

[...]

> The series is much better now thank you. One question: why did you write
> your own init scripts rather than reusing xencommons (with the caveat
> that you would have to add make sure to source_path.sh before running
> xencommons)?  Does it have something to do with systemd?

There are a few related reasons for this.

1. Using runit lets us abstract out our dependency on systemd and
upstart. We can use the same abstraction in containers [1], virtual
machines and on bare metal.

2. In Linux distributions, there is tight coupling between package
management system (rpm/deb), init systems (upstart/systemd), and service
daemons.

With containers, if the expectation is that most service daemons and
apps would be containerized, and managed by a node agent then a natural
question to ask would be what should be the role of init systems like
systemd?

By using runit (on systemd, upstart and within containers), we defer
answering this question. 

3. One of the use cases that we want to support is to have different
versions of xen co-exist on the same filesystem. Then a higher level
tool can do rolling updates and if required rollbacks.

While it is possible to accomplish this on existing init systems,
depending on how xen is packaged and deployed, it might involve using
distro package and repository management tools.

With runit, we can use regular docker tools, which is much more friendly
for mainstream developers and CI systems. We also abstract over init
systems, which is a desirable property to have.

4. I looked into xencommons script and systemd unit files when creating
runit scripts. Our runit scripts is straightforward translation of how
one would start xen manually.

Perhaps the only part of the script that might need some explanation is
in `xen-init-dom0/run`.

```
exec chpst -b xen-init-dom0 runit-pause
```

This is a pattern used to build equivalent of "oneshot" service in
systemd. It was developed in Ignite (a Arch Linux project before they
switched to systemd) and later co-opted by Void Linux [2].

I am not sure if I answered your question. Sometimes I feel, maybe we
should just let questions around init systems be like one of those
"unanswered questions" in theology. :-) [3]

Best,
Rajiv

[1] https://github.com/lambda-linux/baseimage-amzn#adding_additional_daemons

[2] 
https://github.com/voidlinux/void-runit/commit/7aecf46ec589a5bc49ae2392137bcd0e7468dd08

[3] https://en.wikipedia.org/wiki/The_unanswered_questions

___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

[Xen-devel] [stage1-xen PATCH v1 09/10] build/fedora: Add `RUNNING_STAGE1_XEN.md`

2017-08-26 Thread Rajiv Ranganath 
Signed-off-by: Rajiv Ranganath <rajiv.rangan...@atihita.com>
---
 build/fedora/RUNNING_STAGE1_XEN.md |  391 
 1 file changed, 391 insertions(+)
 create mode 100644 build/fedora/RUNNING_STAGE1_XEN.md

diff --git a/build/fedora/RUNNING_STAGE1_XEN.md 
b/build/fedora/RUNNING_STAGE1_XEN.md
new file mode 100644
index 000..867b8b4
--- /dev/null
+++ b/build/fedora/RUNNING_STAGE1_XEN.md
@@ -0,0 +1,391 @@
+# Running Stage1 Xen on Fedora
+
+This document outlines the steps to get started with stage1-xen on Fedora. 
They are 
+
+ * [Preparing your machine and installing minimal 
Fedora](#preparing_your_machine_and_installing_minimal_fedora)
+ * [Booting into Xen](#booting_into_xen)
+ * [Launching Xen services](#launching_xen_services)
+ * [Setting up Xen networking](#setting_up_xen_networking)
+ * [Running stage1-xen](#running_stage1-xen)
+
+
+## Preparing your machine and installing minimal Fedora
+
+On x86 platform there are two ways to start an operating system or a 
hypervisor. They are 
+
+ * Legacy BIOS Mode
+ * EFI Mode
+
+Latest operating systems and hypervisors including Fedora and Xen has support 
for EFI mode. If you are unfamiliar with EFI we recommend checking out this 
[article](http://www.rodsbooks.com/efi-bootloaders/principles.html).
+
+By default, most BIOS now boot using EFI Mode. In your BIOS menu, there might 
be an option to toggle _Legacy BIOS Mode_. Do not toggle that option.
+
+### Enable VT-x and VT-d
+
+Please ensure that you have enabled VT-x and if available VT-d.
+
+### Disable Secure Boot
+
+As we will be booting a custom build of Xen, we need to disable secure boot. 
You will find an option in your BIOS menu to disable secure boot.
+
+### Installing minimal Fedora
+
+The default Fedora installation installs packages that we do not require when 
running Xen. We recommend doing a minimal Fedora as follows.
+
+ 1. Download Net Install image
+
+ 2. Prepare a USB drive
+
+ 3. Do a minimal Fedora Install
+
+You can download the Fedora net install image 
[here](https://alt.fedoraproject.org/). You can select either the Fedora Server 
or Fedora Workstation image, it doesn't really matter.
+
+After downloading the net install images, please copy the raw image onto a USB 
drive. Please see 
[this](https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/html/Installation_Guide/sect-making-usb-media.html)
 link on how to prepare USB drive.
+
+EFI BIOS comes with a _BIOS Boot Menu_ using which you can select the device 
to boot from. Insert the USB drive, then go into your BIOS Boot Menu and boot 
using the USB drive. This should start the Fedora Network Installer.
+
+In the Fedora Installer, there is a section for under _SOFTWARE_ called 
_SOFTWARE SELECTION_. In this section please **select** either _Minimal 
Install_ or _Basic Desktop_, **without** any add-ons. 
+
+**Note:** If there is existing data on the hard disk, please ensure that 
_INSTALLATION DESTINATION_ under _SYSTEM_ section is appropriately configured. 
+
+Then click on _Begin Installation_ to complete the installation.
+
+Once the installation is complete, please disable SELinux by editing 
`/etc/selinux/config`.
+
+You now have a minimal Fedora Installation, which is good for working with Xen.
+
+
+## Booting into Xen
+
+Build and install Xen and stage1-xen. Please see 
[BUILDING.md](/BUILDING.md#build_fedora).
+
+If you followed the container build with Docker, then copy over 
`stage1-xen-build.tar.gz`. Extract `stage1-xen-build.tar.gz` into `/opt` 
directory.
+
+```shell
+[root@localhost ~]# tar zxvf stage1-xen-build.tar.gz -C /opt
+
+[root@localhost ~]# ls /opt
+qemu-unstable  stage1-xen  xen-unstable  xen-unstable-runit
+```
+
+This will extract all the build artifacts into `/opt` directory.
+
+Next we will create a BIOS Boot Menu entry to boot `xen-4.10-unstable.efi`. 
This will start Xen hypervisor. Xen will then start Fedora as Dom-0 guest.
+
+On Fedora, EFI system partition (ESP) is usually mounted at `/boot/efi`. This 
is a `vfat` partition. You can check if EFI system partition is mounted as 
follows 
+
+```shell
+[root@localhost ~]# mount | grep '\/boot\/efi'
+/dev/sda1 on /boot/efi type vfat 
(rw,relatime,fmask=0077,dmask=0077,codepage=437,iocharset=ascii,shortname=winnt,errors=remount-ro)
+```
+
+Create a directory for Xen under `/boot/efi/EFI` and copy over 
`xen-4.10-unstable.efi`.
+
+```shell
+[root@localhost ~]# mkdir -p /boot/efi/EFI/xen
+[root@localhost ~]# cp 
/opt/xen-unstable/boot/efi/EFI/xen/xen-4.10-unstable.efi /boot/efi/EFI/xen/
+```
+
+Inspect `/boot/efi/EFI/fedora/grub.cfg`. Under section `### BEGIN 
/etc/grub.d/10_linux ###` you will find `menuentry` for Fedora kernel and 
initrd. Look for `linuxefi` and `initrdefi`. Copy over the `vmlinuz` and 
`initramfs` files that you want to use for your Dom-0 into `/boot/efi/EFI/xen` 
directory.
+
+```shell
+[root@localhost ~]# cp /boot/vmlinuz-A.B.C-D.fcXX.x86_64 /boot/efi/EFI/xen/
+
+[root@loc

[Xen-devel] [stage1-xen PATCH v1 08/10] README.md: Add CircleCI badge

2017-08-26 Thread Rajiv Ranganath 
From: Rajiv M Ranganath <rajiv.rangan...@atihita.com>

Signed-off-by: Rajiv Ranganath <rajiv.rangan...@atihita.com>
---
 README.md |2 ++
 1 file changed, 2 insertions(+)

diff --git a/README.md b/README.md
index 9ea6adf..e1cd40c 100644
--- a/README.md
+++ b/README.md
@@ -1,5 +1,7 @@
 # stage1-xen - A Xen based stage1 for CoreOS rkt
 
+[![Build 
Status](https://circleci.com/gh/rkt/stage1-xen/tree/master.svg?style=shield=:circle-token)](https://circleci.com/gh/rkt/stage1-xen/tree/master)
+
 ## Goal
 
 CoreOS rkt is a modular container engine with [three stages of 
execution](https://coreos.com/rkt/docs/latest/devel/stage1-implementors-guide.html).
 Stage1 is responsible for creating the execution environment for the contained 
applications.


___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

[Xen-devel] [stage1-xen PATCH v1 06/10] build/fedora: Add `xen-unstable-runit/*` scripts

2017-08-26 Thread Rajiv Ranganath 
From: Rajiv M Ranganath <rajiv.rangan...@atihita.com>

Signed-off-by: Rajiv Ranganath <rajiv.rangan...@atihita.com>
---
 build/fedora/xen-unstable-runit/setup.sh   |   18 
 build/fedora/xen-unstable-runit/teardown.sh|   18 
 .../xen-init-dom0-disk-backend/run |   11 ++
 build/fedora/xen-unstable-runit/xen-init-dom0/run  |9 
 build/fedora/xen-unstable-runit/xenconsoled/run|   13 +++
 build/fedora/xen-unstable-runit/xenstored/run  |   23 
 6 files changed, 92 insertions(+)
 create mode 100755 build/fedora/xen-unstable-runit/setup.sh
 create mode 100755 build/fedora/xen-unstable-runit/teardown.sh
 create mode 100755 
build/fedora/xen-unstable-runit/xen-init-dom0-disk-backend/run
 create mode 100755 build/fedora/xen-unstable-runit/xen-init-dom0/run
 create mode 100755 build/fedora/xen-unstable-runit/xenconsoled/run
 create mode 100755 build/fedora/xen-unstable-runit/xenstored/run

diff --git a/build/fedora/xen-unstable-runit/setup.sh 
b/build/fedora/xen-unstable-runit/setup.sh
new file mode 100755
index 000..b5adf8c
--- /dev/null
+++ b/build/fedora/xen-unstable-runit/setup.sh
@@ -0,0 +1,18 @@
+#!/bin/bash
+
+set -e
+
+# runit RPM creates `/etc/service` directory
+if [ ! -d "/etc/service" ]; then
+echo "/etc/service directory not found. Please install runit RPM."
+exit 1
+fi
+
+runit_services="xenconsoled xen-init-dom0 xen-init-dom0-disk-backend xenstored"
+
+for service in $runit_services; do
+ln -sf /opt/xen-unstable-runit/$service /etc/service/$service
+done
+
+echo "Successfully created symlinks in /etc/service directory."
+exit 0
diff --git a/build/fedora/xen-unstable-runit/teardown.sh 
b/build/fedora/xen-unstable-runit/teardown.sh
new file mode 100755
index 000..d333807
--- /dev/null
+++ b/build/fedora/xen-unstable-runit/teardown.sh
@@ -0,0 +1,18 @@
+#!/bin/bash
+
+set -e
+
+# runit RPM creates `/etc/service` directory
+if [ ! -d "/etc/service" ]; then
+echo "/etc/service directory not found."
+exit 1
+fi
+
+runit_services="xenconsoled xen-init-dom0 xen-init-dom0-disk-backend xenstored"
+
+for service in $runit_services; do
+rm -f /etc/service/$service
+done
+
+echo "Successfully deleted symlinks in /etc/service directory."
+exit 0
diff --git a/build/fedora/xen-unstable-runit/xen-init-dom0-disk-backend/run 
b/build/fedora/xen-unstable-runit/xen-init-dom0-disk-backend/run
new file mode 100755
index 000..6315d48
--- /dev/null
+++ b/build/fedora/xen-unstable-runit/xen-init-dom0-disk-backend/run
@@ -0,0 +1,11 @@
+#!/bin/bash
+
+set -e
+
+sv check xenstored >/dev/null || exit 1
+sv check xenconsoled >/dev/null || exit 1
+
+# In case of failure, allow user to run teardown script
+sleep 5s
+
+exec /opt/xen-unstable/lib/xen/bin/qemu-system-i386 -xen-domid 0 -xen-attach 
-name dom0 -nographic -M xenpv -monitor /dev/null -serial /dev/null -parallel 
/dev/null -nodefaults -no-user-config
diff --git a/build/fedora/xen-unstable-runit/xen-init-dom0/run 
b/build/fedora/xen-unstable-runit/xen-init-dom0/run
new file mode 100755
index 000..193ba19
--- /dev/null
+++ b/build/fedora/xen-unstable-runit/xen-init-dom0/run
@@ -0,0 +1,9 @@
+#!/bin/bash
+
+set -e
+
+sv check xenstored >/dev/null || exit 1
+
+/opt/xen-unstable/lib/xen/bin/xen-init-dom0
+
+exec chpst -b xen-init-dom0 runit-pause
diff --git a/build/fedora/xen-unstable-runit/xenconsoled/run 
b/build/fedora/xen-unstable-runit/xenconsoled/run
new file mode 100755
index 000..b5b7a9f
--- /dev/null
+++ b/build/fedora/xen-unstable-runit/xenconsoled/run
@@ -0,0 +1,13 @@
+#!/bin/bash
+
+set -e
+
+sv check xen-init-dom0 >/dev/null || exit 1
+
+[ ! -d /var/log/xen/console ] && mkdir -p /var/log/xen/console
+
+# In case of failure, allow user to run teardown script
+sleep 5s
+
+# --log=[none|guest|hv|all]
+exec /opt/xen-unstable/sbin/xenconsoled -i --log=none
diff --git a/build/fedora/xen-unstable-runit/xenstored/run 
b/build/fedora/xen-unstable-runit/xenstored/run
new file mode 100755
index 000..beb2a5f
--- /dev/null
+++ b/build/fedora/xen-unstable-runit/xenstored/run
@@ -0,0 +1,23 @@
+#!/bin/bash
+
+set -e
+
+[ ! -d /var/run/xen ] && mkdir -p /var/run/xen
+[ ! -d /var/run/xenstored ] && mkdir -p /var/run/xenstored
+[ ! -d /var/log/xen ] && mkdir -p /var/log/xen
+[ ! -d /var/lib/xen ] && mkdir -p /var/lib/xen
+[ ! -d /var/lib/xen/dump ] && mkdir -p /var/lib/xen/dump
+[ ! -d /var/lib/xen/xenpaging ] && mkdir -p /var/lib/xen/paging
+[ ! -d /var/lib/xenstored ] && mkdir -p /var/lib/xenstored
+modprobe -q xen-evtchn || exit 1
+modprobe -q xen-gntdev || exit 1
+modprobe -q xen-gntalloc || exit 1
+mountpoint -q /proc/xen || mount -t xenfs xenfs /proc/xen
+mountpoint -q /var/lib/xenstored || mount -t tmpfs xenstored /var/lib/xenstored
+grep -q "contr

[Xen-devel] [stage1-xen PATCH v1 10/10] BUILDING.md: Add Fedora instructions

2017-08-26 Thread Rajiv Ranganath 
Signed-off-by: Rajiv Ranganath <rajiv.rangan...@atihita.com>
---
 BUILDING.md |   96 ---
 1 file changed, 91 insertions(+), 5 deletions(-)

diff --git a/BUILDING.md b/BUILDING.md
index 3ef5311..946c799 100644
--- a/BUILDING.md
+++ b/BUILDING.md
@@ -1,7 +1,13 @@
 # Build
-stage1-xen requires new Xen and QEMU versions at the time of writing. You are 
unlikely to find them already packaged with your distro. This document 
describes how to build and install the latest Xen and QEMU from scratch. In 
addition, given that CoreOS rkt is also missing from reasonably new distros 
such as Ubuntu Xenial Xerus, I added instructions on how to build that too. The 
document includes the dependencies needed for the build based on Ubuntu Xenial 
Xerus.
+stage1-xen requires new Xen and QEMU versions at the time of writing. You are 
unlikely to find them already packaged with your distro. This document 
describes how to build and install the latest Xen, QEMU and rkt from scratch 
for Ubuntu Xenial Xerus and Fedora. Differently from documentation for Ubuntu, 
the documentation for Fedora uses a Docker container for the build. There is 
also support for building on host on Fedora.
 
-## Building Xen
+ * [Ubuntu Xenial Xerus](#build_ubuntu)
+ * [Fedora](#build_fedora)
+
+
+## Ubuntu Xenial Xerus
+
+### Building Xen
 ```
 apt-get install git build-essential python-dev gettext uuid-dev 
libncurses5-dev libyajl-dev libaio-dev pkg-config libglib2.0-dev libssl-dev 
libpixman-1-dev bridge-utils wget libfdt-dev bin86 bcc liblzma-dev iasl 
libc6-dev-i386
 
@@ -17,7 +23,7 @@ reboot
 Make sure to select Xen at boot, or edit /boot/grub/grub.cfg to make it the 
default, changing "set default="0" to point to the appropriate entry below (the 
one booting xen.gz), which could be entry number "4" for example.
 
 
-## Building QEMU
+### Building QEMU
 ```
 apt-get install libglib2.0-dev libpixman-1-dev libcap-dev libattr1-dev
 
@@ -54,7 +60,7 @@ make install
 cp i386-softmmu/qemu-system-i386 /usr/lib/xen/bin/
 ```
 
-## Building CoreOS rkt
+### Building CoreOS rkt
 ```
 apt-get install golang automake libacl1-dev libsystemd-dev
 ./configure --disable-tpm --with-stage1-flavors=coreos
@@ -62,7 +68,7 @@ make
 cp build-rkt-1.26.0+git/target/bin/rkt /usr/sbin
 ```
 
-## Building stage1-xen
+### Building stage1-xen
 ```
 apt-get install busybox-static jq
 
@@ -72,3 +78,83 @@ export GOPATH=/path/to/gopath
 bash build.sh
 cp stage1-xen.aci /home/username
 ```
+
+
+## Fedora
+
+On Fedora there are two ways to build stage1-xen artifacts.
+
+ * [Container Build](#build_fedora_container_build)
+ * [Manual Build](#build_fedora_manual_build)
+
+
+### Container Build
+
+We can build stage1-xen artifacts (Xen, QEMU and rkt) automatically in a 
docker container as follows 
+
+```
+cd stage1-xen
+
+docker pull lambdalinuxfedora/stage1-xen-fedora-buildroot
+
+docker run --rm \
+  -v `pwd`:/root/gopath/src/github.com/rkt/stage1-xen \
+  -v /tmp:/tmp \
+  -t -i lambdalinuxfedora/stage1-xen-fedora-buildroot \
+  /sbin/my_init -- /root/bin/run
+```
+
+Once `docker run` completes, the build artifact `stage1-xen-build.tar.gz` is 
generated in `/tmp` directory. Please see 
[RUNNING_STAGE1_XEN.md](build/fedora/RUNNING_STAGE1_XEN.md) for details on how 
to setup Fedora for running stage1-xen.
+
+
+### Manual Build
+
+It is also possible to manually build stage1-xen components on a Fedora host. 
+
+Please ensure that you have all the dependencies installed. The dependencies 
for Xen, QEMU, rkt and stage1-xen is documented in 
[buildroot-Dockerfile](build/fedora/buildroot-Dockerfile). You will also need 
to install [`binutils`](https://github.com/lambda-linux-fedora/binutils) 
package that is compiled with `i386pe` support. You can download the pre-built 
RPMs from [here](https://drive.google.com/open?id=0B_tTbuxmuRzIR05wQ3E1eWVyaGs).
+
+Install `binutils` package.
+
+```
+tar xvf binutils-2.26.1-1.1.fc25.tar
+
+dnf install -y 
./binutils/2.26.1/1.1.fc25/x86_64/binutils-2.26.1-1.1.fc25.x86_64.rpm
+```
+
+You can verify `i386pe` support in `binutils` by doing the following.
+
+```
+[root@localhost]# ld -V
+GNU ld version 2.26.1-1.1.fc25  Supported emulations:
+   elf_x86_64
+   elf32_x86_64
+   elf_i386
+   elf_iamcu
+   i386linux
+   elf_l1om
+   elf_k1om
+   i386pep
+   i386pe
+```
+
+You should see the lines `i386pep` and `i386pe` in the output.
+
+Next you can build Xen, Qemu and rkt using the following scripts 
+
+ * [`build/fedora/components/xen`](build/fedora/components/xen)
+ * [`build/fedora/components/qemu`](build/fedora/components/qemu)
+ * [`build/fedora/components/rkt`](build/fedora/components/rkt)
+
+Please review the scripts and adjust the paths according to your requirements.
+
+Once the dependencies are installed, you can build stage1-xen
+
+```
+git clone https://github.com/rkt/stage1-xen.git
+cd stage1-xen
+export GOPATH=/path/to/gopath
+bash build.sh
+cp stage1-xen.ac

[Xen-devel] [stage1-xen PATCH v1 07/10] .circleci/config.yml: Add

2017-08-26 Thread Rajiv Ranganath 
From: Rajiv M Ranganath <rajiv.rangan...@atihita.com>

Signed-off-by: Rajiv Ranganath <rajiv.rangan...@atihita.com>
---
 .circleci/config.yml |   21 +
 1 file changed, 21 insertions(+)
 create mode 100644 .circleci/config.yml

diff --git a/.circleci/config.yml b/.circleci/config.yml
new file mode 100644
index 000..93315b4
--- /dev/null
+++ b/.circleci/config.yml
@@ -0,0 +1,21 @@
+version: 2
+jobs:
+  build:
+working_directory: /root
+docker:
+  - image: lambdalinuxfedora/stage1-xen-fedora-buildroot:1708260126
+command: /sbin/my_init
+steps:
+  - run:
+  # We create `stage1-xen` directory in Dockerfile for local dev
+  # environment. Removing it here so CircleCI checkout step can work
+  # correctly
+  name: Removing stage1-xen directory from GOPATH...
+  command: |
+rm -rf /root/gopath/src/github.com/rkt/stage1-xen
+  - checkout:
+  path: /root/gopath/src/github.com/rkt/stage1-xen
+  - run:
+  name: Starting run...
+  command: |
+/root/bin/run


___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

[Xen-devel] [stage1-xen PATCH v1 05/10] build/fedora: Add `source_path.sh`

2017-08-26 Thread Rajiv Ranganath 
From: Rajiv M Ranganath <rajiv.rangan...@atihita.com>

Signed-off-by: Rajiv Ranganath <rajiv.rangan...@atihita.com>
---
 build/fedora/source_path.sh |3 +++
 1 file changed, 3 insertions(+)
 create mode 100755 build/fedora/source_path.sh

diff --git a/build/fedora/source_path.sh b/build/fedora/source_path.sh
new file mode 100755
index 000..57ac7ed
--- /dev/null
+++ b/build/fedora/source_path.sh
@@ -0,0 +1,3 @@
+#!/bin/sh
+
+export 
PATH=/opt/xen-unstable/sbin:/opt/xen-unstable/bin:/opt/stage1-xen/bin:/opt/qemu-unstable/bin:$PATH


___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

[Xen-devel] [stage1-xen PATCH v1 00/10] Add Fedora support

2017-08-26 Thread Rajiv Ranganath 
Hi Stefanao,

I've made changes based on RFC feedback. Kindly please review.

Changes in V1:

- build/fedora: Add `run` and `components/*` scripts

Introduce scripts to build individual components (xen, rkt, qemu) both
within a container or directly on the host. It also has constants that
can be used to customize the build. `run` script now uses `component/*`
scripts.

- build/fedora: Add `xen-unstable-runit/*` scripts

Removed usage of `pipefile`, `pushd`, `popd`. Also added `set -e` and
switched to `/bin/bash`.

- build/fedora: Add `RUNNING_STAGE1_XEN.md`
- BUILDING.md: Add Fedora instructions

Updated documentation to highlight both container build and manual
build.

Here is a temporary repo [1] with v1 series.

Best,
Rajiv

[1]: https://github.com/lambda-linux-fedora/stage1-xen/tree/wip1

___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

[Xen-devel] [stage1-xen PATCH v1 03/10] build/fedora: Add `buildroot-Dockerfile`

2017-08-26 Thread Rajiv Ranganath 
From: Rajiv M Ranganath <rajiv.rangan...@atihita.com>

Signed-off-by: Rajiv Ranganath <rajiv.rangan...@atihita.com>
---
 build/fedora/buildroot-Dockerfile |  113 +
 1 file changed, 113 insertions(+)
 create mode 100644 build/fedora/buildroot-Dockerfile

diff --git a/build/fedora/buildroot-Dockerfile 
b/build/fedora/buildroot-Dockerfile
new file mode 100644
index 000..971560e
--- /dev/null
+++ b/build/fedora/buildroot-Dockerfile
@@ -0,0 +1,113 @@
+# tarballs checksum
+# -
+# 974b3091232d781c4fc410ccca98fb62ba9febe9e6a988e348804483c4f66742  
binutils-2.26.1-1.1.fc25.tar
+
+FROM lambdalinuxfedora/baseimage-fedora
+
+CMD ["/sbin/my_init"]
+
+COPY [ \
+  "./binutils-2.26.1-1.1.fc25.tar", \
+  \
+  "./components/*", \
+  "./run", \
+  "/tmp/docker-build/" \
+]
+
+RUN \
+  # dnf
+  echo "Running dnf update..." && \
+  dnf update -y && \
+  dnf install -y less && \
+  dnf install -y sudo && \
+  \
+  # circleci container requirements
+  # 
https://circleci.com/docs/2.0/custom-images/#adding-required-and-custom-tools-or-files
+  dnf install -y git && \
+  dnf install -y openssh-clients && \
+  dnf install -y tar && \
+  dnf install -y gzip && \
+  dnf install -y ca-certificates && \
+  \
+  # install `binutils`
+  pushd /tmp/docker-build && \
+# verify checksum
+echo "974b3091232d781c4fc410ccca98fb62ba9febe9e6a988e348804483c4f66742  
binutils-2.26.1-1.1.fc25.tar" | sha256sum -c - && \
+tar xvf binutils-2.26.1-1.1.fc25.tar && \
+dnf install -y 
./binutils/2.26.1/1.1.fc25/x86_64/binutils-2.26.1-1.1.fc25.x86_64.rpm && \
+  popd && \
+  \
+  dnf install -y @buildsys-build && \
+  \
+  # Having `ipdb` around is useful when debugging `run` script. Uncomment this
+  # section as required
+  # dnf install -y python2-devel && \
+  # dnf install -y python-pip && \
+  # su -l root -c "pip2 install --user ipdb==0.8 ipython==5.3.0" && \
+  \
+  # Note: xen and qemu has some duplicate package dependencies. We are
+  # explicitly calling out dependencies for xen and qemu
+  #
+  # xen build dependencies
+  dnf install -y bridge-utils && \
+  dnf install -y gettext && \
+  dnf install -y glib2-devel && \
+  dnf install -y glibc-devel.i686 && \
+  dnf install -y grub2 && \
+  dnf install -y iasl && \
+  dnf install -y libaio-devel && \
+  dnf install -y libuuid-devel && \
+  dnf install -y ncurses-devel && \
+  dnf install -y openssl-devel && \
+  dnf install -y pixman-devel && \
+  dnf install -y python2-devel && \
+  dnf install -y wget && \
+  dnf install -y yajl-devel && \
+  \
+  # qemu build dependencies
+  dnf install -y glib2-devel && \
+  dnf install -y libaio-devel && \
+  dnf install -y libattr-devel && \
+  dnf install -y libcap-devel && \
+  dnf install -y libcap-ng-devel && \
+  dnf install -y pixman-devel && \
+  dnf install -y zlib-devel && \
+  \
+  # rkt build dependencies
+  dnf install -y autoconf && \
+  dnf install -y automake && \
+  dnf install -y git && \
+  dnf install -y glibc-static && \
+  dnf install -y gnupg && \
+  dnf install -y golang && \
+  dnf install -y libacl-devel && \
+  dnf install -y squashfs-tools && \
+  dnf install -y systemd-devel && \
+  dnf install -y wget && \
+  \
+  # stage1-xen build dependencies
+  dnf install -y bc && \
+  dnf install -y busybox && \
+  dnf install -y glide && \
+  dnf install -y golang && \
+  dnf install -y jq && \
+  dnf install -y libacl-devel && \
+  dnf install -y wget && \
+  \
+  # copy `run` file and `components/{qemu,rkt,xen}`
+  su -l root -c "mkdir /root/bin" && \
+  su -l root -c "cp /tmp/docker-build/run /root/bin" && \
+  su -l root -c "mkdir /root/bin/components" && \
+  su -l root -c "cp /tmp/docker-build/qemu /root/bin/components" && \
+  su -l root -c "cp /tmp/docker-build/rkt /root/bin/components" && \
+  su -l root -c "cp /tmp/docker-build/xen /root/bin/components" && \
+  \
+  # create `stage1-xen` directory
+  mkdir -p /root/gopath/src/github.com/rkt/stage1-xen && \
+  \
+  # cleanup
+  rm -rf /tmp/docker-build && \
+  dnf clean all && \
+  rm -rf /var/cache/dnf/* && \
+  rm -rf /tmp/* && \
+  rm -rf /var/tmp/*


___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

[Xen-devel] [stage1-xen PATCH v1 02/10] build/fedora: Add `buildroot-README.md`

2017-08-26 Thread Rajiv Ranganath 
From: Rajiv M Ranganath <rajiv.rangan...@atihita.com>

Signed-off-by: Rajiv Ranganath <rajiv.rangan...@atihita.com>
---
 build/fedora/buildroot-README.md |   50 ++
 1 file changed, 50 insertions(+)
 create mode 100644 build/fedora/buildroot-README.md

diff --git a/build/fedora/buildroot-README.md b/build/fedora/buildroot-README.md
new file mode 100644
index 000..0efb150
--- /dev/null
+++ b/build/fedora/buildroot-README.md
@@ -0,0 +1,50 @@
+## stage1-xen Fedora Buildroot
+
+stage1-xen build artifacts for Fedora is built in two phases. In the first 
phase
+a docker container is prepared with all the build dependencies. We refer to it
+as `stage1-xen-fedora-buildroot`. In the next phase we execute the `run` script
+that uses `stage1-xen-fedora-buildroot` and to produce the build artifacts.
+
+### Building `stage1-xen-fedora-buildroot`
+
+`stage1-xen-fedora-buildroot` has a external dependency
+on [`binutils`](https://github.com/lambda-linux-fedora/binutils) package that 
is
+compiled with `i386pe` support. You can download the pre-built RPMs
+from [here](https://drive.google.com/open?id=0B_tTbuxmuRzIR05wQ3E1eWVyaGs).
+Please download `binutils-2.26.1-1.1.fc25.tar`.
+
+To build docker image
+
+```
+cd stage1-xen/build/fedora
+
+docker build -f buildroot-Dockerfile -t stage1-xen-fedora-buildroot .
+```
+
+### Running `stage1-xen-fedora-buildroot`
+
+```
+cd stage1-xen
+
+docker run --rm \
+  -v `pwd`:/root/gopath/src/github.com/rkt/stage1-xen \
+  -v /tmp:/tmp \
+  -t -i stage1-xen-fedora-buildroot \
+  /sbin/my_init -- /root/bin/run
+```
+
+The generated build artifacts are in `/tmp` directory.
+
+To debug build issues -
+
+```
+cd stage1-xen
+
+docker run --rm \
+  -v `pwd`:/root/gopath/src/github.com/rkt/stage1-xen \
+  -v /tmp:/tmp \
+  -t -i stage1-xen-fedora-buildroot \
+  /sbin/my_init -- /bin/bash
+```
+
+Also see section on `ipdb` in `buildroot-Dockerfile`.


___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

[Xen-devel] [stage1-xen PATCH v1 01/10] .gitignore: Add

2017-08-26 Thread Rajiv Ranganath 
From: Rajiv M Ranganath <rajiv.rangan...@atihita.com>

Signed-off-by: Rajiv Ranganath <rajiv.rangan...@atihita.com>
---
 .gitignore |2 ++
 1 file changed, 2 insertions(+)
 create mode 100644 .gitignore

diff --git a/.gitignore b/.gitignore
new file mode 100644
index 000..873f8f6
--- /dev/null
+++ b/.gitignore
@@ -0,0 +1,2 @@
+# build/fedora
+build/fedora/binutils-*.tar


___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

[Xen-devel] [stage1-xen PATCH v1 04/10] build/fedora: Add `run` and `components/*` scripts

2017-08-26 Thread Rajiv Ranganath 
From: Rajiv M Ranganath <rajiv.rangan...@atihita.com>

Signed-off-by: Rajiv Ranganath <rajiv.rangan...@atihita.com>
---
 build/fedora/components/qemu |   50 
 build/fedora/components/rkt  |   58 ++
 build/fedora/components/xen  |   46 +
 build/fedora/run |   56 +
 4 files changed, 210 insertions(+)
 create mode 100755 build/fedora/components/qemu
 create mode 100755 build/fedora/components/rkt
 create mode 100755 build/fedora/components/xen
 create mode 100755 build/fedora/run

diff --git a/build/fedora/components/qemu b/build/fedora/components/qemu
new file mode 100755
index 000..6c89e2c
--- /dev/null
+++ b/build/fedora/components/qemu
@@ -0,0 +1,50 @@
+#!/usr/bin/python2
+
+import shlex
+import subprocess
+import sys
+import os
+
+# Modify this if you would like to install Qemu elsewhere on your filesystem or
+# a different version of Qemu
+QEMU_PREFIX = '/opt/qemu-unstable'
+QEMU_BRANCH = 'master'
+
+# This should correspond to your Xen install prefix
+XEN_PREFIX = '/opt/xen-unstable'
+
+
+# helper function to capture stdout from a long running process
+def subprocess_stdout(cmd, cwd, env):
+p = subprocess.Popen(
+shlex.split(cmd), cwd=cwd, env=env, stdout=subprocess.PIPE)
+while p.poll() is None:
+l = p.stdout.readline()
+sys.stdout.write(l)
+if p.returncode != 0:
+sys.exit(1)
+
+env = os.environ.copy()
+
+# build and install qemu
+print "Cloning qemu..."
+cmd = "git clone --branch %(branch)s git://git.qemu.org/qemu.git" % {
+'branch': QEMU_BRANCH
+}
+subprocess.check_output(shlex.split(cmd), cwd='/root')
+
+steps = [
+"./configure --prefix=%(qemu_prefix)s --enable-xen 
--target-list=i386-softmmu --extra-cflags=\"-I%(xen_prefix)s/include\" 
--extra-ldflags=\"-L%(xen_prefix)s/lib -Wl,-rpath,%(xen_prefix)s/lib\" 
--disable-kvm --enable-virtfs --enable-linux-aio"
+% {
+'qemu_prefix': QEMU_PREFIX,
+'xen_prefix': XEN_PREFIX
+}, 'make', 'make install'
+]
+for cmd in steps:
+cwd = '/root/qemu'
+subprocess_stdout(cmd, cwd, env)
+
+cmd = "cp i386-softmmu/qemu-system-i386 
%(xen_prefix)s/lib/xen/bin/qemu-system-i386" % {
+'xen_prefix': XEN_PREFIX
+}
+subprocess.check_output(shlex.split(cmd), cwd='/root/qemu')
diff --git a/build/fedora/components/rkt b/build/fedora/components/rkt
new file mode 100755
index 000..edfdd1c
--- /dev/null
+++ b/build/fedora/components/rkt
@@ -0,0 +1,58 @@
+#!/usr/bin/python2
+
+import shlex
+import subprocess
+import sys
+import os
+
+# `rkt` is installed in the same prefix as `stage1-xen`. Modify this if you
+# would like to install rkt elsewhere on your filesystem.
+STAGE1_XEN_PREFIX = '/opt/stage1-xen'
+RKT_PREFIX = STAGE1_XEN_PREFIX
+RKT_BRANCH = 'master'
+
+# Adjust this according to what RKT_BRANCH generates
+RKT_BUILD_VER = 'rkt-1.28.1+git'
+
+
+# helper function to capture stdout from a long running process
+def subprocess_stdout(cmd, cwd, env):
+p = subprocess.Popen(
+shlex.split(cmd), cwd=cwd, env=env, stdout=subprocess.PIPE)
+while p.poll() is None:
+l = p.stdout.readline()
+sys.stdout.write(l)
+if p.returncode != 0:
+sys.exi(1)
+
+
+env = os.environ.copy()
+
+# build rkt
+print "Cloning rkt..."
+cmd = "git clone --branch %(branch)s https://github.com/rkt/rkt.git; % {
+'branch': RKT_BRANCH
+}
+subprocess.check_output(shlex.split(cmd), cwd='/root')
+
+steps = [
+'./autogen.sh', './configure --disable-tpm --with-stage1-flavors=coreos',
+'make'
+]
+for cmd in steps:
+cwd = '/root/rkt'
+subprocess_stdout(cmd, cwd, env)
+
+# install rkt build artifacts to RKT_PREFIX
+steps = [
+"mkdir -p %(prefix)s/bin" % {
+'prefix': RKT_PREFIX
+},
+"cp /root/rkt/build-%(build_ver)s/target/bin/rkt %(prefix)s/bin/rkt" % {
+'build_ver': RKT_BUILD_VER,
+'prefix': RKT_PREFIX
+}
+]
+for cmd in steps:
+cwd = '/root/rkt'
+subprocess_stdout(cmd, cwd, env)
diff --git a/build/fedora/components/xen b/build/fedora/components/xen
new file mode 100755
index 000..95da9a6
--- /dev/null
+++ b/build/fedora/components/xen
@@ -0,0 +1,46 @@
+#!/usr/bin/python2
+
+import shlex
+import subprocess
+import sys
+import os
+
+# Modify this if you would like to install Xen elsewhere on your filesystem or
+# a different version of Xen
+XEN_PREFIX = '/opt/xen-unstable'
+XEN_BRANCH = 'master'
+
+
+# helper function to capture stdout from a long running process
+def subprocess_stdout(cmd, cwd, env):
+p = subprocess.Popen(
+shlex.split(cmd), cwd=cwd, env=env, stdout=subprocess.PIPE)
+while p.poll() is None:
+l = p.stdout.readline()
+sys.stdout.write(l)
+if p.returncode != 0:
+sys.exit(1)
+
+
+env = os.

Re: [Xen-devel] [stage1-xen (RFC) PATCH 07/10] .circleci/config.yml: Add

2017-08-24 Thread Rajiv Ranganath 
On Thu, Aug 24 2017 at 05:54:05 AM, Stefano Stabellini <sstabell...@kernel.org> 
wrote:
> On Mon, 21 Aug 2017, Rajiv Ranganath wrote:
>> From: Rajiv M Ranganath <rajiv.rangan...@atihita.com>
>
> Does .circleci need to be in the top directory or could it be under
> fedora? If possible, I think it would make more sense to introduce it
> there.
>

I would have also preferred the `.circleci/` directory to be under
`build/fedora/`.

However, I could not find an option to change this directory. From their
documentation [1], I get a sense that this path is hardcoded.

Best,
Rajiv

[1]: https://circleci.com/docs/2.0/configuration-reference/

___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

[Xen-devel] [stage1-xen (RFC) PATCH 08/10] README.md: Add CircleCI badge

2017-08-21 Thread Rajiv Ranganath 
From: Rajiv M Ranganath 


---
 README.md |2 ++
 1 file changed, 2 insertions(+)

diff --git a/README.md b/README.md
index 9ea6adf..e1cd40c 100644
--- a/README.md
+++ b/README.md
@@ -1,5 +1,7 @@
 # stage1-xen - A Xen based stage1 for CoreOS rkt
 
+[![Build 
Status](https://circleci.com/gh/rkt/stage1-xen/tree/master.svg?style=shield=:circle-token)](https://circleci.com/gh/rkt/stage1-xen/tree/master)
+
 ## Goal
 
 CoreOS rkt is a modular container engine with [three stages of 
execution](https://coreos.com/rkt/docs/latest/devel/stage1-implementors-guide.html).
 Stage1 is responsible for creating the execution environment for the contained 
applications.


___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

[Xen-devel] [stage1-xen (RFC) PATCH 05/10] build/fedora: Add `source_path.sh`

2017-08-21 Thread Rajiv Ranganath 
From: Rajiv M Ranganath 


---
 build/fedora/source_path.sh |3 +++
 1 file changed, 3 insertions(+)
 create mode 100755 build/fedora/source_path.sh

diff --git a/build/fedora/source_path.sh b/build/fedora/source_path.sh
new file mode 100755
index 000..57ac7ed
--- /dev/null
+++ b/build/fedora/source_path.sh
@@ -0,0 +1,3 @@
+#!/bin/sh
+
+export 
PATH=/opt/xen-unstable/sbin:/opt/xen-unstable/bin:/opt/stage1-xen/bin:/opt/qemu-unstable/bin:$PATH


___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

[Xen-devel] [stage1-xen (RFC) PATCH 07/10] .circleci/config.yml: Add

2017-08-21 Thread Rajiv Ranganath 
From: Rajiv M Ranganath 


---
 .circleci/config.yml |   21 +
 1 file changed, 21 insertions(+)
 create mode 100644 .circleci/config.yml

diff --git a/.circleci/config.yml b/.circleci/config.yml
new file mode 100644
index 000..67ac964
--- /dev/null
+++ b/.circleci/config.yml
@@ -0,0 +1,21 @@
+version: 2
+jobs:
+  build:
+working_directory: /root
+docker:
+  - image: lambdalinuxfedora/stage1-xen-fedora-buildroot:1708171409
+command: /sbin/my_init
+steps:
+  - run:
+  # We create `stage1-xen` directory in Dockerfile for local dev
+  # environment. Removing it here so CircleCI checkout step can work
+  # correctly
+  name: Removing stage1-xen directory from GOPATH...
+  command: |
+rm -rf /root/gopath/src/github.com/rkt/stage1-xen
+  - checkout:
+  path: /root/gopath/src/github.com/rkt/stage1-xen
+  - run:
+  name: Starting run...
+  command: |
+/root/bin/run


___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

[Xen-devel] [stage1-xen (RFC) PATCH 04/10] build/fedora: Add `run`

2017-08-21 Thread Rajiv Ranganath 
From: Rajiv M Ranganath 


---
 build/fedora/run |   87 ++
 1 file changed, 87 insertions(+)
 create mode 100755 build/fedora/run

diff --git a/build/fedora/run b/build/fedora/run
new file mode 100755
index 000..37e1dac
--- /dev/null
+++ b/build/fedora/run
@@ -0,0 +1,87 @@
+#!/usr/bin/python2
+
+import shlex
+import subprocess
+import sys
+import os
+
+
+# helper function to capture stdout from a long running process
+def subprocess_stdout(cmd, cwd, env):
+p = subprocess.Popen(
+shlex.split(cmd), cwd=cwd, env=env, stdout=subprocess.PIPE)
+while p.poll() is None:
+l = p.stdout.readline()
+sys.stdout.write(l)
+
+
+env = os.environ.copy()
+
+# build and install xen-unstable
+print "Cloning xen-unstable..."
+cmd = 'git clone git://xenbits.xen.org/xen.git'
+subprocess.check_output(shlex.split(cmd), cwd='/root')
+
+steps = [
+'./configure --prefix=/opt/xen-unstable 
--with-system-qemu=/opt/xen-unstable/lib/xen/bin/qemu-system-i386 
--disable-stubdom --disable-qemu-traditional --disable-rombios 
--sysconfdir=/opt/xen-unstable/etc --enable-rpath --disable-systemd',
+'make',
+'make install BOOT_DIR=/opt/xen-unstable/boot 
DEBUG_DIR=/opt/xen-unstable/lib/debug 
EFI_DIR=/opt/xen-unstable/boot/efi/EFI/xen'
+]
+for cmd in steps:
+cwd = '/root/xen'
+subprocess_stdout(cmd, cwd, env)
+
+# build and install qemu-unstable
+print "Cloning qemu-unstable..."
+cmd = 'git clone git://git.qemu.org/qemu.git'
+subprocess.check_output(shlex.split(cmd), cwd='/root')
+
+steps = [
+'./configure --prefix=/opt/qemu-unstable --enable-xen 
--target-list=i386-softmmu --extra-cflags="-I/opt/xen-unstable/include" 
--extra-ldflags="-L/opt/xen-unstable/lib -Wl,-rpath,/opt/xen-unstable/lib" 
--disable-kvm --enable-virtfs --enable-linux-aio',
+'make', 'make install'
+]
+for cmd in steps:
+cwd = '/root/qemu'
+subprocess_stdout(cmd, cwd, env)
+
+cmd = 'cp i386-softmmu/qemu-system-i386 
/opt/xen-unstable/lib/xen/bin/qemu-system-i386'
+subprocess.check_output(shlex.split(cmd), cwd='/root/qemu')
+
+# build rkt
+print "Cloning rkt..."
+cmd = 'git clone https://github.com/rkt/rkt.git'
+subprocess.check_output(shlex.split(cmd), cwd='/root')
+
+steps = [
+'./autogen.sh', './configure --disable-tpm --with-stage1-flavors=coreos',
+'make'
+]
+for cmd in steps:
+cwd = '/root/rkt'
+subprocess_stdout(cmd, cwd, env)
+
+# build stage1-xen
+env['GOPATH'] = '/root/gopath'
+cwd = '/root/gopath/src/github.com/rkt/stage1-xen'
+cmd = 'bash build.sh'
+subprocess_stdout(cmd, cwd, env)
+
+# install build artifacts to `/opt/`
+steps = [
+'mkdir -p /opt/stage1-xen/bin', 'mkdir -p /opt/stage1-xen/aci',
+'cp /root/rkt/build-rkt-1.28.1+git/target/bin/rkt /opt/stage1-xen/bin/rkt',
+'cp /root/gopath/src/github.com/rkt/stage1-xen/stage1-xen.aci 
/opt/stage1-xen/aci/stage1-xen.aci',
+'cp /root/gopath/src/github.com/rkt/stage1-xen/build/fedora/source_path.sh 
/opt/stage1-xen/bin/source_path.sh',
+'cp -r 
/root/gopath/src/github.com/rkt/stage1-xen/build/fedora/xen-unstable-runit 
/opt/xen-unstable-runit'
+]
+for cmd in steps:
+cwd = '/root'
+subprocess_stdout(cmd, cwd, env)
+
+cwd = '/opt'
+cmd = 'tar zcvf /root/stage1-xen-build.tar.gz qemu-unstable/ stage1-xen/ 
xen-unstable/ xen-unstable-runit/'
+subprocess_stdout(cmd, cwd, env)
+
+cwd = '/root'
+cmd = 'mv /root/stage1-xen-build.tar.gz /tmp'
+subprocess_stdout(cmd, cwd, env)


___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

[Xen-devel] [stage1-xen (RFC) PATCH 10/10] BUILDING.md: Add Fedora instructions

2017-08-21 Thread Rajiv Ranganath 

---
 BUILDING.md |   35 ++-
 1 file changed, 30 insertions(+), 5 deletions(-)

diff --git a/BUILDING.md b/BUILDING.md
index 3ef5311..b9ca404 100644
--- a/BUILDING.md
+++ b/BUILDING.md
@@ -1,7 +1,13 @@
 # Build
-stage1-xen requires new Xen and QEMU versions at the time of writing. You are 
unlikely to find them already packaged with your distro. This document 
describes how to build and install the latest Xen and QEMU from scratch. In 
addition, given that CoreOS rkt is also missing from reasonably new distros 
such as Ubuntu Xenial Xerus, I added instructions on how to build that too. The 
document includes the dependencies needed for the build based on Ubuntu Xenial 
Xerus.
+stage1-xen requires new Xen and QEMU versions at the time of writing. You are 
unlikely to find them already packaged with your distro. This document 
describes how to build and install the latest Xen, QEMU and rkt from scratch 
for Ubuntu Xenial Xerus and Fedora.
 
-## Building Xen
+ * [Ubuntu Xenial Xerus](#build_ubuntu)
+ * [Fedora](#build_fedora)
+
+
+## Ubuntu Xenial Xerus
+
+### Building Xen
 ```
 apt-get install git build-essential python-dev gettext uuid-dev 
libncurses5-dev libyajl-dev libaio-dev pkg-config libglib2.0-dev libssl-dev 
libpixman-1-dev bridge-utils wget libfdt-dev bin86 bcc liblzma-dev iasl 
libc6-dev-i386
 
@@ -17,7 +23,7 @@ reboot
 Make sure to select Xen at boot, or edit /boot/grub/grub.cfg to make it the 
default, changing "set default="0" to point to the appropriate entry below (the 
one booting xen.gz), which could be entry number "4" for example.
 
 
-## Building QEMU
+### Building QEMU
 ```
 apt-get install libglib2.0-dev libpixman-1-dev libcap-dev libattr1-dev
 
@@ -54,7 +60,7 @@ make install
 cp i386-softmmu/qemu-system-i386 /usr/lib/xen/bin/
 ```
 
-## Building CoreOS rkt
+### Building CoreOS rkt
 ```
 apt-get install golang automake libacl1-dev libsystemd-dev
 ./configure --disable-tpm --with-stage1-flavors=coreos
@@ -62,7 +68,7 @@ make
 cp build-rkt-1.26.0+git/target/bin/rkt /usr/sbin
 ```
 
-## Building stage1-xen
+### Building stage1-xen
 ```
 apt-get install busybox-static jq
 
@@ -72,3 +78,22 @@ export GOPATH=/path/to/gopath
 bash build.sh
 cp stage1-xen.aci /home/username
 ```
+
+
+## Fedora
+
+On Fedora we build stage1-xen artifacts (Xen, QEMU and rkt) in a docker 
container as follows 
+
+```
+cd stage1-xen
+
+docker pull lambdalinuxfedora/stage1-xen-fedora-buildroot
+
+docker run --rm \
+  -v `pwd`:/root/gopath/src/github.com/rkt/stage1-xen \
+  -v /tmp:/tmp \
+  -t -i lambdalinuxfedora/stage1-xen-fedora-buildroot \
+  /sbin/my_init -- /root/bin/run
+```
+
+Once `docker run` completes, the build artifact `stage1-xen-build.tar.gz` is 
generated in `/tmp` directory. Please see 
[RUNNING_STAGE1_XEN.md](build/fedora/RUNNING_STAGE1_XEN.md) for details on how 
to setup Fedora for running stage1-xen.


___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

[Xen-devel] [stage1-xen (RFC) PATCH 03/10] build/fedora: Add `buildroot-Dockerfile`

2017-08-21 Thread Rajiv Ranganath 
From: Rajiv M Ranganath 


---
 build/fedora/buildroot-Dockerfile |  108 +
 1 file changed, 108 insertions(+)
 create mode 100644 build/fedora/buildroot-Dockerfile

diff --git a/build/fedora/buildroot-Dockerfile 
b/build/fedora/buildroot-Dockerfile
new file mode 100644
index 000..ede6e57
--- /dev/null
+++ b/build/fedora/buildroot-Dockerfile
@@ -0,0 +1,108 @@
+# tarballs checksum
+# -
+# 974b3091232d781c4fc410ccca98fb62ba9febe9e6a988e348804483c4f66742  
binutils-2.26.1-1.1.fc25.tar
+
+FROM lambdalinuxfedora/baseimage-fedora
+
+CMD ["/sbin/my_init"]
+
+COPY [ \
+  "./binutils-2.26.1-1.1.fc25.tar", \
+  \
+  "./run", \
+  "/tmp/docker-build/" \
+]
+
+RUN \
+  # dnf
+  echo "Running dnf update..." && \
+  dnf update -y && \
+  dnf install -y less && \
+  dnf install -y sudo && \
+  \
+  # circleci container requirements
+  # 
https://circleci.com/docs/2.0/custom-images/#adding-required-and-custom-tools-or-files
+  dnf install -y git && \
+  dnf install -y openssh-clients && \
+  dnf install -y tar && \
+  dnf install -y gzip && \
+  dnf install -y ca-certificates && \
+  \
+  # install `binutils`
+  pushd /tmp/docker-build && \
+# verify checksum
+echo "974b3091232d781c4fc410ccca98fb62ba9febe9e6a988e348804483c4f66742  
binutils-2.26.1-1.1.fc25.tar" | sha256sum -c - && \
+tar xvf binutils-2.26.1-1.1.fc25.tar && \
+dnf install -y 
./binutils/2.26.1/1.1.fc25/x86_64/binutils-2.26.1-1.1.fc25.x86_64.rpm && \
+  popd && \
+  \
+  dnf install -y @buildsys-build && \
+  \
+  # Having `ipdb` around is useful when debugging `run` script. Uncomment this
+  # section as required
+  # dnf install -y python2-devel && \
+  # dnf install -y python-pip && \
+  # su -l root -c "pip2 install --user ipdb==0.8 ipython==5.3.0" && \
+  \
+  # Note: xen and qemu has some duplicate package dependencies. We are
+  # explicitly calling out dependencies for xen and qemu
+  #
+  # xen build dependencies
+  dnf install -y bridge-utils && \
+  dnf install -y gettext && \
+  dnf install -y glib2-devel && \
+  dnf install -y glibc-devel.i686 && \
+  dnf install -y grub2 && \
+  dnf install -y iasl && \
+  dnf install -y libaio-devel && \
+  dnf install -y libuuid-devel && \
+  dnf install -y ncurses-devel && \
+  dnf install -y openssl-devel && \
+  dnf install -y pixman-devel && \
+  dnf install -y python2-devel && \
+  dnf install -y wget && \
+  dnf install -y yajl-devel && \
+  \
+  # qemu build dependencies
+  dnf install -y glib2-devel && \
+  dnf install -y libaio-devel && \
+  dnf install -y libattr-devel && \
+  dnf install -y libcap-devel && \
+  dnf install -y libcap-ng-devel && \
+  dnf install -y pixman-devel && \
+  dnf install -y zlib-devel && \
+  \
+  # rkt build dependencies
+  dnf install -y autoconf && \
+  dnf install -y automake && \
+  dnf install -y git && \
+  dnf install -y glibc-static && \
+  dnf install -y gnupg && \
+  dnf install -y golang && \
+  dnf install -y libacl-devel && \
+  dnf install -y squashfs-tools && \
+  dnf install -y systemd-devel && \
+  dnf install -y wget && \
+  \
+  # stage1-xen build dependencies
+  dnf install -y bc && \
+  dnf install -y busybox && \
+  dnf install -y glide && \
+  dnf install -y golang && \
+  dnf install -y jq && \
+  dnf install -y libacl-devel && \
+  dnf install -y wget && \
+  \
+  # copy `run` file
+  su -l root -c "mkdir /root/bin" && \
+  su -l root -c "cp /tmp/docker-build/run /root/bin" && \
+  \
+  # create `stage1-xen` directory
+  mkdir -p /root/gopath/src/github.com/rkt/stage1-xen && \
+  \
+  # cleanup
+  rm -rf /tmp/docker-build && \
+  dnf clean all && \
+  rm -rf /var/cache/dnf/* && \
+  rm -rf /tmp/* && \
+  rm -rf /var/tmp/*


___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

[Xen-devel] [stage1-xen (RFC) PATCH 09/10] build/fedora: Add `RUNNING_STAGE1_XEN.md`

2017-08-21 Thread Rajiv Ranganath 

---
 build/fedora/RUNNING_STAGE1_XEN.md |  383 
 1 file changed, 383 insertions(+)
 create mode 100644 build/fedora/RUNNING_STAGE1_XEN.md

diff --git a/build/fedora/RUNNING_STAGE1_XEN.md 
b/build/fedora/RUNNING_STAGE1_XEN.md
new file mode 100644
index 000..705e7e4
--- /dev/null
+++ b/build/fedora/RUNNING_STAGE1_XEN.md
@@ -0,0 +1,383 @@
+# Running Stage1 Xen on Fedora
+
+This document outlines the steps to get started with stage1-xen on Fedora. 
They are 
+
+ * [Preparing your machine and installing minimal 
Fedora](#preparing_your_machine_and_installing_minimal_fedora)
+ * [Booting into Xen](#booting_into_xen)
+ * [Launching Xen services](#launching_xen_services)
+ * [Setting up Xen networking](#setting_up_xen_networking)
+ * [Running stage1-xen](#running_stage1-xen)
+
+
+## Preparing your machine and installing minimal Fedora
+
+On x86 platform there are two ways to start an operating system or a 
hypervisor. They are 
+
+ * Legacy BIOS Mode
+ * EFI Mode
+
+Latest operating systems and hypervisors including Fedora and Xen has support 
for EFI mode. If you are unfamiliar with EFI we recommend checking out this 
[article](http://www.rodsbooks.com/efi-bootloaders/principles.html).
+
+By default, most BIOS now boot using EFI Mode. In your BIOS menu, there might 
be an option to toggle _Legacy BIOS Mode_. Do not toggle that option.
+
+### Enable VT-x and VT-d
+
+Please ensure that you have enabled VT-x and if available VT-d.
+
+### Disable Secure Boot
+
+As we will be booting a custom build of Xen, we need to disable secure boot. 
You will find an option in your BIOS menu to disable secure boot.
+
+### Installing minimal Fedora
+
+The default Fedora installation installs packages that we do not require when 
running Xen. We recommend doing a minimal Fedora as follows.
+
+ 1. Download Net Install image
+
+ 2. Prepare a USB drive
+
+ 3. Do a minimal Fedora Install
+
+You can download the Fedora net install image 
[here](https://alt.fedoraproject.org/). You can select either the Fedora Server 
or Fedora Workstation image, it doesn't really matter.
+
+After downloading the net install images, please copy the raw image onto a USB 
drive. Please see 
[this](https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/html/Installation_Guide/sect-making-usb-media.html)
 link on how to prepare USB drive.
+
+EFI BIOS comes with a _BIOS Boot Menu_ using which you can select the device 
to boot from. Insert the USB drive, then go into your BIOS Boot Menu and boot 
using the USB drive. This should start the Fedora Network Installer.
+
+In the Fedora Installer, there is a section for under _SOFTWARE_ called 
_SOFTWARE SELECTION_. In this section please **select** either _Minimal 
Install_ or _Basic Desktop_, **without** any add-ons. 
+
+**Note:** If there is existing data on the hard disk, please ensure that 
_INSTALLATION DESTINATION_ under _SYSTEM_ section is appropriately configured. 
+
+Then click on _Begin Installation_ to complete the installation.
+
+Once the installation is complete, please disable SELinux by editing 
`/etc/selinux/config`.
+
+You now have a minimal Fedora Installation, which is good for working with Xen.
+
+
+## Booting into Xen
+
+Log into Fedora as root and copy over `stage1-xen-build.tar.gz`. Extract 
`stage1-xen-build.tar.gz` into `/opt` directory.
+
+```shell
+[root@localhost ~]# tar zxvf stage1-xen-build.tar.gz -C /opt
+
+[root@localhost ~]# ls /opt
+qemu-unstable  stage1-xen  xen-unstable  xen-unstable-runit
+```
+
+This will extract all the build artifacts into `/opt` directory.
+
+Next we will create a BIOS Boot Menu entry to boot `xen-4.10-unstable.efi`. 
This will start Xen hypervisor. Xen will then start Fedora as Dom-0 guest.
+
+On Fedora, EFI system partition (ESP) is usually mounted at `/boot/efi`. This 
is a `vfat` partition. You can check if EFI system partition is mounted as 
follows 
+
+```shell
+[root@localhost ~]# mount | grep '\/boot\/efi'
+/dev/sda1 on /boot/efi type vfat 
(rw,relatime,fmask=0077,dmask=0077,codepage=437,iocharset=ascii,shortname=winnt,errors=remount-ro)
+```
+
+Create a directory for Xen under `/boot/efi/EFI` and copy over 
`xen-4.10-unstable.efi`.
+
+```shell
+[root@localhost ~]# mkdir -p /boot/efi/EFI/xen
+[root@localhost ~]# cp 
/opt/xen-unstable/boot/efi/EFI/xen/xen-4.10-unstable.efi /boot/efi/EFI/xen/
+```
+
+Inspect `/boot/efi/EFI/fedora/grub.cfg`. Under section `### BEGIN 
/etc/grub.d/10_linux ###` you will find `menuentry` for Fedora kernel and 
initrd. Look for `linuxefi` and `initrdefi`. Copy over the `vmlinuz` and 
`initramfs` files that you want to use for your Dom-0 into `/boot/efi/EFI/xen` 
directory.
+
+```shell
+[root@localhost ~]# cp /boot/vmlinuz-A.B.C-D.fcXX.x86_64 /boot/efi/EFI/xen/
+
+[root@localhost ~]# cp /boot/initramfs-A.B.C-D.fcXX.x86_64.img 
/boot/efi/EFI/xen/
+```
+
+Now in `/boot/efi/EFI/xen/` you should have the following files.
+
+```shell
+[root@localhost ~]# ls

[Xen-devel] [stage1-xen (RFC) PATCH 02/10] build/fedora: Add `buildroot-README.md`

2017-08-21 Thread Rajiv Ranganath 
From: Rajiv M Ranganath 


---
 build/fedora/buildroot-README.md |   50 ++
 1 file changed, 50 insertions(+)
 create mode 100644 build/fedora/buildroot-README.md

diff --git a/build/fedora/buildroot-README.md b/build/fedora/buildroot-README.md
new file mode 100644
index 000..0efb150
--- /dev/null
+++ b/build/fedora/buildroot-README.md
@@ -0,0 +1,50 @@
+## stage1-xen Fedora Buildroot
+
+stage1-xen build artifacts for Fedora is built in two phases. In the first 
phase
+a docker container is prepared with all the build dependencies. We refer to it
+as `stage1-xen-fedora-buildroot`. In the next phase we execute the `run` script
+that uses `stage1-xen-fedora-buildroot` and to produce the build artifacts.
+
+### Building `stage1-xen-fedora-buildroot`
+
+`stage1-xen-fedora-buildroot` has a external dependency
+on [`binutils`](https://github.com/lambda-linux-fedora/binutils) package that 
is
+compiled with `i386pe` support. You can download the pre-built RPMs
+from [here](https://drive.google.com/open?id=0B_tTbuxmuRzIR05wQ3E1eWVyaGs).
+Please download `binutils-2.26.1-1.1.fc25.tar`.
+
+To build docker image
+
+```
+cd stage1-xen/build/fedora
+
+docker build -f buildroot-Dockerfile -t stage1-xen-fedora-buildroot .
+```
+
+### Running `stage1-xen-fedora-buildroot`
+
+```
+cd stage1-xen
+
+docker run --rm \
+  -v `pwd`:/root/gopath/src/github.com/rkt/stage1-xen \
+  -v /tmp:/tmp \
+  -t -i stage1-xen-fedora-buildroot \
+  /sbin/my_init -- /root/bin/run
+```
+
+The generated build artifacts are in `/tmp` directory.
+
+To debug build issues -
+
+```
+cd stage1-xen
+
+docker run --rm \
+  -v `pwd`:/root/gopath/src/github.com/rkt/stage1-xen \
+  -v /tmp:/tmp \
+  -t -i stage1-xen-fedora-buildroot \
+  /sbin/my_init -- /bin/bash
+```
+
+Also see section on `ipdb` in `buildroot-Dockerfile`.


___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

[Xen-devel] [stage1-xen (RFC) PATCH 01/10] .gitignore: Add

2017-08-21 Thread Rajiv Ranganath 
From: Rajiv M Ranganath 


---
 .gitignore |2 ++
 1 file changed, 2 insertions(+)
 create mode 100644 .gitignore

diff --git a/.gitignore b/.gitignore
new file mode 100644
index 000..873f8f6
--- /dev/null
+++ b/.gitignore
@@ -0,0 +1,2 @@
+# build/fedora
+build/fedora/binutils-*.tar


___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

[Xen-devel] [stage1-xen (RFC) PATCH 06/10] build/fedora: Add `xen-unstable-runit/*` scripts

2017-08-21 Thread Rajiv Ranganath 
From: Rajiv M Ranganath 


---
 build/fedora/xen-unstable-runit/setup.sh   |   21 
 build/fedora/xen-unstable-runit/teardown.sh|   21 
 .../xen-init-dom0-disk-backend/run |8 
 build/fedora/xen-unstable-runit/xen-init-dom0/run  |6 ++
 build/fedora/xen-unstable-runit/xenconsoled/run|   10 ++
 build/fedora/xen-unstable-runit/xenstored/run  |   20 +++
 6 files changed, 86 insertions(+)
 create mode 100755 build/fedora/xen-unstable-runit/setup.sh
 create mode 100755 build/fedora/xen-unstable-runit/teardown.sh
 create mode 100755 
build/fedora/xen-unstable-runit/xen-init-dom0-disk-backend/run
 create mode 100755 build/fedora/xen-unstable-runit/xen-init-dom0/run
 create mode 100755 build/fedora/xen-unstable-runit/xenconsoled/run
 create mode 100755 build/fedora/xen-unstable-runit/xenstored/run

diff --git a/build/fedora/xen-unstable-runit/setup.sh 
b/build/fedora/xen-unstable-runit/setup.sh
new file mode 100755
index 000..16699be
--- /dev/null
+++ b/build/fedora/xen-unstable-runit/setup.sh
@@ -0,0 +1,21 @@
+#!/bin/bash
+
+set -e
+set -o pipefail
+
+# runit RPM creates `/etc/service` directory
+if [ ! -d "/etc/service" ]; then
+echo "/etc/service directory not found. Please install runit RPM."
+exit 1
+fi
+
+runit_services="xenconsoled xen-init-dom0 xen-init-dom0-disk-backend xenstored"
+
+pushd /etc/service > /dev/null
+for service in $runit_services; do
+ln -sf /opt/xen-unstable-runit/$service $service
+done
+popd > /dev/null
+
+echo "Successfully created symlinks in /etc/service directory."
+exit 0
diff --git a/build/fedora/xen-unstable-runit/teardown.sh 
b/build/fedora/xen-unstable-runit/teardown.sh
new file mode 100755
index 000..c0895f4
--- /dev/null
+++ b/build/fedora/xen-unstable-runit/teardown.sh
@@ -0,0 +1,21 @@
+#!/bin/bash
+
+set -e
+set -o pipefail
+
+# runit RPM creates `/etc/service` directory
+if [ ! -d "/etc/service" ]; then
+echo "/etc/service directory not found."
+exit 1
+fi
+
+runit_services="xenconsoled xen-init-dom0 xen-init-dom0-disk-backend xenstored"
+
+pushd /etc/service > /dev/null
+for service in $runit_services; do
+rm -f $service
+done
+popd > /dev/null
+
+echo "Successfully deleted symlinks in /etc/service directory."
+exit 0
diff --git a/build/fedora/xen-unstable-runit/xen-init-dom0-disk-backend/run 
b/build/fedora/xen-unstable-runit/xen-init-dom0-disk-backend/run
new file mode 100755
index 000..a952890
--- /dev/null
+++ b/build/fedora/xen-unstable-runit/xen-init-dom0-disk-backend/run
@@ -0,0 +1,8 @@
+#!/bin/sh
+sv check xenstored >/dev/null || exit 1
+sv check xenconsoled >/dev/null || exit 1
+
+# In case of failure, allow user to run teardown script
+sleep 5s
+
+exec /opt/xen-unstable/lib/xen/bin/qemu-system-i386 -xen-domid 0 -xen-attach 
-name dom0 -nographic -M xenpv -monitor /dev/null -serial /dev/null -parallel 
/dev/null -nodefaults -no-user-config
diff --git a/build/fedora/xen-unstable-runit/xen-init-dom0/run 
b/build/fedora/xen-unstable-runit/xen-init-dom0/run
new file mode 100755
index 000..f7f56ed
--- /dev/null
+++ b/build/fedora/xen-unstable-runit/xen-init-dom0/run
@@ -0,0 +1,6 @@
+#!/bin/sh
+sv check xenstored >/dev/null || exit 1
+
+/opt/xen-unstable/lib/xen/bin/xen-init-dom0
+
+exec chpst -b xen-init-dom0 runit-pause
diff --git a/build/fedora/xen-unstable-runit/xenconsoled/run 
b/build/fedora/xen-unstable-runit/xenconsoled/run
new file mode 100755
index 000..7a3ab4e
--- /dev/null
+++ b/build/fedora/xen-unstable-runit/xenconsoled/run
@@ -0,0 +1,10 @@
+#!/bin/sh
+sv check xen-init-dom0 >/dev/null || exit 1
+
+[ ! -d /var/log/xen/console ] && mkdir -p /var/log/xen/console
+
+# In case of failure, allow user to run teardown script
+sleep 5s
+
+# --log=[none|guest|hv|all]
+exec /opt/xen-unstable/sbin/xenconsoled -i --log=none
diff --git a/build/fedora/xen-unstable-runit/xenstored/run 
b/build/fedora/xen-unstable-runit/xenstored/run
new file mode 100755
index 000..42e5e41
--- /dev/null
+++ b/build/fedora/xen-unstable-runit/xenstored/run
@@ -0,0 +1,20 @@
+#!/bin/sh
+[ ! -d /var/run/xen ] && mkdir -p /var/run/xen
+[ ! -d /var/run/xenstored ] && mkdir -p /var/run/xenstored
+[ ! -d /var/log/xen ] && mkdir -p /var/log/xen
+[ ! -d /var/lib/xen ] && mkdir -p /var/lib/xen
+[ ! -d /var/lib/xen/dump ] && mkdir -p /var/lib/xen/dump
+[ ! -d /var/lib/xen/xenpaging ] && mkdir -p /var/lib/xen/paging
+[ ! -d /var/lib/xenstored ] && mkdir -p /var/lib/xenstored
+modprobe -q xen-evtchn || exit 1
+modprobe -q xen-gntdev || exit 1
+modprobe -q xen-gntalloc || exit 1
+mountpoint -q /proc/xen || mount -t xenfs xenfs /proc/xen
+mountpoint -q /var/lib/xenstored || mount -t tmpfs xenstored /var/lib/xenstored
+grep -q "control_d" /proc/xen/capabilities || exit 1
+
+# In case of failure, allow user to run teardown script
+sleep 5s
+
+# Use `--trace-file /var/log/xen/xenstored-trace.log` for logs
+exec

[Xen-devel] [stage1-xen (RFC) PATCH 00/10] Add Fedora support

2017-08-21 Thread Rajiv Ranganath 
Hi Stefanao,

Following series adds documentation on how to use stage1-xen on Fedora.
It also adds continuous build support using CircleCI.

Xen, Qemu, rkt and stage1 ACI image are built using a Fedora based
container. This setup allows for a reproducible build environment that
we can use locally and also within a CI system.

Below is an overview of the series -

build/fedora: Add `buildroot-README.md`
build/fedora: Add `buildroot-Dockerfile`
build/fedora: Add `run`

`build/fedora` directory is introduced. In this directory we can
maintain documentation, scripts and patches relating to Fedora support
for stage1-xen.

`buildroot-README.md` describes a two stage container build process.
This is modeled on how we do builds for distro packages, but using
containers instead of a mock/chroot environment. The build artifact
produced by `run` script is a tarball, instead of a rpm or deb file.

build/fedora: Add `source_path.sh`
build/fedora: Add `xen-unstable-runit/*` scripts

Scripts used within build artifact tarball.

.circleci/config.yml: Add
README.md: Add CircleCI badge

Adds CircleCI support. Here [1][2] is a temporary repo with Circle CI
support enabled.

build/fedora: Add `RUNNING_STAGE1_XEN.md`
BUILDING.md: Add Fedora instructions

Update docs.

Please let me know your comments. Thank you!

Best,
Rajiv

[1]: https://github.com/lambda-linux-fedora/stage1-xen/tree/wip
[2]: https://circleci.com/gh/lambda-linux-fedora/stage1-xen/tree/wip

___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] [stage1-xen PATCH v1] init: Add `glide.lock`

2017-08-15 Thread Rajiv Ranganath 
On Tue, Aug 15 2017 at 06:23:07 AM, Stefano Stabellini  
wrote:
> Thank you for the patch. Usually the description that you sent in the
> previous email is written here.
>
> I like the build.sh changes and I think introducing init/glide.yaml is a
> great idea. But I don't think that introducing init/glide.lock is
> necessary, is it? We could let glide generate it on the fly based on the
> key versioning info already specified in glide.yaml.
>
> For example, this patch already introduces:
>
>   - package: github.com/containernetworking/cni
> version: 0.3.0
>
> to glide.yaml. Are there any other reasons for committing glide.lock to
> the repository instead of generating it?

I think the pattern of using `.lock` files to manage nested library
dependencies and semantic versioning for library APIs was initially
championed in the Ruby on Rails community. The idea has since been
adopted by Go community in Glide, Rust community in Cargo and JavaScript
community in Yarn.

Here is the link to the original discussion on whether `Gemfile.lock`
should be checked into the source tree or not. [1]

If we go by author's line of reasoning, then answer would depend on if
we consider init to be an app or a library.

Personally, I feel `init.go` is an app and it would make sense to check
in `glide.lock`.

If for some reason, in future there is a build failure due to a nested
dependency issue with dependent go libraries, then having a working
`.lock` in the git is always useful.

In anycase after sending `BUILDING.md` Fedora patches, I am also
planning on sending patches to do continuous build of `stage1-xen` in a
Fedora based docker container. That should also catch build failures
early.

Please let me know what you prefer. I can send a v2 of the patch with
just `glide.yaml`

Best,
Rajiv

[1] 
http://yehudakatz.com/2010/12/16/clarifying-the-roles-of-the-gemspec-and-gemfile/

___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

[Xen-devel] [stage1-xen PATCH v1] init: Add `glide.lock`

2017-08-12 Thread Rajiv Ranganath 
Signed-off-by: Rajiv Ranganath <rajiv.rangan...@atihita.com>
---
 build.sh|5 +--
 init/glide.lock |   89 +++
 init/glide.yaml |   23 ++
 3 files changed, 114 insertions(+), 3 deletions(-)
 create mode 100644 init/glide.lock
 create mode 100644 init/glide.yaml

diff --git a/build.sh b/build.sh
index ec56093..6c34890 100755
--- a/build.sh
+++ b/build.sh
@@ -83,10 +83,9 @@ if [ -f stage1-xen.aci ]; then
 fi
 
 # Build init
-go get github.com/hashicorp/errwrap
 cd init
-glide init || true
-glide up -v 
+rm -rf vendor
+glide install -v
 cd ..
 go build -o target/rootfs/init init/init.go
 
diff --git a/init/glide.lock b/init/glide.lock
new file mode 100644
index 000..f512bc7
--- /dev/null
+++ b/init/glide.lock
@@ -0,0 +1,89 @@
+hash: eb0d5fbb629911862615dfdc4dde5283949af890a06d3ff70662e507385bd14b
+updated: 2017-08-12T09:56:42.779804672Z
+imports:
+- name: github.com/appc/spec
+  version: 210e2995a04148739121566b71b7440512467cc2
+  subpackages:
+  - aci
+  - pkg/device
+  - pkg/tarheader
+  - schema
+  - schema/common
+  - schema/types
+  - schema/types/resource
+- name: github.com/containernetworking/cni
+  version: 5c3c17164270150467498a32c71436c7cd5501be
+  subpackages:
+  - pkg/ip
+  - pkg/ns
+  - pkg/types
+  - pkg/utils
+  - pkg/utils/sysctl
+- name: github.com/coreos/go-iptables
+  version: f2ede9c85e2fac4d72d5a9af0af59c0858d7a3bd
+  subpackages:
+  - iptables
+- name: github.com/coreos/go-semver
+  version: 1817cd4bea52af76542157eeabd74b057d1a199e
+  subpackages:
+  - semver
+- name: github.com/coreos/go-systemd
+  version: d2196463941895ee908e13531a23a39feb9e1243
+  subpackages:
+  - unit
+- name: github.com/d2g/dhcp4
+  version: fcbeb8a548ebd34b55134f2833c5b036a941aa82
+- name: github.com/d2g/dhcp4client
+  version: 8ca8fe2cad1770f068782377ec6be6733c01a96b
+- name: github.com/hashicorp/errwrap
+  version: 7554cd9344cec97297fa6649b055a8c98c2a1e55
+- name: github.com/rkt/rkt
+  version: 142050d1a558ab07f6eeddea55c0f51053a99b05
+  subpackages:
+  - common
+  - common/cgroup
+  - common/cgroup/v1
+  - common/cgroup/v2
+  - common/networking
+  - networking/netinfo
+  - networking/tuntap
+  - pkg/acl
+  - pkg/fileutil
+  - pkg/flag
+  - pkg/fs
+  - pkg/group
+  - pkg/log
+  - pkg/mountinfo
+  - pkg/passwd
+  - pkg/sys
+  - pkg/user
+  - stage1/common
+  - stage1/common/types
+  - stage1/init/common
+- name: github.com/spf13/pflag
+  version: e57e3eeb33f795204c1ca35f56c44f83227c6e66
+- name: github.com/sstabellini/rkt
+  version: 8a57cb8b6682ed8fef054f57efef292781597fde
+  subpackages:
+  - networking
+- name: github.com/syndtr/gocapability
+  version: db04d3cc01c8b54962a58ec7e491717d06cfcc16
+  subpackages:
+  - capability
+- name: github.com/vishvananda/netlink
+  version: f5a6f697a596c788d474984a38a0ac4ba0719e93
+  subpackages:
+  - nl
+- name: github.com/vishvananda/netns
+  version: 86bef332bfc3b59b7624a600bd53009ce91a9829
+- name: go4.org
+  version: 034d17a462f7b2dcd1a4a73553ec5357ff6e6c6e
+  subpackages:
+  - errorutil
+- name: golang.org/x/sys
+  version: e42485b6e20ae7d2304ec72e535b103ed350cc02
+  subpackages:
+  - unix
+- name: gopkg.in/inf.v0
+  version: 3887ee99ecf07df5b447e9b00d9c0b2adaa9f3e4
+testImports: []
diff --git a/init/glide.yaml b/init/glide.yaml
new file mode 100644
index 000..3919338
--- /dev/null
+++ b/init/glide.yaml
@@ -0,0 +1,23 @@
+package: github.com/rkt/stage1-xen/init
+import:
+- package: github.com/appc/spec
+  subpackages:
+  - schema/types
+- package: github.com/hashicorp/errwrap
+- package: github.com/rkt/rkt
+  subpackages:
+  - common
+  - common/networking
+  - pkg/flag
+  - pkg/log
+  - pkg/sys
+  - stage1/common
+  - stage1/common/types
+  - stage1/init/common
+- package: github.com/sstabellini/rkt
+  subpackages:
+  - networking
+- package: github.com/containernetworking/cni
+  version: 0.3.0
+- package: github.com/d2g/dhcp4
+- package: github.com/d2g/dhcp4client


___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

[Xen-devel] [stage1-xen PATCH v1] init: Add `glide.lock`

2017-08-12 Thread Rajiv Ranganath 
Hi Stefano,

`build.sh` does `glide init` followed by `glide up -v` in order to pull
in dependencies for `init.go`.

I ran into two issues -

1. `glide init` automatically starts in interactive mode, which pauses
`build.sh`. This could be fixed with `glide init --non-interactive`.

2. `glide up -v` 

`glide update` dependency resolution breaks with the following error -

```
[INFO]  --> Fetching updates for github.com/vishvananda/netns.
[ERROR] Error scanning github.com/containernetworking/cni/pkg/ip: open 
/root/.glide/cache/src/https-github.com-containernetworking-cni/pkg/ip: no such 
file or directory
[ERROR] This error means the referenced package was not found.
[ERROR] Missing file or directory errors usually occur when multiple packages
[ERROR] share a common dependency and the first reference encountered by the 
scanner
[ERROR] sets the version to one that does not contain a subpackage needed 
required
[ERROR] by another package that uses the shared dependency. Try setting a
[ERROR] version in your glide.yaml that works for all packages that share this
[ERROR] dependency.

[...]

[ERROR] Failed to retrieve a list of dependencies: Error resolving imports
```

In order to resolve `github.com/vishvananda/netns` dependency and build
correctly we need to use `cni#v0.3.0`.

Following patch proposes introducing `glide.lock` with list of nested
dependencies needed to build `init.go`.

Best,
Rajiv

___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] stage1-xen for Fedora

2017-08-08 Thread Rajiv Ranganath 
Hi Stefano,

On Wed, Aug 2, 2017 at 12:15 AM, Stefano Stabellini
 wrote:

[...]

> The main thing that will be different is the list of dependencies you
> need to install to build Xen. On Fedora it should be (I am using
> Raisin[1] as a reference):

Thank you for the pointer to Raisin.

I have managed to build stage1-xen on Fedora. This project is very
interesting. I have some questions regarding stage1-xen and containers
on Xen.

1. Is there a roadmap/design doc for containers primitives and container
standards that Xen community is looking to support?

The only documentation that I could find were presentations by you.
[1][2]

2. Now that OCI 1.0 is out, are there any plans to create a Xen based
OCI runtime? [3]

A Xen based OCI runtime that can work with containerd and cri-o would be
very interesting to us.

I was wondering if you have thoughts on how xen-stage1 could be evolved
to support rkt and also also a OCI runtime?

3. Are there plans to use PVHv2 guests instead of PV guests?

4. In the presentation I noticed PV Calls for Networking. However when I
did `rkt run ...`, it seems to use netback with vif-nat. How can I try
PV calls for networking?

[...]

> Let me know if you find any issues!

Following are the issues that I ran into -

1. `rkt rm ...` fails with `stage1/rootfs/gc` file not found error. I
think because of this the Xen host gets populated with a lot of
overlayfs mounts. I tried to manually clean up, but that failed too.

2. Upstream cni master seems to have reorganized its directory
structure. So, I had to pin the version to 0.3 to get the build to work.
I also had to manually get dhcp4 and dhcp4client packages. Perhaps we
can add a glide.lock file to lock down the dependencies. I can send a
patch for it.

> I would be very happy to take a patch (or pull request) for
> BUILDING.md to document how to do this on Fedora.

I have a somewhat "non-standard" setup for xen and qemu for Fedora. I'll
briefly describe the setup.

Xen is booted using EFI. This required building a custom binutils
package [4]. Both Xen and qemu are built with a non-standard prefix
(/opt/xen-unstable and /opt/qemu-stable), with RPATHs appropriately
adjusted.

Lastly I don't use systemd to manage Xen on Fedora. In the buildroot,
Xen is explicitly configured using --disable-systemd. We have a version
of runit package that we run under systemd. Runit then launches
xenstore, xenconsole, dom0 qemu disk backend. We frequently toggle
between upstart and systemd based distro, so using runit on both has
been very helpful.

If this setup is okay you, I can open up the Fedora variant of our tools
and packages and send patches to BUILDING.md.

Please let me know.

Thank you!

Best,
Rajiv

[1]: 
https://xendeveloperanddesignsummit2017.sched.com/event/AjGx/keynote-secure-containers-with-xen-and-coreos-rkt-stefano-stabellini-aporeto
[2]: 
https://docs.google.com/presentation/d/1dP_7myrUrtwQHnjgDtlMQkAxJNG6Se9SBl0tdaFIAYQ/edit?usp=sharing
[3]: 
https://github.com/opencontainers/runtime-spec/blob/master/implementations.md
[4]: https://wiki.xenproject.org/wiki/Xen_EFI

___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel