Making Android system properties available to Upstart
As already explained by Oli, the Ubuntu Touch "flipped" architecture runs the Android services within an LXC container. This brings a number of advantages but there is a problem: knowing when specific Android services are ready. This is needed such that the Ubuntu Upstart jobs running on the host side can start as soon as possible, but no sooner ;) Currently, the approach is to use heuristics to determine the "readiness" of the Android services, but this can be unreliable. A better solution would be to have a way to create Upstart events on the host-side when services changes occur in the Android environment. The plan at this stage is to make use of Androids System Properties and create a 2-part solution: 1) Create an Upstart bridge that runs on the host. This bridge will listen on a socket and read data in the form: =\n This will be converted into an upstart event of the form: = The will be specified as an argument to the bridge for security reasons and would be set to something like "android-property" for Touch. The bridge would run at the system level (as root) on the host (since it has to start *before* the Android container, but the events it generates would be accessible to the Session (once that has been converted to use an Upstart Session Init) as ":sys:" allowing Upstart jobs to specify conditions such as: # run job when ueventd has started in Androids LXC container start on :sys:android-property init.svc.ueventd=running 2) Create an Android service that runs early in the Android boot process. This service would: - connect to the well-known socket the bridge is already listening on. - monitor Androids System Properties and write them to the socket in the form: =\n -- Ideally, we would just have a single Upstart bridge on the host side which is able to monitor property changes (*) without the need for "helper" processes on the Android side. There are a number of solutions that involve modifying Androids init, but we're attempting to avoid doing that. If anyone knows of a better way to solve this problem, please jump in. Kind regards, James. -- James Hunt (*) - just like 'watchprop' does in the Android environment. #upstart on freenode http://upstart.ubuntu.com/cookbook https://lists.ubuntu.com/mailman/listinfo/upstart-devel -- ubuntu-devel mailing list ubuntu-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Re: DEP-8 tests
On 13/05/13 13:59, Benjamin Drung wrote: > Am Montag, den 13.05.2013, 13:36 +0100 schrieb James Hunt: >> Great to see all these new DEP-8 tests [1] appearing! Let's keep the momentum >> going so we can turn [2] into a wall of green^H^H^H^H^H err blue sky and >> sunshine! :-) > > I found a bug: [1] and [2] are missing in your email. :) > That got you interested right? :-) [1] - http://developer.ubuntu.com/packaging/html/auto-pkg-test.html [2] - https://jenkins.qa.ubuntu.com/view/Saucy/view/AutoPkgTest/ Kind regards, James. -- James Hunt #upstart on freenode http://upstart.ubuntu.com/cookbook https://lists.ubuntu.com/mailman/listinfo/upstart-devel -- ubuntu-devel mailing list ubuntu-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Patch Pilot report 2013-05-13
https://code.launchpad.net/~yolanda.robla/nut/dep-8-tests/+merge/161414 Approved. https://code.launchpad.net/~yolanda.robla/ubuntu/saucy/postfix/dep-8-tests/+merge/161610 Approved. https://code.launchpad.net/~yolanda.robla/ubuntu/saucy/openldap/dep-8-tests/+merge/162097 Commented. https://code.launchpad.net/~yolanda.robla/ubuntu/saucy/spamassassin/dep-8-tests/+merge/162690 Approved. https://code.launchpad.net/~yolanda.robla/ubuntu/saucy/ganglia/dep-8-tests/+merge/163035 Approved. https://code.launchpad.net/~yolanda.robla/ubuntu/saucy/iscsitarget/dep-8-tests/+merge/163368 Approved. https://code.launchpad.net/~cristiklein/ubuntu/precise/vm-builder/lp468809/+merge/161427 Approved. https://bugs.launchpad.net/ubuntu/+source/mysql-5.5/+bug/1121874 Commented. Great to see all these new DEP-8 tests [1] appearing! Let's keep the momentum going so we can turn [2] into a wall of green^H^H^H^H^H err blue sky and sunshine! :-) Kind regards, James. -- James Hunt #upstart on freenode http://upstart.ubuntu.com/cookbook https://lists.ubuntu.com/mailman/listinfo/upstart-devel -- ubuntu-devel mailing list ubuntu-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Re: Coverity static analysis for C, C++ and Java code
On 10/04/13 13:41, Loïc Minier wrote: > On Mon, Apr 08, 2013, James Hunt wrote: >> We're already using it for critical packages including Upstart and >> Whoopsie [3], but it would be great to expand its scope to make it use >> the norm rather than the exception. > > Cool! How did you hook it up to the Upstart sources though? I haven't done that yet - currently a slightly manual process but looking at ways to automate further (starting with a daily cron :) Ideally, I'd like to have all MP's scanned. at release > time, or e.g. from some Jenkins job pushing the latest version daily? > > Does this scan the Ubuntu branch of Upstart, the upstream one or both? I do both. > > Would it be ok license-wise and hard for us to do this at a larger > scale; e.g. have some kind of daily job that pushes the latest Ubuntu > source packages from a set to be tested? I don't know. Coverity seemed to have relaxed the restriction that the individual that requests Coverity scans for a project be the "project owner". If you look at the "Role with the Project" option on [1], there are now 6 values including "other". I'll contact them and see if it might be possible... Kind regards, James. [1] - http://scan.coverity.com/project_register.html James Hunt #upstart on freenode http://upstart.ubuntu.com/cookbook https://lists.ubuntu.com/mailman/listinfo/upstart-devel -- ubuntu-devel mailing list ubuntu-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Re: Further Coverity info
On 09/04/13 22:28, Scott Ritchie wrote: > On 4/9/13 7:34 AM, Allan LeSage wrote: >> * As part of our Jenkins CI program, we're Coverity-scanning merge >> proposals, and disapproving them upon finding a new defect: >> https://code.launchpad.net/~mrazik/unico/coverity/+merge/156877 . > > As an upstream (wine) that uses Coverity, I'm curious how we can get this sort > of feature in the free tier. From what I can tell Coverity just periodically > scans our git tree periodically and produces a list of reports. > > We have a testbot that scans incoming patches (submitted via mailing list) to > measure new defects: in Wine's case this is defined as tests that fail on one > of > the bot VMs, but if I could invoke coverity directly it could in principle > scan > an arbitrary patchset. > > Do I need to setup some elaborate system of making a new git branch with the > incoming patch set and then automatically asking coverity to scan that > branch? Or can it be manually invoked with arbitrary patches? Yes - you can run it manually once you have a login... Wine is already shown in the list of Coverity projects [1], so all you need to do is: - Request a login by mailing scan-ad...@coverity.com and access to the Wine Coverity project. - Download the Coverity scan tool and run it across any version of the wine codebase. - Submit your "snapshot" (Coverity scan tool output) using [3] or [4]. - Login to http://scan5.coverity.com and view the results. Here's an example of the web interface: http://ubuntuone.com/7Ufq2dHdgGVeqJ16ftqJk1 The first two steps are one-off activities of course. Note that the "snapshots" can be any arbitrary version of wine - you differentiate them by adding a tag and/or version on the upload page [3] or using the -b/-t coverity-scan options. For example, here's how I might upload a scan of Upstart manually using [4]: $ coverity-submit -t lp:upstart-20130410-foobar-baz.2 upstart This will: - clean the build tree - run the build with Coverity - upload the snapshot You'll get a mail from Coverity once the scan is available (takes a few minutes for me, although might take longer for Wine ;-). If you have multiple versions/tags, when you login to http://scan5.coverity.com, select the appropriate version from the Snapshots menu on the left. Kind regards, James. [1] - http://scan.coverity.com/all-projects.html [2] - http://scan.coverity.com/start/ [3] - http://scan.coverity.com/upload.html [4] - http://www.catb.org/~esr/coverity-submit/ James Hunt #upstart on freenode http://upstart.ubuntu.com/cookbook https://lists.ubuntu.com/mailman/listinfo/upstart-devel -- ubuntu-devel mailing list ubuntu-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Re: Coverity static analysis for C, C++ and Java code
On 08/04/13 14:45, Colin Ian King wrote: > On 08/04/13 14:40, James Hunt wrote: >> On 08/04/13 13:57, Matthias Klose wrote: >>> Am 08.04.2013 14:13, schrieb James Hunt: >>>> As a precis of my earlier blog post [1], I'd like to encourage those >>>> involved >>>> with a C, C++ or Java project in Ubuntu to take a look at the Coverity Scan >>>> static-analysis service offered free to OSS projects [2]. >>>> >>>> We're already using it for critical packages including Upstart and >>>> Whoopsie [3], >>>> but it would be great to expand its scope to make it use the norm rather >>>> than >>>> the exception. >>> >>> Did it catch the wrong use of the malloc attribute in upstart? ;) >> I don't know - we were using it in anger then and I've now fixed that gcc >> function attribute issue :) >> >>> >>>> For those who have either never used static analysis tools, or have simply >>>> never >>>> used Coverity, don't fall into the trap of thinking that "gcc -pedantic >>>> -Wall" >>>> should be good enough for anyone - it simply is not. >>> >>> I don't know where you did get this from ... Anyway, not using -Wextra >>> leaves >>> out more things. >>> >>> while not static analysis tools, you might want to look at >>> -fsanitize=address >>> and -fsanitize=thread in GCC 4.8 (available in the ubuntu-toolchain-r/test >>> PPA). >> Will do, thanks. >> >>> >>> There's also clang --analyze, scan-view and scan-build in the clang package >>> as a >>> static analyzer. >> Yes, I have used and continue to use these tools. However, from my >> experiences, >> they are not as thorough as Coverity for the codebases I'm regularly looking >> at. >> >>> >>> And all of these are free software. >> Back in the day, splint [1] rocked on static analysis but the project >> appears to >> have languished - it doesn't even appear to handle C99. YMMV but IMHO, >> Coverity >> Scan is the most thorough static-analysis tool available to OSS developers >> today >> that I've seen. Maybe if splint were to be revived my opinion may change... >> ;) > > smatch [1] is quite a useful tool too, it has helped me find a variety > of bugs in applications I've written, Agreed - I'm using smatch alongside Coverity. however, I'd rather use coverity > if we had access to it. > > [1] http://smatch.sourceforge.net/ > >> >>> >>> Matthias >>> >>> >> >> Kind regards, >> >> James. >> >> [1] - http://splint.sourceforge.net/ >> -- >> James Hunt >> >> #upstart on freenode >> http://upstart.ubuntu.com/cookbook >> https://lists.ubuntu.com/mailman/listinfo/upstart-devel >> > > -- Kind regards, James. -- James Hunt #upstart on freenode http://upstart.ubuntu.com/cookbook https://lists.ubuntu.com/mailman/listinfo/upstart-devel -- ubuntu-devel mailing list ubuntu-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Re: Coverity static analysis for C, C++ and Java code
On 08/04/13 13:57, Matthias Klose wrote: > Am 08.04.2013 14:13, schrieb James Hunt: >> As a precis of my earlier blog post [1], I'd like to encourage those involved >> with a C, C++ or Java project in Ubuntu to take a look at the Coverity Scan >> static-analysis service offered free to OSS projects [2]. >> >> We're already using it for critical packages including Upstart and Whoopsie >> [3], >> but it would be great to expand its scope to make it use the norm rather than >> the exception. > > Did it catch the wrong use of the malloc attribute in upstart? ;) I don't know - we were using it in anger then and I've now fixed that gcc function attribute issue :) > >> For those who have either never used static analysis tools, or have simply >> never >> used Coverity, don't fall into the trap of thinking that "gcc -pedantic >> -Wall" >> should be good enough for anyone - it simply is not. > > I don't know where you did get this from ... Anyway, not using -Wextra leaves > out more things. > > while not static analysis tools, you might want to look at -fsanitize=address > and -fsanitize=thread in GCC 4.8 (available in the ubuntu-toolchain-r/test > PPA). Will do, thanks. > > There's also clang --analyze, scan-view and scan-build in the clang package > as a > static analyzer. Yes, I have used and continue to use these tools. However, from my experiences, they are not as thorough as Coverity for the codebases I'm regularly looking at. > > And all of these are free software. Back in the day, splint [1] rocked on static analysis but the project appears to have languished - it doesn't even appear to handle C99. YMMV but IMHO, Coverity Scan is the most thorough static-analysis tool available to OSS developers today that I've seen. Maybe if splint were to be revived my opinion may change... ;) > > Matthias > > Kind regards, James. [1] - http://splint.sourceforge.net/ -- James Hunt #upstart on freenode http://upstart.ubuntu.com/cookbook https://lists.ubuntu.com/mailman/listinfo/upstart-devel -- ubuntu-devel mailing list ubuntu-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Coverity static analysis for C, C++ and Java code
As a precis of my earlier blog post [1], I'd like to encourage those involved with a C, C++ or Java project in Ubuntu to take a look at the Coverity Scan static-analysis service offered free to OSS projects [2]. We're already using it for critical packages including Upstart and Whoopsie [3], but it would be great to expand its scope to make it use the norm rather than the exception. For those who have either never used static analysis tools, or have simply never used Coverity, don't fall into the trap of thinking that "gcc -pedantic -Wall" should be good enough for anyone - it simply is not. Kind regards, James. [1] - http://ifdeflinux.blogspot.co.uk/2013/04/coverity-static-analysis-for-c-c-and.html [2] - http://scan.coverity.com/ [3] - http://scan.coverity.com/all-projects.html -- James Hunt #upstart on freenode http://upstart.ubuntu.com/cookbook https://lists.ubuntu.com/mailman/listinfo/upstart-devel -- ubuntu-devel mailing list ubuntu-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
procenv and buildd environments
The procenv utility [1], which provides information on the environment it runs in, is now available in Debian Sid [2] and Ubuntu Raring [3]. As part of its package build, 'make check' gets run which does the following: (1) Dumps information on the C pre-processor, compiler and linker. (2) Runs procenv itself. Step (1) is an aid to (2) in case it should fail. (2) is useful since it: - Helps to ensure the package works as expected once installed [*]. - Is a good check that procenv can handle 'odd' environments such as those provided by chroots (which are unusual in that their libc version often does not align with the provided kernel version). The readily accessible build logs for procenv itself ([4], [5], [6]) may in some circumstances help with FTBFS and test failure issues with _other_ packages since they provide an insight into the environment in which a package is built. Using the build logs, informative analysis can be performed: - See the environment a buildd provides. - Compare a 'normal' environment with a buildd-provided one. - Compare a Debian buildd environment to an Ubuntu one. There are other examples and ideas in the initial blog post and the man page. [*] - Note that procenv also has DEP-8 support such that the autopkgtest-provided environment can also be inspected [7]. The plan is to also enable Ubuntu PPA builds, again for comparative purposes. If anyone is interested in adding additional features to procenv, see the TODO and please let me know. Kind regards, James. [1] - https://launchpad.net/procenv [2] - http://packages.debian.org/sid/procenv [3] - http://packages.ubuntu.com/raring/procenv [4] - https://buildd.debian.org/status/package.php?p=procenv&suite=sid [5] - http://buildd.debian-ports.org/status/package.php?p=procenv [6] - https://launchpad.net/ubuntu/+source/procenv (expand version 'twistie' for build logs) [7] - https://jenkins.qa.ubuntu.com/view/Raring/view/AutoPkgTest/job/raring-adt-procenv/ -- James Hunt http://upstart.ubuntu.com/cookbook http://upstart.ubuntu.com/cookbook/upstart_cookbook.pdf -- ubuntu-devel mailing list ubuntu-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Patch Pilot report 2012-11-26
lp:~kampka/ubuntu/quantal/zabbix/upstart-support: Commented that this is now suitable for upload. lp:~kampka/ubuntu/quantal/fwknop/upstart-support: Small change required prior to upload. lp:~wb-munzinger/ubuntu/precise/havp/fix-init-script: Approved (already released). Reviewed patch on bug 1072015 and made some suggestions. Reviewed patch on bug 613231 and applied to branch ready for next upload. lp:~anders-kaseorg/ubuntu/raring/kmod/lp1082598: Approved. lp:~ubuntu-treblig/ubuntu/raring/gdb/bug-1069897: Approved. Also added further comments on bug 1069897 after code analysis. Kind regards, James. -- James Hunt http://upstart.ubuntu.com/cookbook http://upstart.ubuntu.com/cookbook/upstart_cookbook.pdf -- ubuntu-devel mailing list ubuntu-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Re: Enhanced Upstart User Sessions for the Raring desktop
Hi Robie, On 22/11/12 10:24, Robie Basak wrote: > A minor point, speaking as an uninformed observer. > >> /etc/xdg/init > > These files will be Upstart-specific, right? What if XDG want to use > /etc/xdg/init in the future? Would /etc/xdg/upstart avoid a potential > future namespace collision here? > > I understand that /etc/xdg/init would be analogous to /etc/init, but do > we really own the /etc/xdg namespace like we do /etc? And you're using > $HOME/.config/upstart/ so there wouldn't be too much of an inconsistency > there. > Agreed. I've changed the spec to reference /etc/xdg/upstart/ although I wonder if we should make this /etc/xdg/upstart/user/ (*) to make it clear the common jobs in that directory are only for Upstart running as a non-priv Session Init. Kind regards, James. (*) - or even /etc/xdg/init/upstart/user/ maybe. -- James Hunt http://upstart.ubuntu.com/cookbook http://upstart.ubuntu.com/cookbook/upstart_cookbook.pdf -- ubuntu-devel mailing list ubuntu-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Re: Enhanced Upstart User Sessions for the Raring desktop
Original Message Subject: Re: Enhanced Upstart User Sessions for the Raring desktop Date: Tue, 20 Nov 2012 15:53:40 + From: James Hunt Reply-To: james.h...@ubuntu.com To: Evan Huus CC: Upstart Devel List , ja...@ubuntu.com, Kees Cook Hi Evan, On 20/11/12 13:58, Evan Huus wrote: > Hi James, > > I've only scanned it briefly at this point, but I have one question regarding > security. At the moment the spec. states that system events will by default be > available to user sessions as well, but I think this is overly permissive. > While > I can't actually think of a current system event that really must be hidden, I > feel that on principle system events should not be visible to user sessions > unless explicitly marked as such. > > I would suggest a change to the new event syntax such that > > - foo and ::foo are always equivalent, and emit an event only visible within > the > current upstart namespace (system or user) > - :sys:foo and :user:foo are unchanged > - :all:foo (or :global:foo or :public:foo) emits an event visible to other > upstart namespaces as well This would mean user jobs would not be able to react to system job events. That in itself might not be a problem [1], but it would also mean that user jobs have no access to system events (by default) that don't relate to jobs. I think this _is_ potentially more of an issue. My canonical example is being able to plug a USB headset in, and have a user job react to the udev event and start an application of my choice. This currently works if user jobs are enabled in Ubuntu and is a useful feature. If we did implement ':all:', we could change the upstart-udev-bridge to ensure it emits events to ':all:', but we're then back to the concern over permissiveness and it feels wrong to special-case one particular event source. So, if we do 'hide' system events from user jobs by default, we would be breaking the current publish-subscribe behaviour. That's not to say we won't consider doing this [2], but we are striving to retain current behaviour so I'd like to understand better what the potential 'attack vector' might be with some real world examples. I think this needs to be explored further. Can anyone think of scenarios where user jobs should not be privy to system events? Note that Upstart is happy to run system jobs whose configuration files have perms 0400 and the /var/log/upstart/* files are not readable by normal users, so AFAICS, the risk is probably limited to users being able to 'sniff' system event environment variables. This is only a problem if an admin creates a job where they specify sensitive information in one of those variables. However, I have personally never seen such a job and is kind of analogous to an admin doing something inappropriate such as 'sudo chmod 777 /etc/shadow'. > > Thoughts? > > Cheers, > Evan > > > On Tue, Nov 20, 2012 at 5:09 AM, James Hunt <mailto:james.h...@ubuntu.com>> wrote: > > Here is the draft plan for 'Enhanced Upstart User Sessions' ([1]), which > will be > used to supervise desktop sessions in Ubuntu: > > https://wiki.ubuntu.com/FoundationsTeam/Specs/RaringUpstartUserSessions > > = Summary = > > Allow Upstart to run as a non-privileged user to supervise a session in an > event-based manner. > > This brings many advantages including: > > - more dynamic and efficient session startup (desktop services only get > started > *when required*). > - reliable session shutdown. > - automatic job output logging. > - more efficient use of system resources (helping to maximise battery > life for > example). > > Comments Welcome. If you would like to get involved in this project, > please let > me know. > > Kind regards, > > James. > > [1] - > > https://blueprints.launchpad.net/ubuntu/+spec/foundations-r-upstart-user-session-enhancements > -- > James Hunt > > http://upstart.ubuntu.com/cookbook > http://upstart.ubuntu.com/cookbook/upstart_cookbook.pdf > > -- > upstart-devel mailing list > upstart-de...@lists.ubuntu.com <mailto:upstart-de...@lists.ubuntu.com> > Modify settings or unsubscribe at: > https://lists.ubuntu.com/mailman/listinfo/upstart-devel > > Kind regards, James. [1] - Its unclear what use-cases there are for user jobs reacting to system _jobs_, so if you have some, please let us know. To lend weight to this, there is also the issue that even if user jobs can access system job events, they can only see those events that occur after their Session Init has starte
Enhanced Upstart User Sessions for the Raring desktop
Here is the draft plan for 'Enhanced Upstart User Sessions' ([1]), which will be used to supervise desktop sessions in Ubuntu: https://wiki.ubuntu.com/FoundationsTeam/Specs/RaringUpstartUserSessions = Summary = Allow Upstart to run as a non-privileged user to supervise a session in an event-based manner. This brings many advantages including: - more dynamic and efficient session startup (desktop services only get started *when required*). - reliable session shutdown. - automatic job output logging. - more efficient use of system resources (helping to maximise battery life for example). Comments Welcome. If you would like to get involved in this project, please let me know. Kind regards, James. [1] - https://blueprints.launchpad.net/ubuntu/+spec/foundations-r-upstart-user-session-enhancements -- James Hunt http://upstart.ubuntu.com/cookbook http://upstart.ubuntu.com/cookbook/upstart_cookbook.pdf -- ubuntu-devel mailing list ubuntu-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Patch pilot report 2012-07-31
Today was my first Patch Pilot session, so was co-piloting with apw. - https://bugs.launchpad.net/upstart/+bug/1018925 Reviewed and merged upstream. - https://code.launchpad.net/~lool/upstart/document-stop-in-pre-start Reviewed and merged upstream. - https://bugs.launchpad.net/ubuntu/+source/nss-pam-ldapd/+bug/806761 Reviewed but user appears to have attached wrong version of patch. Asked for clarification and provided some advice on possible job modification. Kind regards, James. -- James Hunt http://upstart.ubuntu.com/cookbook http://upstart.ubuntu.com/cookbook/upstart_cookbook.pdf -- ubuntu-devel mailing list ubuntu-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Re: event based initramfs
On 16/05/12 19:09, Phillip Susi wrote: > On 5/16/2012 9:01 AM, James Hunt wrote: >> The most obvious reason is that the initramfs has it's own init >> system (a big shell script). So, an Ubuntu system with an initramfs >> has two different init systems. By reducing this down to just >> Upstart, we get a simpler and hence more readily maintainable system >> that: > > It has its own system because it isn't really an init system at all. > The goal of upstart is to bring up and maintain the system. The goal of > the initramfs is only to find the root filesystem. Sounds like both systems are performing initialisation to me. And as we know, there is symmetry between what happens in the initramfs and what then has to happen again in the main system. By using Upstart, we get one tool to do that job without duplication of code. >> - has the advantage of being event-based (but which of course can >> also run jobs effectively "serially" too if desired). > >> LUKS support in the initramfs is currently handled using calls to >> "sleep 0.1". Who chose this value? How did they decide upon this >> value? Why 0.1 seconds? Why not just wait until the device appears? >> By having an event-based system, we can adopt a saner approach where >> the system simply reacts to the udev event when the kernel generates >> it - no polling required. > > Isn't this what udev is for? Event based handling of hardware. What do > we need upstart for? Upstart provides a "udev bridge" that allows arbitrary jobs to react to udev events without having to install udev rules in one of the 3 udev locations (although I guess that's really potentially 5 if you include the initramfs locations (but exclude /run)). For the cases where you aren't modifying devices, it makes working with udev much simpler. The whole point is that modern Linux systems are highly dynamic in nature. Polling is not required for most operations so why do it? Upstart was designed from the outset to operate in such a dynamic environment so it makes a lot of sense to take advantage of Upstarts ample abilities to make the initramfs less opaque, more efficient and more maintainable. > >> My non-LVM, non-RAID systems initramfs contains 16 calls to >> mount. By using mountall you end up with a single mountall.conf job >> that handles all that complexity. > > I only see one call to mount. There is only one filesystem to mount, so > where does 16 come from? mountall is somewhat complex because it needs > to mount multiple filesystems, in the correct order. The initramfs just > needs one. ? The initramfs performs numerous individual mounts including: root filesystem /proc /sys /dev /dev/pts /run The code for mountall is admittedly more complex than simply calling 'mount', but it has the following advantages: - it actually performs error checking (unlike the initramfs mount code). - it provides user feedback on the operations via Plymouth. - it runs fsck and provides feedback to the user via Plymouth. - it supports delays and timeouts. - it emits Upstart events allowing jobs to react to mount changes. >> A number of upstream projects are starting to move functionality from >> / to /usr (see [3]), hence we intend to support this model too. > > If getting to mountall depends on a file in /usr, then that is an FHS > violation and fundamentally incompatible with having /usr on its own > filesystem. Having the initramfs have its own copy of mountall and its > dependencies in order to mount /usr before the real mountall runs is a > kludgy workaround. If you want to boot without an initramfs at all, > this workaround won't help ( since the kernel only knows how to mount > the root and it is expected to contain everything needed to mount /usr > or other filesystems ), so the proper thing to do is make sure that > dependencies of mountall go in /. > > What does it matter where upstream installs to anyhow? There is an ongoing associated overhead in maintaining patches where we diverge from upstreams. They normally go > to /usr/local and we move them to /usr with a configure switch. Moving > to / instead should be trivial. That's not actually what we're talking about here - the upstreams are moving tooling currently in / to /usr. We might be able to work around this, but it sounds like it will become increasingly difficult to do so. >> - By utilizing Upstart, we get a clean, fast framework for mounting >> the root FS. We should also be able to simplify the initramfs by >> removing duplicate code. - the initramfs spawns daemons (such as >> udevd) which can be better managed via upstart. - the current >> initramfs kills the daemons it starts requiring them to be restarted
Re: event based initramfs
scue mode, then maybe they shouldn't be in /usr in the first place? A number of upstream projects are starting to move functionality from / to /usr (see [3]), hence we intend to support this model too. Additional reasons for supporting an event-based initramfs: - By utilizing Upstart, we get a clean, fast framework for mounting the root FS. We should also be able to simplify the initramfs by removing duplicate code. - the initramfs spawns daemons (such as udevd) which can be better managed via upstart. - the current initramfs kills the daemons it starts requiring them to be restarted in the main system. This has caused a few race conditions and clearly disallows state passing between the two systems. By enhancing Upstart to support stateful re-exec [4] we get huge benefits as we can retain state on processes started in the initramfs and hence don't necessarily need to restart them. I don't know of any other system that is capable of this. - there is potential for extra "smarts" that cannot be achieved easily/cleanly with the current shell approach. "Phase 1" is to add the ability to generate an event-based initramfs. The plan is to make this "opt in" so we can encourage users to try it (they can always revert) and gather user feedback since clearly there are a number of different boot scenarios to consider. Additionally, we will also need to consider carefully the initramfs size and performance impacts of any potential changes. Kind regards, James. -- James Hunt [1] - http://brainstorm.ubuntu.com/?keywords=boot [2] - http://summit.ubuntu.com/uds-q/meeting/20296/foundations-q-usr-merge/ [3] - https://blueprints.launchpad.net/ubuntu/+spec/foundations-q-usr-merge [4] - https://blueprints.launchpad.net/ubuntu/+spec/foundations-q-upstart-stateful-re-exec http://upstart.ubuntu.com/cookbook http://upstart.ubuntu.com/cookbook/upstart_cookbook.pdf signature.asc Description: OpenPGP digital signature -- ubuntu-devel mailing list ubuntu-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Call for testing: latest Upstart with logger improvements.
Hi All, Job logging was temporarily disabled recently in the Ubuntu Upstart package when we discovered a corner case the tests did not take account of [1]. The problem is now fixed and we have extended the tests to guard against regressions in this area [2]. I've put the latest code in a PPA to allow for further testing before we consider re-enabling job logging for all. If you would like to help with this important testing, read on... Getting the Latest Build 1) Enable the ppa:jamesodhunt/upstart-job-logging PPA [3]. 2) Install Upstart version '1.4-0ubuntu5~jh'. 3) Boot adding '--log' on the grub command line (you _must_ do this to enable logging!) 4) Poke around in /var/log/upstart/ and check CPU+mem usage. 5) Provide feedback. Limitations --- Jobs that _end_ prior to a writable log partition being available (around the time the 'filesystem' event is emitted [4]) will not have their data logged. I'm reworking a patch to remove this limitation... Thanks for your help! Again, remember to boot with '--log' for the time being if you want your job output logged! Kind regards, -- James Hunt [1] - https://bugs.launchpad.net/ubuntu/+source/upstart/+bug/922754 [2] - This brings the total number of Upstart tests to 1066 (excluding the 2846 NIH tests). [3] - https://launchpad.net/~jamesodhunt/+archive/upstart-job-logging [4] - upstart-events(7) - http://upstart.ubuntu.com/cookbook/#ubuntu-well-known-events-ubuntu-specific http://upstart.ubuntu.com/cookbook http://upstart.ubuntu.com/cookbook/upstart_cookbook.pdf -- ubuntu-devel mailing list ubuntu-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Re: Call for testing: Upstart 1.4 in Ubuntu
On 23/12/11 13:47, Tim Gardner wrote: > James - The kernel team has plenty of Precise systems that we can > update. Given the fundamental nature of Upstart, is there any way to > recover short of reinstalling if it completely breaks the boot ? Can we > stash the original /sbin/init somewhere and hack the grub command ? Hi Tim, Hopefully it won't completely break your system, but if you do experience problems, I'd recommend rebooting with the '--no-log' option to disable the logging feature. You can't disable the 'setuid'/'setgid' feature currently [1], but you can of course ensure that you've disabled all jobs that make use of those new stanzas. Generally speaking, we do try to ensure that as features are added, a corresponding command-line option is added to revert to the previous behaviour thus disabling the new code path. On the general topic of recovering a broken system, you *could* stash the previous version of upstart as /sbin/init.old or similar (you'd need to do the same for /sbin/initctl too of course), but booting with /sbin/init.old assumes that the new version of init wasn't introduced to cater for some fundamental interface change in the NIH libraries for example. Note too that any .conf files referencing initctl would strictly need to be modified to reference /sbin/initctl.old. For Ubuntu, we could conceivably use the 'alternatives' system to handle this along with static linking to overcome the NIH issue but that comes with its own issues and would need a lot of extra testing to ensure all the recovery code paths work as expected. For testing I do actually maintain a number of inits in /sbin and use a script [2] that boots setting 'init=/sbin/upstart_menu'. The script presents a menu like the following allowing me to select which particular version I want to boot with: http://people.canonical.com/~jhunt/upstart/utils/upstart_menu.png The selected init is then exec'd. This works fine for test systems. The 'upstart_menu' could in principle be hooked into the Ubuntu recovery path, but as stated above, adding this would require a lot of additional effort to ensure all the recovery paths are failsafe and I wonder if that effort wouldn't be better directed at just testing the latest init? Kind regards, James. -- James Hunt [1] - If you think this would be useful please raise a bug. [2] - http://people.canonical.com/~jhunt/upstart/utils/upstart_menu.sh http://upstart.ubuntu.com/cookbook http://upstart.ubuntu.com/cookbook/upstart_cookbook.pdf -- ubuntu-devel mailing list ubuntu-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Call for testing: Upstart 1.4 in Ubuntu
Hi All, We're looking to land Upstart 1.4 in Ubuntu Precise early January 2012. If you have an Ubuntu Precise test system [1] and you'd like to help with testing these new features, read on... = New Features = The two main new features are: - new 'setuid' and 'setgid' stanzas Allows you to specify the user and group a job runs as (this should help minimize the intricate su/sudo/start-stop-daemon command-lines) - Logging of job output for system jobs [2] Job logging is enabled by default in 1.4. Since this is the first version of Upstart that writes to *any* files, this is quite a big change. 3 new command-line options have been added to support this feature: '--no-log' (disable logging entirely) '--logdir=DIR' (specify alternate log directory) '--default-console=VALUE' (specify default value for 'console' stanza). = How to Obtain the Upstart 1.4 Package = please add the following 'upstart-job-logging' PPA [3] to your system and give it a spin: sudo add-apt-repository ppa:jamesodhunt/upstart-job-logging = Feedback = Please provide feedback via a bug report: https://bugs.launchpad.net/ubuntu/+source/upstart/+filebug = Further Details on Features = Full details on these features can be found in the usual places: - init(5) - init(8) - cookbook: http://upstart.ubuntu.com/cookbook/#console-log http://upstart.ubuntu.com/cookbook/#configuration http://upstart.ubuntu.com/cookbook/#setuid http://upstart.ubuntu.com/cookbook/#setgid Thanks for your help. Kind regards, James. [1] - Usual caveats apply: do *not* install this on any critical systems. [2] - Two limitations to be aware of: - logging *only* currently applies to system jobs, - any job that produces output and ends *before* the disk becomes writeable will not currently have output logged. Note: both limitations are currently being addressed. [3] - https://launchpad.net/~jamesodhunt/+archive/upstart-job-logging -- James Hunt http://upstart.ubuntu.com/cookbook http://upstart.ubuntu.com/cookbook/upstart_cookbook.pdf -- ubuntu-devel mailing list ubuntu-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Re: Upstart helpers (abstract jobs and event aliases) for Oneiric
On 07/06/11 18:58, Kees Cook wrote:
> Hi James,
>
> On Tue, Jun 07, 2011 at 03:00:03PM +0100, James Hunt wrote:
>> == Proposal 2: Provide Abstract Jobs for Common Services ==
>> [...]
>> Option 1
>> [...]
>> For example, each display manager package would be updated such that its
>> .conf file specified:
>>
>> envDISPLAY_MANAGER=y
>> export DISPLAY_MANAGER
>>
>> Then, any job that requires a display manager could say:
>>
>> start on starting DISPLAY_MANAGER=y
>> [...]
>> Option 2
>> Create a Job Configuration File that hard-codes the list of known
>> service providers. For example for "display-manger", we could have:
>>
>> start on (starting gdm
>> or (starting kdm
>> or (starting lightdm
>> or (starting lxdm
>> or (starting slim
>> or (starting wdm
>> or starting xdm))
>>
>> stop on (stopping gdm
>> or (stopping kdm
>> or (stopping lightdm
>> or (stopping lxdm
>> or (stopping slim
>> or (stopping wdm
>> or stopping xdm))
>>
>> envABSTRACT_JOB=y
>> export ABSTRACT_JOB
>> [...]
>>
>> - We appear to have simply "moved the problem" since although we are no
>> longer hard-coding the list of display managers in "plymouth.conf", we
>> are hard-coding the list now in "display-manager.conf". However, we
>> have gained by doing this since:
>>
>> - We have still created the level of abstraction desired.
>> - We have contained the problem into a single file: we define the list
>> of display managers *once*.
>> - We don't need to update every Ubuntu (and potentially every Debian)
>> package as would be required for "Option 1".
>> - We *could* conceivably auto-generate "display-manager.conf" from the
>> archive with a simple script which munged the output of:
>>
>> apt-cache search x-display-manager|awk '{print $1}'|sort
>>
>> - The "ABSTRACT_JOB" variable would allow other jobs to detect that a
>> job was abstract (they shouldn't need to care, but just in case...)
>> This also avoids the unwieldliness of "abstract-display-manager" (or
>> even "abstract/display-manager" (were we to put the abstract jobs in
>> /etc/init/abstract/ say).
>
> What about merging the two options? I think it's a mistake to have
> hardcoded lists; this is like having /etc/service.conf instead of
> /etc/service.d/ for things to easily insert themselves into the system.
>
> Imagine, instead:
>
>start on (starting DISPLAY_MANAGER=y
> or (starting gdm
> or (starting kdm
> or (starting lightdm
> or (starting lxdm
> or (starting slim
> or (starting wdm
> or starting xdm)))
>
> Then anything not in the hardcoded list can just define itself as a
> DISPLAY_MANAGER to be included, and everything else can still start on the
> abstract job names too.
Great idea! This makes a lot of sense of we go the "kiss" route rather than
adopting a new stanza
(such as "alias" as suggested by Marc Dahlhaus).
>
> -Kees
>
--
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at:
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Upstart helpers (abstract jobs and event aliases) for Oneiric
es
is:
DISPLAY_MANAGER
FIREWALL
GRAPHICS_CARD
NETWORK
NETWORK_MANAGER
For example, each display manager package would be updated such that its
.conf file specified:
envDISPLAY_MANAGER=y
export DISPLAY_MANAGER
Then, any job that requires a display manager could say:
start on starting DISPLAY_MANAGER=y
= Observations =
- This might *appear* to be inefficient in that Upstart must check every
time *any* job starts to determine if DISPLAY_MANAGER=y is set in that
jobs environment. However, there is no additional overhead beyond how
Upstart currently works. Consider that...
start on starting gdm
... is actually an alias for:
start on stating JOB=gdm
- This would require *every* package that provides a service to be
updated. Admittedly this would only need to be done once though.
- Might be confusing since users you *must* specify "=y" exactly
(dropping the "=y" won't work, and nor will specifying "=Y" (or "=1")).
Option 2
Create a Job Configuration File that hard-codes the list of known
service providers. For example for "display-manger", we could have:
start on (starting gdm
or (starting kdm
or (starting lightdm
or (starting lxdm
or (starting slim
or (starting wdm
or starting xdm))
stop on (stopping gdm
or (stopping kdm
or (stopping lightdm
or (stopping lxdm
or (stopping slim
or (stopping wdm
or stopping xdm))
envABSTRACT_JOB=y
export ABSTRACT_JOB
= Observations =
- Since this is an Abstract Job, it will have no PID, but this job will
"run" for the duration of the first display manager to be invoked.
- We appear to have simply "moved the problem" since although we are no
longer hard-coding the list of display managers in "plymouth.conf", we
are hard-coding the list now in "display-manager.conf". However, we
have gained by doing this since:
- We have still created the level of abstraction desired.
- We have contained the problem into a single file: we define the list
of display managers *once*.
- We don't need to update every Ubuntu (and potentially every Debian)
package as would be required for "Option 1".
- We *could* conceivably auto-generate "display-manager.conf" from the
archive with a simple script which munged the output of:
apt-cache search x-display-manager|awk '{print $1}'|sort
- The "ABSTRACT_JOB" variable would allow other jobs to detect that a
job was abstract (they shouldn't need to care, but just in case...)
This also avoids the unwieldliness of "abstract-display-manager" (or
even "abstract/display-manager" (were we to put the abstract jobs in
/etc/init/abstract/ say).
Personal Preference
Although it looks rather ugly, my preference is currently for Option 2
primarily since it would be quick and easy to modify the single source
for each service. However, I could be persuaded otherwise :)
= Rationale =
By introducing Abstract Jobs and Event Alias "helpers", we can simplify
the existing Job Configuration Files and make them more understandable.
For example, the "start on" condition for "plymouth.conf" could become:
start on (starting mountall
or (shutdown and stopped display-manager))
Similarly, "gdm.conf" could become the much simpler / easier to comprehend:
start on (filesystem and (started dbus and graphics-device-available)
stop on shutdown
These changes may also incidentally minimise changes required by Ubuntu
derivatives once the helpers become pervasive.
= Documentation =
We will of course ensure that all these helpers are documented appropriately
and we plan to make
maximum use of them to ease the work required for [2].
-
[1] -
https://blueprints.launchpad.net/ubuntu/+spec/foundations-o-upstart-convert-main-initd-to-jobs
[2] -
http://summit.ubuntu.com/uds-o/meeting/foundations-o-upstart-convert-main-initd-to-jobs/
--
James Hunt
--
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at:
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Re: systemd for 11.10 ?
On 11/05/11 20:07, Patrick Goetz wrote: > On 05/11/2011 05:25 AM, James Hunt wrote: >> The best place to start is: >> >> https://wiki.ubuntu.com/NattyNarwhal/TechnicalOverviewUpstart >> > > This section: > > ...if a job configuration file specified the following complex condition: > > start on (A and (B or (starting C or (starting D or starting E > > The check-config command would flag an error if for example none of the > jobs 'C', 'D' or 'E' were available since that would indicate the job in > question could never be automatically started (since the start on > condition could never be true). > > > is incorrect as stated. If A and B are true, the job starts. Since the > proposition can be logically flattened > (B or (C or (D or E))) == B or C or D or E > I suspect the typo is the "or" after B should be "and" > > Actually, there is no error here. You are right that *if* events A and B are emitted, then the job will start. But that's not what the check-config command is telling you. The command tries to identify problems that *might* occur "in the future" (ie next boot). It is not considering what *did* happen when the system booted. So, in this example, we don't know if event A or B will actually be emitted. Because of this uncertainty, we also need to check the rest of the expression. See: http://upstart.at/2011/03/16/checking-jobs-and-events-in-ubuntu-natty/ James -- ubuntu-devel mailing list ubuntu-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Re: systemd for 11.10 ?
On 10/05/11 21:45, Phillip Susi wrote: > On 5/10/2011 4:15 PM, James Hunt wrote: >> If you are seeing strange behaviour, please raise bugs so we can look at >> them. > > It is a design limitation AFAIK, not a bug. At least the last time I > asked SJR about it, Upstart doesn't use cgroups to track children like > systemd does, and so it looses track of children of the jobs it creates, > such as programs the logged in user runs, and they can continue running > even though you stop the gdm job. > >> As of natty, Upstart has a socket bridge which is very similar to >> systemd's "socket activation" facility. Note that both products' socket >> features are probably going to be most effective for server-type apps >> though. > > Ahh, I hadn't noticed that. I'll have to read up on it. > The best place to start is: https://wiki.ubuntu.com/NattyNarwhal/TechnicalOverviewUpstart Regards, James. -- ubuntu-devel mailing list ubuntu-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Re: systemd for 11.10 ?
On 10/05/11 21:45, Phillip Susi wrote: > On 5/10/2011 4:15 PM, James Hunt wrote: >> If you are seeing strange behaviour, please raise bugs so we can look at >> them. > > It is a design limitation AFAIK, not a bug. At least the last time I > asked SJR about it, Upstart doesn't use cgroups to track children like > systemd does, and so it looses track of children of the jobs it creates, > such as programs the logged in user runs, and they can continue running > even though you stop the gdm job. > >> As of natty, Upstart has a socket bridge which is very similar to >> systemd's "socket activation" facility. Note that both products' socket >> features are probably going to be most effective for server-type apps >> though. > > Ahh, I hadn't noticed that. I'll have to read up on it. > The best place to start is: https://wiki.ubuntu.com/NattyNarwhal/TechnicalOverviewUpstart Regards, James. -- ubuntu-devel mailing list ubuntu-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Re: systemd for 11.10 ?
Hi Phillip, On 10/05/11 18:39, Phillip Susi wrote: > On 5/10/2011 6:46 AM, Steffen Barszus wrote: >> So the discussion should be on how to evaluate systemd , and set a >> number of criterias to benchmark both. Then the better one should be >> planned for slow migration. >> >> "Look its new and it has bells and whistles lets move to that" is not >> a valid argument for moving to a new init. > > I agree, so let's see if we can get the ball rolling in that direction. > > Some of the shortcomings I see with upstart that systemd sounds like it > addresses are: > > 1) The ability to drop back to single user mode for system > administration. Upstart can not make sure that all processes started > during multi user mode that are not supposed to be running in single > user mode are actually killed. If you are seeing strange behaviour, please raise bugs so we can look at them. > > 2) The ability to serialize state and re-execute itself so it can run > in the initramfs and then hand off to the real system. This is being considerd. I wasn't aware that systemd could run in initramfs. Regardless, I believe that Fedora is moving to dracut for initramfs. Whereas Ubuntu is planning to put Upstart into the initramfs for Oneiric. > > 3) The ability to figure out why a given service runs or does not run > when it does or should. Right now Upstart relies on comments being > added to the config files to give the admin clues about this, but that > is inherently unreliable and still difficult to figure out. See: http://upstart.at/2011/03/25/visualisation-of-jobs-and-events-in-ubuntu-natty/ Tomorrow, here at UDS we will be discussing the provision of new tooling for administrators that would give a more "traditional" view of services: https://blueprints.launchpad.net/ubuntu/+spec/foundations-o-upstart-for-admins > > 4) The increased parallelism systemd offers by running services at the > same time as those they "depend" on by having systemd create the socket > is not something upstart currently can do. I am not sure if it can be > extended to do this or not. > Systemd offers no increased parallelism over Upstart for starting and managing jobs that I'm aware of. Note that Upstart was designed from the ground up for performance and parallelism. As of natty, Upstart has a socket bridge which is very similar to systemd's "socket activation" facility. Note that both products' socket features are probably going to be most effective for server-type apps though. Regards, James. -- ubuntu-devel mailing list ubuntu-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Re: systemd for 11.10 ?
On 10/05/11 11:46, Steffen Barszus wrote: > "Look its new and it has bells and whistles lets move to that" is not > a valid argument for moving to a new init. > +1000. James. -- ubuntu-devel mailing list ubuntu-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Re: systemd for 11.10 ?
On 09/05/11 19:34, Patrick Goetz wrote: > I think the discussion found in this bug report makes a fairly > compelling argument that Ubuntu should probably migrate to systemd > sooner rather than later. Note particularly that a take away message > from this is that services like apache and postfix will need to continue > to use sysV init scripts until a complete rewrite of upstart has been > completed. > > https://bugs.launchpad.net/upstart/+bug/406397 > > Of course by all means let me know if I'm missing something here. I disagree. Yes there are some known issues with Upstart. I am sure there will be atleast as many issues with systemd (after all, nobody is even using it yet). Again, "new" != "superior". This is a nasty bug, but it does not require a complete rewrite of Upstart. Indeed, we are currently considering replacing the process tracking code to resolve this bug. > > > > James -- ubuntu-devel mailing list ubuntu-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Re: systemd for 11.10 ?
Patrick, On 09/05/11 16:58, Patrick Goetz wrote: >> From: Steve Langasek >> Date: Mon, 9 May 2011 11:45:56 +0200 >> >> And let's not forget that, for anyone tracking the LTS, upstart >> is*already* >> the system in use for the previous LTS, 10.04. The fundamentals of how >> upstart will work in 12.04 LTS are the same as in 10.04 LTS; upstart in >> 12.04 will include incremental improvements to the usability and >> instrumentability for system admins, this is not something *new* that >> admins >> are being asked to invest their time in learning. > > > Hi - > > Your points are well taken, and -- especially if not everyone agrees > that systemd is superior -- it's fair to keep upstart going for a while > before jumping to the conclusion that systemd is better. This comment still implies to me you consider systemd better, but I do not understand why you have this view? Everyone > immediately jumped on the X windows bandwagon many years ago (mostly as > a foil to Sun's NeWS system), and we haven't been unable to get out from > underneath using network stacks for locally displayed graphics for the > 20+ years since. > > My point about learning a new system was based on the observation that > there still seem to be a number of kinks in upstart; in particular, > we've had timing problems with autofs and statd, both of which > intermittently refused to start until we tweaked the upstart scripts. Why does your having to change a configuration file imply we should throw away a core part of the Ubuntu system? The problem you have described is a packaging/maintainer configuration file issue for 2 different packages, neither of which are Upstart itself. > Also, the fact that packages like apache still don't have an upstart > script indicates the technology isn't fully mature. Under these > circumstances, if everyone agreed that systemd is better, then it makes > more sense to switch sooner rather than later. > >> http://i.imgur.com/usftZ.png > > This is very funny, but there are some points raised in Lennart's > comparison: > > http://0pointer.de/blog/projects/why.html > > that merit careful reflection; e.g. shell-free bootup, mount/automount > handling, and disabling of services without editing files, to name a > few. There are other points that look interesting, but I don't know > enough about what he means to comment. Lots of features does not necessarily equate to a superior system. the design of Upstart follows "the Unix way": do one job and do it well. When required, Upstart relies on "helpers" (such as upstart-udev-bridge). This design is fundamentally sound - *iff* there were a problem in the udev bridge, it cannot bring down init, which would of course cause a kernel panic. Adding every possible feature into a critical system process such as init should only be considered with extreme caution IMHO. No one, however, can tell me > that the accepted method for disabling services in upstart isn't a > kludge! <:) The rationale for Upstarts behaviour, which is of course different to SysV-like systems, is that if you install a service, you'll want it started on boot: guaranteed. In 9 out of 10 cases, this is the right answer. For the remaining case, say a database dev who has a handful of db engines installed, but doesn't want them all auto-started on her underpowered laptop, we provide the ability to disable that job. Yes, the interface is simple, but does it really need to be any more complex than it is? > > > James. -- James Hunt Ubuntu Foundations Team, Canonical. -- ubuntu-devel mailing list ubuntu-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Re: systemd for 11.10 ?
Hi Martin, On 09/05/11 21:01, Martin Pitt wrote: > Steve Langasek [2011-05-09 12:36 +0200]: >> It's one thing to have a rollback plan for which desktop experience is >> shipped by default; that touches a handful of closely related packages. >> It's quite another to try to roll back an init system change, which touches >> every single package that's involved with system startup and requires >> changes across a very broad set of foundations, desktop, and server >> packages. > > If we should ever do the migration to systemd, .. Which to be clear, we are not considering currently. then we should > certainly not drop the upstart integration/jobs along the way, but > keep both upstart and systemd jobs in the packages for a while (at > least until the next LTS), so that you can switch between the two with > the init= kernel parameter (that's in fact how you can test systemd on > Debian/Ubuntu/everywhere else today). I agree in principle that we should do this *should* we ever consider moving to systemd, but there is the issue of ensuring that both the Upstart and the systemd jobs are guaranteed to result in the same behaviour. How we would do this, I don't currently know. Then rolling back is by and > large just a seed change, and perhaps flipping back an option in grub. Well, let's not forget that we are about to put Upstart into the initramfs for Oneiric. Removing Upstart is not in any plan I'm aware of, but to be clear to all, *if* we ever did do it, it would take a **considerable** effort to do IMHO so would need very careful justification, planning, and extremely thorough testing. > > This would even allow us to do things like installing systemd for new > installs, but keeping upstart for upgrades until there is a transition > plan for how to deal with custom upstart jobs that administrators > created (which seems to be the most difficult thing to handle in the > transition). > > Martin > > James -- ubuntu-devel mailing list ubuntu-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Re: Test version of Upstart with full chroot support available
On 11/04/11 16:01, Clint Byrum wrote: > Excerpts from James Hunt's message of Fri Apr 08 08:51:48 -0700 2011: >> -BEGIN PGP SIGNED MESSAGE- >> Hash: SHA1 >> >> Hi All, >> >> As a precursor to pushing this update out to Natty next week, I've >> updated my upstart-testing PPA with Upstart version 0.9.5-1ubuntu1: >> >> ppa:jamesodhunt/upstart-testing >> >> Code is here: >> >> lp:~jamesodhunt/ubuntu/natty/upstart/fix-chroot-sessions >> >> As the name suggests, chroots should now work fully [1], but we are keen >> to solicit feedback from the community. > > FYI, on my natty box when I was running this, installing dbus in a schroot > session resulted in upstart consuming all available virtual memory and > eventually crashing the box. > > Steps to reproduce: > > (assuming you've setup schroots w/ mk-sbuild): > > schroot -c natty-amd64 -u root > apt-get install dbus > > > At the 'setting up dbus' point, upstart starts to consume memory at an > alarming rate. > > I do need to try and build php w/ sbuild.. now would be an excellent time > > to have chroot support so it doesn't die when it tries to start mysql server. :) > This is likely because the dbus upstart job has a post-start that sends > USR1 to pid 1, which is supposed to tell it to re-connect to dbus. > > I believe the bug is because the USR1 handler needs to ignore requests > to re-connect to dbus from chrooted processes, but I haven't gotten very > deep in to debugging it yet. > Hi Clint, I've now fixed this bug and updated the ppa with upstart-0.9.6-1ubuntu1~jh1. Feel free to re-test and let me know how it goes. Cheers, James. -- James Hunt Ubuntu Foundations Team, Canonical. -- ubuntu-devel mailing list ubuntu-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Re: Test version of Upstart with full chroot support available
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 11/04/11 16:01, Clint Byrum wrote: > Excerpts from James Hunt's message of Fri Apr 08 08:51:48 -0700 2011: >> -BEGIN PGP SIGNED MESSAGE- >> Hash: SHA1 >> >> Hi All, >> >> As a precursor to pushing this update out to Natty next week, I've >> updated my upstart-testing PPA with Upstart version 0.9.5-1ubuntu1: >> >> ppa:jamesodhunt/upstart-testing >> >> Code is here: >> >> lp:~jamesodhunt/ubuntu/natty/upstart/fix-chroot-sessions >> >> As the name suggests, chroots should now work fully [1], but we are keen >> to solicit feedback from the community. > > FYI, on my natty box when I was running this, installing dbus in a schroot > session resulted in upstart consuming all available virtual memory and > eventually crashing the box. > > Steps to reproduce: > > (assuming you've setup schroots w/ mk-sbuild): > > schroot -c natty-amd64 -u root > apt-get install dbus > > > At the 'setting up dbus' point, upstart starts to consume memory at an > alarming rate. > > This is likely because the dbus upstart job has a post-start that sends > USR1 to pid 1, which is supposed to tell it to re-connect to dbus. > > I believe the bug is because the USR1 handler needs to ignore requests > to re-connect to dbus from chrooted processes, but I haven't gotten very > deep in to debugging it yet. > Hi Clint, Thanks for highlighting this. It actually looks like a namespace leak that is causing the issue - I'm investigating now... Cheers, James. - -- James Hunt Ubuntu Foundations Team, Canonical. -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk2jGgcACgkQYBWEaHcQG9f6lQCfZwD+qOMnyUle0HCPZ9vtv6KO FHIAn1MdOsF/FLhToR0qWadRrBoYeviF =cSt6 -END PGP SIGNATURE- -- ubuntu-devel mailing list ubuntu-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Test version of Upstart with full chroot support available
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Hi All, As a precursor to pushing this update out to Natty next week, I've updated my upstart-testing PPA with Upstart version 0.9.5-1ubuntu1: ppa:jamesodhunt/upstart-testing Code is here: lp:~jamesodhunt/ubuntu/natty/upstart/fix-chroot-sessions As the name suggests, chroots should now work fully [1], but we are keen to solicit feedback from the community. Additional changes: - - includes Clints reload fix to /lib/init/upstart-job (lp:~clint-fewbar/ubuntu/natty/upstart/upstart-job-change-restart) - - init-checkconf(8) now checks Upstart stanza syntax and script sections. [1] - https://wiki.ubuntu.com/NattyNarwhal/TechnicalOverviewUpstart Kind regards, James. - -- James Hunt Ubuntu Foundations Team, Canonical. -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk2fLwwACgkQYBWEaHcQG9er9QCfextMgQSczbWLaHACNc6DWLBS EuoAnjf4qUjOxVdPiN6xlPmJ2kpgPdmw =xQuf -END PGP SIGNATURE- -- ubuntu-devel mailing list ubuntu-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Re: Upstart Cookbook
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1
On 24/03/11 22:11, Scott James Remnant wrote:
> On Thu, Mar 24, 2011 at 2:34 PM, James Hunt <mailto:james.h...@canonical.com>> wrote:
>
>> Clint and I have been hard at work on an "Upstart Cookbook". Although it
>> is "early days", we wanted to let you all know we're working on this
>> project. Our (still *very* draft!) efforts can be viewed here:
>>
> Nice work, some comments on the copy of 1-4 below:
>
> Although Upstart is used on on a number of different Operating
> Systems (including Ubuntu, Google's Chromium OS and Google's Chrome
> OS), the Ubuntu version is considered the "reference
> implementation". This is primarily due to the fact that Upstart was
> written specifically for Ubuntu (although this does not mean that it
> cannot run on any other Linux-based system).
>
>
> I'd disagree with this. Reference implementation always implies that
> other implementations should copy it as much as possible, and Ubuntu is
> no way that. As long as Ubuntu still uses a hybrid of Sys V and Upstart
> jobs, it can never be a reference implementation.
Thanks Scott - cookbook updated on this point.
>
> It may be that you mean what the document corresponds to, in which case
> use a different term ;-)
>
> A notification sent by Upstart to all interested parties (either
> jobs or other events). They can be thought of as "signals". Events
> are /emitted/(created and then broadcast) to the entire Upstart system.
>
>
> Events can be more than just signals, I've made a point of documenting
> this recently, so this just confuses the issue.
>
> * Events are like Signals
> <http://upstart.at/2010/12/08/events-are-like-signals/>
> * Events are like Methods
> <http://upstart.at/2010/12/16/events-are-like-methods/>
> * Events are like Hooks
> <http://upstart.at/2011/01/06/events-are-like-hooks/>
I've further clarified this by creating new sections for each type of
event. Note too that the new upstart-events manual page on an Ubuntu
Natty system ("man 7 upstart-events") already shows these types in
Tables (1) and (2).
>
> Scott
Cheers,
James.
- --
James Hunt
Ubuntu Foundations Team, Canonical.
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.10 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iEYEARECAAYFAk2MgS4ACgkQYBWEaHcQG9f7zACdHHjEoXpYa/PMcAxeUTEzjKh3
xyMAnAxPx9zFMpS4y25QaJw1A6lt++HI
=9D8o
-END PGP SIGNATURE-
--
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at:
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Re: Upstart Cookbook
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 29/03/11 02:00, Scott James Remnant wrote: > The simple answer there is that the way in which events interact with > tasks vs. services is that they are *identical* :D > > "started" is always emitted after post-start and before the main > process, whether or not it's a task or service > > On Mon, Mar 28, 2011 at 5:58 PM, Evan Broder wrote: >> On Mon, Mar 28, 2011 at 3:32 PM, Scott James Remnant >> wrote: >>> 4.1.1 & 4.1.2 - worth explaining the real difference between "task" >>> and "service" here, perhaps? or later? >> >> +1 from me on that. In particular, I'm interested in how >> starting/started/stopping/stopped events interact with tasks. >> slangasek and I tried to work this out on IRC earlier and were unable >> to do so to our satisfaction. >> > Scott - thanks for clarifying this. Evan - I've added a "Job Lifecycle" section which (hopefully) covers all this: http://upstart.ubuntu.com/cookbook/#job-lifecycle Cheers, James. - -- James Hunt Ubuntu Foundations Team, Canonical. -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk2RsU4ACgkQYBWEaHcQG9cTOQCfSE02WlKKIx41NOadCzYChgbU c7AAn2A1JChfnGTa9+sqPknXkYKznPbA =CDC/ -END PGP SIGNATURE- -- ubuntu-devel mailing list ubuntu-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Upstart Cookbook
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Hi All, Clint and I have been hard at work on an "Upstart Cookbook". Although it is "early days", we wanted to let you all know we're working on this project. Our (still *very* draft!) efforts can be viewed here: http://upstart.ubuntu.com/cookbook/ Source: lp:upstart-cookbook This document is being updated frequently and is certainly not finished. We're currently collecting content [1] but any comments, suggestions and bug reports welcome: https://bugs.launchpad.net/upstart-cookbook/+filebug Enjoy. James. [1] - note that we haven't added in the Natty Features yet - manana! - -- James Hunt Ubuntu Foundations Team, Canonical. -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk2LuNYACgkQYBWEaHcQG9fxBACfbjHTV0Ijn/YBFd+2WAMEzOVA DEMAn2naiJD0iJJ0ayeQ2T/+NrV/TvGn =v9Eg -END PGP SIGNATURE- -- ubuntu-devel mailing list ubuntu-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Upstart 0.9.2 PPA for testing
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1
Hi All,
As a precursor to pushing this update out to Natty next week, I've
updated my upstart-testing PPA with Upstart 0.9.2:
ppa:jamesodhunt/upstart-testing
Code is here:
lp:~jamesodhunt/ubuntu/natty/upstart/proposed-stage2
The major changes (which will be added to
(https://wiki.ubuntu.com/NattyNarwhal/TechnicalOverview) are...
Basic Job/Event Visualisation
Upstart provides a new Python script `initctl2dot` which converts the
output of the new `initctl show-config` command to
[[http://www.graphviz.org/|GraphViz]] format. By default, all job
configuration files are considered and the links between jobs and events
are displayed graphically. Additionally, it is possible to list a set of
jobs to graph. Run "{{{initctl2dot --help}}}" or "{{{man initctl2dot}}}"
for full options.
New Initctl Commands
Two new `initctl` commands have been added:
1. `show-config`
1. `check-config`
The `show-config` command displays core job configuration details,
namely the `start on`, `stop on` and `emits` stanza information. This is
useful since it allows users to see how Upstart has parsed the job
configuration files. Additionally, the `show-config` command accepts an
optional `--enumerate` option which makes it easy to see which elements
of complex conditions are jobs, which are events and which are
environment details. This option forms the basis of the Visualisation
feature above.
The `check-config` command is useful for System Adminstrators and
tooling to ensure that all jobs are theoretically startable/stoppable.
For example, if a job configuration file specified the following complex
condition:
{{{
start on (A and (B or (starting C or (starting D or starting E
}}}
The `check-config` command would flag an error if for example none of
the jobs 'C', 'D' or 'E' were available since that would indicate
the job in question could never be automatically started (since the
start on condition could never be true). Similar checks are performed on
events, so if jobs 'C', 'D' and 'E' are available but events 'A' and 'B'
are not advertised as being emitted by any job, 'check-config' will
generate an error. If no errors are detected, `check-config` displays no
output and returns zero. If errors are detected for a job, each
condition that is unsatisfiable is displayed with a message.
Miscellaneous
* A new manual page, `upstart-events (8)` summarising well-known
Upstart events.
* An updated [[http://www.vim/org|Vim]] syntax file has been included.
* A Bash completion script has been included for `initctl`.
* A new script `init-checkconf` has been provided which allows a user
to check an individual job configuration file to ensure it is
syntactically correct before installing it in "`/etc/init/`". Run
"{{{init-checkconf -h}}}" or "{{{man init-checkconf}}}" for full options.
* The [[http://www.vim/org|Vim]] package "{{{vim-runtime}}}" now
includes syntax support for upstart job configuration files.
- --
Cheers,
James.
- --
James Hunt
Ubuntu Foundations Team, Canonical.
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.10 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iEYEARECAAYFAk16EFIACgkQYBWEaHcQG9eYSwCfXSGgqDlX8F6mN8UONzDHlyVZ
LAYAnRivuH8Lj1Q1uuqZNn6u7Csr9RtN
=8LZc
-END PGP SIGNATURE-
--
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at:
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
