[Python-Dev] Automatic installer builds (was Re: Fwd: Broken link to download (Mac OS X))

[Nick Coghlan]

Martin v. Löwis wrote:
>> Wasn't that problem fixed weeks ago?  The installer image has been 
>> available there since several days after the release.  And the link 
>> seems fine now.
> 
> The inherent problem remains. There is no binary for 2.7b1, for example.
> The last binaries produced in the 2.7 testing process were for 2.7a2.

I seem to recall there was some issue (aside from the current lack of a
reliable OS X buildbot) that prevented us from regularly grabbing the
head of the tree and using it to automatically build the Windows and Mac
installers (to check that the installers could actually be created,
preventing a repeat of the Mac situation with 2.7b1).

Am I misremembering the discussion from last time this topic came up?

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
---
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Automatic installer builds (was Re: Fwd: Broken link to download (Mac OS X))

[C. Titus Brown]

On Thu, Apr 15, 2010 at 12:47:50AM +1000, Nick Coghlan wrote:
> Martin v. L?wis wrote:
> >> Wasn't that problem fixed weeks ago?  The installer image has been 
> >> available there since several days after the release.  And the link 
> >> seems fine now.
> > 
> > The inherent problem remains. There is no binary for 2.7b1, for example.
> > The last binaries produced in the 2.7 testing process were for 2.7a2.
> 
> I seem to recall there was some issue (aside from the current lack of a
> reliable OS X buildbot) that prevented us from regularly grabbing the
> head of the tree and using it to automatically build the Windows and Mac
> installers (to check that the installers could actually be created,
> preventing a repeat of the Mac situation with 2.7b1).
> 
> Am I misremembering the discussion from last time this topic came up?

Making the Windows binary build process automatic and robust is challenging
if you hate Windows (like I do).  Martin also made the point that it's
been broken forever, so people don't seem to care :).  ISTR Martin just
makes them manually.

It's on my TODO list to robustify this process.

OTOH, my Mac automated builds/tests are working just fine.  I could produce
nightly binaries from those easily enough:

http://lyorn.idyll.org/ctb/pb-dev/p/python/

Just need to figure out the magic doohickey commands... will add to list.

cheers,
--titus
-- 
C. Titus Brown, c...@msu.edu
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Automatic installer builds (was Re: Fwd: Broken link to download (Mac OS X))

[Ronald Oussoren]

 
On Wednesday, April 14, 2010, at 04:47PM, "Nick Coghlan"  
wrote:
>Martin v. Löwis wrote:
>>> Wasn't that problem fixed weeks ago?  The installer image has been 
>>> available there since several days after the release.  And the link 
>>> seems fine now.
>> 
>> The inherent problem remains. There is no binary for 2.7b1, for example.
>> The last binaries produced in the 2.7 testing process were for 2.7a2.

That's because I haven't had time yet to actually build the installer for OSX 
due to lack of free time because both my professional and private lives are 
fully filled at the moment (which is great, but leaves very little time to hack 
on Python or PyObjC).

>
>I seem to recall there was some issue 

> (aside from the current lack of a
>reliable OS X buildbot)

Speaking of which... I have a mac-mini that could be used for a buildbot. How 
much work is needed to kickstart a buildbot, taking into account that I'd 
prefer to have a buildbot with different configure-flags that the default unix 
build (that is, I want to test a framework build that is a universal binary).

 that prevented us from regularly grabbing the
>head of the tree and using it to automatically build the Windows and Mac
>installers (to check that the installers could actually be created,
>preventing a repeat of the Mac situation with 2.7b1).

Creating the Mac installer is easy: just run Mac/BuildScript/build-installer.py 
on an OSX 10.5 system where a local version of Tcl/Tk 8.4 is installed in 
/Library/Frameworks. The system should also not have fink or darwinports and a 
clean /usr/local tree to avoid contaminating the build.

Ronald
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Automatic installer builds (was Re: Fwd: Broken link to download (Mac OS X))

[skip]

Ronald> Creating the Mac installer is easy: just run
Ronald> Mac/BuildScript/build-installer.py on an OSX 10.5 system where a
Ronald> local version of Tcl/Tk 8.4 is installed in
Ronald> /Library/Frameworks. The system should also not have fink or
Ronald> darwinports and a clean /usr/local tree to avoid contaminating
Ronald> the build.

For those of us who live in an X11 world and use tools which Apple doesn't
provide, like it or not, Fink or MacPorts are often a practical necessity.
Would it be sufficient to modify the environment so that /sw, /opt/local and
/usr/local don't appear in any paths when the script is run?  On my laptop
(MacPorts installed, not Fink) I see that MANPATH, PATH and INFOPATH are
currently "polluted" with /opt/local.  MANPATH and PATH contain /usr/local.

Skip
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Automatic installer builds (was Re: Fwd: Broken link to download (Mac OS X))

[Bill Janssen]

s...@pobox.com wrote:

> 
> Ronald> Creating the Mac installer is easy: just run
> Ronald> Mac/BuildScript/build-installer.py on an OSX 10.5 system where a
> Ronald> local version of Tcl/Tk 8.4 is installed in
> Ronald> /Library/Frameworks. The system should also not have fink or
> Ronald> darwinports and a clean /usr/local tree to avoid contaminating
> Ronald> the build.
> 
> For those of us who live in an X11 world and use tools which Apple doesn't
> provide, like it or not,

Actually, Skip, you're talking about my world there (X11 and GNU Emacs
and Gimp and ...), and I don't use Fink or MacPorts on my Macs.

> Fink or MacPorts are often a practical necessity.

Fink is deadly, MacPorts much more benign, in my experience.  Which is
several years out-of-date, before I realized I didn't need either one of
them, and before the UNIX community started adding configure patches to
support OS X builds more widely.  Perhaps they've improved.

In any case, they shouldn't be needed on buildbots maintained by the PSF.

> Would it be sufficient to modify the environment so that /sw, /opt/local and
> /usr/local don't appear in any paths when the script is run?  On my laptop
> (MacPorts installed, not Fink) I see that MANPATH, PATH and INFOPATH are
> currently "polluted" with /opt/local.  MANPATH and PATH contain /usr/local.

Probably fine on your personal Mac.  And the build scripts can probably
mask those out on their own.

Bill
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Automatic installer builds (was Re: Fwd: Broken link to download (Mac OS X))

[Martin v. Löwis]

> I seem to recall there was some issue (aside from the current lack of a
> reliable OS X buildbot) that prevented us from regularly grabbing the
> head of the tree and using it to automatically build the Windows and Mac
> installers (to check that the installers could actually be created,
> preventing a repeat of the Mac situation with 2.7b1).
> 
> Am I misremembering the discussion from last time this topic came up?

I only remember a similar discussion, which was about why the Windows
nightly builds had been dropped. The reason was that the process was too
flaky and required too much manual fixing, and that the demand for these
binaries was too low. I don't recall OSX being mentioned in that
discussion, though.

Regards,
Martin
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Automatic installer builds (was Re: Fwd: Broken link to download (Mac OS X))

[Martin v. Löwis]

> Speaking of which... I have a mac-mini that could be used for a
> buildbot. How much work is needed to kickstart a buildbot, taking
> into account that I'd prefer to have a buildbot with different
> configure-flags that the default unix build (that is, I want to test
> a framework build that is a universal binary).

The configure flags are defined on the master, so if I know what they
should be, I can arrange your slave to see them. You can't then change
them easily.

On the slave, you need to install buildbot, and create a slave
configuration; it would the be good if the slave process would somehow
get restarted automatically after a system reboot (I think there are
recipes for that out there).

> Creating the Mac installer is easy: just run
> Mac/BuildScript/build-installer.py on an OSX 10.5 system where a
> local version of Tcl/Tk 8.4 is installed in /Library/Frameworks. The
> system should also not have fink or darwinports and a clean
> /usr/local tree to avoid contaminating the build.

Buildbot also supports uploading binaries to the master, which we did
for the Windows builds. We could do that for OSX as well (in which case
the release manager might be able to trigger a release build by just
passing the right branch name in the master UI).

Regards,
Martin
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Automatic installer builds (was Re: Fwd: Broken link to download (Mac OS X))

[Martin v. Löwis]

> Probably fine on your personal Mac.  And the build scripts can probably
> mask those out on their own.

For a private Python installation, it doesn't actually hurt to have them
on disk. The binaries may be linked with strange libraries, but it
should work since the libraries themselves are also on disk. It's only
that they would poison an installer - the binaries would work on no
other machine.

Regards,
Martin
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Automatic installer builds (was Re: Fwd: Broken link to download (Mac OS X))

[skip]

Bill> In any case, they shouldn't be needed on buildbots maintained by
Bill> the PSF.

Sure.  My question was related to humans building binary distributions
though.  Unless that becomes fully automated so the release manager can just
push a button and have it built on and as-yet-nonexistent Mac OSX buildbot
machine, somebody will have to generate that installer.  Ronald says Fink,
MacPorts and /usr/local are poison.  If that's truly the case that's fine.
It's just that it reduces the size of the potential binary installer build
machines.

Now that I think about it, it might not be sufficient to just hide those
directories from the environment.  The Python setup.py file has
unconditional hard-coded references to /sw, /opt/local and /usr/local.  That
would probably have to change before you could use an "infected" machine to
build the binary installer.  (At the very least, searching /sw/include (for
instance) could be suppressed if /sw/bin was not found in PATH.  Similarly
for /opt/local and /usr/local.)

(ISTM the same might be true should people ever decide to once again build a
Solaris installer.  /opt/sfw is currently searched for Berkeley DB include
files.)

Skip
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Automatic installer builds (was Re: Fwd: Broken link to download (Mac OS X))

[skip]

Martin> On the slave, you need to install buildbot, and create a slave
Martin> configuration; it would the be good if the slave process would
Martin> somehow get restarted automatically after a system reboot (I
Martin> think there are recipes for that out there).

A static IP address makes things a bit easier as well, though it's not
impossible to run a buildbot using a dynamic IP address assuming you use
dyndns.org or something similar to maintain the name-to-IP mapping.

I ran a Mac OSX buildbot for the community buildbots for awhile but never
did figure out at the time how to get it to fire up on reboot.  That was a
few years ago.  I think today that would easily be accomplished with an
@reboot crontab entry.  Dunno if that was available way back then, but it's
certainly supported on Mac OSX now.

Skip
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Automatic installer builds (was Re: Fwd: Broken link to download (Mac OS X))

[Martin v. Löwis]

> Making the Windows binary build process automatic and robust is challenging
> if you hate Windows (like I do).  Martin also made the point that it's
> been broken forever, so people don't seem to care :).  ISTR Martin just
> makes them manually.

That's true. In particular, people had been asking that the MSI
installers are code-signed, which I now also do. I wouldn't like to keep
the private key for the PSF code signing certificate available to some
machine that is readily accessible from the internet - so at least that
part is manual, in the sense that I have to provide my password.

Regards,
Martin
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Automatic installer builds (was Re: Fwd: Broken link to download (Mac OS X))

[Martin v. Löwis]

> (ISTM the same might be true should people ever decide to once again build a
> Solaris installer.  /opt/sfw is currently searched for Berkeley DB include
> files.)

And rightly so (likewise for Fink). Primarily, the script is there to
help people installing Python; packaging Python can be more difficult.

I guess ActiveState has solved the problem (probably by not populating
/opt/sfw on the Solaris machine that builds ActivePython).

Regards,
Martin
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Automatic installer builds (was Re: Fwd: Broken link to download (Mac OS X))

[Simon Brunning]

On 14 April 2010 20:02,   wrote:
> I ran a Mac OSX buildbot for the community buildbots for awhile but never
> did figure out at the time how to get it to fire up on reboot.  That was a
> few years ago.  I think today that would easily be accomplished with an
> @reboot crontab entry.  Dunno if that was available way back then, but it's
> certainly supported on Mac OSX now.

launchd might be a better bet - .

-- 
Cheers,
Simon B.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Automatic installer builds (was Re: Fwd: Broken link to download (Mac OS X))

[Zvezdan Petkovic]

On Apr 14, 2010, at 3:30 PM, Simon Brunning wrote:

> On 14 April 2010 20:02,   wrote:
>> I ran a Mac OSX buildbot for the community buildbots for awhile but never 
>> did figure out at the time how to get it to fire up on reboot.  That was a 
>> few years ago.  I think today that would easily be accomplished with an 
>> @reboot crontab entry.  Dunno if that was available way back then, but it's 
>> certainly supported on Mac OSX now.
> 
> launchd might be a better bet - .

Exactly, launchd is the program intended for such use.
A small nitpick at that property list shown in the linked example:
It should be "Apple" now instead of "Apple Computer".
They officially changed the name and all the new developer documentation shows 
the property lists with "-//Apple/DTD ..."
:-)

Regards,

Zvezdan

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Automatic installer builds (was Re: Fwd: Broken link to download (Mac OS X))

[Ronald Oussoren]

On 14 Apr, 2010, at 19:41, s...@pobox.com wrote:

> 
>Ronald> Creating the Mac installer is easy: just run
>Ronald> Mac/BuildScript/build-installer.py on an OSX 10.5 system where a
>Ronald> local version of Tcl/Tk 8.4 is installed in
>Ronald> /Library/Frameworks. The system should also not have fink or
>Ronald> darwinports and a clean /usr/local tree to avoid contaminating
>Ronald> the build.
> 
> For those of us who live in an X11 world and use tools which Apple doesn't
> provide, like it or not, Fink or MacPorts are often a practical necessity.
> Would it be sufficient to modify the environment so that /sw, /opt/local and
> /usr/local don't appear in any paths when the script is run?  On my laptop
> (MacPorts installed, not Fink) I see that MANPATH, PATH and INFOPATH are
> currently "polluted" with /opt/local.  MANPATH and PATH contain /usr/local.

/usr/local/lib and /usr/local/include must be empty, or at least not contain 
anything that might be used by the python build. AFAIK it is practically 
impossible[1] to tell the compiler to exclude /usr/local from its search paths. 

Fink and Macports must be disabled because Python's setup.py actively looks in 
those directory for dependencies (such as for BerkelyDB).   I wouldn't mind 
having a configure switch that disables looking in the default Fink and 
Macports trees, but don't have time to work on that myself.

BTW. Excluding these locations is only needed when building the official 
installer, you can obviously just keep them around when you build and use 
Python locally.

I tend to move /usr/local aside when I build the installer and move it back 
afterwards, and have used an mostly empty virtual machine for the last couple 
of releases.

Ronald


[1] that is, the only way I've found it to tell gcc not to use any search paths 
at all and manually specify all search paths, including compiler-specific 
directories.
> 
> Skip



smime.p7s
Description: S/MIME cryptographic signature
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Automatic installer builds (was Re: Fwd: Broken link to download (Mac OS X))

[Ronald Oussoren]

On 14 Apr, 2010, at 20:46, Martin v. Löwis wrote:

>> Speaking of which... I have a mac-mini that could be used for a
>> buildbot. How much work is needed to kickstart a buildbot, taking
>> into account that I'd prefer to have a buildbot with different
>> configure-flags that the default unix build (that is, I want to test
>> a framework build that is a universal binary).
> 
> The configure flags are defined on the master, so if I know what they
> should be, I can arrange your slave to see them. You can't then change
> them easily.

That's fine. I'll first create a stable environment for performing builds/tests 
and contact you when I'm ready to be added to the buildbot farm.

> 
> On the slave, you need to install buildbot, and create a slave
> configuration; it would the be good if the slave process would somehow
> get restarted automatically after a system reboot (I think there are
> recipes for that out there).

That should be easy enough to arange. The buildbot manual mentions an @reboot 
crontab entry, but a launchd item should be easy enough and is slightly cleaner.


> 
>> Creating the Mac installer is easy: just run
>> Mac/BuildScript/build-installer.py on an OSX 10.5 system where a
>> local version of Tcl/Tk 8.4 is installed in /Library/Frameworks. The
>> system should also not have fink or darwinports and a clean
>> /usr/local tree to avoid contaminating the build.
> 
> Buildbot also supports uploading binaries to the master, which we did
> for the Windows builds. We could do that for OSX as well (in which case
> the release manager might be able to trigger a release build by just
> passing the right branch name in the master UI).

How would that work? Creation of the OSX installer is not integrated in the 
regular Makefiles but is a separate script.

A slightly problematic issue is that the machine I want to use for the buildbot 
is running OSX 10.6 and creating the binary installer doesn't work on 10.6 yet, 
but that should be easy enough to fix when I look into it.

Ronald
> 
> Regards,
> Martin



smime.p7s
Description: S/MIME cryptographic signature
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Automatic installer builds (was Re: Fwd: Broken link to download (Mac OS X))

[skip]

>> I ran a Mac OSX buildbot for the community buildbots for awhile but
>> never did figure out at the time how to get it to fire up on
>> reboot.  That was a few years ago.  I think today that would easily
>> be accomplished with an @reboot crontab entry.  Dunno if that was
>> available way back then, but it's certainly supported on Mac OSX now.

Simon> launchd might be a better bet - .

Sure, but it's Mac-specific.  For someone like myself who moves between
different platforms I generally try to use tools which work across those
platforms to the extent that I can.  While @reboot is not universally
available (Solaris doesn't support it as far as I can tell) it is still more
common where I operate than launchd.  Not to mention which I have to horse
around with XML for config files which I completely detest for that purpose.

Skip
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Automatic installer builds (was Re: Fwd: Broken link to download (Mac OS X))

[Martin v. Löwis]

> How would that work? Creation of the OSX installer is not integrated
> in the regular Makefiles but is a separate script.

Again by setting up another builder (and assigning it to either the same
or a different slave). Instead of the regular configure/make/make
test/make clean builder, this one would configure/run build
script/upload/clean. Essentially, a builder will invoke arbitrary shell
scripts on the slave, which will (slavishly :-) perform them.

So make sure this can run under an untrusted account; anybody with
control over the master can get access to all slaves (but we do keep
access to the master very tightly restricted). In addition, all
committers basically have control over all slaves (by committing stuff
into the makefile); this is, of course, monitored through the checkins list.

I recommend we look at daily installers after the regular slave is set up.

> A slightly problematic issue is that the machine I want to use for
> the buildbot is running OSX 10.6 and creating the binary installer
> doesn't work on 10.6 yet, but that should be easy enough to fix when
> I look into it.

So we should definitely take one step at a time.

Regards,
Martin
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Automatic installer builds (was Re: Fwd: Broken link to download (Mac OS X))

[Nick Coghlan]

Martin v. Löwis wrote:
>> I seem to recall there was some issue (aside from the current lack of a
>> reliable OS X buildbot) that prevented us from regularly grabbing the
>> head of the tree and using it to automatically build the Windows and Mac
>> installers (to check that the installers could actually be created,
>> preventing a repeat of the Mac situation with 2.7b1).
>>
>> Am I misremembering the discussion from last time this topic came up?
> 
> I only remember a similar discussion, which was about why the Windows
> nightly builds had been dropped. The reason was that the process was too
> flaky and required too much manual fixing, and that the demand for these
> binaries was too low. I don't recall OSX being mentioned in that
> discussion, though.

Yep, that sounds like the discussion I was thinking of.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
---
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Automatic installer builds (was Re: Fwd: Broken link to download (Mac OS X))

[Bill Janssen]

s...@pobox.com wrote:

> Now that I think about it, it might not be sufficient to just hide those
> directories from the environment.  The Python setup.py file has
> unconditional hard-coded references to /sw, /opt/local and /usr/local.  That

I added that to my virus scanner: check any setup.py files for /sw.  I
autopatch a couple of the commopn ones to get rid of it.

Bill
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Automatic installer builds (was Re: Fwd: Broken link to download (Mac OS X))

[Stephen J. Turnbull]

Bill Janssen writes:

 > > Fink or MacPorts are often a practical necessity.
 > 
 > Fink is deadly, MacPorts much more benign, in my experience.  Which is
 > several years out-of-date, before I realized I didn't need either one of
 > them, and before the UNIX community started adding configure patches to
 > support OS X builds more widely.  Perhaps they've improved.

FWIW, I use MacPorts heavily, and it works fine for the most recent
release of Mac OS X which has been out for at least 6 months.  Bug
fixes for Snow Leopard in MacPorts are typically a matter of a handful
of hours, but many port maintainers don't have full-time access to
Leopard, and few will do more than pay lip service and help install a
patch if you have problems with Tiger.  IOW, the previous release
always has occasional glitches, and the second previous release is a
nightmare.  MacPorts is fine if you don't mind occasionally messing
with it and don't need to deal with Mac OS versions getting even
slightly long in the tooth.

 > In any case, they shouldn't be needed on buildbots maintained by
 > the PSF.

They should be avoided, as many users won't have them.  MacPorts and
presumably Fink have their own testing process, and the quality of
/opt/local/lib varies quite a bit.

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Automatic installer builds (was Re: Fwd: Broken link to download (Mac OS X))

[David Bolen]

Ronald Oussoren  writes:

> Speaking of which... I have a mac-mini that could be used for a
> buildbot. How much work is needed to kickstart a buildbot, taking
> into account that I'd prefer to have a buildbot with different
> configure-flags that the default unix build (that is, I want to test
> a framework build that is a universal binary).

Sort of picking this message to enter the thread ...

Interestingly enough, I had made an offer to Martin to host an OSX
build slave last week (before this thread), since I had a Mini whose
disk crashed and would be reasonably free after I rebuilt it, and I
had noticed there were no longer any OSX build slaves around.

It's online now, though it certainly need not be the only one.
There's not much to setting up a build slave - just easy_install
buildbot on top of a basic Python install (you'll need Twisted), use
the "build-slave" command to create a tree, edit the files in "info",
and "buildbot start " to get it started.  Of course,
there's some overhead to monitoring things over time.

My new slave is a Tiger box, since that's what I still need for my own
application builds. So it won't help with building/testing x64 builds.

Per the "startup on reboot" part of the thread, I typically use the
LaunchAgents setup on my Macs for that purpose, although to be honest,
not many of my build slaves start automatically anyway, as the boxes
don't tend to get restarted other than manually.

In the first builds, I have noticed that the build master seems to
execute the builds as a Unix system, so isn't building with the
universalsdk option or as a framework, though the latter would
probably need a bit of glue to make the framework available just
locally to the buildbot process.

On Windows, the buildbot tasks use scripts that keep the third party
stuff in the buildbot branch tree itself.  I wonder if perhaps with a
little TLC, we could come up with some Mac-specific buildbot scripts
to better have an OSX build slave mimic the final build process.  It
might be as simple as using build-installer with a specified directory
within the build tree, though utilitizing the built framework for the
tests probably needs some support.  The build master could then be
set to execute those scripts (much as it does on Windows) rather than
the stock Unix approach.

I anticipate some tuning to do with respect to external libraries.
Rather than use system libraries or my own (or finks or MacPorts)
version of libraries, I suspect it would be better to download the
same releases of the third party stuff that the installer script
downloads and install them locally for normal buildbot builds, to best
match what would be packaged up with a final binary.  Of course, if
so, then there's the question of keeping the buildbot environment up
to date with the installer script.

As I believe Martin mentioned elsewhere in the thread, he used to have
my Windows slave generate nightly MSI builds and upload them, but the
process became fragile over time, and didn't seem to be missed when we
discontinued them.  But something similar could probably be worked out
for building nightly DMGs for OSX if it were deemed useful.

> Creating the Mac installer is easy: just run
> Mac/BuildScript/build-installer.py on an OSX 10.5 system where a
> local version of Tcl/Tk 8.4 is installed in /Library/Frameworks. The
> system should also not have fink or darwinports and a clean
> /usr/local tree to avoid contaminating the build.

Yes, I've successfully used the script to create a DMG of the latest
2.7 out of trunk, and at least the basic sniff test (install it
locally, run a few things) seems fine.  Did run into one quirk where
if you have already built Python within the tree you run the script
from, it doesn't rebuild everything beneath /tmp/_py/_bld - such as
the pgen stuff.  So you want a clean source tree too.

To the extent that the output (DMG) of the build-installer script is
what is currently needed for OSX, I can generate one if it is still
needed, though I can't promise much beyond just executing the script.
The new build slave can also be made available for RMs (or whomever)
to generate the DMG when needed if that might be helpful.  Though
there should probably be some basic installation test on other systems
prior to publishing any such generated files.

-- David

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Automatic installer builds (was Re: Fwd: Broken link to download (Mac OS X))

[David Cournapeau]

On Thu, Apr 15, 2010 at 3:54 AM,   wrote:
>
>    Bill> In any case, they shouldn't be needed on buildbots maintained by
>    Bill> the PSF.
>
> Sure.  My question was related to humans building binary distributions
> though.  Unless that becomes fully automated so the release manager can just
> push a button and have it built on and as-yet-nonexistent Mac OSX buildbot
> machine, somebody will have to generate that installer.  Ronald says Fink,
> MacPorts and /usr/local are poison.  If that's truly the case that's fine.
> It's just that it reduces the size of the potential binary installer build
> machines.

Actually, you can just use a chroot "jail" to build the binary - I use
this process to build the official numpy/scipy binaries, it works very
well whatever crap there is on my laptop otherwise.

cheers,

David
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Automatic installer builds (was Re: Fwd: Broken link to download (Mac OS X))

[Martin v. Löwis]

> In the first builds, I have noticed that the build master seems to
> execute the builds as a Unix system, so isn't building with the
> universalsdk option or as a framework, though the latter would
> probably need a bit of glue to make the framework available just
> locally to the buildbot process.

Not sure what you mean by "make available" - I thought this is just a
matter of configure options?

Would you like to see such a build on your machine in addition, or
instead of, the Unix build?

> On Windows, the buildbot tasks use scripts that keep the third party
> stuff in the buildbot branch tree itself.  I wonder if perhaps with a
> little TLC, we could come up with some Mac-specific buildbot scripts
> to better have an OSX build slave mimic the final build process.

No. I'd rather create a separate builder on the master.

> I anticipate some tuning to do with respect to external libraries.
> Rather than use system libraries or my own (or finks or MacPorts)
> version of libraries, I suspect it would be better to download the
> same releases of the third party stuff that the installer script
> downloads and install them locally for normal buildbot builds, to best
> match what would be packaged up with a final binary.  Of course, if
> so, then there's the question of keeping the buildbot environment up
> to date with the installer script.

It would be possible to commit these packages into the externals
repository, just as we do for Windows. It would then be possible to
check them out over again all the time, or somehow to detect when the
URL changes so they export a different subversion directory.

> To the extent that the output (DMG) of the build-installer script is
> what is currently needed for OSX, I can generate one if it is still
> needed, though I can't promise much beyond just executing the script.
> The new build slave can also be made available for RMs (or whomever)
> to generate the DMG when needed if that might be helpful.  Though
> there should probably be some basic installation test on other systems
> prior to publishing any such generated files.

For 2.7, I would do that only if the very same build process is also
going to be used for the final release. There is no point in testing
builds when then the final release uses some other process. It would
also be good if anybody who commits to producing OSX binaries now would
also produce them throughout the 2.7 lifetime (which could be a bit
longer than the traditional 18 months).

Regards,
Martin
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Automatic installer builds (was Re: Fwd: Broken link to download (Mac OS X))

[David Bolen]

"Martin v. Löwis"  writes:

> Not sure what you mean by "make available" - I thought this is just a
> matter of configure options?

Building as a framework, yes.  But I think there's some steps to take
to then have the test python binary use the locally built framework
while running the tests (since the framework won't be in a system
location like /Library/Frameworks during the test). Probably similar to
whatever py2app does to use packaged frameworks contained within the
application bundle.

I doubt it's overly complex, but just an extra step that might need to
be incorporated into any buildbot test rules or script to ensure the
just built framework is used in the tests.

> Would you like to see such a build on your machine in addition, or
> instead of, the Unix build?

I suppose an argument could be made to test both, since others have
indicated here they often do a normal Unix-oriented build on OSX, but
I think if I only had to pick one, I'd go with framework to match what
gets published as a binary for downloading.  Theoretically it should
probably also be a universal build, though of course only one variant
would actually get tested on a build slave depending on its platform.

Certainly anything we could do to make the buildbot generated builds
match as closely to possible the distribution binary process would be
beneficial, I'd think.

>> On Windows, the buildbot tasks use scripts that keep the third party
>> stuff in the buildbot branch tree itself.  I wonder if perhaps with a
>> little TLC, we could come up with some Mac-specific buildbot scripts
>> to better have an OSX build slave mimic the final build process.
>
> No. I'd rather create a separate builder on the master.

I'm wondering if I'm saying the same thing but just not using the
right vernacular.  Something on the master (a builder?) for Windows
instructs it to run Tools\buildbot\build.bat for building and test.bat
for testing, rather than each of the individual commands.

Whether we encapsulate the needed logic in separate makefile targets
for the Mac, or independent build scripts like on Windows, the build
master would only need to execute some Mac appropriate command on a
given branch (not sure if that's separate builders), with the
makefile/script in the given checkout handling the third party stuff
and environmental setup.

Keeping the knowledge in the makefile or script in the source tree would
let the rules change across branches without affecting the build master.
Though if having more specific rules in the master was easier, I'd be
fine with that too.

> It would be possible to commit these packages into the externals
> repository, just as we do for Windows. It would then be possible to
> check them out over again all the time, or somehow to detect when the
> URL changes so they export a different subversion directory.

Sure - that's the sort of thing I figured a build script could take
care of, much as it does on Windows.  Or, the current Mac
build-installer script already has the necessary information, and can
be instructed where to place the third party stuff, so it might only
be necessary to have the buildbot process run that script with
appropriate build tree-relative paths.

> For 2.7, I would do that only if the very same build process is also
> going to be used for the final release. There is no point in testing
> builds when then the final release uses some other process.

Agreed, thus my caveat as to the output of the build-installer script
in fact being what has been published for downloads.  In my brief
tests it looks like it created a production DMG ready for publishing
on the download page, and did in fact install correctly, but I'm still
new to the Mac binary building process.

> It would
> also be good if anybody who commits to producing OSX binaries now would
> also produce them throughout the 2.7 lifetime (which could be a bit
> longer than the traditional 18 months).

To the extent that the installer script is in fact the definitive
process, and if the machine (10.4 Intel) is considered suitable as a
binary build platform, then worst case we could probably have the
buildbot run the script and upload the result when needed (sort of a
one-shot version of the old MSI daily builder).  I don't plan on the
machine going away any time soon.

Of course, this is just the DMG construction.  Not sure what amount of
"test the installer" testing should be required prior to publication.

-- David

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Automatic installer builds (was Re: Fwd: Broken link to download (Mac OS X))

[Martin v. Löwis]

> Keeping the knowledge in the makefile or script in the source tree would
> let the rules change across branches without affecting the build master.
> Though if having more specific rules in the master was easier, I'd be
> fine with that too.

Intuitively, I find the set of batch files used for Windows "ugly"
(despite me having created them in the first place). Objectively, I
think that has the risk of getting "out of hands", in the sense that
people start to think that they have to use these scripts in order to
build Python on Mac. This actually happened on Windows - some people now
recommend to run the buildbot scripts on a regular developer checkout,
because they supposedly do the right things.

I would rather prefer to have the buildbot run the same commands that we
recommend developers to run. The knowledge about these should be in the
README, and it should be stable knowledge, i.e. require infrequent
changes. This is true for Unix: configure/make/make test/make clean
had been the correct procedure for ten years now. The Unix buildbot only
deviates slightly, to have the slaves run a more expensive version of
the test suite.

> Agreed, thus my caveat as to the output of the build-installer script
> in fact being what has been published for downloads.  In my brief
> tests it looks like it created a production DMG ready for publishing
> on the download page, and did in fact install correctly, but I'm still
> new to the Mac binary building process.

I'd be interested in setting up daily builds then. For these, I think
it's fine that they may differ slightly from the official binaries -
people would recognize that they are testing a different set of binaries.

> Of course, this is just the DMG construction.  Not sure what amount of
> "test the installer" testing should be required prior to publication.

If it's fully automated, the amount of testing may actually be small,
IMO. I personally would put timeliness over correctness here. Also, it
might be that we can sign up some volunteer to do a smoke test in a
timely manner at release time who will test the installer on, say, a
clean VM.

Regards,
Martin
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Automatic installer builds (was Re: Fwd: Broken link to download (Mac OS X))

[David Bolen]

"Martin v. Löwis"  writes:

>  This actually happened on Windows - some people now
> recommend to run the buildbot scripts on a regular developer checkout,
> because they supposedly do the right things.

I have to admit that I'm guilty of this (though to be fair most of my
test builds are buildbot-related), if only because it takes care of
all the external stuff automatically and self-contained in the build
tree.

> I would rather prefer to have the buildbot run the same commands that we
> recommend developers to run. The knowledge about these should be in the
> README, and it should be stable knowledge, i.e. require infrequent
> changes. This is true for Unix: configure/make/make test/make clean
> had been the correct procedure for ten years now. The Unix buildbot only
> deviates slightly, to have the slaves run a more expensive version of
> the test suite.

In digging around a bit, it would appear that there's actually quite a
bit of OSX support already in the Makefile (either the main one or
the one in Mac).  There's even a target that tests both halves of a
universal build (using rosetta for the PPC version) on an Intel box.

I suspect it's just a question of setting up a Mac-appropriate
builder, using the universal SDK in the configure step, and whatever
Makefile targets are deemed best and current, perhaps with the
addition of support for pulling in the third party stuff through
externals or whatever.  A first pass could just be to factor that
process into a separate stage of build-installer that could be run
independently of the rest of the installer build process.

In the interim, I've generated the third-party libraries via the
current trunk build-installer script and installed them in /usr/local
on my buildbot, so they are found by any of the buildbot builds using
the stock configure/make approach.  Other than a slight update to the
bzip version, looks like the dependency versions in the installer
script haven't changed for over a year, so I suspect the libraries are
fine for any of the branches currently being built.

I also updated to the latest 8.4.19 Tcl/Tk in /Library/Frameworks
since I saw some interpreter crashes in tests in what appeared to be a
Tcl code path.  It had been building against my system 8.4.7 Tcl.  The
Windows buildbot uses Tcl 8.5 - not sure if that should be preferred
for the Mac buildbot as well, but will leave it at 8.4 for now.

I think this should create test builds more representative of the
final binaries.  It's not testing a universal framework build, but the
test target only tests the Intel path anyway, the generated code
should still be the same, and the same libraries are in use.

If anyone more familiar with this side of things for OSX has some
spare time down the road, I'd be happy to help work on improving the
process for the buildbot.

> I'd be interested in setting up daily builds then. For these, I think
> it's fine that they may differ slightly from the official binaries -
> people would recognize that they are testing a different set of binaries.

We can certainly start by just trying to automate the execution of
build-installer, something I suspect can be completely controlled from
the master side, and then uploading the resulting dmg file.  I'd be
happy to help coordinate any experiments offline if you're interested.

I do currently have a DMG built for 2.7 Beta 1, if it would be useful.

-- David

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Automatic installer builds (was Re: Fwd: Broken link to download (Mac OS X))

[Steve Holden]

David Bolen wrote:
> "Martin v. Löwis"  writes:
> 
>>  This actually happened on Windows - some people now
>> recommend to run the buildbot scripts on a regular developer checkout,
>> because they supposedly do the right things.
> 
> I have to admit that I'm guilty of this (though to be fair most of my
> test builds are buildbot-related), if only because it takes care of
> all the external stuff automatically and self-contained in the build
> tree.
> 
>> I would rather prefer to have the buildbot run the same commands that we
>> recommend developers to run. The knowledge about these should be in the
>> README, and it should be stable knowledge, i.e. require infrequent
>> changes. This is true for Unix: configure/make/make test/make clean
>> had been the correct procedure for ten years now. The Unix buildbot only
>> deviates slightly, to have the slaves run a more expensive version of
>> the test suite.
> 
> In digging around a bit, it would appear that there's actually quite a
> bit of OSX support already in the Makefile (either the main one or
> the one in Mac).  There's even a target that tests both halves of a
> universal build (using rosetta for the PPC version) on an Intel box.
> 
> I suspect it's just a question of setting up a Mac-appropriate
> builder, using the universal SDK in the configure step, and whatever
> Makefile targets are deemed best and current, perhaps with the
> addition of support for pulling in the third party stuff through
> externals or whatever.  A first pass could just be to factor that
> process into a separate stage of build-installer that could be run
> independently of the rest of the installer build process.
> 
> In the interim, I've generated the third-party libraries via the
> current trunk build-installer script and installed them in /usr/local
> on my buildbot, so they are found by any of the buildbot builds using
> the stock configure/make approach.  Other than a slight update to the
> bzip version, looks like the dependency versions in the installer
> script haven't changed for over a year, so I suspect the libraries are
> fine for any of the branches currently being built.
> 
> I also updated to the latest 8.4.19 Tcl/Tk in /Library/Frameworks
> since I saw some interpreter crashes in tests in what appeared to be a
> Tcl code path.  It had been building against my system 8.4.7 Tcl.  The
> Windows buildbot uses Tcl 8.5 - not sure if that should be preferred
> for the Mac buildbot as well, but will leave it at 8.4 for now.
> 
> I think this should create test builds more representative of the
> final binaries.  It's not testing a universal framework build, but the
> test target only tests the Intel path anyway, the generated code
> should still be the same, and the same libraries are in use.
> 
> If anyone more familiar with this side of things for OSX has some
> spare time down the road, I'd be happy to help work on improving the
> process for the buildbot.
> 
>> I'd be interested in setting up daily builds then. For these, I think
>> it's fine that they may differ slightly from the official binaries -
>> people would recognize that they are testing a different set of binaries.
> 
> We can certainly start by just trying to automate the execution of
> build-installer, something I suspect can be completely controlled from
> the master side, and then uploading the resulting dmg file.  I'd be
> happy to help coordinate any experiments offline if you're interested.
> 
> I do currently have a DMG built for 2.7 Beta 1, if it would be useful.
> 
Great work David, this is a terrific step forward. Thanks!

regards
 Steve
-- 
Steve Holden   +1 571 484 6266   +1 800 494 3119
See PyCon Talks from Atlanta 2010  http://pycon.blip.tv/
Holden Web LLC http://www.holdenweb.com/
UPCOMING EVENTS:http://holdenweb.eventbrite.com/

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Automatic installer builds (was Re: Fwd: Broken link to download (Mac OS X))

[Martin v. Löwis]

> We can certainly start by just trying to automate the execution of
> build-installer, something I suspect can be completely controlled from
> the master side, and then uploading the resulting dmg file.  I'd be
> happy to help coordinate any experiments offline if you're interested.

Sure!

> I do currently have a DMG built for 2.7 Beta 1, if it would be useful.

As I said before: if you would plan to do this on a regular basis, for
all upcoming releases, that would certainly be a good thing.

Having just a single DMG file for some beta release doesn't really help,
as testing results for that binary might not transfer to the next one
(if the next one comes from a different environment again).

Regards,
Martin
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Automatic installer builds (was Re: Fwd: Broken link to download (Mac OS X))

[Ronald Oussoren]

On 18 Apr, 2010, at 17:20, Martin v. Löwis wrote:

>> We can certainly start by just trying to automate the execution of
>> build-installer, something I suspect can be completely controlled from
>> the master side, and then uploading the resulting dmg file.  I'd be
>> happy to help coordinate any experiments offline if you're interested.
> 
> Sure!
> 
>> I do currently have a DMG built for 2.7 Beta 1, if it would be useful.
> 
> As I said before: if you would plan to do this on a regular basis, for
> all upcoming releases, that would certainly be a good thing.
> 
> Having just a single DMG file for some beta release doesn't really help,
> as testing results for that binary might not transfer to the next one
> (if the next one comes from a different environment again).

I will provide installers for 2.7 starting from beta2,

Ronald

> 
> Regards,
> Martin
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: 
> http://mail.python.org/mailman/options/python-dev/ronaldoussoren%40mac.com



smime.p7s
Description: S/MIME cryptographic signature
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Automatic installer builds (was Re: Fwd: Broken link to download (Mac OS X))

[David Bolen]

"Martin v. Löwis"  writes:

>> I do currently have a DMG built for 2.7 Beta 1, if it would be useful.
>
> As I said before: if you would plan to do this on a regular basis, for
> all upcoming releases, that would certainly be a good thing.

No argument - just figured I'd offer to get past the near term
resource issue, presuming that having a beta installer available is
better than not, even with some build-environment difference risk.

I could probably commit to a longer term too, but it sounds like
Ronald is still on board for that.  I've also been a little hesitant
since there's clearly a lot of effort that has been put into the Mac
build stuff and I didn't want to be the inexperienced bull in the
china shop compared to those who have been doing it for a while.

I'm happy to stick with the buildbot side of things for now, and
remain willing to assist in any changes to better match the buildbot
builds to the final distribution process for more aligned testing.

-- David

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com