Re: [Pythonmac-SIG] Building plans .....

[Christopher Barker]

Thanks Jack.

But I'm still confused about a technical question:

Does having python itself in a Framework enable anything that having it
outside a Framework won't let you do?

This is completely aside from the python in an app bundle question, as you
can certainly have a python binary with no app bundle, and you can have
python in an app bundle with no Framework (indeed, the current conda Python
python.app hack works with an otherwise standard Unix build). Again, I'm
curious what a Framework buys you (or anyone). ANd I'm not talking bout the
"framework build" that is currently provided by the python.org build
scripts, but a Framework in the general sense.

Second: the whole intertwining of conda, Anaconda, brew, etc, is very
confusing, so I'll try to provide a bit of clarity here:

I think we all know what brew (and macports, if anything is still using
that) is.

but "conda" is a package management tool, a bit like yum or apt in the
Linux world, or Brew, for that matter. except that:

* It is platform independent -- it attempts to work the same way on all all
platforms to the degree possible.
* It provides management of isolated "environments", so you can have a
different collection of software and libraries, all with a different
combination of version on the same system, isolated from one another.

* critically, it can manage, in theory, ANY software or library --
including Python, of course.
* conda is written in Python, but otherwise but is not specifically about
Python.
* In practice, it is currently used mostly for Python and associated
libraries, but also R, node.js, geospatial libraries and tools, etc.

"Anaconda" is a particular "distribution" of Python and data science
software that is curated with a bunch of packages that are all tested and
work together. It also comes with nifty GUI tools to manage the packages,
etc.

Importantly, conda the package manager, can be, and is, used to manage
collections of software without Anaconda (see "miniconda)

In theory, anyone could build and distribute conda packages in any way (or
build then themselves and entirely self manage them). In practice, THe
anaconda.org site is used as a repository of packages. Anyone can set up
their own "channel" and distribute whatever they want.

In addition to the packages being distributed by the Anaconda company, and
a whole variety of private channels on anaconda.org, there is the
"conda-forge" project, which is a community-driven curated collection of a
packages available on anaconda.org, and managed by a big ol' gitHub org.

Now that that's done:

On Sat, Mar 14, 2020 at 4:41 PM Jack Jansen  wrote:

> I think you have to decide who the target audience is, and based on the
> needs of the target audience decide. Possibly deciding to go both ways.
>

I agree -- there are different needs for different audiences, and there may
not be one single way to build Python that meats all needs.


> One target audience is Unix developers who happen to use a Mac because
> it’s a convenient Unix development platform (and, practically speaking, the
> only viable BSD development platform left nowadays). These will use brew,
> build things from source, and they’ll be perfectly happy with a
> non-framework build. And if they ever need to use a library that requires
> to be in a framework you teach them to use some magic command to do that
> (tell them to please try `MagicMacPythonWrapper pip install foo` if `pip
> install foo` doesn’t work, that kind of thing).
>

I think that audience is currently satisfied by the regular old *nix builds
(though I think Brew actually does Framework builds, oddly enough -- but I
haven't used brew in a while, since I discovered conda). Though many of
these users would be well served by conda as well, with a stndard *nix
build.

Another target audience is Mac users, with some limited programming needs.
> Think artists, designers, musicians, etc. These will use Python to connect
> strange and wondrous hardware to strange and wondrous software packages,
> and they will definitely need a framework build because otherwise it won’t
> interface to their funny hardware, and they won’t know how to get through
> all the dialogs that say you can’t install this because it isn’t trusted or
> it won’t load this DLL because it is untrusted or yadda yadda yadda.
>

I'm still unclear on that -- do they really need a Framework build? or do
they only need the executable to be in a app bundle? And the current
Framework builds, I think, satisfy these users pretty well.

And a third audience is scientists who are not developers, or at least not
> primarily developers, and who don’t have a strong attachment to either Mac
> or Unix.
>

That's pretty much the audience I'm concerned with now -- and tha one that
is not quite well served at the moment. (though some may well be attached
to the Mac, but aren't writing code specifically for the Mac )


> Here I don’t mean people who don’t develop at all (because then they
> wouldn’t need 

Re: [Pythonmac-SIG] Building plans .....

[Jack Jansen]

I was tempted to answer this point-by-point, but I think that would lead to a 
discussion that’ll quickly branch into many different ways and won’t lead to 
anything.

I think you have to decide who the target audience is, and based on the needs 
of the target audience decide. Possibly deciding to go both ways.

One target audience is Unix developers who happen to use a Mac because it’s a 
convenient Unix development platform (and, practically speaking, the only 
viable BSD development platform left nowadays). These will use brew, build 
things from source, and they’ll be perfectly happy with a non-framework build. 
And if they ever need to use a library that requires to be in a framework you 
teach them to use some magic command to do that (tell them to please try 
`MagicMacPythonWrapper pip install foo` if `pip install foo` doesn’t work, that 
kind of thing).

Another target audience is Mac users, with some limited programming needs. 
Think artists, designers, musicians, etc. These will use Python to connect 
strange and wondrous hardware to strange and wondrous software packages, and 
they will definitely need a framework build because otherwise it won’t 
interface to their funny hardware, and they won’t know how to get through all 
the dialogs that say you can’t install this because it isn’t trusted or it 
won’t load this DLL because it is untrusted or yadda yadda yadda.
 
And a third audience is scientists who are not developers, or at least not 
primarily developers, and who don’t have a strong attachment to either Mac or 
Unix. Here I don’t mean people who don’t develop at all (because then they 
wouldn’t need Anaconda) but who are basically data scientists or deep learning 
people or whatever, and who have been told that there’s a nifty package in 
Python that’ll do just what they need. And for them I think you need to look at 
how Anaconda treats the difference between Windows and Linux: if it tries to 
hide that difference as much as possible (like jupyter does, for example, but 
also completely different things like Adobe CS which really tries to make their 
software look that same independent of platform) you should do the same.

And if you can’t decide: how much work would it be to have two Anaconda 
distributions for the Mac: 
- one that is primarily Unix-y, installed through brew, intended to be used 
from the command line, and with the MagicMacPythonWrapper
- another one that is framework-based, installed as an app bundle, signed, 
notarized, all that jazz, and comes with an “Anaconda Terminal” which is a 
perfectly normal Terminal but with all the environment and things setup so that 
people can use the command line in the Unix way but it would under the hood 
have all the PATH and DYLD_LIBRARY_PATH and whatnot to use the Python from the 
accompanying framework build always.

Jack


> On  14-Mar-2020, at 23:49 , Christopher Barker  wrote:
> 
> Jack et al,
> 
> Yes, lots of deja vu here as well -- I suppose because this hasn't been 
> resolved.
> 
> > Hmm, that was a long sentence for what was intended to be the question: 
> > “What is the problem with using a normal MacOS Framework build of Python 
> > for Anaconda”?
> 
> I can't really answer this -- I was not part of the original discussion, nor 
> am I in an decision making position now. I'm just a user an contributor to 
> conda-forge that's trying to make things easier.
> 
> But I think it more or less comes down to what you said:
> 
> Conda wants to be as similar as possible on all platforms as it can. Yes, 
> there are differences on Windows, but they are keeping those to a minimum, 
> and there is no choice.  OS-X, on the other is, as you mentioned, BSD *nix. 
> So doing something very differently there feels gratuitous.
> 
> And in a practical sense, many folks use OS-X as a development platform, so 
> want to use Python on OS-X in the same way they use it on other *nix systems. 
> I myself have found this to be very helpful for my own work, as well as for 
> teaching. And if you aren't doing any GUI work, then it is nice for the Mad 
> to be "just another *nix". For my part, I develop for the MAc, but only in 
> the context of cross platform tools: I never do anything you can do only on 
> the Mac, and try to do everything in a cross platform way as possible (using 
> wPyrthon for the GUI, for instance). Which is why I want Python, and 
> everything else to work the same .
> 
> That being said, I'm not sure that this was thought through all that 
> carefully initially, and I haven't seen anyone spell out the reasons for not 
> doing a Framework build other than
> 
> "Honestly, I think adding the framework will make this more confusing when 
> building things between OS X and Linux" (from 
> https://github.com/conda-forge/python-feedstock/issues/23 
> ).
> 
> But I'll turn around the question: "What is the problem with using a 
> non-framework build for conda on the Mac" (

Re: [Pythonmac-SIG] Building plans .....

[Christopher Barker]

Jack et al,

Yes, lots of deja vu here as well -- I suppose because this hasn't been
resolved.

> Hmm, that was a long sentence for what was intended to be the question:
“What is the problem with using a normal MacOS Framework build of Python
for Anaconda”?

I can't really answer this -- I was not part of the original discussion,
nor am I in an decision making position now. I'm just a user an contributor
to conda-forge that's trying to make things easier.

But I think it more or less comes down to what you said:

Conda wants to be as similar as possible on all platforms as it can. Yes,
there are differences on Windows, but they are keeping those to a minimum,
and there is no choice.  OS-X, on the other is, as you mentioned, BSD *nix.
So doing something very differently there feels gratuitous.

And in a practical sense, many folks use OS-X as a development platform, so
want to use Python on OS-X in the same way they use it on other *nix
systems. I myself have found this to be very helpful for my own work, as
well as for teaching. And if you aren't doing any GUI work, then it is nice
for the Mad to be "just another *nix". For my part, I develop for the MAc,
but only in the context of cross platform tools: I never do anything you
can do only on the Mac, and try to do everything in a cross platform way as
possible (using wPyrthon for the GUI, for instance). Which is why I want
Python, and everything else to work the same .

That being said, I'm not sure that this was thought through all that
carefully initially, and I haven't seen anyone spell out the reasons for
not doing a Framework build other than

"Honestly, I think adding the framework will make this more confusing when
building things between OS X and Linux" (from
https://github.com/conda-forge/python-feedstock/issues/23).

But I'll turn around the question: "What is the problem with using a
non-framework build for conda on the Mac" (or even outside of conda,
e.g.brew or macports)

It's unclear to me is what the advantages of a Framework Build are in this
context. Other than it being the "Mac way of doing things". It's my
impression that Apple simply has not thought out how it should support
things like Python -- the whole system is very much designed around
"Applications", but the Python interpreter is not an application in the
usual sense. But it is also not a library in the usual sense, either. So
I'm not sure there IS an obvious right way to support Python on a a Mac.

And I understand the idea behind frameworks, and like them. I've never much
liked the scattering of things all over the filesystem approach that *nix
usually does. But in this case, conda is doing a similar thing, but in a
more powerful and flexible way. So cramming a Framework into conda seems
gratuitous.

As far as I know, the only issue people have had with using a standard Unix
build in conda is the whole GUI thing.Jack mentioned a lot of others,
essentially interacting the the non-kernel part of the OS, but I haven't
heard complaints about that -- maybe because no one's doing that with conda.

But the key technical question I'm still not totally clear on is:

I understand (vaguely) what having Python running from a App bundle
provides -- and these are important features, so we do want to support that.

And I I thought that the executable re-directs to one in a bundle could be
done independently of the Framework, and the fact that the two is
essentially and accident of how they build scripts were set up (and
probably easier to do). If I'm correct, then Python could be built to
provide an app bundle, but otherwise be just like a "normal" unix build.

But what does having Python built as a Framework actually provide, other
than making more like the usual Mac setup? As far as I can tell, it would
provide no new functionality. In which case, given a choice, for, e.g. a
conda build, I'd rather have it be more like the other *nixes than more
like other Mac software.

In short: as far as I can tell, the only reason to use a Framework build in
conda is that it's the only configuration that's already set up to make
pythonw in the build scripts, and autoconf is enough of a pain that one
wants to make a new configuration.

By the way, I chatted with Ned Diely a few years ago about this, and I
think he was thinking of moving away form Frameworks for python.org builds,
though I don't know if he decided not to make that change, or simply hasn't
gotten around to it.

-CHB

References to Discussions:

https://github.com/conda-forge/python-feedstock/issues/23





> Jack
>
> On  14-Mar-2020, at 14:54 , Ronald Oussoren 
> wrote:
>
>
>
> On 13 Mar 2020, at 20:36, Christopher Barker  wrote:
>
> There is a private API that makes it possible to use GUI libraries outside
>> of an app bundle and that’s used by a number of projects, but I wouldn’t
>> want to use that in Python.
>>
>
> Is that what TK is doing?
>
>
> I don’t know.
>
>
> And while probably not good to build it into Python itself, what about

Re: [Pythonmac-SIG] Building plans .....

[Jack Jansen]

I’m having an incredible amount of deja-vu in this conversation. We seem to be 
heading down the path of talking about little-documented APIs and such the same 
way we were doing 15 or 18 years ago….

Christopher,
can you explain what the problem is that you’re running into? Why is there a 
problem with the Mac Python builds having a different structure than other Unix 
Python builds? Windows Python builds also have a different structure, does the 
problem occur there as well?

From your original post it seems that the problems are with the 
Anaconda/Conda-forge distribution treating MacOS as “just another unix 
distribution”, with the idea that “Mac GUI compatibility” wouldn’t be too 
important for the Anaconda users, or could be handled with the pythonw 
workaround shellscript.

But actually “MacOS-compatible” would be a more accurate term here, because I’m 
pretty sure there are a whole lot of APIs that have nothing to do with the GUI 
but won’t work unless you’re running from a bundle: definitely Open Scripting, 
but probably also audio/video capture, probably anything using keychain like 
crypto serveices, maybe bluetooth, maybe USB, probably anything using the GPU 
for computations….. 

So basically if you build Python for MacOS as if it is “just another BSD 
distribution” then that is indeed what you get: a Python for just another BSD 
distribution.

Hmm, that was a long sentence for what was intended to be the question: “What 
is the problem with using a normal MacOS Framework build of Python for 
Anaconda”?

Jack

> On  14-Mar-2020, at 14:54 , Ronald Oussoren  wrote:
> 
> 
> 
>> On 13 Mar 2020, at 20:36, Christopher Barker > > wrote:
>> 
>> There is a private API that makes it possible to use GUI libraries outside 
>> of an app bundle and that’s used by a number of projects, but I wouldn’t 
>> want to use that in Python.
>> 
>> Is that what TK is doing? 
> 
> I don’t know.
>> 
>> And while probably not good to build it into Python itself, what about 
>> building into the GUI toolkits? 
>> 
>> Do you have a reference to that API?
> 
> I don’t have a reference handy, it has come up before either on this list or 
> the python tracker (years back). IIRC someone found the API by checking how 
> the bundled java VM managed to use GUI APIs without being in an app bundle.
> 
> Apple is clearly preferring application code packaged as app bundles, and 
> effectively doesn’t care about traditional Unix-y use of macOS. The need for 
> an app bundle when using frameworks is just one example of that. Another 
> example are the issues that keep popping up when trying to use Apple 
> frameworks in child processes after calling os.fork(). 
> 
> Because of this I don’t think it is worthwhile to try to work around this 
> limitation by calling private APIs. That’s even before considering use cases 
> that involve distributing code through the macOS app store.
> 
> Ronald
> —
> 
> Twitter: @ronaldoussoren
> Blog: https://blog.ronaldoussoren.net/ 

--
Jack Jansen, , http://www.cwi.nl/~jack
If I can't dance I don't want to be part of your revolution -- Emma Goldman



___
Pythonmac-SIG maillist  -  Pythonmac-SIG@python.org
https://mail.python.org/mailman/listinfo/pythonmac-sig
unsubscribe: https://mail.python.org/mailman/options/Pythonmac-SIG

Re: [Pythonmac-SIG] Building plans .....

[Ronald Oussoren via Pythonmac-SIG]

> On 13 Mar 2020, at 20:36, Christopher Barker  wrote:
> 
> There is a private API that makes it possible to use GUI libraries outside of 
> an app bundle and that’s used by a number of projects, but I wouldn’t want to 
> use that in Python.
> 
> Is that what TK is doing? 

I don’t know.
> 
> And while probably not good to build it into Python itself, what about 
> building into the GUI toolkits? 
> 
> Do you have a reference to that API?

I don’t have a reference handy, it has come up before either on this list or 
the python tracker (years back). IIRC someone found the API by checking how the 
bundled java VM managed to use GUI APIs without being in an app bundle.

Apple is clearly preferring application code packaged as app bundles, and 
effectively doesn’t care about traditional Unix-y use of macOS. The need for an 
app bundle when using frameworks is just one example of that. Another example 
are the issues that keep popping up when trying to use Apple frameworks in 
child processes after calling os.fork(). 

Because of this I don’t think it is worthwhile to try to work around this 
limitation by calling private APIs. That’s even before considering use cases 
that involve distributing code through the macOS app store.

Ronald
—

Twitter: @ronaldoussoren
Blog: https://blog.ronaldoussoren.net/

___
Pythonmac-SIG maillist  -  Pythonmac-SIG@python.org
https://mail.python.org/mailman/listinfo/pythonmac-sig
unsubscribe: https://mail.python.org/mailman/options/Pythonmac-SIG