Re: [Pythonmac-SIG] py2app and nested packages

[Russell E. Owen]

In article <7da2de5d-dcbc-404e-83cd-0a243374e...@mac.com>,
 Ronald Oussoren  wrote:

> On 25 Apr, 2012, at 2:17, Chris Barker wrote:
> 
> > On Tue, Apr 24, 2012 at 4:29 AM, Ronald Oussoren
> >> The option "packages" currently always stores the included package outside 
> >> of site-packages.zip to ensure that data files can be loaded in the old 
> >> pre-pkgresources way (that is, by opening paths relative to 
> >> somepackage.__file__).
> > 
> > which is useful, but another thought -- maybe there could be two options:
> > 
> > "packages" and "full_packages" or something -- so one would simple add
> > the package to the modulegraph, to catch dynamic imports, and the
> > other would do what is done now -- include the whole darn thing.
> > 
> > One other issue with the "whole darn thing" option, is that you get
> > the *.py, *.pyc, Na *.pyo files -- so it's maybe three times as big as
> > it could be -- I wonder if it's worth cleaning that up.
> 
> Adding "full_packages"  for the current behavior and giving "packages" a 
> semantics that's simular to "modules" would be useful.

Please consider retaining the current meaning for the current name and 
pick a new name for the new meaning (if you can figure one out that 
makes sense). That would preserve backwards compatibility.

Is the inclusion of .pyc and .pyo files a significant problem? If they 
are not already present then they will be regenerated next time the 
application is run. Though if some .py files are never loaded then the 
associated .pyc and .pyo files are wasted space.

-- Russell

___
Pythonmac-SIG maillist  -  Pythonmac-SIG@python.org
http://mail.python.org/mailman/listinfo/pythonmac-sig
unsubscribe: http://mail.python.org/mailman/options/Pythonmac-SIG

Re: [Pythonmac-SIG] py2app and nested packages

[Ronald Oussoren]

On 25 Apr, 2012, at 2:17, Chris Barker wrote:

> On Tue, Apr 24, 2012 at 4:29 AM, Ronald Oussoren
> 
>>> example: the "pubsub" package is delivered with wxPython, so it is
>>> commonly imported thusly:
>>> 
>>> from wx.lib.pubsub import Publisher
>> 
>> wx.lib.pubsub is a bad example, it manipulates __path__ which causes 
>> problems with py2app because it cannot detect this.  I added a recipe to 
>> py2app's repository a while ago that should fix this particular problem.
> 
> 
> I thought I had the latest -- I'll go check that out.

Knowing me I probably forgot to check in the updated recipe :-(.  I also lost 
track on whether or not all changes are released on PyPI.

> 
> Also, there is a bit of trick if you force it to import it's dyanamic
> module, but if you've solved that one, great!

IIRC that dynamic import is completely unnecessary and probably won't work when 
wx is in a zip file.

> 
> 
>> That's a bug, the code that implements the packages options assumes that all 
>> mentioned packages are toplevel packages.  It should be fairly easy to fix 
>> that, although this requires some path manipulation trickery because the 
>> subpackage will be in a different location that the parent: the parent is in 
>> site-packages.zip while the child is not.
> 
> right -- do you have an idea how to solve that? I don't!

The copying part is easy, just create the right directory structure (packages = 
["wx.pubsub"] -> create "wx/pubsub" in the application bundle). Ensuring that 
__import__ finds the subpackage is harder, I'll have to experiment to find a 
working solution that won't cause nausea in anyone looking at the code. 

> 
>> The option "packages" currently always stores the included package outside 
>> of site-packages.zip to ensure that data files can be loaded in the old 
>> pre-pkgresources way (that is, by opening paths relative to 
>> somepackage.__file__).
> 
> which is useful, but another thought -- maybe there could be two options:
> 
> "packages" and "full_packages" or something -- so one would simple add
> the package to the modulegraph, to catch dynamic imports, and the
> other would do what is done now -- include the whole darn thing.
> 
> One other issue with the "whole darn thing" option, is that you get
> the *.py, *.pyc, Na *.pyo files -- so it's maybe three times as big as
> it could be -- I wonder if it's worth cleaning that up.

Adding "full_packages"  for the current behavior and giving "packages" a 
semantics that's simular to "modules" would be useful.

> 
>>> Is there a way to "turn off" a recipe (other than deleting it from the 
>>> install)?
> 
>> No, and I'd prefer to fix recipes instead of having a way to disable them.
> 
> I understand that, and generally agree -- but it may be that there is
> no one recipe that works for everyone. For example, Scipy is weird
> enough that probably the only way to be sure it'll work in all cases
> is to include the whole darn thing -- but in my case at least, I'm
> using a couple functions in scipy.special, and it works fine if that
> is all I include.
> 
> Maybe if there is a way to pass options to recipes?

I guess you're right, there has to be a way to configure recipes and that 
includes disabling them completely.

Matplotlib is another example where this would be useful: it should be possible 
to specify that your app uses a particular backend without including the entire 
tree.

> 
>> The (scipy) recipe should get a way to specify which dynamic imports are 
>> present, simular to the hardcoded knowledge about the stdlib in modulegraph. 
>> That way py2app can include only the bits that are really necessary.
> 
> If you give me a pointer - maybe to the similar modulegraph code,
> perhaps I could work on that.

The modulegraph code I refer to is modulegraph.find_moduels.get_implies 
().
  The wx recipe in the py2app tree 
() 
basicly does the same thing from the outside.

> 
>>> IIRC the matplotlib recipe includes the entire package because of the data 
>>> files that are in the package, including only the code doesn't work.
> 
> I've poked into this more -- it turn out that there is special-case
> code in matplotlib itself for py2exe that looks for the data-files in
> a different place when sys.frozen -- I'm adding some code for py2app
> as well -- if I get that working, I can submit a patch to MPL and the
> recipe.
> 
> Is there a way to specify data files in a recipe?

Good question, and it seems the anwer is no (looking at 
).
  Adding an option for that would be easy enough though.

> 
>>> - Is there a way to apply excludes after the recipe? As far as I can
>>> tell, if the recipe includes it, excludes doesn't remove it.
> 
>> That's correct, and is a bug: user specified actions should ove

Re: [Pythonmac-SIG] py2app and nested packages

[Chris Barker]

On Tue, Apr 24, 2012 at 4:29 AM, Ronald Oussoren

>> example: the "pubsub" package is delivered with wxPython, so it is
>> commonly imported thusly:
>>
>> from wx.lib.pubsub import Publisher
>
> wx.lib.pubsub is a bad example, it manipulates __path__ which causes problems 
> with py2app because it cannot detect this.  I added a recipe to py2app's 
> repository a while ago that should fix this particular problem.


I thought I had the latest -- I'll go check that out.

Also, there is a bit of trick if you force it to import it's dyanamic
module, but if you've solved that one, great!


> That's a bug, the code that implements the packages options assumes that all 
> mentioned packages are toplevel packages.  It should be fairly easy to fix 
> that, although this requires some path manipulation trickery because the 
> subpackage will be in a different location that the parent: the parent is in 
> site-packages.zip while the child is not.

right -- do you have an idea how to solve that? I don't!

> The option "packages" currently always stores the included package outside of 
> site-packages.zip to ensure that data files can be loaded in the old 
> pre-pkgresources way (that is, by opening paths relative to 
> somepackage.__file__).

which is useful, but another thought -- maybe there could be two options:

"packages" and "full_packages" or something -- so one would simple add
the package to the modulegraph, to catch dynamic imports, and the
other would do what is done now -- include the whole darn thing.

One other issue with the "whole darn thing" option, is that you get
the *.py, *.pyc, Na *.pyo files -- so it's maybe three times as big as
it could be -- I wonder if it's worth cleaning that up.

>> Is there a way to "turn off" a recipe (other than deleting it from the 
>> install)?

> No, and I'd prefer to fix recipes instead of having a way to disable them.

I understand that, and generally agree -- but it may be that there is
no one recipe that works for everyone. For example, Scipy is weird
enough that probably the only way to be sure it'll work in all cases
is to include the whole darn thing -- but in my case at least, I'm
using a couple functions in scipy.special, and it works fine if that
is all I include.

Maybe if there is a way to pass options to recipes?

> The (scipy) recipe should get a way to specify which dynamic imports are 
> present, simular to the hardcoded knowledge about the stdlib in modulegraph. 
> That way py2app can include only the bits that are really necessary.

If you give me a pointer - maybe to the similar modulegraph code,
perhaps I could work on that.

>> IIRC the matplotlib recipe includes the entire package because of the data 
>> files that are in the package, including only the code doesn't work.

I've poked into this more -- it turn out that there is special-case
code in matplotlib itself for py2exe that looks for the data-files in
a different place when sys.frozen -- I'm adding some code for py2app
as well -- if I get that working, I can submit a patch to MPL and the
recipe.

Is there a way to specify data files in a recipe?

>>- Is there a way to apply excludes after the recipe? As far as I can
>> tell, if the recipe includes it, excludes doesn't remove it.

> That's correct, and is a bug: user specified actions should override > 
> automatic actions, including the recipes.

OK -- I suspect the issue is there is no way to exclude stuff that is
going to go outside of the zip file -- the full packages. It looks
like what it does is remove it from modulegraph only.

Thanks for your work (and ideas) on this.

-Chris




-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Pythonmac-SIG maillist  -  Pythonmac-SIG@python.org
http://mail.python.org/mailman/listinfo/pythonmac-sig
unsubscribe: http://mail.python.org/mailman/options/Pythonmac-SIG

Re: [Pythonmac-SIG] py2app and nested packages

[Ronald Oussoren]

On 23 Apr, 2012, at 23:13, Chris Barker wrote:

> Another couple py2app questions:
> 
> I"ve discovered that my app is HUGE! -- I already expected that, but
> this is a bit out of bounds. The reason:
> 
> The scipy and matplotlib packages bring in all of scipy, numpy, and
> matplotlib uncompressed. This gets pretty darn big, particularly when
> you're using one or two simple functions from scipy, for instance.
> 
> I know why they do that -- there are a number of dynamic inports, and
> folks often include matplotlib and/or scipy in apps where people may
> be scripting, etc, and using features that aren't always imported by
> the app. And this way, apps using those libs "just work" which is
> great.
> 
> However, it would be nice to be able to override soem of this:
> 
> Is there a way to "turn off" a recipe (other than deleting it from the 
> install)?

No, and I'd prefer to fix recipes instead of having a way to disable them.

The numpy/scipy recipes are probably as aggressive as they are because that 
fixes the immediate problem with minimal effort while writing the recipe.  The 
recipe should get a way to specify which dynamic imports are present, simular 
to the hardcoded knowlegde about the stdlib in modulegraph. That way py2app can 
include only the bits that are really necessary.

IIRC the matplotlib recipe includes the entire package because of the data 
files that are in the package, including only the code doesn't work.

> 
> - Is there a way to apply excludes after the recipe? As far as I can
> tell, if the recipe includes it, excludes doesn't remove it.

That's correct, and is a bug: user specified actions should override automatic 
actions, including the recipes.

Ronald
> 
> 
> -Chris
> 
> 
> 
> On Mon, Apr 23, 2012 at 11:42 AM, Chris Barker  wrote:
>> Folks,
>> 
>> py2app doesn't seem to do "the right thing" with nested packages.
>> However, it's not entirely clear what it should do...
>> 
>> example: the "pubsub" package is delivered with wxPython, so it is
>> commonly imported thusly:
>> 
>> from wx.lib.pubsub import Publisher
>> 
>> However, in wx.lib.pubsub, there is some trickery in the __init___, so
>> as py2app installs it, ot doesn't work (see previous messages to this
>> and the wxPython list). However, if you do:
>> 
>> packages = ['wx.lib.pubsub']  ...
>> 
>> in your py2app setup.py, you get the entire pubsub package, but installed in 
>> :
>> 
>> Peppy.app/Contents/Resources/lib/python2.7/pubsub
>> 
>> so it will work, but only if you now import it as:
>> 
>> from pubsub import Publisher
>> 
>> not too heinous, but it's really nice if we don't have to change
>> imports for the bundled version.
>> 
>> if I add 'wx' to packages, It does work, but I've then got all of wx,
>> all of wx.lib, etc, which is  really a bit much (the resulting app
>> bundle is 124MB, vs. 60 without specifying the wx package)
>> 
>> At first, I thought I'd want:
>> 
>> wx/lib/pubsub/
>> 
>> put on sys.path, with wx and lib empty except the specified dirs.
>> However, that would put two "wx" dirs on sys.path, which would not be
>> good (assuming wx is used...)
>> 
>> Would it be possible for the packages specification to be processed
>> after modulegraph is run, and it would then see that "wx" and "wx.lib"
>> are there, and then put all of pubsub in there? Or maybe put the parts
>> of wx and wx.lib that were already going to be included outside the
>> zip bundle?
>> 
>> -Chris
>> 
>> 
>> --
>> 
>> Christopher Barker, Ph.D.
>> Oceanographer
>> 
>> Emergency Response Division
>> NOAA/NOS/OR&R(206) 526-6959   voice
>> 7600 Sand Point Way NE   (206) 526-6329   fax
>> Seattle, WA  98115   (206) 526-6317   main reception
>> 
>> chris.bar...@noaa.gov
> 
> 
> 
> -- 
> 
> Christopher Barker, Ph.D.
> Oceanographer
> 
> Emergency Response Division
> NOAA/NOS/OR&R(206) 526-6959   voice
> 7600 Sand Point Way NE   (206) 526-6329   fax
> Seattle, WA  98115   (206) 526-6317   main reception
> 
> chris.bar...@noaa.gov
> ___
> Pythonmac-SIG maillist  -  Pythonmac-SIG@python.org
> http://mail.python.org/mailman/listinfo/pythonmac-sig
> unsubscribe: http://mail.python.org/mailman/options/Pythonmac-SIG



smime.p7s
Description: S/MIME cryptographic signature
___
Pythonmac-SIG maillist  -  Pythonmac-SIG@python.org
http://mail.python.org/mailman/listinfo/pythonmac-sig
unsubscribe: http://mail.python.org/mailman/options/Pythonmac-SIG

Re: [Pythonmac-SIG] py2app and nested packages

[Ronald Oussoren]

On 23 Apr, 2012, at 20:42, Chris Barker wrote:

> Folks,
> 
> py2app doesn't seem to do "the right thing" with nested packages.
> However, it's not entirely clear what it should do...
> 
> example: the "pubsub" package is delivered with wxPython, so it is
> commonly imported thusly:
> 
> from wx.lib.pubsub import Publisher

wx.lib.pubsub is a bad example, it manipulates __path__ which causes problems 
with py2app because it cannot detect this.  I added a recipe to py2app's 
repository a while ago that should fix this particular problem.

> 
> However, in wx.lib.pubsub, there is some trickery in the __init___, so
> as py2app installs it, ot doesn't work (see previous messages to this
> and the wxPython list). However, if you do:
> 
> packages = ['wx.lib.pubsub']  ...
> 
> in your py2app setup.py, you get the entire pubsub package, but installed in :
> 
> Peppy.app/Contents/Resources/lib/python2.7/pubsub

That's a bug, the code that implements the packages options assumes that all 
mentioned packages are toplevel packages.  It should be fairly easy to fix 
that, although this requires some path manipulation trickery because the 
subpackage will be in a different location that the parent: the parent is in 
site-packages.zip while the child is not.
> 
> so it will work, but only if you now import it as:
> 
> from pubsub import Publisher
> 
> not too heinous, but it's really nice if we don't have to change
> imports for the bundled version.
> 
> if I add 'wx' to packages, It does work, but I've then got all of wx,
> all of wx.lib, etc, which is  really a bit much (the resulting app
> bundle is 124MB, vs. 60 without specifying the wx package)
> 
> At first, I thought I'd want:
> 
> wx/lib/pubsub/
> 
> put on sys.path, with wx and lib empty except the specified dirs.
> However, that would put two "wx" dirs on sys.path, which would not be
> good (assuming wx is used...)
> 
> Would it be possible for the packages specification to be processed
> after modulegraph is run, and it would then see that "wx" and "wx.lib"
> are there, and then put all of pubsub in there? Or maybe put the parts
> of wx and wx.lib that were already going to be included outside the
> zip bundle?

The option "packages" currently always stores the included package outside of 
site-packages.zip to ensure that data files can be loaded in the old 
pre-pkgresources way (that is, by opening paths relative to 
somepackage.__file__). 

Ronald



smime.p7s
Description: S/MIME cryptographic signature
___
Pythonmac-SIG maillist  -  Pythonmac-SIG@python.org
http://mail.python.org/mailman/listinfo/pythonmac-sig
unsubscribe: http://mail.python.org/mailman/options/Pythonmac-SIG

Re: [Pythonmac-SIG] py2app and nested packages

[Chris Barker]

Another couple py2app questions:

I"ve discovered that my app is HUGE! -- I already expected that, but
this is a bit out of bounds. The reason:

The scipy and matplotlib packages bring in all of scipy, numpy, and
matplotlib uncompressed. This gets pretty darn big, particularly when
you're using one or two simple functions from scipy, for instance.

I know why they do that -- there are a number of dynamic inports, and
folks often include matplotlib and/or scipy in apps where people may
be scripting, etc, and using features that aren't always imported by
the app. And this way, apps using those libs "just work" which is
great.

However, it would be nice to be able to override soem of this:

Is there a way to "turn off" a recipe (other than deleting it from the install)?

- Is there a way to apply excludes after the recipe? As far as I can
tell, if the recipe includes it, excludes doesn't remove it.


-Chris



On Mon, Apr 23, 2012 at 11:42 AM, Chris Barker  wrote:
> Folks,
>
> py2app doesn't seem to do "the right thing" with nested packages.
> However, it's not entirely clear what it should do...
>
> example: the "pubsub" package is delivered with wxPython, so it is
> commonly imported thusly:
>
> from wx.lib.pubsub import Publisher
>
> However, in wx.lib.pubsub, there is some trickery in the __init___, so
> as py2app installs it, ot doesn't work (see previous messages to this
> and the wxPython list). However, if you do:
>
> packages = ['wx.lib.pubsub']  ...
>
> in your py2app setup.py, you get the entire pubsub package, but installed in :
>
> Peppy.app/Contents/Resources/lib/python2.7/pubsub
>
> so it will work, but only if you now import it as:
>
> from pubsub import Publisher
>
> not too heinous, but it's really nice if we don't have to change
> imports for the bundled version.
>
> if I add 'wx' to packages, It does work, but I've then got all of wx,
> all of wx.lib, etc, which is  really a bit much (the resulting app
> bundle is 124MB, vs. 60 without specifying the wx package)
>
> At first, I thought I'd want:
>
> wx/lib/pubsub/
>
> put on sys.path, with wx and lib empty except the specified dirs.
> However, that would put two "wx" dirs on sys.path, which would not be
> good (assuming wx is used...)
>
> Would it be possible for the packages specification to be processed
> after modulegraph is run, and it would then see that "wx" and "wx.lib"
> are there, and then put all of pubsub in there? Or maybe put the parts
> of wx and wx.lib that were already going to be included outside the
> zip bundle?
>
> -Chris
>
>
> --
>
> Christopher Barker, Ph.D.
> Oceanographer
>
> Emergency Response Division
> NOAA/NOS/OR&R            (206) 526-6959   voice
> 7600 Sand Point Way NE   (206) 526-6329   fax
> Seattle, WA  98115       (206) 526-6317   main reception
>
> chris.bar...@noaa.gov



-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Pythonmac-SIG maillist  -  Pythonmac-SIG@python.org
http://mail.python.org/mailman/listinfo/pythonmac-sig
unsubscribe: http://mail.python.org/mailman/options/Pythonmac-SIG