[Distutils] distlib and data files = resources ?
Hey I was reading this : http://distlib.readthedocs.org/en/latest/overview.html#what-was-the-problem-with-packaging and I thought I should bring up that distutils2 has implemented a way to define data files that can be installed in the right places on the target system. right here being what the linux distros decides, or the default win location etc. There is a description here: http://alexis.notmyidea.org/distutils2/setupcfg.html#resources I think it's a better system than the data files that are located/and stay in the python package, maybe it would be interesting not to lose it. Cheers Tarek ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] distlib and data files = resources ?
Tarek Ziadé tarek at ziade.org writes: and I thought I should bring up that distutils2 has implemented a way to define data files that can be installed in the right places on the target system. right here being what the linux distros decides, or the default win location etc. There is a description here: http://alexis.notmyidea.org/distutils2/setupcfg.html#resources I think it's a better system than the data files that are located/and stay in the python package, maybe it would be interesting not to lose it. I would agree that it shouldn't be lost, but there are some scenarios where data located in packages might be desirable - perhaps, for example, - translation files (.mo) - templates Particularly where these relate to plugins of some sort. At the moment distlib functionality is lower-level than pysetup, so I haven't looked at these issues in detail yet. However, I agree that an important element is the mapping of logical paths to physical paths in a standardised but extensible way which is distro packager friendly and also sensible on Windows/ OS X. Regards, Vinay Sajip ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] distlib and data files = resources ?
On 11/18/12 10:15 PM, Vinay Sajip wrote:
Tarek Ziadé tarek at ziade.org writes:
and I thought I should bring up that distutils2 has implemented a way to
define data files that can be installed in the right places on the
target system. right here being what the linux distros decides, or the
default win location etc.
There is a description here:
http://alexis.notmyidea.org/distutils2/setupcfg.html#resources
I think it's a better system than the data files that are located/and
stay in the python package, maybe it would be interesting not to lose it.
I would agree that it shouldn't be lost, but there are some scenarios where data
located in packages might be desirable - perhaps, for example,
- translation files (.mo)
- templates
Those two examples have usually a specific location on the various linux
distros.
Particularly where these relate to plugins of some sort.
Having a indirection like distutils2's resources allows the data files
to live alongside the code
in development and to be installed wherever that's desired by the
distro, without breaking
the code as long as it uses the indirection function to find back the file.
Since the indirection is provided by a file that can be browsed in the
metadata directory,
it means anyone can get the file location by using the API we provided.
e.g.
get_file_path('ProjectName', 'relative/path/file.mo')
see
http://hg.python.org/distutils2/file/65e23cccd0db/distutils2/database.py#l595
At the moment distlib functionality is lower-level than pysetup, so I haven't
looked at these issues in detail yet. However, I agree that an important element
is the mapping of logical paths to physical paths in a standardised but
extensible way which is distro packager friendly and also sensible on Windows/
OS X.
That's been an issue for OS packagers since day 1, so it would be great
if we solve this in this round of new tools :)
Regards,
Vinay Sajip
___
Distutils-SIG maillist - Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig
___
Distutils-SIG maillist - Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] distlib and data files = resources ?
Tarek Ziadé tarek at ziade.org writes: Having a indirection like distutils2's resources allows the data files to live alongside the code in development and to be installed wherever that's desired by the distro, without breaking the code as long as it uses the indirection function to find back the file. Since the indirection is provided by a file that can be browsed in the metadata directory, it means anyone can get the file location by using the API we provided. Yes, but it seems like you're assuming everything's always installed in a conventional way and not, say, deployed in a .zip. I'm not disputing what you said - get_file_path and get_resource_path are still there in distlib - but I'm not sure that there's *never* a case for data located in packages. Regards, Vinay Sajip ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] distlib and data files = resources ?
On 11/18/2012 05:08 PM, Vinay Sajip wrote: Tarek Ziadé tarek at ziade.org writes: Having a indirection like distutils2's resources allows the data files to live alongside the code in development and to be installed wherever that's desired by the distro, without breaking the code as long as it uses the indirection function to find back the file. Since the indirection is provided by a file that can be browsed in the metadata directory, it means anyone can get the file location by using the API we provided. Yes, but it seems like you're assuming everything's always installed in a conventional way and not, say, deployed in a .zip. I'm not disputing what you said - get_file_path and get_resource_path are still there in distlib - but I'm not sure that there's *never* a case for data located in packages. Requiring that developers build an alternate-structure-of-files-that-arent-Python-files-just-because-they-aren't-Python-files outside the package structure seems like a terrible idea. - C ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] distlib and data files = resources ?
Just come to my mind data test related to a package would be a natural candidate: mypack |- __init__.py |- test |- __init__.py |- test0.py |- data |- data.for.test0.dat It is natural deploying them as part of an installer file. I don't see why another indirection layer is required here: you have the test.__file__ or the inspect.getfile *already*. What makes so special this case or a very similar one (eg. po files, images files for a gui based app, etc.)? Is there any reason to invent/introduce something else? I hope this helps, Antonio Vinay Sajip wrote: Tarek Ziadétarekat ziade.org writes: Having a indirection like distutils2's resources allows the data files to live alongside the code in development and to be installed wherever that's desired by the distro, without breaking the code as long as it uses the indirection function to find back the file. Since the indirection is provided by a file that can be browsed in the metadata directory, it means anyone can get the file location by using the API we provided. Yes, but it seems like you're assuming everything's always installed in a conventional way and not, say, deployed in a .zip. I'm not disputing what you said - get_file_path and get_resource_path are still there in distlib - but I'm not sure that there's *never* a case for data located in packages. Regards, Vinay Sajip ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] distlib and data files = resources ?
On 11/18/12 11:15 PM, Chris McDonough wrote: On 11/18/2012 05:08 PM, Vinay Sajip wrote: Tarek Ziadé tarek at ziade.org writes: Having a indirection like distutils2's resources allows the data files to live alongside the code in development and to be installed wherever that's desired by the distro, without breaking the code as long as it uses the indirection function to find back the file. Since the indirection is provided by a file that can be browsed in the metadata directory, it means anyone can get the file location by using the API we provided. Yes, but it seems like you're assuming everything's always installed in a conventional way and not, say, deployed in a .zip. I'm not disputing what you said - get_file_path and get_resource_path are still there in distlib - but I'm not sure that there's *never* a case for data located in packages. Requiring that developers build an alternate-structure-of-files-that-arent-Python-files-just-because-they-aren't-Python-files outside the package structure seems like a terrible idea. I think you did not get it. If you look/read carefully, it's not an alternate structure. The files are not moved - they are staying at the same place alongside the Python files in your source tree. The difference is only at installation time: they may be installed somewhere else. And when you access them - there's an API to read them instead of using __file__. Tarek ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] distlib and data files = resources ?
On 11/18/12 11:25 PM, Antonio Cavallo wrote: Just come to my mind data test related to a package would be a natural candidate: mypack |- __init__.py |- test |- __init__.py |- test0.py |- data |- data.for.test0.dat It is natural deploying them as part of an installer file. I don't see why another indirection layer is required here: you have the test.__file__ or the inspect.getfile *already*. What makes so special this case or a very similar one (eg. po files, images files for a gui based app, etc.)? Is there any reason to invent/introduce something else? One example: - if you include test that means you want test to be run - your test writes in that .dat file what happens in distros that have a read-only site-packages and only allow writes under /var/xxx ? if you define all your data files under some categories, suddenly, you are allowing OS packagers to tweak your package config to move these files to another spot, without having to change your package's code. Today, OS packagers will patch your code. I hope this helps, Antonio Vinay Sajip wrote: Tarek Ziadétarekat ziade.org writes: Having a indirection like distutils2's resources allows the data files to live alongside the code in development and to be installed wherever that's desired by the distro, without breaking the code as long as it uses the indirection function to find back the file. Since the indirection is provided by a file that can be browsed in the metadata directory, it means anyone can get the file location by using the API we provided. Yes, but it seems like you're assuming everything's always installed in a conventional way and not, say, deployed in a .zip. I'm not disputing what you said - get_file_path and get_resource_path are still there in distlib - but I'm not sure that there's *never* a case for data located in packages. Regards, Vinay Sajip ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] distlib and data files = resources ?
On 11/18/2012 05:33 PM, Tarek Ziadé wrote: On 11/18/12 11:15 PM, Chris McDonough wrote: On 11/18/2012 05:08 PM, Vinay Sajip wrote: Tarek Ziadé tarek at ziade.org writes: Having a indirection like distutils2's resources allows the data files to live alongside the code in development and to be installed wherever that's desired by the distro, without breaking the code as long as it uses the indirection function to find back the file. Since the indirection is provided by a file that can be browsed in the metadata directory, it means anyone can get the file location by using the API we provided. Yes, but it seems like you're assuming everything's always installed in a conventional way and not, say, deployed in a .zip. I'm not disputing what you said - get_file_path and get_resource_path are still there in distlib - but I'm not sure that there's *never* a case for data located in packages. Requiring that developers build an alternate-structure-of-files-that-arent-Python-files-just-because-they-aren't-Python-files outside the package structure seems like a terrible idea. I think you did not get it. If you look/read carefully, it's not an alternate structure. The files are not moved - they are staying at the same place alongside the Python files in your source tree. The difference is only at installation time: they may be installed somewhere else. And when you access them - there's an API to read them instead of using __file__. I don't really take exception to having such a feature (although I would be unlikely to use it), but I do take exception to the first paragraph of the description you pointed at: This section describes the files used by the project which must not be installed in the same place that python modules or libraries, they are called resources. They are for example documentation files, script files, databases, etc... That is not the definition of resource in setuptools-derived systems. The definition of resource in setuptools-derived systems is a non-Python file that lives in a package, which is exactly the opposite of the above definition. I'd suggest to avoid confusion we don't call files used by the project that must not be installed in the same place that python modules or libraries, and we call this other class of things something else. - C ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] distlib and data files = resources ?
On 11/18/12 11:45 PM, Chris McDonough wrote: I don't really take exception to having such a feature (although I would be unlikely to use it), but I do take exception to the first paragraph of the description you pointed at: This section describes the files used by the project which must not be installed in the same place that python modules or libraries, they are called resources. They are for example documentation files, script files, databases, etc... That is not the definition of resource in setuptools-derived systems. The definition of resource in setuptools-derived systems is a non-Python file that lives in a package, which is exactly the opposite of the above definition. I'd suggest to avoid confusion we don't call files used by the project that must not be installed in the same place that python modules or libraries, and we call this other class of things something else. I think there's a difference between: 1/ the nature of the file = e.g. not a source code file but a data file 2/ the location of the file = in site-packages as opposed to anywhere else The name resource for me is only defined by 1/ and I don't see any confusion or opposed definitions in that naming. Neither did the people that worked on this feature. But if you feel it's confusing, I am all for a separate name - I can't think of another one though, do you have one in mind ? 'data' seems to much generic... Tarek ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] distlib and data files = resources ?
On 11/18/2012 05:55 PM, Tarek Ziadé wrote: On 11/18/12 11:45 PM, Chris McDonough wrote: I don't really take exception to having such a feature (although I would be unlikely to use it), but I do take exception to the first paragraph of the description you pointed at: This section describes the files used by the project which must not be installed in the same place that python modules or libraries, they are called resources. They are for example documentation files, script files, databases, etc... That is not the definition of resource in setuptools-derived systems. The definition of resource in setuptools-derived systems is a non-Python file that lives in a package, which is exactly the opposite of the above definition. I'd suggest to avoid confusion we don't call files used by the project that must not be installed in the same place that python modules or libraries, and we call this other class of things something else. I think there's a difference between: 1/ the nature of the file = e.g. not a source code file but a data file 2/ the location of the file = in site-packages as opposed to anywhere else The name resource for me is only defined by 1/ and I don't see any confusion or opposed definitions in that naming. Neither did the people that worked on this feature. But if you feel it's confusing, I am all for a separate name - I can't think of another one though, do you have one in mind ? Seems to me that ... files used by the project which must not be installed in the same place that python modules or libraries, they are called resources and resources = ... in the setup.cfg contradicts existing understanding of what a resource is in terms of Python packaging. At least I was confused. If you don't want another name, I suppose you could define the keys as actions you take against resources in setup.cfg copy_resources = doc/doc/man = destination_doc and/or move_resources = ... doc/doc/man = destination_doc And change the docs to define resource as something that indeed does live in a package, at least until it's moved by the installer, and that you can ensure they are copied or moved by using copy_resources and/or move_resources. - C ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
