Re: [Distutils] unparseable sdist filenames
On Mon, Sep 30, 2013 at 6:06 PM, Marcus Smith wrote: >> >> > how will context decide between the version being "dev" or "xdist-dev"? >> >> By whether you asked to install "pytest-xdist" or "pytest", and by >> whether "dev" or "xdist-dev" match your version requirements. In >> practice this tends to only be a problem if you are: >> >> 1. Installing the two packages during the same run, and >> 2. Aren't using version specifiers. > > > #2 is a big deal. the pip issue (#1192) that motivated this was a #2. local > find-links with non-versioned requirements. I should probably add a #3 to that list: it's also rarely a problem because usually there's at least one *numbered* version of the desired package available, which will always prioritize newer than an arbitrary alphabetic version like "xdist", even if you haven't requested a specific version. IOW, you need to not only not request a version and come across the other package in the same run, but you *also* need for there not to be any valid versions of your intended target present. > so, suppose you have "pytest-xdist-dev.tar.gz" in a find-links location. > whether you're trying to install "pytest" or "pytest-xdist" doesn't help the > installer determine how to parse that archive. In easy_install's case, it will take the one with the highest version number, which means it'll try to install pytest-xdist-dev.tar.gz, on the theory that 'xdist-dev' is a newer version than 'dev'. This is kind of silly, but it was the Simplest Possible Thing That Could Work, and I figured I'd change it when there was enough feedback/field experience to decide what to change it to. ;-) Probably the "right" thing to do would be to warn or decline on ambiguous filenames in the future, since it is quite possible to detect if a filename has more than one possible interpretation. Another possibility would be to treat them analagously to stable/unstable versions, prioritize them below regular matches, etc. ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] unparseable sdist filenames
> > > > how will context decide between the version being "dev" or "xdist-dev"? > > By whether you asked to install "pytest-xdist" or "pytest", and by > whether "dev" or "xdist-dev" match your version requirements. In > practice this tends to only be a problem if you are: > > 1. Installing the two packages during the same run, and > 2. Aren't using version specifiers. > #2 is a big deal. the pip issue (#1192) that motivated this was a #2. local find-links with non-versioned requirements. so, suppose you have "pytest-xdist-dev.tar.gz" in a find-links location. whether you're trying to install "pytest" or "pytest-xdist" doesn't help the installer determine how to parse that archive. ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] unparseable sdist filenames
On Mon, Sep 30, 2013 at 4:11 PM, Marcus Smith wrote: > >> The setuptools.package_index API, however, *does* support parsing >> sdist names, it's just that it generates a *list* of possibilities, > > > oh, ok, "setuptools.package_index.distros_for_url" > >> >> Thus, tools using this API can contextually decide which to consider >> the "real" interpretation, based on context. This method is used by >> easy_install; I don't know if pip does as well. > > > how will context decide between the version being "dev" or "xdist-dev"? By whether you asked to install "pytest-xdist" or "pytest", and by whether "dev" or "xdist-dev" match your version requirements. In practice this tends to only be a problem if you are: 1. Installing the two packages during the same run, and 2. Aren't using version specifiers. ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] unparseable sdist filenames
> The setuptools.package_index API, however, *does* support parsing > sdist names, it's just that it generates a *list* of possibilities, > oh, ok, "setuptools.package_index.distros_for_url" > Thus, tools using this API can contextually decide which to consider > the "real" interpretation, based on context. This method is used by > easy_install; I don't know if pip does as well. > how will context decide between the version being "dev" or "xdist-dev"? > - pip will accept this as a "pytest" archive and install it potentially if > > no other version matches greater. > > Does it also accept it as "pytest-xdist"? yes. without getting into too many details, pip basically searches for matches by comparing the archive name with "-". I.e when looking for "pytest" archives, match on "pytest-". and the version is the rest. ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] unparseable sdist filenames
> > what's the right answer? > > IMO it should be ignored, i.e. be rejected as a candidate for installation. > (If pointing pip to the local file it's different of course). > This particular name isn't a real-life use case, is it? > the real life issue that motivated this is https://github.com/pypa/pip/issues/1192 what we discovered was that pip was allowing 'pytest-xdist' archives to fulfill a pytest requirement. to fix it, I was going to start using distlib's split_filename, but then noticed it didn't accept "dev" as a version, and one of pip's tests were failing as a result (which was specifically confirming pip could install "dev" versions) Vinay has just pushed an update to split_filename to handle the dev case (and others) https://bitbucket.org/pypa/distlib/commits/22c38563185bf13bfd267efc88d147a51e9880e3 ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] unparseable sdist filenames
On Mon, Sep 30, 2013 at 10:55 -0700, Marcus Smith wrote: > so, take a case like so "pytest-xdist-dev.tar.gz" (or any sdist with "-" > in the project name, and a version starting with a string) > > I think it's like so: > - pkg_resources.Distribution.from_location will treat "xdist-dev" as the > version. > - distlib.util.split_filename won't parse it because versions have to start > with [0-9]. > - pip will accept this as a "pytest" archive and install it potentially if > no other version matches greater. > > what's the right answer? IMO it should be ignored, i.e. be rejected as a candidate for installation. (If pointing pip to the local file it's different of course). This particular name isn't a real-life use case, is it? best, holger > The historical-compatible answer is to be confused when projects have "-". > so stay confused? or get rigid like distlib? > > Marcus > ___ > Distutils-SIG maillist - Distutils-SIG@python.org > https://mail.python.org/mailman/listinfo/distutils-sig ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] unparseable sdist filenames
On Mon, Sep 30, 2013 at 1:55 PM, Marcus Smith wrote: > so, take a case like so "pytest-xdist-dev.tar.gz" (or any sdist with "-" in > the project name, and a version starting with a string) > > I think it's like so: > - pkg_resources.Distribution.from_location will treat "xdist-dev" as the > version. > - distlib.util.split_filename won't parse it because versions have to start > with [0-9]. For these APIs, these are reasonable defaults, because they are intended only to parse *unambiguous* filenames. Neither API should be used with sdist files, because sdist filenames are ambiguous and cannot therefore be parsed to a single name. The setuptools.package_index API, however, *does* support parsing sdist names, it's just that it generates a *list* of possibilities, rather than a single unambiguous interpretation of the filename. If you ask it to parse the example you gave, you will get a list of: * Project "pydist", version "xdist-dev" * Project "pydist-xdist", version "dev" * Project "pydist-xdist-dev", no version Thus, tools using this API can contextually decide which to consider the "real" interpretation, based on context. This method is used by easy_install; I don't know if pip does as well. > - pip will accept this as a "pytest" archive and install it potentially if > no other version matches greater. Does it also accept it as "pytest-xdist"? > what's the right answer? It depends on what you want, and whether what you want is compatible with reality. ;-) > The historical-compatible answer is to be confused when projects have "-". > so stay confused? or get rigid like distlib? The future sdist format should not be ambiguous like this. Dealing with ambiguity in past packages is for the moment something of a requirement if you are making a general-use tool. ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] unparseable sdist filenames
On 30 September 2013 18:55, Marcus Smith wrote: > so, take a case like so "pytest-xdist-dev.tar.gz" (or any sdist with "-" in > the project name, and a version starting with a string) > > I think it's like so: > - pkg_resources.Distribution.from_location will treat "xdist-dev" as the > version. > - distlib.util.split_filename won't parse it because versions have to start > with [0-9]. > - pip will accept this as a "pytest" archive and install it potentially if > no other version matches greater. > > what's the right answer? > > The historical-compatible answer is to be confused when projects have "-". > so stay confused? or get rigid like distlib? tl;dr Let's have a strict standard and then tools should go for the usual "be lenient in what you accept, but strict in what you produce" principle. To expand on that, IMO, we should specify in one of the metadata-type PEPs a set of precise, parseable rules (I avoid the word "rigid" here because of the negative connotations, but that's what I mean). Projects that need to may or may not change to conform to the spec, so there may still be cases where tools have to be confused for backward compatibility reasons, but at least we then make those cases into clearly expressed "undefined behaviour" cases. Whether tools converge on a common understanding for those cases, or remain confused as now, is less of an issue to me, as the clear message is that the project has a means to get consistent behaviour by changing to match the standard, and if they choose not to they can accept the consequences with no worse impact than we currently have. But I'd also agree with Donald that stats on the scale of the problem would help to better inform a decision here. Paul. ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] unparseable sdist filenames
On Sep 30, 2013, at 1:55 PM, Marcus Smith wrote: > so, take a case like so "pytest-xdist-dev.tar.gz" (or any sdist with "-" in > the project name, and a version starting with a string) > > I think it's like so: > - pkg_resources.Distribution.from_location will treat "xdist-dev" as the > version. > - distlib.util.split_filename won't parse it because versions have to start > with [0-9]. > - pip will accept this as a "pytest" archive and install it potentially if no > other version matches greater. > > what's the right answer? > > The historical-compatible answer is to be confused when projects have "-". > so stay confused? or get rigid like distlib? > > Marcus > ___ > Distutils-SIG maillist - Distutils-SIG@python.org > https://mail.python.org/mailman/listinfo/distutils-sig I think we're going to need more data to figure out a solution. We need to see how many things are going to be affected by this. If it's not a massive amount I think we should get strict. - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
[Distutils] unparseable sdist filenames
so, take a case like so "pytest-xdist-dev.tar.gz" (or any sdist with "-" in the project name, and a version starting with a string) I think it's like so: - pkg_resources.Distribution.from_location will treat "xdist-dev" as the version. - distlib.util.split_filename won't parse it because versions have to start with [0-9]. - pip will accept this as a "pytest" archive and install it potentially if no other version matches greater. what's the right answer? The historical-compatible answer is to be confused when projects have "-". so stay confused? or get rigid like distlib? Marcus ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Problem: couldn't find a setup script in C:\Users\[user]\[package]
Indeed, if there is a folder called "foobar" in your current directory, and you run "easy_install foobar", easy_install will assume you are trying to install the package in the folder "foobar", which fails if it is not a package. One solution to that is to change your current directory to a directory that does not include the folder "foobar". //Lennart On Mon, Sep 30, 2013 at 11:19 AM, Martin Fiers wrote: > Hello! When a folder inside a user's home directory has the same name as a > package, then installation fails. > > What steps will reproduce the problem? > 1. On a Windows system, start with a vanilla python, with pip not installed > 2. Create the folder pip in C:\Users\[username] (so now there's a folder > C:\Users\[username]\pip) > 3. run easy_install pip > > (the error is reproduced with any other package as well, as long as there's > a home folder). > > What is the expected output? What do you see instead? > pip is installed > Instead, this gives the error: > Couldn't find a setup script in /home/[username]/pip > > What version of the product are you using? On what operating system? > Windows 7, Python 2.7.5, with ez_setup.py called so setuptools is installed. > > Please provide any additional information below. > This is rather annoying, since our software creates a folder in the user's > home directory. > But also the pip packages creates a log folder. So when I accidentially > remove pip and want to > re-install, easy_install does not work. > > Thanks very much, > Martin > > > > ___ > Distutils-SIG maillist - Distutils-SIG@python.org > https://mail.python.org/mailman/listinfo/distutils-sig > ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
[Distutils] Problem: couldn't find a setup script in C:\Users\[user]\[package]
Hello! When a folder inside a user's home directory has the same name as a package, then installation fails. *What steps will reproduce the problem?* 1. On a Windows system, start with a vanilla python, with pip not installed 2. Create the folder pip in C:\Users\[username] (so now there's a folder C:\Users\[username]\pip) 3. run easy_install pip (the error is reproduced with any other package as well, as long as there's a home folder). *What is the expected output? What do you see instead?* pip is installed Instead, this gives the error: Couldn't find a setup script in /home/[username]/pip *What version of the product are you using? On what operating system?* Windows 7, Python 2.7.5, with ez_setup.py called so setuptools is installed. *Please provide any additional information below.* This is rather annoying, since our software creates a folder in the user's home directory. But also the pip packages creates a log folder. So when I accidentially remove pip and want to re-install, easy_install does not work. Thanks very much, Martin ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
