I have rarely* "pointed at a tarball and installed" but used `pip install <package expression>` which I understand is implemented as you describe, but with some naive version logic. Maybe just personal style, I just can't go back from package management, even if it lacks real dependency resolution. I do know that pip is flexible enough that it is very easy to install from a custom directory/prefix, since like Maven it really is just a directory. I rolled this a couple times so internal-only packages had the same install story as public code. We could use this to set things up in a temp prefix.
Kenn *I have more Python experience than Java ;-) though it is stale by ~3.5 years On Fri, Mar 9, 2018 at 1:47 PM Robert Bradshaw <rober...@google.com> wrote: > Python development isn't generally like Java in this sense--you just point > people at a tarball that they install. Granted, this doesn't work as well > when one has a set of 100 tarballs that all should be installed together, > but again that's not the Python way to release a project. > > I did find https://test.pypi.org/project/apache-beam/#files which seems > the > closest thing to this. They're marked with the release candidate name > because (again unlike java) it's considered bad practice to release an > artifact under one name, and then push a new "version" of that artifact > (the way we do with https://dist.apache.org/repos/dist/dev/beam/2.4.0/ > where the binary blobs there are whatever was push last). > > > On Fri, Mar 9, 2018 at 1:27 PM Kenneth Knowles <k...@google.com> wrote: > > > I think the thing is just that you can point a user env at it to test > like a user. Like you can do this with > https://repository.apache.org/content/repositories/orgapachebeam-1030/ but > not https://dist.apache.org/repos/dist/dev/beam/2.4.0/ > > > On Wed, Mar 7, 2018 at 5:03 PM Ahmet Altay <al...@google.com> wrote: > > >> I do not know what is the best practice. For practical purposes it makes > sense to stage to the same svn repo, so that we can test it as part of the > release process. > > >> On Wed, Mar 7, 2018 at 4:22 PM, Robert Bradshaw <rober...@google.com> > wrote: > > >>> Yes, we should. There's a bit of an open question of where these > release artifacts should be staged. (Eventually, of course, they'll be > published to PyPi). Should they be placed alongside the source artifacts in > the svn repository? > > > >>> On Wed, Mar 7, 2018 at 3:00 PM Ahmet Altay <al...@google.com> wrote: > > >>>> Are we planning to do this for the 2.4.0 release? I am asking, because > they were not part of RC1 artifacts. > > >>>> On Tue, Feb 13, 2018 at 9:18 AM, Robert Bradshaw <rober...@google.com > > > wrote: > > >>>>> On Tue, Feb 13, 2018 at 8:31 AM, Nima Mousavi < > nima.mous...@gmail.com> > wrote: > >>>>> > Related question: > >>>>> > > >>>>> > How can we tell if the docker image of our binary contains the > cython > >>>>> > optimized beam or the slower codepath? > >>>>> > The image was built on Google cloud (using gcloud container builds > submit). > > >>>>> There are certain modules (corresponding to the pyx files) that are > >>>>> only built if Cython is present. We can (1) make sure Cython is > >>>>> installed before installing apache beam into the container, and (2) > >>>>> assert as part of the build process that these modules exist. > > >>>>> > On Mon, Feb 12, 2018 at 9:32 PM, Ahmet Altay <al...@google.com> > wrote: > >>>>> >> > >>>>> >> +1 to wheels. The main effort for this would be updating the > release > >>>>> >> guide, and adding support for other platforms in Jenkins for > building and > >>>>> >> testing wheels. In light of this, maybe we can prioritize having > test > >>>>> >> infrastructure for other platforms. > >>>>> >> > >>>>> >> On Mon, Feb 12, 2018 at 1:47 PM, Ismaël Mejía <ieme...@gmail.com> > wrote: > >>>>> >>> > >>>>> >>> +1 for wheels, they are the standard binary distribution format > so it > >>>>> >>> makes sense. Also wheels support packaging python 2 and 3 on > universal > >>>>> >>> packages so they are future proof. > >>>>> >>> > >>>>> >>> On Mon, Feb 12, 2018 at 10:26 PM, Robert Bradshaw < > rober...@google.com> > >>>>> >>> wrote: > >>>>> >>> > +1, is it too late to try to release these as part of the 2.3 > release > >>>>> >>> > (to get familiar with the process, no code changes should be > needed)? > >>>>> >> > >>>>> >> > >>>>> >> It would be nice to have this for the current release. How can we > build > >>>>> >> and test these binaries? I think it will be prudent to waIt until > we have > >>>>> >> infrastructure. > >>>>> >> > >>>>> >>> > >>>>> >>> > > >>>>> >>> > The wheels are advantageous when running locally (e.g. during > testing > >>>>> >>> > and development) where requiring containers will probably be > overkill. > >>>>> >>> > This will become especially relevant with the switch to use the > >>>>> >>> > FnApiRunner. > >>>>> >>> > > >>>>> >>> > On Mon, Feb 12, 2018 at 1:22 PM, Lukasz Cwik <lc...@google.com > > > wrote: > >>>>> >>> >> If we want all our code related to pipeline execution to be in > a > >>>>> >>> >> container, > >>>>> >>> >> what value does building wheel distributions provide? > >>>>> >>> >> > >>>>> >>> >> > >>>>> >>> >> On Mon, Feb 12, 2018 at 1:18 PM, Kenneth Knowles < > k...@google.com> > >>>>> >>> >> wrote: > >>>>> >>> >>> > >>>>> >>> >>> +1 > >>>>> >>> >>> > >>>>> >>> >>> On Mon, Feb 12, 2018 at 1:04 PM, Charles Chen < > c...@google.com> > wrote: > >>>>> >>> >>>> > >>>>> >>> >>>> Currently, Apache Beam distributes Python packages through > pip and > >>>>> >>> >>>> PyPI. > >>>>> >>> >>>> On PyPI, developers can release either source tarballs, and > / or > >>>>> >>> >>>> precompiled > >>>>> >>> >>>> "wheel" distributions for each platform, which would be used > if > >>>>> >>> >>>> available > >>>>> >>> >>>> for a particular platform. Currently, we only distribute > the source > >>>>> >>> >>>> tarballs, so any user who installs Beam using "pip install > >>>>> >>> >>>> apache_beam" has > >>>>> >>> >>>> to have a compiler and toolchain installed to take advantage > of > >>>>> >>> >>>> Cython > >>>>> >>> >>>> optimizations in Beam (which require compiled C code). If > such a > >>>>> >>> >>>> compiler > >>>>> >>> >>>> is not available, Beam is currently configured to install > anyway, > >>>>> >>> >>>> but will > >>>>> >>> >>>> use slower Python codepaths instead of the more optimized > ones (for > >>>>> >>> >>>> example, > >>>>> >>> >>>> for Coder encoding / decoding). > >>>>> >>> >>>> > >>>>> >>> >>>> I would like to propose that we start distributing binary > wheel > >>>>> >>> >>>> distributions for our releases, for common platforms like > Windows / > >>>>> >>> >>>> Mac / > >>>>> >>> >>>> Linux. We could potentially use a method similar to this > one > >>>>> >>> >>>> (https://github.com/MacPython/cython-wheels) for building > these > >>>>> >>> >>>> wheel > >>>>> >>> >>>> distributions. Thoughts? > >>>>> >>> >>>> > >>>>> >>> >>>> Best, > >>>>> >>> >>>> Charles > >>>>> >>> >>> > >>>>> >>> >>> > >>>>> >>> >> > >>>>> >> > >>>>> >> > >>>>> > >
