[Distutils] Re: Docker builds, ~/.pip/cache, and O(1) or O(n) bandwidth
RUN --mount needs a magic comment to opt into experimental syntax: # syntax=docker/dockerfile:experimental https://github.com/moby/buildkit/blob/master/frontend/dockerfile/docs/experimental.md but AFAIK, was designed for this purpose, among others. Thanks, On Tue, May 7, 2019, 11:33 AM Daniel Holth wrote: > Also you must set an environment variable for docker build to be able to > use --mount > > On Tue, May 7, 2019, 12:28 Daniel Holth wrote: > >> That one is multistage, great feature, but I think cache mounts would >> work fine with any number of stages. The mount option is a pretty new >> feature. >> >> On Tue, May 7, 2019, 12:24 Wes Turner wrote: >> >>> >>> >>> On Tuesday, May 7, 2019, Daniel Holth wrote: >>> Have you tried using buildkit and the RUN --mount option? I've done extra stuff here (downloading rpms in a second image first) but I think you could just use the cache option. It would also be easy to use a second image and COPY in old docker. https://github.com/dholth/vagrant-docker/blob/master/Dockerfile#L86 >>> >>> Thanks. Is this a multi-stage Dockerfile (with multiple FROM >>> instructions)? >>> >>> Is there a guide for a PyPA recommended way to create network-efficient >>> reproducible (manylinux2010) wheels? >>> >>> On Tue, May 7, 2019, 11:59 Alex Becker wrote: > You can use a local PyPI mirror, e.g. devpi, and point your docker > builds at that, basically tricking docker by going through the (local) > network stack instead of the filesystem. > > On Tue, May 7, 2019 at 8:12 AM Wes Turner > wrote: > >> What is the best way to build docker images without constantly >> re-downloading packages from PyPI (to use ~O(1) bandwidth instead of O(n) >> for every build) >> >> (AFAIK, nobody has any issue with the amount of bandwidth PyPI uses) >> >> Thus far, Docker doesn't want to support a build-time -v option (that >> could be used to bind-mount .pip/cache in at build time): >> >> "build time only -v option" >> https://github.com/moby/moby/issues/14080 >> >> Buildah *does* support a build-time -v option (and can also do >> rootless builds without a docker socket) >> -- >> Distutils-SIG mailing list -- distutils-sig@python.org >> To unsubscribe send an email to distutils-sig-le...@python.org >> https://mail.python.org/mailman3/lists/distutils-sig.python.org/ >> Message archived at >> https://mail.python.org/archives/list/distutils-sig@python.org/message/IL6MGTSAA4X5UZDERCAAAFGGX5DGRPF3/ >> > -- > Distutils-SIG mailing list -- distutils-sig@python.org > To unsubscribe send an email to distutils-sig-le...@python.org > https://mail.python.org/mailman3/lists/distutils-sig.python.org/ > Message archived at > https://mail.python.org/archives/list/distutils-sig@python.org/message/2MNMGYBNFOOVGXA2JQ2PLDTZITZXANJU/ > -- > Distutils-SIG mailing list -- distutils-sig@python.org > To unsubscribe send an email to distutils-sig-le...@python.org > https://mail.python.org/mailman3/lists/distutils-sig.python.org/ > Message archived at > https://mail.python.org/archives/list/distutils-sig@python.org/message/JXVF2IQKGCAGLH2A6KYOIQARJGXVKTI5/ > -- Distutils-SIG mailing list -- distutils-sig@python.org To unsubscribe send an email to distutils-sig-le...@python.org https://mail.python.org/mailman3/lists/distutils-sig.python.org/ Message archived at https://mail.python.org/archives/list/distutils-sig@python.org/message/K6UW3XEYYGPYX7VZVJOCQQIMFE7VJGDK/
[Distutils] Re: Docker builds, ~/.pip/cache, and O(1) or O(n) bandwidth
Also you must set an environment variable for docker build to be able to use --mount On Tue, May 7, 2019, 12:28 Daniel Holth wrote: > That one is multistage, great feature, but I think cache mounts would work > fine with any number of stages. The mount option is a pretty new feature. > > On Tue, May 7, 2019, 12:24 Wes Turner wrote: > >> >> >> On Tuesday, May 7, 2019, Daniel Holth wrote: >> >>> Have you tried using buildkit and the RUN --mount option? I've done >>> extra stuff here (downloading rpms in a second image first) but I think you >>> could just use the cache option. >>> >>> It would also be easy to use a second image and COPY in old docker. >>> >>> https://github.com/dholth/vagrant-docker/blob/master/Dockerfile#L86 >>> >> >> Thanks. Is this a multi-stage Dockerfile (with multiple FROM >> instructions)? >> >> Is there a guide for a PyPA recommended way to create network-efficient >> reproducible (manylinux2010) wheels? >> >> >>> >>> On Tue, May 7, 2019, 11:59 Alex Becker wrote: >>> You can use a local PyPI mirror, e.g. devpi, and point your docker builds at that, basically tricking docker by going through the (local) network stack instead of the filesystem. On Tue, May 7, 2019 at 8:12 AM Wes Turner wrote: > What is the best way to build docker images without constantly > re-downloading packages from PyPI (to use ~O(1) bandwidth instead of O(n) > for every build) > > (AFAIK, nobody has any issue with the amount of bandwidth PyPI uses) > > Thus far, Docker doesn't want to support a build-time -v option (that > could be used to bind-mount .pip/cache in at build time): > > "build time only -v option" > https://github.com/moby/moby/issues/14080 > > Buildah *does* support a build-time -v option (and can also do > rootless builds without a docker socket) > -- > Distutils-SIG mailing list -- distutils-sig@python.org > To unsubscribe send an email to distutils-sig-le...@python.org > https://mail.python.org/mailman3/lists/distutils-sig.python.org/ > Message archived at > https://mail.python.org/archives/list/distutils-sig@python.org/message/IL6MGTSAA4X5UZDERCAAAFGGX5DGRPF3/ > -- Distutils-SIG mailing list -- distutils-sig@python.org To unsubscribe send an email to distutils-sig-le...@python.org https://mail.python.org/mailman3/lists/distutils-sig.python.org/ Message archived at https://mail.python.org/archives/list/distutils-sig@python.org/message/2MNMGYBNFOOVGXA2JQ2PLDTZITZXANJU/ >>> -- Distutils-SIG mailing list -- distutils-sig@python.org To unsubscribe send an email to distutils-sig-le...@python.org https://mail.python.org/mailman3/lists/distutils-sig.python.org/ Message archived at https://mail.python.org/archives/list/distutils-sig@python.org/message/JXVF2IQKGCAGLH2A6KYOIQARJGXVKTI5/
[Distutils] Re: Docker builds, ~/.pip/cache, and O(1) or O(n) bandwidth
That one is multistage, great feature, but I think cache mounts would work fine with any number of stages. The mount option is a pretty new feature. On Tue, May 7, 2019, 12:24 Wes Turner wrote: > > > On Tuesday, May 7, 2019, Daniel Holth wrote: > >> Have you tried using buildkit and the RUN --mount option? I've done extra >> stuff here (downloading rpms in a second image first) but I think you could >> just use the cache option. >> >> It would also be easy to use a second image and COPY in old docker. >> >> https://github.com/dholth/vagrant-docker/blob/master/Dockerfile#L86 >> > > Thanks. Is this a multi-stage Dockerfile (with multiple FROM instructions)? > > Is there a guide for a PyPA recommended way to create network-efficient > reproducible (manylinux2010) wheels? > > >> >> On Tue, May 7, 2019, 11:59 Alex Becker wrote: >> >>> You can use a local PyPI mirror, e.g. devpi, and point your docker >>> builds at that, basically tricking docker by going through the (local) >>> network stack instead of the filesystem. >>> >>> On Tue, May 7, 2019 at 8:12 AM Wes Turner wrote: >>> What is the best way to build docker images without constantly re-downloading packages from PyPI (to use ~O(1) bandwidth instead of O(n) for every build) (AFAIK, nobody has any issue with the amount of bandwidth PyPI uses) Thus far, Docker doesn't want to support a build-time -v option (that could be used to bind-mount .pip/cache in at build time): "build time only -v option" https://github.com/moby/moby/issues/14080 Buildah *does* support a build-time -v option (and can also do rootless builds without a docker socket) -- Distutils-SIG mailing list -- distutils-sig@python.org To unsubscribe send an email to distutils-sig-le...@python.org https://mail.python.org/mailman3/lists/distutils-sig.python.org/ Message archived at https://mail.python.org/archives/list/distutils-sig@python.org/message/IL6MGTSAA4X5UZDERCAAAFGGX5DGRPF3/ >>> -- >>> Distutils-SIG mailing list -- distutils-sig@python.org >>> To unsubscribe send an email to distutils-sig-le...@python.org >>> https://mail.python.org/mailman3/lists/distutils-sig.python.org/ >>> Message archived at >>> https://mail.python.org/archives/list/distutils-sig@python.org/message/2MNMGYBNFOOVGXA2JQ2PLDTZITZXANJU/ >>> >> -- Distutils-SIG mailing list -- distutils-sig@python.org To unsubscribe send an email to distutils-sig-le...@python.org https://mail.python.org/mailman3/lists/distutils-sig.python.org/ Message archived at https://mail.python.org/archives/list/distutils-sig@python.org/message/UWNYYGT7BGCEOM4JP7WMMPFQJZ2HBGHC/
[Distutils] Re: Docker builds, ~/.pip/cache, and O(1) or O(n) bandwidth
On Tuesday, May 7, 2019, Daniel Holth wrote: > Have you tried using buildkit and the RUN --mount option? I've done extra > stuff here (downloading rpms in a second image first) but I think you could > just use the cache option. > > It would also be easy to use a second image and COPY in old docker. > > https://github.com/dholth/vagrant-docker/blob/master/Dockerfile#L86 > Thanks. Is this a multi-stage Dockerfile (with multiple FROM instructions)? Is there a guide for a PyPA recommended way to create network-efficient reproducible (manylinux2010) wheels? > > On Tue, May 7, 2019, 11:59 Alex Becker wrote: > >> You can use a local PyPI mirror, e.g. devpi, and point your docker builds >> at that, basically tricking docker by going through the (local) network >> stack instead of the filesystem. >> >> On Tue, May 7, 2019 at 8:12 AM Wes Turner wrote: >> >>> What is the best way to build docker images without constantly >>> re-downloading packages from PyPI (to use ~O(1) bandwidth instead of O(n) >>> for every build) >>> >>> (AFAIK, nobody has any issue with the amount of bandwidth PyPI uses) >>> >>> Thus far, Docker doesn't want to support a build-time -v option (that >>> could be used to bind-mount .pip/cache in at build time): >>> >>> "build time only -v option" >>> https://github.com/moby/moby/issues/14080 >>> >>> Buildah *does* support a build-time -v option (and can also do rootless >>> builds without a docker socket) >>> -- >>> Distutils-SIG mailing list -- distutils-sig@python.org >>> To unsubscribe send an email to distutils-sig-le...@python.org >>> https://mail.python.org/mailman3/lists/distutils-sig.python.org/ >>> Message archived at https://mail.python.org/archives/list/distutils-sig@ >>> python.org/message/IL6MGTSAA4X5UZDERCAAAFGGX5DGRPF3/ >>> >> -- >> Distutils-SIG mailing list -- distutils-sig@python.org >> To unsubscribe send an email to distutils-sig-le...@python.org >> https://mail.python.org/mailman3/lists/distutils-sig.python.org/ >> Message archived at https://mail.python.org/archives/list/distutils-sig@ >> python.org/message/2MNMGYBNFOOVGXA2JQ2PLDTZITZXANJU/ >> > -- Distutils-SIG mailing list -- distutils-sig@python.org To unsubscribe send an email to distutils-sig-le...@python.org https://mail.python.org/mailman3/lists/distutils-sig.python.org/ Message archived at https://mail.python.org/archives/list/distutils-sig@python.org/message/WHEVRTBBCTRKJX6BKT43DXJMOLFUEUB7/
[Distutils] Re: Docker builds, ~/.pip/cache, and O(1) or O(n) bandwidth
Have you tried using buildkit and the RUN --mount option? I've done extra stuff here (downloading rpms in a second image first) but I think you could just use the cache option. It would also be easy to use a second image and COPY in old docker. https://github.com/dholth/vagrant-docker/blob/master/Dockerfile#L86 On Tue, May 7, 2019, 11:59 Alex Becker wrote: > You can use a local PyPI mirror, e.g. devpi, and point your docker builds > at that, basically tricking docker by going through the (local) network > stack instead of the filesystem. > > On Tue, May 7, 2019 at 8:12 AM Wes Turner wrote: > >> What is the best way to build docker images without constantly >> re-downloading packages from PyPI (to use ~O(1) bandwidth instead of O(n) >> for every build) >> >> (AFAIK, nobody has any issue with the amount of bandwidth PyPI uses) >> >> Thus far, Docker doesn't want to support a build-time -v option (that >> could be used to bind-mount .pip/cache in at build time): >> >> "build time only -v option" >> https://github.com/moby/moby/issues/14080 >> >> Buildah *does* support a build-time -v option (and can also do rootless >> builds without a docker socket) >> -- >> Distutils-SIG mailing list -- distutils-sig@python.org >> To unsubscribe send an email to distutils-sig-le...@python.org >> https://mail.python.org/mailman3/lists/distutils-sig.python.org/ >> Message archived at >> https://mail.python.org/archives/list/distutils-sig@python.org/message/IL6MGTSAA4X5UZDERCAAAFGGX5DGRPF3/ >> > -- > Distutils-SIG mailing list -- distutils-sig@python.org > To unsubscribe send an email to distutils-sig-le...@python.org > https://mail.python.org/mailman3/lists/distutils-sig.python.org/ > Message archived at > https://mail.python.org/archives/list/distutils-sig@python.org/message/2MNMGYBNFOOVGXA2JQ2PLDTZITZXANJU/ > -- Distutils-SIG mailing list -- distutils-sig@python.org To unsubscribe send an email to distutils-sig-le...@python.org https://mail.python.org/mailman3/lists/distutils-sig.python.org/ Message archived at https://mail.python.org/archives/list/distutils-sig@python.org/message/HUYZYFQVUGBVCQWT5AEJUTXQW4VJTZ6L/
[Distutils] Re: Docker builds, ~/.pip/cache, and O(1) or O(n) bandwidth
You can use a local PyPI mirror, e.g. devpi, and point your docker builds at that, basically tricking docker by going through the (local) network stack instead of the filesystem. On Tue, May 7, 2019 at 8:12 AM Wes Turner wrote: > What is the best way to build docker images without constantly > re-downloading packages from PyPI (to use ~O(1) bandwidth instead of O(n) > for every build) > > (AFAIK, nobody has any issue with the amount of bandwidth PyPI uses) > > Thus far, Docker doesn't want to support a build-time -v option (that > could be used to bind-mount .pip/cache in at build time): > > "build time only -v option" > https://github.com/moby/moby/issues/14080 > > Buildah *does* support a build-time -v option (and can also do rootless > builds without a docker socket) > -- > Distutils-SIG mailing list -- distutils-sig@python.org > To unsubscribe send an email to distutils-sig-le...@python.org > https://mail.python.org/mailman3/lists/distutils-sig.python.org/ > Message archived at > https://mail.python.org/archives/list/distutils-sig@python.org/message/IL6MGTSAA4X5UZDERCAAAFGGX5DGRPF3/ > -- Distutils-SIG mailing list -- distutils-sig@python.org To unsubscribe send an email to distutils-sig-le...@python.org https://mail.python.org/mailman3/lists/distutils-sig.python.org/ Message archived at https://mail.python.org/archives/list/distutils-sig@python.org/message/2MNMGYBNFOOVGXA2JQ2PLDTZITZXANJU/
