[openstack-dev] [tc][infra][release][security][stable][kolla][loci][tripleo][docker][kubernetes] do we want to be publishing binary container images?
Last week at the Forum we had a couple of discussions about collaboration between the various teams building or consuming container images. One topic that came up was deciding how to publish images from the various teams to docker hub or other container registries. While the technical bits seem easy enough to work out, there is still the question of precedence and whether it's a good idea to do so at all. In the past, we have refrained from publishing binary packages in other formats such as debs and RPMs. (We did publish debs way back in the beginning, for testing IIRC, but switched away from them to sdists to be more inclusive.) Since then, we have said it is the responsibility of downstream consumers to build production packages, either as distributors or as a deployer that is rolling their own. We do package sdists for python libraries, push some JavaScript to the NPM registries, and have tarballs of those and a bunch of other artifacts that we build out of our release tools. But none of those is declared as "production ready," and so the community is not sending the signal that we are responsible for maintaining them in the context of production deployments, beyond continuing to produce new releases when there are bugs. Container images introduce some extra complexity, over the basic operating system style packages mentioned above. Due to the way they are constructed, they are likely to include content we don't produce ourselves (either in the form of base layers or via including build tools or other things needed when assembling the full image). That extra content means there would need to be more tracking of upstream issues (bugs, CVEs, etc.) to ensure the images are updated as needed. Given our security and stable team resources, I'm not entirely comfortable with us publishing these images, and giving the appearance that the community *as a whole* is committing to supporting them. I don't have any objection to someone from the community publishing them, as long as it is made clear who the actual owner is. I'm not sure how easy it is to make that distinction if we publish them through infra jobs, so that may mean some outside process. I also don't think there would be any problem in building images on our infrastructure for our own gate jobs, as long as they are just for testing and we don't push those to any other registries. I'm raising the issue here to get some more input into how to proceed. Do other people think this concern is overblown? Can we mitigate the risk by communicating through metadata for the images? Should we stick to publishing build instructions (Dockerfiles, or whatever) instead of binary images? Are there other options I haven't mentioned? Doug __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [tc][infra][release][security][stable][kolla][loci][tripleo][docker][kubernetes] do we want to be publishing binary container images?
For starters, I want to emphasize that fresh set of dockerhub images was one of most requested features from Kolla on this summit and few other features more or less requires readily-available docker registry. Features like full release upgrade gates. This will have numerous benefits for users that doesn't have resources to put sophisticated CI/staging env, which, I'm willing to bet, is still quite significant user base. If we do it correctly (and we will do it correctly), images we'll going to push will go through series of gates which we have in Kolla (and will have more). So when you pull image, you know that it was successfully deployed within scenerios available in our gates, maybe even upgrade and increase scenerio coverage later? That is a huge benefit for actual users. On 15 May 2017 at 10:34, Doug Hellmann wrote: > Last week at the Forum we had a couple of discussions about > collaboration between the various teams building or consuming > container images. One topic that came up was deciding how to publish > images from the various teams to docker hub or other container > registries. While the technical bits seem easy enough to work out, > there is still the question of precedence and whether it's a good > idea to do so at all. > > In the past, we have refrained from publishing binary packages in > other formats such as debs and RPMs. (We did publish debs way back > in the beginning, for testing IIRC, but switched away from them to > sdists to be more inclusive.) Since then, we have said it is the > responsibility of downstream consumers to build production packages, > either as distributors or as a deployer that is rolling their own. > We do package sdists for python libraries, push some JavaScript to > the NPM registries, and have tarballs of those and a bunch of other > artifacts that we build out of our release tools. But none of those > is declared as "production ready," and so the community is not > sending the signal that we are responsible for maintaining them in > the context of production deployments, beyond continuing to produce > new releases when there are bugs. So for us that would mean something really hacky and bad. We are community driven not company driven project. We don't have Red Hat or Canonical teams behind us (we have contributors, but that's different). > Container images introduce some extra complexity, over the basic > operating system style packages mentioned above. Due to the way > they are constructed, they are likely to include content we don't > produce ourselves (either in the form of base layers or via including > build tools or other things needed when assembling the full image). > That extra content means there would need to be more tracking of > upstream issues (bugs, CVEs, etc.) to ensure the images are updated > as needed. We can do this by building daily, which was the plan in fact. If we build every day you have at most 24hrs old packages, CVEs and things like that on non-openstack packages are still maintained by distro maintainers. > Given our security and stable team resources, I'm not entirely > comfortable with us publishing these images, and giving the appearance > that the community *as a whole* is committing to supporting them. > I don't have any objection to someone from the community publishing > them, as long as it is made clear who the actual owner is. I'm not > sure how easy it is to make that distinction if we publish them > through infra jobs, so that may mean some outside process. I also > don't think there would be any problem in building images on our > infrastructure for our own gate jobs, as long as they are just for > testing and we don't push those to any other registries. Today we use Kolla account for that and I'm more than happy to keep it this way. We license our code with ASL which gives no guarantees. Containers will be licensed this way too, so they're available as-is and "production readiness" should be decided by everyone who runs it. That being said what we *can* promise is that our containers passed through more or less rigorous gates and that's more than most of packages/self-built containers ever do. I think that value would be appreciated by small to mid companies that just want to work with openstack and don't have means to spare teams/resources for CI. > I'm raising the issue here to get some more input into how to > proceed. Do other people think this concern is overblown? Can we > mitigate the risk by communicating through metadata for the images? > Should we stick to publishing build instructions (Dockerfiles, or > whatever) instead of binary images? Are there other options I haven't > mentioned? Today we do publish build instructions, that's what Kolla is. We also publish built containers already, just we do it manually on release today. If we decide to block it, I assume we should stop doing that too? That will hurt users who uses this piece of Kolla, and I'd hate to hurt our users:( > Doug > >
Re: [openstack-dev] [tc][infra][release][security][stable][kolla][loci][tripleo][docker][kubernetes] do we want to be publishing binary container images?
Sorry for the top post, Michal, Can you please clarify a couple of things: 1) Can folks install just one or two services for their specific scenario? 2) Can the container images from kolla be run on bare docker daemon? 3) Can someone take the kolla container images from say dockerhub and use it without the Kolla framework? Thanks, Dims On Mon, May 15, 2017 at 1:52 PM, Michał Jastrzębski wrote: > For starters, I want to emphasize that fresh set of dockerhub images > was one of most requested features from Kolla on this summit and few > other features more or less requires readily-available docker > registry. Features like full release upgrade gates. > > This will have numerous benefits for users that doesn't have resources > to put sophisticated CI/staging env, which, I'm willing to bet, is > still quite significant user base. If we do it correctly (and we will > do it correctly), images we'll going to push will go through series of > gates which we have in Kolla (and will have more). So when you pull > image, you know that it was successfully deployed within scenerios > available in our gates, maybe even upgrade and increase scenerio > coverage later? That is a huge benefit for actual users. > > On 15 May 2017 at 10:34, Doug Hellmann wrote: >> Last week at the Forum we had a couple of discussions about >> collaboration between the various teams building or consuming >> container images. One topic that came up was deciding how to publish >> images from the various teams to docker hub or other container >> registries. While the technical bits seem easy enough to work out, >> there is still the question of precedence and whether it's a good >> idea to do so at all. >> >> In the past, we have refrained from publishing binary packages in >> other formats such as debs and RPMs. (We did publish debs way back >> in the beginning, for testing IIRC, but switched away from them to >> sdists to be more inclusive.) Since then, we have said it is the >> responsibility of downstream consumers to build production packages, >> either as distributors or as a deployer that is rolling their own. >> We do package sdists for python libraries, push some JavaScript to >> the NPM registries, and have tarballs of those and a bunch of other >> artifacts that we build out of our release tools. But none of those >> is declared as "production ready," and so the community is not >> sending the signal that we are responsible for maintaining them in >> the context of production deployments, beyond continuing to produce >> new releases when there are bugs. > > So for us that would mean something really hacky and bad. We are > community driven not company driven project. We don't have Red Hat or > Canonical teams behind us (we have contributors, but that's > different). > >> Container images introduce some extra complexity, over the basic >> operating system style packages mentioned above. Due to the way >> they are constructed, they are likely to include content we don't >> produce ourselves (either in the form of base layers or via including >> build tools or other things needed when assembling the full image). >> That extra content means there would need to be more tracking of >> upstream issues (bugs, CVEs, etc.) to ensure the images are updated >> as needed. > > We can do this by building daily, which was the plan in fact. If we > build every day you have at most 24hrs old packages, CVEs and things > like that on non-openstack packages are still maintained by distro > maintainers. > >> Given our security and stable team resources, I'm not entirely >> comfortable with us publishing these images, and giving the appearance >> that the community *as a whole* is committing to supporting them. >> I don't have any objection to someone from the community publishing >> them, as long as it is made clear who the actual owner is. I'm not >> sure how easy it is to make that distinction if we publish them >> through infra jobs, so that may mean some outside process. I also >> don't think there would be any problem in building images on our >> infrastructure for our own gate jobs, as long as they are just for >> testing and we don't push those to any other registries. > > Today we use Kolla account for that and I'm more than happy to keep it > this way. We license our code with ASL which gives no guarantees. > Containers will be licensed this way too, so they're available as-is > and "production readiness" should be decided by everyone who runs it. > That being said what we *can* promise is that our containers passed > through more or less rigorous gates and that's more than most of > packages/self-built containers ever do. I think that value would be > appreciated by small to mid companies that just want to work with > openstack and don't have means to spare teams/resources for CI. > >> I'm raising the issue here to get some more input into how to >> proceed. Do other people think this concern is overblown? Can we >> mitigate the risk by commu
Re: [openstack-dev] [tc][infra][release][security][stable][kolla][loci][tripleo][docker][kubernetes] do we want to be publishing binary container images?
On 05/15/2017 01:52 PM, Michał Jastrzębski wrote: > For starters, I want to emphasize that fresh set of dockerhub images > was one of most requested features from Kolla on this summit and few > other features more or less requires readily-available docker > registry. Features like full release upgrade gates. > > This will have numerous benefits for users that doesn't have resources > to put sophisticated CI/staging env, which, I'm willing to bet, is > still quite significant user base. If we do it correctly (and we will > do it correctly), images we'll going to push will go through series of > gates which we have in Kolla (and will have more). So when you pull > image, you know that it was successfully deployed within scenerios > available in our gates, maybe even upgrade and increase scenerio > coverage later? That is a huge benefit for actual users. That concerns me quite a bit. Given the nature of the patch story on containers (which is a rebuild), I really feel like users should have their own build / CI pipeline locally to be deploying this way. Making that easy for them to do, is great, but skipping that required local infrastructure puts them in a bad position should something go wrong. I do get that many folks want that, but I think it builds in a set of expectations that it's not possible to actually meet from an upstream perspective. > On 15 May 2017 at 10:34, Doug Hellmann wrote: >> Last week at the Forum we had a couple of discussions about >> collaboration between the various teams building or consuming >> container images. One topic that came up was deciding how to publish >> images from the various teams to docker hub or other container >> registries. While the technical bits seem easy enough to work out, >> there is still the question of precedence and whether it's a good >> idea to do so at all. >> >> In the past, we have refrained from publishing binary packages in >> other formats such as debs and RPMs. (We did publish debs way back >> in the beginning, for testing IIRC, but switched away from them to >> sdists to be more inclusive.) Since then, we have said it is the >> responsibility of downstream consumers to build production packages, >> either as distributors or as a deployer that is rolling their own. >> We do package sdists for python libraries, push some JavaScript to >> the NPM registries, and have tarballs of those and a bunch of other >> artifacts that we build out of our release tools. But none of those >> is declared as "production ready," and so the community is not >> sending the signal that we are responsible for maintaining them in >> the context of production deployments, beyond continuing to produce >> new releases when there are bugs. > > So for us that would mean something really hacky and bad. We are > community driven not company driven project. We don't have Red Hat or > Canonical teams behind us (we have contributors, but that's > different). > >> Container images introduce some extra complexity, over the basic >> operating system style packages mentioned above. Due to the way >> they are constructed, they are likely to include content we don't >> produce ourselves (either in the form of base layers or via including >> build tools or other things needed when assembling the full image). >> That extra content means there would need to be more tracking of >> upstream issues (bugs, CVEs, etc.) to ensure the images are updated >> as needed. > > We can do this by building daily, which was the plan in fact. If we > build every day you have at most 24hrs old packages, CVEs and things > like that on non-openstack packages are still maintained by distro > maintainers. There have been many instances where 24 hours wasn't good enough as embargoes end up pretty weird in terms of when things hit mirrors. It also assumes that when a CVE hits some other part of the gate or infrastructure isn't wedged so that it's not possible to build new packages. Or the capacity demands happen during a feature freeze, with tons of delay in there. There are many single points of failure in this process. >> Given our security and stable team resources, I'm not entirely >> comfortable with us publishing these images, and giving the appearance >> that the community *as a whole* is committing to supporting them. >> I don't have any objection to someone from the community publishing >> them, as long as it is made clear who the actual owner is. I'm not >> sure how easy it is to make that distinction if we publish them >> through infra jobs, so that may mean some outside process. I also >> don't think there would be any problem in building images on our >> infrastructure for our own gate jobs, as long as they are just for >> testing and we don't push those to any other registries. > > Today we use Kolla account for that and I'm more than happy to keep it > this way. We license our code with ASL which gives no guarantees. > Containers will be licensed this way too, so they're available as-i
Re: [openstack-dev] [tc][infra][release][security][stable][kolla][loci][tripleo][docker][kubernetes] do we want to be publishing binary container images?
On 15 May 2017 at 11:19, Davanum Srinivas wrote: > Sorry for the top post, Michal, Can you please clarify a couple of things: > > 1) Can folks install just one or two services for their specific scenario? Yes, that's more of a kolla-ansible feature and require a little bit of ansible know-how, but entirely possible. Kolla-k8s is built to allow maximum flexibility in that space. > 2) Can the container images from kolla be run on bare docker daemon? Yes, but they need to either override our default CMD (kolla_start) or provide ENVs requred by it, not a huge deal > 3) Can someone take the kolla container images from say dockerhub and > use it without the Kolla framework? Yes, there is no such thing as kolla framework really. Our images follow stable ABI and they can be deployed by any deploy mechanism that will follow it. We have several users who wrote their own deploy mechanism from scratch. Containers are just blobs with binaries in it. Little things that we add are kolla_start script to allow our config file management and some custom startup scripts for things like mariadb to help with bootstrapping, both are entirely optional. > > Thanks, > Dims > > On Mon, May 15, 2017 at 1:52 PM, Michał Jastrzębski wrote: >> For starters, I want to emphasize that fresh set of dockerhub images >> was one of most requested features from Kolla on this summit and few >> other features more or less requires readily-available docker >> registry. Features like full release upgrade gates. >> >> This will have numerous benefits for users that doesn't have resources >> to put sophisticated CI/staging env, which, I'm willing to bet, is >> still quite significant user base. If we do it correctly (and we will >> do it correctly), images we'll going to push will go through series of >> gates which we have in Kolla (and will have more). So when you pull >> image, you know that it was successfully deployed within scenerios >> available in our gates, maybe even upgrade and increase scenerio >> coverage later? That is a huge benefit for actual users. >> >> On 15 May 2017 at 10:34, Doug Hellmann wrote: >>> Last week at the Forum we had a couple of discussions about >>> collaboration between the various teams building or consuming >>> container images. One topic that came up was deciding how to publish >>> images from the various teams to docker hub or other container >>> registries. While the technical bits seem easy enough to work out, >>> there is still the question of precedence and whether it's a good >>> idea to do so at all. >>> >>> In the past, we have refrained from publishing binary packages in >>> other formats such as debs and RPMs. (We did publish debs way back >>> in the beginning, for testing IIRC, but switched away from them to >>> sdists to be more inclusive.) Since then, we have said it is the >>> responsibility of downstream consumers to build production packages, >>> either as distributors or as a deployer that is rolling their own. >>> We do package sdists for python libraries, push some JavaScript to >>> the NPM registries, and have tarballs of those and a bunch of other >>> artifacts that we build out of our release tools. But none of those >>> is declared as "production ready," and so the community is not >>> sending the signal that we are responsible for maintaining them in >>> the context of production deployments, beyond continuing to produce >>> new releases when there are bugs. >> >> So for us that would mean something really hacky and bad. We are >> community driven not company driven project. We don't have Red Hat or >> Canonical teams behind us (we have contributors, but that's >> different). >> >>> Container images introduce some extra complexity, over the basic >>> operating system style packages mentioned above. Due to the way >>> they are constructed, they are likely to include content we don't >>> produce ourselves (either in the form of base layers or via including >>> build tools or other things needed when assembling the full image). >>> That extra content means there would need to be more tracking of >>> upstream issues (bugs, CVEs, etc.) to ensure the images are updated >>> as needed. >> >> We can do this by building daily, which was the plan in fact. If we >> build every day you have at most 24hrs old packages, CVEs and things >> like that on non-openstack packages are still maintained by distro >> maintainers. >> >>> Given our security and stable team resources, I'm not entirely >>> comfortable with us publishing these images, and giving the appearance >>> that the community *as a whole* is committing to supporting them. >>> I don't have any objection to someone from the community publishing >>> them, as long as it is made clear who the actual owner is. I'm not >>> sure how easy it is to make that distinction if we publish them >>> through infra jobs, so that may mean some outside process. I also >>> don't think there would be any problem in building images on our >>> infrastructure
Re: [openstack-dev] [tc][infra][release][security][stable][kolla][loci][tripleo][docker][kubernetes] do we want to be publishing binary container images?
On 15 May 2017 at 11:47, Sean Dague wrote: > On 05/15/2017 01:52 PM, Michał Jastrzębski wrote: >> For starters, I want to emphasize that fresh set of dockerhub images >> was one of most requested features from Kolla on this summit and few >> other features more or less requires readily-available docker >> registry. Features like full release upgrade gates. >> >> This will have numerous benefits for users that doesn't have resources >> to put sophisticated CI/staging env, which, I'm willing to bet, is >> still quite significant user base. If we do it correctly (and we will >> do it correctly), images we'll going to push will go through series of >> gates which we have in Kolla (and will have more). So when you pull >> image, you know that it was successfully deployed within scenerios >> available in our gates, maybe even upgrade and increase scenerio >> coverage later? That is a huge benefit for actual users. > > That concerns me quite a bit. Given the nature of the patch story on > containers (which is a rebuild), I really feel like users should have > their own build / CI pipeline locally to be deploying this way. Making > that easy for them to do, is great, but skipping that required local > infrastructure puts them in a bad position should something go wrong. I totally agree they should. Even if they do, it's still would be additive to gating that we run, so it's even better. > I do get that many folks want that, but I think it builds in a set of > expectations that it's not possible to actually meet from an upstream > perspective. > >> On 15 May 2017 at 10:34, Doug Hellmann wrote: >>> Last week at the Forum we had a couple of discussions about >>> collaboration between the various teams building or consuming >>> container images. One topic that came up was deciding how to publish >>> images from the various teams to docker hub or other container >>> registries. While the technical bits seem easy enough to work out, >>> there is still the question of precedence and whether it's a good >>> idea to do so at all. >>> >>> In the past, we have refrained from publishing binary packages in >>> other formats such as debs and RPMs. (We did publish debs way back >>> in the beginning, for testing IIRC, but switched away from them to >>> sdists to be more inclusive.) Since then, we have said it is the >>> responsibility of downstream consumers to build production packages, >>> either as distributors or as a deployer that is rolling their own. >>> We do package sdists for python libraries, push some JavaScript to >>> the NPM registries, and have tarballs of those and a bunch of other >>> artifacts that we build out of our release tools. But none of those >>> is declared as "production ready," and so the community is not >>> sending the signal that we are responsible for maintaining them in >>> the context of production deployments, beyond continuing to produce >>> new releases when there are bugs. >> >> So for us that would mean something really hacky and bad. We are >> community driven not company driven project. We don't have Red Hat or >> Canonical teams behind us (we have contributors, but that's >> different). >> >>> Container images introduce some extra complexity, over the basic >>> operating system style packages mentioned above. Due to the way >>> they are constructed, they are likely to include content we don't >>> produce ourselves (either in the form of base layers or via including >>> build tools or other things needed when assembling the full image). >>> That extra content means there would need to be more tracking of >>> upstream issues (bugs, CVEs, etc.) to ensure the images are updated >>> as needed. >> >> We can do this by building daily, which was the plan in fact. If we >> build every day you have at most 24hrs old packages, CVEs and things >> like that on non-openstack packages are still maintained by distro >> maintainers. > > There have been many instances where 24 hours wasn't good enough as > embargoes end up pretty weird in terms of when things hit mirrors. It > also assumes that when a CVE hits some other part of the gate or > infrastructure isn't wedged so that it's not possible to build new > packages. Or the capacity demands happen during a feature freeze, with > tons of delay in there. There are many single points of failure in this > process. > >>> Given our security and stable team resources, I'm not entirely >>> comfortable with us publishing these images, and giving the appearance >>> that the community *as a whole* is committing to supporting them. >>> I don't have any objection to someone from the community publishing >>> them, as long as it is made clear who the actual owner is. I'm not >>> sure how easy it is to make that distinction if we publish them >>> through infra jobs, so that may mean some outside process. I also >>> don't think there would be any problem in building images on our >>> infrastructure for our own gate jobs, as long as they are just for >>> testing an
Re: [openstack-dev] [tc][infra][release][security][stable][kolla][loci][tripleo][docker][kubernetes] do we want to be publishing binary container images?
Excerpts from Michał Jastrzębski's message of 2017-05-15 10:52:12 -0700: > For starters, I want to emphasize that fresh set of dockerhub images > was one of most requested features from Kolla on this summit and few > other features more or less requires readily-available docker > registry. Features like full release upgrade gates. > > This will have numerous benefits for users that doesn't have resources > to put sophisticated CI/staging env, which, I'm willing to bet, is > still quite significant user base. If we do it correctly (and we will > do it correctly), images we'll going to push will go through series of > gates which we have in Kolla (and will have more). So when you pull > image, you know that it was successfully deployed within scenerios > available in our gates, maybe even upgrade and increase scenerio > coverage later? That is a huge benefit for actual users. I have no doubt that consumers of the images would like us to keep creating them. We had lots of discussions last week about resource constraints and sustainable practices, though, and this strikes me as an area where we're deviating from our history in a way that will require more maintenance work upstream. > On 15 May 2017 at 10:34, Doug Hellmann wrote: > > Last week at the Forum we had a couple of discussions about > > collaboration between the various teams building or consuming > > container images. One topic that came up was deciding how to publish > > images from the various teams to docker hub or other container > > registries. While the technical bits seem easy enough to work out, > > there is still the question of precedence and whether it's a good > > idea to do so at all. > > > > In the past, we have refrained from publishing binary packages in > > other formats such as debs and RPMs. (We did publish debs way back > > in the beginning, for testing IIRC, but switched away from them to > > sdists to be more inclusive.) Since then, we have said it is the > > responsibility of downstream consumers to build production packages, > > either as distributors or as a deployer that is rolling their own. > > We do package sdists for python libraries, push some JavaScript to > > the NPM registries, and have tarballs of those and a bunch of other > > artifacts that we build out of our release tools. But none of those > > is declared as "production ready," and so the community is not > > sending the signal that we are responsible for maintaining them in > > the context of production deployments, beyond continuing to produce > > new releases when there are bugs. > > So for us that would mean something really hacky and bad. We are > community driven not company driven project. We don't have Red Hat or > Canonical teams behind us (we have contributors, but that's > different). Although I work at Red Hat, I want to make sure it's clear that my objection is purely related to community concerns. For this conversation, I'm wearing my upstream TC and Release team hats. > > Container images introduce some extra complexity, over the basic > > operating system style packages mentioned above. Due to the way > > they are constructed, they are likely to include content we don't > > produce ourselves (either in the form of base layers or via including > > build tools or other things needed when assembling the full image). > > That extra content means there would need to be more tracking of > > upstream issues (bugs, CVEs, etc.) to ensure the images are updated > > as needed. > > We can do this by building daily, which was the plan in fact. If we > build every day you have at most 24hrs old packages, CVEs and things > like that on non-openstack packages are still maintained by distro > maintainers. A daily build job introduces new questions about how big the images are and how many of them we keep, but let's focus on whether the change in policy is something we want to adopt before we consider those questions. > > Given our security and stable team resources, I'm not entirely > > comfortable with us publishing these images, and giving the appearance > > that the community *as a whole* is committing to supporting them. > > I don't have any objection to someone from the community publishing > > them, as long as it is made clear who the actual owner is. I'm not > > sure how easy it is to make that distinction if we publish them > > through infra jobs, so that may mean some outside process. I also > > don't think there would be any problem in building images on our > > infrastructure for our own gate jobs, as long as they are just for > > testing and we don't push those to any other registries. > > Today we use Kolla account for that and I'm more than happy to keep it > this way. We license our code with ASL which gives no guarantees. > Containers will be licensed this way too, so they're available as-is > and "production readiness" should be decided by everyone who runs it. > That being said what we *can* promise is that our containers passed > t
Re: [openstack-dev] [tc][infra][release][security][stable][kolla][loci][tripleo][docker][kubernetes] do we want to be publishing binary container images?
On 15 May 2017 at 12:12, Doug Hellmann wrote: > Excerpts from Michał Jastrzębski's message of 2017-05-15 10:52:12 -0700: >> For starters, I want to emphasize that fresh set of dockerhub images >> was one of most requested features from Kolla on this summit and few >> other features more or less requires readily-available docker >> registry. Features like full release upgrade gates. >> >> This will have numerous benefits for users that doesn't have resources >> to put sophisticated CI/staging env, which, I'm willing to bet, is >> still quite significant user base. If we do it correctly (and we will >> do it correctly), images we'll going to push will go through series of >> gates which we have in Kolla (and will have more). So when you pull >> image, you know that it was successfully deployed within scenerios >> available in our gates, maybe even upgrade and increase scenerio >> coverage later? That is a huge benefit for actual users. > > I have no doubt that consumers of the images would like us to keep > creating them. We had lots of discussions last week about resource > constraints and sustainable practices, though, and this strikes me > as an area where we're deviating from our history in a way that > will require more maintenance work upstream. > >> On 15 May 2017 at 10:34, Doug Hellmann wrote: >> > Last week at the Forum we had a couple of discussions about >> > collaboration between the various teams building or consuming >> > container images. One topic that came up was deciding how to publish >> > images from the various teams to docker hub or other container >> > registries. While the technical bits seem easy enough to work out, >> > there is still the question of precedence and whether it's a good >> > idea to do so at all. >> > >> > In the past, we have refrained from publishing binary packages in >> > other formats such as debs and RPMs. (We did publish debs way back >> > in the beginning, for testing IIRC, but switched away from them to >> > sdists to be more inclusive.) Since then, we have said it is the >> > responsibility of downstream consumers to build production packages, >> > either as distributors or as a deployer that is rolling their own. >> > We do package sdists for python libraries, push some JavaScript to >> > the NPM registries, and have tarballs of those and a bunch of other >> > artifacts that we build out of our release tools. But none of those >> > is declared as "production ready," and so the community is not >> > sending the signal that we are responsible for maintaining them in >> > the context of production deployments, beyond continuing to produce >> > new releases when there are bugs. >> >> So for us that would mean something really hacky and bad. We are >> community driven not company driven project. We don't have Red Hat or >> Canonical teams behind us (we have contributors, but that's >> different). > > Although I work at Red Hat, I want to make sure it's clear that my > objection is purely related to community concerns. For this > conversation, I'm wearing my upstream TC and Release team hats. > >> > Container images introduce some extra complexity, over the basic >> > operating system style packages mentioned above. Due to the way >> > they are constructed, they are likely to include content we don't >> > produce ourselves (either in the form of base layers or via including >> > build tools or other things needed when assembling the full image). >> > That extra content means there would need to be more tracking of >> > upstream issues (bugs, CVEs, etc.) to ensure the images are updated >> > as needed. >> >> We can do this by building daily, which was the plan in fact. If we >> build every day you have at most 24hrs old packages, CVEs and things >> like that on non-openstack packages are still maintained by distro >> maintainers. > > A daily build job introduces new questions about how big the images > are and how many of them we keep, but let's focus on whether the > change in policy is something we want to adopt before we consider > those questions. http://tarballs.openstack.org/kolla/images/ we are already doing this for last few months. Only difference is that it's hacky and we want something that's not hacky. Let's separate resource constrains for now please, because from current standpoint all the resources we need is a single vm that's gonna run 1hr every day and some uplink megabytes (probably less than 1gig every day as Docker will cache a lot). If that's an issue, we can work on it and limit amount of pushes to just version changes, something we were discussing anyway. > >> > Given our security and stable team resources, I'm not entirely >> > comfortable with us publishing these images, and giving the appearance >> > that the community *as a whole* is committing to supporting them. >> > I don't have any objection to someone from the community publishing >> > them, as long as it is made clear who the actual owner is. I'm not >> > sure how easy it is
Re: [openstack-dev] [tc][infra][release][security][stable][kolla][loci][tripleo][docker][kubernetes] do we want to be publishing binary container images?
On 15/05/17 11:49 -0700, Michał Jastrzębski wrote: On 15 May 2017 at 11:19, Davanum Srinivas wrote: Sorry for the top post, Michal, Can you please clarify a couple of things: 1) Can folks install just one or two services for their specific scenario? Yes, that's more of a kolla-ansible feature and require a little bit of ansible know-how, but entirely possible. Kolla-k8s is built to allow maximum flexibility in that space. 2) Can the container images from kolla be run on bare docker daemon? Yes, but they need to either override our default CMD (kolla_start) or provide ENVs requred by it, not a huge deal 3) Can someone take the kolla container images from say dockerhub and use it without the Kolla framework? Yes, there is no such thing as kolla framework really. Our images follow stable ABI and they can be deployed by any deploy mechanism that will follow it. We have several users who wrote their own deploy mechanism from scratch. Containers are just blobs with binaries in it. Little things that we add are kolla_start script to allow our config file management and some custom startup scripts for things like mariadb to help with bootstrapping, both are entirely optional. Just as a bonus example, TripleO is currently using kolla images. They used to be vanilla and they are not anymore but only because TripleO depends on puppet being in the image, which has nothing to do with kolla. Flavio -- @flaper87 Flavio Percoco signature.asc Description: PGP signature __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [tc][infra][release][security][stable][kolla][loci][tripleo][docker][kubernetes] do we want to be publishing binary container images?
On 15/05/17 12:32 -0700, Michał Jastrzębski wrote: On 15 May 2017 at 12:12, Doug Hellmann wrote: [huge snip] > I'm raising the issue here to get some more input into how to > proceed. Do other people think this concern is overblown? Can we > mitigate the risk by communicating through metadata for the images? > Should we stick to publishing build instructions (Dockerfiles, or > whatever) instead of binary images? Are there other options I haven't > mentioned? Today we do publish build instructions, that's what Kolla is. We also publish built containers already, just we do it manually on release today. If we decide to block it, I assume we should stop doing that too? That will hurt users who uses this piece of Kolla, and I'd hate to hurt our users:( Well, that's the question. Today we have teams publishing those images themselves, right? And the proposal is to have infra do it? That change could be construed to imply that there is more of a relationship with the images and the rest of the community (remember, folks outside of the main community activities do not always make the same distinctions we do about teams). So, before we go ahead with that, I want to make sure that we all have a chance to discuss the policy change and its implications. Infra as vm running with infra, but team to publish it can be Kolla team. I assume we'll be responsible to keep these images healthy... I think this is the gist of the concern and I'd like us to focus on it. As someone that used to consume these images from kolla's dockerhub account directly, I can confirm they are useful. However, I do share Doug's concern and the impact this may have on the community. From a release perspective, as Doug mentioned, we've avoided releasing projects in any kind of built form. This was also one of the concerns I raised when working on the proposal to support other programming languages. The problem of releasing built images goes beyond the infrastructure requirements. It's the message and the guarantees implied with the built product itself that are the concern here. And I tend to agree with Doug that this might be a problem for us as a community. Unfortunately, putting your name, Michal, as contact point is not enough. Kolla is not the only project producing container images and we need to be consistent in the way we release these images. Nothing prevents people for building their own images and uploading them to dockerhub. Having this as part of the OpenStack's pipeline is a problem. Flavio P.S: note this goes against my container(ish) interests but it's a community-wide problem. -- @flaper87 Flavio Percoco signature.asc Description: PGP signature __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [tc][infra][release][security][stable][kolla][loci][tripleo][docker][kubernetes] do we want to be publishing binary container images?
Flavio, Forgive the top post – outlook ftw. I understand the concerns raised in this thread. It is unclear if this thread is the feeling of two TC members or enough TC members care deeply about this issue to permanently limit OpenStack big tent projects’ ability to generate container images in various external artifact storage systems. The point of discussion I see effectively raised in this thread is “OpenStack infra will not push images to dockerhub”. I’d like clarification if this is a ruling from the TC, or simply an exploratory discussion. If it is exploratory, it is prudent that OpenStack projects not be blocked by debate on this issue until the TC has made such ruling as to banning the creation of container images via OpenStack infrastructure. Regards -steve -Original Message- From: Flavio Percoco Reply-To: "OpenStack Development Mailing List (not for usage questions)" Date: Monday, May 15, 2017 at 7:00 PM To: "OpenStack Development Mailing List (not for usage questions)" Subject: Re: [openstack-dev] [tc][infra][release][security][stable][kolla][loci][tripleo][docker][kubernetes] do we want to be publishing binary container images? On 15/05/17 12:32 -0700, Michał Jastrzębski wrote: >On 15 May 2017 at 12:12, Doug Hellmann wrote: [huge snip] >>> > I'm raising the issue here to get some more input into how to >>> > proceed. Do other people think this concern is overblown? Can we >>> > mitigate the risk by communicating through metadata for the images? >>> > Should we stick to publishing build instructions (Dockerfiles, or >>> > whatever) instead of binary images? Are there other options I haven't >>> > mentioned? >>> >>> Today we do publish build instructions, that's what Kolla is. We also >>> publish built containers already, just we do it manually on release >>> today. If we decide to block it, I assume we should stop doing that >>> too? That will hurt users who uses this piece of Kolla, and I'd hate >>> to hurt our users:( >> >> Well, that's the question. Today we have teams publishing those >> images themselves, right? And the proposal is to have infra do it? >> That change could be construed to imply that there is more of a >> relationship with the images and the rest of the community (remember, >> folks outside of the main community activities do not always make >> the same distinctions we do about teams). So, before we go ahead >> with that, I want to make sure that we all have a chance to discuss >> the policy change and its implications. > >Infra as vm running with infra, but team to publish it can be Kolla >team. I assume we'll be responsible to keep these images healthy... I think this is the gist of the concern and I'd like us to focus on it. As someone that used to consume these images from kolla's dockerhub account directly, I can confirm they are useful. However, I do share Doug's concern and the impact this may have on the community. From a release perspective, as Doug mentioned, we've avoided releasing projects in any kind of built form. This was also one of the concerns I raised when working on the proposal to support other programming languages. The problem of releasing built images goes beyond the infrastructure requirements. It's the message and the guarantees implied with the built product itself that are the concern here. And I tend to agree with Doug that this might be a problem for us as a community. Unfortunately, putting your name, Michal, as contact point is not enough. Kolla is not the only project producing container images and we need to be consistent in the way we release these images. Nothing prevents people for building their own images and uploading them to dockerhub. Having this as part of the OpenStack's pipeline is a problem. Flavio P.S: note this goes against my container(ish) interests but it's a community-wide problem. -- @flaper87 Flavio Percoco __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [tc][infra][release][security][stable][kolla][loci][tripleo][docker][kubernetes] do we want to be publishing binary container images?
On Monday, 15 May 2017 21:12:16 CEST Doug Hellmann wrote: > Excerpts from Michał Jastrzębski's message of 2017-05-15 10:52:12 -0700: > > > On 15 May 2017 at 10:34, Doug Hellmann wrote: > > > I'm raising the issue here to get some more input into how to > > > proceed. Do other people think this concern is overblown? Can we > > > mitigate the risk by communicating through metadata for the images? > > > Should we stick to publishing build instructions (Dockerfiles, or > > > whatever) instead of binary images? Are there other options I haven't > > > mentioned? > > > > Today we do publish build instructions, that's what Kolla is. We also > > publish built containers already, just we do it manually on release > > today. If we decide to block it, I assume we should stop doing that > > too? That will hurt users who uses this piece of Kolla, and I'd hate > > to hurt our users:( > > Well, that's the question. Today we have teams publishing those > images themselves, right? And the proposal is to have infra do it? > That change could be construed to imply that there is more of a > relationship with the images and the rest of the community (remember, > folks outside of the main community activities do not always make > the same distinctions we do about teams). So, before we go ahead > with that, I want to make sure that we all have a chance to discuss > the policy change and its implications. Sorry for hijacking the thread, but we have a similar scenario for example in Sahara. It is about full VM images containing Hadoop/Spark/other_big_data stuff, and not containers, but it's looks really the same. So far ready-made images have been published under http://sahara-files.mirantis.com/images/upstream/, but we are looking to have them hosted on openstack.org, just like other artifacts. We asked about this few days ago on openstack-infra@, but no answer so far (the Summit didn't help): http://lists.openstack.org/pipermail/openstack-infra/2017-April/005312.html I think that the answer to the question raised in this thread is definitely going to be relevant for our use case. Ciao -- Luigi __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [tc][infra][release][security][stable][kolla][loci][tripleo][docker][kubernetes] do we want to be publishing binary container images?
Why drag TC into this discussion Steven? If the TC has something to say, it will be in the form of a resolution with topic "formal-vote". So please Stop! Thanks, Dims On Tue, May 16, 2017 at 12:22 AM, Steven Dake (stdake) wrote: > Flavio, > > Forgive the top post – outlook ftw. > > I understand the concerns raised in this thread. It is unclear if this > thread is the feeling of two TC members or enough TC members care deeply > about this issue to permanently limit OpenStack big tent projects’ ability to > generate container images in various external artifact storage systems. The > point of discussion I see effectively raised in this thread is “OpenStack > infra will not push images to dockerhub”. > > I’d like clarification if this is a ruling from the TC, or simply an > exploratory discussion. > > If it is exploratory, it is prudent that OpenStack projects not be blocked by > debate on this issue until the TC has made such ruling as to banning the > creation of container images via OpenStack infrastructure. > > Regards > -steve > > -Original Message- > From: Flavio Percoco > Reply-To: "OpenStack Development Mailing List (not for usage questions)" > > Date: Monday, May 15, 2017 at 7:00 PM > To: "OpenStack Development Mailing List (not for usage questions)" > > Subject: Re: [openstack-dev] > [tc][infra][release][security][stable][kolla][loci][tripleo][docker][kubernetes] > do we want to be publishing binary container images? > > On 15/05/17 12:32 -0700, Michał Jastrzębski wrote: > >On 15 May 2017 at 12:12, Doug Hellmann wrote: > > [huge snip] > > >>> > I'm raising the issue here to get some more input into how to > >>> > proceed. Do other people think this concern is overblown? Can we > >>> > mitigate the risk by communicating through metadata for the images? > >>> > Should we stick to publishing build instructions (Dockerfiles, or > >>> > whatever) instead of binary images? Are there other options I > haven't > >>> > mentioned? > >>> > >>> Today we do publish build instructions, that's what Kolla is. We also > >>> publish built containers already, just we do it manually on release > >>> today. If we decide to block it, I assume we should stop doing that > >>> too? That will hurt users who uses this piece of Kolla, and I'd hate > >>> to hurt our users:( > >> > >> Well, that's the question. Today we have teams publishing those > >> images themselves, right? And the proposal is to have infra do it? > >> That change could be construed to imply that there is more of a > >> relationship with the images and the rest of the community (remember, > >> folks outside of the main community activities do not always make > >> the same distinctions we do about teams). So, before we go ahead > >> with that, I want to make sure that we all have a chance to discuss > >> the policy change and its implications. > > > >Infra as vm running with infra, but team to publish it can be Kolla > >team. I assume we'll be responsible to keep these images healthy... > > I think this is the gist of the concern and I'd like us to focus on it. > > As someone that used to consume these images from kolla's dockerhub > account > directly, I can confirm they are useful. However, I do share Doug's > concern and > the impact this may have on the community. > > From a release perspective, as Doug mentioned, we've avoided releasing > projects > in any kind of built form. This was also one of the concerns I raised when > working on the proposal to support other programming languages. The > problem of > releasing built images goes beyond the infrastructure requirements. It's > the > message and the guarantees implied with the built product itself that are > the > concern here. And I tend to agree with Doug that this might be a problem > for us > as a community. Unfortunately, putting your name, Michal, as contact > point is > not enough. Kolla is not the only project producing container images and > we need > to be consistent in the way we release these images. > > Nothing prevents people for building their own images and uploading them > to > dockerhub. Having this as part of the OpenStack's pipeline is a problem. > > Flavio > > P.S: note this goes against my container(ish
Re: [openstack-dev] [tc][infra][release][security][stable][kolla][loci][tripleo][docker][kubernetes] do we want to be publishing binary container images?
Flavio Percoco wrote: > From a release perspective, as Doug mentioned, we've avoided releasing > projects > in any kind of built form. This was also one of the concerns I raised when > working on the proposal to support other programming languages. The problem of > releasing built images goes beyond the infrastructure requirements. It's the > message and the guarantees implied with the built product itself that are the > concern here. And I tend to agree with Doug that this might be a problem for > us > as a community. Unfortunately, putting your name, Michal, as contact point is > not enough. Kolla is not the only project producing container images and we > need > to be consistent in the way we release these images. > > Nothing prevents people for building their own images and uploading them to > dockerhub. Having this as part of the OpenStack's pipeline is a problem. I totally subscribe to the concerns around publishing binaries (under any form), and the expectations in terms of security maintenance that it would set on the publisher. At the same time, we need to have images available, for convenience and testing. So what is the best way to achieve that without setting strong security maintenance expectations for the OpenStack community ? We have several options: 1/ Have third-parties publish images It is the current situation. The issue is that the Kolla team (and likely others) would rather automate the process and use OpenStack infrastructure for it. 2/ Have third-parties publish images, but through OpenStack infra This would allow to automate the process, but it would be a bit weird to use common infra resources to publish in a private repo. 3/ Publish transient (per-commit or daily) images A "daily build" (especially if you replace it every day) would set relatively-limited expectations in terms of maintenance. It would end up picking up security updates in upstream layers, even if not immediately. 4/ Publish images and own them Staff release / VMT / stable team in a way that lets us properly own those images and publish them officially. Personally I think (4) is not realistic. I think we could make (3) work, and I prefer it to (2). If all else fails, we should keep (1). -- Thierry Carrez (ttx) signature.asc Description: OpenPGP digital signature __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [tc][infra][release][security][stable][kolla][loci][tripleo][docker][kubernetes] do we want to be publishing binary container images?
On 16/05/17 04:22 +, Steven Dake (stdake) wrote: Flavio, Forgive the top post – outlook ftw. I understand the concerns raised in this thread. It is unclear if this thread is the feeling of two TC members or enough TC members care deeply about this issue to permanently limit OpenStack big tent projects’ ability to generate container images in various external artifact storage systems. The point of discussion I see effectively raised in this thread is “OpenStack infra will not push images to dockerhub”. I’d like clarification if this is a ruling from the TC, or simply an exploratory discussion. If it is exploratory, it is prudent that OpenStack projects not be blocked by debate on this issue until the TC has made such ruling as to banning the creation of container images via OpenStack infrastructure. Hey Steven, It's nothing to do with the TC. It's a release management concern and I just happen to have an opinion on it. :) As Doug mentioned, OpenStack has (almost) never released binaries in any form. This doesn't mean we can't revisit this "rule" but until that happens, the concern stands. Flavio Regards -steve -Original Message- From: Flavio Percoco Reply-To: "OpenStack Development Mailing List (not for usage questions)" Date: Monday, May 15, 2017 at 7:00 PM To: "OpenStack Development Mailing List (not for usage questions)" Subject: Re: [openstack-dev] [tc][infra][release][security][stable][kolla][loci][tripleo][docker][kubernetes] do we want to be publishing binary container images? On 15/05/17 12:32 -0700, Michał Jastrzębski wrote: >On 15 May 2017 at 12:12, Doug Hellmann wrote: [huge snip] >>> > I'm raising the issue here to get some more input into how to >>> > proceed. Do other people think this concern is overblown? Can we >>> > mitigate the risk by communicating through metadata for the images? >>> > Should we stick to publishing build instructions (Dockerfiles, or >>> > whatever) instead of binary images? Are there other options I haven't >>> > mentioned? >>> >>> Today we do publish build instructions, that's what Kolla is. We also >>> publish built containers already, just we do it manually on release >>> today. If we decide to block it, I assume we should stop doing that >>> too? That will hurt users who uses this piece of Kolla, and I'd hate >>> to hurt our users:( >> >> Well, that's the question. Today we have teams publishing those >> images themselves, right? And the proposal is to have infra do it? >> That change could be construed to imply that there is more of a >> relationship with the images and the rest of the community (remember, >> folks outside of the main community activities do not always make >> the same distinctions we do about teams). So, before we go ahead >> with that, I want to make sure that we all have a chance to discuss >> the policy change and its implications. > >Infra as vm running with infra, but team to publish it can be Kolla >team. I assume we'll be responsible to keep these images healthy... I think this is the gist of the concern and I'd like us to focus on it. As someone that used to consume these images from kolla's dockerhub account directly, I can confirm they are useful. However, I do share Doug's concern and the impact this may have on the community. From a release perspective, as Doug mentioned, we've avoided releasing projects in any kind of built form. This was also one of the concerns I raised when working on the proposal to support other programming languages. The problem of releasing built images goes beyond the infrastructure requirements. It's the message and the guarantees implied with the built product itself that are the concern here. And I tend to agree with Doug that this might be a problem for us as a community. Unfortunately, putting your name, Michal, as contact point is not enough. Kolla is not the only project producing container images and we need to be consistent in the way we release these images. Nothing prevents people for building their own images and uploading them to dockerhub. Having this as part of the OpenStack's pipeline is a problem. Flavio P.S: note this goes against my container(ish) interests but it's a community-wide problem. -- @flaper87 Flavio Percoco __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/list
Re: [openstack-dev] [tc][infra][release][security][stable][kolla][loci][tripleo][docker][kubernetes] do we want to be publishing binary container images?
Dims, The [tc] was in the subject tag, and the message was represented as indicating some TC directive and has had several tc members comment on the thread. I did nothing wrong. Regards -steve -Original Message- From: Davanum Srinivas Reply-To: "OpenStack Development Mailing List (not for usage questions)" Date: Tuesday, May 16, 2017 at 4:34 AM To: "OpenStack Development Mailing List (not for usage questions)" Subject: Re: [openstack-dev] [tc][infra][release][security][stable][kolla][loci][tripleo][docker][kubernetes] do we want to be publishing binary container images? Why drag TC into this discussion Steven? If the TC has something to say, it will be in the form of a resolution with topic "formal-vote". So please Stop! Thanks, Dims On Tue, May 16, 2017 at 12:22 AM, Steven Dake (stdake) wrote: > Flavio, > > Forgive the top post – outlook ftw. > > I understand the concerns raised in this thread. It is unclear if this thread is the feeling of two TC members or enough TC members care deeply about this issue to permanently limit OpenStack big tent projects’ ability to generate container images in various external artifact storage systems. The point of discussion I see effectively raised in this thread is “OpenStack infra will not push images to dockerhub”. > > I’d like clarification if this is a ruling from the TC, or simply an exploratory discussion. > > If it is exploratory, it is prudent that OpenStack projects not be blocked by debate on this issue until the TC has made such ruling as to banning the creation of container images via OpenStack infrastructure. > > Regards > -steve > > -Original Message- > From: Flavio Percoco > Reply-To: "OpenStack Development Mailing List (not for usage questions)" > Date: Monday, May 15, 2017 at 7:00 PM > To: "OpenStack Development Mailing List (not for usage questions)" > Subject: Re: [openstack-dev] [tc][infra][release][security][stable][kolla][loci][tripleo][docker][kubernetes] do we want to be publishing binary container images? > > On 15/05/17 12:32 -0700, Michał Jastrzębski wrote: > >On 15 May 2017 at 12:12, Doug Hellmann wrote: > > [huge snip] > > >>> > I'm raising the issue here to get some more input into how to > >>> > proceed. Do other people think this concern is overblown? Can we > >>> > mitigate the risk by communicating through metadata for the images? > >>> > Should we stick to publishing build instructions (Dockerfiles, or > >>> > whatever) instead of binary images? Are there other options I haven't > >>> > mentioned? > >>> > >>> Today we do publish build instructions, that's what Kolla is. We also > >>> publish built containers already, just we do it manually on release > >>> today. If we decide to block it, I assume we should stop doing that > >>> too? That will hurt users who uses this piece of Kolla, and I'd hate > >>> to hurt our users:( > >> > >> Well, that's the question. Today we have teams publishing those > >> images themselves, right? And the proposal is to have infra do it? > >> That change could be construed to imply that there is more of a > >> relationship with the images and the rest of the community (remember, > >> folks outside of the main community activities do not always make > >> the same distinctions we do about teams). So, before we go ahead > >> with that, I want to make sure that we all have a chance to discuss > >> the policy change and its implications. > > > >Infra as vm running with infra, but team to publish it can be Kolla > >team. I assume we'll be responsible to keep these images healthy... > > I think this is the gist of the concern and I'd like us to focus on it. > > As someone that used to consume these images from kolla's dockerhub account > directly, I can confirm they are useful. However, I do share Doug's concern and > the impact this may have on the community. > > From a release perspective, as Doug mentioned, we've avoided releasing projects > in any kind of built form. This was also one of the concerns I raised when > working on the proposal to support other programming languages. The pro
Re: [openstack-dev] [tc][infra][release][security][stable][kolla][loci][tripleo][docker][kubernetes] do we want to be publishing binary container images?
Excerpts from Thierry Carrez's message of 2017-05-16 14:08:07 +0200: > Flavio Percoco wrote: > > From a release perspective, as Doug mentioned, we've avoided releasing > > projects > > in any kind of built form. This was also one of the concerns I raised when > > working on the proposal to support other programming languages. The problem > > of > > releasing built images goes beyond the infrastructure requirements. It's the > > message and the guarantees implied with the built product itself that are > > the > > concern here. And I tend to agree with Doug that this might be a problem > > for us > > as a community. Unfortunately, putting your name, Michal, as contact point > > is > > not enough. Kolla is not the only project producing container images and we > > need > > to be consistent in the way we release these images. > > > > Nothing prevents people for building their own images and uploading them to > > dockerhub. Having this as part of the OpenStack's pipeline is a problem. > > I totally subscribe to the concerns around publishing binaries (under > any form), and the expectations in terms of security maintenance that it > would set on the publisher. At the same time, we need to have images > available, for convenience and testing. So what is the best way to > achieve that without setting strong security maintenance expectations > for the OpenStack community ? We have several options: > > 1/ Have third-parties publish images > It is the current situation. The issue is that the Kolla team (and > likely others) would rather automate the process and use OpenStack > infrastructure for it. > > 2/ Have third-parties publish images, but through OpenStack infra > This would allow to automate the process, but it would be a bit weird to > use common infra resources to publish in a private repo. > > 3/ Publish transient (per-commit or daily) images > A "daily build" (especially if you replace it every day) would set > relatively-limited expectations in terms of maintenance. It would end up > picking up security updates in upstream layers, even if not immediately. > > 4/ Publish images and own them > Staff release / VMT / stable team in a way that lets us properly own > those images and publish them officially. > > Personally I think (4) is not realistic. I think we could make (3) work, > and I prefer it to (2). If all else fails, we should keep (1). > At the forum we talked about putting test images on a "private" repository hosted on openstack.org somewhere. I think that's option 3 from your list? Paul may be able to shed more light on the details of the technology (maybe it's just an Apache-served repo, rather than a full blown instance of Docker's service, for example). Doug __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [tc][infra][release][security][stable][kolla][loci][tripleo][docker][kubernetes] do we want to be publishing binary container images?
Excerpts from Luigi Toscano's message of 2017-05-16 11:50:53 +0200: > On Monday, 15 May 2017 21:12:16 CEST Doug Hellmann wrote: > > Excerpts from Michał Jastrzębski's message of 2017-05-15 10:52:12 -0700: > > > > > On 15 May 2017 at 10:34, Doug Hellmann wrote: > > > > I'm raising the issue here to get some more input into how to > > > > proceed. Do other people think this concern is overblown? Can we > > > > mitigate the risk by communicating through metadata for the images? > > > > Should we stick to publishing build instructions (Dockerfiles, or > > > > whatever) instead of binary images? Are there other options I haven't > > > > mentioned? > > > > > > Today we do publish build instructions, that's what Kolla is. We also > > > publish built containers already, just we do it manually on release > > > today. If we decide to block it, I assume we should stop doing that > > > too? That will hurt users who uses this piece of Kolla, and I'd hate > > > to hurt our users:( > > > > Well, that's the question. Today we have teams publishing those > > images themselves, right? And the proposal is to have infra do it? > > That change could be construed to imply that there is more of a > > relationship with the images and the rest of the community (remember, > > folks outside of the main community activities do not always make > > the same distinctions we do about teams). So, before we go ahead > > with that, I want to make sure that we all have a chance to discuss > > the policy change and its implications. > > Sorry for hijacking the thread, but we have a similar scenario for example in > Sahara. It is about full VM images containing Hadoop/Spark/other_big_data > stuff, and not containers, but it's looks really the same. > So far ready-made images have been published under > http://sahara-files.mirantis.com/images/upstream/, but we are looking to have > them hosted on > openstack.org, just like other artifacts. > > We asked about this few days ago on openstack-infra@, but no answer so far > (the Summit didn't help): > > http://lists.openstack.org/pipermail/openstack-infra/2017-April/005312.html > > I think that the answer to the question raised in this thread is definitely > going to be relevant for our use case. > > Ciao Thanks for raising this. I think the same concerns apply to VM images. Doug __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [tc][infra][release][security][stable][kolla][loci][tripleo][docker][kubernetes] do we want to be publishing binary container images?
On 16/05/17 14:08 +0200, Thierry Carrez wrote: Flavio Percoco wrote: From a release perspective, as Doug mentioned, we've avoided releasing projects in any kind of built form. This was also one of the concerns I raised when working on the proposal to support other programming languages. The problem of releasing built images goes beyond the infrastructure requirements. It's the message and the guarantees implied with the built product itself that are the concern here. And I tend to agree with Doug that this might be a problem for us as a community. Unfortunately, putting your name, Michal, as contact point is not enough. Kolla is not the only project producing container images and we need to be consistent in the way we release these images. Nothing prevents people for building their own images and uploading them to dockerhub. Having this as part of the OpenStack's pipeline is a problem. I totally subscribe to the concerns around publishing binaries (under any form), and the expectations in terms of security maintenance that it would set on the publisher. At the same time, we need to have images available, for convenience and testing. So what is the best way to achieve that without setting strong security maintenance expectations for the OpenStack community ? We have several options: 1/ Have third-parties publish images It is the current situation. The issue is that the Kolla team (and likely others) would rather automate the process and use OpenStack infrastructure for it. 2/ Have third-parties publish images, but through OpenStack infra This would allow to automate the process, but it would be a bit weird to use common infra resources to publish in a private repo. 3/ Publish transient (per-commit or daily) images A "daily build" (especially if you replace it every day) would set relatively-limited expectations in terms of maintenance. It would end up picking up security updates in upstream layers, even if not immediately. 4/ Publish images and own them Staff release / VMT / stable team in a way that lets us properly own those images and publish them officially. Personally I think (4) is not realistic. I think we could make (3) work, and I prefer it to (2). If all else fails, we should keep (1). Agreed #4 is a bit unrealistic. Not sure I understand the difference between #2 and #3. Is it just the cadence? I'd prefer for these builds to have a daily cadence because it sets the expectations w.r.t maintenance right: "These images are daily builds and not certified releases. For stable builds you're better off building it yourself" Flavio -- @flaper87 Flavio Percoco signature.asc Description: PGP signature __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [tc][infra][release][security][stable][kolla][loci][tripleo][docker][kubernetes] do we want to be publishing binary container images?
On 05/16/2017 09:24 AM, Doug Hellmann wrote: > Excerpts from Luigi Toscano's message of 2017-05-16 11:50:53 +0200: >> On Monday, 15 May 2017 21:12:16 CEST Doug Hellmann wrote: >>> Excerpts from Michał Jastrzębski's message of 2017-05-15 10:52:12 -0700: >>> On 15 May 2017 at 10:34, Doug Hellmann wrote: > I'm raising the issue here to get some more input into how to > proceed. Do other people think this concern is overblown? Can we > mitigate the risk by communicating through metadata for the images? > Should we stick to publishing build instructions (Dockerfiles, or > whatever) instead of binary images? Are there other options I haven't > mentioned? Today we do publish build instructions, that's what Kolla is. We also publish built containers already, just we do it manually on release today. If we decide to block it, I assume we should stop doing that too? That will hurt users who uses this piece of Kolla, and I'd hate to hurt our users:( >>> >>> Well, that's the question. Today we have teams publishing those >>> images themselves, right? And the proposal is to have infra do it? >>> That change could be construed to imply that there is more of a >>> relationship with the images and the rest of the community (remember, >>> folks outside of the main community activities do not always make >>> the same distinctions we do about teams). So, before we go ahead >>> with that, I want to make sure that we all have a chance to discuss >>> the policy change and its implications. >> >> Sorry for hijacking the thread, but we have a similar scenario for example >> in >> Sahara. It is about full VM images containing Hadoop/Spark/other_big_data >> stuff, and not containers, but it's looks really the same. >> So far ready-made images have been published under >> http://sahara-files.mirantis.com/images/upstream/, but we are looking to >> have them hosted on >> openstack.org, just like other artifacts. >> >> We asked about this few days ago on openstack-infra@, but no answer so far >> (the Summit didn't help): >> >> http://lists.openstack.org/pipermail/openstack-infra/2017-April/005312.html >> >> I think that the answer to the question raised in this thread is definitely >> going to be relevant for our use case. >> >> Ciao > > Thanks for raising this. I think the same concerns apply to VM images. Agreed. -Sean -- Sean Dague http://dague.net __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [tc][infra][release][security][stable][kolla][loci][tripleo][docker][kubernetes] do we want to be publishing binary container images?
Steve, We should not always ask "if this is a ruling from the TC", the default is that it's a discussion/exploration. If it is a "ruling", it won't be on a ML thread. Thanks, Dims On Tue, May 16, 2017 at 9:22 AM, Steven Dake (stdake) wrote: > Dims, > > The [tc] was in the subject tag, and the message was represented as > indicating some TC directive and has had several tc members comment on the > thread. I did nothing wrong. > > Regards > -steve > > > -Original Message- > From: Davanum Srinivas > Reply-To: "OpenStack Development Mailing List (not for usage questions)" > > Date: Tuesday, May 16, 2017 at 4:34 AM > To: "OpenStack Development Mailing List (not for usage questions)" > > Subject: Re: [openstack-dev] > [tc][infra][release][security][stable][kolla][loci][tripleo][docker][kubernetes] > do we want to be publishing binary container images? > > Why drag TC into this discussion Steven? If the TC has something to > say, it will be in the form of a resolution with topic "formal-vote". > So please Stop! > > Thanks, > Dims > > On Tue, May 16, 2017 at 12:22 AM, Steven Dake (stdake) > wrote: > > Flavio, > > > > Forgive the top post – outlook ftw. > > > > I understand the concerns raised in this thread. It is unclear if this > thread is the feeling of two TC members or enough TC members care deeply > about this issue to permanently limit OpenStack big tent projects’ ability to > generate container images in various external artifact storage systems. The > point of discussion I see effectively raised in this thread is “OpenStack > infra will not push images to dockerhub”. > > > > I’d like clarification if this is a ruling from the TC, or simply an > exploratory discussion. > > > > If it is exploratory, it is prudent that OpenStack projects not be > blocked by debate on this issue until the TC has made such ruling as to > banning the creation of container images via OpenStack infrastructure. > > > > Regards > > -steve > > > > -Original Message- > > From: Flavio Percoco > > Reply-To: "OpenStack Development Mailing List (not for usage > questions)" > > Date: Monday, May 15, 2017 at 7:00 PM > > To: "OpenStack Development Mailing List (not for usage questions)" > > > Subject: Re: [openstack-dev] > [tc][infra][release][security][stable][kolla][loci][tripleo][docker][kubernetes] > do we want to be publishing binary container images? > > > > On 15/05/17 12:32 -0700, Michał Jastrzębski wrote: > > >On 15 May 2017 at 12:12, Doug Hellmann > wrote: > > > > [huge snip] > > > > >>> > I'm raising the issue here to get some more input into how to > > >>> > proceed. Do other people think this concern is overblown? Can > we > > >>> > mitigate the risk by communicating through metadata for the > images? > > >>> > Should we stick to publishing build instructions > (Dockerfiles, or > > >>> > whatever) instead of binary images? Are there other options I > haven't > > >>> > mentioned? > > >>> > > >>> Today we do publish build instructions, that's what Kolla is. > We also > > >>> publish built containers already, just we do it manually on > release > > >>> today. If we decide to block it, I assume we should stop doing > that > > >>> too? That will hurt users who uses this piece of Kolla, and I'd > hate > > >>> to hurt our users:( > > >> > > >> Well, that's the question. Today we have teams publishing those > > >> images themselves, right? And the proposal is to have infra do > it? > > >> That change could be construed to imply that there is more of a > > >> relationship with the images and the rest of the community > (remember, > > >> folks outside of the main community activities do not always make > > >> the same distinctions we do about teams). So, before we go ahead > > >> with that, I want to make sure that we all have a chance to > discuss > > >> the policy change and its implications. > > > > > >Infra as vm running w
Re: [openstack-dev] [tc][infra][release][security][stable][kolla][loci][tripleo][docker][kubernetes] do we want to be publishing binary container images?
This is one of those areas that was shared understanding for a long time, and seems less "shared" now that we've grown and added new projects to the community. I intended to prepare a governance resolution *after* having some public discussion, so that we can restore that common understanding through documentation. I didn't prepare the resolution as a first step, because if the consensus is that we've changed our collective minds about whether publishing binary artifacts is a good idea then the wording of the resolution needs to reflect that. Doug Excerpts from Davanum Srinivas (dims)'s message of 2017-05-16 09:25:56 -0400: > Steve, > > We should not always ask "if this is a ruling from the TC", the > default is that it's a discussion/exploration. If it is a "ruling", it > won't be on a ML thread. > > Thanks, > Dims > > On Tue, May 16, 2017 at 9:22 AM, Steven Dake (stdake) > wrote: > > Dims, > > > > The [tc] was in the subject tag, and the message was represented as > > indicating some TC directive and has had several tc members comment on the > > thread. I did nothing wrong. > > > > Regards > > -steve > > > > > > -Original Message- > > From: Davanum Srinivas > > Reply-To: "OpenStack Development Mailing List (not for usage questions)" > > > > Date: Tuesday, May 16, 2017 at 4:34 AM > > To: "OpenStack Development Mailing List (not for usage questions)" > > > > Subject: Re: [openstack-dev] > > [tc][infra][release][security][stable][kolla][loci][tripleo][docker][kubernetes] > > do we want to be publishing binary container images? > > > > Why drag TC into this discussion Steven? If the TC has something to > > say, it will be in the form of a resolution with topic "formal-vote". > > So please Stop! > > > > Thanks, > > Dims > > > > On Tue, May 16, 2017 at 12:22 AM, Steven Dake (stdake) > > wrote: > > > Flavio, > > > > > > Forgive the top post – outlook ftw. > > > > > > I understand the concerns raised in this thread. It is unclear if > > this thread is the feeling of two TC members or enough TC members care > > deeply about this issue to permanently limit OpenStack big tent projects’ > > ability to generate container images in various external artifact storage > > systems. The point of discussion I see effectively raised in this thread > > is “OpenStack infra will not push images to dockerhub”. > > > > > > I’d like clarification if this is a ruling from the TC, or simply an > > exploratory discussion. > > > > > > If it is exploratory, it is prudent that OpenStack projects not be > > blocked by debate on this issue until the TC has made such ruling as to > > banning the creation of container images via OpenStack infrastructure. > > > > > > Regards > > > -steve > > > > > > -Original Message- > > > From: Flavio Percoco > > > Reply-To: "OpenStack Development Mailing List (not for usage > > questions)" > > > Date: Monday, May 15, 2017 at 7:00 PM > > > To: "OpenStack Development Mailing List (not for usage questions)" > > > > > Subject: Re: [openstack-dev] > > [tc][infra][release][security][stable][kolla][loci][tripleo][docker][kubernetes] > > do we want to be publishing binary container images? > > > > > > On 15/05/17 12:32 -0700, Michał Jastrzębski wrote: > > > >On 15 May 2017 at 12:12, Doug Hellmann > > wrote: > > > > > > [huge snip] > > > > > > >>> > I'm raising the issue here to get some more input into how > > to > > > >>> > proceed. Do other people think this concern is overblown? > > Can we > > > >>> > mitigate the risk by communicating through metadata for the > > images? > > > >>> > Should we stick to publishing build instructions > > (Dockerfiles, or > > > >>> > whatever) instead of binary images? Are there other options > > I haven't > > > >>> > mentioned? > > > >>> > > > >>> Today we do publish build instructions, that's what Kolla is. > > We also > > > >>> publish built containers already, j
Re: [openstack-dev] [tc][infra][release][security][stable][kolla][loci][tripleo][docker][kubernetes] do we want to be publishing binary container images?
Excerpts from Flavio Percoco's message of 2017-05-15 21:50:23 -0400: > On 15/05/17 11:49 -0700, Michał Jastrzębski wrote: > >On 15 May 2017 at 11:19, Davanum Srinivas wrote: > >> Sorry for the top post, Michal, Can you please clarify a couple of things: > >> > >> 1) Can folks install just one or two services for their specific scenario? > > > >Yes, that's more of a kolla-ansible feature and require a little bit > >of ansible know-how, but entirely possible. Kolla-k8s is built to > >allow maximum flexibility in that space. > > > >> 2) Can the container images from kolla be run on bare docker daemon? > > > >Yes, but they need to either override our default CMD (kolla_start) or > >provide ENVs requred by it, not a huge deal > > > >> 3) Can someone take the kolla container images from say dockerhub and > >> use it without the Kolla framework? > > > >Yes, there is no such thing as kolla framework really. Our images > >follow stable ABI and they can be deployed by any deploy mechanism > >that will follow it. We have several users who wrote their own deploy > >mechanism from scratch. > > > >Containers are just blobs with binaries in it. Little things that we > >add are kolla_start script to allow our config file management and > >some custom startup scripts for things like mariadb to help with > >bootstrapping, both are entirely optional. > > Just as a bonus example, TripleO is currently using kolla images. They used to > be vanilla and they are not anymore but only because TripleO depends on puppet > being in the image, which has nothing to do with kolla. > > Flavio > When you say "using kolla images," what do you mean? In upstream CI tests? On contributors' dev/test systems? Production deployments? Doug __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [tc][infra][release][security][stable][kolla][loci][tripleo][docker][kubernetes] do we want to be publishing binary container images?
On 16 May 2017 at 06:20, Flavio Percoco wrote: > On 16/05/17 14:08 +0200, Thierry Carrez wrote: >> >> Flavio Percoco wrote: >>> >>> From a release perspective, as Doug mentioned, we've avoided releasing >>> projects >>> in any kind of built form. This was also one of the concerns I raised >>> when >>> working on the proposal to support other programming languages. The >>> problem of >>> releasing built images goes beyond the infrastructure requirements. It's >>> the >>> message and the guarantees implied with the built product itself that are >>> the >>> concern here. And I tend to agree with Doug that this might be a problem >>> for us >>> as a community. Unfortunately, putting your name, Michal, as contact >>> point is >>> not enough. Kolla is not the only project producing container images and >>> we need >>> to be consistent in the way we release these images. >>> >>> Nothing prevents people for building their own images and uploading them >>> to >>> dockerhub. Having this as part of the OpenStack's pipeline is a problem. >> >> >> I totally subscribe to the concerns around publishing binaries (under >> any form), and the expectations in terms of security maintenance that it >> would set on the publisher. At the same time, we need to have images >> available, for convenience and testing. So what is the best way to >> achieve that without setting strong security maintenance expectations >> for the OpenStack community ? We have several options: >> >> 1/ Have third-parties publish images >> It is the current situation. The issue is that the Kolla team (and >> likely others) would rather automate the process and use OpenStack >> infrastructure for it. >> >> 2/ Have third-parties publish images, but through OpenStack infra >> This would allow to automate the process, but it would be a bit weird to >> use common infra resources to publish in a private repo. >> >> 3/ Publish transient (per-commit or daily) images >> A "daily build" (especially if you replace it every day) would set >> relatively-limited expectations in terms of maintenance. It would end up >> picking up security updates in upstream layers, even if not immediately. >> >> 4/ Publish images and own them >> Staff release / VMT / stable team in a way that lets us properly own >> those images and publish them officially. >> >> Personally I think (4) is not realistic. I think we could make (3) work, >> and I prefer it to (2). If all else fails, we should keep (1). > > > Agreed #4 is a bit unrealistic. > > Not sure I understand the difference between #2 and #3. Is it just the > cadence? > > I'd prefer for these builds to have a daily cadence because it sets the > expectations w.r.t maintenance right: "These images are daily builds and not > certified releases. For stable builds you're better off building it > yourself" And daily builds are exactly what I wanted in the first place:) We probably will keep publishing release packages too, but we can be so called 3rd party. I also agree [4] is completely unrealistic and I would be against putting such heavy burden of responsibility on any community, including Kolla. While daily cadence will send message that it's not stable, truth will be that it will be more stable than what people would normally build locally (again, it passes more gates), but I'm totally fine in not saying that and let people decide how they want to use it. So, can we move on with implementation? Thanks! Michal > > Flavio > > -- > @flaper87 > Flavio Percoco > > __ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [tc][infra][release][security][stable][kolla][loci][tripleo][docker][kubernetes] do we want to be publishing binary container images?
On 16 May 2017 at 06:22, Doug Hellmann wrote: > Excerpts from Thierry Carrez's message of 2017-05-16 14:08:07 +0200: >> Flavio Percoco wrote: >> > From a release perspective, as Doug mentioned, we've avoided releasing >> > projects >> > in any kind of built form. This was also one of the concerns I raised when >> > working on the proposal to support other programming languages. The >> > problem of >> > releasing built images goes beyond the infrastructure requirements. It's >> > the >> > message and the guarantees implied with the built product itself that are >> > the >> > concern here. And I tend to agree with Doug that this might be a problem >> > for us >> > as a community. Unfortunately, putting your name, Michal, as contact point >> > is >> > not enough. Kolla is not the only project producing container images and >> > we need >> > to be consistent in the way we release these images. >> > >> > Nothing prevents people for building their own images and uploading them to >> > dockerhub. Having this as part of the OpenStack's pipeline is a problem. >> >> I totally subscribe to the concerns around publishing binaries (under >> any form), and the expectations in terms of security maintenance that it >> would set on the publisher. At the same time, we need to have images >> available, for convenience and testing. So what is the best way to >> achieve that without setting strong security maintenance expectations >> for the OpenStack community ? We have several options: >> >> 1/ Have third-parties publish images >> It is the current situation. The issue is that the Kolla team (and >> likely others) would rather automate the process and use OpenStack >> infrastructure for it. >> >> 2/ Have third-parties publish images, but through OpenStack infra >> This would allow to automate the process, but it would be a bit weird to >> use common infra resources to publish in a private repo. >> >> 3/ Publish transient (per-commit or daily) images >> A "daily build" (especially if you replace it every day) would set >> relatively-limited expectations in terms of maintenance. It would end up >> picking up security updates in upstream layers, even if not immediately. >> >> 4/ Publish images and own them >> Staff release / VMT / stable team in a way that lets us properly own >> those images and publish them officially. >> >> Personally I think (4) is not realistic. I think we could make (3) work, >> and I prefer it to (2). If all else fails, we should keep (1). >> > > At the forum we talked about putting test images on a "private" > repository hosted on openstack.org somewhere. I think that's option > 3 from your list? > > Paul may be able to shed more light on the details of the technology > (maybe it's just an Apache-served repo, rather than a full blown > instance of Docker's service, for example). > > Doug > > __ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [tc][infra][release][security][stable][kolla][loci][tripleo][docker][kubernetes] do we want to be publishing binary container images?
On 16 May 2017 at 06:22, Doug Hellmann wrote: > Excerpts from Thierry Carrez's message of 2017-05-16 14:08:07 +0200: >> Flavio Percoco wrote: >> > From a release perspective, as Doug mentioned, we've avoided releasing >> > projects >> > in any kind of built form. This was also one of the concerns I raised when >> > working on the proposal to support other programming languages. The >> > problem of >> > releasing built images goes beyond the infrastructure requirements. It's >> > the >> > message and the guarantees implied with the built product itself that are >> > the >> > concern here. And I tend to agree with Doug that this might be a problem >> > for us >> > as a community. Unfortunately, putting your name, Michal, as contact point >> > is >> > not enough. Kolla is not the only project producing container images and >> > we need >> > to be consistent in the way we release these images. >> > >> > Nothing prevents people for building their own images and uploading them to >> > dockerhub. Having this as part of the OpenStack's pipeline is a problem. >> >> I totally subscribe to the concerns around publishing binaries (under >> any form), and the expectations in terms of security maintenance that it >> would set on the publisher. At the same time, we need to have images >> available, for convenience and testing. So what is the best way to >> achieve that without setting strong security maintenance expectations >> for the OpenStack community ? We have several options: >> >> 1/ Have third-parties publish images >> It is the current situation. The issue is that the Kolla team (and >> likely others) would rather automate the process and use OpenStack >> infrastructure for it. >> >> 2/ Have third-parties publish images, but through OpenStack infra >> This would allow to automate the process, but it would be a bit weird to >> use common infra resources to publish in a private repo. >> >> 3/ Publish transient (per-commit or daily) images >> A "daily build" (especially if you replace it every day) would set >> relatively-limited expectations in terms of maintenance. It would end up >> picking up security updates in upstream layers, even if not immediately. >> >> 4/ Publish images and own them >> Staff release / VMT / stable team in a way that lets us properly own >> those images and publish them officially. >> >> Personally I think (4) is not realistic. I think we could make (3) work, >> and I prefer it to (2). If all else fails, we should keep (1). >> > > At the forum we talked about putting test images on a "private" > repository hosted on openstack.org somewhere. I think that's option > 3 from your list? > > Paul may be able to shed more light on the details of the technology > (maybe it's just an Apache-served repo, rather than a full blown > instance of Docker's service, for example). Issue with that is 1. Apache served is harder to use because we want to follow docker API and we'd have to reimplement it 2. Running registry is single command 3. If we host in in infra, in case someone actually uses it (there will be people like that), that will eat up lot of network traffic potentially 4. With local caching of images (working already) in nodepools we loose complexity of mirroring registries across nodepools So bottom line, having dockerhub/quay.io is simply easier. > Doug > > __ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [tc][infra][release][security][stable][kolla][loci][tripleo][docker][kubernetes] do we want to be publishing binary container images?
I would like to bring up a subject that hasn't really been discussed in this thread yet, forgive me if I missed an email mentioning this. What I personally would like to see is a publishing infrastructure to allow pushing built images to an internal infra mirror/repo/registry for consumption of internal infra jobs (deployment tools like kolla-ansible and openstack-ansible). The images built from infra mirrors with security turned off are perfect for testing internally to infra. If you build images properly in infra, then you will have an image that is not security checked (no gpg verification of packages) and completely unverifiable. These are absolutely not images we want to push to DockerHub/quay for obvious reasons. Security and verification being chief among them. They are absolutely not images that should ever be run in production and are only suited for testing. These are the only types of images that can come out of infra. Thanks, SamYaple On Tue, May 16, 2017 at 1:57 PM, Michał Jastrzębski wrote: > On 16 May 2017 at 06:22, Doug Hellmann wrote: > > Excerpts from Thierry Carrez's message of 2017-05-16 14:08:07 +0200: > >> Flavio Percoco wrote: > >> > From a release perspective, as Doug mentioned, we've avoided > releasing projects > >> > in any kind of built form. This was also one of the concerns I raised > when > >> > working on the proposal to support other programming languages. The > problem of > >> > releasing built images goes beyond the infrastructure requirements. > It's the > >> > message and the guarantees implied with the built product itself that > are the > >> > concern here. And I tend to agree with Doug that this might be a > problem for us > >> > as a community. Unfortunately, putting your name, Michal, as contact > point is > >> > not enough. Kolla is not the only project producing container images > and we need > >> > to be consistent in the way we release these images. > >> > > >> > Nothing prevents people for building their own images and uploading > them to > >> > dockerhub. Having this as part of the OpenStack's pipeline is a > problem. > >> > >> I totally subscribe to the concerns around publishing binaries (under > >> any form), and the expectations in terms of security maintenance that it > >> would set on the publisher. At the same time, we need to have images > >> available, for convenience and testing. So what is the best way to > >> achieve that without setting strong security maintenance expectations > >> for the OpenStack community ? We have several options: > >> > >> 1/ Have third-parties publish images > >> It is the current situation. The issue is that the Kolla team (and > >> likely others) would rather automate the process and use OpenStack > >> infrastructure for it. > >> > >> 2/ Have third-parties publish images, but through OpenStack infra > >> This would allow to automate the process, but it would be a bit weird to > >> use common infra resources to publish in a private repo. > >> > >> 3/ Publish transient (per-commit or daily) images > >> A "daily build" (especially if you replace it every day) would set > >> relatively-limited expectations in terms of maintenance. It would end up > >> picking up security updates in upstream layers, even if not immediately. > >> > >> 4/ Publish images and own them > >> Staff release / VMT / stable team in a way that lets us properly own > >> those images and publish them officially. > >> > >> Personally I think (4) is not realistic. I think we could make (3) work, > >> and I prefer it to (2). If all else fails, we should keep (1). > >> > > > > At the forum we talked about putting test images on a "private" > > repository hosted on openstack.org somewhere. I think that's option > > 3 from your list? > > > > Paul may be able to shed more light on the details of the technology > > (maybe it's just an Apache-served repo, rather than a full blown > > instance of Docker's service, for example). > > Issue with that is > > 1. Apache served is harder to use because we want to follow docker API > and we'd have to reimplement it > 2. Running registry is single command > 3. If we host in in infra, in case someone actually uses it (there > will be people like that), that will eat up lot of network traffic > potentially > 4. With local caching of images (working already) in nodepools we > loose complexity of mirroring registries across nodepools > > So bottom line, having dockerhub/quay.io is simply easier. > > > Doug > > > > > __ > > OpenStack Development Mailing List (not for usage questions) > > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject: > unsubscribe > > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > __ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe >
Re: [openstack-dev] [tc][infra][release][security][stable][kolla][loci][tripleo][docker][kubernetes] do we want to be publishing binary container images?
On 16/05/17 09:45 -0400, Doug Hellmann wrote: Excerpts from Flavio Percoco's message of 2017-05-15 21:50:23 -0400: On 15/05/17 11:49 -0700, Michał Jastrzębski wrote: >On 15 May 2017 at 11:19, Davanum Srinivas wrote: >> Sorry for the top post, Michal, Can you please clarify a couple of things: >> >> 1) Can folks install just one or two services for their specific scenario? > >Yes, that's more of a kolla-ansible feature and require a little bit >of ansible know-how, but entirely possible. Kolla-k8s is built to >allow maximum flexibility in that space. > >> 2) Can the container images from kolla be run on bare docker daemon? > >Yes, but they need to either override our default CMD (kolla_start) or >provide ENVs requred by it, not a huge deal > >> 3) Can someone take the kolla container images from say dockerhub and >> use it without the Kolla framework? > >Yes, there is no such thing as kolla framework really. Our images >follow stable ABI and they can be deployed by any deploy mechanism >that will follow it. We have several users who wrote their own deploy >mechanism from scratch. > >Containers are just blobs with binaries in it. Little things that we >add are kolla_start script to allow our config file management and >some custom startup scripts for things like mariadb to help with >bootstrapping, both are entirely optional. Just as a bonus example, TripleO is currently using kolla images. They used to be vanilla and they are not anymore but only because TripleO depends on puppet being in the image, which has nothing to do with kolla. Flavio When you say "using kolla images," what do you mean? In upstream CI tests? On contributors' dev/test systems? Production deployments? All of them. Note that TripleO now builds its own "kolla images" (it uses the kolla Dockerfiles and kolla-build) because the dependency of puppet. When I said, TripleO uses kolla images was intended to answer Dims question on whether these images (or Dockerfiles) can be consumed by other projects. Flavio -- @flaper87 Flavio Percoco signature.asc Description: PGP signature __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [tc][infra][release][security][stable][kolla][loci][tripleo][docker][kubernetes] do we want to be publishing binary container images?
On May 15, 2017, at 9:00 PM, Flavio Percoco wrote: > [huge snip] Thank you! We don’t need 50K of repeated text in every response. -- Ed Leafe __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [tc][infra][release][security][stable][kolla][loci][tripleo][docker][kubernetes] do we want to be publishing binary container images?
On 16 May 2017 at 07:11, Sam Yaple wrote: > I would like to bring up a subject that hasn't really been discussed in this > thread yet, forgive me if I missed an email mentioning this. > > What I personally would like to see is a publishing infrastructure to allow > pushing built images to an internal infra mirror/repo/registry for > consumption of internal infra jobs (deployment tools like kolla-ansible and > openstack-ansible). The images built from infra mirrors with security turned > off are perfect for testing internally to infra. > > If you build images properly in infra, then you will have an image that is > not security checked (no gpg verification of packages) and completely > unverifiable. These are absolutely not images we want to push to > DockerHub/quay for obvious reasons. Security and verification being chief > among them. They are absolutely not images that should ever be run in > production and are only suited for testing. These are the only types of > images that can come out of infra. So I guess we need new feature:) since we can test gpg packages... > Thanks, > SamYaple > > On Tue, May 16, 2017 at 1:57 PM, Michał Jastrzębski > wrote: >> >> On 16 May 2017 at 06:22, Doug Hellmann wrote: >> > Excerpts from Thierry Carrez's message of 2017-05-16 14:08:07 +0200: >> >> Flavio Percoco wrote: >> >> > From a release perspective, as Doug mentioned, we've avoided >> >> > releasing projects >> >> > in any kind of built form. This was also one of the concerns I raised >> >> > when >> >> > working on the proposal to support other programming languages. The >> >> > problem of >> >> > releasing built images goes beyond the infrastructure requirements. >> >> > It's the >> >> > message and the guarantees implied with the built product itself that >> >> > are the >> >> > concern here. And I tend to agree with Doug that this might be a >> >> > problem for us >> >> > as a community. Unfortunately, putting your name, Michal, as contact >> >> > point is >> >> > not enough. Kolla is not the only project producing container images >> >> > and we need >> >> > to be consistent in the way we release these images. >> >> > >> >> > Nothing prevents people for building their own images and uploading >> >> > them to >> >> > dockerhub. Having this as part of the OpenStack's pipeline is a >> >> > problem. >> >> >> >> I totally subscribe to the concerns around publishing binaries (under >> >> any form), and the expectations in terms of security maintenance that >> >> it >> >> would set on the publisher. At the same time, we need to have images >> >> available, for convenience and testing. So what is the best way to >> >> achieve that without setting strong security maintenance expectations >> >> for the OpenStack community ? We have several options: >> >> >> >> 1/ Have third-parties publish images >> >> It is the current situation. The issue is that the Kolla team (and >> >> likely others) would rather automate the process and use OpenStack >> >> infrastructure for it. >> >> >> >> 2/ Have third-parties publish images, but through OpenStack infra >> >> This would allow to automate the process, but it would be a bit weird >> >> to >> >> use common infra resources to publish in a private repo. >> >> >> >> 3/ Publish transient (per-commit or daily) images >> >> A "daily build" (especially if you replace it every day) would set >> >> relatively-limited expectations in terms of maintenance. It would end >> >> up >> >> picking up security updates in upstream layers, even if not >> >> immediately. >> >> >> >> 4/ Publish images and own them >> >> Staff release / VMT / stable team in a way that lets us properly own >> >> those images and publish them officially. >> >> >> >> Personally I think (4) is not realistic. I think we could make (3) >> >> work, >> >> and I prefer it to (2). If all else fails, we should keep (1). >> >> >> > >> > At the forum we talked about putting test images on a "private" >> > repository hosted on openstack.org somewhere. I think that's option >> > 3 from your list? >> > >> > Paul may be able to shed more light on the details of the technology >> > (maybe it's just an Apache-served repo, rather than a full blown >> > instance of Docker's service, for example). >> >> Issue with that is >> >> 1. Apache served is harder to use because we want to follow docker API >> and we'd have to reimplement it >> 2. Running registry is single command >> 3. If we host in in infra, in case someone actually uses it (there >> will be people like that), that will eat up lot of network traffic >> potentially >> 4. With local caching of images (working already) in nodepools we >> loose complexity of mirroring registries across nodepools >> >> So bottom line, having dockerhub/quay.io is simply easier. >> >> > Doug >> > >> > >> > __ >> > OpenStack Development Mailing List (not for usage questions) >> > Unsubscribe: >> > openstack-dev-requ...@lists.openstac
Re: [openstack-dev] [tc][infra][release][security][stable][kolla][loci][tripleo][docker][kubernetes] do we want to be publishing binary container images?
Flavio Percoco wrote: > On 16/05/17 14:08 +0200, Thierry Carrez wrote: >> 1/ Have third-parties publish images >> It is the current situation. The issue is that the Kolla team (and >> likely others) would rather automate the process and use OpenStack >> infrastructure for it. >> >> 2/ Have third-parties publish images, but through OpenStack infra >> This would allow to automate the process, but it would be a bit weird to >> use common infra resources to publish in a private repo. >> >> 3/ Publish transient (per-commit or daily) images >> A "daily build" (especially if you replace it every day) would set >> relatively-limited expectations in terms of maintenance. It would end up >> picking up security updates in upstream layers, even if not immediately. >> >> 4/ Publish images and own them >> Staff release / VMT / stable team in a way that lets us properly own >> those images and publish them officially. >> >> Personally I think (4) is not realistic. I think we could make (3) work, >> and I prefer it to (2). If all else fails, we should keep (1). > > Agreed #4 is a bit unrealistic. > > Not sure I understand the difference between #2 and #3. Is it just the > cadence? In #3 the infrastructure ends up publishing to an official "openstack-daily" repository. In #2 the infrastructure job ends up publishing to some "flavios-garage" repository. -- Thierry Carrez (ttx) signature.asc Description: OpenPGP digital signature __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [tc][infra][release][security][stable][kolla][loci][tripleo][docker][kubernetes] do we want to be publishing binary container images?
Michał Jastrzębski wrote: > On 16 May 2017 at 06:20, Flavio Percoco wrote: >> I'd prefer for these builds to have a daily cadence because it sets the >> expectations w.r.t maintenance right: "These images are daily builds and not >> certified releases. For stable builds you're better off building it >> yourself" > > And daily builds are exactly what I wanted in the first place:) We > probably will keep publishing release packages too, but we can be so > called 3rd party. I also agree [4] is completely unrealistic and I > would be against putting such heavy burden of responsibility on any > community, including Kolla. > > While daily cadence will send message that it's not stable, truth will > be that it will be more stable than what people would normally build > locally (again, it passes more gates), but I'm totally fine in not > saying that and let people decide how they want to use it. > > So, can we move on with implementation? I'm just listing options to help frame the discussion. I still think we need a global answer on this (for container images and VMs) so I think it would be great to have a clear TC resolution (picking one of those options) before moving on with implementation. -- Thierry Carrez (ttx) __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [tc][infra][release][security][stable][kolla][loci][tripleo][docker][kubernetes] do we want to be publishing binary container images?
Excerpts from Flavio Percoco's message of 2017-05-16 10:07:52 -0400: > On 16/05/17 09:45 -0400, Doug Hellmann wrote: > >Excerpts from Flavio Percoco's message of 2017-05-15 21:50:23 -0400: > >> On 15/05/17 11:49 -0700, Michał Jastrzębski wrote: > >> >On 15 May 2017 at 11:19, Davanum Srinivas wrote: > >> >> Sorry for the top post, Michal, Can you please clarify a couple of > >> >> things: > >> >> > >> >> 1) Can folks install just one or two services for their specific > >> >> scenario? > >> > > >> >Yes, that's more of a kolla-ansible feature and require a little bit > >> >of ansible know-how, but entirely possible. Kolla-k8s is built to > >> >allow maximum flexibility in that space. > >> > > >> >> 2) Can the container images from kolla be run on bare docker daemon? > >> > > >> >Yes, but they need to either override our default CMD (kolla_start) or > >> >provide ENVs requred by it, not a huge deal > >> > > >> >> 3) Can someone take the kolla container images from say dockerhub and > >> >> use it without the Kolla framework? > >> > > >> >Yes, there is no such thing as kolla framework really. Our images > >> >follow stable ABI and they can be deployed by any deploy mechanism > >> >that will follow it. We have several users who wrote their own deploy > >> >mechanism from scratch. > >> > > >> >Containers are just blobs with binaries in it. Little things that we > >> >add are kolla_start script to allow our config file management and > >> >some custom startup scripts for things like mariadb to help with > >> >bootstrapping, both are entirely optional. > >> > >> Just as a bonus example, TripleO is currently using kolla images. They > >> used to > >> be vanilla and they are not anymore but only because TripleO depends on > >> puppet > >> being in the image, which has nothing to do with kolla. > >> > >> Flavio > >> > > > >When you say "using kolla images," what do you mean? In upstream > >CI tests? On contributors' dev/test systems? Production deployments? > > All of them. Note that TripleO now builds its own "kolla images" (it uses the > kolla Dockerfiles and kolla-build) because the dependency of puppet. When I > said, TripleO uses kolla images was intended to answer Dims question on > whether > these images (or Dockerfiles) can be consumed by other projects. > > Flavio > Ah, OK. So TripleO is using the build instructions for kolla images, but not the binary images being produced today? Doug __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [tc][infra][release][security][stable][kolla][loci][tripleo][docker][kubernetes] do we want to be publishing binary container images?
Excerpts from Michał Jastrzębski's message of 2017-05-16 06:52:12 -0700: > On 16 May 2017 at 06:20, Flavio Percoco wrote: > > On 16/05/17 14:08 +0200, Thierry Carrez wrote: > >> > >> Flavio Percoco wrote: > >>> > >>> From a release perspective, as Doug mentioned, we've avoided releasing > >>> projects > >>> in any kind of built form. This was also one of the concerns I raised > >>> when > >>> working on the proposal to support other programming languages. The > >>> problem of > >>> releasing built images goes beyond the infrastructure requirements. It's > >>> the > >>> message and the guarantees implied with the built product itself that are > >>> the > >>> concern here. And I tend to agree with Doug that this might be a problem > >>> for us > >>> as a community. Unfortunately, putting your name, Michal, as contact > >>> point is > >>> not enough. Kolla is not the only project producing container images and > >>> we need > >>> to be consistent in the way we release these images. > >>> > >>> Nothing prevents people for building their own images and uploading them > >>> to > >>> dockerhub. Having this as part of the OpenStack's pipeline is a problem. > >> > >> > >> I totally subscribe to the concerns around publishing binaries (under > >> any form), and the expectations in terms of security maintenance that it > >> would set on the publisher. At the same time, we need to have images > >> available, for convenience and testing. So what is the best way to > >> achieve that without setting strong security maintenance expectations > >> for the OpenStack community ? We have several options: > >> > >> 1/ Have third-parties publish images > >> It is the current situation. The issue is that the Kolla team (and > >> likely others) would rather automate the process and use OpenStack > >> infrastructure for it. > >> > >> 2/ Have third-parties publish images, but through OpenStack infra > >> This would allow to automate the process, but it would be a bit weird to > >> use common infra resources to publish in a private repo. > >> > >> 3/ Publish transient (per-commit or daily) images > >> A "daily build" (especially if you replace it every day) would set > >> relatively-limited expectations in terms of maintenance. It would end up > >> picking up security updates in upstream layers, even if not immediately. > >> > >> 4/ Publish images and own them > >> Staff release / VMT / stable team in a way that lets us properly own > >> those images and publish them officially. > >> > >> Personally I think (4) is not realistic. I think we could make (3) work, > >> and I prefer it to (2). If all else fails, we should keep (1). > > > > > > Agreed #4 is a bit unrealistic. > > > > Not sure I understand the difference between #2 and #3. Is it just the > > cadence? > > > > I'd prefer for these builds to have a daily cadence because it sets the > > expectations w.r.t maintenance right: "These images are daily builds and not > > certified releases. For stable builds you're better off building it > > yourself" > > And daily builds are exactly what I wanted in the first place:) We > probably will keep publishing release packages too, but we can be so > called 3rd party. I also agree [4] is completely unrealistic and I > would be against putting such heavy burden of responsibility on any > community, including Kolla. > > While daily cadence will send message that it's not stable, truth will > be that it will be more stable than what people would normally build > locally (again, it passes more gates), but I'm totally fine in not > saying that and let people decide how they want to use it. > > So, can we move on with implementation? I don't want the images published to docker hub. Are they still useful to you if they aren't published? Doug > > Thanks! > Michal > > > > > Flavio > > > > -- > > @flaper87 > > Flavio Percoco > > > > __ > > OpenStack Development Mailing List (not for usage questions) > > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > > __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [tc][infra][release][security][stable][kolla][loci][tripleo][docker][kubernetes] do we want to be publishing binary container images?
Excerpts from Sam Yaple's message of 2017-05-16 14:11:18 +: > I would like to bring up a subject that hasn't really been discussed in > this thread yet, forgive me if I missed an email mentioning this. > > What I personally would like to see is a publishing infrastructure to allow > pushing built images to an internal infra mirror/repo/registry for > consumption of internal infra jobs (deployment tools like kolla-ansible and > openstack-ansible). The images built from infra mirrors with security > turned off are perfect for testing internally to infra. > > If you build images properly in infra, then you will have an image that is > not security checked (no gpg verification of packages) and completely > unverifiable. These are absolutely not images we want to push to > DockerHub/quay for obvious reasons. Security and verification being chief > among them. They are absolutely not images that should ever be run in > production and are only suited for testing. These are the only types of > images that can come out of infra. > > Thanks, > SamYaple This sounds like an implementation detail of option 3? I think not signing the images does help indicate that they're not meant to be used in production environments. Is some sort of self-hosted solution a reasonable compromise between building images in test jobs (which I understand makes them take extra time) and publishing images to public registries (which is the thing I object to)? If self-hosting is reasonable, then we can work out which tool to use to do it as a second question. Doug __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [tc][infra][release][security][stable][kolla][loci][tripleo][docker][kubernetes] do we want to be publishing binary container images?
On 16 May 2017 at 08:12, Doug Hellmann wrote: > Excerpts from Michał Jastrzębski's message of 2017-05-16 06:52:12 -0700: >> On 16 May 2017 at 06:20, Flavio Percoco wrote: >> > On 16/05/17 14:08 +0200, Thierry Carrez wrote: >> >> >> >> Flavio Percoco wrote: >> >>> >> >>> From a release perspective, as Doug mentioned, we've avoided releasing >> >>> projects >> >>> in any kind of built form. This was also one of the concerns I raised >> >>> when >> >>> working on the proposal to support other programming languages. The >> >>> problem of >> >>> releasing built images goes beyond the infrastructure requirements. It's >> >>> the >> >>> message and the guarantees implied with the built product itself that are >> >>> the >> >>> concern here. And I tend to agree with Doug that this might be a problem >> >>> for us >> >>> as a community. Unfortunately, putting your name, Michal, as contact >> >>> point is >> >>> not enough. Kolla is not the only project producing container images and >> >>> we need >> >>> to be consistent in the way we release these images. >> >>> >> >>> Nothing prevents people for building their own images and uploading them >> >>> to >> >>> dockerhub. Having this as part of the OpenStack's pipeline is a problem. >> >> >> >> >> >> I totally subscribe to the concerns around publishing binaries (under >> >> any form), and the expectations in terms of security maintenance that it >> >> would set on the publisher. At the same time, we need to have images >> >> available, for convenience and testing. So what is the best way to >> >> achieve that without setting strong security maintenance expectations >> >> for the OpenStack community ? We have several options: >> >> >> >> 1/ Have third-parties publish images >> >> It is the current situation. The issue is that the Kolla team (and >> >> likely others) would rather automate the process and use OpenStack >> >> infrastructure for it. >> >> >> >> 2/ Have third-parties publish images, but through OpenStack infra >> >> This would allow to automate the process, but it would be a bit weird to >> >> use common infra resources to publish in a private repo. >> >> >> >> 3/ Publish transient (per-commit or daily) images >> >> A "daily build" (especially if you replace it every day) would set >> >> relatively-limited expectations in terms of maintenance. It would end up >> >> picking up security updates in upstream layers, even if not immediately. >> >> >> >> 4/ Publish images and own them >> >> Staff release / VMT / stable team in a way that lets us properly own >> >> those images and publish them officially. >> >> >> >> Personally I think (4) is not realistic. I think we could make (3) work, >> >> and I prefer it to (2). If all else fails, we should keep (1). >> > >> > >> > Agreed #4 is a bit unrealistic. >> > >> > Not sure I understand the difference between #2 and #3. Is it just the >> > cadence? >> > >> > I'd prefer for these builds to have a daily cadence because it sets the >> > expectations w.r.t maintenance right: "These images are daily builds and >> > not >> > certified releases. For stable builds you're better off building it >> > yourself" >> >> And daily builds are exactly what I wanted in the first place:) We >> probably will keep publishing release packages too, but we can be so >> called 3rd party. I also agree [4] is completely unrealistic and I >> would be against putting such heavy burden of responsibility on any >> community, including Kolla. >> >> While daily cadence will send message that it's not stable, truth will >> be that it will be more stable than what people would normally build >> locally (again, it passes more gates), but I'm totally fine in not >> saying that and let people decide how they want to use it. >> >> So, can we move on with implementation? > > I don't want the images published to docker hub. Are they still useful > to you if they aren't published? What do you mean? We need images available...whether it's dockerhub, infra-hosted registry or any other way to have them, we need to be able to have images that are available and fresh without building. Dockerhub/quay.io is least problems for infra team/resources. > Doug > >> >> Thanks! >> Michal >> >> > >> > Flavio >> > >> > -- >> > @flaper87 >> > Flavio Percoco >> > >> > __ >> > OpenStack Development Mailing List (not for usage questions) >> > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe >> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >> > >> > > __ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage qu
Re: [openstack-dev] [tc][infra][release][security][stable][kolla][loci][tripleo][docker][kubernetes] do we want to be publishing binary container images?
On Tue, May 16, 2017 at 02:08:07PM +0200, Thierry Carrez wrote: > > I totally subscribe to the concerns around publishing binaries (under > any form), and the expectations in terms of security maintenance that it > would set on the publisher. At the same time, we need to have images > available, for convenience and testing. So what is the best way to > achieve that without setting strong security maintenance expectations > for the OpenStack community ? We have several options: > > 1/ Have third-parties publish images > It is the current situation. The issue is that the Kolla team (and > likely others) would rather automate the process and use OpenStack > infrastructure for it. > > 2/ Have third-parties publish images, but through OpenStack infra > This would allow to automate the process, but it would be a bit weird to > use common infra resources to publish in a private repo. > > 3/ Publish transient (per-commit or daily) images > A "daily build" (especially if you replace it every day) would set > relatively-limited expectations in terms of maintenance. It would end up > picking up security updates in upstream layers, even if not immediately. > I share the concerns around implying support for any of these. But I also think they could be incredibly useful, and if we don't do it, there is even more of a chance of multiple "bad" images being published by others. I agree having an automated daily image published should give a reasonable expectation that there is not long term maintenance for these. > 4/ Publish images and own them > Staff release / VMT / stable team in a way that lets us properly own > those images and publish them officially. > > Personally I think (4) is not realistic. I think we could make (3) work, > and I prefer it to (2). If all else fails, we should keep (1). > > -- > Thierry Carrez (ttx) > __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [tc][infra][release][security][stable][kolla][loci][tripleo][docker][kubernetes] do we want to be publishing binary container images?
On Tue, May 16, 2017 at 11:12 AM, Doug Hellmann wrote: > Excerpts from Flavio Percoco's message of 2017-05-16 10:07:52 -0400: >> On 16/05/17 09:45 -0400, Doug Hellmann wrote: >> >Excerpts from Flavio Percoco's message of 2017-05-15 21:50:23 -0400: >> >> On 15/05/17 11:49 -0700, Michał Jastrzębski wrote: >> >> >On 15 May 2017 at 11:19, Davanum Srinivas wrote: >> >> >> Sorry for the top post, Michal, Can you please clarify a couple of >> >> >> things: >> >> >> >> >> >> 1) Can folks install just one or two services for their specific >> >> >> scenario? >> >> > >> >> >Yes, that's more of a kolla-ansible feature and require a little bit >> >> >of ansible know-how, but entirely possible. Kolla-k8s is built to >> >> >allow maximum flexibility in that space. >> >> > >> >> >> 2) Can the container images from kolla be run on bare docker daemon? >> >> > >> >> >Yes, but they need to either override our default CMD (kolla_start) or >> >> >provide ENVs requred by it, not a huge deal >> >> > >> >> >> 3) Can someone take the kolla container images from say dockerhub and >> >> >> use it without the Kolla framework? >> >> > >> >> >Yes, there is no such thing as kolla framework really. Our images >> >> >follow stable ABI and they can be deployed by any deploy mechanism >> >> >that will follow it. We have several users who wrote their own deploy >> >> >mechanism from scratch. >> >> > >> >> >Containers are just blobs with binaries in it. Little things that we >> >> >add are kolla_start script to allow our config file management and >> >> >some custom startup scripts for things like mariadb to help with >> >> >bootstrapping, both are entirely optional. >> >> >> >> Just as a bonus example, TripleO is currently using kolla images. They >> >> used to >> >> be vanilla and they are not anymore but only because TripleO depends on >> >> puppet >> >> being in the image, which has nothing to do with kolla. >> >> >> >> Flavio >> >> >> > >> >When you say "using kolla images," what do you mean? In upstream >> >CI tests? On contributors' dev/test systems? Production deployments? >> >> All of them. Note that TripleO now builds its own "kolla images" (it uses the >> kolla Dockerfiles and kolla-build) because the dependency of puppet. When I >> said, TripleO uses kolla images was intended to answer Dims question on >> whether >> these images (or Dockerfiles) can be consumed by other projects. >> >> Flavio >> > > Ah, OK. So TripleO is using the build instructions for kolla images, but > not the binary images being produced today? Exactly. We have to add Puppet packaging into the list of things we want in the binary, that's why we don't consume the binary directly. > Doug > > __ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- Emilien Macchi __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [tc][infra][release][security][stable][kolla][loci][tripleo][docker][kubernetes] do we want to be publishing binary container images?
On 16 May 2017 at 08:30, Emilien Macchi wrote: > On Tue, May 16, 2017 at 11:12 AM, Doug Hellmann wrote: >> Excerpts from Flavio Percoco's message of 2017-05-16 10:07:52 -0400: >>> On 16/05/17 09:45 -0400, Doug Hellmann wrote: >>> >Excerpts from Flavio Percoco's message of 2017-05-15 21:50:23 -0400: >>> >> On 15/05/17 11:49 -0700, Michał Jastrzębski wrote: >>> >> >On 15 May 2017 at 11:19, Davanum Srinivas wrote: >>> >> >> Sorry for the top post, Michal, Can you please clarify a couple of >>> >> >> things: >>> >> >> >>> >> >> 1) Can folks install just one or two services for their specific >>> >> >> scenario? >>> >> > >>> >> >Yes, that's more of a kolla-ansible feature and require a little bit >>> >> >of ansible know-how, but entirely possible. Kolla-k8s is built to >>> >> >allow maximum flexibility in that space. >>> >> > >>> >> >> 2) Can the container images from kolla be run on bare docker daemon? >>> >> > >>> >> >Yes, but they need to either override our default CMD (kolla_start) or >>> >> >provide ENVs requred by it, not a huge deal >>> >> > >>> >> >> 3) Can someone take the kolla container images from say dockerhub and >>> >> >> use it without the Kolla framework? >>> >> > >>> >> >Yes, there is no such thing as kolla framework really. Our images >>> >> >follow stable ABI and they can be deployed by any deploy mechanism >>> >> >that will follow it. We have several users who wrote their own deploy >>> >> >mechanism from scratch. >>> >> > >>> >> >Containers are just blobs with binaries in it. Little things that we >>> >> >add are kolla_start script to allow our config file management and >>> >> >some custom startup scripts for things like mariadb to help with >>> >> >bootstrapping, both are entirely optional. >>> >> >>> >> Just as a bonus example, TripleO is currently using kolla images. They >>> >> used to >>> >> be vanilla and they are not anymore but only because TripleO depends on >>> >> puppet >>> >> being in the image, which has nothing to do with kolla. >>> >> >>> >> Flavio >>> >> >>> > >>> >When you say "using kolla images," what do you mean? In upstream >>> >CI tests? On contributors' dev/test systems? Production deployments? >>> >>> All of them. Note that TripleO now builds its own "kolla images" (it uses >>> the >>> kolla Dockerfiles and kolla-build) because the dependency of puppet. When I >>> said, TripleO uses kolla images was intended to answer Dims question on >>> whether >>> these images (or Dockerfiles) can be consumed by other projects. >>> >>> Flavio >>> >> >> Ah, OK. So TripleO is using the build instructions for kolla images, but >> not the binary images being produced today? > > Exactly. We have to add Puppet packaging into the list of things we > want in the binary, that's why we don't consume the binary directly. And frankly, if we get this thing agreed on, I don't see why TripleO couldn't publish their images too. If we build technical infra in Kolla, everyone else can benefit from it. >> Doug >> >> __ >> OpenStack Development Mailing List (not for usage questions) >> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > > > -- > Emilien Macchi > > __ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [tc][infra][release][security][stable][kolla][loci][tripleo][docker][kubernetes] do we want to be publishing binary container images?
Excerpts from Michał Jastrzębski's message of 2017-05-15 10:52:12 -0700: > > Container images introduce some extra complexity, over the basic > > operating system style packages mentioned above. Due to the way > > they are constructed, they are likely to include content we don't > > produce ourselves (either in the form of base layers or via including > > build tools or other things needed when assembling the full image). > > That extra content means there would need to be more tracking of > > upstream issues (bugs, CVEs, etc.) to ensure the images are updated > > as needed. > > We can do this by building daily, which was the plan in fact. If we > build every day you have at most 24hrs old packages, CVEs and things > like that on non-openstack packages are still maintained by distro > maintainers. > What's at stake isn't so much "how do we get the bits to the users" but "how do we only get bits to users that they need". If you build and push daily, do you expect all of your users to also _pull_ daily? Redeploy all their containers? How do you detect that there's new CVE-fixing stuff in a daily build? This is really the realm of distributors that have full-time security teams tracking issues and providing support to paying customers. So I think this is a fine idea, however, it needs to include a commitment for a full-time paid security team who weighs in on every change to the manifest. Otherwise we're just lobbing time bombs into our users' data-centers. __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [tc][infra][release][security][stable][kolla][loci][tripleo][docker][kubernetes] do we want to be publishing binary container images?
So another consideration. Do you think whole rule of "not building binares" should be reconsidered? We are kind of new use case here. We aren't distro but we are packagers (kind of). I don't think putting us on equal footing as Red Hat, Canonical or other companies is correct here. K8s is something we want to work with, and what we are discussing is central to how k8s is used. K8s community creates this culture of "organic packages" built by anyone, most of companies/projects already have semi-official container images and I think expectations on quality of these are well...none? You get what you're given and if you don't agree, there is always way to reproduce this yourself. [Another huge snip] __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [tc][infra][release][security][stable][kolla][loci][tripleo][docker][kubernetes] do we want to be publishing binary container images?
On 16 May 2017 at 09:40, Clint Byrum wrote: > Excerpts from Michał Jastrzębski's message of 2017-05-15 10:52:12 -0700: >> > Container images introduce some extra complexity, over the basic >> > operating system style packages mentioned above. Due to the way >> > they are constructed, they are likely to include content we don't >> > produce ourselves (either in the form of base layers or via including >> > build tools or other things needed when assembling the full image). >> > That extra content means there would need to be more tracking of >> > upstream issues (bugs, CVEs, etc.) to ensure the images are updated >> > as needed. >> >> We can do this by building daily, which was the plan in fact. If we >> build every day you have at most 24hrs old packages, CVEs and things >> like that on non-openstack packages are still maintained by distro >> maintainers. >> > > What's at stake isn't so much "how do we get the bits to the users" but > "how do we only get bits to users that they need". If you build and push > daily, do you expect all of your users to also _pull_ daily? Redeploy > all their containers? How do you detect that there's new CVE-fixing > stuff in a daily build? > > This is really the realm of distributors that have full-time security > teams tracking issues and providing support to paying customers. > > So I think this is a fine idea, however, it needs to include a commitment > for a full-time paid security team who weighs in on every change to > the manifest. Otherwise we're just lobbing time bombs into our users' > data-centers. One thing I struggle with is...well...how does *not having* built containers help with that? If your company have full time security team, they can check our containers prior to deployment. If your company doesn't, then building locally will be subject to same risks as downloading from dockerhub. Difference is, dockerhub containers were tested in our CI to extend that our CI allows. No matter whether or not you have your own security team, local CI, staging env, that will be just a little bit of testing on top of that which you get for free, and I think that's value enough for users to push for this. > __ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [tc][infra][release][security][stable][kolla][loci][tripleo][docker][kubernetes] do we want to be publishing binary container images?
On 2017-05-16 11:17:31 -0400 (-0400), Doug Hellmann wrote: > Excerpts from Sam Yaple's message of 2017-05-16 14:11:18 +: [...] > > If you build images properly in infra, then you will have an image that is > > not security checked (no gpg verification of packages) and completely > > unverifiable. These are absolutely not images we want to push to > > DockerHub/quay for obvious reasons. Security and verification being chief > > among them. They are absolutely not images that should ever be run in > > production and are only suited for testing. These are the only types of > > images that can come out of infra. > > This sounds like an implementation detail of option 3? I think not > signing the images does help indicate that they're not meant to be used > in production environments. [...] I'm pretty sure Sam wasn't talking about whether or not the images which get built are signed, but whether or not the package manager used when building the images vets the distro packages it retrieves (the Ubuntu package mirror we maintain in our CI doesn't have "secure APT" signatures available for its indices so we disable that security measure by default in the CI system to allow us to use those mirrors). Point being, if images are built in the upstream CI with packages from our Ubuntu package mirror then they are (at least at present) not suitable for production use from a security perspective for this particular reason even in absence of the other concerns expressed. -- Jeremy Stanley signature.asc Description: Digital signature __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [tc][infra][release][security][stable][kolla][loci][tripleo][docker][kubernetes] do we want to be publishing binary container images?
On 16 May 2017 at 10:41, Jeremy Stanley wrote: > On 2017-05-16 11:17:31 -0400 (-0400), Doug Hellmann wrote: >> Excerpts from Sam Yaple's message of 2017-05-16 14:11:18 +: > [...] >> > If you build images properly in infra, then you will have an image that is >> > not security checked (no gpg verification of packages) and completely >> > unverifiable. These are absolutely not images we want to push to >> > DockerHub/quay for obvious reasons. Security and verification being chief >> > among them. They are absolutely not images that should ever be run in >> > production and are only suited for testing. These are the only types of >> > images that can come out of infra. >> >> This sounds like an implementation detail of option 3? I think not >> signing the images does help indicate that they're not meant to be used >> in production environments. > [...] > > I'm pretty sure Sam wasn't talking about whether or not the images > which get built are signed, but whether or not the package manager > used when building the images vets the distro packages it retrieves > (the Ubuntu package mirror we maintain in our CI doesn't have > "secure APT" signatures available for its indices so we disable that > security measure by default in the CI system to allow us to use > those mirrors). Point being, if images are built in the upstream CI > with packages from our Ubuntu package mirror then they are (at least > at present) not suitable for production use from a security > perspective for this particular reason even in absence of the other > concerns expressed. > -- > Jeremy Stanley This is valid concern, but also particularly easy to solve. If we decide to use nightly builds (or midday in Hawaii? Any timezone with least traffic would do), we can skip infra mirrors. In fact, that approach would help us in a different sense as well. Since these wouldn't be bound to any particular patchset, we could test it to an extreme, so voting gates for both kolla-ansible and kolla-kubernetes deployment. I was reluctant to have deploy gates voting inside Kolla, but that would allow us to do it. In fact, net uplink consumption from infra would go down, as we won't need to publish tarballs of registry every commit, we'll do it once a day in a most convenient hour. > __ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [tc][infra][release][security][stable][kolla][loci][tripleo][docker][kubernetes] do we want to be publishing binary container images?
Sorry, I'm a bit late to the discussion. Over the time I've maintained openstack clouds, I've seen many times, integration issues pop up between distro packages and openstack packages. The released openstack packages are usually fine. Changes in the distros break kolla builds over time. In the past Kolla's dealt with this by having users show up, say "the *stuff* doesn't build" and then someone on irc scrambles to figure out why it broke. This is not a good place to be in. We even suffer through it ourselves with the gates. An upstream releases something, everything breaks, and then we spend all our time debugging the same problem rather then some of us debugging that and the rest making forward progress. So, we're starting to break up the gate jobs into periodic gates that test known good kolla stuff against newer upstream stuff to see if it works. if it does, it caches it for the regular gate tests for new patch sets. If it fails, then someone has time to look at it without the rest of the gates being broken. As part of this process we also produce known working, up to date, openstack stable release containers, since we need them for upgrade gate testing. So, basically to have proper gates in kolla, we have to do all the work of building up to date stable/trunk containers that are tested with the gate suite of tests. So, the real question is, can we go the last 2 feet and push them to the docker hub rather then doing it manually like we do today and they tend to rot there. There is a fair amount of benefit to some users and very little additional cost to openstack. I see very little reason not to. We should still recommend users build it themselves. But for many use cases, such as testing the waters, an operator might just want the easiest way to see the thing work (pull updated containers from the hub), prove out that its worth doing it with Kolla, then either building their own containers for production, or better yet, paying a Distro for support. We want to make it as easy as possible to try out OpenStack. This is one of the biggest/most reported problems with OpenStack. Saying, step 1, you must always build all the containers yourself is not a part of solving that problem. Thanks, Kevin From: Flavio Percoco [fla...@redhat.com] Sent: Tuesday, May 16, 2017 6:20 AM To: OpenStack Development Mailing List (not for usage questions) Subject: Re: [openstack-dev] [tc][infra][release][security][stable][kolla][loci][tripleo][docker][kubernetes] do we want to be publishing binary container images? On 16/05/17 14:08 +0200, Thierry Carrez wrote: >Flavio Percoco wrote: >> From a release perspective, as Doug mentioned, we've avoided releasing >> projects >> in any kind of built form. This was also one of the concerns I raised when >> working on the proposal to support other programming languages. The problem >> of >> releasing built images goes beyond the infrastructure requirements. It's the >> message and the guarantees implied with the built product itself that are the >> concern here. And I tend to agree with Doug that this might be a problem for >> us >> as a community. Unfortunately, putting your name, Michal, as contact point is >> not enough. Kolla is not the only project producing container images and we >> need >> to be consistent in the way we release these images. >> >> Nothing prevents people for building their own images and uploading them to >> dockerhub. Having this as part of the OpenStack's pipeline is a problem. > >I totally subscribe to the concerns around publishing binaries (under >any form), and the expectations in terms of security maintenance that it >would set on the publisher. At the same time, we need to have images >available, for convenience and testing. So what is the best way to >achieve that without setting strong security maintenance expectations >for the OpenStack community ? We have several options: > >1/ Have third-parties publish images >It is the current situation. The issue is that the Kolla team (and >likely others) would rather automate the process and use OpenStack >infrastructure for it. > >2/ Have third-parties publish images, but through OpenStack infra >This would allow to automate the process, but it would be a bit weird to >use common infra resources to publish in a private repo. > >3/ Publish transient (per-commit or daily) images >A "daily build" (especially if you replace it every day) would set >relatively-limited expectations in terms of maintenance. It would end up >picking up security updates in upstream layers, even if not immediately. > >4/ Publish images and own them >Staff release / VMT / stable team in a way that lets us properly own >th
Re: [openstack-dev] [tc][infra][release][security][stable][kolla][loci][tripleo][docker][kubernetes] do we want to be publishing binary container images?
+1. ironic and trove have the same issues as well. lowering the bar in order to kick the tires will help OpenStack a lot in adoption. From: Sean Dague [s...@dague.net] Sent: Tuesday, May 16, 2017 6:28 AM To: openstack-dev@lists.openstack.org Subject: Re: [openstack-dev] [tc][infra][release][security][stable][kolla][loci][tripleo][docker][kubernetes] do we want to be publishing binary container images? On 05/16/2017 09:24 AM, Doug Hellmann wrote: > Excerpts from Luigi Toscano's message of 2017-05-16 11:50:53 +0200: >> On Monday, 15 May 2017 21:12:16 CEST Doug Hellmann wrote: >>> Excerpts from Michał Jastrzębski's message of 2017-05-15 10:52:12 -0700: >>> >>>> On 15 May 2017 at 10:34, Doug Hellmann wrote: >>>>> I'm raising the issue here to get some more input into how to >>>>> proceed. Do other people think this concern is overblown? Can we >>>>> mitigate the risk by communicating through metadata for the images? >>>>> Should we stick to publishing build instructions (Dockerfiles, or >>>>> whatever) instead of binary images? Are there other options I haven't >>>>> mentioned? >>>> >>>> Today we do publish build instructions, that's what Kolla is. We also >>>> publish built containers already, just we do it manually on release >>>> today. If we decide to block it, I assume we should stop doing that >>>> too? That will hurt users who uses this piece of Kolla, and I'd hate >>>> to hurt our users:( >>> >>> Well, that's the question. Today we have teams publishing those >>> images themselves, right? And the proposal is to have infra do it? >>> That change could be construed to imply that there is more of a >>> relationship with the images and the rest of the community (remember, >>> folks outside of the main community activities do not always make >>> the same distinctions we do about teams). So, before we go ahead >>> with that, I want to make sure that we all have a chance to discuss >>> the policy change and its implications. >> >> Sorry for hijacking the thread, but we have a similar scenario for example in >> Sahara. It is about full VM images containing Hadoop/Spark/other_big_data >> stuff, and not containers, but it's looks really the same. >> So far ready-made images have been published under >> http://sahara-files.mirantis.com/images/upstream/, but we are looking to >> have them hosted on >> openstack.org, just like other artifacts. >> >> We asked about this few days ago on openstack-infra@, but no answer so far >> (the Summit didn't help): >> >> http://lists.openstack.org/pipermail/openstack-infra/2017-April/005312.html >> >> I think that the answer to the question raised in this thread is definitely >> going to be relevant for our use case. >> >> Ciao > > Thanks for raising this. I think the same concerns apply to VM images. Agreed. -Sean -- Sean Dague http://dague.net __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [tc][infra][release][security][stable][kolla][loci][tripleo][docker][kubernetes] do we want to be publishing binary container images?
Excerpts from Michał Jastrzębski's message of 2017-05-16 09:46:19 -0700: > So another consideration. Do you think whole rule of "not building > binares" should be reconsidered? We are kind of new use case here. We > aren't distro but we are packagers (kind of). I don't think putting us > on equal footing as Red Hat, Canonical or other companies is correct > here. > > K8s is something we want to work with, and what we are discussing is > central to how k8s is used. K8s community creates this culture of > "organic packages" built by anyone, most of companies/projects already > have semi-official container images and I think expectations on > quality of these are well...none? You get what you're given and if you > don't agree, there is always way to reproduce this yourself. > > [Another huge snip] > I wanted to have the discussion, but my position for now is that we should continue as we have been and not change the policy. I don't have a problem with any individual or group of individuals publishing their own organic packages. The issue I have is with making sure it is clear those *are* "organic" and not officially supported by the broader community. One way to do that is to say they need to be built somewhere other than on our shared infrastructure. There may be other ways, though, so I'm looking for input on that. Doug __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [tc][infra][release][security][stable][kolla][loci][tripleo][docker][kubernetes] do we want to be publishing binary container images?
Excerpts from Jeremy Stanley's message of 2017-05-16 17:41:28 +: > On 2017-05-16 11:17:31 -0400 (-0400), Doug Hellmann wrote: > > Excerpts from Sam Yaple's message of 2017-05-16 14:11:18 +: > [...] > > > If you build images properly in infra, then you will have an image that is > > > not security checked (no gpg verification of packages) and completely > > > unverifiable. These are absolutely not images we want to push to > > > DockerHub/quay for obvious reasons. Security and verification being chief > > > among them. They are absolutely not images that should ever be run in > > > production and are only suited for testing. These are the only types of > > > images that can come out of infra. > > > > This sounds like an implementation detail of option 3? I think not > > signing the images does help indicate that they're not meant to be used > > in production environments. > [...] > > I'm pretty sure Sam wasn't talking about whether or not the images > which get built are signed, but whether or not the package manager > used when building the images vets the distro packages it retrieves > (the Ubuntu package mirror we maintain in our CI doesn't have > "secure APT" signatures available for its indices so we disable that > security measure by default in the CI system to allow us to use > those mirrors). Point being, if images are built in the upstream CI > with packages from our Ubuntu package mirror then they are (at least > at present) not suitable for production use from a security > perspective for this particular reason even in absence of the other > concerns expressed. Thanks for clarifying; that makes more sense. __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [tc][infra][release][security][stable][kolla][loci][tripleo][docker][kubernetes] do we want to be publishing binary container images?
Excerpts from Michał Jastrzębski's message of 2017-05-16 08:20:17 -0700: > On 16 May 2017 at 08:12, Doug Hellmann wrote: > > Excerpts from Michał Jastrzębski's message of 2017-05-16 06:52:12 -0700: > >> On 16 May 2017 at 06:20, Flavio Percoco wrote: > >> > On 16/05/17 14:08 +0200, Thierry Carrez wrote: > >> >> > >> >> Flavio Percoco wrote: > >> >>> > >> >>> From a release perspective, as Doug mentioned, we've avoided releasing > >> >>> projects > >> >>> in any kind of built form. This was also one of the concerns I raised > >> >>> when > >> >>> working on the proposal to support other programming languages. The > >> >>> problem of > >> >>> releasing built images goes beyond the infrastructure requirements. > >> >>> It's > >> >>> the > >> >>> message and the guarantees implied with the built product itself that > >> >>> are > >> >>> the > >> >>> concern here. And I tend to agree with Doug that this might be a > >> >>> problem > >> >>> for us > >> >>> as a community. Unfortunately, putting your name, Michal, as contact > >> >>> point is > >> >>> not enough. Kolla is not the only project producing container images > >> >>> and > >> >>> we need > >> >>> to be consistent in the way we release these images. > >> >>> > >> >>> Nothing prevents people for building their own images and uploading > >> >>> them > >> >>> to > >> >>> dockerhub. Having this as part of the OpenStack's pipeline is a > >> >>> problem. > >> >> > >> >> > >> >> I totally subscribe to the concerns around publishing binaries (under > >> >> any form), and the expectations in terms of security maintenance that it > >> >> would set on the publisher. At the same time, we need to have images > >> >> available, for convenience and testing. So what is the best way to > >> >> achieve that without setting strong security maintenance expectations > >> >> for the OpenStack community ? We have several options: > >> >> > >> >> 1/ Have third-parties publish images > >> >> It is the current situation. The issue is that the Kolla team (and > >> >> likely others) would rather automate the process and use OpenStack > >> >> infrastructure for it. > >> >> > >> >> 2/ Have third-parties publish images, but through OpenStack infra > >> >> This would allow to automate the process, but it would be a bit weird to > >> >> use common infra resources to publish in a private repo. > >> >> > >> >> 3/ Publish transient (per-commit or daily) images > >> >> A "daily build" (especially if you replace it every day) would set > >> >> relatively-limited expectations in terms of maintenance. It would end up > >> >> picking up security updates in upstream layers, even if not immediately. > >> >> > >> >> 4/ Publish images and own them > >> >> Staff release / VMT / stable team in a way that lets us properly own > >> >> those images and publish them officially. > >> >> > >> >> Personally I think (4) is not realistic. I think we could make (3) work, > >> >> and I prefer it to (2). If all else fails, we should keep (1). > >> > > >> > > >> > Agreed #4 is a bit unrealistic. > >> > > >> > Not sure I understand the difference between #2 and #3. Is it just the > >> > cadence? > >> > > >> > I'd prefer for these builds to have a daily cadence because it sets the > >> > expectations w.r.t maintenance right: "These images are daily builds and > >> > not > >> > certified releases. For stable builds you're better off building it > >> > yourself" > >> > >> And daily builds are exactly what I wanted in the first place:) We > >> probably will keep publishing release packages too, but we can be so > >> called 3rd party. I also agree [4] is completely unrealistic and I > >> would be against putting such heavy burden of responsibility on any > >> community, including Kolla. > >> > >> While daily cadence will send message that it's not stable, truth will > >> be that it will be more stable than what people would normally build > >> locally (again, it passes more gates), but I'm totally fine in not > >> saying that and let people decide how they want to use it. > >> > >> So, can we move on with implementation? > > > > I don't want the images published to docker hub. Are they still useful > > to you if they aren't published? > > What do you mean? We need images available...whether it's dockerhub, > infra-hosted registry or any other way to have them, we need to be > able to have images that are available and fresh without building. > Dockerhub/quay.io is least problems for infra team/resources. There are 2 separate concerns. The first concern is whether this is a good idea at all, from a policy perspective. Do we have the people to maintain the images, track CVEs, etc.? Do we have the response time to update or remove bad images? Can we, as a community, actually staff the support to an appropriate level? Or, can we clearly communicate that we do not support the images for production use and effectively avoid having someone start to rely on them? The second concern is whether we have the compute r
Re: [openstack-dev] [tc][infra][release][security][stable][kolla][loci][tripleo][docker][kubernetes] do we want to be publishing binary container images?
On 16 May 2017 at 11:27, Doug Hellmann wrote: > Excerpts from Michał Jastrzębski's message of 2017-05-16 09:46:19 -0700: >> So another consideration. Do you think whole rule of "not building >> binares" should be reconsidered? We are kind of new use case here. We >> aren't distro but we are packagers (kind of). I don't think putting us >> on equal footing as Red Hat, Canonical or other companies is correct >> here. >> >> K8s is something we want to work with, and what we are discussing is >> central to how k8s is used. K8s community creates this culture of >> "organic packages" built by anyone, most of companies/projects already >> have semi-official container images and I think expectations on >> quality of these are well...none? You get what you're given and if you >> don't agree, there is always way to reproduce this yourself. >> >> [Another huge snip] >> > > I wanted to have the discussion, but my position for now is that > we should continue as we have been and not change the policy. > > I don't have a problem with any individual or group of individuals > publishing their own organic packages. The issue I have is with > making sure it is clear those *are* "organic" and not officially > supported by the broader community. One way to do that is to say > they need to be built somewhere other than on our shared infrastructure. > There may be other ways, though, so I'm looking for input on that. What I was trying to say here is, current discussion aside, maybe we should revise this "not supported by broader community" rule. They may very well be supported to a certain point. Support is not just yes or no, it's all the levels in between. I think we can afford *some* level of official support, even if that some level means best effort made by community. If Kolla community, not an individual like myself, would like to support these images best to our ability, why aren't we allowed? As long as we are crystal clear what is scope of our support, why can't we do it? I think we've already proven that it's going to be tremendously useful for a lot of people, even in a shape we discuss today, that is "best effort, you still need to validate it for yourself"... > Doug > > __ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [tc][infra][release][security][stable][kolla][loci][tripleo][docker][kubernetes] do we want to be publishing binary container images?
I'm not sure I follow the problem. The containers being built don't pull from infra's mirrors, so they can be secure containers. Once built, during deploy/testing, they don't install anything, so shouldn't have any issues there either. Am I misunderstanding? Thanks, Kevin From: Sam Yaple [sam...@yaple.net] Sent: Tuesday, May 16, 2017 7:11 AM To: OpenStack Development Mailing List (not for usage questions) Subject: Re: [openstack-dev] [tc][infra][release][security][stable][kolla][loci][tripleo][docker][kubernetes] do we want to be publishing binary container images? I would like to bring up a subject that hasn't really been discussed in this thread yet, forgive me if I missed an email mentioning this. What I personally would like to see is a publishing infrastructure to allow pushing built images to an internal infra mirror/repo/registry for consumption of internal infra jobs (deployment tools like kolla-ansible and openstack-ansible). The images built from infra mirrors with security turned off are perfect for testing internally to infra. If you build images properly in infra, then you will have an image that is not security checked (no gpg verification of packages) and completely unverifiable. These are absolutely not images we want to push to DockerHub/quay for obvious reasons. Security and verification being chief among them. They are absolutely not images that should ever be run in production and are only suited for testing. These are the only types of images that can come out of infra. Thanks, SamYaple On Tue, May 16, 2017 at 1:57 PM, Michał Jastrzębski mailto:inc...@gmail.com>> wrote: On 16 May 2017 at 06:22, Doug Hellmann mailto:d...@doughellmann.com>> wrote: > Excerpts from Thierry Carrez's message of 2017-05-16 14:08:07 +0200: >> Flavio Percoco wrote: >> > From a release perspective, as Doug mentioned, we've avoided releasing >> > projects >> > in any kind of built form. This was also one of the concerns I raised when >> > working on the proposal to support other programming languages. The >> > problem of >> > releasing built images goes beyond the infrastructure requirements. It's >> > the >> > message and the guarantees implied with the built product itself that are >> > the >> > concern here. And I tend to agree with Doug that this might be a problem >> > for us >> > as a community. Unfortunately, putting your name, Michal, as contact point >> > is >> > not enough. Kolla is not the only project producing container images and >> > we need >> > to be consistent in the way we release these images. >> > >> > Nothing prevents people for building their own images and uploading them to >> > dockerhub. Having this as part of the OpenStack's pipeline is a problem. >> >> I totally subscribe to the concerns around publishing binaries (under >> any form), and the expectations in terms of security maintenance that it >> would set on the publisher. At the same time, we need to have images >> available, for convenience and testing. So what is the best way to >> achieve that without setting strong security maintenance expectations >> for the OpenStack community ? We have several options: >> >> 1/ Have third-parties publish images >> It is the current situation. The issue is that the Kolla team (and >> likely others) would rather automate the process and use OpenStack >> infrastructure for it. >> >> 2/ Have third-parties publish images, but through OpenStack infra >> This would allow to automate the process, but it would be a bit weird to >> use common infra resources to publish in a private repo. >> >> 3/ Publish transient (per-commit or daily) images >> A "daily build" (especially if you replace it every day) would set >> relatively-limited expectations in terms of maintenance. It would end up >> picking up security updates in upstream layers, even if not immediately. >> >> 4/ Publish images and own them >> Staff release / VMT / stable team in a way that lets us properly own >> those images and publish them officially. >> >> Personally I think (4) is not realistic. I think we could make (3) work, >> and I prefer it to (2). If all else fails, we should keep (1). >> > > At the forum we talked about putting test images on a "private" > repository hosted on openstack.org<http://openstack.org> somewhere. I think > that's option > 3 from your list? > > Paul may be able to shed more light on the details of the technology > (maybe it's just an Apache-served repo, rather than a fu
Re: [openstack-dev] [tc][infra][release][security][stable][kolla][loci][tripleo][docker][kubernetes] do we want to be publishing binary container images?
Excerpts from Michał Jastrzębski's message of 2017-05-16 09:51:00 -0700: > On 16 May 2017 at 09:40, Clint Byrum wrote: > > > > What's at stake isn't so much "how do we get the bits to the users" but > > "how do we only get bits to users that they need". If you build and push > > daily, do you expect all of your users to also _pull_ daily? Redeploy > > all their containers? How do you detect that there's new CVE-fixing > > stuff in a daily build? > > > > This is really the realm of distributors that have full-time security > > teams tracking issues and providing support to paying customers. > > > > So I think this is a fine idea, however, it needs to include a commitment > > for a full-time paid security team who weighs in on every change to > > the manifest. Otherwise we're just lobbing time bombs into our users' > > data-centers. > > One thing I struggle with is...well...how does *not having* built > containers help with that? If your company have full time security > team, they can check our containers prior to deployment. If your > company doesn't, then building locally will be subject to same risks > as downloading from dockerhub. Difference is, dockerhub containers > were tested in our CI to extend that our CI allows. No matter whether > or not you have your own security team, local CI, staging env, that > will be just a little bit of testing on top of that which you get for > free, and I think that's value enough for users to push for this. The benefit of not building images ourselves is that we are clearly communicating that the responsibility for maintaining the images falls on whoever *does* build them. There can be no question in any user's mind that the community somehow needs to maintain the content of the images for them, just because we're publishing new images at some regular cadence. Doug __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [tc][infra][release][security][stable][kolla][loci][tripleo][docker][kubernetes] do we want to be publishing binary container images?
My guess is same with octavia. https://github.com/openstack/octavia/tree/master/diskimage-create#diskimage-builder-script-for-creating-octavia-amphora-images -Josh Fox, Kevin M wrote: +1. ironic and trove have the same issues as well. lowering the bar in order to kick the tires will help OpenStack a lot in adoption. From: Sean Dague [s...@dague.net] Sent: Tuesday, May 16, 2017 6:28 AM To: openstack-dev@lists.openstack.org Subject: Re: [openstack-dev] [tc][infra][release][security][stable][kolla][loci][tripleo][docker][kubernetes] do we want to be publishing binary container images? On 05/16/2017 09:24 AM, Doug Hellmann wrote: __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [tc][infra][release][security][stable][kolla][loci][tripleo][docker][kubernetes] do we want to be publishing binary container images?
On 16 May 2017 at 11:33, Doug Hellmann wrote: > Excerpts from Michał Jastrzębski's message of 2017-05-16 08:20:17 -0700: >> On 16 May 2017 at 08:12, Doug Hellmann wrote: >> > Excerpts from Michał Jastrzębski's message of 2017-05-16 06:52:12 -0700: >> >> On 16 May 2017 at 06:20, Flavio Percoco wrote: >> >> > On 16/05/17 14:08 +0200, Thierry Carrez wrote: >> >> >> >> >> >> Flavio Percoco wrote: >> >> >>> >> >> >>> From a release perspective, as Doug mentioned, we've avoided releasing >> >> >>> projects >> >> >>> in any kind of built form. This was also one of the concerns I raised >> >> >>> when >> >> >>> working on the proposal to support other programming languages. The >> >> >>> problem of >> >> >>> releasing built images goes beyond the infrastructure requirements. >> >> >>> It's >> >> >>> the >> >> >>> message and the guarantees implied with the built product itself that >> >> >>> are >> >> >>> the >> >> >>> concern here. And I tend to agree with Doug that this might be a >> >> >>> problem >> >> >>> for us >> >> >>> as a community. Unfortunately, putting your name, Michal, as contact >> >> >>> point is >> >> >>> not enough. Kolla is not the only project producing container images >> >> >>> and >> >> >>> we need >> >> >>> to be consistent in the way we release these images. >> >> >>> >> >> >>> Nothing prevents people for building their own images and uploading >> >> >>> them >> >> >>> to >> >> >>> dockerhub. Having this as part of the OpenStack's pipeline is a >> >> >>> problem. >> >> >> >> >> >> >> >> >> I totally subscribe to the concerns around publishing binaries (under >> >> >> any form), and the expectations in terms of security maintenance that >> >> >> it >> >> >> would set on the publisher. At the same time, we need to have images >> >> >> available, for convenience and testing. So what is the best way to >> >> >> achieve that without setting strong security maintenance expectations >> >> >> for the OpenStack community ? We have several options: >> >> >> >> >> >> 1/ Have third-parties publish images >> >> >> It is the current situation. The issue is that the Kolla team (and >> >> >> likely others) would rather automate the process and use OpenStack >> >> >> infrastructure for it. >> >> >> >> >> >> 2/ Have third-parties publish images, but through OpenStack infra >> >> >> This would allow to automate the process, but it would be a bit weird >> >> >> to >> >> >> use common infra resources to publish in a private repo. >> >> >> >> >> >> 3/ Publish transient (per-commit or daily) images >> >> >> A "daily build" (especially if you replace it every day) would set >> >> >> relatively-limited expectations in terms of maintenance. It would end >> >> >> up >> >> >> picking up security updates in upstream layers, even if not >> >> >> immediately. >> >> >> >> >> >> 4/ Publish images and own them >> >> >> Staff release / VMT / stable team in a way that lets us properly own >> >> >> those images and publish them officially. >> >> >> >> >> >> Personally I think (4) is not realistic. I think we could make (3) >> >> >> work, >> >> >> and I prefer it to (2). If all else fails, we should keep (1). >> >> > >> >> > >> >> > Agreed #4 is a bit unrealistic. >> >> > >> >> > Not sure I understand the difference between #2 and #3. Is it just the >> >> > cadence? >> >> > >> >> > I'd prefer for these builds to have a daily cadence because it sets the >> >> > expectations w.r.t maintenance right: "These images are daily builds >> >> > and not >> >> > certified releases. For stable builds you're better off building it >> >> > yourself" >> >> >> >> And daily builds are exactly what I wanted in the first place:) We >> >> probably will keep publishing release packages too, but we can be so >> >> called 3rd party. I also agree [4] is completely unrealistic and I >> >> would be against putting such heavy burden of responsibility on any >> >> community, including Kolla. >> >> >> >> While daily cadence will send message that it's not stable, truth will >> >> be that it will be more stable than what people would normally build >> >> locally (again, it passes more gates), but I'm totally fine in not >> >> saying that and let people decide how they want to use it. >> >> >> >> So, can we move on with implementation? >> > >> > I don't want the images published to docker hub. Are they still useful >> > to you if they aren't published? >> >> What do you mean? We need images available...whether it's dockerhub, >> infra-hosted registry or any other way to have them, we need to be >> able to have images that are available and fresh without building. >> Dockerhub/quay.io is least problems for infra team/resources. > > There are 2 separate concerns. > > The first concern is whether this is a good idea at all, from a > policy perspective. Do we have the people to maintain the images, > track CVEs, etc.? Do we have the response time to update or remove > bad images? Can we, as a community, actually staff the support to > an appropr
Re: [openstack-dev] [tc][infra][release][security][stable][kolla][loci][tripleo][docker][kubernetes] do we want to be publishing binary container images?
Excerpts from Michał Jastrzębski's message of 2017-05-16 11:38:19 -0700: > On 16 May 2017 at 11:27, Doug Hellmann wrote: > > Excerpts from Michał Jastrzębski's message of 2017-05-16 09:46:19 -0700: > >> So another consideration. Do you think whole rule of "not building > >> binares" should be reconsidered? We are kind of new use case here. We > >> aren't distro but we are packagers (kind of). I don't think putting us > >> on equal footing as Red Hat, Canonical or other companies is correct > >> here. > >> > >> K8s is something we want to work with, and what we are discussing is > >> central to how k8s is used. K8s community creates this culture of > >> "organic packages" built by anyone, most of companies/projects already > >> have semi-official container images and I think expectations on > >> quality of these are well...none? You get what you're given and if you > >> don't agree, there is always way to reproduce this yourself. > >> > >> [Another huge snip] > >> > > > > I wanted to have the discussion, but my position for now is that > > we should continue as we have been and not change the policy. > > > > I don't have a problem with any individual or group of individuals > > publishing their own organic packages. The issue I have is with > > making sure it is clear those *are* "organic" and not officially > > supported by the broader community. One way to do that is to say > > they need to be built somewhere other than on our shared infrastructure. > > There may be other ways, though, so I'm looking for input on that. > > What I was trying to say here is, current discussion aside, maybe we > should revise this "not supported by broader community" rule. They may > very well be supported to a certain point. Support is not just yes or > no, it's all the levels in between. I think we can afford *some* level > of official support, even if that some level means best effort made by > community. If Kolla community, not an individual like myself, would > like to support these images best to our ability, why aren't we > allowed? As long as we are crystal clear what is scope of our support, > why can't we do it? I think we've already proven that it's going to be > tremendously useful for a lot of people, even in a shape we discuss > today, that is "best effort, you still need to validate it for > yourself"... Right, I understood that. So far I haven't heard anything to change my mind, though. I think you're underestimating the amount of risk you're taking on for yourselves and by extension the rest of the community, and introducing to potential consumers of the images, by promising to support production deployments with a small team of people without the economic structure in place to sustain the work. Doug __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [tc][infra][release][security][stable][kolla][loci][tripleo][docker][kubernetes] do we want to be publishing binary container images?
On 05/16/2017 02:39 PM, Doug Hellmann wrote: > Excerpts from Michał Jastrzębski's message of 2017-05-16 09:51:00 -0700: >> On 16 May 2017 at 09:40, Clint Byrum wrote: >>> >>> What's at stake isn't so much "how do we get the bits to the users" but >>> "how do we only get bits to users that they need". If you build and push >>> daily, do you expect all of your users to also _pull_ daily? Redeploy >>> all their containers? How do you detect that there's new CVE-fixing >>> stuff in a daily build? >>> >>> This is really the realm of distributors that have full-time security >>> teams tracking issues and providing support to paying customers. >>> >>> So I think this is a fine idea, however, it needs to include a commitment >>> for a full-time paid security team who weighs in on every change to >>> the manifest. Otherwise we're just lobbing time bombs into our users' >>> data-centers. >> >> One thing I struggle with is...well...how does *not having* built >> containers help with that? If your company have full time security >> team, they can check our containers prior to deployment. If your >> company doesn't, then building locally will be subject to same risks >> as downloading from dockerhub. Difference is, dockerhub containers >> were tested in our CI to extend that our CI allows. No matter whether >> or not you have your own security team, local CI, staging env, that >> will be just a little bit of testing on top of that which you get for >> free, and I think that's value enough for users to push for this. > > The benefit of not building images ourselves is that we are clearly > communicating that the responsibility for maintaining the images > falls on whoever *does* build them. There can be no question in any > user's mind that the community somehow needs to maintain the content > of the images for them, just because we're publishing new images > at some regular cadence. +1. It is really easy to think that saying "don't use this in production" prevents people from using it in production. See: User Survey 2017 and the number of folks reporting DevStack as their production deployment tool. We need to not only manage artifacts, but expectations. And with all the confusion of projects in the openstack git namespace being officially blessed openstack projects over the past few years, I can't imagine people not thinking that openstack infra generated content in dockerhub is officially supported content. -Sean -- Sean Dague http://dague.net __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [tc][infra][release][security][stable][kolla][loci][tripleo][docker][kubernetes] do we want to be publishing binary container images?
On 16 May 2017 at 11:49, Doug Hellmann wrote: > Excerpts from Michał Jastrzębski's message of 2017-05-16 11:38:19 -0700: >> On 16 May 2017 at 11:27, Doug Hellmann wrote: >> > Excerpts from Michał Jastrzębski's message of 2017-05-16 09:46:19 -0700: >> >> So another consideration. Do you think whole rule of "not building >> >> binares" should be reconsidered? We are kind of new use case here. We >> >> aren't distro but we are packagers (kind of). I don't think putting us >> >> on equal footing as Red Hat, Canonical or other companies is correct >> >> here. >> >> >> >> K8s is something we want to work with, and what we are discussing is >> >> central to how k8s is used. K8s community creates this culture of >> >> "organic packages" built by anyone, most of companies/projects already >> >> have semi-official container images and I think expectations on >> >> quality of these are well...none? You get what you're given and if you >> >> don't agree, there is always way to reproduce this yourself. >> >> >> >> [Another huge snip] >> >> >> > >> > I wanted to have the discussion, but my position for now is that >> > we should continue as we have been and not change the policy. >> > >> > I don't have a problem with any individual or group of individuals >> > publishing their own organic packages. The issue I have is with >> > making sure it is clear those *are* "organic" and not officially >> > supported by the broader community. One way to do that is to say >> > they need to be built somewhere other than on our shared infrastructure. >> > There may be other ways, though, so I'm looking for input on that. >> >> What I was trying to say here is, current discussion aside, maybe we >> should revise this "not supported by broader community" rule. They may >> very well be supported to a certain point. Support is not just yes or >> no, it's all the levels in between. I think we can afford *some* level >> of official support, even if that some level means best effort made by >> community. If Kolla community, not an individual like myself, would >> like to support these images best to our ability, why aren't we >> allowed? As long as we are crystal clear what is scope of our support, >> why can't we do it? I think we've already proven that it's going to be >> tremendously useful for a lot of people, even in a shape we discuss >> today, that is "best effort, you still need to validate it for >> yourself"... > > Right, I understood that. So far I haven't heard anything to change > my mind, though. > > I think you're underestimating the amount of risk you're taking on > for yourselves and by extension the rest of the community, and > introducing to potential consumers of the images, by promising to > support production deployments with a small team of people without > the economic structure in place to sustain the work. Again, we tell what it is and what it is not. I think support is loaded term here. Instead we can create lengthy documentation explaining to a detail lifecycle and testing certain container had to pass before it lands in dockerhub. Maybe add link to particular set of jobs that container had passed. Only thing we can offer is automated and transparent process of publishing. On top of that? You are on your own. But even within these boundaries, a lot of people could have better experience of running OpenStack... > Doug > > __ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [tc][infra][release][security][stable][kolla][loci][tripleo][docker][kubernetes] do we want to be publishing binary container images?
And bandwidth can be conserved by only uploading images that actually changed in non trivial ways (packages were updated, not just logfile with a new timestamp) Thanks, Keivn From: Michał Jastrzębski [inc...@gmail.com] Sent: Tuesday, May 16, 2017 11:46 AM To: OpenStack Development Mailing List (not for usage questions) Subject: Re: [openstack-dev] [tc][infra][release][security][stable][kolla][loci][tripleo][docker][kubernetes] do we want to be publishing binary container images? On 16 May 2017 at 11:33, Doug Hellmann wrote: > Excerpts from Michał Jastrzębski's message of 2017-05-16 08:20:17 -0700: >> On 16 May 2017 at 08:12, Doug Hellmann wrote: >> > Excerpts from Michał Jastrzębski's message of 2017-05-16 06:52:12 -0700: >> >> On 16 May 2017 at 06:20, Flavio Percoco wrote: >> >> > On 16/05/17 14:08 +0200, Thierry Carrez wrote: >> >> >> >> >> >> Flavio Percoco wrote: >> >> >>> >> >> >>> From a release perspective, as Doug mentioned, we've avoided releasing >> >> >>> projects >> >> >>> in any kind of built form. This was also one of the concerns I raised >> >> >>> when >> >> >>> working on the proposal to support other programming languages. The >> >> >>> problem of >> >> >>> releasing built images goes beyond the infrastructure requirements. >> >> >>> It's >> >> >>> the >> >> >>> message and the guarantees implied with the built product itself that >> >> >>> are >> >> >>> the >> >> >>> concern here. And I tend to agree with Doug that this might be a >> >> >>> problem >> >> >>> for us >> >> >>> as a community. Unfortunately, putting your name, Michal, as contact >> >> >>> point is >> >> >>> not enough. Kolla is not the only project producing container images >> >> >>> and >> >> >>> we need >> >> >>> to be consistent in the way we release these images. >> >> >>> >> >> >>> Nothing prevents people for building their own images and uploading >> >> >>> them >> >> >>> to >> >> >>> dockerhub. Having this as part of the OpenStack's pipeline is a >> >> >>> problem. >> >> >> >> >> >> >> >> >> I totally subscribe to the concerns around publishing binaries (under >> >> >> any form), and the expectations in terms of security maintenance that >> >> >> it >> >> >> would set on the publisher. At the same time, we need to have images >> >> >> available, for convenience and testing. So what is the best way to >> >> >> achieve that without setting strong security maintenance expectations >> >> >> for the OpenStack community ? We have several options: >> >> >> >> >> >> 1/ Have third-parties publish images >> >> >> It is the current situation. The issue is that the Kolla team (and >> >> >> likely others) would rather automate the process and use OpenStack >> >> >> infrastructure for it. >> >> >> >> >> >> 2/ Have third-parties publish images, but through OpenStack infra >> >> >> This would allow to automate the process, but it would be a bit weird >> >> >> to >> >> >> use common infra resources to publish in a private repo. >> >> >> >> >> >> 3/ Publish transient (per-commit or daily) images >> >> >> A "daily build" (especially if you replace it every day) would set >> >> >> relatively-limited expectations in terms of maintenance. It would end >> >> >> up >> >> >> picking up security updates in upstream layers, even if not >> >> >> immediately. >> >> >> >> >> >> 4/ Publish images and own them >> >> >> Staff release / VMT / stable team in a way that lets us properly own >> >> >> those images and publish them officially. >> >> >> >> >> >> Personally I think (4) is not realistic. I think we could make (3) >> >> >> work, >> >> >> and I prefer it to (2). If all else fails, we sh
Re: [openstack-dev] [tc][infra][release][security][stable][kolla][loci][tripleo][docker][kubernetes] do we want to be publishing binary container images?
We can put warnings all over it and if folks choose to ignore them, then its they who took the risk and get to keep the pieces when it breaks. Some folks are crazy enough to run devstack in production. But does that mean we should just abandon devstack? No. of course not. I don't think we should hold back OpenStack from the benefits this provides just because a few folks might possibly do something bad with it. We do what we can to allow folks to recognize bad ideas. But if they want to do it anyway, thats the freedom of open source. They can. They get to deal with the fallout though, not us. If we always catered to the lowest common denominator we would never get anywhere. Thanks, Kevin From: Doug Hellmann [d...@doughellmann.com] Sent: Tuesday, May 16, 2017 11:49 AM To: openstack-dev Subject: Re: [openstack-dev] [tc][infra][release][security][stable][kolla][loci][tripleo][docker][kubernetes] do we want to be publishing binary container images? Excerpts from Michał Jastrzębski's message of 2017-05-16 11:38:19 -0700: > On 16 May 2017 at 11:27, Doug Hellmann wrote: > > Excerpts from Michał Jastrzębski's message of 2017-05-16 09:46:19 -0700: > >> So another consideration. Do you think whole rule of "not building > >> binares" should be reconsidered? We are kind of new use case here. We > >> aren't distro but we are packagers (kind of). I don't think putting us > >> on equal footing as Red Hat, Canonical or other companies is correct > >> here. > >> > >> K8s is something we want to work with, and what we are discussing is > >> central to how k8s is used. K8s community creates this culture of > >> "organic packages" built by anyone, most of companies/projects already > >> have semi-official container images and I think expectations on > >> quality of these are well...none? You get what you're given and if you > >> don't agree, there is always way to reproduce this yourself. > >> > >> [Another huge snip] > >> > > > > I wanted to have the discussion, but my position for now is that > > we should continue as we have been and not change the policy. > > > > I don't have a problem with any individual or group of individuals > > publishing their own organic packages. The issue I have is with > > making sure it is clear those *are* "organic" and not officially > > supported by the broader community. One way to do that is to say > > they need to be built somewhere other than on our shared infrastructure. > > There may be other ways, though, so I'm looking for input on that. > > What I was trying to say here is, current discussion aside, maybe we > should revise this "not supported by broader community" rule. They may > very well be supported to a certain point. Support is not just yes or > no, it's all the levels in between. I think we can afford *some* level > of official support, even if that some level means best effort made by > community. If Kolla community, not an individual like myself, would > like to support these images best to our ability, why aren't we > allowed? As long as we are crystal clear what is scope of our support, > why can't we do it? I think we've already proven that it's going to be > tremendously useful for a lot of people, even in a shape we discuss > today, that is "best effort, you still need to validate it for > yourself"... Right, I understood that. So far I haven't heard anything to change my mind, though. I think you're underestimating the amount of risk you're taking on for yourselves and by extension the rest of the community, and introducing to potential consumers of the images, by promising to support production deployments with a small team of people without the economic structure in place to sustain the work. Doug __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [tc][infra][release][security][stable][kolla][loci][tripleo][docker][kubernetes] do we want to be publishing binary container images?
On 2017-05-16 11:46:14 -0700 (-0700), Michał Jastrzębski wrote: [...] > So CVE tracking might not be required by us. Since we still use > distro packages under the hood, we can just use these. [...] I think the question is how I, as a semi-clueful downstream user of your images, can tell whether the image I'm deploying has fixes for some specific recently disclosed vulnerability. It sounds like your answer is that I should compare the package manifest against the versions listed on the distro's CVE tracker or similar service? That should be prominently documented, perhaps in a highly visible FAQ list. > Since we'd rebuild daily, that alone would ensure timely update to > our containers. What we can promise to potential users is that > containers out there were built lately (24hrs) [...] As outlined elsewhere in the thread, there are a myriad of reasons why this could end up not being the case from time to time so I can only assume your definition of "promise" differs from mine (and unfortunately, from most people who might be trying to decide whether it's safe to rely on these images in a sensitive/production environment). -- Jeremy Stanley signature.asc Description: Digital signature __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [tc][infra][release][security][stable][kolla][loci][tripleo][docker][kubernetes] do we want to be publishing binary container images?
On 16 May 2017 at 12:36, Jeremy Stanley wrote: > On 2017-05-16 11:46:14 -0700 (-0700), Michał Jastrzębski wrote: > [...] >> So CVE tracking might not be required by us. Since we still use >> distro packages under the hood, we can just use these. > [...] > > I think the question is how I, as a semi-clueful downstream user of > your images, can tell whether the image I'm deploying has fixes for > some specific recently disclosed vulnerability. It sounds like your > answer is that I should compare the package manifest against the > versions listed on the distro's CVE tracker or similar service? That > should be prominently documented, perhaps in a highly visible FAQ > list. One thing we've been working on prior to summit was manifesto of versions - I think we can provide single file with all the versions of packages in container, we can add track of CI jobs that led containers to this place, all the informations that semi-careful downstream user can use to help him/her to determine what's that they're getting. I'm all for that kind of features. >> Since we'd rebuild daily, that alone would ensure timely update to >> our containers. What we can promise to potential users is that >> containers out there were built lately (24hrs) > [...] > > As outlined elsewhere in the thread, there are a myriad of reasons > why this could end up not being the case from time to time so I can > only assume your definition of "promise" differs from mine (and > unfortunately, from most people who might be trying to decide > whether it's safe to rely on these images in a sensitive/production > environment). By "promise" I mean clear documentation of where containers came from and what did they pass. After that, take it or leave it. > -- > Jeremy Stanley > > __ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [tc][infra][release][security][stable][kolla][loci][tripleo][docker][kubernetes] do we want to be publishing binary container images?
Security is a spectrum, not a boolean. I know some sites that have instituted super long/complex password requirements. The end result is usually humans just writing pw's down on stickies then since its too hard to remember, making security worse, not better. Humans are always the weakest link in the security system and must be taken into account. There are people that will do things insecurely. If we can make it much easier for them to get access to much more fresh stuff rather then build it themselves (which they wont), the community as a whole will be better off. There will be far fewer clouds out there with known bad CVE's baked in. Lets provide the tools to make it as easy as possible to identify containers with issues, and allow upgrading the system to newer ones. Which CVE's are on the system is somewhat less important then being able to get to newer versions installed easily. Right now, thats probably harder then it should be. If its hard, people won't do it. Fresh images are just a step in that process. Thanks, Kevin From: Jeremy Stanley [fu...@yuggoth.org] Sent: Tuesday, May 16, 2017 12:36 PM To: OpenStack Development Mailing List (not for usage questions) Subject: Re: [openstack-dev] [tc][infra][release][security][stable][kolla][loci][tripleo][docker][kubernetes] do we want to be publishing binary container images? On 2017-05-16 11:46:14 -0700 (-0700), Michał Jastrzębski wrote: [...] > So CVE tracking might not be required by us. Since we still use > distro packages under the hood, we can just use these. [...] I think the question is how I, as a semi-clueful downstream user of your images, can tell whether the image I'm deploying has fixes for some specific recently disclosed vulnerability. It sounds like your answer is that I should compare the package manifest against the versions listed on the distro's CVE tracker or similar service? That should be prominently documented, perhaps in a highly visible FAQ list. > Since we'd rebuild daily, that alone would ensure timely update to > our containers. What we can promise to potential users is that > containers out there were built lately (24hrs) [...] As outlined elsewhere in the thread, there are a myriad of reasons why this could end up not being the case from time to time so I can only assume your definition of "promise" differs from mine (and unfortunately, from most people who might be trying to decide whether it's safe to rely on these images in a sensitive/production environment). -- Jeremy Stanley __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [tc][infra][release][security][stable][kolla][loci][tripleo][docker][kubernetes] do we want to be publishing binary container images?
On 2017-05-16 19:56:31 + (+), Fox, Kevin M wrote: [...] > Lets provide the tools to make it as easy as possible to identify > containers with issues, and allow upgrading the system to newer > ones. > > Which CVE's are on the system is somewhat less important then > being able to get to newer versions installed easily. Right now, > thats probably harder then it should be. If its hard, people won't > do it. [...] My point (which I've trimmed because I don't have the patience to undo your top-posting at the moment) was that security expectations for these images should be clearly documented and communicated, that's all. I'm not sure what you were reading into it. -- Jeremy Stanley signature.asc Description: Digital signature __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [tc][infra][release][security][stable][kolla][loci][tripleo][docker][kubernetes] do we want to be publishing binary container images?
Sean Dague wrote: > On 05/16/2017 02:39 PM, Doug Hellmann wrote: >> Excerpts from Michał Jastrzębski's message of 2017-05-16 09:51:00 -0700: >>> One thing I struggle with is...well...how does *not having* built >>> containers help with that? If your company have full time security >>> team, they can check our containers prior to deployment. If your >>> company doesn't, then building locally will be subject to same risks >>> as downloading from dockerhub. Difference is, dockerhub containers >>> were tested in our CI to extend that our CI allows. No matter whether >>> or not you have your own security team, local CI, staging env, that >>> will be just a little bit of testing on top of that which you get for >>> free, and I think that's value enough for users to push for this. >> >> The benefit of not building images ourselves is that we are clearly >> communicating that the responsibility for maintaining the images >> falls on whoever *does* build them. There can be no question in any >> user's mind that the community somehow needs to maintain the content >> of the images for them, just because we're publishing new images >> at some regular cadence. > > +1. It is really easy to think that saying "don't use this in > production" prevents people from using it in production. See: User > Survey 2017 and the number of folks reporting DevStack as their > production deployment tool. > > We need to not only manage artifacts, but expectations. And with all the > confusion of projects in the openstack git namespace being officially > blessed openstack projects over the past few years, I can't imagine > people not thinking that openstack infra generated content in dockerhub > is officially supported content. I totally agree, although I think daily rebuilds / per-commit rebuilds, together with a properly named repository, might limit expectations enough to remove the "supported" part of your sentence. As a parallel, we refresh per-commit a Nova master source code tarball (nova-master.tar.gz). If a vulnerability is introduced in master but was never "released" with a version number, we silently fix it in master (no OSSA advisory published). People tracking master are supposed to be continuously tracking master. Back to container image world, if we refresh those images daily and they are not versioned or archived (basically you can only use the latest and can't really access past dailies), I think we'd be in a similar situation ? -- Thierry Carrez (ttx) __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [tc][infra][release][security][stable][kolla][loci][tripleo][docker][kubernetes] do we want to be publishing binary container images?
On Wed, 17 May 2017, Thierry Carrez wrote: Back to container image world, if we refresh those images daily and they are not versioned or archived (basically you can only use the latest and can't really access past dailies), I think we'd be in a similar situation ? Yes, this. -- Chris Dent ┬──┬◡ノ(° -°ノ) https://anticdent.org/ freenode: cdent tw: @anticdent__ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [tc][infra][release][security][stable][kolla][loci][tripleo][docker][kubernetes] do we want to be publishing binary container images?
On 17 May 2017 at 04:14, Chris Dent wrote: > On Wed, 17 May 2017, Thierry Carrez wrote: > >> Back to container image world, if we refresh those images daily and they >> are not versioned or archived (basically you can only use the latest and >> can't really access past dailies), I think we'd be in a similar situation >> ? > > > Yes, this. I think it's not a bad idea to message "you are responsible for archving your containers". Do that, combine it with good toolset that helps users determine versions of packages and other metadata and we'll end up with something that itself would be greatly appreciated. Few potential user stories. I have OpenStack <100 nodes and need every single one of them, hence no CI. At the same time I want to have fresh packages to avoid CVEs. I deploy kolla with tip-of-the-stable-branch and setup cronjob that will upgrade it every week. Because my scenerio is quite typical and containers already ran through gates that tests my scenerio, I'm good. Another one: I have 300+ node cloud, heavy CI and security team examining every container. While I could build containers locally, downloading them is just simpler and effectively the same (after all, it's containers being tested not build process). Every download our security team scrutinize contaniers and uses toolset Kolla provides to help them. Additional benefit is that on top of our CI these images went through Kolla CI which is nice, more testing is always good. And another one We are Kolla community. We want to provide testing for full release upgrades every day in gates, to make sure OpenStack and Kolla is upgradable and improve general user experience of upgrades. Because infra is resource constrained, we cannot afford building 2 sets of containers (stable and master) and doing deploy->test->upgrade->test. However because we have these cached containers, that are fresh and passed CI for deploy, we can just use them! Now effectively we're not only testing Kolla's correctness of upgrade procedure but also all the other project team upgrades! Oh, it seems Nova merged something that negatively affects upgrades, let's make sure they are aware! And last one, which cannot be underestimated I am CTO of some company and I've heard OpenStack is no longer hard to deploy, I'll just download kolla-ansible and try. I'll follow this guide that deploys simple OpenStack with 2 commands and few small configs, and it's done! Super simple! We're moving to OpenStack and start contributing tomorrow! Please, let's solve messaging problems, put burden of archiving on users, whatever it takes to protect our community from wrong expectations, but not kill this effort. There are very real and immediate benefits to OpenStack as a whole if we do this. Cheers, Michal > -- > Chris Dent ┬──┬◡ノ(° -°ノ) https://anticdent.org/ > freenode: cdent tw: @anticdent > __ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [tc][infra][release][security][stable][kolla][loci][tripleo][docker][kubernetes] do we want to be publishing binary container images?
Michael, There has been a lot of cost and risk analysis in this thread. What hasn’t really been discussed at great detail is the “benefit analysis” which you have started. I think we are all clear on the risks and the costs. If we as a technical community are going to place a line in the sand and state “thou shall not ship containers to dockerhub” because of the risks inherent in such behavior, we are not integrating properly with the emerging container ecosystem. Expecting operators to build their own images is a viable path forward. Unfortunately, the lack of automation introduces significant cognitive load for many (based upon the Q&A in the #openstack-kolla channel on a daily basis). This cognitive load could be (incorrectly) perceived by many to be “OpenStack doesn’t care about integrating with adjacent communities.” On balance, the benefits to OpenStack of your proposal outweigh the costs. Regards -steve -Original Message- From: Michał Jastrzębski Reply-To: "OpenStack Development Mailing List (not for usage questions)" Date: Wednesday, May 17, 2017 at 7:47 AM To: "OpenStack Development Mailing List (not for usage questions)" Subject: Re: [openstack-dev] [tc][infra][release][security][stable][kolla][loci][tripleo][docker][kubernetes] do we want to be publishing binary container images? On 17 May 2017 at 04:14, Chris Dent wrote: > On Wed, 17 May 2017, Thierry Carrez wrote: > >> Back to container image world, if we refresh those images daily and they >> are not versioned or archived (basically you can only use the latest and >> can't really access past dailies), I think we'd be in a similar situation >> ? > > > Yes, this. I think it's not a bad idea to message "you are responsible for archving your containers". Do that, combine it with good toolset that helps users determine versions of packages and other metadata and we'll end up with something that itself would be greatly appreciated. Few potential user stories. I have OpenStack <100 nodes and need every single one of them, hence no CI. At the same time I want to have fresh packages to avoid CVEs. I deploy kolla with tip-of-the-stable-branch and setup cronjob that will upgrade it every week. Because my scenerio is quite typical and containers already ran through gates that tests my scenerio, I'm good. Another one: I have 300+ node cloud, heavy CI and security team examining every container. While I could build containers locally, downloading them is just simpler and effectively the same (after all, it's containers being tested not build process). Every download our security team scrutinize contaniers and uses toolset Kolla provides to help them. Additional benefit is that on top of our CI these images went through Kolla CI which is nice, more testing is always good. And another one We are Kolla community. We want to provide testing for full release upgrades every day in gates, to make sure OpenStack and Kolla is upgradable and improve general user experience of upgrades. Because infra is resource constrained, we cannot afford building 2 sets of containers (stable and master) and doing deploy->test->upgrade->test. However because we have these cached containers, that are fresh and passed CI for deploy, we can just use them! Now effectively we're not only testing Kolla's correctness of upgrade procedure but also all the other project team upgrades! Oh, it seems Nova merged something that negatively affects upgrades, let's make sure they are aware! And last one, which cannot be underestimated I am CTO of some company and I've heard OpenStack is no longer hard to deploy, I'll just download kolla-ansible and try. I'll follow this guide that deploys simple OpenStack with 2 commands and few small configs, and it's done! Super simple! We're moving to OpenStack and start contributing tomorrow! Please, let's solve messaging problems, put burden of archiving on users, whatever it takes to protect our community from wrong expectations, but not kill this effort. There are very real and immediate benefits to OpenStack as a whole if we do this. Cheers, Michal > -- > Chris Dent ┬──┬◡ノ(° -°ノ) https://anticdent.org/ > freenode: cdent tw: @anticdent > __ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mai
Re: [openstack-dev] [tc][infra][release][security][stable][kolla][loci][tripleo][docker][kubernetes] do we want to be publishing binary container images?
Excerpts from Chris Dent's message of 2017-05-17 12:14:40 +0100: > On Wed, 17 May 2017, Thierry Carrez wrote: > > > Back to container image world, if we refresh those images daily and they > > are not versioned or archived (basically you can only use the latest and > > can't really access past dailies), I think we'd be in a similar situation ? > > Yes, this. > Is that how container publishing works? Can we overwrite an existing archive, so that there is only ever 1 version of a published container at any given time? Doug __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [tc][infra][release][security][stable][kolla][loci][tripleo][docker][kubernetes] do we want to be publishing binary container images?
Excerpts from Thierry Carrez's message of 2017-05-17 12:19:22 +0200: > Sean Dague wrote: > > On 05/16/2017 02:39 PM, Doug Hellmann wrote: > >> Excerpts from Michał Jastrzębski's message of 2017-05-16 09:51:00 -0700: > >>> One thing I struggle with is...well...how does *not having* built > >>> containers help with that? If your company have full time security > >>> team, they can check our containers prior to deployment. If your > >>> company doesn't, then building locally will be subject to same risks > >>> as downloading from dockerhub. Difference is, dockerhub containers > >>> were tested in our CI to extend that our CI allows. No matter whether > >>> or not you have your own security team, local CI, staging env, that > >>> will be just a little bit of testing on top of that which you get for > >>> free, and I think that's value enough for users to push for this. > >> > >> The benefit of not building images ourselves is that we are clearly > >> communicating that the responsibility for maintaining the images > >> falls on whoever *does* build them. There can be no question in any > >> user's mind that the community somehow needs to maintain the content > >> of the images for them, just because we're publishing new images > >> at some regular cadence. > > > > +1. It is really easy to think that saying "don't use this in > > production" prevents people from using it in production. See: User > > Survey 2017 and the number of folks reporting DevStack as their > > production deployment tool. > > > > We need to not only manage artifacts, but expectations. And with all the > > confusion of projects in the openstack git namespace being officially > > blessed openstack projects over the past few years, I can't imagine > > people not thinking that openstack infra generated content in dockerhub > > is officially supported content. > > I totally agree, although I think daily rebuilds / per-commit rebuilds, > together with a properly named repository, might limit expectations > enough to remove the "supported" part of your sentence. > > As a parallel, we refresh per-commit a Nova master source code tarball > (nova-master.tar.gz). If a vulnerability is introduced in master but was > never "released" with a version number, we silently fix it in master (no > OSSA advisory published). People tracking master are supposed to be > continuously tracking master. > > Back to container image world, if we refresh those images daily and they > are not versioned or archived (basically you can only use the latest and > can't really access past dailies), I think we'd be in a similar situation ? > The source tarballs are not production deployment tools and only contain code for one project at a time and it is all our code, so we don't have to track issues in any other components. The same differences apply to the artifacts we publish to PyPI and NPM. So it's similar, but different. Doug __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [tc][infra][release][security][stable][kolla][loci][tripleo][docker][kubernetes] do we want to be publishing binary container images?
On 17 May 2017 at 08:55, Doug Hellmann wrote: > Excerpts from Chris Dent's message of 2017-05-17 12:14:40 +0100: >> On Wed, 17 May 2017, Thierry Carrez wrote: >> >> > Back to container image world, if we refresh those images daily and they >> > are not versioned or archived (basically you can only use the latest and >> > can't really access past dailies), I think we'd be in a similar situation ? >> >> Yes, this. >> > > Is that how container publishing works? Can we overwrite an existing > archive, so that there is only ever 1 version of a published container > at any given time? We can do it either way, but that's how we want it, top of stable branch daily + top of master. > Doug > > __ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [tc][infra][release][security][stable][kolla][loci][tripleo][docker][kubernetes] do we want to be publishing binary container images?
Doug, When a docker image is pushed to dockerhub (or quay.io, etc) a tag is specified. If the tag already exists, it is overwritten. Regards -steve -Original Message- From: Doug Hellmann Reply-To: "OpenStack Development Mailing List (not for usage questions)" Date: Wednesday, May 17, 2017 at 8:55 AM To: openstack-dev Subject: Re: [openstack-dev] [tc][infra][release][security][stable][kolla][loci][tripleo][docker][kubernetes] do we want to be publishing binary container images? Excerpts from Chris Dent's message of 2017-05-17 12:14:40 +0100: > On Wed, 17 May 2017, Thierry Carrez wrote: > > > Back to container image world, if we refresh those images daily and they > > are not versioned or archived (basically you can only use the latest and > > can't really access past dailies), I think we'd be in a similar situation ? > > Yes, this. > Is that how container publishing works? Can we overwrite an existing archive, so that there is only ever 1 version of a published container at any given time? Doug __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [tc][infra][release][security][stable][kolla][loci][tripleo][docker][kubernetes] do we want to be publishing binary container images?
What kolla's been discussing is having something like: 4.0.0-1, 4.0.0-2, 4.0.0-3, etc. only keeping the most recent two. and then aliases for: 4.0.0 pointing to the newest. This allows helm upgrade to atomically roll/forward back properly. If you drop releases, k8s can't properly do atomic upgrades. You will get inconsistent rollouts and will not know which containers are old and have the security issues. Knowing there is a newer -revision number also notifies you that you are running something old and need to update. Thanks, Kevin From: Chris Dent [cdent...@anticdent.org] Sent: Wednesday, May 17, 2017 4:14 AM To: OpenStack Development Mailing List (not for usage questions) Subject: Re: [openstack-dev] [tc][infra][release][security][stable][kolla][loci][tripleo][docker][kubernetes] do we want to be publishing binary container images? On Wed, 17 May 2017, Thierry Carrez wrote: > Back to container image world, if we refresh those images daily and they > are not versioned or archived (basically you can only use the latest and > can't really access past dailies), I think we'd be in a similar situation ? Yes, this. -- Chris Dent ┬──┬◡ノ(° -°ノ) https://anticdent.org/ freenode: cdent tw: @anticdent __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [tc][infra][release][security][stable][kolla][loci][tripleo][docker][kubernetes] do we want to be publishing binary container images?
You can do that, but doesn't play well with orchestration systems such as k8s as it removes its ability to know when upgraded containers appear. Thanks, Kevin * As always, sorry for top posting, but my organization does not allow me the choice of mail software. From: Doug Hellmann [d...@doughellmann.com] Sent: Wednesday, May 17, 2017 8:55 AM To: openstack-dev Subject: Re: [openstack-dev] [tc][infra][release][security][stable][kolla][loci][tripleo][docker][kubernetes] do we want to be publishing binary container images? Excerpts from Chris Dent's message of 2017-05-17 12:14:40 +0100: > On Wed, 17 May 2017, Thierry Carrez wrote: > > > Back to container image world, if we refresh those images daily and they > > are not versioned or archived (basically you can only use the latest and > > can't really access past dailies), I think we'd be in a similar situation ? > > Yes, this. > Is that how container publishing works? Can we overwrite an existing archive, so that there is only ever 1 version of a published container at any given time? Doug __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [tc][infra][release][security][stable][kolla][loci][tripleo][docker][kubernetes] do we want to be publishing binary container images?
Excerpts from Michał Jastrzębski's message of 2017-05-17 07:47:31 -0700: > On 17 May 2017 at 04:14, Chris Dent wrote: > > On Wed, 17 May 2017, Thierry Carrez wrote: > > > >> Back to container image world, if we refresh those images daily and they > >> are not versioned or archived (basically you can only use the latest and > >> can't really access past dailies), I think we'd be in a similar situation > >> ? > > > > > > Yes, this. > > I think it's not a bad idea to message "you are responsible for > archving your containers". Do that, combine it with good toolset that > helps users determine versions of packages and other metadata and > we'll end up with something that itself would be greatly appreciated. > > Few potential user stories. > > I have OpenStack <100 nodes and need every single one of them, hence > no CI. At the same time I want to have fresh packages to avoid CVEs. I > deploy kolla with tip-of-the-stable-branch and setup cronjob that will > upgrade it every week. Because my scenerio is quite typical and > containers already ran through gates that tests my scenerio, I'm good. > > Another one: > > I have 300+ node cloud, heavy CI and security team examining every > container. While I could build containers locally, downloading them is > just simpler and effectively the same (after all, it's containers > being tested not build process). Every download our security team > scrutinize contaniers and uses toolset Kolla provides to help them. > Additional benefit is that on top of our CI these images went through > Kolla CI which is nice, more testing is always good. > > And another one > > We are Kolla community. We want to provide testing for full release > upgrades every day in gates, to make sure OpenStack and Kolla is > upgradable and improve general user experience of upgrades. Because > infra is resource constrained, we cannot afford building 2 sets of > containers (stable and master) and doing deploy->test->upgrade->test. > However because we have these cached containers, that are fresh and > passed CI for deploy, we can just use them! Now effectively we're not > only testing Kolla's correctness of upgrade procedure but also all the > other project team upgrades! Oh, it seems Nova merged something that > negatively affects upgrades, let's make sure they are aware! > > And last one, which cannot be underestimated > > I am CTO of some company and I've heard OpenStack is no longer hard to > deploy, I'll just download kolla-ansible and try. I'll follow this > guide that deploys simple OpenStack with 2 commands and few small > configs, and it's done! Super simple! We're moving to OpenStack and > start contributing tomorrow! > > Please, let's solve messaging problems, put burden of archiving on > users, whatever it takes to protect our community from wrong > expectations, but not kill this effort. There are very real and > immediate benefits to OpenStack as a whole if we do this. > > Cheers, > Michal You've presented some positive scenarios. Here's a worst case situation that I'm worried about. Suppose in a few months the top several companies contributing to kolla decide to pull out of or reduce their contributions to OpenStack. IBM, Intel, Oracle, and Cisco either lay folks off or redirect their efforts to other projects. Maybe they start contributing directly to kubernetes. The kolla team is hit badly, and all of the people from that team who know how the container publishing jobs work are gone. The day after everyone says goodbye, the build breaks. Maybe a bad patch lands, or maybe some upstream assumption changes. The issue isn't with the infra jobs themselves. The break means no new container images are being published. Since there's not much of a kolla team any more, it looks like it will be a while before anyone has time to figure out how to fix the problem. Later that same day, a new zero-day exploit is announced in a component included in all or most of those images. Something that isn't developed in the community, such as OpenSSL or glibc. The exploit allows a complete breach of any app running with it. All existing published containers include the bad bits and need to be updated. We now have an unknown number of clouds running containers built by the community with major security holes. The team responsible for maintaining those images is a shambles, but even if they weren't the automation isn't working, so no new images can be published. The consumers of the existing containers haven't bothered to set up build pipelines of their own, because why bother? Even though we've clearly said the images "we" publish are for our own testing, they have found it irresistibly convenient to use them and move on with their lives. When the exploit is announced, they start clamoring for new container images, and become understandably irate when we say we didn't think they would be using them in production and they *shouldn't have* and their problems are not our problems because we tol
Re: [openstack-dev] [tc][infra][release][security][stable][kolla][loci][tripleo][docker][kubernetes] do we want to be publishing binary container images?
On 17 May 2017 at 11:04, Doug Hellmann wrote: > Excerpts from Michał Jastrzębski's message of 2017-05-17 07:47:31 -0700: >> On 17 May 2017 at 04:14, Chris Dent wrote: >> > On Wed, 17 May 2017, Thierry Carrez wrote: >> > >> >> Back to container image world, if we refresh those images daily and they >> >> are not versioned or archived (basically you can only use the latest and >> >> can't really access past dailies), I think we'd be in a similar situation >> >> ? >> > >> > >> > Yes, this. >> >> I think it's not a bad idea to message "you are responsible for >> archving your containers". Do that, combine it with good toolset that >> helps users determine versions of packages and other metadata and >> we'll end up with something that itself would be greatly appreciated. >> >> Few potential user stories. >> >> I have OpenStack <100 nodes and need every single one of them, hence >> no CI. At the same time I want to have fresh packages to avoid CVEs. I >> deploy kolla with tip-of-the-stable-branch and setup cronjob that will >> upgrade it every week. Because my scenerio is quite typical and >> containers already ran through gates that tests my scenerio, I'm good. >> >> Another one: >> >> I have 300+ node cloud, heavy CI and security team examining every >> container. While I could build containers locally, downloading them is >> just simpler and effectively the same (after all, it's containers >> being tested not build process). Every download our security team >> scrutinize contaniers and uses toolset Kolla provides to help them. >> Additional benefit is that on top of our CI these images went through >> Kolla CI which is nice, more testing is always good. >> >> And another one >> >> We are Kolla community. We want to provide testing for full release >> upgrades every day in gates, to make sure OpenStack and Kolla is >> upgradable and improve general user experience of upgrades. Because >> infra is resource constrained, we cannot afford building 2 sets of >> containers (stable and master) and doing deploy->test->upgrade->test. >> However because we have these cached containers, that are fresh and >> passed CI for deploy, we can just use them! Now effectively we're not >> only testing Kolla's correctness of upgrade procedure but also all the >> other project team upgrades! Oh, it seems Nova merged something that >> negatively affects upgrades, let's make sure they are aware! >> >> And last one, which cannot be underestimated >> >> I am CTO of some company and I've heard OpenStack is no longer hard to >> deploy, I'll just download kolla-ansible and try. I'll follow this >> guide that deploys simple OpenStack with 2 commands and few small >> configs, and it's done! Super simple! We're moving to OpenStack and >> start contributing tomorrow! >> >> Please, let's solve messaging problems, put burden of archiving on >> users, whatever it takes to protect our community from wrong >> expectations, but not kill this effort. There are very real and >> immediate benefits to OpenStack as a whole if we do this. >> >> Cheers, >> Michal > > You've presented some positive scenarios. Here's a worst case > situation that I'm worried about. > > Suppose in a few months the top several companies contributing to > kolla decide to pull out of or reduce their contributions to > OpenStack. IBM, Intel, Oracle, and Cisco either lay folks off or > redirect their efforts to other projects. Maybe they start > contributing directly to kubernetes. The kolla team is hit badly, > and all of the people from that team who know how the container > publishing jobs work are gone. There are only 2 ways to defend against that: diverse community, which we have. If Intel, Red Hat, Oracle, Cisco and IBM back out of OpenStack, we'd still have almost 50% of contributors. I think we'll much more likely to survive than most of other Big Tent projects. In fact, I'd think with our current diversity, that we'll survive for as long as OpenStack survives. Also all the more reasons why *we shouldn't build images personally*, we should have autonomous process to do it for us. > The day after everyone says goodbye, the build breaks. Maybe a bad > patch lands, or maybe some upstream assumption changes. The issue > isn't with the infra jobs themselves. The break means no new container > images are being published. Since there's not much of a kolla team > any more, it looks like it will be a while before anyone has time > to figure out how to fix the problem. > Later that same day, a new zero-day exploit is announced in a > component included in all or most of those images. Something that > isn't developed in the community, such as OpenSSL or glibc. The > exploit allows a complete breach of any app running with it. All > existing published containers include the bad bits and need to be > updated. I guess this is problem of all the software ever written. If community dies around it, people who uses it are in lots of trouble. One way to make sure it won't happen is
Re: [openstack-dev] [tc][infra][release][security][stable][kolla][loci][tripleo][docker][kubernetes] do we want to be publishing binary container images?
On 17 May 2017 at 11:36, Michał Jastrzębski wrote: > On 17 May 2017 at 11:04, Doug Hellmann wrote: >> Excerpts from Michał Jastrzębski's message of 2017-05-17 07:47:31 -0700: >>> On 17 May 2017 at 04:14, Chris Dent wrote: >>> > On Wed, 17 May 2017, Thierry Carrez wrote: >>> > >>> >> Back to container image world, if we refresh those images daily and they >>> >> are not versioned or archived (basically you can only use the latest and >>> >> can't really access past dailies), I think we'd be in a similar situation >>> >> ? >>> > >>> > >>> > Yes, this. >>> >>> I think it's not a bad idea to message "you are responsible for >>> archving your containers". Do that, combine it with good toolset that >>> helps users determine versions of packages and other metadata and >>> we'll end up with something that itself would be greatly appreciated. >>> >>> Few potential user stories. >>> >>> I have OpenStack <100 nodes and need every single one of them, hence >>> no CI. At the same time I want to have fresh packages to avoid CVEs. I >>> deploy kolla with tip-of-the-stable-branch and setup cronjob that will >>> upgrade it every week. Because my scenerio is quite typical and >>> containers already ran through gates that tests my scenerio, I'm good. >>> >>> Another one: >>> >>> I have 300+ node cloud, heavy CI and security team examining every >>> container. While I could build containers locally, downloading them is >>> just simpler and effectively the same (after all, it's containers >>> being tested not build process). Every download our security team >>> scrutinize contaniers and uses toolset Kolla provides to help them. >>> Additional benefit is that on top of our CI these images went through >>> Kolla CI which is nice, more testing is always good. >>> >>> And another one >>> >>> We are Kolla community. We want to provide testing for full release >>> upgrades every day in gates, to make sure OpenStack and Kolla is >>> upgradable and improve general user experience of upgrades. Because >>> infra is resource constrained, we cannot afford building 2 sets of >>> containers (stable and master) and doing deploy->test->upgrade->test. >>> However because we have these cached containers, that are fresh and >>> passed CI for deploy, we can just use them! Now effectively we're not >>> only testing Kolla's correctness of upgrade procedure but also all the >>> other project team upgrades! Oh, it seems Nova merged something that >>> negatively affects upgrades, let's make sure they are aware! >>> >>> And last one, which cannot be underestimated >>> >>> I am CTO of some company and I've heard OpenStack is no longer hard to >>> deploy, I'll just download kolla-ansible and try. I'll follow this >>> guide that deploys simple OpenStack with 2 commands and few small >>> configs, and it's done! Super simple! We're moving to OpenStack and >>> start contributing tomorrow! >>> >>> Please, let's solve messaging problems, put burden of archiving on >>> users, whatever it takes to protect our community from wrong >>> expectations, but not kill this effort. There are very real and >>> immediate benefits to OpenStack as a whole if we do this. >>> >>> Cheers, >>> Michal >> >> You've presented some positive scenarios. Here's a worst case >> situation that I'm worried about. >> >> Suppose in a few months the top several companies contributing to >> kolla decide to pull out of or reduce their contributions to >> OpenStack. IBM, Intel, Oracle, and Cisco either lay folks off or >> redirect their efforts to other projects. Maybe they start >> contributing directly to kubernetes. The kolla team is hit badly, >> and all of the people from that team who know how the container >> publishing jobs work are gone. > > There are only 2 ways to defend against that: diverse community, which > we have. If Intel, Red Hat, Oracle, Cisco and IBM back out of > OpenStack, we'd still have almost 50% of contributors. I think we'll > much more likely to survive than most of other Big Tent projects. In > fact, I'd think with our current diversity, that we'll survive for as > long as OpenStack survives. Diverse community and off-by-one errors;) I was meaning to say diverse community and involvement. > > Also all the more reasons why *we shouldn't build images personally*, > we should have autonomous process to do it for us. > >> The day after everyone says goodbye, the build breaks. Maybe a bad >> patch lands, or maybe some upstream assumption changes. The issue >> isn't with the infra jobs themselves. The break means no new container >> images are being published. Since there's not much of a kolla team >> any more, it looks like it will be a while before anyone has time >> to figure out how to fix the problem. > >> Later that same day, a new zero-day exploit is announced in a >> component included in all or most of those images. Something that >> isn't developed in the community, such as OpenSSL or glibc. The >> exploit allows a complete breach of any app running with
Re: [openstack-dev] [tc][infra][release][security][stable][kolla][loci][tripleo][docker][kubernetes] do we want to be publishing binary container images?
Excerpts from Michał Jastrzębski's message of 2017-05-17 11:36:40 -0700: > On 17 May 2017 at 11:04, Doug Hellmann wrote: > > > You've presented some positive scenarios. Here's a worst case > > situation that I'm worried about. > > > > Suppose in a few months the top several companies contributing to > > kolla decide to pull out of or reduce their contributions to > > OpenStack. IBM, Intel, Oracle, and Cisco either lay folks off or > > redirect their efforts to other projects. Maybe they start > > contributing directly to kubernetes. The kolla team is hit badly, > > and all of the people from that team who know how the container > > publishing jobs work are gone. > > There are only 2 ways to defend against that: diverse community, which > we have. If Intel, Red Hat, Oracle, Cisco and IBM back out of > OpenStack, we'd still have almost 50% of contributors. I think we'll > much more likely to survive than most of other Big Tent projects. In > fact, I'd think with our current diversity, that we'll survive for as > long as OpenStack survives. > > Also all the more reasons why *we shouldn't build images personally*, > we should have autonomous process to do it for us. > > > The day after everyone says goodbye, the build breaks. Maybe a bad > > patch lands, or maybe some upstream assumption changes. The issue > > isn't with the infra jobs themselves. The break means no new container > > images are being published. Since there's not much of a kolla team > > any more, it looks like it will be a while before anyone has time > > to figure out how to fix the problem. > > > Later that same day, a new zero-day exploit is announced in a > > component included in all or most of those images. Something that > > isn't developed in the community, such as OpenSSL or glibc. The > > exploit allows a complete breach of any app running with it. All > > existing published containers include the bad bits and need to be > > updated. > > I guess this is problem of all the software ever written. If community > dies around it, people who uses it are in lots of trouble. One way to > make sure it won't happen is to get involved yourself to make sure you > can fix what is broken for you. This is how open source works. In > Kolla most of our contributors are actually operators who run these > very containers in their own infrastructure. This is where our > diversity comes from. We aren't distro and that makes us, and our > users, more protected from this scenario. > > If nova loses all of it's community, and someone finds critical bug in > nova that allows hackers to gain access to vm data, there will be > nobody to fix it, that's bad right? Same argument can be made. We > aren't discussing deleting Nova tho right? I think there's a difference there, because of the way nova and the other components currently have an intermediary doing the distribution. > > Contrast that with a scenario in which consumers either take > > responsibility for their systems by building their own images, by > > collaborating directly with other consumers to share the resources > > needed to build those images, or by paying a third-party a sustainable > > amount of money to build images for them. In any of those cases, > > there is an incentive for the responsible party to be ready and > > able to produce new images in a timely manner. Consumers of the > > images know exactly where to go for support when they have problems. > > Issues in those images don't reflect on the community in any way, > > because we were not involved in producing them. > > Unless as you said build system breaks, then they are equally screwed > locally. Unless someone fix it, and they can fix it for openstack > infra too. Difference is, for OpenStack infra it's whole community > that can fix it where local it's just you. That's the strength of open > source. The difference is that it's definitely not the community's problem in that case. I'm looking at this from a community perspective, and not the deployer or operator. > > As I said at the start of this thread, we've long avoided building > > and supporting simple operating system style packages of the > > components we produce. I am still struggling to understand how > > building more complex artifacts, including bits over which we have > > little or no control, is somehow more sustainable than those simple > > packages. > > Binaries are built as standalone projects. Nova-api has no > dependencies build into .rpm. If issue you just described would happen > in any of projects openstack uses as dependency, in any of these [1], > same argument applies. We pin specific versions in upper constraints. > I'm willing to bet money of it that if today one of these libs would > release CVE, there is good change we won't find out. I don't understand what you're saying here. How do constraints we use in our test gate enter into this? > Bottom line, yes, we do *package* today with PIP. Exactly same issues > apply to pip packages if ve
Re: [openstack-dev] [tc][infra][release][security][stable][kolla][loci][tripleo][docker][kubernetes] do we want to be publishing binary container images?
On 16.05.2017 20:57, Michał Jastrzębski wrote: > On 16 May 2017 at 11:49, Doug Hellmann wrote: >> Excerpts from Michał Jastrzębski's message of 2017-05-16 11:38:19 -0700: >>> On 16 May 2017 at 11:27, Doug Hellmann wrote: Excerpts from Michał Jastrzębski's message of 2017-05-16 09:46:19 -0700: > So another consideration. Do you think whole rule of "not building > binares" should be reconsidered? We are kind of new use case here. We > aren't distro but we are packagers (kind of). I don't think putting us > on equal footing as Red Hat, Canonical or other companies is correct > here. > > K8s is something we want to work with, and what we are discussing is > central to how k8s is used. K8s community creates this culture of > "organic packages" built by anyone, most of companies/projects already > have semi-official container images and I think expectations on > quality of these are well...none? You get what you're given and if you > don't agree, there is always way to reproduce this yourself. > > [Another huge snip] > I wanted to have the discussion, but my position for now is that we should continue as we have been and not change the policy. I don't have a problem with any individual or group of individuals publishing their own organic packages. The issue I have is with making sure it is clear those *are* "organic" and not officially supported by the broader community. One way to do that is to say they need to be built somewhere other than on our shared infrastructure. There may be other ways, though, so I'm looking for input on that. >>> >>> What I was trying to say here is, current discussion aside, maybe we >>> should revise this "not supported by broader community" rule. They may >>> very well be supported to a certain point. Support is not just yes or >>> no, it's all the levels in between. I think we can afford *some* level >>> of official support, even if that some level means best effort made by >>> community. If Kolla community, not an individual like myself, would >>> like to support these images best to our ability, why aren't we >>> allowed? As long as we are crystal clear what is scope of our support, >>> why can't we do it? I think we've already proven that it's going to be >>> tremendously useful for a lot of people, even in a shape we discuss >>> today, that is "best effort, you still need to validate it for >>> yourself"... >> >> Right, I understood that. So far I haven't heard anything to change >> my mind, though. >> >> I think you're underestimating the amount of risk you're taking on >> for yourselves and by extension the rest of the community, and >> introducing to potential consumers of the images, by promising to >> support production deployments with a small team of people without >> the economic structure in place to sustain the work. > > Again, we tell what it is and what it is not. I think support is > loaded term here. Instead we can create lengthy documentation > explaining to a detail lifecycle and testing certain container had to > pass before it lands in dockerhub. Maybe add link to particular set of > jobs that container had passed. Only thing we can offer is automated > and transparent process of publishing. On top of that? You are on your > own. But even within these boundaries, a lot of people could have > better experience of running OpenStack... That totally makes sense. Supporting builds like "a published container passed some test scenarios for our CI gates, here is a link to particular set of jobs that container had passed" benefits all and has nothing to the production use cases and guarantees nothing in terms of supporting them. > >> Doug >> >> __ >> OpenStack Development Mailing List (not for usage questions) >> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > __ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > -- Best regards, Bogdan Dobrelya, Irc #bogdando __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [tc][infra][release][security][stable][kolla][loci][tripleo][docker][kubernetes] do we want to be publishing binary container images?
On Tue, May 16, 2017 at 06:57:04AM -0700, Michał Jastrzębski wrote: > On 16 May 2017 at 06:22, Doug Hellmann wrote: > > Excerpts from Thierry Carrez's message of 2017-05-16 14:08:07 +0200: > >> Flavio Percoco wrote: > >> > From a release perspective, as Doug mentioned, we've avoided releasing > >> > projects > >> > in any kind of built form. This was also one of the concerns I raised > >> > when > >> > working on the proposal to support other programming languages. The > >> > problem of > >> > releasing built images goes beyond the infrastructure requirements. It's > >> > the > >> > message and the guarantees implied with the built product itself that > >> > are the > >> > concern here. And I tend to agree with Doug that this might be a problem > >> > for us > >> > as a community. Unfortunately, putting your name, Michal, as contact > >> > point is > >> > not enough. Kolla is not the only project producing container images and > >> > we need > >> > to be consistent in the way we release these images. > >> > > >> > Nothing prevents people for building their own images and uploading them > >> > to > >> > dockerhub. Having this as part of the OpenStack's pipeline is a problem. > >> > >> I totally subscribe to the concerns around publishing binaries (under > >> any form), and the expectations in terms of security maintenance that it > >> would set on the publisher. At the same time, we need to have images > >> available, for convenience and testing. So what is the best way to > >> achieve that without setting strong security maintenance expectations > >> for the OpenStack community ? We have several options: > >> > >> 1/ Have third-parties publish images > >> It is the current situation. The issue is that the Kolla team (and > >> likely others) would rather automate the process and use OpenStack > >> infrastructure for it. > >> > >> 2/ Have third-parties publish images, but through OpenStack infra > >> This would allow to automate the process, but it would be a bit weird to > >> use common infra resources to publish in a private repo. > >> > >> 3/ Publish transient (per-commit or daily) images > >> A "daily build" (especially if you replace it every day) would set > >> relatively-limited expectations in terms of maintenance. It would end up > >> picking up security updates in upstream layers, even if not immediately. > >> > >> 4/ Publish images and own them > >> Staff release / VMT / stable team in a way that lets us properly own > >> those images and publish them officially. > >> > >> Personally I think (4) is not realistic. I think we could make (3) work, > >> and I prefer it to (2). If all else fails, we should keep (1). > >> > > > > At the forum we talked about putting test images on a "private" > > repository hosted on openstack.org somewhere. I think that's option > > 3 from your list? > > > > Paul may be able to shed more light on the details of the technology > > (maybe it's just an Apache-served repo, rather than a full blown > > instance of Docker's service, for example). > > Issue with that is > > 1. Apache served is harder to use because we want to follow docker API > and we'd have to reimplement it No, the idea is apache is transparent, for now we have been using proxypass module in apache. I think what Doug was mentioning was have a primary docker registery, with is RW for a publisher, then proxy it to regional mirrors as RO. > 2. Running registry is single command > I've seen this mentioned a few times before, just because it is one command or 'simple' to do, doesn't mean we want to or can. Currently our infrastructure is complicated, for various reasons. I am sure we'll get to the right technical solution for making jobs happy. Remember our infrastructure spans 6 clouds and 15 regions and want to make sure it is done correctly. > 3. If we host in in infra, in case someone actually uses it (there > will be people like that), that will eat up lot of network traffic > potentially We can monitor this and adjust as needed. > 4. With local caching of images (working already) in nodepools we > loose complexity of mirroring registries across nodepools > > So bottom line, having dockerhub/quay.io is simply easier. > See comment above. > > Doug > > > > __ > > OpenStack Development Mailing List (not for usage questions) > > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > __ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-r
Re: [openstack-dev] [tc][infra][release][security][stable][kolla][loci][tripleo][docker][kubernetes] do we want to be publishing binary container images?
On Tue, May 16, 2017 at 02:11:18PM +, Sam Yaple wrote: > I would like to bring up a subject that hasn't really been discussed in > this thread yet, forgive me if I missed an email mentioning this. > > What I personally would like to see is a publishing infrastructure to allow > pushing built images to an internal infra mirror/repo/registry for > consumption of internal infra jobs (deployment tools like kolla-ansible and > openstack-ansible). The images built from infra mirrors with security > turned off are perfect for testing internally to infra. > Zuulv3 should have a little with this, it will allow for DAG graph for jobs, which means the top level job could be an image build then all jobs below can now consume said image. The steps we are still working on is artifact handling but long term, it should be possible for the testing jobs to setup the dynamic infrastructure needed themselves. > If you build images properly in infra, then you will have an image that is > not security checked (no gpg verification of packages) and completely > unverifiable. These are absolutely not images we want to push to > DockerHub/quay for obvious reasons. Security and verification being chief > among them. They are absolutely not images that should ever be run in > production and are only suited for testing. These are the only types of > images that can come out of infra. > We disable gpg for Ubuntu packaging for a specific reason, most this is because our APT repos are not official mirrors of upstream. We regenerate indexes every 2 hours as not to break long running jobs. We have talked in the past of fixing this, but it requires openstack-infra to move to a new mirroring tool for APT. > Thanks, > SamYaple > > On Tue, May 16, 2017 at 1:57 PM, Michał Jastrzębski > wrote: > > > On 16 May 2017 at 06:22, Doug Hellmann wrote: > > > Excerpts from Thierry Carrez's message of 2017-05-16 14:08:07 +0200: > > >> Flavio Percoco wrote: > > >> > From a release perspective, as Doug mentioned, we've avoided > > releasing projects > > >> > in any kind of built form. This was also one of the concerns I raised > > when > > >> > working on the proposal to support other programming languages. The > > problem of > > >> > releasing built images goes beyond the infrastructure requirements. > > It's the > > >> > message and the guarantees implied with the built product itself that > > are the > > >> > concern here. And I tend to agree with Doug that this might be a > > problem for us > > >> > as a community. Unfortunately, putting your name, Michal, as contact > > point is > > >> > not enough. Kolla is not the only project producing container images > > and we need > > >> > to be consistent in the way we release these images. > > >> > > > >> > Nothing prevents people for building their own images and uploading > > them to > > >> > dockerhub. Having this as part of the OpenStack's pipeline is a > > problem. > > >> > > >> I totally subscribe to the concerns around publishing binaries (under > > >> any form), and the expectations in terms of security maintenance that it > > >> would set on the publisher. At the same time, we need to have images > > >> available, for convenience and testing. So what is the best way to > > >> achieve that without setting strong security maintenance expectations > > >> for the OpenStack community ? We have several options: > > >> > > >> 1/ Have third-parties publish images > > >> It is the current situation. The issue is that the Kolla team (and > > >> likely others) would rather automate the process and use OpenStack > > >> infrastructure for it. > > >> > > >> 2/ Have third-parties publish images, but through OpenStack infra > > >> This would allow to automate the process, but it would be a bit weird to > > >> use common infra resources to publish in a private repo. > > >> > > >> 3/ Publish transient (per-commit or daily) images > > >> A "daily build" (especially if you replace it every day) would set > > >> relatively-limited expectations in terms of maintenance. It would end up > > >> picking up security updates in upstream layers, even if not immediately. > > >> > > >> 4/ Publish images and own them > > >> Staff release / VMT / stable team in a way that lets us properly own > > >> those images and publish them officially. > > >> > > >> Personally I think (4) is not realistic. I think we could make (3) work, > > >> and I prefer it to (2). If all else fails, we should keep (1). > > >> > > > > > > At the forum we talked about putting test images on a "private" > > > repository hosted on openstack.org somewhere. I think that's option > > > 3 from your list? > > > > > > Paul may be able to shed more light on the details of the technology > > > (maybe it's just an Apache-served repo, rather than a full blown > > > instance of Docker's service, for example). > > > > Issue with that is > > > > 1. Apache served is harder to use because we want to follow docker API > > and we'd have to reimplement it > > 2. Ru
Re: [openstack-dev] [tc][infra][release][security][stable][kolla][loci][tripleo][docker][kubernetes] do we want to be publishing binary container images?
>> Issue with that is >> >> 1. Apache served is harder to use because we want to follow docker API >> and we'd have to reimplement it > > No, the idea is apache is transparent, for now we have been using proxypass > module in apache. I think what Doug was mentioning was have a primary docker > registery, with is RW for a publisher, then proxy it to regional mirrors as > RO. That would also work, yes >> 2. Running registry is single command >> > I've seen this mentioned a few times before, just because it is one command or > 'simple' to do, doesn't mean we want to or can. Currently our infrastructure > is > complicated, for various reasons. I am sure we'll get to the right technical > solution for making jobs happy. Remember our infrastructure spans 6 clouds > and 15 > regions and want to make sure it is done correctly. And that's why we discussed dockerhub. Remember that I was willing to implement proper registry, but we decided to go with dockerhub simply because it puts less stress on both infra and infra team. And I totally agree with that statement. Dockerhub publisher + apache caching was our working idea. >> 3. If we host in in infra, in case someone actually uses it (there >> will be people like that), that will eat up lot of network traffic >> potentially > > We can monitor this and adjust as needed. > >> 4. With local caching of images (working already) in nodepools we >> loose complexity of mirroring registries across nodepools >> >> So bottom line, having dockerhub/quay.io is simply easier. >> > See comment above. > >> > Doug >> > >> > __ >> > OpenStack Development Mailing List (not for usage questions) >> > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe >> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >> >> __ >> OpenStack Development Mailing List (not for usage questions) >> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > __ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [tc][infra][release][security][stable][kolla][loci][tripleo][docker][kubernetes] do we want to be publishing binary container images?
On 18 May 2017 at 08:03, Paul Belanger wrote: > On Tue, May 16, 2017 at 02:11:18PM +, Sam Yaple wrote: >> I would like to bring up a subject that hasn't really been discussed in >> this thread yet, forgive me if I missed an email mentioning this. >> >> What I personally would like to see is a publishing infrastructure to allow >> pushing built images to an internal infra mirror/repo/registry for >> consumption of internal infra jobs (deployment tools like kolla-ansible and >> openstack-ansible). The images built from infra mirrors with security >> turned off are perfect for testing internally to infra. >> > Zuulv3 should have a little with this, it will allow for DAG graph for jobs, > which means the top level job could be an image build then all jobs below can > now consume said image. The steps we are still working on is artifact > handling > but long term, it should be possible for the testing jobs to setup the dynamic > infrastructure needed themselves. > >> If you build images properly in infra, then you will have an image that is >> not security checked (no gpg verification of packages) and completely >> unverifiable. These are absolutely not images we want to push to >> DockerHub/quay for obvious reasons. Security and verification being chief >> among them. They are absolutely not images that should ever be run in >> production and are only suited for testing. These are the only types of >> images that can come out of infra. >> > We disable gpg for Ubuntu packaging for a specific reason, most this is > because > our APT repos are not official mirrors of upstream. We regenerate indexes > every > 2 hours as not to break long running jobs. We have talked in the past of > fixing > this, but it requires openstack-infra to move to a new mirroring tool for APT. So idea to solve this particular problem goes like this: Publish job is not a change-driven, it'll be periodical (24h?) during low time. Then in this job we can turn off using infra mirrors and just use upstream signed. That being said, all the technical issues we saw so far (unless I'm missing something) are solvable and we (kolla community) would love to do all the heavy lifting to solve it. We need to wait for TC to resolve non-technical issues before we can proceed tho. >> Thanks, >> SamYaple >> >> On Tue, May 16, 2017 at 1:57 PM, Michał Jastrzębski >> wrote: >> >> > On 16 May 2017 at 06:22, Doug Hellmann wrote: >> > > Excerpts from Thierry Carrez's message of 2017-05-16 14:08:07 +0200: >> > >> Flavio Percoco wrote: >> > >> > From a release perspective, as Doug mentioned, we've avoided >> > releasing projects >> > >> > in any kind of built form. This was also one of the concerns I raised >> > when >> > >> > working on the proposal to support other programming languages. The >> > problem of >> > >> > releasing built images goes beyond the infrastructure requirements. >> > It's the >> > >> > message and the guarantees implied with the built product itself that >> > are the >> > >> > concern here. And I tend to agree with Doug that this might be a >> > problem for us >> > >> > as a community. Unfortunately, putting your name, Michal, as contact >> > point is >> > >> > not enough. Kolla is not the only project producing container images >> > and we need >> > >> > to be consistent in the way we release these images. >> > >> > >> > >> > Nothing prevents people for building their own images and uploading >> > them to >> > >> > dockerhub. Having this as part of the OpenStack's pipeline is a >> > problem. >> > >> >> > >> I totally subscribe to the concerns around publishing binaries (under >> > >> any form), and the expectations in terms of security maintenance that it >> > >> would set on the publisher. At the same time, we need to have images >> > >> available, for convenience and testing. So what is the best way to >> > >> achieve that without setting strong security maintenance expectations >> > >> for the OpenStack community ? We have several options: >> > >> >> > >> 1/ Have third-parties publish images >> > >> It is the current situation. The issue is that the Kolla team (and >> > >> likely others) would rather automate the process and use OpenStack >> > >> infrastructure for it. >> > >> >> > >> 2/ Have third-parties publish images, but through OpenStack infra >> > >> This would allow to automate the process, but it would be a bit weird to >> > >> use common infra resources to publish in a private repo. >> > >> >> > >> 3/ Publish transient (per-commit or daily) images >> > >> A "daily build" (especially if you replace it every day) would set >> > >> relatively-limited expectations in terms of maintenance. It would end up >> > >> picking up security updates in upstream layers, even if not immediately. >> > >> >> > >> 4/ Publish images and own them >> > >> Staff release / VMT / stable team in a way that lets us properly own >> > >> those images and publish them officially. >> > >> >> > >> Personally I think (4) is not realistic. I th
Re: [openstack-dev] [tc][infra][release][security][stable][kolla][loci][tripleo][docker][kubernetes] do we want to be publishing binary container images?
On Thu, May 18, 2017 at 09:34:44AM -0700, Michał Jastrzębski wrote: > >> Issue with that is > >> > >> 1. Apache served is harder to use because we want to follow docker API > >> and we'd have to reimplement it > > > > No, the idea is apache is transparent, for now we have been using proxypass > > module in apache. I think what Doug was mentioning was have a primary > > docker > > registery, with is RW for a publisher, then proxy it to regional mirrors as > > RO. > > That would also work, yes > > >> 2. Running registry is single command > >> > > I've seen this mentioned a few times before, just because it is one command > > or > > 'simple' to do, doesn't mean we want to or can. Currently our > > infrastructure is > > complicated, for various reasons. I am sure we'll get to the right > > technical > > solution for making jobs happy. Remember our infrastructure spans 6 clouds > > and 15 > > regions and want to make sure it is done correctly. > > And that's why we discussed dockerhub. Remember that I was willing to > implement proper registry, but we decided to go with dockerhub simply > because it puts less stress on both infra and infra team. And I > totally agree with that statement. Dockerhub publisher + apache > caching was our working idea. > yes, we still want to implement a docker registry for openstack, maybe for testing, maybe for production. From the technical side, we have a good handle now how that would look. However, even if we decide to publish to docker, I don't think we'd push directly. Maybe push to our docker registry then mirror to docker hub. That is something we can figure out a little later. > >> 3. If we host in in infra, in case someone actually uses it (there > >> will be people like that), that will eat up lot of network traffic > >> potentially > > > > We can monitor this and adjust as needed. > > > >> 4. With local caching of images (working already) in nodepools we > >> loose complexity of mirroring registries across nodepools > >> > >> So bottom line, having dockerhub/quay.io is simply easier. > >> > > See comment above. > > > >> > Doug > >> > > >> > __ > >> > OpenStack Development Mailing List (not for usage questions) > >> > Unsubscribe: > >> > openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > >> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > >> > >> __ > >> OpenStack Development Mailing List (not for usage questions) > >> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > > > __ > > OpenStack Development Mailing List (not for usage questions) > > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > __ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [tc][infra][release][security][stable][kolla][loci][tripleo][docker][kubernetes] do we want to be publishing binary container images?
On 2017-05-18 18:04:35 -0400 (-0400), Paul Belanger wrote: [...] > if we decide to publish to docker, I don't think we'd push > directly. Maybe push to our docker registry then mirror to docker > hub. That is something we can figure out a little later. [...] Ideally by iterating on https://review.openstack.org/447524 where the details for what that would look like are being hashed out. -- Jeremy Stanley signature.asc Description: Digital signature __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev