Re: Specs? Re: salsa.debian.org maintenance (GitLab 11.1.4 upgrade, external storage migration)
On 2018-08-20 20:58:47 +0200 (+0200), Bastian Blank wrote: > On Mon, Aug 20, 2018 at 08:55:46PM +0200, Alexander Wirt wrote: > > if you can replace ee with ce in the url it is also valid for > > ce. jftr afaik gitlab uses fog[1] for cloud storage, maybe that > > knowledge helps. > > They use CarrierWave, which supports the following fog backends: > > - AWS (aka S3) > - Google > - Rackspace > - OpenStack I've just spoken to the CEO of a service provider in Canada who is interested in talking about donating OpenStack Swift object storage for this (the storage documentation[*] for the fog-openstack backend indicates it's a supported option). The same provider has also donated a variety of hosting resources to the Linux Foundation, OpenStack Foundation and other free software communities for many years now and is remarkably stable. If this is an alternative you're willing to consider, feel free to follow up with me privately and I'll make introductions. [*] https://github.com/fog/fog-openstack/blob/master/docs/storage.md -- Jeremy Stanley signature.asc Description: PGP signature
Re: Specs? Re: salsa.debian.org maintenance (GitLab 11.1.4 upgrade, external storage migration)
On Mon, Aug 20, 2018 at 08:55:46PM +0200, Alexander Wirt wrote: > if you can replace ee with ce in the url it is also valid for ce. > jftr afaik gitlab uses fog[1] for cloud storage, maybe that knowledge helps. They use CarrierWave, which supports the following fog backends: - AWS (aka S3) - Google - Rackspace - OpenStack Bastian -- Killing is stupid; useless! -- McCoy, "A Private Little War", stardate 4211.8
Re: Specs? Re: salsa.debian.org maintenance (GitLab 11.1.4 upgrade, external storage migration)
On 2018-08-20 20:55:46 +0200 (+0200), Alexander Wirt wrote: > On Mon, 20 Aug 2018, Jeremy Stanley wrote: > > > On 2018-08-20 20:05:42 +0200 (+0200), Alexander Wirt wrote: [...] > > > https://docs.gitlab.com/ee/administration/repository_storage_paths.html > > > https://docs.gitlab.com/ee/workflow/lfs/lfs_administration.html#storing-lfs-objects-in-remote-object-storage > > > https://gitlab.com/gitlab-org/omnibus-gitlab/blob/master/files/gitlab-config-template/gitlab.rb.template#L176 > > > https://salsa.debian.org/salsa/salsa-ansible/compare/e709949d0e174f9503b757c95cebee79f0ffe9b0...aafc7392e90efc21fa7e1858eee214029b29764c > > > > > > its all there. > > > > Part of the open-core challenge I'm afraid. I already spoke with one > > "cloud" service provider last week who was willing to follow up to > > this thread with an offer to donate whatever storage types are > > needed, but we couldn't find documentation explaining supported > > storage options for Gitlab *CE* (note the "ee" in those > > documentation URLs, they're the same ones I already found). Is it to > > be assumed, generally, that Gitlab's Enterprise Edition > > documentation is also appropriate for Community Edition deployments? > https://docs.gitlab.com/ce/administration/repository_storage_paths.html > > if you can replace ee with ce in the url it is also valid for ce. > jftr afaik gitlab uses fog[1] for cloud storage, maybe that knowledge helps. > > Alex > > [1] https://github.com/fog/fog I had actually tried that on a whim with some of the EE documents I found last week, but I guess I tried it on the wrong ones. In particular https://docs.gitlab.com/ce/workflow/lfs/lfs_administration.html#storing-lfs-objects-in-remote-object-storage does seem to confirm CE supports whatever Fog does, so pretty much anything in that case. Thanks! (Also, no need to Cc me, I've been subscribed to and at least skim most everything which passes through this list for something like two decades now.) -- Jeremy Stanley signature.asc Description: PGP signature
Re: Specs? Re: salsa.debian.org maintenance (GitLab 11.1.4 upgrade, external storage migration)
On Mon, 20 Aug 2018, Jeremy Stanley wrote: > On 2018-08-20 20:05:42 +0200 (+0200), Alexander Wirt wrote: > > On Mon, 20 Aug 2018, Thomas Goirand wrote: > [...] > > > Could you please at least define what is "some of the large data stores" > > > and explain where it is configured in Gitlab? A possible pointer to the > > > Gitlab documentation would probably help as well. > > I find it very frustrating that you expect us to tell you where the docs > > are. > > However: > > > > https://docs.gitlab.com/ee/administration/repository_storage_paths.html > > https://docs.gitlab.com/ee/workflow/lfs/lfs_administration.html#storing-lfs-objects-in-remote-object-storage > > https://gitlab.com/gitlab-org/omnibus-gitlab/blob/master/files/gitlab-config-template/gitlab.rb.template#L176 > > https://salsa.debian.org/salsa/salsa-ansible/compare/e709949d0e174f9503b757c95cebee79f0ffe9b0...aafc7392e90efc21fa7e1858eee214029b29764c > > > > its all there. > > Part of the open-core challenge I'm afraid. I already spoke with one > "cloud" service provider last week who was willing to follow up to > this thread with an offer to donate whatever storage types are > needed, but we couldn't find documentation explaining supported > storage options for Gitlab *CE* (note the "ee" in those > documentation URLs, they're the same ones I already found). Is it to > be assumed, generally, that Gitlab's Enterprise Edition > documentation is also appropriate for Community Edition deployments? https://docs.gitlab.com/ce/administration/repository_storage_paths.html if you can replace ee with ce in the url it is also valid for ce. jftr afaik gitlab uses fog[1] for cloud storage, maybe that knowledge helps. Alex [1] https://github.com/fog/fog signature.asc Description: PGP signature
Re: Specs? Re: salsa.debian.org maintenance (GitLab 11.1.4 upgrade, external storage migration)
On 2018-08-20 20:05:42 +0200 (+0200), Alexander Wirt wrote: > On Mon, 20 Aug 2018, Thomas Goirand wrote: [...] > > Could you please at least define what is "some of the large data stores" > > and explain where it is configured in Gitlab? A possible pointer to the > > Gitlab documentation would probably help as well. > I find it very frustrating that you expect us to tell you where the docs are. > However: > > https://docs.gitlab.com/ee/administration/repository_storage_paths.html > https://docs.gitlab.com/ee/workflow/lfs/lfs_administration.html#storing-lfs-objects-in-remote-object-storage > https://gitlab.com/gitlab-org/omnibus-gitlab/blob/master/files/gitlab-config-template/gitlab.rb.template#L176 > https://salsa.debian.org/salsa/salsa-ansible/compare/e709949d0e174f9503b757c95cebee79f0ffe9b0...aafc7392e90efc21fa7e1858eee214029b29764c > > its all there. Part of the open-core challenge I'm afraid. I already spoke with one "cloud" service provider last week who was willing to follow up to this thread with an offer to donate whatever storage types are needed, but we couldn't find documentation explaining supported storage options for Gitlab *CE* (note the "ee" in those documentation URLs, they're the same ones I already found). Is it to be assumed, generally, that Gitlab's Enterprise Edition documentation is also appropriate for Community Edition deployments? -- Jeremy Stanley signature.asc Description: PGP signature
Re: Specs? Re: salsa.debian.org maintenance (GitLab 11.1.4 upgrade, external storage migration)
On Mon, 20 Aug 2018, Thomas Goirand wrote: > On 08/20/2018 11:40 AM, Alexander Wirt wrote: > > On Mon, 20 Aug 2018, Ulrike Uhlig wrote: > > > >> Hi! > >> > >> Bastian Blank: > >>> On Sat, Aug 18, 2018 at 11:34:53PM +0200, Thomas Goirand wrote: > Ok, I'm getting in touch with the DSA team to see how it can be done. > Let's see first if we have hardware, and then how I can help for the > setup and maintenance. > >>> > >>> If you want to do something, please show us the plan _before_ you are > >>> implementing anything. We have to vet that it is actually usable for > >>> the purpose. So just running in one direction does not help your case. > >> > >> Has the Salsa team redacted specs prior to implementing the current > >> cloud storage solution? If yes, where can these specs be found? > >> If you haven't redacted such specs, would you be willing to provide > >> those somewhere, even in a keywordish manner? Otherwise, is there any > >> technical documentation (or as a fallback: code & manifests) about this > >> implementation publicly accessible? > >> > >> Having such documents might help avoid running in the wrong direction > >> and would also allow to gather more overall insight. > > No, but gitlab is open source, you should find all the specs you need in the > > docs. > > > > Alex > > Alex, > > We do run a Gitlab instance at $work, so I may be able to try on a local > setup before affecting all of Debian. However, I'd appreciate a bit of > pointers. I've asked numerous questions, and so far got zero answer. > > Since you've configured it, you probably know where its done. Having a > quick look at the documentation over here: > https://docs.gitlab.com/ce/administration/ > > I haven't find in the TOC something that looks relevant. > > The only info I have from you is: > > > We will also start moving some of the large data stores with public > > accessible files off to Google Cloud storage. Using an external > > storage allows us to store a much larger amount of data in our GitLab > > instance. All access to it will be proxied, without providing user > > identifying data to the storage provider. > > Could you please at least define what is "some of the large data stores" > and explain where it is configured in Gitlab? A possible pointer to the > Gitlab documentation would probably help as well. I find it very frustrating that you expect us to tell you where the docs are. However: https://docs.gitlab.com/ee/administration/repository_storage_paths.html https://docs.gitlab.com/ee/workflow/lfs/lfs_administration.html#storing-lfs-objects-in-remote-object-storage https://gitlab.com/gitlab-org/omnibus-gitlab/blob/master/files/gitlab-config-template/gitlab.rb.template#L176 https://salsa.debian.org/salsa/salsa-ansible/compare/e709949d0e174f9503b757c95cebee79f0ffe9b0...aafc7392e90efc21fa7e1858eee214029b29764c its all there. Alex
Re: Specs? Re: salsa.debian.org maintenance (GitLab 11.1.4 upgrade, external storage migration)
On 08/20/2018 11:40 AM, Alexander Wirt wrote: > On Mon, 20 Aug 2018, Ulrike Uhlig wrote: > >> Hi! >> >> Bastian Blank: >>> On Sat, Aug 18, 2018 at 11:34:53PM +0200, Thomas Goirand wrote: Ok, I'm getting in touch with the DSA team to see how it can be done. Let's see first if we have hardware, and then how I can help for the setup and maintenance. >>> >>> If you want to do something, please show us the plan _before_ you are >>> implementing anything. We have to vet that it is actually usable for >>> the purpose. So just running in one direction does not help your case. >> >> Has the Salsa team redacted specs prior to implementing the current >> cloud storage solution? If yes, where can these specs be found? >> If you haven't redacted such specs, would you be willing to provide >> those somewhere, even in a keywordish manner? Otherwise, is there any >> technical documentation (or as a fallback: code & manifests) about this >> implementation publicly accessible? >> >> Having such documents might help avoid running in the wrong direction >> and would also allow to gather more overall insight. > No, but gitlab is open source, you should find all the specs you need in the > docs. > > Alex Alex, We do run a Gitlab instance at $work, so I may be able to try on a local setup before affecting all of Debian. However, I'd appreciate a bit of pointers. I've asked numerous questions, and so far got zero answer. Since you've configured it, you probably know where its done. Having a quick look at the documentation over here: https://docs.gitlab.com/ce/administration/ I haven't find in the TOC something that looks relevant. The only info I have from you is: > We will also start moving some of the large data stores with public > accessible files off to Google Cloud storage. Using an external > storage allows us to store a much larger amount of data in our GitLab > instance. All access to it will be proxied, without providing user > identifying data to the storage provider. Could you please at least define what is "some of the large data stores" and explain where it is configured in Gitlab? A possible pointer to the Gitlab documentation would probably help as well. Cheers, Thomas Goirand (zigo)
Re: Specs? Re: salsa.debian.org maintenance (GitLab 11.1.4 upgrade, external storage migration)
On Mon, 20 Aug 2018, Ulrike Uhlig wrote: > Hi! > > Bastian Blank: > > On Sat, Aug 18, 2018 at 11:34:53PM +0200, Thomas Goirand wrote: > >> Ok, I'm getting in touch with the DSA team to see how it can be done. > >> Let's see first if we have hardware, and then how I can help for the > >> setup and maintenance. > > > > If you want to do something, please show us the plan _before_ you are > > implementing anything. We have to vet that it is actually usable for > > the purpose. So just running in one direction does not help your case. > > Has the Salsa team redacted specs prior to implementing the current > cloud storage solution? If yes, where can these specs be found? > If you haven't redacted such specs, would you be willing to provide > those somewhere, even in a keywordish manner? Otherwise, is there any > technical documentation (or as a fallback: code & manifests) about this > implementation publicly accessible? > > Having such documents might help avoid running in the wrong direction > and would also allow to gather more overall insight. No, but gitlab is open source, you should find all the specs you need in the docs. Alex
Specs? Re: salsa.debian.org maintenance (GitLab 11.1.4 upgrade, external storage migration)
Hi! Bastian Blank: > On Sat, Aug 18, 2018 at 11:34:53PM +0200, Thomas Goirand wrote: >> Ok, I'm getting in touch with the DSA team to see how it can be done. >> Let's see first if we have hardware, and then how I can help for the >> setup and maintenance. > > If you want to do something, please show us the plan _before_ you are > implementing anything. We have to vet that it is actually usable for > the purpose. So just running in one direction does not help your case. Has the Salsa team redacted specs prior to implementing the current cloud storage solution? If yes, where can these specs be found? If you haven't redacted such specs, would you be willing to provide those somewhere, even in a keywordish manner? Otherwise, is there any technical documentation (or as a fallback: code & manifests) about this implementation publicly accessible? Having such documents might help avoid running in the wrong direction and would also allow to gather more overall insight. Cheers, Ulrike
Re: salsa.debian.org maintenance (GitLab 11.1.4 upgrade, external storage migration)
On 08/19/2018 09:53 AM, Bastian Blank wrote:
> On Sat, Aug 18, 2018 at 11:55:19PM +0200, Thomas Goirand wrote:
>> On 08/18/2018 01:11 PM, Bastian Blank wrote:
>> First, there's dozens of OpenStack public cloud out there, so you're not
>> locked-in with a single operator.
>
> There exists thousand variants how to setup an OpenStack instance.
Really? It's been years I'm working on OpenStack, that's not what I saw.
Could you explain a bit more?
> Just leave out some specific service and it's not longer compatible with your
> project.
What kind of specific server do you have in mind here?
> So you not just need to find an OpenStack operator, you need
> to find an operator who provides the correct set of services you need.
The bare minimum one is to expect is: keystone, nova, glance, cinder,
neutron, heat. And generally, you also find swift. I don't see how a
public cloud provider would not provide at least these. Barbican is also
slowly becoming a standard too.
Now, if your provider also has trove, designate, magnum, manilla,
murano, neutron-{fwaas,lbaas,vpnaas} and so on, that's only a bonus, but
you often can deal without them.
>> Then there's not a single contributor
>> to the OpenStack source code, the contributors are quite diverse.
>
> A friend of mine called OpenStack a dysfunctional open source project.
> Especially the number of different people giving the shots and running
> in different directions does not help. (No project in OpenStack looks
> alike. Just look at all the different variants for how to maintain the
> database schema.)
This is somewhat exaggerated. Yes, because of historical reasons,
there's both Alembic and SQLAlchemy-migrate. Though oslo.db supports
both, and in practice, it's not a problem. Though in general, I do see
that most projects are disciplined and do work on the same direction. If
you consider the amount of dependencies (539 for the last Rocky
release), it's kind of nice that they are all worked out as a whole.
Lots of things have been standardized. The technical committee is also
there to steer in a single direction.
So, if that's your feeling, I'd suggest you look a 2nd time. I really
don't think OpenStack is a dysfunctional project.
Cheers,
Thomas Goirand (zigo)
Re: salsa.debian.org maintenance (GitLab 11.1.4 upgrade, external storage migration)
Le dimanche 19 août 2018 à 09:11:23+0200, Alexander Wirt a écrit : > On Sat, 18 Aug 2018, Thomas Goirand wrote: > > > On 08/18/2018 07:42 AM, Alexander Wirt wrote: > > >> I also don't understand why we're not attempting to build a Ceph cluster > > >> at UBC. Why not? > > > go ahead. if it works well we can switch to it. > > > > > > Alex > > > > Ok, I'm getting in touch with the DSA team to see how it can be done. > > Let's see first if we have hardware, and then how I can help for the > > setup and maintenance. > Of course we also need docker. You will have to ensure that everythings work > with gitlab, that its performant enough and you will have to maintain it. > Otherwise we won't switch. Hi, I'm not sure how serious you are about adopting a home made solution. If you really consider this as preferred, I'd be happy to try helping, to the extents of my capabilities, setting such a solution up. Would you really consider such a thing seriously? Cheers. -- Pierre-Elliott Bécue GPG: 9AE0 4D98 6400 E3B6 7528 F493 0D44 2664 1949 74E2 It's far easier to fight for one's principles than to live up to them. signature.asc Description: PGP signature
Re: salsa.debian.org maintenance (GitLab 11.1.4 upgrade, external storage migration)
On 08/19/2018 09:11 AM, Alexander Wirt wrote: > On Sat, 18 Aug 2018, Thomas Goirand wrote: > >> On 08/18/2018 07:42 AM, Alexander Wirt wrote: I also don't understand why we're not attempting to build a Ceph cluster at UBC. Why not? >>> go ahead. if it works well we can switch to it. >>> >>> Alex >> >> Ok, I'm getting in touch with the DSA team to see how it can be done. >> Let's see first if we have hardware, and then how I can help for the >> setup and maintenance. > Of course we also need docker. Hang on here. Are you saying that Salsa is using GKE? Or is it still for storage? In the later case, how is setting-up a Ceph cluster or Swift related to the storage need? Could you also describe what the need is? If I understood correctly, it would be an object store, right? Or do you also need block storage? In the former case, swift is best. In the later case, Ceph is preferred. We could even do both if we need to, and have enough hardware to play with. > You will have to ensure that everythings work > with gitlab, that its performant enough and you will have to maintain it. > Otherwise we won't switch. I get that. I suppose you remembered that I volunteered twice already for helping with Salsa. It'd be with great pleasure that I get involved. The only thing, is that I need hardware to play with, and probably will need the root to be able to do the setup. This means dealing with DSA team, and I'm not sure how they see it yet. It'd be great if we could have their view here. Cheers, Thomas Goirand (zigo)
English language & completely off-topic Re: salsa.debian.org maintenance (GitLab 11.1.4 upgrade, external storage migration)
Hi, Bastian Blank: >> And there's what Jeremy replied to you. We shall not endorse non-free. > > No, he just said we should prefer to use free ones. > > Endorse is something different, please read yourself > https://www.merriam-webster.com/dictionary/endorse or As your link says Endorse: to approve openly, to recommend. The use of this word is entirely correct. Cheers, Ulrike
Re: salsa.debian.org maintenance (GitLab 11.1.4 upgrade, external storage migration)
On Sat, Aug 18, 2018 at 11:55:19PM +0200, Thomas Goirand wrote: > On 08/18/2018 01:11 PM, Bastian Blank wrote: > First, there's dozens of OpenStack public cloud out there, so you're not > locked-in with a single operator. There exists thousand variants how to setup an OpenStack instance. Just leave out some specific service and it's not longer compatible with your project. So you not just need to find an OpenStack operator, you need to find an operator who provides the correct set of services you need. > Then there's not a single contributor > to the OpenStack source code, the contributors are quite diverse. A friend of mine called OpenStack a dysfunctional open source project. Especially the number of different people giving the shots and running in different directions does not help. (No project in OpenStack looks alike. Just look at all the different variants for how to maintain the database schema.) > To the contrary, if you're using a proprietary cloud like AWS, you have > no choice but to continue with them. If you depend on the _interfaces_ they provide. AWS, at least most parts of it, is easily replaceable. > > Endorse is something different, please read yourself > > https://www.merriam-webster.com/dictionary/endorse or > > http://www.learnersdictionary.com/definition/endorse > > Well, it's a question of view. To me, having Salsa to host some of its > data on Google is a kind of tacit endorsement. Even worse, it gives the > message that, from the viewpoint of the whole Debian community, it's ok > to host there. Clearly, I'm not the only one bothered in this way. No, it's how the word is defined in the english language. Bastian -- Emotions are alien to me. I'm a scientist. -- Spock, "This Side of Paradise", stardate 3417.3
Re: salsa.debian.org maintenance (GitLab 11.1.4 upgrade, external storage migration)
Moin On Sat, Aug 18, 2018 at 11:34:53PM +0200, Thomas Goirand wrote: > Ok, I'm getting in touch with the DSA team to see how it can be done. > Let's see first if we have hardware, and then how I can help for the > setup and maintenance. If you want to do something, please show us the plan _before_ you are implementing anything. We have to vet that it is actually usable for the purpose. So just running in one direction does not help your case. Bastian -- I'm a soldier, not a diplomat. I can only tell the truth. -- Kirk, "Errand of Mercy", stardate 3198.9
Re: salsa.debian.org maintenance (GitLab 11.1.4 upgrade, external storage migration)
On Sat, 18 Aug 2018, Thomas Goirand wrote: > On 08/18/2018 07:42 AM, Alexander Wirt wrote: > >> I also don't understand why we're not attempting to build a Ceph cluster > >> at UBC. Why not? > > go ahead. if it works well we can switch to it. > > > > Alex > > Ok, I'm getting in touch with the DSA team to see how it can be done. > Let's see first if we have hardware, and then how I can help for the > setup and maintenance. Of course we also need docker. You will have to ensure that everythings work with gitlab, that its performant enough and you will have to maintain it. Otherwise we won't switch. Alex
Re: salsa.debian.org maintenance (GitLab 11.1.4 upgrade, external storage migration)
* Bastian Blank: " Re: salsa.debian.org maintenance (GitLab 11.1.4 upgrade, external storage migration)" (Sat, 18 Aug 2018 08:26:11 +0200): Erroneously sent as PM. > On Sat, Aug 18, 2018 at 02:16:33AM +0200, Mathias Behrle wrote: > > > Please explain. Google Cloud storage is just a large disk. The > > > analytics stuff can access the data if it got the authorization to > > > access it. > > I have quite some difficulties to believe that Google respects privacy > > settings: > > https://apnews.com/828aefab64d4411bac257a07c1af0ecb/AP-Exclusive:-Google-tracks-your-movements,-like-it-or-not > > > > I have no idea what you are trying to tell me. This article not even > once talks about the storage. You spoke about authorization to Google and I gave an example where Google obviously doesn't care about authorization. Mathias -- Mathias Behrle PGP/GnuPG key availabable from any keyserver, ID: 0xD6D09BE48405BBF6 AC29 7E5C 46B9 D0B6 1C71 7681 D6D0 9BE4 8405 BBF6
Re: salsa.debian.org maintenance (GitLab 11.1.4 upgrade, external storage migration)
On 08/18/2018 08:26 AM, Bastian Blank wrote: > On Sat, Aug 18, 2018 at 02:16:33AM +0200, Mathias Behrle wrote: >>> Please explain. Google Cloud storage is just a large disk. The >>> analytics stuff can access the data if it got the authorization to >>> access it. >> I have quite some difficulties to believe that Google respects privacy >> settings: >> https://apnews.com/828aefab64d4411bac257a07c1af0ecb/AP-Exclusive:-Google-tracks-your-movements,-like-it-or-not > > This article not even once talks about the storage. Let's blindly trust then...
Re: salsa.debian.org maintenance (GitLab 11.1.4 upgrade, external storage migration)
On 08/18/2018 01:11 PM, Bastian Blank wrote: >>Also, using a free >> implementation avoids vendor lock-in, > > Could you please be more specific on what vendor lock-in you mean? > > If I build an application using Kubernetes and Helm, I'm using only free > components, but I'm still locked-in, as there is only one > implementation. If I do the same using the OpenStack API, I'm locked-in > as there exist only one implementation of this API. First, there's dozens of OpenStack public cloud out there, so you're not locked-in with a single operator. Then there's not a single contributor to the OpenStack source code, the contributors are quite diverse. To the contrary, if you're using a proprietary cloud like AWS, you have no choice but to continue with them. > In our case all the components can utilize multiple backends, so we can > freely choose which one to use. Including completely free alternatives > like ceph with radosgw or minio. I'd like to understand better what the need and usage is. Could you please expand? Do you need to store blobs, like with Swift? Maybe swift is more adapted to the use case, if we only need to store blobs? Swift is also a way more easy to deploy and maintain than a Ceph cluster, which requires careful monitoring of daemons if you don't want it to fall apart. >> and provides proven (and tested on >> a CI, for the case of OpenStack) interoperability. > > I'm pretty sure all the Cloud vendors do heavy tests on all sorts. To > be exact, I know they do. They just don't show you all of it. So this > is no unique selling point for OpenStack. I was talking specifically about interoperability tests from one OpenStack version to the next. OpenStack clients are designed to work with *any* version of OpenStack, and they are supposed to work even some very old deployments. And that's the thing that is tested in the CI. I may be wrong, but so far, I haven't head that any of the big 4 non-free clouds having such thing for their client API, and even if they did, it would make little sense, as they don't have the interoperability problem to take care of (they just need to be compatible with themselves). Other free IaaS implementation may have the feature also, but free-software public cloud providers are almost always using OpenStack anyway, so interoperability, really, is a unique selling point here for OpenStack. >> And there's what Jeremy replied to you. We shall not endorse non-free. > > No, he just said we should prefer to use free ones. > > Endorse is something different, please read yourself > https://www.merriam-webster.com/dictionary/endorse or > http://www.learnersdictionary.com/definition/endorse Well, it's a question of view. To me, having Salsa to host some of its data on Google is a kind of tacit endorsement. Even worse, it gives the message that, from the viewpoint of the whole Debian community, it's ok to host there. Clearly, I'm not the only one bothered in this way. >> Though what I agree very much about, is that we'd get more freedom if we >> were self-hosting fully. One always do. > > You primarily got more work to do. Sure, of course.
Re: salsa.debian.org maintenance (GitLab 11.1.4 upgrade, external storage migration)
On 08/18/2018 07:42 AM, Alexander Wirt wrote: >> I also don't understand why we're not attempting to build a Ceph cluster >> at UBC. Why not? > go ahead. if it works well we can switch to it. > > Alex Ok, I'm getting in touch with the DSA team to see how it can be done. Let's see first if we have hardware, and then how I can help for the setup and maintenance. Cheers, Thomas Goirand (zigo)
Re: salsa.debian.org maintenance (GitLab 11.1.4 upgrade, external storage migration)
Hi Thomas On Fri, Aug 17, 2018 at 11:12:01PM +0200, Thomas Goirand wrote: > Wouter, I very much do not agree with your argumentation. Please read > this video: > https://meetings-archive.debian.net/pub/debian-meetings/2018/DebConf18/2018-07-30/server-freedom-why-choosing-the-cloud-op.webm Do you have some abstract for this? I watched it and still don't know what you want to say here. >Also, using a free > implementation avoids vendor lock-in, Could you please be more specific on what vendor lock-in you mean? If I build an application using Kubernetes and Helm, I'm using only free components, but I'm still locked-in, as there is only one implementation. If I do the same using the OpenStack API, I'm locked-in as there exist only one implementation of this API. In our case all the components can utilize multiple backends, so we can freely choose which one to use. Including completely free alternatives like ceph with radosgw or minio. > and provides proven (and tested on > a CI, for the case of OpenStack) interoperability. I'm pretty sure all the Cloud vendors do heavy tests on all sorts. To be exact, I know they do. They just don't show you all of it. So this is no unique selling point for OpenStack. > And there's what Jeremy replied to you. We shall not endorse non-free. No, he just said we should prefer to use free ones. Endorse is something different, please read yourself https://www.merriam-webster.com/dictionary/endorse or http://www.learnersdictionary.com/definition/endorse > Though what I agree very much about, is that we'd get more freedom if we > were self-hosting fully. One always do. You primarily got more work to do. Bastian -- You! What PLANET is this! -- McCoy, "The City on the Edge of Forever", stardate 3134.0
Re: salsa.debian.org maintenance (GitLab 11.1.4 upgrade, external storage migration)
On Fri, Aug 17, 2018 at 11:12:01PM +0200, Thomas Goirand wrote: > On 08/17/2018 04:11 PM, Wouter Verhelst wrote: > > But if we're going to > > be using an external cloud provider for such things, then it doesn't > > matter whether that external cloud provider runs a free software cloud > > implementation or a proprietary one, since we wouldn't have any of the > > benefits of the free software implementation anyway. In that background, > > it makes much more sense to use free software which can talk to multiple > > cloud storage implementations; that gives us the ability to move from > > one cloud provider to another by simply copying some data followed by > > the flick of a switch (or "just" that flick, if the free software in > > question copies the data for you). > > > > It is true that the whole "cloud" thing does have some impact on free > > software, and it is indeed also true that this is something that should > > be considered when deciding on a strategy; however, it is *not* correct > > to assume that a cloud provider which uses free software is in any way > > less of a problem in that context than a cloud provider which does not. > > Both are cases of you relying on a third party for part of the service > > which you provide, and so there is no real difference. > > Wouter, I very much do not agree with your argumentation. Please read > this video: You mean watch, I take it? ;-) > https://meetings-archive.debian.net/pub/debian-meetings/2018/DebConf18/2018-07-30/server-freedom-why-choosing-the-cloud-op.webm > > Having the underlying infrastructure to fully run free software is super > important, for many reasons. For example, because you can completely > reproduce it on your own hardware if you feel like it's the correct > thing to do, and then migrate your data to it. Also, using a free > implementation avoids vendor lock-in, and provides proven (and tested on > a CI, for the case of OpenStack) interoperability. That does make sense. However, if the free software on top of whatever cloud infrastructure we choose to use already supports multiple cloud providers (as is the case here), then you already do avoid that same vendor lock-in. It would be totally different if we were talking about something that can only be done by Google Compute Engine; say, if the salsa administrators wrote the necessary code themselves and did not even consider looking at a free implementation. Were that the case, I would agree with you. But that's not what this is about. > And there's what Jeremy replied to you. We shall not endorse non-free. > > I've been fighting for software freedom on the servers for nearly all of > my professional career, and reading what you wrote makes me really sad. > IMO you haven't understood what the problem is. I do; I just don't agree that, *in this specific instance*, it applies. > Though what I agree very much about, is that we'd get more freedom if we > were self-hosting fully. One always do. Exactly. We should encourage that when we can; but we should not fault the salsa administrators for deciding they don't have the time for that, and for choosing a third party cloud infrastructure, *for storage only*. When the cloud provider is used for only a minimal part of the problem space (as is the case here), and when the software in use allows for easy switching to another cloud provider (as is the case here), then the advantages of choosing a cloud provider that uses free software to provide the required service become very slim indeed. What use is free software when you can't exercise the rights anyway, as is the case with a third-party installation of free software cloud infrastructure? One could say that we might want to do so anyway, for political reasons, and I would definitely agree with you. But that ship has sailed when we chose gitlab over the really free software alternatives that also exist. -- Could you people please use IRC like normal people?!? -- Amaya Rodrigo Sastre, trying to quiet down the buzz in the DebConf 2008 Hacklab
Re: salsa.debian.org maintenance (GitLab 11.1.4 upgrade, external storage migration)
On Sat, Aug 18, 2018 at 02:16:33AM +0200, Mathias Behrle wrote: > > Please explain. Google Cloud storage is just a large disk. The > > analytics stuff can access the data if it got the authorization to > > access it. > I have quite some difficulties to believe that Google respects privacy > settings: > https://apnews.com/828aefab64d4411bac257a07c1af0ecb/AP-Exclusive:-Google-tracks-your-movements,-like-it-or-not I have no idea what you are trying to tell me. This article not even once talks about the storage. Bastian -- The sooner our happiness together begins, the longer it will last. -- Miramanee, "The Paradise Syndrome", stardate 4842.6
Re: salsa.debian.org maintenance (GitLab 11.1.4 upgrade, external storage migration)
On Fri, 17 Aug 2018, Thomas Goirand wrote: > On 08/17/2018 10:52 AM, Andrey Rahmatullin wrote: > > On Fri, Aug 17, 2018 at 08:27:00AM +, Ulrike Uhlig wrote: > >> While I understand the simplicity of using $company's cloud storage, I'd > >> rather not rely on some external company and in particular not on this > >> one. This company does not exactly represent what I would call ethical, > >> non-proprietary, and decentralized. > > Is that a problem? > > > >> Are there no partners that would kindly provide such storage to Debian > >> (Gandi?). > > Are they ethical, non-proprietary, and decentralized? > > To me, none of the above. > > On 08/17/2018 11:30 AM, Yao Wei wrote: > > Still, it is up to their implementation how we can access their > > storage, and as long as we can access it with free software > > (JavaScript stuff could be a pitfall though) it shouldn't be too much > > problem for us. > > I very much do not agree with this. The full software stack used by the > provider *MUST* be free as well. I have made a full talk at Debconf > explaining why it is important, I would appreciate if you could have a > look to the video. > > On 08/17/2018 03:46 PM, Ulrike Uhlig wrote: > > I feel like we're currently balancing on a thin cobweb of fait > > accompli. > > I very much agree with this, and I kind of feel we're all gamed into > using Salsa, and then it moves to using non-free solutions. And I'm not > even going back to the previous thread where Alexander wrote he would > *never* switch to a packaged solution. That one as well, is an > unilateral decision. > > > Should we make it known and visible to people who use Salsa that some > > of their data might be stored at a 3rd party company? Is this a > > consent issue? > > It's more than this, unfortunately. After we've all been invited to > migrate to Salsa, and all spent time on it, now it's partly non-free. If > I knew it would go this way, it's possible that I would have chosen > another place to host my Debian work. Possibly self-hosting it. > > On 08/17/2018 03:46 PM, Ulrike Uhlig wrote: > > Have Salsa maintainers enabled the least invasive privacy features for > > this service? > > Do you really trust this? I don't... > > Also, have we ever thought that Google is completely banned in China? > Can users in China access the data? If it's direct access to things > hosted by Google, then the answer is probably that it's also blocked. > > On 08/17/2018 03:49 PM, Alexander Wirt wrote: > > why should gandi be better? Do you have access to all of their source > > code (managementfrontend, storagebackend, billingbackend and so on?) > > > > Unless debian is doing the whole thing on its own, we are out of luck. > > > > Alex > > As I mentioned in my talk at Debconf18, there's 18 listed public cloud > providers using OpenStack here: > > https://www.openstack.org/marketplace/public-clouds/ > > OVH, CloudWatt, Rackspace, Vexxhost, and also probably the company I > work for (Infomaniak) are potential candidates for sponsoring storage, > and all of them are using free-software hosting platforms. > > I also don't understand why we're not attempting to build a Ceph cluster > at UBC. Why not? go ahead. if it works well we can switch to it. Alex P.S. I will not react to the other nonsense in that mail.
Re: salsa.debian.org maintenance (GitLab 11.1.4 upgrade, external storage migration)
* Bastian Blank: " Re: salsa.debian.org maintenance (GitLab 11.1.4 upgrade, external storage migration)" (Fri, 17 Aug 2018 22:58:18 +0200): > > Google Cloud storage is tightly linked to their AI & big data analytics > > features which I personally find highly questionable. > > Please explain. Google Cloud storage is just a large disk. The > analytics stuff can access the data if it got the authorization to > access it. I have quite some difficulties to believe that Google respects privacy settings: https://apnews.com/828aefab64d4411bac257a07c1af0ecb/AP-Exclusive:-Google-tracks-your-movements,-like-it-or-not Mathias -- Mathias Behrle ✧ Debian Developer PGP/GnuPG key availabable from any keyserver, ID: 0xD6D09BE48405BBF6 AC29 7E5C 46B9 D0B6 1C71 7681 D6D0 9BE4 8405 BBF6
Re: salsa.debian.org maintenance (GitLab 11.1.4 upgrade, external storage migration)
On Fri, Aug 17, 2018 at 01:46:00PM +, Ulrike Uhlig wrote: > Consent > --- > > I feel like we're currently balancing on a thin cobweb of fait accompli. > Are such decisions team internal or do they require the consent of the > project? There is no notion of a project consent in Debian, apart from a GR. > How does this entire issue relate to the GDPR? It does not, as this information does not contain personal data. > How does this relate to our Social Contract? > > - 4. "Our priorities are our users and free software". This is very > vague but one could argue that this paragraph is not entirely > respected. It is. We try to provide our users the best service we can. > Data > > > Google Cloud storage is tightly linked to their AI & big data analytics > features which I personally find highly questionable. Please explain. Google Cloud storage is just a large disk. The analytics stuff can access the data if it got the authorization to access it. > As this > intelligent cute monster feeds on data and metadata, it's part of its > ecosystem to provide free services in order to get more free food. > (Mentioning this because Marco d'Itri was raising the issue of having to > pay for storage.) Google Cloud storage is no free service. We certainly pay for it; with Google's money, but still. > To end with, I believe that a self-hosted solution would address all of > these issues. This is nice, but incorrect. Regards, Bastian -- The sight of death frightens them [Earthers]. -- Kras the Klingon, "Friday's Child", stardate 3497.2
Re: salsa.debian.org maintenance (GitLab 11.1.4 upgrade, external storage migration)
On 08/17/2018 04:11 PM, Wouter Verhelst wrote: > But if we're going to > be using an external cloud provider for such things, then it doesn't > matter whether that external cloud provider runs a free software cloud > implementation or a proprietary one, since we wouldn't have any of the > benefits of the free software implementation anyway. In that background, > it makes much more sense to use free software which can talk to multiple > cloud storage implementations; that gives us the ability to move from > one cloud provider to another by simply copying some data followed by > the flick of a switch (or "just" that flick, if the free software in > question copies the data for you). > > It is true that the whole "cloud" thing does have some impact on free > software, and it is indeed also true that this is something that should > be considered when deciding on a strategy; however, it is *not* correct > to assume that a cloud provider which uses free software is in any way > less of a problem in that context than a cloud provider which does not. > Both are cases of you relying on a third party for part of the service > which you provide, and so there is no real difference. Wouter, I very much do not agree with your argumentation. Please read this video: https://meetings-archive.debian.net/pub/debian-meetings/2018/DebConf18/2018-07-30/server-freedom-why-choosing-the-cloud-op.webm Having the underlying infrastructure to fully run free software is super important, for many reasons. For example, because you can completely reproduce it on your own hardware if you feel like it's the correct thing to do, and then migrate your data to it. Also, using a free implementation avoids vendor lock-in, and provides proven (and tested on a CI, for the case of OpenStack) interoperability. And there's what Jeremy replied to you. We shall not endorse non-free. I've been fighting for software freedom on the servers for nearly all of my professional career, and reading what you wrote makes me really sad. IMO you haven't understood what the problem is. Though what I agree very much about, is that we'd get more freedom if we were self-hosting fully. One always do. Cheers, Thomas Goirand (zigo)
Re: salsa.debian.org maintenance (GitLab 11.1.4 upgrade, external storage migration)
On 08/17/2018 10:52 AM, Andrey Rahmatullin wrote: > On Fri, Aug 17, 2018 at 08:27:00AM +, Ulrike Uhlig wrote: >> While I understand the simplicity of using $company's cloud storage, I'd >> rather not rely on some external company and in particular not on this >> one. This company does not exactly represent what I would call ethical, >> non-proprietary, and decentralized. > Is that a problem? > >> Are there no partners that would kindly provide such storage to Debian >> (Gandi?). > Are they ethical, non-proprietary, and decentralized? To me, none of the above. On 08/17/2018 11:30 AM, Yao Wei wrote: > Still, it is up to their implementation how we can access their > storage, and as long as we can access it with free software > (JavaScript stuff could be a pitfall though) it shouldn't be too much > problem for us. I very much do not agree with this. The full software stack used by the provider *MUST* be free as well. I have made a full talk at Debconf explaining why it is important, I would appreciate if you could have a look to the video. On 08/17/2018 03:46 PM, Ulrike Uhlig wrote: > I feel like we're currently balancing on a thin cobweb of fait > accompli. I very much agree with this, and I kind of feel we're all gamed into using Salsa, and then it moves to using non-free solutions. And I'm not even going back to the previous thread where Alexander wrote he would *never* switch to a packaged solution. That one as well, is an unilateral decision. > Should we make it known and visible to people who use Salsa that some > of their data might be stored at a 3rd party company? Is this a > consent issue? It's more than this, unfortunately. After we've all been invited to migrate to Salsa, and all spent time on it, now it's partly non-free. If I knew it would go this way, it's possible that I would have chosen another place to host my Debian work. Possibly self-hosting it. On 08/17/2018 03:46 PM, Ulrike Uhlig wrote: > Have Salsa maintainers enabled the least invasive privacy features for > this service? Do you really trust this? I don't... Also, have we ever thought that Google is completely banned in China? Can users in China access the data? If it's direct access to things hosted by Google, then the answer is probably that it's also blocked. On 08/17/2018 03:49 PM, Alexander Wirt wrote: > why should gandi be better? Do you have access to all of their source > code (managementfrontend, storagebackend, billingbackend and so on?) > > Unless debian is doing the whole thing on its own, we are out of luck. > > Alex As I mentioned in my talk at Debconf18, there's 18 listed public cloud providers using OpenStack here: https://www.openstack.org/marketplace/public-clouds/ OVH, CloudWatt, Rackspace, Vexxhost, and also probably the company I work for (Infomaniak) are potential candidates for sponsoring storage, and all of them are using free-software hosting platforms. I also don't understand why we're not attempting to build a Ceph cluster at UBC. Why not? Cheers, Thomas Goirand (zigo)
Re: salsa.debian.org maintenance (GitLab 11.1.4 upgrade, external storage migration)
On 2018-08-17 16:11:22 +0200 (+0200), Wouter Verhelst wrote: [...] > It is true that the whole "cloud" thing does have some impact on free > software, and it is indeed also true that this is something that should > be considered when deciding on a strategy; however, it is *not* correct > to assume that a cloud provider which uses free software is in any way > less of a problem in that context than a cloud provider which does not. > Both are cases of you relying on a third party for part of the service > which you provide, and so there is no real difference. Unless there is also a goal to show support for the projects which create those free software backends, by choosing the providers who deploy them over providers who build their own proprietary backends rather than participating in open community development efforts. -- Jeremy Stanley signature.asc Description: PGP signature
Re: salsa.debian.org maintenance (GitLab 11.1.4 upgrade, external storage migration)
On Tue, Aug 14, 2018 at 01:25:22PM +0200, Jonas Meurer wrote: > [...] free software cloud providers [...] No such thing. The whole concept of "cloud provider" is that you have a company which provides "hardware" or "infrastructure" as a service. This service is provided "as is"; you wouldn't be able to make any changes to the service -- and that makes sense, since the service isn't used just by you, it's used by all users of the same company. Debian could indeed produce its own generic storage system somewhere, and use that as a backend for a service that requires a lot of storage, such as salsa; that would certainly improve the "eat our own dogfood" ratio somewhat, and could possibly be a good idea. But if we're going to be using an external cloud provider for such things, then it doesn't matter whether that external cloud provider runs a free software cloud implementation or a proprietary one, since we wouldn't have any of the benefits of the free software implementation anyway. In that background, it makes much more sense to use free software which can talk to multiple cloud storage implementations; that gives us the ability to move from one cloud provider to another by simply copying some data followed by the flick of a switch (or "just" that flick, if the free software in question copies the data for you). It is true that the whole "cloud" thing does have some impact on free software, and it is indeed also true that this is something that should be considered when deciding on a strategy; however, it is *not* correct to assume that a cloud provider which uses free software is in any way less of a problem in that context than a cloud provider which does not. Both are cases of you relying on a third party for part of the service which you provide, and so there is no real difference. -- Could you people please use IRC like normal people?!? -- Amaya Rodrigo Sastre, trying to quiet down the buzz in the DebConf 2008 Hacklab
Re: salsa.debian.org maintenance (GitLab 11.1.4 upgrade, external storage migration)
Hello, Alexander Wirt: > On Fri, 17 Aug 2018, Ulrike Uhlig wrote: >> Jonas Meurer: >>> Am 13.08.2018 um 20:36 schrieb Alexander Wirt: > why should gandi be better? Do you have access to all of their source code > (managementfrontend, storagebackend, billingbackend and so on?) > > Unless debian is doing the whole thing on its own, we are out of luck. Right, it was a bad example, disregard it. u.
Re: salsa.debian.org maintenance (GitLab 11.1.4 upgrade, external storage migration)
Hello, Andrey Rahmatullin: > On Fri, Aug 17, 2018 at 10:12:00AM +, u wrote: While I understand the simplicity of using $company's cloud storage, I'd rather not rely on some external company and in particular not on this one. This company does not exactly represent what I would call ethical, non-proprietary, and decentralized. >>> Is that a problem?>> Rhetorical questions are a means of communication >>> which are not only a >> means of linguistic manipulation, but also counterproductive in such a >> discussion. May you reformulate? > They are not rhetorical. In that case let me answer your question "Is that a problem?". Note that ideally I would not want to create a gigantic discussion thread with people throwing virtual mud at each other and unfortunately this thread has all the potential to end up in exactly that kind of scenario. So I'll try to stay factual and hope others will do the same. <3 First of all, I'm very grateful that Salsa exists and works so well. Thank you for working on it. Thank you too for thinking about solving I/O problems and for considering proxying in first place. And thank you for announcing the technical possibility of switching to another storage. Besides the bad taste in Jonas' mouth, I want to try to frame where the bad taste in my mouth comes from by asking some questions. Note that I have no ready responses to these questions, but I think we need to ask them. If I understand correctly this storage was enabled on August 11th. Consent --- I feel like we're currently balancing on a thin cobweb of fait accompli. Are such decisions team internal or do they require the consent of the project? Should we make it known and visible to people who use Salsa that some of their data might be stored at a 3rd party company? Is this a consent issue? How does this entire issue relate to the GDPR? - I'm not knowledgeable enough in this area so I cannot reply to this question myself. How does this relate to our Social Contract? - 4. "Our priorities are our users and free software". This is very vague but one could argue that this paragraph is not entirely respected. Data Google Cloud storage is tightly linked to their AI & big data analytics features which I personally find highly questionable. As this intelligent cute monster feeds on data and metadata, it's part of its ecosystem to provide free services in order to get more free food. (Mentioning this because Marco d'Itri was raising the issue of having to pay for storage.) Even though the initial email mentioned that only files that are already publicly accessible are stored there, we have no means to know if this changes at some point due to some configuration modification on our side, be it accidental or not. How can we guarantee it won't? Is this process currently transparent? And if not, how can we make it so? Have Salsa maintainers enabled the least invasive privacy features for this service? [0] Is there a means for us as Salsa users to know when those change? Is there a means to know what they are exactly? I'm not only concerned about sending identifying information about Salsa users but also about making it easier for a 3rd party to do metadata analysis. (I agree this can already be done with data that is public, so this is not a privacy issue per se. But: it cannot be done as easily, hence my concern is more about providing free food, i.e. cheap work [1][2] to a company, and again another concern is about consenting to do so.) Access -- While it has been said that all access is currently proxied and no user identifying data provided to Google, how can we know this remains the case in the future? Will we be consulted for consent? Is this process transparent, i.e. can we see this configuration publicly? And if not how can we make it so? Thanks for thinking about this kind of proxying in first place, Salsa team! By proxying, accessing this data will work over Tor and in places where Google exerts censorship, I suppose and hope. If ever the Salsa team considers changing this setup, or Google cloud storage changes their inner functioning, we might lack the possibility for anonymous access which should be taken into account in order to provide free, decentralized, captcha-free access to Debian's source code repositories. Transparency I believe that privacy should always be the default, as you cannot opt-out of already leaked information after the fact: there is no way to go backwards in time. Hence I believe that such issues need to be treated carefully, transparently and consensually. To end with, I believe that a self-hosted solution would address all of these issues. Cheers, Ulrike PS: I have no interest in replying to the question "Are they ethical, non-proprietary, and decentralized?" on this mailing list as I believe it will not advance the discussion any further. Notes: [0] https://cloud.google.com/security/privacy/ [1]
Re: salsa.debian.org maintenance (GitLab 11.1.4 upgrade, external storage migration)
On Fri, 17 Aug 2018, Ulrike Uhlig wrote: > Hi! > > Jonas Meurer: > > Am 13.08.2018 um 20:36 schrieb Alexander Wirt: > >>> Hrmpf! I have to say that I was somewhat surprised by this announcement. > >>> To be honest, I don't like the idea of making our infrastructure as a > >>> project rely on closed and proprietary systems like Google Cloud. Isn't > >>> it important to us as a project anymore to run our infrastructure on > >>> free software and under our own control? [1] > >>> We don't rely on it. There will be a backup on debian infastructure so > > that > >> we will be able to change to different providers at every time. > > > > That's good to know! > > > Your explanation definitely helps with understanding the rationale > > behind your decision to switch to Google Cloud for some storage. And if > > Salsa indeed has I/O problems already, it's much appreciated that you do > > something about it. Again, thanks for this. > > Ack. > > > I just wonder why we don't consider and prefer free solution (either by > > running an own external storage or by using free software cloud > > providers) over the proprietary ones. In my eyes, this conflicts with > > our social contract and with prioritizing Free Software. That's, why I > > brought it up here. > > > > What do others think about it? > > Thanks for bringing it up, Jonas! I absolutely agree with your reasoning > and I also think that it conflicts with Debian's Social Contract; hence > I'd like to see this discussed before silently implementing such a thing. > > While I understand the simplicity of using $company's cloud storage, I'd > rather not rely on some external company and in particular not on this > one. This company does not exactly represent what I would call ethical, > non-proprietary, and decentralized. > > Are there no partners that would kindly provide such storage to Debian > (Gandi?). Sometimes this may take more time, and be harder to implement. > It might be more expensive, and that's exactly the point: whenever > something is free, you've become the product. why should gandi be better? Do you have access to all of their source code (managementfrontend, storagebackend, billingbackend and so on?) Unless debian is doing the whole thing on its own, we are out of luck. Alex
Re: salsa.debian.org maintenance (GitLab 11.1.4 upgrade, external storage migration)
On Fri, Aug 17, 2018 at 10:12:00AM +, u wrote: > >> While I understand the simplicity of using $company's cloud storage, I'd > >> rather not rely on some external company and in particular not on this > >> one. This company does not exactly represent what I would call ethical, > >> non-proprietary, and decentralized. > > Is that a problem? > >> Are there no partners that would kindly provide such storage to Debian > >> (Gandi?). > > Are they ethical, non-proprietary, and decentralized? > > Rhetorical questions are a means of communication which are not only a > means of linguistic manipulation, but also counterproductive in such a > discussion. May you reformulate? They are not rhetorical. -- WBR, wRAR signature.asc Description: PGP signature
Re: salsa.debian.org maintenance (GitLab 11.1.4 upgrade, external storage migration)
Hello, Andrey Rahmatullin: > On Fri, Aug 17, 2018 at 08:27:00AM +, Ulrike Uhlig wrote: >> While I understand the simplicity of using $company's cloud storage, I'd >> rather not rely on some external company and in particular not on this >> one. This company does not exactly represent what I would call ethical, >> non-proprietary, and decentralized. > Is that a problem? >> Are there no partners that would kindly provide such storage to Debian >> (Gandi?). > Are they ethical, non-proprietary, and decentralized? Rhetorical questions are a means of communication which are not only a means of linguistic manipulation, but also counterproductive in such a discussion. May you reformulate? Cheers, u.
Re: salsa.debian.org maintenance (GitLab 11.1.4 upgrade, external storage migration)
Hi, I believe by decentralization we can just implement it by not relying our data on single company but multiple. Still, it is up to their implementation how we can access their storage, and as long as we can access it with free software (JavaScript stuff could be a pitfall though) it shouldn't be too much problem for us. Yao Wei On Fri, Aug 17, 2018 at 16:52 Andrey Rahmatullin wrote: > On Fri, Aug 17, 2018 at 08:27:00AM +, Ulrike Uhlig wrote: > > While I understand the simplicity of using $company's cloud storage, I'd > > rather not rely on some external company and in particular not on this > > one. This company does not exactly represent what I would call ethical, > > non-proprietary, and decentralized. > Is that a problem? > > > Are there no partners that would kindly provide such storage to Debian > > (Gandi?). > Are they ethical, non-proprietary, and decentralized? > > -- > WBR, wRAR >
Re: salsa.debian.org maintenance (GitLab 11.1.4 upgrade, external storage migration)
On Fri, Aug 17, 2018 at 08:27:00AM +, Ulrike Uhlig wrote: > While I understand the simplicity of using $company's cloud storage, I'd > rather not rely on some external company and in particular not on this > one. This company does not exactly represent what I would call ethical, > non-proprietary, and decentralized. Is that a problem? > Are there no partners that would kindly provide such storage to Debian > (Gandi?). Are they ethical, non-proprietary, and decentralized? -- WBR, wRAR signature.asc Description: PGP signature
Re: salsa.debian.org maintenance (GitLab 11.1.4 upgrade, external storage migration)
Hi! Jonas Meurer: > Am 13.08.2018 um 20:36 schrieb Alexander Wirt: >>> Hrmpf! I have to say that I was somewhat surprised by this announcement. >>> To be honest, I don't like the idea of making our infrastructure as a >>> project rely on closed and proprietary systems like Google Cloud. Isn't >>> it important to us as a project anymore to run our infrastructure on >>> free software and under our own control? [1] >>> We don't rely on it. There will be a backup on debian infastructure so > that >> we will be able to change to different providers at every time. > > That's good to know! > Your explanation definitely helps with understanding the rationale > behind your decision to switch to Google Cloud for some storage. And if > Salsa indeed has I/O problems already, it's much appreciated that you do > something about it. Again, thanks for this. Ack. > I just wonder why we don't consider and prefer free solution (either by > running an own external storage or by using free software cloud > providers) over the proprietary ones. In my eyes, this conflicts with > our social contract and with prioritizing Free Software. That's, why I > brought it up here. > > What do others think about it? Thanks for bringing it up, Jonas! I absolutely agree with your reasoning and I also think that it conflicts with Debian's Social Contract; hence I'd like to see this discussed before silently implementing such a thing. While I understand the simplicity of using $company's cloud storage, I'd rather not rely on some external company and in particular not on this one. This company does not exactly represent what I would call ethical, non-proprietary, and decentralized. Are there no partners that would kindly provide such storage to Debian (Gandi?). Sometimes this may take more time, and be harder to implement. It might be more expensive, and that's exactly the point: whenever something is free, you've become the product. Cheers, Ulrike
Re: salsa.debian.org maintenance (GitLab 11.1.4 upgrade, external storage migration)
Hello, Am 13.08.2018 um 20:36 schrieb Alexander Wirt: >> Hrmpf! I have to say that I was somewhat surprised by this announcement. >> To be honest, I don't like the idea of making our infrastructure as a >> project rely on closed and proprietary systems like Google Cloud. Isn't >> it important to us as a project anymore to run our infrastructure on >> free software and under our own control? [1] >> We don't rely on it. There will be a backup on debian infastructure so that > we will be able to change to different providers at every time. That's good to know! > Additionally its only subsidiary data. Git is and will be only on debian > infrastructure. If you don't use lfs or ci, you are safe (whatever safe > means). > > But using gce allows us to to support use cases different use case than just > git (like lfs, build artificats, build logs and so on) without consuming IO > on debian infrastructure (we are already seeing IO problems on high traffic). > > Hope that helps Your explanation definitely helps with understanding the rationale behind your decision to switch to Google Cloud for some storage. And if Salsa indeed has I/O problems already, it's much appreciated that you do something about it. Again, thanks for this. I just wonder why we don't consider and prefer free solution (either by running an own external storage or by using free software cloud providers) over the proprietary ones. In my eyes, this conflicts with our social contract and with prioritizing Free Software. That's, why I brought it up here. What do others think about it? Cheers jonas signature.asc Description: OpenPGP digital signature
Re: salsa.debian.org maintenance (GitLab 11.1.4 upgrade, external storage migration)
Am 13.08.2018 um 17:50 schrieb Marco d'Itri: > On Aug 13, Jonas Meurer wrote: > >> To be honest, I don't like the idea of making our infrastructure as a >> project rely on closed and proprietary systems like Google Cloud. Isn't >> it important to us as a project anymore to run our infrastructure on >> free software and under our own control? [1] > > Sure. Do you have the source for the firmwares of all your hard disks? Yeah, thanks for this very reasonable argument. Not. Our Social Contract says that we adhere to Free Software, no? So even though there are limitations (but we're working on those as well), I don't see why this justifies switching to proprietary solutions where free ones are available. >> Github and to make us independent from proprietary solutions. If we now >> start moving the salsa storage to a proprietary cloud solution, this >> leaves a bad taste in my mouth.> > We can still easily move that data to a different free as in freedom > cloud solution if needed, it would just be much more expensive because > then I expect that we would actually have to pay for these resources. I don't buy this argument either. If you follow this logic, why not switch to use Github and Travis CI only? Would be the cheapest and we even wouldn't have to maintain it on our own. Yay! And yes, maybe free cloud solutions are more expensive. Do you have calculations? Apart from that, it's not an argument for me to not consider them. And besides, that's your assumption only. It hasn't been discussed in public before and the project was not asked for better alternatives either. Cheers jonas signature.asc Description: OpenPGP digital signature
Re: salsa.debian.org maintenance (GitLab 11.1.4 upgrade, external storage migration)
On 08/13/2018 05:50 PM, Marco d'Itri wrote: > On Aug 13, Jonas Meurer wrote: > >> To be honest, I don't like the idea of making our infrastructure as a >> project rely on closed and proprietary systems like Google Cloud. Isn't >> it important to us as a project anymore to run our infrastructure on >> free software and under our own control? [1] > Sure. Do you have the source for the firmwares of all your hard disks? Well, that's an issue as well, but in such case, we don't have a choice. Unless ... do you know any hard disk manufacturer that are shipping free software firmware? >> Github and to make us independent from proprietary solutions. If we now >> start moving the salsa storage to a proprietary cloud solution, this >> leaves a bad taste in my mouth. > We can still easily move that data to a different free as in freedom > cloud solution if needed, it would just be much more expensive because > then I expect that we would actually have to pay for these resources. I don't think we even tried. I'm convinced that many OpenStack providers would happily sponsor Debian. There's 18 providers listed on the OpenStack market place. Also, numerous times, I offered my help in setting-up an IaaS for Debian, but it doesn't seem like anyone is interested. I also offered my help for maintaining Salsa by the way. Cheers, Thomas Goirand (zigo)
Re: salsa.debian.org maintenance (GitLab 11.1.4 upgrade, external storage migration)
On Mon, 13 Aug 2018, Alexander Wirt wrote: > We don't rely on it. There will be a backup on debian infastructure so that > we will be able to change to different providers at every time. ... > But using gce allows us to to support use cases different use case than just > git (like lfs, build artificats, build logs and so on) without consuming IO > on debian infrastructure (we are already seeing IO problems on high traffic). > > Hope that helps It does. Thank you very much for doing this! -- Henrique Holschuh
Re: salsa.debian.org maintenance (GitLab 11.1.4 upgrade, external storage migration)
On Mon, 13 Aug 2018, Jonas Meurer wrote: > Hello, > > Am 11.08.2018 um 16:20 schrieb Bastian Blank: > > We will do maintenance on salsa.debian.org today, 2018-08-11, between > > 1600 and 1800 UTC. > > > > We will upgrade the GitLab instance to 11.1.4. > > Thanks a ton for all your maintenance work for salsa. It's a huge > improvement for packaging and team maintenance work to have salsa around! > > > We will also start moving some of the large data stores with public > > accessible files off to Google Cloud storage. Using an external storage > > allows us to store a much larger amount of data in our GitLab instance. > > All access to it will be proxied, without providing user identifying > > data to the storage provider. > > Hrmpf! I have to say that I was somewhat surprised by this announcement. > To be honest, I don't like the idea of making our infrastructure as a > project rely on closed and proprietary systems like Google Cloud. Isn't > it important to us as a project anymore to run our infrastructure on > free software and under our own control? [1] We don't rely on it. There will be a backup on debian infastructure so that we will be able to change to different providers at every time. Additionally its only subsidiary data. Git is and will be only on debian infrastructure. If you don't use lfs or ci, you are safe (whatever safe means). But using gce allows us to to support use cases different use case than just git (like lfs, build artificats, build logs and so on) without consuming IO on debian infrastructure (we are already seeing IO problems on high traffic). Hope that helps Alex signature.asc Description: PGP signature
Re: salsa.debian.org maintenance (GitLab 11.1.4 upgrade, external storage migration)
On Aug 13, Jonas Meurer wrote: > To be honest, I don't like the idea of making our infrastructure as a > project rely on closed and proprietary systems like Google Cloud. Isn't > it important to us as a project anymore to run our infrastructure on > free software and under our own control? [1] Sure. Do you have the source for the firmwares of all your hard disks? > Github and to make us independent from proprietary solutions. If we now > start moving the salsa storage to a proprietary cloud solution, this > leaves a bad taste in my mouth. We can still easily move that data to a different free as in freedom cloud solution if needed, it would just be much more expensive because then I expect that we would actually have to pay for these resources. -- ciao, Marco signature.asc Description: PGP signature
Re: salsa.debian.org maintenance (GitLab 11.1.4 upgrade, external storage migration)
Hello, Am 11.08.2018 um 16:20 schrieb Bastian Blank: > We will do maintenance on salsa.debian.org today, 2018-08-11, between > 1600 and 1800 UTC. > > We will upgrade the GitLab instance to 11.1.4. Thanks a ton for all your maintenance work for salsa. It's a huge improvement for packaging and team maintenance work to have salsa around! > We will also start moving some of the large data stores with public > accessible files off to Google Cloud storage. Using an external storage > allows us to store a much larger amount of data in our GitLab instance. > All access to it will be proxied, without providing user identifying > data to the storage provider. Hrmpf! I have to say that I was somewhat surprised by this announcement. To be honest, I don't like the idea of making our infrastructure as a project rely on closed and proprietary systems like Google Cloud. Isn't it important to us as a project anymore to run our infrastructure on free software and under our own control? [1] [1] https://mako.cc/writing/hill-free_tools.html We already switched to proprietary CDNs as our default mirrors. While I don't like this decision either, there's a difference: most mirrors where not maintained by the DSA or other project-internal teams either before. But since there's hundreds of them, we don't rely on a single one to keep working and we could easily change the DNS records for deb.debian.org if one of the CDN providers does evil things. With salsa it is different. It is meant to be a home for our packages and software projects, and to my knowledge its predecessor Alioth had been invented to provide an alternative to platforms Sourceforge and Github and to make us independent from proprietary solutions. If we now start moving the salsa storage to a proprietary cloud solution, this leaves a bad taste in my mouth. At the very least, I would love to have seen this discussed publically before action was taken. Cheers, jonas signature.asc Description: OpenPGP digital signature
