Re: Sysadmin Load Reduction: Subversion Infrastructure

[Māris Nartišs]

pirmd., 2019. g. 11. nov., plkst. 16:02 — lietotājs Luigi Toscano
() rakstīja:
>
> Alexander Potashev ha scritto:
> > вс, 10 нояб. 2019 г. в 18:09, Luigi Toscano :
> >>
> >> Nate Graham ha scritto:
> >>> On 11/9/19 4:50 PM, Nicolás Alvarez wrote:
>  El sáb., 9 de nov. de 2019 a la(s) 20:29, Nate Graham (n...@kde.org) 
>  escribió:
>  - Experiment conversion to git and see if the resulting repo(s) are of
>  a reasonable size (I vaguely recall seeing bad results with this)
> >
> > I expect that a single Git repository would be several GBs in size.
>
> For the record, the current checkout (without svn metadata) of trunk/l10n-kde4
> (ok, no more needed), branches/stable/l10n-kde4, branches/stable/l10n-kf5,
> branches/stable/l10n-kf5-plasma-lts, trunk/l10n-kf5 and trunk/l10n-summit is
> ~11 GiB. The trunk/l10n-kf5 branch alone is 5.3 GiB.

One of pro sides of SVN still is ability to work with a single
directory instead of whole repo. For Latvian language it means using
only ~300MB of disk space (I have less than 1GB free space on my
laptop). Thus if a move to GIT is planned, I would vote to have a
separate repo for each language.

Māris.

Re: Sysadmin Load Reduction: Subversion Infrastructure

[Philippe Cloutier]

Thank you very much for this summary Ben

One addition to the discussion (not to the summary): For my part, every 
time I have to deal with Git, I end up shaking my head in disbelief at 
its* immaturity (and I've only started using it 3 years ago). And I've 
been using VCS-s for over 15 years, so I understand the will of 
translators to remain on Subversion (or at least, to avoid Git).


*: I speak not only of Git here but also of its front-ends like 
TortoiseGit and GitLab's web view


Just one correction below...

On 11/11/2019 05:26, Ben Cooksley wrote:

Hi all,

This has been quite the long thread for one that was started only
around 2 days ago, so thought I would summarise things:

On the topic of removal of the Subversion mirrors, there have been no
objections, so we will proceed with this.

With regards to removal of access for everyone but translators, there
have been two objections noted to this. One concerned compliance with
the Manifesto, whilst the other was concerned with creating obstacles.
Given that restricting access to the websites is still compliant with
the Manifesto despite the lack of a clear clause permitting this I
don't believe that not granting access by default to Subversion should
be an issue, so would like to proceed with this as well.

That leaves the last point, which has generated the most contention,
the removal of WebSVN (websvn.kde.org).


The engine used by websvn.kde.org is not WebSVN but - as can be seen in 
HTML source - ViewVC.





Based on the responses, it would appear that translation workflows are
currently heavily embedded with and tightly dependent on this service.
This would indicate that we would need to keep this service around for
now.

With just a few websites left on Subversion though, I would like to
know if the translation teams have a plan for how they will continue
to do things in the long term, and whether this includes a migration
away from Subversion (as otherwise we are maintaining indefinitely a
Subversion repository, customised hooks, mirroring infrastructure,
WebSVN, and support for the email notifications from those customised
hooks within other systems such as our IRC Bots and Bugzilla)

Regards,
Ben


--
Philippe Cloutier
http://www.philippecloutier.com

Re: Sysadmin Load Reduction: Subversion Infrastructure

[Ingo Klöcker]

On Dienstag, 12. November 2019 13:11:43 CET Luigi Toscano wrote:
> Māris Nartišs ha scritto:
> > pirmd., 2019. g. 11. nov., plkst. 16:02 — lietotājs Luigi Toscano
> > 
> > () rakstīja:
> >> Alexander Potashev ha scritto:
> >>> вс, 10 нояб. 2019 г. в 18:09, Luigi Toscano :
>  Nate Graham ha scritto:
> > On 11/9/19 4:50 PM, Nicolás Alvarez wrote:
> >> El sáb., 9 de nov. de 2019 a la(s) 20:29, Nate Graham (n...@kde.org)
> >> escribió: - Experiment conversion to git and see if the resulting
> >> repo(s) are of a reasonable size (I vaguely recall seeing bad
> >> results with this)
> >>> 
> >>> I expect that a single Git repository would be several GBs in size.
> >> 
> >> For the record, the current checkout (without svn metadata) of
> >> trunk/l10n-kde4 (ok, no more needed), branches/stable/l10n-kde4,
> >> branches/stable/l10n-kf5, branches/stable/l10n-kf5-plasma-lts,
> >> trunk/l10n-kf5 and trunk/l10n-summit is ~11 GiB. The trunk/l10n-kf5
> >> branch alone is 5.3 GiB.
> > 
> > One of pro sides of SVN still is ability to work with a single
> > directory instead of whole repo. For Latvian language it means using
> > only ~300MB of disk space (I have less than 1GB free space on my
> > laptop). Thus if a move to GIT is planned, I would vote to have a
> > separate repo for each language.
> 
> Right, but that would make the life of people doing bulk changes (like me)
> much more complicated. Right now I can do all the changes in one shot.
> That's part of the problem.

It's seems this could be solved with tooling. Jack already proposed using git 
submodules. At work I'm solving the problem with a simple bash-script which 
loops over 10-20 repos to do a git pull/rebase/push on all of them. Yes, merge 
conflicts occur from time to time, but they would occur regardless of the 
number of repos involved.

Regards,
Ingo


signature.asc
Description: This is a digitally signed message part.

Re: Sysadmin Load Reduction: Subversion Infrastructure

[Ben Cooksley]

On Wed, Nov 13, 2019 at 12:12 AM Luigi Toscano  wrote:
>
> Ben Cooksley ha scritto:
> > On Tue, Nov 12, 2019 at 10:00 AM Alexander Potashev
> >  wrote:
> >>
> >> пн, 11 нояб. 2019 г. в 17:02, Luigi Toscano :
> >>>
> >>> Alexander Potashev ha scritto:
>  вс, 10 нояб. 2019 г. в 18:09, Luigi Toscano :
> > Most of translators are not so technical as the developers. And even
> > developers can shoot them in the foot with git, I see many issues 
> > coming from
> > unwanted merges.
> 
>  We can block unwanted merged on the server with a Git hook like this:
>  https://github.com/FabreFrederic/git-hook-pre-receive-reject-merge-commit
> >>>
> >>> When I talk about local merges, I foresee a bit of mess when resolving a 
> >>> merge
> >>> locally, which may result in proper commits whose content has an invalid 
> >>> syntax.
> >>
> >> I don't get it.
> >>
> >> If you have conflicting changes, either with SVN or Git, you have to
> >> resolve the conflict manually. Of course it's easy to break the syntax
> >> while doing so, however SVN does not make it any easier than Git.
> >>
>  Cloning the whole repo would be too much for some translators, however
>  the new Git feature "git clone --filter" might be a solution:
>  https://unix.stackexchange.com/a/468182
> >>>
> >>> Documentation does not seem to help (or at least I don't seem to find it 
> >>> in
> >>> the man pages for git 2.24). Do you know how it could filter just some
> >>> directories? It seems more focused on filtering by object type.
> >>
> >> OK, I now tried this approach with Git 2.23 (see attached log) and
> >> found horrible problems that make it basically unusable:
> >>
> >>  1. git-checkout downloads each file in a new SSH connection. It make
> >> the download really slow: less than 1 file per second in my setup.
> >> That means downloading of translations into one language would take
> >> more than an hour.
> >>
> >>  2. git-status silently tries to download to whole Git repository.
> >>
> >>  3. git-commit tries to download some files one by one again.
> >>
>  Shall I create a new Phabricator task "[WIP] Migrate translations from
>  SVN to Git" so we can put relevant ideas in one place?
> >>>
> >>> I don't know. For me having a board and tasks (it's not just a single 
> >>> task,
> >>> it's an entire project) sounds like there is something that can be
> >>> implemented, and I don't think it's the case.
> >>
> >> The tag [WIP] would hint that we are not sure if it can be
> >> implemented. We can even name it "Reasons why translations cannot
> >> migrate to Git", if necessary.
> >
> > Sorry, but if you proceed with using a title such as that one, then
> > you are not being constructive or helpful towards this conversation.
> >
> > That title would indicate to me that the translation community is
> > completely unwilling to consider change of any form.
>
> That's why it shouldn't be a task at this point, but a document showing why we
> can't migrate.
> It's not a matter that we are unwilling to consider. The problem is that the
> current state makes it almost impossible to migrate. The document would
> explain why.

Spin that around please. It should instead be 'Challenges to migrate to Git'.

This is an exercise Sysadmin is reasonably familiar with when it comes
to any system transition - there are always workflows and processes
that need to be adjusted, tooling that needs to be tweaked or
rewritten, data that needs to be migrated, etc.

>
>
> --
> Luigi

Cheers,
Ben

Re: Sysadmin Load Reduction: Subversion Infrastructure

[Jack]

On 11/12/19 7:11 AM, Luigi Toscano wrote:

Māris Nartišs ha scritto:

pirmd., 2019. g. 11. nov., plkst. 16:02 — lietotājs Luigi Toscano
() rakstīja:

Alexander Potashev ha scritto:

вс, 10 нояб. 2019 г. в 18:09, Luigi Toscano :

Nate Graham ha scritto:

On 11/9/19 4:50 PM, Nicolás Alvarez wrote:

El sáb., 9 de nov. de 2019 a la(s) 20:29, Nate Graham (n...@kde.org) escribió:
- Experiment conversion to git and see if the resulting repo(s) are of
a reasonable size (I vaguely recall seeing bad results with this)

I expect that a single Git repository would be several GBs in size.

For the record, the current checkout (without svn metadata) of trunk/l10n-kde4
(ok, no more needed), branches/stable/l10n-kde4, branches/stable/l10n-kf5,
branches/stable/l10n-kf5-plasma-lts, trunk/l10n-kf5 and trunk/l10n-summit is
~11 GiB. The trunk/l10n-kf5 branch alone is 5.3 GiB.

One of pro sides of SVN still is ability to work with a single
directory instead of whole repo. For Latvian language it means using
only ~300MB of disk space (I have less than 1GB free space on my
laptop). Thus if a move to GIT is planned, I would vote to have a
separate repo for each language.

Right, but that would make the life of people doing bulk changes (like me)
much more complicated. Right now I can do all the changes in one shot. That's
part of the problem.


Would git submodules help with this?  I can imagine it would be an 
effort to get it set up, but then you have everything in one place.


Jack

Re: Sysadmin Load Reduction: Subversion Infrastructure

[Luigi Toscano]

Māris Nartišs ha scritto:
> pirmd., 2019. g. 11. nov., plkst. 16:02 — lietotājs Luigi Toscano
> () rakstīja:
>>
>> Alexander Potashev ha scritto:
>>> вс, 10 нояб. 2019 г. в 18:09, Luigi Toscano :

 Nate Graham ha scritto:
> On 11/9/19 4:50 PM, Nicolás Alvarez wrote:
>> El sáb., 9 de nov. de 2019 a la(s) 20:29, Nate Graham (n...@kde.org) 
>> escribió:
>> - Experiment conversion to git and see if the resulting repo(s) are of
>> a reasonable size (I vaguely recall seeing bad results with this)
>>>
>>> I expect that a single Git repository would be several GBs in size.
>>
>> For the record, the current checkout (without svn metadata) of 
>> trunk/l10n-kde4
>> (ok, no more needed), branches/stable/l10n-kde4, branches/stable/l10n-kf5,
>> branches/stable/l10n-kf5-plasma-lts, trunk/l10n-kf5 and trunk/l10n-summit is
>> ~11 GiB. The trunk/l10n-kf5 branch alone is 5.3 GiB.
> 
> One of pro sides of SVN still is ability to work with a single
> directory instead of whole repo. For Latvian language it means using
> only ~300MB of disk space (I have less than 1GB free space on my
> laptop). Thus if a move to GIT is planned, I would vote to have a
> separate repo for each language.

Right, but that would make the life of people doing bulk changes (like me)
much more complicated. Right now I can do all the changes in one shot. That's
part of the problem.

-- 
Luigi

Re: Sysadmin Load Reduction: Subversion Infrastructure

[Luigi Toscano]

Ilya Bizyaev ha scritto:
> A task shows this attitude much better as it is something that will be
> completed eventually. "[WIP] Migrate translations from SVN to Git" to me
> sounds like a clear intention.

A task and a document are different thing. A set of tasks are action items
which specifies is an action item which
> 
> Docs, in contrast, are static and do not display any motivation for migration.
> "Top 10 Reasons Why We Can't Use Git".

We may have a different ideas about documents, maybe.

A living document on the wiki (I read what Alexander wrote, but I think that
the wiki is the best place) is something that can evolve. When it's clear that
there are possible follow-ups, tasks can be created (this is not specific to
this issue, it's a more general process).

I'm not sure also why it should have a snarky title like that. It could be
something on the line of "Requirements and issues for migrating translations
to git".

-- 
Luigi

Re: Sysadmin Load Reduction: Subversion Infrastructure

[Ilya Bizyaev]

A task shows this attitude much better as it is something that will be 
completed eventually. "[WIP] Migrate translations from SVN to Git" to me sounds 
like a clear intention.



Docs, in contrast, are static and do not display any motivation for migration. 
"Top 10 Reasons Why We Can't Use Git".



Best regards,

Ilya.



 Дата: Вт, 12 ноя 2019 14:12:28 +0300 Luigi Toscano 
 написал(а) 



Ben Cooksley ha scritto: 
> On Tue, Nov 12, 2019 at 10:00 AM Alexander Potashev 
>  wrote: 
>> 
>> пн, 11 нояб. 2019 г. в 17:02, Luigi Toscano 
>> : 
>>> 
>>> Alexander Potashev ha scritto: 
 вс, 10 нояб. 2019 г. в 18:09, Luigi Toscano 
 : 
> Most of translators are not so technical as the developers. And even 
> developers can shoot them in the foot with git, I see many issues coming 
> from 
> unwanted merges. 
 
 We can block unwanted merged on the server with a Git hook like this: 
 https://github.com/FabreFrederic/git-hook-pre-receive-reject-merge-commit 
>>> 
>>> When I talk about local merges, I foresee a bit of mess when resolving a 
>>> merge 
>>> locally, which may result in proper commits whose content has an invalid 
>>> syntax. 
>> 
>> I don't get it. 
>> 
>> If you have conflicting changes, either with SVN or Git, you have to 
>> resolve the conflict manually. Of course it's easy to break the syntax 
>> while doing so, however SVN does not make it any easier than Git. 
>> 
 Cloning the whole repo would be too much for some translators, however 
 the new Git feature "git clone --filter" might be a solution: 
 https://unix.stackexchange.com/a/468182 
>>> 
>>> Documentation does not seem to help (or at least I don't seem to find it in 
>>> the man pages for git 2.24). Do you know how it could filter just some 
>>> directories? It seems more focused on filtering by object type. 
>> 
>> OK, I now tried this approach with Git 2.23 (see attached log) and 
>> found horrible problems that make it basically unusable: 
>> 
>>  1. git-checkout downloads each file in a new SSH connection. It make 
>> the download really slow: less than 1 file per second in my setup. 
>> That means downloading of translations into one language would take 
>> more than an hour. 
>> 
>>  2. git-status silently tries to download to whole Git repository. 
>> 
>>  3. git-commit tries to download some files one by one again. 
>> 
 Shall I create a new Phabricator task "[WIP] Migrate translations from 
 SVN to Git" so we can put relevant ideas in one place? 
>>> 
>>> I don't know. For me having a board and tasks (it's not just a single task, 
>>> it's an entire project) sounds like there is something that can be 
>>> implemented, and I don't think it's the case. 
>> 
>> The tag [WIP] would hint that we are not sure if it can be 
>> implemented. We can even name it "Reasons why translations cannot 
>> migrate to Git", if necessary. 
> 
> Sorry, but if you proceed with using a title such as that one, then 
> you are not being constructive or helpful towards this conversation. 
> 
> That title would indicate to me that the translation community is 
> completely unwilling to consider change of any form. 
 
That's why it shouldn't be a task at this point, but a document showing why we 
can't migrate. 
It's not a matter that we are unwilling to consider. The problem is that the 
current state makes it almost impossible to migrate. The document would 
explain why. 
 
 
-- 
Luigi

Re: Sysadmin Load Reduction: Subversion Infrastructure

[Luigi Toscano]

Ben Cooksley ha scritto:
> On Tue, Nov 12, 2019 at 10:00 AM Alexander Potashev
>  wrote:
>>
>> пн, 11 нояб. 2019 г. в 17:02, Luigi Toscano :
>>>
>>> Alexander Potashev ha scritto:
 вс, 10 нояб. 2019 г. в 18:09, Luigi Toscano :
> Most of translators are not so technical as the developers. And even
> developers can shoot them in the foot with git, I see many issues coming 
> from
> unwanted merges.

 We can block unwanted merged on the server with a Git hook like this:
 https://github.com/FabreFrederic/git-hook-pre-receive-reject-merge-commit
>>>
>>> When I talk about local merges, I foresee a bit of mess when resolving a 
>>> merge
>>> locally, which may result in proper commits whose content has an invalid 
>>> syntax.
>>
>> I don't get it.
>>
>> If you have conflicting changes, either with SVN or Git, you have to
>> resolve the conflict manually. Of course it's easy to break the syntax
>> while doing so, however SVN does not make it any easier than Git.
>>
 Cloning the whole repo would be too much for some translators, however
 the new Git feature "git clone --filter" might be a solution:
 https://unix.stackexchange.com/a/468182
>>>
>>> Documentation does not seem to help (or at least I don't seem to find it in
>>> the man pages for git 2.24). Do you know how it could filter just some
>>> directories? It seems more focused on filtering by object type.
>>
>> OK, I now tried this approach with Git 2.23 (see attached log) and
>> found horrible problems that make it basically unusable:
>>
>>  1. git-checkout downloads each file in a new SSH connection. It make
>> the download really slow: less than 1 file per second in my setup.
>> That means downloading of translations into one language would take
>> more than an hour.
>>
>>  2. git-status silently tries to download to whole Git repository.
>>
>>  3. git-commit tries to download some files one by one again.
>>
 Shall I create a new Phabricator task "[WIP] Migrate translations from
 SVN to Git" so we can put relevant ideas in one place?
>>>
>>> I don't know. For me having a board and tasks (it's not just a single task,
>>> it's an entire project) sounds like there is something that can be
>>> implemented, and I don't think it's the case.
>>
>> The tag [WIP] would hint that we are not sure if it can be
>> implemented. We can even name it "Reasons why translations cannot
>> migrate to Git", if necessary.
> 
> Sorry, but if you proceed with using a title such as that one, then
> you are not being constructive or helpful towards this conversation.
> 
> That title would indicate to me that the translation community is
> completely unwilling to consider change of any form.

That's why it shouldn't be a task at this point, but a document showing why we
can't migrate.
It's not a matter that we are unwilling to consider. The problem is that the
current state makes it almost impossible to migrate. The document would
explain why.


-- 
Luigi

Re: Sysadmin Load Reduction: Subversion Infrastructure

[Ben Cooksley]

On Tue, Nov 12, 2019 at 10:00 AM Alexander Potashev
 wrote:
>
> пн, 11 нояб. 2019 г. в 17:02, Luigi Toscano :
> >
> > Alexander Potashev ha scritto:
> > > вс, 10 нояб. 2019 г. в 18:09, Luigi Toscano :
> > >> Most of translators are not so technical as the developers. And even
> > >> developers can shoot them in the foot with git, I see many issues coming 
> > >> from
> > >> unwanted merges.
> > >
> > > We can block unwanted merged on the server with a Git hook like this:
> > > https://github.com/FabreFrederic/git-hook-pre-receive-reject-merge-commit
> >
> > When I talk about local merges, I foresee a bit of mess when resolving a 
> > merge
> > locally, which may result in proper commits whose content has an invalid 
> > syntax.
>
> I don't get it.
>
> If you have conflicting changes, either with SVN or Git, you have to
> resolve the conflict manually. Of course it's easy to break the syntax
> while doing so, however SVN does not make it any easier than Git.
>
> > > Cloning the whole repo would be too much for some translators, however
> > > the new Git feature "git clone --filter" might be a solution:
> > > https://unix.stackexchange.com/a/468182
> >
> > Documentation does not seem to help (or at least I don't seem to find it in
> > the man pages for git 2.24). Do you know how it could filter just some
> > directories? It seems more focused on filtering by object type.
>
> OK, I now tried this approach with Git 2.23 (see attached log) and
> found horrible problems that make it basically unusable:
>
>  1. git-checkout downloads each file in a new SSH connection. It make
> the download really slow: less than 1 file per second in my setup.
> That means downloading of translations into one language would take
> more than an hour.
>
>  2. git-status silently tries to download to whole Git repository.
>
>  3. git-commit tries to download some files one by one again.
>
> > > Shall I create a new Phabricator task "[WIP] Migrate translations from
> > > SVN to Git" so we can put relevant ideas in one place?
> >
> > I don't know. For me having a board and tasks (it's not just a single task,
> > it's an entire project) sounds like there is something that can be
> > implemented, and I don't think it's the case.
>
> The tag [WIP] would hint that we are not sure if it can be
> implemented. We can even name it "Reasons why translations cannot
> migrate to Git", if necessary.

Sorry, but if you proceed with using a title such as that one, then
you are not being constructive or helpful towards this conversation.

That title would indicate to me that the translation community is
completely unwilling to consider change of any form.

>
> > I think we should first document all the answers to this recurring questions
> > first.
>
> Agreed.
>
> IMO a Phabricator "task" would be the most appropriate place to
> document these answers because we don't have better options: Community
> Wiki and Nextcloud are not very suitable for discussions.
>
> --
> Alexander Potashev

Regards,
Ben

Re: Sysadmin Load Reduction: Subversion Infrastructure

[Ben Cooksley]

On Tue, Nov 12, 2019 at 8:14 AM Ingo Klöcker  wrote:
>
> On Montag, 11. November 2019 11:26:23 CET Ben Cooksley wrote:
> > With regards to removal of access for everyone but translators, there
> > have been two objections noted to this. One concerned compliance with
> > the Manifesto, whilst the other was concerned with creating obstacles.
> > Given that restricting access to the websites is still compliant with
> > the Manifesto despite the lack of a clear clause permitting this I
> > don't believe that not granting access by default to Subversion should
> > be an issue, so would like to proceed with this as well.
>
> FWIW, I withdraw my objection concerning compliance with the Manifesto since,
> according to Ben, anyone with a contributor account can, if needed, request
> access by filing a Sysadmin ticket.

Thanks for confirming this Ingo, it's appreciated.

>
> Regards,
> Ingo

Cheers,
Ben

Re: Sysadmin Load Reduction: Subversion Infrastructure

[Alexander Potashev]

пн, 11 нояб. 2019 г. в 17:02, Luigi Toscano :
>
> Alexander Potashev ha scritto:
> > вс, 10 нояб. 2019 г. в 18:09, Luigi Toscano :
> >> Most of translators are not so technical as the developers. And even
> >> developers can shoot them in the foot with git, I see many issues coming 
> >> from
> >> unwanted merges.
> >
> > We can block unwanted merged on the server with a Git hook like this:
> > https://github.com/FabreFrederic/git-hook-pre-receive-reject-merge-commit
>
> When I talk about local merges, I foresee a bit of mess when resolving a merge
> locally, which may result in proper commits whose content has an invalid 
> syntax.

I don't get it.

If you have conflicting changes, either with SVN or Git, you have to
resolve the conflict manually. Of course it's easy to break the syntax
while doing so, however SVN does not make it any easier than Git.

> > Cloning the whole repo would be too much for some translators, however
> > the new Git feature "git clone --filter" might be a solution:
> > https://unix.stackexchange.com/a/468182
>
> Documentation does not seem to help (or at least I don't seem to find it in
> the man pages for git 2.24). Do you know how it could filter just some
> directories? It seems more focused on filtering by object type.

OK, I now tried this approach with Git 2.23 (see attached log) and
found horrible problems that make it basically unusable:

 1. git-checkout downloads each file in a new SSH connection. It make
the download really slow: less than 1 file per second in my setup.
That means downloading of translations into one language would take
more than an hour.

 2. git-status silently tries to download to whole Git repository.

 3. git-commit tries to download some files one by one again.

> > Shall I create a new Phabricator task "[WIP] Migrate translations from
> > SVN to Git" so we can put relevant ideas in one place?
>
> I don't know. For me having a board and tasks (it's not just a single task,
> it's an entire project) sounds like there is something that can be
> implemented, and I don't think it's the case.

The tag [WIP] would hint that we are not sure if it can be
implemented. We can even name it "Reasons why translations cannot
migrate to Git", if necessary.

> I think we should first document all the answers to this recurring questions
> first.

Agreed.

IMO a Phabricator "task" would be the most appropriate place to
document these answers because we don't have better options: Community
Wiki and Nextcloud are not very suitable for discussions.

-- 
Alexander Potashev
[aspotashev@candy kde-git]$ git clone --no-checkout --filter=blob:none 
"aspotashev@10.8.0.4:/home/aspotashev/kde-git/krita/.git" krita
Cloning into 'krita'...
remote: Enumerating objects: 321084, done.
remote: Counting objects: 100% (321084/321084), done.
remote: Compressing objects: 100% (64401/64401), done.  

remote: Total 321084 (delta 257379), reused 319306 (delta 255807)
Receiving objects: 100% (321084/321084), 36.15 MiB | 593.00 KiB/s, done.
Resolving deltas: 100% (257379/257379), done.
[aspotashev@candy kde-git]$ cd krita/
[aspotashev@candy krita]$ git checkout master -- cmake/
remote: Enumerating objects: 1, done.
remote: Counting objects: 100% (1/1), done.
remote: Total 1 (delta 0), reused 1 (delta 0)
Receiving objects: 100% (1/1), 445 bytes | 445.00 KiB/s, done.
remote: Enumerating objects: 1, done.
remote: Counting objects: 100% (1/1), done.
remote: Total 1 (delta 0), reused 0 (delta 0)
Receiving objects: 100% (1/1), 294 bytes | 294.00 KiB/s, done.
remote: Enumerating objects: 1, done.
remote: Counting objects: 100% (1/1), done.
remote: Total 1 (delta 0), reused 1 (delta 0)
Receiving objects: 100% (1/1), 1020 bytes | 1020.00 KiB/s, done.
remote: Enumerating objects: 1, done.
remote: Counting objects: 100% (1/1), done.
remote: Total 1 (delta 0), reused 0 (delta 0)
Receiving objects: 100% (1/1), 730 bytes | 730.00 KiB/s, done.
remote: Enumerating objects: 1, done.
remote: Counting objects: 100% (1/1), done.
remote: Total 1 (delta 0), reused 1 (delta 0)
Receiving objects: 100% (1/1), 3.42 KiB | 3.42 MiB/s, done.
remote: Enumerating objects: 1, done.
remote: Counting objects: 100% (1/1), done.
remote: Total 1 (delta 0), reused 0 (delta 0)
Receiving objects: 100% (1/1), 362 bytes | 362.00 KiB/s, done.
remote: Enumerating objects: 1, done.
remote: Counting objects: 100% (1/1), done.
remote: Total 1 (delta 0), reused 1 (delta 0)
Receiving objects: 100% (1/1), 358 bytes | 358.00 KiB/s, done.
remote: Enumerating objects: 1, done.
remote: Counting objects: 100% (1/1), done.
remote: Total 1 (delta 0), reused 1 (delta 0)
Receiving objects: 100% (1/1), 465 bytes | 465.00 KiB/s, done.
remote: Enumerating objects: 1, done.
remote: Counting objects: 100% (1/1), done.
remote: Total 1 (delta 0), reused 1 (delta 0)
Receiving objects: 100% (1/1), 416 bytes | 416.00 KiB/s, done.
remote: Enumerating objects: 1, done.
remote: Counting objects: 100% (1/1), done.
remote: Total 1 

Re: Sysadmin Load Reduction: Subversion Infrastructure

[Ingo Klöcker]

On Montag, 11. November 2019 04:21:44 CET Alexander Potashev wrote:
> Also, Lokalize has much more features that a plain text editor:
>  1. Interactive merging and reviewing of changes from other .po
> translation files. I use it all the time to review and commit
> translations from people without SVN accounts.
>  2. Translation memory. Boost productivity, improves consistency of
> translations.
>  3. Other features that improve productivity (insertion of XML/HTML
> tags, source string diffs, ...)

Sounds as if all that's missing in Lokalize is built-in support for git. Maybe 
a GSoC or SoK project?

Regards,
Ingo


signature.asc
Description: This is a digitally signed message part.

Re: Sysadmin Load Reduction: Subversion Infrastructure

[Ingo Klöcker]

On Montag, 11. November 2019 11:26:23 CET Ben Cooksley wrote:
> With regards to removal of access for everyone but translators, there
> have been two objections noted to this. One concerned compliance with
> the Manifesto, whilst the other was concerned with creating obstacles.
> Given that restricting access to the websites is still compliant with
> the Manifesto despite the lack of a clear clause permitting this I
> don't believe that not granting access by default to Subversion should
> be an issue, so would like to proceed with this as well.

FWIW, I withdraw my objection concerning compliance with the Manifesto since, 
according to Ben, anyone with a contributor account can, if needed, request 
access by filing a Sysadmin ticket.

Regards,
Ingo


signature.asc
Description: This is a digitally signed message part.

Re: Sysadmin Load Reduction: Subversion Infrastructure

[Luigi Toscano]

Alexander Potashev ha scritto:
> вс, 10 нояб. 2019 г. в 18:09, Luigi Toscano :
>>
>> Nate Graham ha scritto:
>>> On 11/9/19 4:50 PM, Nicolás Alvarez wrote:
 El sáb., 9 de nov. de 2019 a la(s) 20:29, Nate Graham (n...@kde.org) 
 escribió:
>
> Not knowing anything about the translation system we use... what are the
> blockers to moving it over to git?
>
 Not necessarily in order:
 - Experiment conversion to git and see if the resulting repo(s) are of
 a reasonable size (I vaguely recall seeing bad results with this)
 - Convince and teach every translator to use git (expect lots of friction 
 here)
 - Make scripty use git instead of svn (likely lots of work)
 - Some language teams have their own tooling (local for translation,
 or web for statistics) that would need to be gitified too. For
 example, the Spanish team developed KSvnUpdater.
 - Update lots of documentation in different languages (eg.
 https://es.l10n.kde.org/repositorio.php)

 You may even need to migrate languages one by one (as you convince
 each language team?), so in the meantime all tooling would need to
 support both repos at the same time.

 It doesn't sound like fun...
>>>
>>> Thanks for the background.
>>>
>>> If we're going to keep SVN, I think we need to fully support it. If we don't
>>> have the resources to do this on an ongoing basis, then we need to invest 
>>> the
>>> resources now/soon to move away from it. I don't see any other realistic 
>>> options.
>>
>> New resources is the solution. Please see the other emails: the problem
>> discussed is not subversion itself, but the web interface.
>>
>>>
>>> Again, from a totally non-translation perspective, I am somewhat surprised 
>>> to
>>> hear that individual translators are required to be proficient in a source
>>> control system. Perhaps de-coupling the workflow from direct interaction 
>>> with
>>> the SCM would be beneficial? Isn't this what GUI apps like KLocalize do? If
>>> not, can it be modified to do the SCM interaction? This seems like a 
>>> solvable
>>> problem.
>>
>> Most of translators are not so technical as the developers. And even
>> developers can shoot them in the foot with git, I see many issues coming from
>> unwanted merges.
>> Also, in addition to some of the problem described above (not all of them are
>> blocker IMHO), more relevant: how would you convert the SVN repositories 
>> into git?
>> - a unique repository would be the easier way, and required by people who do
>> changes to all the languages (renamed and moves) but I can only foresee tons
>> of merging issues when committing (see above).
>> - a repository per language would mean a tons of work whenever you need to do
>> a global operation, and right now it's a no go.
> 
> Hi Luigi,
> 
> We can block unwanted merged on the server with a Git hook like this:
> https://github.com/FabreFrederic/git-hook-pre-receive-reject-merge-commit

When I talk about local merges, I foresee a bit of mess when resolving a merge
locally, which may result in proper commits whose content has an invalid syntax.

> 
 - Experiment conversion to git and see if the resulting repo(s) are of
 a reasonable size (I vaguely recall seeing bad results with this)
> 
> I expect that a single Git repository would be several GBs in size.

For the record, the current checkout (without svn metadata) of trunk/l10n-kde4
(ok, no more needed), branches/stable/l10n-kde4, branches/stable/l10n-kf5,
branches/stable/l10n-kf5-plasma-lts, trunk/l10n-kf5 and trunk/l10n-summit is
~11 GiB. The trunk/l10n-kf5 branch alone is 5.3 GiB.


> Cloning the whole repo would be too much for some translators, however
> the new Git feature "git clone --filter" might be a solution:
> https://unix.stackexchange.com/a/468182

Documentation does not seem to help (or at least I don't seem to find it in
the man pages for git 2.24). Do you know how it could filter just some
directories? It seems more focused on filtering by object type.

> Shall I create a new Phabricator task "[WIP] Migrate translations from
> SVN to Git" so we can put relevant ideas in one place?

I don't know. For me having a board and tasks (it's not just a single task,
it's an entire project) sounds like there is something that can be
implemented, and I don't think it's the case.
I think we should first document all the answers to this recurring questions
first.
-- 
Luigi

Re: Sysadmin Load Reduction: Subversion Infrastructure

[Alexander Potashev]

вс, 10 нояб. 2019 г. в 18:09, Luigi Toscano :
>
> Nate Graham ha scritto:
> > On 11/9/19 4:50 PM, Nicolás Alvarez wrote:
> >> El sáb., 9 de nov. de 2019 a la(s) 20:29, Nate Graham (n...@kde.org) 
> >> escribió:
> >>>
> >>> Not knowing anything about the translation system we use... what are the
> >>> blockers to moving it over to git?
> >>>
> >> Not necessarily in order:
> >> - Experiment conversion to git and see if the resulting repo(s) are of
> >> a reasonable size (I vaguely recall seeing bad results with this)
> >> - Convince and teach every translator to use git (expect lots of friction 
> >> here)
> >> - Make scripty use git instead of svn (likely lots of work)
> >> - Some language teams have their own tooling (local for translation,
> >> or web for statistics) that would need to be gitified too. For
> >> example, the Spanish team developed KSvnUpdater.
> >> - Update lots of documentation in different languages (eg.
> >> https://es.l10n.kde.org/repositorio.php)
> >>
> >> You may even need to migrate languages one by one (as you convince
> >> each language team?), so in the meantime all tooling would need to
> >> support both repos at the same time.
> >>
> >> It doesn't sound like fun...
> >
> > Thanks for the background.
> >
> > If we're going to keep SVN, I think we need to fully support it. If we don't
> > have the resources to do this on an ongoing basis, then we need to invest 
> > the
> > resources now/soon to move away from it. I don't see any other realistic 
> > options.
>
> New resources is the solution. Please see the other emails: the problem
> discussed is not subversion itself, but the web interface.
>
> >
> > Again, from a totally non-translation perspective, I am somewhat surprised 
> > to
> > hear that individual translators are required to be proficient in a source
> > control system. Perhaps de-coupling the workflow from direct interaction 
> > with
> > the SCM would be beneficial? Isn't this what GUI apps like KLocalize do? If
> > not, can it be modified to do the SCM interaction? This seems like a 
> > solvable
> > problem.
>
> Most of translators are not so technical as the developers. And even
> developers can shoot them in the foot with git, I see many issues coming from
> unwanted merges.
> Also, in addition to some of the problem described above (not all of them are
> blocker IMHO), more relevant: how would you convert the SVN repositories into 
> git?
> - a unique repository would be the easier way, and required by people who do
> changes to all the languages (renamed and moves) but I can only foresee tons
> of merging issues when committing (see above).
> - a repository per language would mean a tons of work whenever you need to do
> a global operation, and right now it's a no go.

Hi Luigi,

We can block unwanted merged on the server with a Git hook like this:
https://github.com/FabreFrederic/git-hook-pre-receive-reject-merge-commit

> >> - Experiment conversion to git and see if the resulting repo(s) are of
> >> a reasonable size (I vaguely recall seeing bad results with this)

I expect that a single Git repository would be several GBs in size.
Cloning the whole repo would be too much for some translators, however
the new Git feature "git clone --filter" might be a solution:
https://unix.stackexchange.com/a/468182


Shall I create a new Phabricator task "[WIP] Migrate translations from
SVN to Git" so we can put relevant ideas in one place?

-- 
Alexander Potashev

Re: Sysadmin Load Reduction: Subversion Infrastructure

[Ben Cooksley]

Hi all,

This has been quite the long thread for one that was started only
around 2 days ago, so thought I would summarise things:

On the topic of removal of the Subversion mirrors, there have been no
objections, so we will proceed with this.

With regards to removal of access for everyone but translators, there
have been two objections noted to this. One concerned compliance with
the Manifesto, whilst the other was concerned with creating obstacles.
Given that restricting access to the websites is still compliant with
the Manifesto despite the lack of a clear clause permitting this I
don't believe that not granting access by default to Subversion should
be an issue, so would like to proceed with this as well.

That leaves the last point, which has generated the most contention,
the removal of WebSVN (websvn.kde.org).

Based on the responses, it would appear that translation workflows are
currently heavily embedded with and tightly dependent on this service.
This would indicate that we would need to keep this service around for
now.

With just a few websites left on Subversion though, I would like to
know if the translation teams have a plan for how they will continue
to do things in the long term, and whether this includes a migration
away from Subversion (as otherwise we are maintaining indefinitely a
Subversion repository, customised hooks, mirroring infrastructure,
WebSVN, and support for the email notifications from those customised
hooks within other systems such as our IRC Bots and Bugzilla)

Regards,
Ben

Re: Sysadmin Load Reduction: Subversion Infrastructure

[Ben Cooksley]

On Mon, Nov 11, 2019 at 3:58 AM Luigi Toscano  wrote:
>
> Ben Cooksley ha scritto:
> > On Sun, Nov 10, 2019 at 11:19 AM Albert Astals Cid  wrote:
> >>
> >> El dissabte, 9 de novembre de 2019, a les 21:22:16 CET, Ben Cooksley va 
> >> escriure:
> >>> On Sun, Nov 10, 2019 at 8:37 AM Albert Astals Cid  wrote:
> 
>  El dissabte, 9 de novembre de 2019, a les 19:27:07 CET, Ben Cooksley va 
>  escriure:
> > On Sun, Nov 10, 2019 at 7:18 AM Alexander Potashev 
> >  wrote:
> >>
> >> сб, 9 нояб. 2019 г. в 03:20, Ben Cooksley :
> >>> This would include the shutdown of WebSVN in particular, which when
> >>> coupled with the shutdown of our two CGit instances would also allow
> >>> for us to eliminate an entire virtual machine from our systems.
> >>
> >> Will there be any web interface for SVN after shutdown of WebSVN?
> >>
> >> Can we assume https://phabricator.kde.org/source/svn/ remains
> >> available during the next 10 years?
> >>
> >
> > Phabricator's browser will be retired as part of the shutdown of
> > Phabricator, which will take place once Gitlab has assumed
> > responsibility for code hosting and review, and the tasks have been
> > migrated from Phabricator.
> >
> > Should WebSVN be shutdown as well, then there would be no web
> > interface to our SVN repository.
> 
>  That's not acceptable.
> >>>
> >>> Mind explaining why?
> >>
> >> Because we use it in l10n.kde.org to link to po files.
> >
> > Mind detailing what those links are used for?
> >
> >>
> >>> Bear in mind that there is a cost both in terms of infrastructure, and
> >>> people time to maintain a service such as WebSVN.
> >>
> >> We have money, we don't have to shut down things we use because there is a 
> >> cost.
> >
> > I wasn't referring to monetary cost there, I was referring to the flow
> > on effects (such as having to maintain the necessary components on the
> > master server to allow for the Subversion repository to be mirrored).
> >
> > Note also the "people time" component there.
>
> Sure, but please see my previous questions:
> - can we extend the space of rosetta? It already has a partial checkout, and
> 100 GB of free space (which can be kept down

We would need to ask the system hosters to provide this, as Rosetta is
a donated system (if memory serves, Rosetta is currently IOPS
constrainted, so using it to host WebSVN may over-burden the system)

> - if that's not enough, can we simply setup a machine which periodically sync
> from the svn repository? You are probably going to tell me that it does not
> work without server support, but from what I'm reading about svnsync, I don't
> think it should overload the server if executed every 30 minutes.
> Are we sure that we still need something on the master server? Let's try it 
> first.

I'm afraid you cannot use svnsync with KDE's Subversion repository,
that utility hasn't been able to handle it for many, many years now
(it crashes if memory serves - [ade] hit this and documented it on his
blog back around the time it broke which was before the switch to Git
got underway)

We have custom tooling which uses rsync instead to mirror the repository.

Custom tooling is necessary because plain rsync cannot be used to
reliably mirror the repository - because it has 1.5 million commits in
it, which means over 3 million inodes on disk. Each of these would
have to be checked by plain usage of rsync, which takes a substantial
amount of time even on NVMe SSD storage - during which time the local
mirror of the repository may be inconsistent (rendering it unusable),
and which also generates a matching disk load on the master server,
reducing it's performance.

The RSync endpoint for the mirrors to contact is the component on the
master server that has to be maintained.

> - in case it's not clear, I'm volunteering to maintain that system.
>
> Ciao
> --
> Luigi

Cheers,
Ben

Re: Sysadmin Load Reduction: Subversion Infrastructure

[Alexander Potashev]

пн, 11 нояб. 2019 г. в 02:01, Nate Graham :
>
> On 11/10/19 8:09 AM, Luigi Toscano wrote:
> > Most of translators are not so technical as the developers. And even
> > developers can shoot them in the foot with git, I see many issues coming 
> > from
> > unwanted merges.
>
> That's why I would expect that non-technical translators would be able
> to use a nice GUI app that abstracts away all the technical details,
> including the backend SCM. Doesn't Lokalize do this?

No, Lokalize cannot interact with SCMs.

Until 2015 there was a Lokalize plugin "newprojectwizard" - all it
could do were "svn checkout" and "svn update". Once there is an SVN
conflict, you are doomed to using fully functional SVN clients.

-- 
Alexander Potashev

Re: Sysadmin Load Reduction: Subversion Infrastructure

[Alexander Potashev]

вс, 10 нояб. 2019 г. в 23:56, Ingo Klöcker :
>
> On Sonntag, 10. November 2019 16:09:34 CET Luigi Toscano wrote:
> > Nate Graham ha scritto:
> > > Again, from a totally non-translation perspective, I am somewhat surprised
> > > to hear that individual translators are required to be proficient in a
> > > source control system. Perhaps de-coupling the workflow from direct
> > > interaction with the SCM would be beneficial? Isn't this what GUI apps
> > > like KLocalize do? If not, can it be modified to do the SCM interaction?
> > > This seems like a solvable problem.
> >
> > Most of translators are not so technical as the developers. And even
> > developers can shoot them in the foot with git, I see many issues coming
> > from unwanted merges.
>
> GitLab allows editing files directly in the browser. Maybe that's an
> alternative to having to deal with an SCM-CLI.
>
> > Also, in addition to some of the problem described above (not all of them
> > are blocker IMHO), more relevant: how would you convert the SVN
> > repositories into git?
> > - a unique repository would be the easier way, and
> > required by people who do changes to all the languages (renamed and moves)
> > but I can only foresee tons of merging issues when committing (see above).
>
> Why do those merging issues not occur with svn?

Because git-pull merges by default. And even if you teach all
translators to use rebase, there is another obstacle: "git pull
--rebase" does not work with uncommitted changes while "svn update"
works.


Editing translation files on GitLab is by complexity close to editing
SVG images... on GitLab. Some experienced people can get it right, but
most of the time you will produce syntax errors.

Also, Lokalize has much more features that a plain text editor:
 1. Interactive merging and reviewing of changes from other .po
translation files. I use it all the time to review and commit
translations from people without SVN accounts.
 2. Translation memory. Boost productivity, improves consistency of
translations.
 3. Other features that improve productivity (insertion of XML/HTML
tags, source string diffs, ...)

-- 
Alexander Potashev

Re: Sysadmin Load Reduction: Subversion Infrastructure

[Nate Graham]

On 11/10/19 8:09 AM, Luigi Toscano wrote:

Most of translators are not so technical as the developers. And even
developers can shoot them in the foot with git, I see many issues coming from
unwanted merges.


That's why I would expect that non-technical translators would be able 
to use a nice GUI app that abstracts away all the technical details, 
including the backend SCM. Doesn't Lokalize do this?


FWIW, GitLab offers a web editor so in the worst case scenario where 
translators have to directly interact with git, that could be used 
instead of command line tools that make it easy for git novices to shoot 
themselves in the foot.


Nate

Re: Sysadmin Load Reduction: Subversion Infrastructure

[Ingo Klöcker]

On Sonntag, 10. November 2019 16:09:34 CET Luigi Toscano wrote:
> Nate Graham ha scritto:
> > Again, from a totally non-translation perspective, I am somewhat surprised
> > to hear that individual translators are required to be proficient in a
> > source control system. Perhaps de-coupling the workflow from direct
> > interaction with the SCM would be beneficial? Isn't this what GUI apps
> > like KLocalize do? If not, can it be modified to do the SCM interaction?
> > This seems like a solvable problem.
> 
> Most of translators are not so technical as the developers. And even
> developers can shoot them in the foot with git, I see many issues coming
> from unwanted merges.

GitLab allows editing files directly in the browser. Maybe that's an 
alternative to having to deal with an SCM-CLI.

> Also, in addition to some of the problem described above (not all of them
> are blocker IMHO), more relevant: how would you convert the SVN
> repositories into git?
> - a unique repository would be the easier way, and
> required by people who do changes to all the languages (renamed and moves)
> but I can only foresee tons of merging issues when committing (see above).

Why do those merging issues not occur with svn?

Regards,
Ingo


signature.asc
Description: This is a digitally signed message part.

Re: Sysadmin Load Reduction: Subversion Infrastructure

[Luigi Toscano]

Nate Graham ha scritto:
> On 11/9/19 4:50 PM, Nicolás Alvarez wrote:
>> El sáb., 9 de nov. de 2019 a la(s) 20:29, Nate Graham (n...@kde.org) 
>> escribió:
>>>
>>> Not knowing anything about the translation system we use... what are the
>>> blockers to moving it over to git?
>>>
>> Not necessarily in order:
>> - Experiment conversion to git and see if the resulting repo(s) are of
>> a reasonable size (I vaguely recall seeing bad results with this)
>> - Convince and teach every translator to use git (expect lots of friction 
>> here)
>> - Make scripty use git instead of svn (likely lots of work)
>> - Some language teams have their own tooling (local for translation,
>> or web for statistics) that would need to be gitified too. For
>> example, the Spanish team developed KSvnUpdater.
>> - Update lots of documentation in different languages (eg.
>> https://es.l10n.kde.org/repositorio.php)
>>
>> You may even need to migrate languages one by one (as you convince
>> each language team?), so in the meantime all tooling would need to
>> support both repos at the same time.
>>
>> It doesn't sound like fun...
>
> Thanks for the background.
> 
> If we're going to keep SVN, I think we need to fully support it. If we don't
> have the resources to do this on an ongoing basis, then we need to invest the
> resources now/soon to move away from it. I don't see any other realistic 
> options.

New resources is the solution. Please see the other emails: the problem
discussed is not subversion itself, but the web interface.

> 
> Again, from a totally non-translation perspective, I am somewhat surprised to
> hear that individual translators are required to be proficient in a source
> control system. Perhaps de-coupling the workflow from direct interaction with
> the SCM would be beneficial? Isn't this what GUI apps like KLocalize do? If
> not, can it be modified to do the SCM interaction? This seems like a solvable
> problem.

Most of translators are not so technical as the developers. And even
developers can shoot them in the foot with git, I see many issues coming from
unwanted merges.
Also, in addition to some of the problem described above (not all of them are
blocker IMHO), more relevant: how would you convert the SVN repositories into 
git?
- a unique repository would be the easier way, and required by people who do
changes to all the languages (renamed and moves) but I can only foresee tons
of merging issues when committing (see above).
- a repository per language would mean a tons of work whenever you need to do
a global operation, and right now it's a no go.


-- 
Luigi

Re: Sysadmin Load Reduction: Subversion Infrastructure

[Martin Steigerwald]

Albert Astals Cid - 10.11.19, 11:51:00 CET:
> > I wasn't referring to monetary cost there, I was referring to the
> > flow on effects (such as having to maintain the necessary
> > components on the master server to allow for the Subversion
> > repository to be mirrored).
> > 
> > Note also the "people time" component there.
> 
> That can also be solved with money 

Not necessarily or not necessarily easily. I see it in the company I 
work for and I read about it that at least in Europe IT many companies 
currently do not find the amount of skilled people they would like to.

-- 
Martin

Re: Sysadmin Load Reduction: Subversion Infrastructure

[Albert Astals Cid]

El diumenge, 10 de novembre de 2019, a les 3:04:03 CET, Alexander Potashev va 
escriure:
> вс, 10 нояб. 2019 г. в 04:10, Ben Cooksley :
> >
> > On Sun, Nov 10, 2019 at 11:19 AM Albert Astals Cid  wrote:
> > >
> > > El dissabte, 9 de novembre de 2019, a les 21:22:16 CET, Ben Cooksley va 
> > > escriure:
> > > > On Sun, Nov 10, 2019 at 8:37 AM Albert Astals Cid  wrote:
> > > > >
> > > > > El dissabte, 9 de novembre de 2019, a les 19:27:07 CET, Ben Cooksley 
> > > > > va escriure:
> > > > > > On Sun, Nov 10, 2019 at 7:18 AM Alexander Potashev 
> > > > > >  wrote:
> > > > > > >
> > > > > > > сб, 9 нояб. 2019 г. в 03:20, Ben Cooksley :
> > > > > > > > This would include the shutdown of WebSVN in particular, which 
> > > > > > > > when
> > > > > > > > coupled with the shutdown of our two CGit instances would also 
> > > > > > > > allow
> > > > > > > > for us to eliminate an entire virtual machine from our systems.
> > > > > > >
> > > > > > > Will there be any web interface for SVN after shutdown of WebSVN?
> > > > > > >
> > > > > > > Can we assume https://phabricator.kde.org/source/svn/ remains
> > > > > > > available during the next 10 years?
> > > > > > >
> > > > > >
> > > > > > Phabricator's browser will be retired as part of the shutdown of
> > > > > > Phabricator, which will take place once Gitlab has assumed
> > > > > > responsibility for code hosting and review, and the tasks have been
> > > > > > migrated from Phabricator.
> > > > > >
> > > > > > Should WebSVN be shutdown as well, then there would be no web
> > > > > > interface to our SVN repository.
> > > > >
> > > > > That's not acceptable.
> > > >
> > > > Mind explaining why?
> > >
> > > Because we use it in l10n.kde.org to link to po files.
> >
> > Mind detailing what those links are used for?
> 
> A common workflow suggested for translators without SVN account
> (and/or those who find it hard to use SVN) is as folllows:
> 
>  1. Go to https://l10n.kde.org/stats/gui/trunk-kf5/team/ru/ ("ru" for
> Russian) and pick a .po translation file that needs update - e.g.
> incomplete translation or one containing mistakes.
> 
>  2. Click on the .po file at l10n.kde.org to downloaded. WebSVN is
> used for this.
> 
>  3. Edit the downloaded .po file, send for review, etc.
> 
> 
> WebSVN is even mentioned in KDE's primary translation how-to:
> https://l10n.kde.org/docs/translation-howto/gui-translation.html

In addition to that we also use it in check-kde-tp-results to link to the 
particular version the check was run
e.g.
  https://l10n.kde.org/check-kde-tp-results/trunk-kf5/et/messages.html
links to 
  
https://websvn.kde.org/trunk/l10n-kf5/et/messages/kdemultimedia/kwave.po?revision=1555006=markup
in case it was fixed later since the checks only run weekly.

Of course all all of this can be "fixed" by us providing having a "mirror of 
the l10n files" on l10n.kde.org but without having a clue on how websvn works 
it seems it'd be much easier to just keep running that instead of having to 
code our own mirror.

Cheers,
  Albert

Re: Sysadmin Load Reduction: Subversion Infrastructure

[Albert Astals Cid]

El diumenge, 10 de novembre de 2019, a les 2:09:19 CET, Ben Cooksley va 
escriure:
> On Sun, Nov 10, 2019 at 11:19 AM Albert Astals Cid  wrote:
> >
> > El dissabte, 9 de novembre de 2019, a les 21:22:16 CET, Ben Cooksley va 
> > escriure:
> > > On Sun, Nov 10, 2019 at 8:37 AM Albert Astals Cid  wrote:
> > > >
> > > > El dissabte, 9 de novembre de 2019, a les 19:27:07 CET, Ben Cooksley va 
> > > > escriure:
> > > > > On Sun, Nov 10, 2019 at 7:18 AM Alexander Potashev 
> > > > >  wrote:
> > > > > >
> > > > > > сб, 9 нояб. 2019 г. в 03:20, Ben Cooksley :
> > > > > > > This would include the shutdown of WebSVN in particular, which 
> > > > > > > when
> > > > > > > coupled with the shutdown of our two CGit instances would also 
> > > > > > > allow
> > > > > > > for us to eliminate an entire virtual machine from our systems.
> > > > > >
> > > > > > Will there be any web interface for SVN after shutdown of WebSVN?
> > > > > >
> > > > > > Can we assume https://phabricator.kde.org/source/svn/ remains
> > > > > > available during the next 10 years?
> > > > > >
> > > > >
> > > > > Phabricator's browser will be retired as part of the shutdown of
> > > > > Phabricator, which will take place once Gitlab has assumed
> > > > > responsibility for code hosting and review, and the tasks have been
> > > > > migrated from Phabricator.
> > > > >
> > > > > Should WebSVN be shutdown as well, then there would be no web
> > > > > interface to our SVN repository.
> > > >
> > > > That's not acceptable.
> > >
> > > Mind explaining why?
> >
> > Because we use it in l10n.kde.org to link to po files.
> 
> Mind detailing what those links are used for?
> 
> >
> > > Bear in mind that there is a cost both in terms of infrastructure, and
> > > people time to maintain a service such as WebSVN.
> >
> > We have money, we don't have to shut down things we use because there is a 
> > cost.
> 
> I wasn't referring to monetary cost there, I was referring to the flow
> on effects (such as having to maintain the necessary components on the
> master server to allow for the Subversion repository to be mirrored).
> 
> Note also the "people time" component there.

That can also be solved with money :)

Cheers,
  Albert

Re: Sysadmin Load Reduction: Subversion Infrastructure

[Māris Nartišs]

svētd., 2019. g. 10. nov., plkst. 04:09 — lietotājs Alexander Potashev
() rakstīja:
>
> вс, 10 нояб. 2019 г. в 04:10, Ben Cooksley :
> >
> > On Sun, Nov 10, 2019 at 11:19 AM Albert Astals Cid  wrote:
> > >
> > > El dissabte, 9 de novembre de 2019, a les 21:22:16 CET, Ben Cooksley va 
> > > escriure:
> > > Because we use it in l10n.kde.org to link to po files.
> >
> > Mind detailing what those links are used for?
>
> A common workflow suggested for translators without SVN account
> (and/or those who find it hard to use SVN) is as folllows:
>
>  1. Go to https://l10n.kde.org/stats/gui/trunk-kf5/team/ru/ ("ru" for
> Russian) and pick a .po translation file that needs update - e.g.
> incomplete translation or one containing mistakes.
>
>  2. Click on the .po file at l10n.kde.org to downloaded. WebSVN is
> used for this.
>
>  3. Edit the downloaded .po file, send for review, etc.

Besides easier way of handling some corner cases, as exampled by
Friedrich W. H. Kossebau, this is the main reason. If WebSVN goes
away, it means redesign of l10n.kde.org to extract PO/POT files to
some storage place for serving. It might take less space than a full
copy of SVN repo, but who volunteers to write the code (before WebSVN
is shut down)?


Māris Nartišs.

Re: Sysadmin Load Reduction: Subversion Infrastructure

[Nate Graham]

On 11/9/19 4:50 PM, Nicolás Alvarez wrote:

El sáb., 9 de nov. de 2019 a la(s) 20:29, Nate Graham (n...@kde.org) escribió:


Not knowing anything about the translation system we use... what are the
blockers to moving it over to git?

Nate


Not necessarily in order:
- Experiment conversion to git and see if the resulting repo(s) are of
a reasonable size (I vaguely recall seeing bad results with this)
- Convince and teach every translator to use git (expect lots of friction here)
- Make scripty use git instead of svn (likely lots of work)
- Some language teams have their own tooling (local for translation,
or web for statistics) that would need to be gitified too. For
example, the Spanish team developed KSvnUpdater.
- Update lots of documentation in different languages (eg.
https://es.l10n.kde.org/repositorio.php)

You may even need to migrate languages one by one (as you convince
each language team?), so in the meantime all tooling would need to
support both repos at the same time.

It doesn't sound like fun...


Thanks for the background.

If we're going to keep SVN, I think we need to fully support it. If we 
don't have the resources to do this on an ongoing basis, then we need to 
invest the resources now/soon to move away from it. I don't see any 
other realistic options.


Again, from a totally non-translation perspective, I am somewhat 
surprised to hear that individual translators are required to be 
proficient in a source control system. Perhaps de-coupling the workflow 
from direct interaction with the SCM would be beneficial? Isn't this 
what GUI apps like KLocalize do? If not, can it be modified to do the 
SCM interaction? This seems like a solvable problem.


Nate

Re: Sysadmin Load Reduction: Subversion Infrastructure

[Alexander Potashev]

вс, 10 нояб. 2019 г. в 04:10, Ben Cooksley :
>
> On Sun, Nov 10, 2019 at 11:19 AM Albert Astals Cid  wrote:
> >
> > El dissabte, 9 de novembre de 2019, a les 21:22:16 CET, Ben Cooksley va 
> > escriure:
> > > On Sun, Nov 10, 2019 at 8:37 AM Albert Astals Cid  wrote:
> > > >
> > > > El dissabte, 9 de novembre de 2019, a les 19:27:07 CET, Ben Cooksley va 
> > > > escriure:
> > > > > On Sun, Nov 10, 2019 at 7:18 AM Alexander Potashev 
> > > > >  wrote:
> > > > > >
> > > > > > сб, 9 нояб. 2019 г. в 03:20, Ben Cooksley :
> > > > > > > This would include the shutdown of WebSVN in particular, which 
> > > > > > > when
> > > > > > > coupled with the shutdown of our two CGit instances would also 
> > > > > > > allow
> > > > > > > for us to eliminate an entire virtual machine from our systems.
> > > > > >
> > > > > > Will there be any web interface for SVN after shutdown of WebSVN?
> > > > > >
> > > > > > Can we assume https://phabricator.kde.org/source/svn/ remains
> > > > > > available during the next 10 years?
> > > > > >
> > > > >
> > > > > Phabricator's browser will be retired as part of the shutdown of
> > > > > Phabricator, which will take place once Gitlab has assumed
> > > > > responsibility for code hosting and review, and the tasks have been
> > > > > migrated from Phabricator.
> > > > >
> > > > > Should WebSVN be shutdown as well, then there would be no web
> > > > > interface to our SVN repository.
> > > >
> > > > That's not acceptable.
> > >
> > > Mind explaining why?
> >
> > Because we use it in l10n.kde.org to link to po files.
>
> Mind detailing what those links are used for?

A common workflow suggested for translators without SVN account
(and/or those who find it hard to use SVN) is as folllows:

 1. Go to https://l10n.kde.org/stats/gui/trunk-kf5/team/ru/ ("ru" for
Russian) and pick a .po translation file that needs update - e.g.
incomplete translation or one containing mistakes.

 2. Click on the .po file at l10n.kde.org to downloaded. WebSVN is
used for this.

 3. Edit the downloaded .po file, send for review, etc.


WebSVN is even mentioned in KDE's primary translation how-to:
https://l10n.kde.org/docs/translation-howto/gui-translation.html

-- 
Alexander Potashev

Re: Sysadmin Load Reduction: Subversion Infrastructure

[Ben Cooksley]

On Sun, Nov 10, 2019 at 2:13 PM Philippe Cloutier  wrote:
>
>
> On 09/11/2019 16:07, Ben Cooksley wrote:
> > On Sun, Nov 10, 2019 at 9:32 AM Albert Vaca Cintora
> >  wrote:
> >> On Sat, Nov 9, 2019 at 6:53 PM Ben Cooksley  wrote:
> >>> Unfortunately Rosetta does not have the necessary free disk space, as
> >>> the Subversion repository is approximately 80GB these days in size,
> >>> and operating WebSVN requires a full copy of the Subversion repository
> >>> locally in order to operate.
> >> Is this also the case if you run WebSVN next to the SVN server itself
> >> (ie: on the same machine)? If a single repo copy can be shared by
> >> WebSVN and the actual SVN server, that would reduce the storage needs.
> > Running WebSVN on the same machine is not safe for us to do, because
> > certain requests cause WebSVN to generate a substantial amount of
> > system load, and have caused out of memory events on systems running
> > it in the past.
>
> I know nothing about WebSVN internals, but if that's the only issue,
> PHP's memory_limit setting should allow avoiding such as issue:
> https://www.php.net/manual/fr/ini.core.php
> Do realize that this setting only affects a single script's usage though
> - it will avoid one script execution going berserk, but if we really
> want to set an overall WebSVN limit of 4 GB for example, we probably
> need to set memory_limit to - say - 512 MB - but also the max number of
> concurrent requests to 8.

Unfortunately WebSVN is a CGI/Python based bit of software, so the PHP
memory_limit doesn't help us here unfortunately.
(Side note: the software running at websvn.kde.org is actually ViewVC
and has no relation to the software called WebSVN, which is PHP based
- apologies for any confusion there)

>
> I personally used WebSVN for translation issues a few times. I wouldn't
> have helped as much without it. I would find it unfortunate if
> translations could no longer be viewed from the web. I would consider
> that issue as somewhat strategic, since many start contributing by
> getting involved in translations... and a web access to translation
> helps getting involved in translation.
>
>
> > [...]
> > Regards,
> > Ben
>

Cheers,
Ben

> --
> Philippe Cloutier
> http://www.philippecloutier.com
>

Re: Sysadmin Load Reduction: Subversion Infrastructure

[Philippe Cloutier]

On 09/11/2019 16:07, Ben Cooksley wrote:

On Sun, Nov 10, 2019 at 9:32 AM Albert Vaca Cintora
 wrote:

On Sat, Nov 9, 2019 at 6:53 PM Ben Cooksley  wrote:

Unfortunately Rosetta does not have the necessary free disk space, as
the Subversion repository is approximately 80GB these days in size,
and operating WebSVN requires a full copy of the Subversion repository
locally in order to operate.

Is this also the case if you run WebSVN next to the SVN server itself
(ie: on the same machine)? If a single repo copy can be shared by
WebSVN and the actual SVN server, that would reduce the storage needs.

Running WebSVN on the same machine is not safe for us to do, because
certain requests cause WebSVN to generate a substantial amount of
system load, and have caused out of memory events on systems running
it in the past.


I know nothing about WebSVN internals, but if that's the only issue, 
PHP's memory_limit setting should allow avoiding such as issue: 
https://www.php.net/manual/fr/ini.core.php
Do realize that this setting only affects a single script's usage though 
- it will avoid one script execution going berserk, but if we really 
want to set an overall WebSVN limit of 4 GB for example, we probably 
need to set memory_limit to - say - 512 MB - but also the max number of 
concurrent requests to 8.


I personally used WebSVN for translation issues a few times. I wouldn't 
have helped as much without it. I would find it unfortunate if 
translations could no longer be viewed from the web. I would consider 
that issue as somewhat strategic, since many start contributing by 
getting involved in translations... and a web access to translation 
helps getting involved in translation.




[...]
Regards,
Ben


--
Philippe Cloutier
http://www.philippecloutier.com

Re: Sysadmin Load Reduction: Subversion Infrastructure

[Ben Cooksley]

On Sun, Nov 10, 2019 at 11:19 AM Albert Astals Cid  wrote:
>
> El dissabte, 9 de novembre de 2019, a les 21:22:16 CET, Ben Cooksley va 
> escriure:
> > On Sun, Nov 10, 2019 at 8:37 AM Albert Astals Cid  wrote:
> > >
> > > El dissabte, 9 de novembre de 2019, a les 19:27:07 CET, Ben Cooksley va 
> > > escriure:
> > > > On Sun, Nov 10, 2019 at 7:18 AM Alexander Potashev 
> > > >  wrote:
> > > > >
> > > > > сб, 9 нояб. 2019 г. в 03:20, Ben Cooksley :
> > > > > > This would include the shutdown of WebSVN in particular, which when
> > > > > > coupled with the shutdown of our two CGit instances would also allow
> > > > > > for us to eliminate an entire virtual machine from our systems.
> > > > >
> > > > > Will there be any web interface for SVN after shutdown of WebSVN?
> > > > >
> > > > > Can we assume https://phabricator.kde.org/source/svn/ remains
> > > > > available during the next 10 years?
> > > > >
> > > >
> > > > Phabricator's browser will be retired as part of the shutdown of
> > > > Phabricator, which will take place once Gitlab has assumed
> > > > responsibility for code hosting and review, and the tasks have been
> > > > migrated from Phabricator.
> > > >
> > > > Should WebSVN be shutdown as well, then there would be no web
> > > > interface to our SVN repository.
> > >
> > > That's not acceptable.
> >
> > Mind explaining why?
>
> Because we use it in l10n.kde.org to link to po files.

Mind detailing what those links are used for?

>
> > Bear in mind that there is a cost both in terms of infrastructure, and
> > people time to maintain a service such as WebSVN.
>
> We have money, we don't have to shut down things we use because there is a 
> cost.

I wasn't referring to monetary cost there, I was referring to the flow
on effects (such as having to maintain the necessary components on the
master server to allow for the Subversion repository to be mirrored).

Note also the "people time" component there.

>
> Cheers,
>   Albert

Thanks,
Ben

>
> > This includes having to maintain a mirror of the repository on the
> > machine that runs WebSVN, along with the associated infrastructure for
> > allowing that mirroring to happen on the master server.
> >
> > >
> > > Cheers,
> > >   Albert
> >
> > Regards,
> > Ben
> >
> > >
> > > >
> > > > >
> > > > > I'm not sure what use case Luigi has in mind, but among Russian l10n
> > > > > team we often use WebSVN to refer to specific SVN revisions, for
> > > > > example: https://websvn.kde.org/?view=revision=1555342
> > > > >
> > > > > --
> > > > > Alexander Potashev
> > > >
> > > > Regards,
> > > > Ben
> > > >
> > >
> > >
> > >
> > >
> >
>
>
>
>

Re: Sysadmin Load Reduction: Subversion Infrastructure

[Ben Cooksley]

On Sun, Nov 10, 2019 at 10:21 AM Friedrich W. H. Kossebau
 wrote:
>
> Am Samstag, 9. November 2019, 21:35:47 CET schrieb Ben Cooksley:
> > On Sun, Nov 10, 2019 at 9:14 AM Friedrich W. H. Kossebau
> >
> >  wrote:
> > > Am Samstag, 9. November 2019, 19:16:20 CET schrieb Ben Cooksley:
> > > > On Sun, Nov 10, 2019 at 6:04 AM Ingo Klöcker  wrote:
> > > > > On Samstag, 9. November 2019 01:20:01 CET Ben Cooksley wrote:
> > > > > > On top of this, i'd also like to remove commit access to it for
> > > > > > everyone but translators and those who need to work on the small
> > > > > > number of websites remaining on Subversion and only provision this
> > > > > > for
> > > > > > people on an as-needed basis.
> > > > > >
> > > > > > In the next year or so i'd expect the remaining websites to complete
> > > > > > their migrations to Git, after which only translators would receive
> > > > > > access.
> > > > >
> > > > > Restricting access to the translations repository is against the
> > > > > letter of
> > > > > our manifesto which states
> > > > > "All source materials are hosted on infrastructure available to and
> > > > > writable by all KDE contributor accounts"
> > > > > https://manifesto.kde.org/commitments.html
> > > > >
> > > > > AFAIK, "all source materials" includes translations.
> > > > >
> > > > > There are a few reasonable exceptions for this requirement, e.g. for
> > > > > the
> > > > > sources of our websites, but I don't see a good reason for restricting
> > > > > access to the translations.
> > > > >
> > > > > I think restricting access to the translations will create a precedent
> > > > > for
> > > > > restricting access to other source materials and undermines the values
> > > > > stated in our manifesto. Therefore, I don't think we should go down
> > > > > this
> > > > > route.
> > > >
> > > > The access isn't being *restricted* at all.
> > > > It is just something you have to request be enabled separately, and it
> > > > won't be withheld from anyone with a developer account should they
> > > > feel they need it.
> > >
> > > This is a model we also see with rights to kick off build jobs on
> > > build.kde.org, and I think it does not work out:
> > > having to beg for access and having to wait for access being granted is an
> > > obstacle.
> > > Even more when this is nowhere documented, but a secret traded only in
> > > people's minds.
> >
> > As a general principle, filing a Sysadmin ticket has always been the
> > way to get access to systems (developer accounts being the only
> > exception), and the same applies to the CI system. Hence why there is
> > no explicit documentation around this.
>
> It stays an obstacle for everyone on first contact though. And makes one feel
> in begging position, when one should not, by ideas of the manifesto.

As noted by Michael, this issue will likely be resolved as part of the
transition to Gitlab CI.

While it was never our intention to not provide developers with the
ability to start jobs should they wish to, it has unfortunately ended
up being something we can't provide on a group basis with the current
Jenkins setup using Phabricator authentication (we did provide that in
the past). Hence the current model of that access being provided on
request.

> And often enough one needs access _now_, instead of having to hope some
> sysadmin is around in the near time.
>
> > > So, by default there are de-facto restrictions experienced, and they get
> > > in
> > > the way of developer work-flows.
> >
> > It's a bit unusual that a lack of access to the CI system would impact
> > someone's workflow, as the CI system is supposed to automatically
> > trigger itself.
> > Do you have a specific scenario in mind here?
>
> The usual scenarios where one needs to manually trigger build jobs:
> a) build metadata changed (e.g. dependencies)
> b) builds failed for bko-internal issues
> c) branches changed, need manual first-run trigger
>
> All things normal developers want or need to do once in a while, if they care
> for their project on KDE CI.
>
> > > My 2 cent on this matter when it comes to conditions decided about.
> > >
> > > Otherwise, thanks for all the admin work you are doing here once again :)
> > >
> > > Cheers
> > > Friedrich, who only an hour ago had to help someone kicking off builds
> > > jobs on CI, as he once got the access granted after poking a few times
> > > informally...
> > Fortunately switching to Gitlab CI will resolve that specific scenario
> > (needing the DSL Job run when a release is being made) as the DSL Job
> > will be rendered unnecessary.
>
> "When a release is being made" should mean, when a new stabe branch has been
> created, I guess? That would be an improvement. Still leaves scenarios a) &
> b).

That is correct.

>
> Cheers
> Friedrich
>

Regards,
Ben

Re: Sysadmin Load Reduction: Subversion Infrastructure

[Ben Cooksley]

On Sun, Nov 10, 2019 at 11:47 AM Friedrich W. H. Kossebau
 wrote:
>
> Am Samstag, 9. November 2019, 21:22:16 CET schrieb Ben Cooksley:
> > On Sun, Nov 10, 2019 at 8:37 AM Albert Astals Cid  wrote:
> > > El dissabte, 9 de novembre de 2019, a les 19:27:07 CET, Ben Cooksley va
> escriure:
> > > > On Sun, Nov 10, 2019 at 7:18 AM Alexander Potashev
>  wrote:
> > > > > сб, 9 нояб. 2019 г. в 03:20, Ben Cooksley :
> > > > > > This would include the shutdown of WebSVN in particular, which when
> > > > > > coupled with the shutdown of our two CGit instances would also allow
> > > > > > for us to eliminate an entire virtual machine from our systems.
> > > > >
> > > > > Will there be any web interface for SVN after shutdown of WebSVN?
> > > > >
> > > > > Can we assume https://phabricator.kde.org/source/svn/ remains
> > > > > available during the next 10 years?
> > > >
> > > > Phabricator's browser will be retired as part of the shutdown of
> > > > Phabricator, which will take place once Gitlab has assumed
> > > > responsibility for code hosting and review, and the tasks have been
> > > > migrated from Phabricator.
> > > >
> > > > Should WebSVN be shutdown as well, then there would be no web
> > > > interface to our SVN repository.
> > >
> > > That's not acceptable.
> >
> > Mind explaining why?
>
> FWIW, everytime I had to deal with translations as developer (like checking
> pot files as well as .po files contents) I found having the web interface and
> its browsing feature very valuable to quickly find what I was looking for,
> over having to locally mess around with svn commands and juggling between
> commandline & file viewers. Including url bookmarks for quick access to
> browsing certain sets of files.
>
> Incidents which I remember right now included:
> * finding out whether extraction scripts were working as intended
> * comparing translations seen by users over what they should see
>
> Are there any other KDE clients of the svn repos still around, besides
> translation system?

The only other clients I am aware of are a small number of websites,
which can be found at /trunk/www/ in SVN.
The list of ones we serve can be found at
https://invent.kde.org/sysadmin/binary-factory-tooling/blob/master/staticweb/standard-jobs.yaml#L69

> Perhaps the "full clone" needed for WebSVN could be reduced to the translation
> subtrees, would that improve situation to a degree if possible? (well, you
> surely thought of this yourself, just in case)

Unfortunately WebSVN needs a full copy of the underlying repository to
be able to work.

>
> For me as developer contributor to projects in KDE spheres, losing the web
> browsing interface for raw translation files would be a regression in
> developer experience.
>
> Cheers
> Friedrich
>
>

Regards,
Ben

Re: Sysadmin Load Reduction: Subversion Infrastructure

[Nicolás Alvarez]

El sáb., 9 de nov. de 2019 a la(s) 20:29, Nate Graham (n...@kde.org) escribió:
>
> Not knowing anything about the translation system we use... what are the
> blockers to moving it over to git?
>
> Nate

Not necessarily in order:
- Experiment conversion to git and see if the resulting repo(s) are of
a reasonable size (I vaguely recall seeing bad results with this)
- Convince and teach every translator to use git (expect lots of friction here)
- Make scripty use git instead of svn (likely lots of work)
- Some language teams have their own tooling (local for translation,
or web for statistics) that would need to be gitified too. For
example, the Spanish team developed KSvnUpdater.
- Update lots of documentation in different languages (eg.
https://es.l10n.kde.org/repositorio.php)

You may even need to migrate languages one by one (as you convince
each language team?), so in the meantime all tooling would need to
support both repos at the same time.

It doesn't sound like fun...

--
Nicolás

Re: Sysadmin Load Reduction: Subversion Infrastructure

[Nate Graham]

Not knowing anything about the translation system we use... what are the 
blockers to moving it over to git?


Nate

Re: Sysadmin Load Reduction: Subversion Infrastructure

[Nicolás Alvarez]

El sáb., 9 de nov. de 2019 a la(s) 19:48, Friedrich W. H. Kossebau
(kosse...@kde.org) escribió:
> FWIW, everytime I had to deal with translations as developer (like checking
> pot files as well as .po files contents) I found having the web interface and
> its browsing feature very valuable to quickly find what I was looking for,
> over having to locally mess around with svn commands and juggling between
> commandline & file viewers. Including url bookmarks for quick access to
> browsing certain sets of files.
>
> Incidents which I remember right now included:
> * finding out whether extraction scripts were working as intended
> * comparing translations seen by users over what they should see
>
> Are there any other KDE clients of the svn repos still around, besides
> translation system?
> Perhaps the "full clone" needed for WebSVN could be reduced to the translation
> subtrees, would that improve situation to a degree if possible? (well, you
> surely thought of this yourself, just in case)

This is unfortunately not possible. WebSVN needs a full copy of the
repository with its history, not a svn checkout, and that can't be
trimmed to a subtree. Or maybe you *can* extract a subtree, but then
you can't keep that in sync with new changes that appear in the master
repository. Even in git that's a giant pain.

-- 
Nicolás

Re: Sysadmin Load Reduction: Subversion Infrastructure

[Friedrich W. H. Kossebau]

Am Samstag, 9. November 2019, 21:22:16 CET schrieb Ben Cooksley:
> On Sun, Nov 10, 2019 at 8:37 AM Albert Astals Cid  wrote:
> > El dissabte, 9 de novembre de 2019, a les 19:27:07 CET, Ben Cooksley va 
escriure:
> > > On Sun, Nov 10, 2019 at 7:18 AM Alexander Potashev 
 wrote:
> > > > сб, 9 нояб. 2019 г. в 03:20, Ben Cooksley :
> > > > > This would include the shutdown of WebSVN in particular, which when
> > > > > coupled with the shutdown of our two CGit instances would also allow
> > > > > for us to eliminate an entire virtual machine from our systems.
> > > > 
> > > > Will there be any web interface for SVN after shutdown of WebSVN?
> > > > 
> > > > Can we assume https://phabricator.kde.org/source/svn/ remains
> > > > available during the next 10 years?
> > > 
> > > Phabricator's browser will be retired as part of the shutdown of
> > > Phabricator, which will take place once Gitlab has assumed
> > > responsibility for code hosting and review, and the tasks have been
> > > migrated from Phabricator.
> > > 
> > > Should WebSVN be shutdown as well, then there would be no web
> > > interface to our SVN repository.
> > 
> > That's not acceptable.
> 
> Mind explaining why?

FWIW, everytime I had to deal with translations as developer (like checking 
pot files as well as .po files contents) I found having the web interface and 
its browsing feature very valuable to quickly find what I was looking for, 
over having to locally mess around with svn commands and juggling between 
commandline & file viewers. Including url bookmarks for quick access to 
browsing certain sets of files.

Incidents which I remember right now included:
* finding out whether extraction scripts were working as intended
* comparing translations seen by users over what they should see

Are there any other KDE clients of the svn repos still around, besides 
translation system?
Perhaps the "full clone" needed for WebSVN could be reduced to the translation 
subtrees, would that improve situation to a degree if possible? (well, you 
surely thought of this yourself, just in case)

For me as developer contributor to projects in KDE spheres, losing the web 
browsing interface for raw translation files would be a regression in 
developer experience.

Cheers
Friedrich

Re: Sysadmin Load Reduction: Subversion Infrastructure

[Michael Reeves]

On Sat, Nov 9, 2019, 4:21 PM Friedrich W. H. Kossebau 
wrote:

> Am Samstag, 9. November 2019, 21:35:47 CET schrieb Ben Cooksley:
> > On Sun, Nov 10, 2019 at 9:14 AM Friedrich W. H. Kossebau
> >
> >  wrote:
> > > Am Samstag, 9. November 2019, 19:16:20 CET schrieb Ben Cooksley:
> > > > On Sun, Nov 10, 2019 at 6:04 AM Ingo Klöcker 
> wrote:
> > > > > On Samstag, 9. November 2019 01:20:01 CET Ben Cooksley wrote:
> > > > > > On top of this, i'd also like to remove commit access to it for
> > > > > > everyone but translators and those who need to work on the small
> > > > > > number of websites remaining on Subversion and only provision
> this
> > > > > > for
> > > > > > people on an as-needed basis.
> > > > > >
> > > > > > In the next year or so i'd expect the remaining websites to
> complete
> > > > > > their migrations to Git, after which only translators would
> receive
> > > > > > access.
> > > > >
> > > > > Restricting access to the translations repository is against the
> > > > > letter of
> > > > > our manifesto which states
> > > > > "All source materials are hosted on infrastructure available to and
> > > > > writable by all KDE contributor accounts"
> > > > > https://manifesto.kde.org/commitments.html
> > > > >
> > > > > AFAIK, "all source materials" includes translations.
> > > > >
> > > > > There are a few reasonable exceptions for this requirement, e.g.
> for
> > > > > the
> > > > > sources of our websites, but I don't see a good reason for
> restricting
> > > > > access to the translations.
> > > > >
> > > > > I think restricting access to the translations will create a
> precedent
> > > > > for
> > > > > restricting access to other source materials and undermines the
> values
> > > > > stated in our manifesto. Therefore, I don't think we should go down
> > > > > this
> > > > > route.
> > > >
> > > > The access isn't being *restricted* at all.
> > > > It is just something you have to request be enabled separately, and
> it
> > > > won't be withheld from anyone with a developer account should they
> > > > feel they need it.
> > >
> > > This is a model we also see with rights to kick off build jobs on
> > > build.kde.org, and I think it does not work out:
> > > having to beg for access and having to wait for access being granted
> is an
> > > obstacle.
> > > Even more when this is nowhere documented, but a secret traded only in
> > > people's minds.
> >
> > As a general principle, filing a Sysadmin ticket has always been the
> > way to get access to systems (developer accounts being the only
> > exception), and the same applies to the CI system. Hence why there is
> > no explicit documentation around this.
>
> It stays an obstacle for everyone on first contact though. And makes one
> feel
> in begging position, when one should not, by ideas of the manifesto.
> And often enough one needs access _now_, instead of having to hope some
> sysadmin is around in the near time.
>
> > > So, by default there are de-facto restrictions experienced, and they
> get
> > > in
> > > the way of developer work-flows.
> >
> > It's a bit unusual that a lack of access to the CI system would impact
> > someone's workflow, as the CI system is supposed to automatically
> > trigger itself.
> > Do you have a specific scenario in mind here?
>
> The usual scenarios where one needs to manually trigger build jobs:
> a) build metadata changed (e.g. dependencies)
> b) builds failed for bko-internal issues
> c) branches changed, need manual first-run trigger
>
> All things normal developers want or need to do once in a while, if they
> care
> for their project on KDE CI.
>
> > > My 2 cent on this matter when it comes to conditions decided about.
> > >
> > > Otherwise, thanks for all the admin work you are doing here once again
> :)
> > >
> > > Cheers
> > > Friedrich, who only an hour ago had to help someone kicking off builds
> > > jobs on CI, as he once got the access granted after poking a few times
> > > informally...
> > Fortunately switching to Gitlab CI will resolve that specific scenario
> > (needing the DSL Job run when a release is being made) as the DSL Job
> > will be rendered unnecessary.
>
> "When a release is being made" should mean, when a new stabe branch has
> been
> created, I guess? That would be an improvement. Still leaves scenarios a)
> &

b).
>
> Cheers
> Friedrich
>
>
> As far as b) goes I had just this scenario with kdiff3 on the new gitlab
> ci. I was able to manually restart the job without sysadmin intervention.
> It's even possible to have gitlab automatically retry for such errors.
> That's what I have done for kdiff3.

Re: Sysadmin Load Reduction: Subversion Infrastructure

[Albert Astals Cid]

El dissabte, 9 de novembre de 2019, a les 21:22:16 CET, Ben Cooksley va 
escriure:
> On Sun, Nov 10, 2019 at 8:37 AM Albert Astals Cid  wrote:
> >
> > El dissabte, 9 de novembre de 2019, a les 19:27:07 CET, Ben Cooksley va 
> > escriure:
> > > On Sun, Nov 10, 2019 at 7:18 AM Alexander Potashev  
> > > wrote:
> > > >
> > > > сб, 9 нояб. 2019 г. в 03:20, Ben Cooksley :
> > > > > This would include the shutdown of WebSVN in particular, which when
> > > > > coupled with the shutdown of our two CGit instances would also allow
> > > > > for us to eliminate an entire virtual machine from our systems.
> > > >
> > > > Will there be any web interface for SVN after shutdown of WebSVN?
> > > >
> > > > Can we assume https://phabricator.kde.org/source/svn/ remains
> > > > available during the next 10 years?
> > > >
> > >
> > > Phabricator's browser will be retired as part of the shutdown of
> > > Phabricator, which will take place once Gitlab has assumed
> > > responsibility for code hosting and review, and the tasks have been
> > > migrated from Phabricator.
> > >
> > > Should WebSVN be shutdown as well, then there would be no web
> > > interface to our SVN repository.
> >
> > That's not acceptable.
> 
> Mind explaining why?

Because we use it in l10n.kde.org to link to po files.

> Bear in mind that there is a cost both in terms of infrastructure, and
> people time to maintain a service such as WebSVN.

We have money, we don't have to shut down things we use because there is a cost.

Cheers,
  Albert

> This includes having to maintain a mirror of the repository on the
> machine that runs WebSVN, along with the associated infrastructure for
> allowing that mirroring to happen on the master server.
> 
> >
> > Cheers,
> >   Albert
> 
> Regards,
> Ben
> 
> >
> > >
> > > >
> > > > I'm not sure what use case Luigi has in mind, but among Russian l10n
> > > > team we often use WebSVN to refer to specific SVN revisions, for
> > > > example: https://websvn.kde.org/?view=revision=1555342
> > > >
> > > > --
> > > > Alexander Potashev
> > >
> > > Regards,
> > > Ben
> > >
> >
> >
> >
> >
> 

Re: Sysadmin Load Reduction: Subversion Infrastructure

[Friedrich W. H. Kossebau]

Am Samstag, 9. November 2019, 21:35:47 CET schrieb Ben Cooksley:
> On Sun, Nov 10, 2019 at 9:14 AM Friedrich W. H. Kossebau
> 
>  wrote:
> > Am Samstag, 9. November 2019, 19:16:20 CET schrieb Ben Cooksley:
> > > On Sun, Nov 10, 2019 at 6:04 AM Ingo Klöcker  wrote:
> > > > On Samstag, 9. November 2019 01:20:01 CET Ben Cooksley wrote:
> > > > > On top of this, i'd also like to remove commit access to it for
> > > > > everyone but translators and those who need to work on the small
> > > > > number of websites remaining on Subversion and only provision this
> > > > > for
> > > > > people on an as-needed basis.
> > > > > 
> > > > > In the next year or so i'd expect the remaining websites to complete
> > > > > their migrations to Git, after which only translators would receive
> > > > > access.
> > > > 
> > > > Restricting access to the translations repository is against the
> > > > letter of
> > > > our manifesto which states
> > > > "All source materials are hosted on infrastructure available to and
> > > > writable by all KDE contributor accounts"
> > > > https://manifesto.kde.org/commitments.html
> > > > 
> > > > AFAIK, "all source materials" includes translations.
> > > > 
> > > > There are a few reasonable exceptions for this requirement, e.g. for
> > > > the
> > > > sources of our websites, but I don't see a good reason for restricting
> > > > access to the translations.
> > > > 
> > > > I think restricting access to the translations will create a precedent
> > > > for
> > > > restricting access to other source materials and undermines the values
> > > > stated in our manifesto. Therefore, I don't think we should go down
> > > > this
> > > > route.
> > > 
> > > The access isn't being *restricted* at all.
> > > It is just something you have to request be enabled separately, and it
> > > won't be withheld from anyone with a developer account should they
> > > feel they need it.
> > 
> > This is a model we also see with rights to kick off build jobs on
> > build.kde.org, and I think it does not work out:
> > having to beg for access and having to wait for access being granted is an
> > obstacle.
> > Even more when this is nowhere documented, but a secret traded only in
> > people's minds.
> 
> As a general principle, filing a Sysadmin ticket has always been the
> way to get access to systems (developer accounts being the only
> exception), and the same applies to the CI system. Hence why there is
> no explicit documentation around this.

It stays an obstacle for everyone on first contact though. And makes one feel 
in begging position, when one should not, by ideas of the manifesto.
And often enough one needs access _now_, instead of having to hope some 
sysadmin is around in the near time.

> > So, by default there are de-facto restrictions experienced, and they get
> > in
> > the way of developer work-flows.
> 
> It's a bit unusual that a lack of access to the CI system would impact
> someone's workflow, as the CI system is supposed to automatically
> trigger itself.
> Do you have a specific scenario in mind here?

The usual scenarios where one needs to manually trigger build jobs:
a) build metadata changed (e.g. dependencies)
b) builds failed for bko-internal issues
c) branches changed, need manual first-run trigger

All things normal developers want or need to do once in a while, if they care 
for their project on KDE CI.

> > My 2 cent on this matter when it comes to conditions decided about.
> > 
> > Otherwise, thanks for all the admin work you are doing here once again :)
> > 
> > Cheers
> > Friedrich, who only an hour ago had to help someone kicking off builds
> > jobs on CI, as he once got the access granted after poking a few times
> > informally...
> Fortunately switching to Gitlab CI will resolve that specific scenario
> (needing the DSL Job run when a release is being made) as the DSL Job
> will be rendered unnecessary.

"When a release is being made" should mean, when a new stabe branch has been 
created, I guess? That would be an improvement. Still leaves scenarios a) & 
b).

Cheers
Friedrich

Re: Sysadmin Load Reduction: Subversion Infrastructure

[Ben Cooksley]

On Sun, Nov 10, 2019 at 9:32 AM Albert Vaca Cintora
 wrote:
>
> On Sat, Nov 9, 2019 at 6:53 PM Ben Cooksley  wrote:
> >
> > Unfortunately Rosetta does not have the necessary free disk space, as
> > the Subversion repository is approximately 80GB these days in size,
> > and operating WebSVN requires a full copy of the Subversion repository
> > locally in order to operate.
>
> Is this also the case if you run WebSVN next to the SVN server itself
> (ie: on the same machine)? If a single repo copy can be shared by
> WebSVN and the actual SVN server, that would reduce the storage needs.

Running WebSVN on the same machine is not safe for us to do, because
certain requests cause WebSVN to generate a substantial amount of
system load, and have caused out of memory events on systems running
it in the past.

This poses an unacceptable risk to the canonical copy of our
repositories, as well as to Gitlab (which will be hosted on the same
machine as the master Subversion repository)

>
> Albert

Regards,
Ben

Re: Sysadmin Load Reduction: Subversion Infrastructure

[Ben Cooksley]

On Sun, Nov 10, 2019 at 9:14 AM Friedrich W. H. Kossebau
 wrote:
>
> Am Samstag, 9. November 2019, 19:16:20 CET schrieb Ben Cooksley:
> > On Sun, Nov 10, 2019 at 6:04 AM Ingo Klöcker  wrote:
> > > On Samstag, 9. November 2019 01:20:01 CET Ben Cooksley wrote:
> > > > On top of this, i'd also like to remove commit access to it for
> > > > everyone but translators and those who need to work on the small
> > > > number of websites remaining on Subversion and only provision this for
> > > > people on an as-needed basis.
> > > >
> > > > In the next year or so i'd expect the remaining websites to complete
> > > > their migrations to Git, after which only translators would receive
> > > > access.
> > >
> > > Restricting access to the translations repository is against the letter of
> > > our manifesto which states
> > > "All source materials are hosted on infrastructure available to and
> > > writable by all KDE contributor accounts"
> > > https://manifesto.kde.org/commitments.html
> > >
> > > AFAIK, "all source materials" includes translations.
> > >
> > > There are a few reasonable exceptions for this requirement, e.g. for the
> > > sources of our websites, but I don't see a good reason for restricting
> > > access to the translations.
> > >
> > > I think restricting access to the translations will create a precedent for
> > > restricting access to other source materials and undermines the values
> > > stated in our manifesto. Therefore, I don't think we should go down this
> > > route.
> > The access isn't being *restricted* at all.
> > It is just something you have to request be enabled separately, and it
> > won't be withheld from anyone with a developer account should they
> > feel they need it.
>
> This is a model we also see with rights to kick off build jobs on
> build.kde.org, and I think it does not work out:
> having to beg for access and having to wait for access being granted is an
> obstacle.
> Even more when this is nowhere documented, but a secret traded only in
> people's minds.

As a general principle, filing a Sysadmin ticket has always been the
way to get access to systems (developer accounts being the only
exception), and the same applies to the CI system. Hence why there is
no explicit documentation around this.

For the most part, we've not seen many people request those rights -
and all the requests we have received have been granted.

> So, by default there are de-facto restrictions experienced, and they get in
> the way of developer work-flows.
>

It's a bit unusual that a lack of access to the CI system would impact
someone's workflow, as the CI system is supposed to automatically
trigger itself.
Do you have a specific scenario in mind here?

> My 2 cent on this matter when it comes to conditions decided about.
>
> Otherwise, thanks for all the admin work you are doing here once again :)
>
> Cheers
> Friedrich, who only an hour ago had to help someone kicking off builds jobs on
> CI, as he once got the access granted after poking a few times informally...
>
>

Fortunately switching to Gitlab CI will resolve that specific scenario
(needing the DSL Job run when a release is being made) as the DSL Job
will be rendered unnecessary.

Cheers,
Ben

Re: Sysadmin Load Reduction: Subversion Infrastructure

[Albert Vaca Cintora]

On Sat, Nov 9, 2019 at 6:53 PM Ben Cooksley  wrote:
>
> Unfortunately Rosetta does not have the necessary free disk space, as
> the Subversion repository is approximately 80GB these days in size,
> and operating WebSVN requires a full copy of the Subversion repository
> locally in order to operate.

Is this also the case if you run WebSVN next to the SVN server itself
(ie: on the same machine)? If a single repo copy can be shared by
WebSVN and the actual SVN server, that would reduce the storage needs.

Albert

Re: Sysadmin Load Reduction: Subversion Infrastructure

[Ben Cooksley]

On Sun, Nov 10, 2019 at 8:37 AM Albert Astals Cid  wrote:
>
> El dissabte, 9 de novembre de 2019, a les 19:27:07 CET, Ben Cooksley va 
> escriure:
> > On Sun, Nov 10, 2019 at 7:18 AM Alexander Potashev  
> > wrote:
> > >
> > > сб, 9 нояб. 2019 г. в 03:20, Ben Cooksley :
> > > > This would include the shutdown of WebSVN in particular, which when
> > > > coupled with the shutdown of our two CGit instances would also allow
> > > > for us to eliminate an entire virtual machine from our systems.
> > >
> > > Will there be any web interface for SVN after shutdown of WebSVN?
> > >
> > > Can we assume https://phabricator.kde.org/source/svn/ remains
> > > available during the next 10 years?
> > >
> >
> > Phabricator's browser will be retired as part of the shutdown of
> > Phabricator, which will take place once Gitlab has assumed
> > responsibility for code hosting and review, and the tasks have been
> > migrated from Phabricator.
> >
> > Should WebSVN be shutdown as well, then there would be no web
> > interface to our SVN repository.
>
> That's not acceptable.

Mind explaining why?

Bear in mind that there is a cost both in terms of infrastructure, and
people time to maintain a service such as WebSVN.
This includes having to maintain a mirror of the repository on the
machine that runs WebSVN, along with the associated infrastructure for
allowing that mirroring to happen on the master server.

>
> Cheers,
>   Albert

Regards,
Ben

>
> >
> > >
> > > I'm not sure what use case Luigi has in mind, but among Russian l10n
> > > team we often use WebSVN to refer to specific SVN revisions, for
> > > example: https://websvn.kde.org/?view=revision=1555342
> > >
> > > --
> > > Alexander Potashev
> >
> > Regards,
> > Ben
> >
>
>
>
>

Re: Sysadmin Load Reduction: Subversion Infrastructure

[Friedrich W. H. Kossebau]

Am Samstag, 9. November 2019, 19:16:20 CET schrieb Ben Cooksley:
> On Sun, Nov 10, 2019 at 6:04 AM Ingo Klöcker  wrote:
> > On Samstag, 9. November 2019 01:20:01 CET Ben Cooksley wrote:
> > > On top of this, i'd also like to remove commit access to it for
> > > everyone but translators and those who need to work on the small
> > > number of websites remaining on Subversion and only provision this for
> > > people on an as-needed basis.
> > > 
> > > In the next year or so i'd expect the remaining websites to complete
> > > their migrations to Git, after which only translators would receive
> > > access.
> > 
> > Restricting access to the translations repository is against the letter of
> > our manifesto which states
> > "All source materials are hosted on infrastructure available to and
> > writable by all KDE contributor accounts"
> > https://manifesto.kde.org/commitments.html
> > 
> > AFAIK, "all source materials" includes translations.
> > 
> > There are a few reasonable exceptions for this requirement, e.g. for the
> > sources of our websites, but I don't see a good reason for restricting
> > access to the translations.
> > 
> > I think restricting access to the translations will create a precedent for
> > restricting access to other source materials and undermines the values
> > stated in our manifesto. Therefore, I don't think we should go down this
> > route.
> The access isn't being *restricted* at all.
> It is just something you have to request be enabled separately, and it
> won't be withheld from anyone with a developer account should they
> feel they need it.

This is a model we also see with rights to kick off build jobs on 
build.kde.org, and I think it does not work out:
having to beg for access and having to wait for access being granted is an 
obstacle.
Even more when this is nowhere documented, but a secret traded only in 
people's minds.
So, by default there are de-facto restrictions experienced, and they get in 
the way of developer work-flows.

My 2 cent on this matter when it comes to conditions decided about.

Otherwise, thanks for all the admin work you are doing here once again :)

Cheers
Friedrich, who only an hour ago had to help someone kicking off builds jobs on 
CI, as he once got the access granted after poking a few times informally...

Re: Sysadmin Load Reduction: Subversion Infrastructure

[Albert Astals Cid]

El dissabte, 9 de novembre de 2019, a les 19:27:07 CET, Ben Cooksley va 
escriure:
> On Sun, Nov 10, 2019 at 7:18 AM Alexander Potashev  
> wrote:
> >
> > сб, 9 нояб. 2019 г. в 03:20, Ben Cooksley :
> > > This would include the shutdown of WebSVN in particular, which when
> > > coupled with the shutdown of our two CGit instances would also allow
> > > for us to eliminate an entire virtual machine from our systems.
> >
> > Will there be any web interface for SVN after shutdown of WebSVN?
> >
> > Can we assume https://phabricator.kde.org/source/svn/ remains
> > available during the next 10 years?
> >
> 
> Phabricator's browser will be retired as part of the shutdown of
> Phabricator, which will take place once Gitlab has assumed
> responsibility for code hosting and review, and the tasks have been
> migrated from Phabricator.
> 
> Should WebSVN be shutdown as well, then there would be no web
> interface to our SVN repository.

That's not acceptable.

Cheers,
  Albert

> 
> >
> > I'm not sure what use case Luigi has in mind, but among Russian l10n
> > team we often use WebSVN to refer to specific SVN revisions, for
> > example: https://websvn.kde.org/?view=revision=1555342
> >
> > --
> > Alexander Potashev
> 
> Regards,
> Ben
> 

Re: Sysadmin Load Reduction: Subversion Infrastructure

[Luigi Toscano]

Ben Cooksley ha scritto:
> On Sun, Nov 10, 2019 at 12:20 AM Luigi Toscano  
> wrote:
>>
>> Ben Cooksley ha scritto:
>>> Hi all,
>>>
>>> When KDE committed to performing a migration to Git back in 2010, one
>>> of the things that was agreed at the time was that translators could
>>> remain on Subversion to avoid disrupting their workflows.
>>>
>>> This however has led to a certain amount of additional infrastructure
>>> which Sysadmin needs to continue to maintain.
>>>
>>> In recognition of the fact that with few exceptions, everything has
>>> now migrated from Subversion aside from Translations, i'd like to
>>> reduce the level of infrastructure supporting our Subversion
>>> repository to the bare minimum necessary.
>>>
>>> This would include the shutdown of WebSVN in particular, which when
>>> coupled with the shutdown of our two CGit instances would also allow
>>> for us to eliminate an entire virtual machine from our systems.
>>
>> As websvn has proven to be useful, would it help the sysadmin (pending
>> agreement with aacid and the other people with root access on rosetta) to 
>> move
>> websvn to rosetta and under localization team's responsibility?
> 
> Unfortunately Rosetta does not have the necessary free disk space, as
> the Subversion repository is approximately 80GB these days in size,
> and operating WebSVN requires a full copy of the Subversion repository
> locally in order to operate.

Can we extend rosetta's disk then, or add an external volume?

In case it's not clear, I'd like to do everything possible to have websvn
still up.

-- 
Luigi

Re: Sysadmin Load Reduction: Subversion Infrastructure

[Ben Cooksley]

On Sun, Nov 10, 2019 at 7:18 AM Alexander Potashev  wrote:
>
> сб, 9 нояб. 2019 г. в 03:20, Ben Cooksley :
> > This would include the shutdown of WebSVN in particular, which when
> > coupled with the shutdown of our two CGit instances would also allow
> > for us to eliminate an entire virtual machine from our systems.
>
> Will there be any web interface for SVN after shutdown of WebSVN?
>
> Can we assume https://phabricator.kde.org/source/svn/ remains
> available during the next 10 years?
>

Phabricator's browser will be retired as part of the shutdown of
Phabricator, which will take place once Gitlab has assumed
responsibility for code hosting and review, and the tasks have been
migrated from Phabricator.

Should WebSVN be shutdown as well, then there would be no web
interface to our SVN repository.

>
> I'm not sure what use case Luigi has in mind, but among Russian l10n
> team we often use WebSVN to refer to specific SVN revisions, for
> example: https://websvn.kde.org/?view=revision=1555342
>
> --
> Alexander Potashev

Regards,
Ben

Re: Sysadmin Load Reduction: Subversion Infrastructure

[Alexander Potashev]

сб, 9 нояб. 2019 г. в 03:20, Ben Cooksley :
> This would include the shutdown of WebSVN in particular, which when
> coupled with the shutdown of our two CGit instances would also allow
> for us to eliminate an entire virtual machine from our systems.

Will there be any web interface for SVN after shutdown of WebSVN?

Can we assume https://phabricator.kde.org/source/svn/ remains
available during the next 10 years?


I'm not sure what use case Luigi has in mind, but among Russian l10n
team we often use WebSVN to refer to specific SVN revisions, for
example: https://websvn.kde.org/?view=revision=1555342

-- 
Alexander Potashev

Re: Sysadmin Load Reduction: Subversion Infrastructure

[Ben Cooksley]

On Sun, Nov 10, 2019 at 6:04 AM Ingo Klöcker  wrote:
>
> On Samstag, 9. November 2019 01:20:01 CET Ben Cooksley wrote:
> > On top of this, i'd also like to remove commit access to it for
> > everyone but translators and those who need to work on the small
> > number of websites remaining on Subversion and only provision this for
> > people on an as-needed basis.
> >
> > In the next year or so i'd expect the remaining websites to complete
> > their migrations to Git, after which only translators would receive
> > access.
>
> Restricting access to the translations repository is against the letter of our
> manifesto which states
> "All source materials are hosted on infrastructure available to and writable
> by all KDE contributor accounts"
> https://manifesto.kde.org/commitments.html
>
> AFAIK, "all source materials" includes translations.
>
> There are a few reasonable exceptions for this requirement, e.g. for the
> sources of our websites, but I don't see a good reason for restricting access
> to the translations.
>
> I think restricting access to the translations will create a precedent for
> restricting access to other source materials and undermines the values stated
> in our manifesto. Therefore, I don't think we should go down this route.

The access isn't being *restricted* at all.
It is just something you have to request be enabled separately, and it
won't be withheld from anyone with a developer account should they
feel they need it.

There will be a proportion of newer contributors who will have never
worked with SVN, and will likely never need to.

Continuing to provision access to our Subversion repository for every
developer means that Sysadmin will continue to be limited in the
manner in which we handle developer account applications (because
people need to be allocated a username which can never be changed as
it gets baked into the metadata within the repository).

This set of requirements is part of the reason why we have issues with
Identity currently, and why some developers actually have multiple
accounts (as they've changed names, and therefore wanted to change
their username as well - we usually grant them a new account and
disable the old one when this issue occurs).

Given that people are likely to never use the service, and because it
creates issues for us to continue to provision this access as part of
the normal developer account application workflow, it makes sense for
people to need to request access to Subversion separately going
forward to reduce the cost burden of providing this service.

Should you feel it necessary, i'm more than happy to start the process
of amending the Manifesto to explicitly permit this activity although
I feel that what we're proposing here is more than permitted within
the Manifesto as it currently stands (there is no exception for
website repositories in the Manifesto either)

>
> Regards,
> Ingo

Regards,
Ben

Re: Sysadmin Load Reduction: Subversion Infrastructure

[Ben Cooksley]

On Sun, Nov 10, 2019 at 12:20 AM Luigi Toscano  wrote:
>
> Ben Cooksley ha scritto:
> > Hi all,
> >
> > When KDE committed to performing a migration to Git back in 2010, one
> > of the things that was agreed at the time was that translators could
> > remain on Subversion to avoid disrupting their workflows.
> >
> > This however has led to a certain amount of additional infrastructure
> > which Sysadmin needs to continue to maintain.
> >
> > In recognition of the fact that with few exceptions, everything has
> > now migrated from Subversion aside from Translations, i'd like to
> > reduce the level of infrastructure supporting our Subversion
> > repository to the bare minimum necessary.
> >
> > This would include the shutdown of WebSVN in particular, which when
> > coupled with the shutdown of our two CGit instances would also allow
> > for us to eliminate an entire virtual machine from our systems.
>
> As websvn has proven to be useful, would it help the sysadmin (pending
> agreement with aacid and the other people with root access on rosetta) to move
> websvn to rosetta and under localization team's responsibility?

Unfortunately Rosetta does not have the necessary free disk space, as
the Subversion repository is approximately 80GB these days in size,
and operating WebSVN requires a full copy of the Subversion repository
locally in order to operate.

>
> Ciao
> --
> Luigi

Cheers,
Ben

Re: Sysadmin Load Reduction: Subversion Infrastructure

[Ingo Klöcker]

On Samstag, 9. November 2019 01:20:01 CET Ben Cooksley wrote:
> On top of this, i'd also like to remove commit access to it for
> everyone but translators and those who need to work on the small
> number of websites remaining on Subversion and only provision this for
> people on an as-needed basis.
> 
> In the next year or so i'd expect the remaining websites to complete
> their migrations to Git, after which only translators would receive
> access.

Restricting access to the translations repository is against the letter of our 
manifesto which states
"All source materials are hosted on infrastructure available to and writable 
by all KDE contributor accounts"
https://manifesto.kde.org/commitments.html

AFAIK, "all source materials" includes translations.

There are a few reasonable exceptions for this requirement, e.g. for the 
sources of our websites, but I don't see a good reason for restricting access 
to the translations.

I think restricting access to the translations will create a precedent for 
restricting access to other source materials and undermines the values stated 
in our manifesto. Therefore, I don't think we should go down this route.

Regards,
Ingo


signature.asc
Description: This is a digitally signed message part.

Re: Sysadmin Load Reduction: Subversion Infrastructure

[Luigi Toscano]

Ben Cooksley ha scritto:
> Hi all,
> 
> When KDE committed to performing a migration to Git back in 2010, one
> of the things that was agreed at the time was that translators could
> remain on Subversion to avoid disrupting their workflows.
> 
> This however has led to a certain amount of additional infrastructure
> which Sysadmin needs to continue to maintain.
> 
> In recognition of the fact that with few exceptions, everything has
> now migrated from Subversion aside from Translations, i'd like to
> reduce the level of infrastructure supporting our Subversion
> repository to the bare minimum necessary.
> 
> This would include the shutdown of WebSVN in particular, which when
> coupled with the shutdown of our two CGit instances would also allow
> for us to eliminate an entire virtual machine from our systems.

As websvn has proven to be useful, would it help the sysadmin (pending
agreement with aacid and the other people with root access on rosetta) to move
websvn to rosetta and under localization team's responsibility?

Ciao
-- 
Luigi

Sysadmin Load Reduction: Subversion Infrastructure

[Ben Cooksley]

Hi all,

When KDE committed to performing a migration to Git back in 2010, one
of the things that was agreed at the time was that translators could
remain on Subversion to avoid disrupting their workflows.

This however has led to a certain amount of additional infrastructure
which Sysadmin needs to continue to maintain.

In recognition of the fact that with few exceptions, everything has
now migrated from Subversion aside from Translations, i'd like to
reduce the level of infrastructure supporting our Subversion
repository to the bare minimum necessary.

This would include the shutdown of WebSVN in particular, which when
coupled with the shutdown of our two CGit instances would also allow
for us to eliminate an entire virtual machine from our systems.

On top of this, i'd also like to remove commit access to it for
everyone but translators and those who need to work on the small
number of websites remaining on Subversion and only provision this for
people on an as-needed basis.

In the next year or so i'd expect the remaining websites to complete
their migrations to Git, after which only translators would receive
access.

We would also cease providing geographically distributed anonsvn
service, with anonymous access only being provided by the master
server going forward.

Any comments?

Thanks,
Ben