Re: Splitting the security-tracker repo [was Re: [PATCH 00/12] Plannings for secure-testing repository migration to git]

[Moritz Mühlenhoff]

On Thu, Dec 28, 2017 at 09:02:56PM +0100, Salvatore Bonaccorso wrote:
> With that i think we have most all together to do the switch, any
> comments from others? Objections? Concerns? Or somewhing we have
> overseen?

Thanks for your and Guido's work on this!

>From my point of view I'm in favour of moving to salsa ASAP, we can
only really work out the kinks by using this in production.

Cheers,
Moritz

Re: Splitting the security-tracker repo [was Re: [PATCH 00/12] Plannings for secure-testing repository migration to git]

[Salvatore Bonaccorso]

Hi Guido,

On Thu, Dec 28, 2017 at 09:11:35PM +0100, Guido Günther wrote:
> Hi Salvatore,
> On Thu, Dec 28, 2017 at 09:02:56PM +0100, Salvatore Bonaccorso wrote:
> > Hi Guido,
> > 
> > On Thu, Dec 28, 2017 at 06:59:11PM +0100, Guido Günther wrote:
> > > Hi,
> > > On Thu, Dec 28, 2017 at 12:33:27PM +0100, Guido Günther wrote:
> > > > Hi,
> > > > On Wed, Dec 27, 2017 at 11:32:53PM +0100, Salvatore Bonaccorso wrote:
> > > > [..snip..]
> > > > > have that already :). But I'm unahappy with my current versions, thus
> > > > > I have not sent those yet. Basically: I'm struggling with a proper
> > > > > replacement of such constructs:
> > > > > 
> > > > > cd /srv/security-tracker.debian.org/website/secure-testing/data && 
> > > > > svn update && ../bin/update && svn commit -qm "automatic update" CVE 
> > > > > && cd CVE && ../../bin/bts-update list
> > > > 
> > > >cd /srv/security-tracker.debian.org/website/security-tracker/data && 
> > > > git pull origin master && ../bin/update && git commit -qm "automatic 
> > > > update" CVE && git push origin master && cd CVE && ../../bin/bts-update 
> > > > list
> > > > 
> > > > git commit might fail if there are no changes so we need to either guard
> > > > for that or use --allow-empty. git push might fail as well if there were
> > > > pushes in between. Since there are several commands that
> > > > 
> > > > pull, modify, commit push
> > > > 
> > > > should we move this to a script that
> > > > 
> > > > pulls, invokes the modification command, commits when there are
> > > > changes, pushes. If that fails merges again && pushes again.
> > > > 
> > > > That won't save you from merge conflicts but that's the same with SVN a
> > > > assume?
> > > 
> > > Salvatore pointed out to me that it would be good to not mix tracker
> > > code updates with data updates so that the above commands that pull and
> > > push data to/from git would only operate on the data part but not on the
> > > tracker itself. One possible solution would be to split the current
> > > secure-testing repository into two:
> > > 
> > > security-tracker: containing the data that is the data/ packages/ org/
> > > doc/ and conf/ directories.
> > > security-tracker-bin: containing the security tracker service and other
> > > scripts and python modules: check-external, bin, tools, lib,
> > > templates, website
> > > 
> > > security-tracker-bin could then become a submodule of security
> > > tracker. To ease migration we would create symlinks so the directory
> > > layout stays the same. The top level of the secure-testing repo would
> > > look something like this:
> > > 
> > > -rw-r--r--  1 agx agx 3449 Dez 28 11:56 TODO.gitmigration
> > > drwxr-sr-x  9 agx agx 4096 Dez 28 11:56 data
> > > drwxr-sr-x  2 agx agx 4096 Dez 28 11:56 packages
> > > drwxr-sr-x  2 agx agx 4096 Dez 28 11:56 org
> > > drwxr-sr-x  5 agx agx 4096 Dez 28 11:56 doc
> > > -rw-r--r--  1 agx agx 8450 Dez 28 12:01 Makefile
> > > drwxr-sr-x  2 agx agx 4096 Dez 28 13:30 conf
> > > -rw-r--r--  1 agx agx  179 Dez 28 17:20 .gitmodules
> > > lrwxrwxrwx  1 agx agx   35 Dez 28 17:20 check-external -> 
> > > security-tracker-bin/check-external
> > > lrwxrwxrwx  1 agx agx   24 Dez 28 17:20 bin -> 
> > > security-tracker-bin/bin
> > > lrwxrwxrwx  1 agx agx   26 Dez 28 17:20 tools -> 
> > > security-tracker-bin/tools
> > > lrwxrwxrwx  1 agx agx   30 Dez 28 17:20 templates -> 
> > > security-tracker-bin/templates
> > > lrwxrwxrwx  1 agx agx   27 Dez 28 17:20 static -> 
> > > security-tracker-bin/static
> > > lrwxrwxrwx  1 agx agx   24 Dez 28 17:20 lib -> 
> > > security-tracker-bin/lib
> > > lrwxrwxrwx  1 agx agx   28 Dez 28 17:20 website -> 
> > > security-tracker-bin/website
> > > drwxr-sr-x 10 agx agx 4096 Dez 28 17:20 .
> > > drwxr-sr-x  9 agx agx 4096 Dez 28 17:25 security-tracker-bin
> > > drwxr-sr-x  2 agx agx 4096 Dez 28 17:26 stamps
> > > drwxr-sr-x  9 agx agx 4096 Dez 28 17:26 .git
> > > drwxr-sr-x  4 agx agx 4096 Dez 28 18:17 ..
> > > 
> > > I have scripts¹ that produce such a layout and pushed temporary repos
> > > here
> > 
> > That looks good to me, can you push your conversion scripts to
> > tools/git-migration?
> 
> Comitted to the security-testing svn repository.

Thank you. There is one further change needed I think, and one
question/concern.

The Makefile must be as well not in the security-tracker.git, but
linked from there. But now this is still not safe to have
security-tracker-bin git as submodule, because one can just replace
the symlinks we created.

So in the long run we would better of to decouple the data part and
code part defintively, but I wonder if for now the safest road would
be to go back one step, do the svn to git migration in one repository
and for the autmoatic updates triggeres, operate in a separate
workdir.

Thoughs?

Regards,
Salvatore

Re: Splitting the security-tracker repo [was Re: [PATCH 00/12] Plannings for secure-testing repository migration to git]

[Guido Günther]

Hi Salvatore,
On Thu, Dec 28, 2017 at 09:02:56PM +0100, Salvatore Bonaccorso wrote:
> Hi Guido,
> 
> On Thu, Dec 28, 2017 at 06:59:11PM +0100, Guido Günther wrote:
> > Hi,
> > On Thu, Dec 28, 2017 at 12:33:27PM +0100, Guido Günther wrote:
> > > Hi,
> > > On Wed, Dec 27, 2017 at 11:32:53PM +0100, Salvatore Bonaccorso wrote:
> > > [..snip..]
> > > > have that already :). But I'm unahappy with my current versions, thus
> > > > I have not sent those yet. Basically: I'm struggling with a proper
> > > > replacement of such constructs:
> > > > 
> > > > cd /srv/security-tracker.debian.org/website/secure-testing/data && svn 
> > > > update && ../bin/update && svn commit -qm "automatic update" CVE && cd 
> > > > CVE && ../../bin/bts-update list
> > > 
> > >cd /srv/security-tracker.debian.org/website/security-tracker/data && 
> > > git pull origin master && ../bin/update && git commit -qm "automatic 
> > > update" CVE && git push origin master && cd CVE && ../../bin/bts-update 
> > > list
> > > 
> > > git commit might fail if there are no changes so we need to either guard
> > > for that or use --allow-empty. git push might fail as well if there were
> > > pushes in between. Since there are several commands that
> > > 
> > > pull, modify, commit push
> > > 
> > > should we move this to a script that
> > > 
> > > pulls, invokes the modification command, commits when there are
> > > changes, pushes. If that fails merges again && pushes again.
> > > 
> > > That won't save you from merge conflicts but that's the same with SVN a
> > > assume?
> > 
> > Salvatore pointed out to me that it would be good to not mix tracker
> > code updates with data updates so that the above commands that pull and
> > push data to/from git would only operate on the data part but not on the
> > tracker itself. One possible solution would be to split the current
> > secure-testing repository into two:
> > 
> > security-tracker: containing the data that is the data/ packages/ org/
> > doc/ and conf/ directories.
> > security-tracker-bin: containing the security tracker service and other
> > scripts and python modules: check-external, bin, tools, lib,
> > templates, website
> > 
> > security-tracker-bin could then become a submodule of security
> > tracker. To ease migration we would create symlinks so the directory
> > layout stays the same. The top level of the secure-testing repo would
> > look something like this:
> > 
> > -rw-r--r--  1 agx agx 3449 Dez 28 11:56 TODO.gitmigration
> > drwxr-sr-x  9 agx agx 4096 Dez 28 11:56 data
> > drwxr-sr-x  2 agx agx 4096 Dez 28 11:56 packages
> > drwxr-sr-x  2 agx agx 4096 Dez 28 11:56 org
> > drwxr-sr-x  5 agx agx 4096 Dez 28 11:56 doc
> > -rw-r--r--  1 agx agx 8450 Dez 28 12:01 Makefile
> > drwxr-sr-x  2 agx agx 4096 Dez 28 13:30 conf
> > -rw-r--r--  1 agx agx  179 Dez 28 17:20 .gitmodules
> > lrwxrwxrwx  1 agx agx   35 Dez 28 17:20 check-external -> 
> > security-tracker-bin/check-external
> > lrwxrwxrwx  1 agx agx   24 Dez 28 17:20 bin -> security-tracker-bin/bin
> > lrwxrwxrwx  1 agx agx   26 Dez 28 17:20 tools -> 
> > security-tracker-bin/tools
> > lrwxrwxrwx  1 agx agx   30 Dez 28 17:20 templates -> 
> > security-tracker-bin/templates
> > lrwxrwxrwx  1 agx agx   27 Dez 28 17:20 static -> 
> > security-tracker-bin/static
> > lrwxrwxrwx  1 agx agx   24 Dez 28 17:20 lib -> security-tracker-bin/lib
> > lrwxrwxrwx  1 agx agx   28 Dez 28 17:20 website -> 
> > security-tracker-bin/website
> > drwxr-sr-x 10 agx agx 4096 Dez 28 17:20 .
> > drwxr-sr-x  9 agx agx 4096 Dez 28 17:25 security-tracker-bin
> > drwxr-sr-x  2 agx agx 4096 Dez 28 17:26 stamps
> > drwxr-sr-x  9 agx agx 4096 Dez 28 17:26 .git
> > drwxr-sr-x  4 agx agx 4096 Dez 28 18:17 ..
> > 
> > I have scripts¹ that produce such a layout and pushed temporary repos
> > here
> 
> That looks good to me, can you push your conversion scripts to
> tools/git-migration?

Comitted to the security-testing svn repository.

Cheers,
 -- Guido

Re: Splitting the security-tracker repo [was Re: [PATCH 00/12] Plannings for secure-testing repository migration to git]

[Salvatore Bonaccorso]

Hi Guido,

On Thu, Dec 28, 2017 at 06:59:11PM +0100, Guido Günther wrote:
> Hi,
> On Thu, Dec 28, 2017 at 12:33:27PM +0100, Guido Günther wrote:
> > Hi,
> > On Wed, Dec 27, 2017 at 11:32:53PM +0100, Salvatore Bonaccorso wrote:
> > [..snip..]
> > > have that already :). But I'm unahappy with my current versions, thus
> > > I have not sent those yet. Basically: I'm struggling with a proper
> > > replacement of such constructs:
> > > 
> > > cd /srv/security-tracker.debian.org/website/secure-testing/data && svn 
> > > update && ../bin/update && svn commit -qm "automatic update" CVE && cd 
> > > CVE && ../../bin/bts-update list
> > 
> >cd /srv/security-tracker.debian.org/website/security-tracker/data && git 
> > pull origin master && ../bin/update && git commit -qm "automatic update" 
> > CVE && git push origin master && cd CVE && ../../bin/bts-update list
> > 
> > git commit might fail if there are no changes so we need to either guard
> > for that or use --allow-empty. git push might fail as well if there were
> > pushes in between. Since there are several commands that
> > 
> > pull, modify, commit push
> > 
> > should we move this to a script that
> > 
> > pulls, invokes the modification command, commits when there are
> > changes, pushes. If that fails merges again && pushes again.
> > 
> > That won't save you from merge conflicts but that's the same with SVN a
> > assume?
> 
> Salvatore pointed out to me that it would be good to not mix tracker
> code updates with data updates so that the above commands that pull and
> push data to/from git would only operate on the data part but not on the
> tracker itself. One possible solution would be to split the current
> secure-testing repository into two:
> 
> security-tracker: containing the data that is the data/ packages/ org/
> doc/ and conf/ directories.
> security-tracker-bin: containing the security tracker service and other
> scripts and python modules: check-external, bin, tools, lib,
> templates, website
> 
> security-tracker-bin could then become a submodule of security
> tracker. To ease migration we would create symlinks so the directory
> layout stays the same. The top level of the secure-testing repo would
> look something like this:
> 
> -rw-r--r--  1 agx agx 3449 Dez 28 11:56 TODO.gitmigration
> drwxr-sr-x  9 agx agx 4096 Dez 28 11:56 data
> drwxr-sr-x  2 agx agx 4096 Dez 28 11:56 packages
> drwxr-sr-x  2 agx agx 4096 Dez 28 11:56 org
> drwxr-sr-x  5 agx agx 4096 Dez 28 11:56 doc
> -rw-r--r--  1 agx agx 8450 Dez 28 12:01 Makefile
> drwxr-sr-x  2 agx agx 4096 Dez 28 13:30 conf
> -rw-r--r--  1 agx agx  179 Dez 28 17:20 .gitmodules
> lrwxrwxrwx  1 agx agx   35 Dez 28 17:20 check-external -> 
> security-tracker-bin/check-external
> lrwxrwxrwx  1 agx agx   24 Dez 28 17:20 bin -> security-tracker-bin/bin
> lrwxrwxrwx  1 agx agx   26 Dez 28 17:20 tools -> 
> security-tracker-bin/tools
> lrwxrwxrwx  1 agx agx   30 Dez 28 17:20 templates -> 
> security-tracker-bin/templates
> lrwxrwxrwx  1 agx agx   27 Dez 28 17:20 static -> 
> security-tracker-bin/static
> lrwxrwxrwx  1 agx agx   24 Dez 28 17:20 lib -> security-tracker-bin/lib
> lrwxrwxrwx  1 agx agx   28 Dez 28 17:20 website -> 
> security-tracker-bin/website
> drwxr-sr-x 10 agx agx 4096 Dez 28 17:20 .
> drwxr-sr-x  9 agx agx 4096 Dez 28 17:25 security-tracker-bin
> drwxr-sr-x  2 agx agx 4096 Dez 28 17:26 stamps
> drwxr-sr-x  9 agx agx 4096 Dez 28 17:26 .git
> drwxr-sr-x  4 agx agx 4096 Dez 28 18:17 ..
> 
> I have scripts¹ that produce such a layout and pushed temporary repos
> here

That looks good to me, can you push your conversion scripts to
tools/git-migration?

> https://salsa.debian.org/agx/security-tracker-with-submodule
> https://salsa.debian.org/agx/security-tracker-bin
> 
> So if you want to see how it looks to a
> 
> git clone --recursive 
> https://salsa.debian.org/agx/security-tracker-with-submodule
> 
> With some follow up work we can get rid of most of the symlinks but
> this could happen after the git migration. So soriano would only see an
> update of security-tracker-bin code if we bump the submodule revision
> (end even then only if we do a "git submodule update").

Indeed, if we want that we should do so only after migration.
> 
> Does this make sense or does it introduce too much complexity? Is the
> above split about right or should some of the directories move between
> security-tracker and security-tracker-bin (e.g. website or static).

This looks good to me (still testing a bit right now), but I guess we
should to that change during the migration because of the above
mentioned problem where we only want changed to data/ autmoatically
updated.

With that i think we have most all together to do the switch, any
comments from others? Objections? Concerns? Or somewhing we have
overseen?

I will write up a migration plan/checklist, and would be happy to work

Splitting the security-tracker repo [was Re: [PATCH 00/12] Plannings for secure-testing repository migration to git]

[Guido Günther]

Hi,
On Thu, Dec 28, 2017 at 12:33:27PM +0100, Guido Günther wrote:
> Hi,
> On Wed, Dec 27, 2017 at 11:32:53PM +0100, Salvatore Bonaccorso wrote:
> [..snip..]
> > have that already :). But I'm unahappy with my current versions, thus
> > I have not sent those yet. Basically: I'm struggling with a proper
> > replacement of such constructs:
> > 
> > cd /srv/security-tracker.debian.org/website/secure-testing/data && svn 
> > update && ../bin/update && svn commit -qm "automatic update" CVE && cd CVE 
> > && ../../bin/bts-update list
> 
>cd /srv/security-tracker.debian.org/website/security-tracker/data && git 
> pull origin master && ../bin/update && git commit -qm "automatic update" CVE 
> && git push origin master && cd CVE && ../../bin/bts-update list
> 
> git commit might fail if there are no changes so we need to either guard
> for that or use --allow-empty. git push might fail as well if there were
> pushes in between. Since there are several commands that
> 
> pull, modify, commit push
> 
> should we move this to a script that
> 
> pulls, invokes the modification command, commits when there are
> changes, pushes. If that fails merges again && pushes again.
> 
> That won't save you from merge conflicts but that's the same with SVN a
> assume?

Salvatore pointed out to me that it would be good to not mix tracker
code updates with data updates so that the above commands that pull and
push data to/from git would only operate on the data part but not on the
tracker itself. One possible solution would be to split the current
secure-testing repository into two:

security-tracker: containing the data that is the data/ packages/ org/
doc/ and conf/ directories.
security-tracker-bin: containing the security tracker service and other
scripts and python modules: check-external, bin, tools, lib,
templates, website

security-tracker-bin could then become a submodule of security
tracker. To ease migration we would create symlinks so the directory
layout stays the same. The top level of the secure-testing repo would
look something like this:

-rw-r--r--  1 agx agx 3449 Dez 28 11:56 TODO.gitmigration
drwxr-sr-x  9 agx agx 4096 Dez 28 11:56 data
drwxr-sr-x  2 agx agx 4096 Dez 28 11:56 packages
drwxr-sr-x  2 agx agx 4096 Dez 28 11:56 org
drwxr-sr-x  5 agx agx 4096 Dez 28 11:56 doc
-rw-r--r--  1 agx agx 8450 Dez 28 12:01 Makefile
drwxr-sr-x  2 agx agx 4096 Dez 28 13:30 conf
-rw-r--r--  1 agx agx  179 Dez 28 17:20 .gitmodules
lrwxrwxrwx  1 agx agx   35 Dez 28 17:20 check-external -> 
security-tracker-bin/check-external
lrwxrwxrwx  1 agx agx   24 Dez 28 17:20 bin -> security-tracker-bin/bin
lrwxrwxrwx  1 agx agx   26 Dez 28 17:20 tools -> security-tracker-bin/tools
lrwxrwxrwx  1 agx agx   30 Dez 28 17:20 templates -> 
security-tracker-bin/templates
lrwxrwxrwx  1 agx agx   27 Dez 28 17:20 static -> 
security-tracker-bin/static
lrwxrwxrwx  1 agx agx   24 Dez 28 17:20 lib -> security-tracker-bin/lib
lrwxrwxrwx  1 agx agx   28 Dez 28 17:20 website -> 
security-tracker-bin/website
drwxr-sr-x 10 agx agx 4096 Dez 28 17:20 .
drwxr-sr-x  9 agx agx 4096 Dez 28 17:25 security-tracker-bin
drwxr-sr-x  2 agx agx 4096 Dez 28 17:26 stamps
drwxr-sr-x  9 agx agx 4096 Dez 28 17:26 .git
drwxr-sr-x  4 agx agx 4096 Dez 28 18:17 ..

I have scripts¹ that produce such a layout and pushed temporary repos
here

https://salsa.debian.org/agx/security-tracker-with-submodule
https://salsa.debian.org/agx/security-tracker-bin

So if you want to see how it looks to a

git clone --recursive 
https://salsa.debian.org/agx/security-tracker-with-submodule

With some follow up work we can get rid of most of the symlinks but
this could happen after the git migration. So soriano would only see an
update of security-tracker-bin code if we bump the submodule revision
(end even then only if we do a "git submodule update").

Does this make sense or does it introduce too much complexity? Is the
above split about right or should some of the directories move between
security-tracker and security-tracker-bin (e.g. website or static).

Feedback would be welcome.

Cheers,
 -- Guido

¹: 
https://salsa.debian.org/agx/security-tracker/commit/4a4a83117e3d6791fbe091a4de2a2bc1ad0abf98

Re: [PATCH 00/12] Plannings for secure-testing repository migration to git

[Salvatore Bonaccorso]

Hi Guido,

On Thu, Dec 28, 2017 at 01:28:26PM +0100, Guido Günther wrote:
> Hi,
> On Thu, Dec 28, 2017 at 12:45:38PM +0100, Guido Günther wrote:
> [..snip..]
> > We could provide a pre-commit hook:
> > 
> > https://git-scm.com/book/gr/v2/Customizing-Git-Git-Hooks
> > 
> > they're not enforced though. User would have to set it up via
> > 
> > ln -s bin/pre-commit .git/hooks/pre-commit
> > 
> > (which we could put into a "bin/setup-repo"). Since the hook can
> > identify changed files we could only run tests on changed files so it
> > would be reasonably fast. I'll have a look.
> 
> What about adding this to the secure-testing-repo (the makefile already
> creates stamps so we don't need to differentiate in the hook again). We
> might add other stuff like installing dependencies to run the various
> commands to setup-repo later on.

Thank you looks good and have applied it as well!

Regards,
Salvatore

Re: [PATCH 00/12] Plannings for secure-testing repository migration to git

[Salvatore Bonaccorso]

Hi Guido,

On Thu, Dec 28, 2017 at 12:04:12PM +0100, Guido Günther wrote:
> Hi Salvatore,
> On Wed, Dec 27, 2017 at 11:32:53PM +0100, Salvatore Bonaccorso wrote:
> > Hi Guido
> > 
> > Much appreciated you looked at the patchset!
> > 
> > On Wed, Dec 27, 2017 at 09:14:52PM +0100, Guido Günther wrote:
> > > Hi,
> > > On Wed, Dec 27, 2017 at 08:42:49PM +0100, Salvatore Bonaccorso wrote:
> > > > This *preliminary* patchset for review is for the changes needed after
> > > > we would switch from alioth to salsa for the secure-testing repository.
> > > > While needing to do the svn to git conversion the opportunity was taken
> > > > to rename the repository to security-tracker as the naming
> > > > secure-testing was for historical reason when the testing security
> > > > support arised.
> > > 
> > > This looks good! I applied your patches to my checkout I found some more
> > > svn references in
> > > 
> > > * tracker 'latest_revision' method
> > 
> > Jupp that was what I mean that I need here some insights from a LTS
> > member and how the script functions. 
> 
> Here's a patch also available at
> 
>
> https://salsa.debian.org/agx/security-tracker/commit/805a2b5fe27f9812dc679c231ab4dd1a1c9613c1

Thank you applied, with one change:

> +GIT_URL = 
> "https://salsa.debian.org/security-tracker-team/security-tracker-service.git;

Only changed that to 

https://salsa.debian.org/security-tracker-team/security-tracker.git

Regards,
Salvatore

Re: [PATCH 00/12] Plannings for secure-testing repository migration to git

[Guido Günther]

Hi,
On Thu, Dec 28, 2017 at 12:45:38PM +0100, Guido Günther wrote:
[..snip..]
> We could provide a pre-commit hook:
> 
> https://git-scm.com/book/gr/v2/Customizing-Git-Git-Hooks
> 
> they're not enforced though. User would have to set it up via
> 
> ln -s bin/pre-commit .git/hooks/pre-commit
> 
> (which we could put into a "bin/setup-repo"). Since the hook can
> identify changed files we could only run tests on changed files so it
> would be reasonably fast. I'll have a look.

What about adding this to the secure-testing-repo (the makefile already
creates stamps so we don't need to differentiate in the hook again). We
might add other stuff like installing dependencies to run the various
commands to setup-repo later on.

>From 9e0f77e41b4e194e320609c0d74ced3a82b06f05 Mon Sep 17 00:00:00 2001
Message-Id: 
<9e0f77e41b4e194e320609c0d74ced3a82b06f05.1514463957.git@sigxcpu.org>
From: =?UTF-8?q?Guido=20G=C3=BCnther?= 
Date: Thu, 28 Dec 2017 13:05:05 +0100
Subject: [PATCH] Add pre-commit hook to check syntax

---
 bin/setup-repo  | 31 +++
 conf/pre-commit | 12 
 2 files changed, 43 insertions(+)
 create mode 100755 bin/setup-repo
 create mode 100755 conf/pre-commit

diff --git a/bin/setup-repo b/bin/setup-repo
new file mode 100755
index 00..ef7cf2c95b
--- /dev/null
+++ b/bin/setup-repo
@@ -0,0 +1,31 @@
+#!/bin/sh
+#
+# Set up a clone of the security-tracker git repo
+
+set -e
+
+SRC=../../conf/pre-commit
+HOOK=.git/hooks/pre-commit
+
+install_pre_commit_hook() {
+  if [ -L "${HOOK}" ] && [ "$(readlink ${HOOK})" = "${SRC}" ]; then
+  echo "pre-commit hook already set up"
+  return
+  fi
+
+  if [ -e "${HOOK}" ] || [ -L "${HOOK}" ]; then
+echo "Moving old pre-commit hook"
+ mv -f "${HOOK}" "${HOOK}.$(date '+%s')"
+  fi
+
+  echo "Installing pre-commit hook"
+  ln -s "${SRC}" "${HOOK}"
+}
+
+
+if [ "$(git rev-parse --show-cdup)" != '' ] || [ -d data/CVE/list ]; then
+ echo "This does not look like the git repo of the security tracker" 1>&2
+ exit 1
+fi
+
+install_pre_commit_hook
diff --git a/conf/pre-commit b/conf/pre-commit
new file mode 100755
index 00..f097a88abd
--- /dev/null
+++ b/conf/pre-commit
@@ -0,0 +1,12 @@
+#!/bin/sh
+
+set -e
+
+if [ -z "${GIT_DIR}" ]; then
+echo "GIT_DIR not set" 1>&2
+exit 1
+fi
+
+exec 1>&2
+
+make check-syntax
-- 
2.15.1

Re: [PATCH 00/12] Plannings for secure-testing repository migration to git

[Guido Günther]

Hi,
On Thu, Dec 28, 2017 at 09:13:09AM +0100, Salvatore Bonaccorso wrote:
> Hi
> 
> On Wed, Dec 27, 2017 at 11:32:53PM +0100, Salvatore Bonaccorso wrote:
> > 1/ desire of commit mailinglist
> 
> Would there be objection to switch those to
> debian-security-tracker@l.d.o? A dedicated list IHMO would be better
> to distinguish between the commit review list and the general
> discussion/bug list.
> 
> > 2/ how to do the updates with the sectracker role account specific to
> > some files (e.g. automatic updates, data/CVE/list update, dsa
> > canditate finding).
> 
> Btw, 3/ is there a way to force a pre-commit/pre-push action with git
> repositories (so client side hooks forced on every cloned repository)
> which would run 'make check-syntax' before we push something? For the
> subversion repository this was implemented with a pre-commit svn hook,
> which called for changes with bin/check-syntax, or
> bin/check-new-issues -c (for the data/embedded-code-copies file).
> 
> I guess the short answer is 'no'.

We could provide a pre-commit hook:

https://git-scm.com/book/gr/v2/Customizing-Git-Git-Hooks

they're not enforced though. User would have to set it up via

ln -s bin/pre-commit .git/hooks/pre-commit

(which we could put into a "bin/setup-repo"). Since the hook can
identify changed files we could only run tests on changed files so it
would be reasonably fast. I'll have a look.

> I got the reply for Salsa, for the custom_hooks referring to
> https://docs.gitlab.com/ce/administration/custom_hooks.html but as far
> I understand those are server-side running hooks.

Will it be possible for us to add such a hook on salsa? AFAIK adding
these needs admin permissions on the GitLab instance. Having an
in-flight syntax-check is a nice feature of the current setup which
would be good to preserve.
While we could set up a something that tests every merge request using
 merge requests looks like overkill here.

Cheers,
 -- Guido

Re: [PATCH 00/12] Plannings for secure-testing repository migration to git

[Guido Günther]

Hi,
On Wed, Dec 27, 2017 at 11:32:53PM +0100, Salvatore Bonaccorso wrote:
[..snip..]
> have that already :). But I'm unahappy with my current versions, thus
> I have not sent those yet. Basically: I'm struggling with a proper
> replacement of such constructs:
> 
> cd /srv/security-tracker.debian.org/website/secure-testing/data && svn update 
> && ../bin/update && svn commit -qm "automatic update" CVE && cd CVE && 
> ../../bin/bts-update list

   cd /srv/security-tracker.debian.org/website/security-tracker/data && git 
pull origin master && ../bin/update && git commit -qm "automatic update" CVE && 
git push origin master && cd CVE && ../../bin/bts-update list

git commit might fail if there are no changes so we need to either guard
for that or use --allow-empty. git push might fail as well if there were
pushes in between. Since there are several commands that

pull, modify, commit push

should we move this to a script that

pulls, invokes the modification command, commits when there are
changes, pushes. If that fails merges again && pushes again.

That won't save you from merge conflicts but that's the same with SVN a
assume?
Cheers,
 -- Guido

> > Should we create WiP/git-migration branches in the secure-testing and
> > security-tracker-service repos so others can send in pull requests?
> > 
> > We could rebase the branch in the secure-testing repo from time to time
> > so catch up with the changes in the svn repo.
> 
> Give me a bit of time to think about it. My intention was to not
> clutter now what we have on salsa as mirror and constantly push
> updates there. And once we go live, add the Debian group as allowed to
> push, and block the original svn repository.  I would like to avoid
> that people already start pushing to the git repository directly.
> 
> Or do you mean to still create the branches on the git repo on salsa
> (accessible readonly, so that people can contribute based on that,
> rather based on patch for pull requests? As said, I would like to not
> operate directly on the git repo until the switch  to not cause
> problems. Our live instance is currently still the svn repo).
> 
> I would expect we solve the open problems soonish enough so we can do
> the switch. That is:
> 
> 1/ desire of commit mailinglist
> 2/ how to do the updates with the sectracker role account specific to
> some files (e.g. automatic updates, data/CVE/list update, dsa
> canditate finding).
> 
> Thanks a lot for your contribution Guido!
> 
> Salvatore
> 

Re: [PATCH 00/12] Plannings for secure-testing repository migration to git

[Guido Günther]

Hi Salvatore,
On Wed, Dec 27, 2017 at 11:32:53PM +0100, Salvatore Bonaccorso wrote:
> Hi Guido
> 
> Much appreciated you looked at the patchset!
> 
> On Wed, Dec 27, 2017 at 09:14:52PM +0100, Guido Günther wrote:
> > Hi,
> > On Wed, Dec 27, 2017 at 08:42:49PM +0100, Salvatore Bonaccorso wrote:
> > > This *preliminary* patchset for review is for the changes needed after
> > > we would switch from alioth to salsa for the secure-testing repository.
> > > While needing to do the svn to git conversion the opportunity was taken
> > > to rename the repository to security-tracker as the naming
> > > secure-testing was for historical reason when the testing security
> > > support arised.
> > 
> > This looks good! I applied your patches to my checkout I found some more
> > svn references in
> > 
> > * tracker 'latest_revision' method
> 
> Jupp that was what I mean that I need here some insights from a LTS
> member and how the script functions. 

Here's a patch also available at

   
https://salsa.debian.org/agx/security-tracker/commit/805a2b5fe27f9812dc679c231ab4dd1a1c9613c1

>From 81025922f7037c4e3b86ac1aff9cd15432f42409 Mon Sep 17 00:00:00 2001
Message-Id: 
<81025922f7037c4e3b86ac1aff9cd15432f42409.1514457764.git@sigxcpu.org>
From: =?UTF-8?q?Guido=20G=C3=BCnther?= 
Date: Thu, 28 Dec 2017 11:36:55 +0100
Subject: [PATCH] TrackerData: use git in instead of svn

Use git-ls-remote instead of svn-info to determine the head revision.
---
 bin/tracker_data.py | 15 ---
 1 file changed, 8 insertions(+), 7 deletions(-)

diff --git a/bin/tracker_data.py b/bin/tracker_data.py
index 3bbac16603..3aaefbe707 100644
--- a/bin/tracker_data.py
+++ b/bin/tracker_data.py
@@ -45,14 +45,15 @@ def normalize_release(release):
 
 class TrackerData(object):
 DATA_URL = "https://security-tracker.debian.org/tracker/data/json;
+GIT_URL = 
"https://salsa.debian.org/security-tracker-team/security-tracker-service.git;
 CACHED_DATA_PATH = "~/.cache/debian_security_tracker.json"
 CACHED_REVISION_PATH = "~/.cache/debian_security_tracker.rev"
 GET_REVISION_COMMAND = \
-"LC_ALL=C svn info svn://anonscm.debian.org/secure-testing|"\
-"awk '/^Revision:/ { print $2 }'"
+"LC_ALL=C git ls-remote %s | awk '/HEAD$/ { print $1 }'" % GIT_URL
 DATA_DIR = os.path.join(os.path.dirname(os.path.dirname(__file__)), 'data')
 
 def __init__(self, update_cache=True):
+self._latest_revision = None
 self.cached_data_path = os.path.expanduser(self.CACHED_DATA_PATH)
 self.cached_revision_path = os.path.expanduser(
 self.CACHED_REVISION_PATH)
@@ -62,14 +63,14 @@ class TrackerData(object):
 
 @property
 def latest_revision(self):
-"""Return the current revision of the SVN repository"""
+"""Return the current revision of the Git repository"""
 # Return cached value if available
-if hasattr(self, '_latest_revision'):
+if self._latest_revision is not None:
 return self._latest_revision
-# Otherwise call out to svn to get the latest revision
+# Otherwise call out to git to get the latest revision
 output = subprocess.check_output(self.GET_REVISION_COMMAND,
  shell=True)
-self._latest_revision = int(output)
+self._latest_revision = output.strip()
 return self._latest_revision
 
 def _cache_must_be_updated(self):
@@ -78,7 +79,7 @@ class TrackerData(object):
 self.cached_revision_path):
 with open(self.cached_revision_path, 'r') as f:
 try:
-revision = int(f.readline())
+revision = f.read()
 except ValueError:
 revision = None
 if revision == self.latest_revision:
-- 
2.15.1

Re: [PATCH 00/12] Plannings for secure-testing repository migration to git

[Salvatore Bonaccorso]

Hi

On Thu, Dec 28, 2017 at 09:23:11AM +0100, Moritz Mühlenhoff wrote:
> On Thu, Dec 28, 2017 at 09:13:09AM +0100, Salvatore Bonaccorso wrote:
> > Hi
> > 
> > On Wed, Dec 27, 2017 at 11:32:53PM +0100, Salvatore Bonaccorso wrote:
> > > 1/ desire of commit mailinglist
> > 
> > Would there be objection to switch those to
> > debian-security-tracker@l.d.o? A dedicated list IHMO would be better
> > to distinguish between the commit review list and the general
> > discussion/bug list.
> 
> I think that's fine. Or we simply don't have a separate commit mailing
> list until Dominic's replacement server goes live and use the Gitlab-
> internal notifications for now.

I for now have configured it to continue sending commit to
secure-testint-commits@l.a.d.o . That is not ideal due to the naming,
but we have this way a little bit more time to sort out that part. I
did a test (the commit email will be replaced if the commiter is in
the same domain as salsa.d.o, that is carnil@salsa.d.o -> carnil@d.o
so we can directly reply to a commit in case of clarification/review).

I have though now made it again 'inactive' until we switch.

There is some news regarding the alioth lists:

https://wiki.debian.org/Alioth/MailingListContinuation

so technically (if I undesrstand correctly) that will be as well a
unofficial system maintained for a longer migration period.

Regards,
Salvatore

Re: [PATCH 00/12] Plannings for secure-testing repository migration to git

[Moritz Mühlenhoff]

On Thu, Dec 28, 2017 at 09:13:09AM +0100, Salvatore Bonaccorso wrote:
> Hi
> 
> On Wed, Dec 27, 2017 at 11:32:53PM +0100, Salvatore Bonaccorso wrote:
> > 1/ desire of commit mailinglist
> 
> Would there be objection to switch those to
> debian-security-tracker@l.d.o? A dedicated list IHMO would be better
> to distinguish between the commit review list and the general
> discussion/bug list.

I think that's fine. Or we simply don't have a separate commit mailing
list until Dominic's replacement server goes live and use the Gitlab-
internal notifications for now.

Cheers,
Moritz

Re: [PATCH 00/12] Plannings for secure-testing repository migration to git

[Salvatore Bonaccorso]

Hi

On Wed, Dec 27, 2017 at 11:32:53PM +0100, Salvatore Bonaccorso wrote:
> 1/ desire of commit mailinglist

Would there be objection to switch those to
debian-security-tracker@l.d.o? A dedicated list IHMO would be better
to distinguish between the commit review list and the general
discussion/bug list.

> 2/ how to do the updates with the sectracker role account specific to
> some files (e.g. automatic updates, data/CVE/list update, dsa
> canditate finding).

Btw, 3/ is there a way to force a pre-commit/pre-push action with git
repositories (so client side hooks forced on every cloned repository)
which would run 'make check-syntax' before we push something? For the
subversion repository this was implemented with a pre-commit svn hook,
which called for changes with bin/check-syntax, or
bin/check-new-issues -c (for the data/embedded-code-copies file).

I guess the short answer is 'no'.

I got the reply for Salsa, for the custom_hooks referring to
https://docs.gitlab.com/ce/administration/custom_hooks.html but as far
I understand those are server-side running hooks.

Regards,
Salvatore

Re: [PATCH 00/12] Plannings for secure-testing repository migration to git

[Salvatore Bonaccorso]

Hi Guido

Much appreciated you looked at the patchset!

On Wed, Dec 27, 2017 at 09:14:52PM +0100, Guido Günther wrote:
> Hi,
> On Wed, Dec 27, 2017 at 08:42:49PM +0100, Salvatore Bonaccorso wrote:
> > This *preliminary* patchset for review is for the changes needed after
> > we would switch from alioth to salsa for the secure-testing repository.
> > While needing to do the svn to git conversion the opportunity was taken
> > to rename the repository to security-tracker as the naming
> > secure-testing was for historical reason when the testing security
> > support arised.
> 
> This looks good! I applied your patches to my checkout I found some more
> svn references in
> 
> * tracker 'latest_revision' method

Jupp that was what I mean that I need here some insights from a LTS
member and how the script functions. 

> * website/uploading.html

That was intentional, because the uploading.html is mostly for
historic reference there when the secure-testing project arised. It is
about DTSA. We can change that but I though we could leave that site
as it is.

> * doc/soriano.txt (which needs to match what we do in
>   security-tracker-service I guess)

Yes that is pending, I waited for that part before
security-tracker-service are sorted out, so that the documentation
then can mirror how it's now then actually set up on soriano.

> * the security-tracker-service repo 

have that already :). But I'm unahappy with my current versions, thus
I have not sent those yet. Basically: I'm struggling with a proper
replacement of such constructs:

cd /srv/security-tracker.debian.org/website/secure-testing/data && svn update 
&& ../bin/update && svn commit -qm "automatic update" CVE && cd CVE && 
../../bin/bts-update list

> Should we create WiP/git-migration branches in the secure-testing and
> security-tracker-service repos so others can send in pull requests?
> 
> We could rebase the branch in the secure-testing repo from time to time
> so catch up with the changes in the svn repo.

Give me a bit of time to think about it. My intention was to not
clutter now what we have on salsa as mirror and constantly push
updates there. And once we go live, add the Debian group as allowed to
push, and block the original svn repository.  I would like to avoid
that people already start pushing to the git repository directly.

Or do you mean to still create the branches on the git repo on salsa
(accessible readonly, so that people can contribute based on that,
rather based on patch for pull requests? As said, I would like to not
operate directly on the git repo until the switch  to not cause
problems. Our live instance is currently still the svn repo).

I would expect we solve the open problems soonish enough so we can do
the switch. That is:

1/ desire of commit mailinglist
2/ how to do the updates with the sectracker role account specific to
some files (e.g. automatic updates, data/CVE/list update, dsa
canditate finding).

Thanks a lot for your contribution Guido!

Salvatore

Re: [PATCH 00/12] Plannings for secure-testing repository migration to git

[Guido Günther]

Hi,
On Wed, Dec 27, 2017 at 08:42:49PM +0100, Salvatore Bonaccorso wrote:
> This *preliminary* patchset for review is for the changes needed after
> we would switch from alioth to salsa for the secure-testing repository.
> While needing to do the svn to git conversion the opportunity was taken
> to rename the repository to security-tracker as the naming
> secure-testing was for historical reason when the testing security
> support arised.

This looks good! I applied your patches to my checkout I found some more
svn references in

* tracker 'latest_revision' method
* website/uploading.html
* doc/soriano.txt (which needs to match what we do in
  security-tracker-service I guess)

* the security-tracker-service repo 

Should we create WiP/git-migration branches in the secure-testing and
security-tracker-service repos so others can send in pull requests?

We could rebase the branch in the secure-testing repo from time to time
so catch up with the changes in the svn repo.

Thanks a lot for working on this!
 -- Guido

[PATCH 00/12] Plannings for secure-testing repository migration to git

[Salvatore Bonaccorso]

This *preliminary* patchset for review is for the changes needed after
we would switch from alioth to salsa for the secure-testing repository.
While needing to do the svn to git conversion the opportunity was taken
to rename the repository to security-tracker as the naming
secure-testing was for historical reason when the testing security
support arised.

Important note: Patch to update_lists triggering is defintively to be
reworked, in conjuction with the automatic updates performed on the
security-tracker host itself.

Furthermore there is a yet untouched script (bin/tracker_data.py) which
might need a rewrite to work properly with a git based security-tracker
repository.

Proposed patchset following.

Salvatore Bonaccorso (12):
  review-update-needed: Adjust error message to only mention git
checkout
  check-new-issues: Do not mention svn command for review process
  update-lists: Update lists in data in git checkout
  In DLA mode: if git checkout found ask to push changes
  Cleanup documentation after move from Alioth to Salsa
  tracker service: Refer to the salsa project on the help out section
  tracker service: Update references in footer to salsa
  Overview page: Reference to the git repository on the salsa gitlab
instance
  lts-bts: Reference dla-needed.txt from security-tracker git repository
  lts-update-planned template: Reference dla-needed from git repository
  report-vuln: Adjust example usage path to report-vuln scirpt
  Reference files from security-tracker git repository on security-team
website

 Makefile   |  2 +-
 bin/check-new-issues   |  2 +-
 bin/gen-DSA| 14 ---
 bin/lts-bts|  2 +-
 bin/report-vuln|  2 +-
 bin/review-update-needed   |  2 +-
 bin/tracker_service.py | 12 +++---
 doc/security-team.d.o/index|  6 +--
 doc/security-team.d.o/security_tracker | 68 +++---
 templates/lts-update-planned.txt   |  2 +-
 10 files changed, 44 insertions(+), 68 deletions(-)

-- 
2.15.1