Hi,
On Fri, 2 Oct 2020 22:04:32 +0200 Sylvain Beucler <[email protected]> wrote:
It seems a bit difficult to make a big switch, probably because it's not
easy to know and test all the various involved scripts.
Considering a more progressive approach, is there something preventing
us from switching to the rewritten repository and split/merging the
file
As discussed at #debian-security in July, I refined that approach.
There's an on-going effort to tackle the issue through multiple means,
including a different splitting approach, and/or regular repacks
(https://www.freexian.com/blog/debian-contributions-08-2026/), so my
refinement isn't likely to be used as-is.
For posterity, in case this helps similar issues, I'm attaching the
scripts, as well as some design notes.
Cheers!
Sylvain
# Copyright (C) 2026 Sylvain Beucler
import glob, itertools, os, sys
def main():
# Group splits by year
split_files = sorted(glob.glob('data/CVE/*.list'))
years = { year : [open(f, 'rb') for f in volumes]
for year,volumes in itertools.groupby(split_files,
key=lambda x: x.lstrip('data/CVE/main-')[0:4]) }
# 1-line read-ahead, as we can't push back readline() when reaching a new CVE entry
years_readahead = { year : years[year][-1].readline() for year in years }
# Read next year entry, from newest to oldest
def copy_year_cve(year, out):
volumes = years[year]
if years_readahead[year]:
out.write(years_readahead[year])
years_readahead[year] = None
else:
raise EOFError(f"No more content from '{year}'")
while len(volumes) > 0:
line = volumes[-1].readline()
if line.startswith(b'CVE-'):
# stop here and store for next call
years_readahead[year] = line
break
elif line == b'':
# next file
volumes.pop()
continue
else:
out.write(line)
# Merge file
with (open('data/CVE/order', 'r') as order,
open('data/CVE/list', 'wb') as out):
for cur_range in order:
year,nb = cur_range.split(':')
for i in range(int(nb)):
copy_year_cve(year, out)
# Safety check
for year in years:
if len(years[year]) > 0:
raise Exception(f"Lost content from '{year}'")
# Put main code inside a function to avoid slow globals
main()
# Copyright (C) 2026 Sylvain Beucler
import os, sys
# 'order' file to rebuild data/CVE/list in the same order
class OrderFile:
def __init__(self):
self.lines = []
def add_range(self, year, length):
line = b"%s:%d\n" % (year, length)
self.lines.append(line)
def write(self):
with open('data/CVE/order', 'wb') as f:
f.write(b''.join(self.lines))
# Year-based, then volume-based split
class YearSplit:
#SIZE_LIMIT = 5*1024*1024
#MAX_ENTRIES = 30000 # <5MB, 1 file/year until 2020, growable/stable first slot
MAX_ENTRIES = 10000 # generate a different git-filter-branch rewrite for analysis
def __init__(self, year):
self.entries = []
self.year = year
def write(self):
"""
write a yearly split, multi-volume, never split a CVE entry
"""
# MAX_ENTRIES per volume
range_starts = range(0, len(self.entries), self.MAX_ENTRIES)
ranges = [(i,i+self.MAX_ENTRIES) for i in range_starts]
# Oldest entries in first volumes for Git blob and filename stability
entries = list(reversed(self.entries))
file_count = 0
for start,end in ranges:
file_count += 1
# Use a "main" prefix to help Git, which packs better with unique filenames
fd = open(b"data/CVE/main-%s-%d.list" % (self.year, file_count), 'wb')
volume = entries[start:end]
# Prepend newer entries on top to mimic data/CVE/list order
# (optional, appending new content may make sense too)
volume = reversed(volume)
for entry in volume:
fd.write(entry)
fd.close()
# Split, in memory
# (two-pass, as we read newest entries first but write oldest entries first,
# for content stability; also group random writes to yearly files)
def main():
years = {}
with open('data/CVE/list', 'rb') as f: # 'b' for robustness against encoding errors
cur_year = b'0000' # handle invalid first entry for robustness and data integrity
years[cur_year] = YearSplit(cur_year)
cur_range_len = 0
cur_entry = b''
index = OrderFile()
for line in f:
if line.startswith(b'CVE-'):
# Store last entry (inlined code for efficiency)
if len(cur_entry) > 0: # handle invalid first line
years[cur_year].entries.append(cur_entry)
cur_range_len += 1
new_year = line[4:8]
if not new_year.isdigit():
new_year = b'0000' # store entries with invalid years there
if new_year != cur_year:
if cur_range_len > 0:
index.add_range(cur_year, cur_range_len)
cur_range_len = 0
cur_year = new_year
if not cur_year in years:
years[cur_year] = YearSplit(cur_year)
cur_entry = line
else:
# Everything else is the CVE content (this isn't a syntax checker).
# If there's an syntax error, store too under the previous entry.
# (No data loss, especially during the Git full rewrite.)
cur_entry += line
# End Of File, store last entry
if len(cur_entry) > 0:
years[cur_year].entries.append(cur_entry)
cur_range_len += 1
index.add_range(cur_year, cur_range_len)
# Write to disk
for year in years:
years[year].write()
index.write()
# Put main code inside a function to avoid slow globals
main()
Context: man git: "git - the stupid content tracker"
git is a content-indexed tracker. At heart it blindly stores each and every
file revision in full, gzip'd. Its limitations from day 1 are: not handling
large files well (hence git-lfs, etc.), and not handling renames natively.
In a second step, git can search for diffs ("deltas") and generate packs
(git-gc, gitlab "maintenance"), which is costly.
We're storing an ever-growing data/CVE/list in Git, so this triggered the
follow issues:
- git-blame has long stopped being usable.
- Repacking is now taking half a day, even optimized, because diff-ing 50MB
remains costly no matter what.
- Reaching 50MB triggered all kinds of safe limits in Salsa, resulting in
inefficient packs and high network&disk usage.
- We need a work-around for diffs in merge requests.
- We no longer can host the security-tracker (or forks/mirrors) at other forges
(e.g. GitHub).
- New clones are timing out.
Guiding principles:
- Remain compatible with secteam's workflow
I proposed splitting the file before committing (pre-commit hook), and merging
it when working on it (post-merge). We keep the same local data/CVE/list, only
it's stored more efficiently in Git.
We also keep the file order to recreate data/CVE/list exactly as-is (new CVEs
on top by order of addition, no sort), if secteam wants it.
All tools should keep working.
(Except maybe for those that specifically 'git add data/CVE/list', and even
then the fix is trivial.)
- Split data/CVE/list in smaller parts
Split it in smaller files, which Git/Salsa/etc. handles better.
Git operations will be fast again (diff, pull, merge, blame, rebase, computing
deltas, etc.), without the current work-arounds.
This allows for other optimization:
* the lengthy 'make check-syntax' now could only re-check the modified
sub-parts,
* the web tracker import could perform a multi-step, cached yearly import, then
only reimport the last modified years,
* etc.
- Minimize Git objects (blobs, trees)
In https://salsa.debian.org/lts-team/cvehist I split 1 file per-CVE, but I was
focusing on optimizing git-log / git-blame specifically. This creates large
tree objects (Git file listing).
$ git ls-tree --object-only HEAD^{tree} 20* | xargs -n1 git cat-file -s
=> 300k for 2026/, 200k gzip'd; new object for *every* content change
It also creates many files (400k CVEs). cvehist is optimized and drops from
400k CVEs to 70k CVEs with interesting history, but that's not applicable here.
Splitting by year is a good bargain, but even then, each year is growing
exponentially. So I suggest splitting by year, and then by year volumes
(2026-1.list, 2026-2.list), with a limit per volume (see below).
Prior art: Daniel Lange ran a naive yearly split up to 2022:
https://salsa.debian.org/dlange/debian_security_security-tracker_split_files_v2/
It can be repacked to 120MB in 30mn.
(compared to, current Git repacked to 1-2GB in 10-20h)
The CVE order/index should also be small. A trivial CVE list takes 4MB. I
propose listing a series of year ranges (e.g. 112 "2026" entries, then 10
"2025" entries, then 7 "2026" entries, etc.). We're down to 60k.
- Minimize new blobs, i.e. keep file content stable
No new blob means no new content, diff/deltas to compute, data to download,
etc. This is the ideal Git situation.
Old CVE years are rarely touched, so once e.g. 2020-1.list is written, it
should stay unmodified (no new blob), even with new 2020 CVEs. So -1.list gets
the oldest CVEs (by addition), -2.list get newer ones, etc.
2026-2.list should also stay stable when 2026-1.list entries are triaged and
expanded. Splitting with a hard file size limit means entries will be moved
from -1 to -2 as -1 grows, and so on, modifying all year volumes. Instead, I
suggest capping the number of CVE entries per volume (30000 is a good bargain,
<5MB/volume, single-volume until 2020; 10000 packs even better).
- Cons
This is a full Git rewrite. All local clones need to be re-cloned. All
forks/rewrites need to be rewritten (ELTS, cvehist).
Also I didn't fully test how to work in a rewritten repo. 'git commit' will
work, but 'git diff' may need another hook/wrapper to split before diffing, for
instance.
A sample rewrite is available at:
https://salsa.debian.org/beuc/sc-split-v2-10k
