Re: Securely retrieving dscs from snapshot.debian.org

[peter green]

On 27/12/17 23:42, Paul Wise wrote:

On Thu, Dec 28, 2017 at 5:41 AM, peter green wrote:


Unfortunately there doesn't seem to be a good way to securely retrive a dsc
from snapshot.debian.org given a package name and version number.

At this time there isn't any good way to do that securely, until
#763419 gets implemented.

That may help a little, though it raises questions of it's own, like

* what keys would be used to sign these re-signed release files? You wouldn't 
want to use a regular Debian archive key because you wouldn't want people to be 
able to use snapshots to attack Debian users.
* How secure would the re-signing infrastructure be?

And it would only solve one aspect of the problem, the fact that verifying 
Release signatures may involve old keys. It wouldn't solve the issue of how to 
find that damn Release/Sources pair in the first place.

I have attatched my attempt at a tool for downloading source packages securely 
from snapshot.debian.org. It seems to work, comments/improvements welcome.
#!/usr/bin/python3
import sys
import urllib.request
import urllib.error
import json
from bs4 import BeautifulSoup
import re
import deb822
import io
import gzip
import bz2
import lzma
import os
import subprocess
import hashlib
import argparse

parser = argparse.ArgumentParser(description="retrieve a source package from 
snapshot.debian.org with gpg verification\n"+
"the source package will be stored in the current directory\n"+
"in the process of verification files source_version_Release and 
source_version_Release.gpg will be created in the current directory, these will 
be "+
"removed unless --keepevidence is specified"
)
parser.add_argument("source", help="source package name")
parser.add_argument("version", help="source package version")
parser.add_argument("--keepevidence", help="keep Release.gpg, Release and 
Sources files as evidence of package integrity", action="store_true")
parser.add_argument("--1024", help="allow 1024 bit keys, this is needed for old 
packages but may leave you vulnerable to well-funded 
attackers",action="store_true",dest='allow_1024')
args = parser.parse_args()

package=args.source
version=args.version
scriptdir=os.path.dirname(os.path.realpath(__file__))

colonpos = version.find(':')

#regex used for checking  package name
pnallowed = re.compile('[a-z0-9][a-z0-9\-\+\.]+',re.ASCII)

#regex used for checking version number
#this is not a full implementation of Debian version number rules
#just a sanity check for unexcepted characters
vallowed = re.compile('[a-z0-9\-:\+~\.]+',re.ASCII)

#regex used for checking allowed characters in package filenames
#and a few other things.
pfnallowed = re.compile('[a-z0-9\-_:\+~\.]+',re.ASCII)

#regex used for checked allowed characters in timestamp strings
tsallowed = re.compile('[A-Z0-9]+',re.ASCII)

#regex used for matching duplicate or aincient (no Release.gpg) distribution 
names
dupain = 
re.compile('(Debian.*|bo.*|buzz.*|hamm.*|potato|rex|slink.*)/',re.ASCII)

if not pnallowed.fullmatch(package):
print('package name fails to match required format')

if not vallowed.fullmatch(package):
print('version number fails to match required format')

if colonpos >= 0:
versionnoepoch=version[colonpos+1:]
else:
versionnoepoch=version

url='http://snapshot.debian.org/mr/package/'+package+'/'+version+'/srcfiles?fileinfo=1'

with urllib.request.urlopen(url) as response:
jsondata = response.read()

jsondecoded = json.loads(jsondata.decode("utf-8"))

instances = []
for sha1, info in jsondecoded['fileinfo'].items():
for instance in info:
#print(repr(instance))
if instance['name'] == package+'_'+versionnoepoch+'.dsc':
#print(repr(instance))
instances.append(instance)

#unfortunately snapshot.debian.org doesn't seem to provide a mr interface for 
file listings, so we have to screen scrape
#the aim here is to only get true subdirs, not files or symlinks, these seem to 
be indicated by a trailing / in the link
#string. We also need to avoid any links with complex urls, which likely 
represent page chrome.
def snapshotsubdirlist(url):
result = []
with urllib.request.urlopen(url) as response:
pagedata = response.read()
soup = BeautifulSoup(pagedata, "lxml")
p = re.compile('[a-zA-Z0-9\-]+/',re.ASCII)
for item in soup.find_all('a'):
if not (finalentry is None): break
link = item['href']
if (p.fullmatch(link)):
result.append(link)
return result

#ideally we want sha256 but sometimes that doesn't exist
def findmostsecurereleasefiles(deb822):
if 'SHA256' in deb822:
return deb822['SHA256']
if 

extracting upstream source.

[Peter Green]

As part of my dgit based autoforwardporter I want to add a script that will 
de-fuzz patches with fuzz and remove patches that cannot be applied.

To do this I need the "upstream" source tree. However the process of extracting 
the upstream source is quite fiddly, there may be multiple tarballs to deal with, 
tarballs may or may not have a top-level directory that needs to be removed from the 
paths etc.

Both dpkg-source ("commit" and "build")and dgit (quilt fixup) clearly extract 
the upstream source tree as part of their processing, so the code to do it already exists but i'm 
not sure if there is a convenient way to access it.

Any hints?



___
vcs-pkg-discuss mailing list
vcs-pkg-discuss@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/vcs-pkg-discuss

on quilt, dpkg, dgit and symlinks.

[peter green]

I recently discovered some unusual behaviour in a source package I was working 
on.

I was using some scripts I put together myself to generate patch series for a 
debian package.

dgit claimed I was creating a new symlink and that creation of a new symlink could not be 
represented by 3.0 (quilt). So I removed the symlink, dpkg-source called by dgit then 
warned about "deletion" of said symlink. Furthermore when I extracted the 
resulting source package the symlink reappeared.

After some tracking down it seems a patch *can* create a symlink and that both 
dpkg-source and "quilt push" will happily apply such a patch.

quilt pop on the other hand is not so happy. Sometimes it will remove the patch, 
sometimes it will complain that it "won't remove cleanly". I haven't figured 
out the exact trigger.

dgit will import a dsc with such a patch, then fail to build a source package 
from the tree it has just imported complaining about changes that cannot be 
represented by 3.0 (quilt).

not sure what to make of all this.

I have attached a source package demonstrating the issue (you will need to grab 
the orig tarball from the debian archive)




hello_2.10-1+symlink.debian.tar.xz
Description: application/xz
Format: 3.0 (quilt)
Source: hello
Binary: hello
Architecture: any
Version: 2.10-1+symlink
Maintainer: Santiago Vila 
Homepage: http://www.gnu.org/software/hello/
Standards-Version: 3.9.6
Build-Depends: debhelper (>= 9.20120311)
Package-List:
 hello deb devel optional arch=any
Checksums-Sha1:
 f7bebf6f9c62a2295e889f66e05ce9bfaed9ace3 725946 hello_2.10.orig.tar.gz
 692a93f818ec349fbe130bfa3c6f5aff20b01e24 6408 
hello_2.10-1+symlink.debian.tar.xz
Checksums-Sha256:
 31e066137a962676e89f69d1b65382de95a7ef7d914b8cb956f41ea72e0f516b 725946 
hello_2.10.orig.tar.gz
 f8eb5f4161568f7e30874f12bf2d5262ec7cef8ce2eef8353a681a320a81d002 6408 
hello_2.10-1+symlink.debian.tar.xz
Files:
 6cd0ffea3884a4e79330338dcc2987d6 725946 hello_2.10.orig.tar.gz
 3a0426222a13f7f40147fd865088f032 6408 hello_2.10-1+symlink.debian.tar.xz
___
vcs-pkg-discuss mailing list
vcs-pkg-discuss@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/vcs-pkg-discuss

Re: PSA: github file size limit means that some debian package imports can't be pushed to github.

[peter green]

On 09/03/17 16:05, s...@vilain.net wrote:

git-lfs is your friend. And supported natively by GitHub.


Thanks but it seems like a highly undesirable soloution to me.

Firstly it seems to require mangling your git repo. That is going to make it at 
best a PITA to use with things like dgit.

Secondly AIUI git-lfs stores every version of the file seperately. If so then 
one would very quickly blow through the default storage and bandwidth 
limitations for githubs lfs offering.



On March 9, 2017 7:42:20 AM PST, peter green <plugw...@p10link.net> wrote:

I have recently started pushing source for packages where we carry 
modifications for in raspbian to github. The packages are imported into git 
using dgit.

However I have discovered that this is not possible for all packages. In 
particular github rejects files over 100 megabytes and the current libreoffice 
package has a 107 megabyte (upstream) ChangeLog in it.

You might want to bear this in mind when considering packaging workflows 
that may involve github.


--

vcs-pkg-discuss mailing list
vcs-pkg-discuss@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/vcs-pkg-discuss

-- Sent from my Android device with K-9 Mail. Please excuse my brevity.

___
vcs-pkg-discuss mailing list
vcs-pkg-discuss@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/vcs-pkg-discuss


___
vcs-pkg-discuss mailing list
vcs-pkg-discuss@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/vcs-pkg-discuss

PSA: github file size limit means that some debian package imports can't be pushed to github.

[peter green]

I have recently started pushing source for packages where we carry 
modifications for in raspbian to github. The packages are imported into git 
using dgit.

However I have discovered that this is not possible for all packages. In 
particular github rejects files over 100 megabytes and the current libreoffice 
package has a 107 megabyte (upstream) ChangeLog in it.

You might want to bear this in mind when considering packaging workflows that 
may involve github.

___
vcs-pkg-discuss mailing list
vcs-pkg-discuss@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/vcs-pkg-discuss

Re: git based autoforwardporter.

[peter green]

On 19/01/17 19:36, Ian Jackson wrote:


I think such a tool would be very useful in general, if it could be
used for individual users.

I see no reason why it couldn't be used by an individual user.

I have posted the current code at .

https://github.com/plugwash/dscdirtogit (import tool)
https://github.com/plugwash/autoforwardporter (actual autoforwardporter)

Feel free to take a look at it and comment/criticize.

Currently there are a bunch of assumptions in the code that are either raspbian-specific 
or specific to the box I run the code on. I guess I will need to introduce some sort of 
config file so that "distro" names, file paths, urls, committer emails etc can 
be configured.

The code is a mixture of python3 and shell script, anyone have any suggestions 
for the best way to do a config file that can be easilly read from both python3 
and shell script?

___
vcs-pkg-discuss mailing list
vcs-pkg-discuss@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/vcs-pkg-discuss

re: patches-applied historical imports (usd import)

[peter green]

I haven't run dgit's dsc importer on a whole historical archive but
Peter Green of Raspian has been running it and filing bugs.  I haven't
seen such a bug recently so I hope it has been working for all the
packages he's seen.

I haven't really been doing large scale importing, I have just imported 
relatively shallow histories of a relative handful of packages.



___
vcs-pkg-discuss mailing list
vcs-pkg-discuss@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/vcs-pkg-discuss

git based autoforwardporter.

[peter green]

First let me introduce myself, I am Peter Michael Green, cofounder of the 
Raspbian project. I have quite a bit of experiance with Debian

Some time ago I put together what I call the "autoforwardporter". The aim of 
this is to take downstream changes and apply them on top of new debian uploads.

Originally this was diff/patch based but I got into a conversation with Ian 
Jackson at the minidebconf and he convinced me it would be a good idea to use 
dgit.

I can't imagine raspbian is the only project who would find such a tool useful. 
OTOH it would be a fair bit of work to clean up the tool to seperate local 
raspbian assumptions from more generally applicable functionality.

Are there people who would be interested in using and collaborating on such a 
tool?


___
vcs-pkg-discuss mailing list
vcs-pkg-discuss@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/vcs-pkg-discuss