Bug#700309: wotsap: Wotsap can't read latest .wot files from http://wot.christoph-egger.org/

[Giovanni Mascellani]

Hi.

Il 17/07/2016 22:52, Christoph Egger ha scritto:
>   As wotsap can't cleanly handle the first case right now (it might when
> making sure the new fingerprints file is added *at*the*end*) I agree
> making an incompatible change is masically unavoidable. Plkease, when
> implementing the client consider unsing python-arpy (I started getting
> arpy in a shape where we can use it for that but never did anything
> more) so the client no longer depends on the ordering of files in the
> ar-archive!

Uhm, are you sure that would work? Skimming the source code, it appears
that it must seek through the archive in order to decode chunks in an
arbitrary order (this is basically unavoidable anyway). Since xz does
not support seek, this means that it must decompress the whole archive
more than once, which does not seems to be a great idea.

The nice thing of having the chunks in this order is that you can read
them sequentially and build the internal structures directly from them,
without caching or seeking. If it is not too difficult for you to keep
the order, I really would do it.

>   I think the new format should have the folloing changes
> 
>  + Get full fingerprints into the keys file
>  + change compression from bzip to xz
>  + maybe make keys a textfile (newline terminated hex-encoded strings)
>After encryption there should be hardly any difference in size

I did these. Please see


https://github.com/giomasce/wotsap/commit/cb2f81fd8d5c73365532234cf6351fd7504d5b49

and let me know what you think.

Since we are moving from binary to text, we could do that all the way
and change also the "signatures" chunk. For example, something like:
each line describes the signatures of a key; it contains a decimal
number (the number of signatures) and then another number for each
signature, separated by spaces. Each of these numbers is followed
(without spaces) by a letter 'P' or 'p' (standing for the primary key
flag) and another digit (the cert level). The old "bit 3" is not
encoded, since was never used.

Once we agree on the format, I can start coding for wotsap (well, as
soon as I find free time).

Thanks, Giovanni.
-- 
Giovanni Mascellani 
PhD Student - Scuola Normale Superiore, Pisa, Italy

http://poisson.phc.unipi.it/~mascellani



signature.asc
Description: OpenPGP digital signature

Bug#700309: wotsap: Wotsap can't read latest .wot files from http://wot.christoph-egger.org/

[Christoph Egger]

Hi Giovanni!

Giovanni Mascellani  writes:
> Since the original wotsap is not maintained any more, but there are
> quite a few patches that could be applied and improvements that could be
> made, I decided to fork it and begin again its development here:
>
>   https://github.com/giomasce/wotsap
>
> Actually, I will not have a lot of time for it, but hopefully enough to
> fix the current most outstanding issues. I will also update the Debian
> package after this repository.


  That's great news, thank you! also means I don't need to care for the
client side and can concentrate on getting the extraction part right!

> I see two main alternatives:
>
>  * The file "keys" is extended to 8 or 20 bytes record. This is the
> sanest thing, but it is also backward-incompatible. The WOTVERSION
> number should be bumped to 0.3, old clients could not use new files and
> new clients could not use old files (unless they retain compatibility,
> of course).
>
>  * Old files are kept as they are, but another file (e.g., "longkeys")
> is added, which contains 4 or 16 bytes record: each record is the part
> that must be pasted before the corresponding "keys" record in order to
> have the long ID. This is backward-compatible, but really ugly in itself.
>
> I tend to prefer the first alternative, especially if you can keep
> providing 0.2 .wots for some time after the change.

  As wotsap can't cleanly handle the first case right now (it might when
making sure the new fingerprints file is added *at*the*end*) I agree
making an incompatible change is masically unavoidable. Plkease, when
implementing the client consider unsing python-arpy (I started getting
arpy in a shape where we can use it for that but never did anything
more) so the client no longer depends on the ordering of files in the
ar-archive!

  I think the new format should have the folloing changes

 + Get full fingerprints into the keys file
 + change compression from bzip to xz
 + maybe make keys a textfile (newline terminated hex-encoded strings)
   After encryption there should be hardly any difference in size

Christoph

-- 
9FED 5C6C E206 B70A 5857  70CA 9655 22B9 D49A E731
Debian Developer | Lisp Hacker | CaCert Assurer

Bug#700309: wotsap: Wotsap can't read latest .wot files from http://wot.christoph-egger.org/

[Giovanni Mascellani]

Dear Christoph and Elena.

Since the original wotsap is not maintained any more, but there are
quite a few patches that could be applied and improvements that could be
made, I decided to fork it and begin again its development here:

  https://github.com/giomasce/wotsap

Actually, I will not have a lot of time for it, but hopefully enough to
fix the current most outstanding issues. I will also update the Debian
package after this repository.

Il 11/02/2013 20:36, Christoph Egger ha scritto:
> Hi!
> 
>   The next .wot files (generated every weekend) should now work again
> with wotsap, I've commented out the relevant part of the generation
> script causing the problems.
> 
>   I'd still really appreciate if wotsap will be changed to handle .wot
> files as the bzip2 compressed ar archives they are. Would also be cool
> to figure out if we can add an extra component to the archive in a way
> that does not break wotsap (as the problem with 4byte short IDs only is
> still unresolved) or if we need to keep both formats in parallel (and as
> a consequence have more freedom in how the "new" wots will look like)

In theory it should, but there may be bug. I agree that wotsap should be
updated to support longer IDs (and possibly also the whole fingerprint).
 If you want we can discuss the file format changes to support this, so
that you can apply them on the .wot generator and I on wotsap.

I see two main alternatives:

 * The file "keys" is extended to 8 or 20 bytes record. This is the
sanest thing, but it is also backward-incompatible. The WOTVERSION
number should be bumped to 0.3, old clients could not use new files and
new clients could not use old files (unless they retain compatibility,
of course).

 * Old files are kept as they are, but another file (e.g., "longkeys")
is added, which contains 4 or 16 bytes record: each record is the part
that must be pasted before the corresponding "keys" record in order to
have the long ID. This is backward-compatible, but really ugly in itself.

I tend to prefer the first alternative, especially if you can keep
providing 0.2 .wots for some time after the change.

Let me know what you think.

Have a nice day, Giovanni.
-- 
Giovanni Mascellani 
PhD Student - Scuola Normale Superiore, Pisa, Italy

http://poisson.phc.unipi.it/~mascellani



signature.asc
Description: OpenPGP digital signature

Bug#700309: wotsap: Wotsap can't read latest .wot files from http://wot.christoph-egger.org/

[Giovanni Mascellani]

Ciao Christoph.

Il 11/02/2013 20:36, Christoph Egger ha scritto:
 Hi!
 
   The next .wot files (generated every weekend) should now work again
 with wotsap, I've commented out the relevant part of the generation
 script causing the problems.
 
   I'd still really appreciate if wotsap will be changed to handle .wot
 files as the bzip2 compressed ar archives they are. Would also be cool
 to figure out if we can add an extra component to the archive in a way
 that does not break wotsap (as the problem with 4byte short IDs only is
 still unresolved) or if we need to keep both formats in parallel (and as
 a consequence have more freedom in how the new wots will look like)

At the moment wotsap is not maintained anymore upstream:

http://www.lysator.liu.se/~jc/wotsap/news.html

So I guess that nobody is going to complain if you want to take it over.
I would like too, but at the moment I can spend no time with it. If you
decide to take over wotsap, I can also pass you the Debian package
maintainership (and maybe remain as comaintainer).

BTW, thank you very much for starting your WOT generation service.

Giovanni.
-- 
Giovanni Mascellani mascell...@poisson.phc.unipi.it
Pisa, Italy

Web: http://poisson.phc.unipi.it/~mascellani
Jabber: g.mascell...@jabber.org / giova...@elabor.homelinux.org



signature.asc
Description: OpenPGP digital signature

Bug#700309: wotsap: Wotsap can't read latest .wot files from http://wot.christoph-egger.org/

[Elena Grandi]

Package: wotsap
Version: 0.7-2
Severity: normal

When trying to use a recent (  2013-01-30 ) .wot file from 
http://wot.christoph-egger.org/ I get the following error:

$ wotsap -w 2013-02-09.wot 01809E2A
Strange. Found longkeys file.
Traceback (most recent call last):
  File /usr/bin/wotsap, line 1903, in module
wotsapmain(sys.argv)
  File /usr/bin/wotsap, line 1751, in wotsapmain
wot = Wot(wotfile)
  File /usr/bin/wotsap, line 1471, in __init__
self.loadfile(dumpfile)
  File /usr/bin/wotsap, line 1484, in loadfile
self.filename, self.fileversion, self.debug) = loadfile(dumpfile)
  File /usr/bin/wotsap, line 462, in loadfile
f.seek(size + (size1), 1)
IOError: [Errno 29] Illegal seek

bunzip2: I/O or other error, bailing out.  Possible reason follows.
bunzip2: Broken pipe
Input file = (stdin), output file = (stdout)

The .wot file is a valid bzip2 file (tested with bunzip2 2013-02-09.wot) 
and it seems that the pathfinder at http://pgp.cs.uu.nl/ is able to 
read the new files, which is why I suspect (but I'm not sure) that the problem 
could be in wotsap itself.

-- System Information:
Debian Release: 7.0
  APT prefers testing
  APT policy: (500, 'testing')
Architecture: amd64 (x86_64)

Kernel: Linux 3.2.0-4-amd64 (SMP w/1 CPU core)
Locale: LANG=en_GB.UTF-8, LC_CTYPE=en_GB.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash

Versions of packages wotsap depends on:
ii  fonts-freefont-ttf [ttf-freefont]  20120503-1
ii  python 2.7.3~rc2-1
ii  python-imaging 1.1.7-4
ii  ttf-dejavu 2.33-3

wotsap recommends no packages.

Versions of packages wotsap suggests:
ii  gnupg  1.4.12-7
ii  wget   1.13.4-3

-- no debconf information


-- 
To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org

Bug#700309: wotsap: Wotsap can't read latest .wot files from http://wot.christoph-egger.org/

[Christoph Egger]

Hi!

Elena Grandi elena.valha...@gmail.com writes:
 When trying to use a recent (  2013-01-30 ) .wot file from 
 http://wot.christoph-egger.org/ I get the following error:

 $ wotsap -w 2013-02-09.wot 01809E2A
 Strange. Found longkeys file.
 Traceback (most recent call last):
   File /usr/bin/wotsap, line 1903, in module
 wotsapmain(sys.argv)
   File /usr/bin/wotsap, line 1751, in wotsapmain
 wot = Wot(wotfile)
   File /usr/bin/wotsap, line 1471, in __init__
 self.loadfile(dumpfile)
   File /usr/bin/wotsap, line 1484, in loadfile
 self.filename, self.fileversion, self.debug) = loadfile(dumpfile)
   File /usr/bin/wotsap, line 462, in loadfile
 f.seek(size + (size1), 1)
 IOError: [Errno 29] Illegal seek

 bunzip2: I/O or other error, bailing out.  Possible reason follows.
 bunzip2: Broken pipe
 Input file = (stdin), output file = (stdout)

 The .wot file is a valid bzip2 file (tested with bunzip2 2013-02-09.wot) 
 and it seems that the pathfinder at http://pgp.cs.uu.nl/ is able to 
 read the new files, which is why I suspect (but I'm not sure) that the 
 problem 
 could be in wotsap itself.

Newer wotsap files were created with an additional file content (I'm
trying to get something longer than 4byte short IDs in). Must have
missed to check with wotsap for compatibility there. I'll probably
revert the change for now (doesn't work as intended right now anyway)
but would be cool if wotsap could deal wiuth an additional component in
the ar archive!

Christoph


-- 
To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org

Bug#700309: wotsap: Wotsap can't read latest .wot files from http://wot.christoph-egger.org/

[Christoph Egger]

Hi!

  The next .wot files (generated every weekend) should now work again
with wotsap, I've commented out the relevant part of the generation
script causing the problems.

  I'd still really appreciate if wotsap will be changed to handle .wot
files as the bzip2 compressed ar archives they are. Would also be cool
to figure out if we can add an extra component to the archive in a way
that does not break wotsap (as the problem with 4byte short IDs only is
still unresolved) or if we need to keep both formats in parallel (and as
a consequence have more freedom in how the new wots will look like)

Regards

Christoph


-- 
To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org