Re: [PATCH v2 0/9] Tools and fixes for parallel parsing

2018-02-25 Thread Daniel Axtens
Stephen Finucane writes: > On Sun, 2018-02-25 at 01:50 +1100, Daniel Axtens wrote: >> Thomas Petazzoni reported that Patchwork would occasionally lose >> Buildroot email. Andrew - having talked to jk and sfr - suggested that >> this may be race-condition related. >> >> I investigated and found s

Re: [PATCH v2 9/9] parser: don't fail on multiple SeriesReferences

2018-02-25 Thread Daniel Axtens
Stephen Finucane writes: > On Sun, 2018-02-25 at 01:50 +1100, Daniel Axtens wrote: >> Parallel parsing would occasonally fail with: >> >> patchwork.models.MultipleObjectsReturned: get() returned more than one >> SeriesReference -- it returned 2! >> >> I think these are happening if you have di

Re: [PATCH v2 3/9] tools/scripts: split a mbox N ways

2018-02-25 Thread Daniel Axtens
Stephen Finucane writes: > On Sun, 2018-02-25 at 01:50 +1100, Daniel Axtens wrote: >> To test parallel loading of mail, it's handy to be able to split >> an existing mbox file into N mbox files in an alternating pattern >> (e.g. 1 2 1 2 or 1 2 3 4 1 2 3 4 etc) >> >> Introduce tools/scripts as a

Re: [PATCH] [RFC] tools: drop vagrant

2018-02-25 Thread Andrew Donnellan
On 24/02/18 12:22, Daniel Axtens wrote: It served us well, but it's now outdated (Trusty, Python 3.4, etc) There is no indication that anyone uses it or keeps it up to date. Signed-off-by: Daniel Axtens Vagrant is dead, long live Docker! Reviewed-by: Andrew Donnellan -- Andrew Donnellan

Re: [PATCH v2 0/9] Tools and fixes for parallel parsing

2018-02-25 Thread Stephen Finucane
On Sun, 2018-02-25 at 01:50 +1100, Daniel Axtens wrote: > Thomas Petazzoni reported that Patchwork would occasionally lose > Buildroot email. Andrew - having talked to jk and sfr - suggested that > this may be race-condition related. > > I investigated and found some bugs. I first had to develop s

Re: [PATCH v2 9/9] parser: don't fail on multiple SeriesReferences

2018-02-25 Thread Stephen Finucane
On Sun, 2018-02-25 at 01:50 +1100, Daniel Axtens wrote: > Parallel parsing would occasonally fail with: > > patchwork.models.MultipleObjectsReturned: get() returned more than one > SeriesReference -- it returned 2! > > I think these are happening if you have different processes parsing > e.g. 1/

Re: [PATCH v2 7/9] parser: avoid an unnecessary UPDATE of Person

2018-02-25 Thread Stephen Finucane
On Sun, 2018-02-25 at 01:50 +1100, Daniel Axtens wrote: > Analysis of SQL statements showed that when parsing an email, the row > for the Person who sent the email was always getting updated. This is > because the test for updating it only checks if the incoming mail has > *a* name attached to the

Re: [PATCH v2 8/9] parser: use Patch.objects.create instead of save()

2018-02-25 Thread Stephen Finucane
On Sun, 2018-02-25 at 01:50 +1100, Daniel Axtens wrote: > Attempts to do parallel parsing with MySQL threw the following errors: > > _mysql_exceptions.OperationalError: (1213, 'Deadlock found when trying to get > lock; try restarting transaction') > > Looking at the code, it was thrown when we c

Re: [PATCH v2 5/9] parser: Handle even more exotically broken headers

2018-02-25 Thread Stephen Finucane
On Sun, 2018-02-25 at 01:50 +1100, Daniel Axtens wrote: > An archive of the Ubuntu kernel team mailing list contains a > fascinating email that causes the following parse error: > > email.errors.HeaderParseError: header value appears to contain an embedded > header: > '4Mf^tnii7k\\_EnR5aobBm6Di

Re: [PATCH v2 4/9] tools/scripts: parallel_parsearchive - load archives in parallel

2018-02-25 Thread Stephen Finucane
On Sun, 2018-02-25 at 01:50 +1100, Daniel Axtens wrote: > If you have multiple archives, you quickly tire of typing stuff like > python3 manage.py parsearchive --list-id=patchwork.ozlabs.org foo-1 & > python3 manage.py parsearchive --list-id=patchwork.ozlabs.org foo-2 & > python3 manage.py parsearc

Re: [PATCH v2 3/9] tools/scripts: split a mbox N ways

2018-02-25 Thread Stephen Finucane
On Sun, 2018-02-25 at 01:50 +1100, Daniel Axtens wrote: > To test parallel loading of mail, it's handy to be able to split > an existing mbox file into N mbox files in an alternating pattern > (e.g. 1 2 1 2 or 1 2 3 4 1 2 3 4 etc) > > Introduce tools/scripts as a place to put things like this. >

Re: [PATCH v2 2/9] debugging: add command to dump patches and series

2018-02-25 Thread Stephen Finucane
On Sun, 2018-02-25 at 01:50 +1100, Daniel Axtens wrote: > I don't want a full dump, just enough to know if the same patches > and series have been created with roughly the same properties. This > seemed like the easiest way to do it. > > Usage: > python3 manage.py debug_dump > file > ... make

Re: [PATCH v2 1/9] tools/docker: assume terminal supports utf-8

2018-02-25 Thread Stephen Finucane
On Sun, 2018-02-25 at 01:50 +1100, Daniel Axtens wrote: > Set PYTHONIOENCODING to UTF-8, which allows Python3 to print UTF-8 > directly to the terminal (to a pipe or shell-redirected file) rather > than throwing an error. > > Reviewed-by: Andrew Donnellan > Signed-off-by: Daniel Axtens Fine by

Re: [PATCH] [RFC] tools: drop vagrant

2018-02-25 Thread Stephen Finucane
On Sat, 2018-02-24 at 12:22 +1100, Daniel Axtens wrote: > It served us well, but it's now outdated (Trusty, Python 3.4, etc) > There is no indication that anyone uses it or keeps it up to date. > > Signed-off-by: Daniel Axtens I'm good to drop this. I've been using Docker exclusively for a while