On Mon, Jan 29, 2018 at 11:36:36PM +1100, Daniel Axtens wrote:
> Hi Don,
>
> >> I suppose to put a finer point on it - what is your usecase here? What
> >> are you trying to achieve, and can we help you do that in a way that
> >> requires smaller changes to Patchwork, and is less fragile? (For example
> >> this is going to break if someone mis-spells the keyword you're looking
> >> for in the subject_match.)
> >
> > So here is our use case. Internally at Red Hat we use one mailing list to
> > post all of our kernel patches. However, being a distro company, patches
> > can be applied to either RHEL-6,7,X. For the last 15 years we have always
> > used the method:
> >
> > [RHEL-7 PATCH] instead of [PATCH].
>
> Ah yes. I work for Canonical (although I do Patchwork in a private
> capacity only) and our kernel team does something very similar.
>
> > The project inside the []s has been what we filter through our regex. Is it
> > prone to human typos? Yes. Most folks have stuck this in the .git/config
> > subjectprefix option. That limited the typos. It isn't perfect.
> >
> > We have been using a hacked up PatchWork1 for the last 7 or so years. This
> > is one of those features we need (or something to solve our problem) if we
> > want to migrate to a PatchWork2 instance.
> >
> > I hope that adds a little bit of context on our thinking. Thoughts?
>
> That's a compelling use-case and I'm happy to look at supporting it. I
> will review the patch more closely in the coming days.
Thanks for your understanding and support!
Again, we know the solution has its 'human' limitations. :-) We just never
came up with a better idea. So any ideas there are welcomed too! :-)
Cheers,
Don
>
> Regards,
> Daniel
>
> > Cheers,
> > Don
> >
> >>
> >> Regards,
> >> Daniel
> >>
> >> >
> >> > Signed-off-by: Veronika Kabatova <vkaba...@redhat.com>
> >> > ---
> >> > docs/api.yaml | 3 +++
> >> > docs/deployment/management.rst | 4 +--
> >> > patchwork/api/project.py | 5 ++--
> >> > .../0021_add_subject_match_to_project.py | 29
> >> > ++++++++++++++++++++++
> >> > patchwork/models.py | 9 ++++++-
> >> > patchwork/parser.py | 19 +++++++++++++-
> >> > .../notes/list-filtering-4643d98b4064367a.yaml | 10 ++++++++
> >> > 7 files changed, 73 insertions(+), 6 deletions(-)
> >> > create mode 100644
> >> > patchwork/migrations/0021_add_subject_match_to_project.py
> >> > create mode 100644
> >> > releasenotes/notes/list-filtering-4643d98b4064367a.yaml
> >> >
> >> > diff --git a/docs/api.yaml b/docs/api.yaml
> >> > index 3e79f0b..3373226 100644
> >> > --- a/docs/api.yaml
> >> > +++ b/docs/api.yaml
> >> > @@ -374,6 +374,9 @@ definitions:
> >> > list_id:
> >> > type: string
> >> > description: Mailing list identifier for project.
> >> > + subject_match:
> >> > + type: string
> >> > + description: Regex used for email filtering.
> >> > list_email:
> >> > type: string
> >> > description: Mailing list email address for project.
> >> > diff --git a/docs/deployment/management.rst
> >> > b/docs/deployment/management.rst
> >> > index c50b7b6..870d7ee 100644
> >> > --- a/docs/deployment/management.rst
> >> > +++ b/docs/deployment/management.rst
> >> > @@ -63,7 +63,7 @@ due to, for example, an outage.
> >> > .. option:: --list-id <list-id>
> >> >
> >> > mailing list ID. If not supplied, this will be extracted from the
> >> > mail
> >> > - headers.
> >> > + headers. Don't use this option if you require filtering based on
> >> > subjects.
> >> >
> >> > .. option:: infile
> >> >
> >> > @@ -88,7 +88,7 @@ the :ref:`deployment installation guide
> >> > <deployment-parsemail>`.
> >> > .. option:: --list-id <list-id>
> >> >
> >> > mailing list ID. If not supplied, this will be extracted from the
> >> > mail
> >> > - headers.
> >> > + headers. Don't use this option if you require filtering based on
> >> > subjects.
> >> >
> >> > .. option:: infile
> >> >
> >> > diff --git a/patchwork/api/project.py b/patchwork/api/project.py
> >> > index 446c473..597f605 100644
> >> > --- a/patchwork/api/project.py
> >> > +++ b/patchwork/api/project.py
> >> > @@ -39,8 +39,9 @@ class ProjectSerializer(HyperlinkedModelSerializer):
> >> > class Meta:
> >> > model = Project
> >> > fields = ('id', 'url', 'name', 'link_name', 'list_id',
> >> > 'list_email',
> >> > - 'web_url', 'scm_url', 'webscm_url', 'maintainers')
> >> > - read_only_fields = ('name', 'maintainers')
> >> > + 'web_url', 'scm_url', 'webscm_url', 'maintainers',
> >> > + 'subject_match')
> >> > + read_only_fields = ('name', 'maintainers', 'subject_match')
> >> > extra_kwargs = {
> >> > 'url': {'view_name': 'api-project-detail'},
> >> > }
> >> > diff --git a/patchwork/migrations/0021_add_subject_match_to_project.py
> >> > b/patchwork/migrations/0021_add_subject_match_to_project.py
> >> > new file mode 100644
> >> > index 0000000..6066266
> >> > --- /dev/null
> >> > +++ b/patchwork/migrations/0021_add_subject_match_to_project.py
> >> > @@ -0,0 +1,29 @@
> >> > +# -*- coding: utf-8 -*-
> >> > +# Generated by Django 1.10.8 on 2018-01-19 18:16
> >> > +from __future__ import unicode_literals
> >> > +
> >> > +from django.db import migrations, models
> >> > +
> >> > +
> >> > +class Migration(migrations.Migration):
> >> > +
> >> > + dependencies = [
> >> > + ('patchwork', '0020_tag_show_column'),
> >> > + ]
> >> > +
> >> > + operations = [
> >> > + migrations.AddField(
> >> > + model_name='project',
> >> > + name='subject_match',
> >> > + field=models.CharField(blank=True, default=b'',
> >> > help_text=b'Regex to match the subject against if only part of emails
> >> > sent to the list belongs to this project. Will be used with IGNORECASE
> >> > and MULTILINE flags. If rules for more projects match the first one
> >> > returned from DB is chosen.', max_length=64),
> >> > + ),
> >> > + migrations.AlterField(
> >> > + model_name='project',
> >> > + name='listid',
> >> > + field=models.CharField(max_length=255),
> >> > + ),
> >> > + migrations.AlterUniqueTogether(
> >> > + name='project',
> >> > + unique_together=set([('listid', 'subject_match')]),
> >> > + ),
> >> > + ]
> >> > diff --git a/patchwork/models.py b/patchwork/models.py
> >> > index 11886f1..907707f 100644
> >> > --- a/patchwork/models.py
> >> > +++ b/patchwork/models.py
> >> > @@ -71,8 +71,14 @@ class Project(models.Model):
> >> >
> >> > linkname = models.CharField(max_length=255, unique=True)
> >> > name = models.CharField(max_length=255, unique=True)
> >> > - listid = models.CharField(max_length=255, unique=True)
> >> > + listid = models.CharField(max_length=255)
> >> > listemail = models.CharField(max_length=200)
> >> > + subject_match = models.CharField(
> >> > + max_length=64, blank=True, default='', help_text='Regex to
> >> > match the '
> >> > + 'subject against if only part of emails sent to the list
> >> > belongs to '
> >> > + 'this project. Will be used with IGNORECASE and MULTILINE
> >> > flags. If '
> >> > + 'rules for more projects match the first one returned from DB
> >> > is '
> >> > + 'chosen.')
> >> >
> >> > # url metadata
> >> >
> >> > @@ -100,6 +106,7 @@ class Project(models.Model):
> >> > return self.name
> >> >
> >> > class Meta:
> >> > + unique_together = (('listid', 'subject_match'),)
> >> > ordering = ['linkname']
> >> >
> >> >
> >> > diff --git a/patchwork/parser.py b/patchwork/parser.py
> >> > index ac7dc5f..015f709 100644
> >> > --- a/patchwork/parser.py
> >> > +++ b/patchwork/parser.py
> >> > @@ -157,9 +157,25 @@ def find_project_by_id(list_id):
> >> > project = Project.objects.get(listid=list_id)
> >> > except Project.DoesNotExist:
> >> > logger.debug("'%s' if not a valid project list-id", list_id)
> >> > + except Project.MultipleObjectsReturned:
> >> > + logger.debug("Multiple projects with list-id '%s' found",
> >> > list_id)
> >> > return project
> >> >
> >> >
> >> > +def find_project_by_id_and_subject(list_id, subject):
> >> > + """Find a `project` object based on `list_id` and subject prefix
> >> > match."""
> >> > + projects = Project.objects.filter(listid=list_id)
> >> > + for project in projects:
> >> > + if (not project.subject_match or
> >> > + re.search(project.subject_match, subject,
> >> > + re.MULTILINE | re.IGNORECASE)):
> >> > + return project
> >> > +
> >> > + logger.debug("No project to match email with list-id '%s', subject
> >> > '%s' "
> >> > + "found", list_id, subject)
> >> > + return None
> >> > +
> >> > +
> >> > def find_project_by_header(mail):
> >> > project = None
> >> > listid_res = [re.compile(r'.*<([^>]+)>.*', re.S),
> >> > @@ -181,7 +197,8 @@ def find_project_by_header(mail):
> >> >
> >> > listid = match.group(1)
> >> >
> >> > - project = find_project_by_id(listid)
> >> > + project = find_project_by_id_and_subject(listid,
> >> > +
> >> > mail.get('Subject'))
> >> > if project:
> >> > break
> >> >
> >> > diff --git a/releasenotes/notes/list-filtering-4643d98b4064367a.yaml
> >> > b/releasenotes/notes/list-filtering-4643d98b4064367a.yaml
> >> > new file mode 100644
> >> > index 0000000..21c1680
> >> > --- /dev/null
> >> > +++ b/releasenotes/notes/list-filtering-4643d98b4064367a.yaml
> >> > @@ -0,0 +1,10 @@
> >> > +---
> >> > +features:
> >> > + - |
> >> > + Allow list filtering into multiple projects (and email dropping)
> >> > based on
> >> > + subject prefixes. Disabled by default, enable by specifying a
> >> > regular
> >> > + expression which needs to be matched in the subject on a
> >> > per-project basis
> >> > + (field `subject_match`).
> >> > +api:
> >> > + - |
> >> > + Expose `subject_match` in REST API.
> >> > --
> >> > 2.13.6
> >> >
> >> > _______________________________________________
> >> > Patchwork mailing list
> >> > Patchwork@lists.ozlabs.org
> >> > https://lists.ozlabs.org/listinfo/patchwork
> >> _______________________________________________
> >> Patchwork mailing list
> >> Patchwork@lists.ozlabs.org
> >> https://lists.ozlabs.org/listinfo/patchwork
_______________________________________________
Patchwork mailing list
Patchwork@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/patchwork
