Re: [Patchwork-maintainers] patchwork.ozlabs.org downtime for maintenance - 15/16 August

[Konstantin Ryabitsev]

On Tue, Jan 16, 2024 at 11:05:42AM -0600, Rob Herring wrote:
> > This improved things, but it seems like the DT PW has gotten slower
> > again in the last few weeks. It's taking ~12sec to load the patch
> > list.
> 
> I'm still seeing this slowness and now to add to that intermittent 503 errors.

In case it helps, and just for the record, the following index really helped
to improve some of the patch list views when delegates were used:

ALTER TABLE patchwork_patch ADD INDEX 
patchwork_patch_delegate_id_state_id_archived (delegate_id, state_id, archived);

-K
___
Patchwork mailing list
Patchwork@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/patchwork

Re: PyPatchwork

[Konstantin Ryabitsev]

On Tue, Feb 21, 2023 at 11:21:49AM -0800, Tedd Ho-Jeong An wrote:
> Hi
> 
> I wrote the python library to use Patchwork REST API for my own project
> and I would like to share with Patchwork community.

This is nice, thank you for putting that together!

-K
___
Patchwork mailing list
Patchwork@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/patchwork

Re: [PATCH 1/2] REST: Don't error if a versioned field we would remove is absent

[Konstantin Ryabitsev]

On Wed, Oct 27, 2021 at 05:00:34PM +0100, Stephen Finucane wrote:
> v2.2.6 is available now. Let me know if there are any issues.

Thanks!

> Somewhat related: are there any large blockers preventing you moving to 
> v3.0.x?
> Is it the major version bump or removal of Python 2.7 support that's 
> preventing
> the upgrade, or is there something else? In particular, is there anything that
> we could address? I'd like to get v3.1.0 out before the end of year but I'm
> slightly concerned that no one (?) has rolled forward to v3.0.x yet :D

The only concern, really, is the data migration and the potential for
regressions. There are several subsystems now that are reliant on the CI
aspects of patchwork.kernel.org to track their work, so I'm trying to be
careful not to disrupt their work. I know that there are some cool new
features in 3.x that would warrant migrating to it sooner than later -- just
trying to plan out when is the best time to do it.

On the upside, we're on the cusp of moving the web ui parts to a containerized
deployment, so perhaps I can trial out the migration to 3.0 as part of that
work.

-K
___
Patchwork mailing list
Patchwork@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/patchwork

Re: [PATCH 1/2] REST: Don't error if a versioned field we would remove is absent

[Konstantin Ryabitsev]

On Fri, 20 Aug 2021 at 17:59, Stephen Finucane  wrote:
> > We remove fields that shouldn't be seen on old versions of the API.
> > This was done with `pop(field name)`, which will throw an exception
> > if the named field is absent from the data. However, sometimes if
> > a patch request is via an old API version, we hit this line without
> > ever having the field present.
> >
> > This is odd, but not harmful and we definitely shouldn't 500.
> >
> > Fixes: d944f17ec059 ("REST: Use versioning for modified responses")
> > Signed-off-by: Daniel Axtens 
>
> Looks good to me.
>
> Reviewed-by: Stephen Finucane 
>
> I squashed the test into this (with a tweak in the name) and applied it.

Hi, all:

Could this please be backported to stable/2.2 and maybe released as 2.2.6?

-K
___
Patchwork mailing list
Patchwork@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/patchwork

Re: Feature request: mbox for series

[Konstantin Ryabitsev]

On Thu, Sep 16, 2021 at 08:51:21AM +0200, Rafał Miłecki wrote:
> Could you consider support for downloading mbox for all patches in a
> series? I'd like to use "curl  | git am" to download & apply all
> patches at once. That would save me quite some time of copying URLs of
> single patches.

Rafał:

You can already do this, e.g.:

curl -sL https://patchwork.kernel.org/series/547921/mbox/

Alternatively, if you're not actually using patchwork's CI and state tracking
features, you can use b4 (assuming you're working with lists that are archived
on lore.kernel.org).

-K
___
Patchwork mailing list
Patchwork@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/patchwork

Re: PaReD: a patch relations detector for patchwork

[Konstantin Ryabitsev]

On Sat, Sep 04, 2021 at 08:04:48AM +0200, Lukas Bulwahn wrote:
> For completeness, I need to mention that Konstantin's b4 tool also
> detects the "latest patch series" when you ask it to pick a patch
> series from a kernel mailing list. I do not know how it determines
> that (and I hope that Konstantin can comment here), but it is probably
> also a simple heuristics searching for similar/same subject lines of
> the patch series cover letter.

There are two ways that b4 supports. One is a simple query for the same author
and subject, which obviously doesn't work if the subject changes.

The second one is support for the Obsoleted-by: follow-up tag, but nobody
really uses it because it's not widely known. I really envisioned it to be
used by automated tooling like the upcoming auto-pull-request-exploder.

I am leaning towards embracing the Change-id: header, which would reference
the message-id of the initial patch/series (or URL like
lore.kernel.org/r/message-id). That would allow us to more effectively track
the series progression.

-K
___
Patchwork mailing list
Patchwork@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/patchwork

Re: [PATCH 1/2] REST: Don't error if a versioned field we would remove is absent

[Konstantin Ryabitsev]

On Sat, Aug 21, 2021 at 12:57:58AM +1000, Daniel Axtens wrote:
> This is odd, but not harmful and we definitely shouldn't 500.

I can confirm that it fixes the 500 error and restores previous functionality.
Thanks very much!

Tested-by: Konstantin Ryabitsev 

-K
___
Patchwork mailing list
Patchwork@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/patchwork

2.2.5 release api-1.1 500 errors

[Konstantin Ryabitsev]

Hello:

After the 2.2.2->2.2.5 upgrade, we seem to be getting lots of /api/1.1/ errors
for PATCH calls:

PATCH /api/1.1/patches/{id}/

switching to /api/1.2/ seems to fix it, but requires other client app fixes
for API incompatibility.

Some of the others returning 500 errors are:

GET /api/patches/{msgid}/comments/ 

I'd be happy to provide tracebacks if I can figure out how to get them.

-K
___
Patchwork mailing list
Patchwork@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/patchwork

Re: Introduction + Importing git mailing list archives

[Konstantin Ryabitsev]

On Wed, Jun 16, 2021 at 05:14:23PM -0400, Raxel Gutierrez wrote:
> I set up my development environment for Patchwork through the manual
> installation steps. Since I am focusing on the Git project, I am
> trying to set up a test instance using the Git mailing list archive. I
> was able to successfully import the Patchwork mailing list archives as
> outlined [1]. The Patchwork mailing list uses a mailman instance but
> the Git mailing list doesn't, so the instructions wouldn’t work [2].
> My host (jrnieder) helped me locate a file [3] that Konstantin wrote
> which seems to already solve this issue, so I am interested in any
> pointers you have about the setup.

It's easy to use public-inbox archives as your patchwork sources, after some
initial configuration (primarily, so you properly import GitGitGadget series).

Here's how to go about it:

1. Read the following post and configure grokmirror with grok-pi-piper. Do NOT
   set shallow=yes, as you'll want to import the entirety of archives.
   
https://people.kernel.org/monsieuricon/subscribing-to-lore-lists-with-grokmirror

2. Adapt the following procmail rules here to properly import 
   GitGitGadget-created patch series:
   https://git.kernel.org/pub/scm/infra/patchwork/procmail.git/tree/git.rc 3. 

3. Find the file git/git/0.git/pi-piper.latest and change it to contain the
   following hash: dabc1bccbf7b767275a4a46d3f11ef50151027db (this is the first
   commit in the public-inbox repository archive).

4. Now you can run:
   grok-pi-piper -c path/to/pi-piper.conf /path/to/git/git/0.git

This should import the entirety of git archives (15 years worth) into
patchwork. Once it's done, you can continuously import new messages by setting
up the grok-pull service as described in the people.kernel.org post I linked
above.

Please let me know if you run into any troubles.

-K
___
Patchwork mailing list
Patchwork@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/patchwork

Re: Patchwork gets confused about authorship

[Konstantin Ryabitsev]

On Thu, May 20, 2021 at 12:39:43PM -0700, Jonathan Nieder wrote:
> > It's only required if there is no DKIM signature on the original message 
> > (see
> > my musings on this subject here:
> > https://people.kernel.org/monsieuricon/subspace-mailing-list-server#appeasing-dmarc
> >  )
> >
> > If there is no DKIM signature on the original mail, then yes, the DMARC
> > verification server is supposed to perform an "alignment check" to make sure
> > that the envelope-from domain matches the in-header From: domain.
> [...]
> > A much better solution is to have everyone properly DKIM-sign
> > their outgoing mail. :)
> 
> I just wanted to thank you for this.  I remember when DMARC+SPF broke
> the mailing list ecosystem, and I had no idea that increased adoption
> of DKIM is going to allow un-breaking it.  That little glimmer of hope
> was a great addition to my day.

Good to hear. And DKIM signing also helps with patch attestation, so it's a
win-win if more people start doing it. :) We check DKIM signatures with b4, so
hopefully it wouldn't be too hard to add something to patchwork that would
show whether the DKIM verification passes. Or this could be an external CI
check, I guess.

Anyway, everyone should DKIM-sign their mail!

-K
___
Patchwork mailing list
Patchwork@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/patchwork

Re: Patchwork gets confused about authorship

[Konstantin Ryabitsev]

On Thu, May 20, 2021 at 09:04:43PM +0200, Yann E. MORIN wrote:
> > There's not even an X-Original-From: or anything like that, which would help
> > avoid this problem, since Patchwork will recognize this situation and
> > correctly substitute the list From with the original From.
> 
> OK, so if we could get the list to set X-Original-From, that would
> solve the issue.

Yes -- at least, it should.

> > You should fix the list to not substitute the From: header with the list
> > address.
> 
> Ah, but this is required when the sender is from a domain with DMARC
> with a Reject or Quarantine policy, and the recipient server checks that
> setting.

It's only required if there is no DKIM signature on the original message (see
my musings on this subject here:
https://people.kernel.org/monsieuricon/subspace-mailing-list-server#appeasing-dmarc
 )

If there is no DKIM signature on the original mail, then yes, the DMARC
verification server is supposed to perform an "alignment check" to make sure
that the envelope-from domain matches the in-header From: domain.

Rewriting the From: header is a terrible fix for this, though, as it breaks
the patch flow (unless tools have specific kludges to fix this, like
Patchwork). A much better solution is to have everyone properly DKIM-sign
their outgoing mail. :)

-K
___
Patchwork mailing list
Patchwork@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/patchwork

Re: Patchwork gets confused about authorship

[Konstantin Ryabitsev]

On Thu, May 20, 2021 at 06:15:51PM +0200, Yann E. MORIN wrote:
> Hello,
> 
> With recently noticed that patchwork gets confused about the authorship
> of some patches.

I wouldn't say that it's patchwork that gets confused about the authorship. If
you examine the headers, it's clear that the list causes this problem by
rewriting the "From: " header:

From: "Chan, Donald via buildroot" 

The original From: header is completely obliterated, though I expect it's the
list that inserts the following:

Reply-To: "Chan, Donald" 

There's not even an X-Original-From: or anything like that, which would help
avoid this problem, since Patchwork will recognize this situation and
correctly substitute the list From with the original From.

> For example:
> 
> 
> https://patchwork.ozlabs.org/project/buildroot/patch/1621475246377.74...@lab126.com/
> 
> has Eugen.Hristev set as author, but it was never sent by then. The
> original mail can be had on our mailing list:
> 
> http://lists.busybox.net/pipermail/buildroot/2021-May/310923.html
> 
> which was sent by Chan, Donald.  So, when we apply patches, they get the
> wrong authorship...
> 
> The mbox link (to use with git-am for example), has the wrong From
> header:
> 
> $ curl -s 
> 'https://patchwork.ozlabs.org/project/buildroot/patch/1621475246377.74...@lab126.com/mbox/'
>  |grep -E '^From:'
> From: "Eugen.Hristev--- via buildroot" 

This is because when patchwork received an email with the address
"buildr...@busybox.net" it was sent by Eugen.Hristev -- so the
buildr...@busybox.net account in patchwork has that name associated with it
and will put that into all patches.

Again, not really the problem of patchwork, but the side-effect of how the
list operates.

> Any idea? How can we avoid that?

You should fix the list to not substitute the From: header with the list
address.

-K
___
Patchwork mailing list
Patchwork@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/patchwork

Re: Pain points in Git's patch flow

[Konstantin Ryabitsev]

On Mon, Apr 19, 2021 at 09:49:46PM +0200, Sebastian Schuberth wrote:
> > To watch a particular filename, the "dfn:" prefix may be used.
> > The prefixes supported for a particular instance are documented in
> > , and you
> > can watch multiple files by combining with "OR".
> 
> Thanks for pointing out these interesting features, I wasn't aware of them.

Eric is being modest. There are very cool things brewing in public-inbox, like
ability to create saved searches and follow threads you're interested in.
E.g. you should be able to define something like "whenever someone mentions my
favourite file, function name, or term, copy the entire thread into my inbox
and continuously update it with new messages."

I'm hoping that this will help turn the concept of mailing lists on their head
-- instead of subscribing to a list, folks will instead subscribe to closely
relevant saved searches across any number of remote and local sources.

> > Email is already well-established with a good amount of small
> > players, and plain-text is relatively inexpensive.  So it seems
> > best to build off the only halfway-decentralized thing we have
> > in wide use, rather than trying to start from scratch.
> 
> While I can understand that conservative approach for a community
> around a tool as important as Git, I still fear that only ever
> sticking to technology that is already in wide use will hinder to look
> over the rim of the tea cup.

I view email as merely one way of exchanging RFC2822-formatted messages. 
There are others and RFC2822 is robust enough to serve as a good standard
base that allows both free-form and structured content, including mixed.

-K
___
Patchwork mailing list
Patchwork@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/patchwork

Re: Pain points in Git's patch flow

[Konstantin Ryabitsev]

On Mon, Apr 19, 2021 at 07:54:37AM +0200, Sebastian Schuberth wrote:
> > of these proposed alternatives involve moving away from something that's
> > a distributed system today (E-Mail infrastructure, local clients), to
> > what's essentially some website run by a centralized entity, in some
> > cases proprietary.
> 
> That's a good point, I admit I haven't thought of that. Probably
> because I also don't care much. So *does* it really matter? What
> exactly concerns you about a "centralized entity"? Is it the technical
> aspect of a single point of failure, or the political / social aspect
> of being dependent on someone you do not want to get influenced by? I
> guess it's a bit of both.

It's all of the above, and really should not be discounted. Let's take what
Russian government is doing lately as an example. In its effort to control
social dissent, Russian censorship organization RosKomNadzor (RKN) has taken
steps to deliberately break internet operation -- in a very ham-fisted way.
Just a month ago they tried to "slow down" Twitter by blocking DNS queries for
any domains containing the substring "t.co" -- which, hey, broke
gihubusercontent.com among many other sites. There's every reason to believe
that this won't be the only time they do something idiotic like that, so as a
result it is increasingly difficult for Russian contributors to justify
participating in projects that are hosted on GitHub -- one day they may not be
able to reach it reliably (or at all).

(If you think the answer to that would be "just use a VPN", it's one of those
recommendations that are easy to make for someone not worried about their ISP
reporting "sketchy encrypted traffic" to "the authorities.")

Patches sent via email remain immune to this. Even if vger falls over, it's
merely a list service -- there are alternative ways of transmitting RFC2822
messages that don't involve a central host (such as via a NNTP gateway,
publishing a public-inbox "feed", etc). Email remains one of the few protocols
that are designed ground-up to be decentralized and I'm afraid that we are
again finding ourselves in a world where this is increasingly relevant.

> While these concerns could probably be addressed somewhat e.g. by
> multiple independently operated Gerrit servers that are kept in sync,
> I was curious and quickly search for more fitting "truly
> decentralized" solutions, and came across radicle [1]. Just FYI.

I know Radicle folks -- I was on their technical board. A lot of what they
have implemented is very similar to my initial thoughts expressed in
https://people.kernel.org/monsieuricon/patches-carved-into-developer-sigchains

I have high hopes for the project, but it's not ready to take on the world
until they implement code collaboration aspects (issue tracking, change
requests, etc). It's going to be tough and I really hope they succeed.

-K
___
Patchwork mailing list
Patchwork@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/patchwork

Re: Autodelegation based on more complex rules

[Konstantin Ryabitsev]

On Wed, Feb 17, 2021 at 02:15:23PM +, Ali Alnubani wrote:
> > I suggest that you do this outside of patchwork, using procmail and 
> > filtering
> > hooks to set X-Patchwork-Delegate. This is what we do at kernel.org, though
> > we
> > don't directly parse the MAINTAINERS file. You can see our scripts here:
> > 
> > https://git.kernel.org/pub/scm/infra/patchwork/procmail.git/tree/
> > 
> 
> Thanks for the suggestion, If I understand your configurations correctly,
> they don't consider the hierarchy of trees though (e.g., when submitters
> change files in multiple trees), do they?

No, because we don't have a reason to do so. Moreover, I'm not sure how it
would work, as a submission can only be delegated to a single person/account.
If a patch touches multiple files, you still have to pick a single delegate.

-K
___
Patchwork mailing list
Patchwork@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/patchwork

Re: Autodelegation based on more complex rules

[Konstantin Ryabitsev]

On Mon, Feb 15, 2021 at 12:14:47PM +, Ali Alnubani wrote:
> Hi all,
> 
> There are currently 2 ways in Patchwork to autodelegate patches. Patchwork
> can either parse the hint header (X-Patchwork-Delegate), or it can lookup
> the fnmatch-formatted rules for the project.
> 
> Projects using a MAINTAINERS file might want to use the same rules they
> already have there for delegation. These rules can be in the hundreds, and
> they can change very often. Managing these manually in django's admin
> interface can be a bit hard (I have even seen that Django will render them
> extremely slowly, will cause very high CPU usage, and 5xx timeouts when
> loading a project's admin page).  Adding these rules can be automated using
> django's management shell, however, there are cases where more complex rules
> might be needed to find the right delegate. An example is projects with a
> hierarchy of trees and maintainers, were patches changing files in more than
> a single sub-tree usually require delegating to the maintainer of a
> higher-level tree. Current rule priorities don't support such a case,
> because we would need to consider all files changed at once, not separately.
> 
> Can we implement a hook that would call an external and project-specific
> script in parser.py (specifically in find_delegate_by_filename()) to resolve
> the issues above? The script would be expected to take the filenames as an
> argument, and output a delegates email. This way, any complicated rules
> could be managed by that script.  I realize there might be security
> implications to this though. Please let me know what you think.

I suggest that you do this outside of patchwork, using procmail and filtering
hooks to set X-Patchwork-Delegate. This is what we do at kernel.org, though we
don't directly parse the MAINTAINERS file. You can see our scripts here:

https://git.kernel.org/pub/scm/infra/patchwork/procmail.git/tree/

If you end up writing a hook that will set X-Patchwork-Delegate based on what
is in MAINTAINERS, I'm sure we'll be happy to adopt it. :)

-K
___
Patchwork mailing list
Patchwork@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/patchwork

Re: UTF-8-challenged checkpatch on patchwork

[Konstantin Ryabitsev]

On Tue, Dec 01, 2020 at 10:36:43AM -0800, Jakub Kicinski wrote:
> > Not sure if this is a bug in requests or not, but the corruption is 
> > introduced in .text. Here's POC:
> > 
> > 
> > #!/usr/bin/env python3
> > import requests
> > 
> > pmbx = 
> > 'https://patchwork.kernel.org/project/netdevbpf/patch/20201130183705.17540-1-t...@redhat.com/mbox/'
> > res = requests.get(pmbx)
> > 
> > print('content: ' + res.content.decode().split('\n')[102])
> > print('   text: ' + res.text.split('\n')[102])
> > 
> > 
> > $ python3 test.py
> > content: Signed-off-by: Toke Høiland-Jørgensen 
> >text: Signed-off-by: Toke Høiland-Jørgensen 
> 
> Ah, great! Thank you! I pushed out a patch, I will put it in prod after 
> the mid-day rush is over.

Following up, this is not a bug in requests, but a bug in patchwork.  
Here are the headers it sends:

$ wget -S 
https://patchwork.kernel.org/project/netdevbpf/patch/20201130183705.17540-1-t...@redhat.com/mbox/
--2020-12-01 13:45:02--  
https://patchwork.kernel.org/project/netdevbpf/patch/20201130183705.17540-1-t...@redhat.com/mbox/
Resolving patchwork.kernel.org (patchwork.kernel.org)... 198.145.29.83
Connecting to patchwork.kernel.org (patchwork.kernel.org)|198.145.29.83|:443... 
connected.
HTTP request sent, awaiting response...
  HTTP/1.1 200 OK
  Server: nginx
  Date: Tue, 01 Dec 2020 18:45:03 GMT
  Content-Type: text/plain
  Content-Length: 6151
  Connection: keep-alive
  Content-Disposition: attachment; 
filename=net-inet_ecn-Fix-endianness-of-checksum-update-when-setting-ECT-1.patch
  X-Frame-Options: SAMEORIGIN
  proxy_redirect: off

The HTTP/1.1 standard defines the default character encoding as 
iso-8859-1, so response.text is not wrong. Patchwork should be setting 
the correct encoding in the header to be utf-8:

Content-Type: text/plain; charset=utf-8

(here: 
https://github.com/getpatchwork/patchwork/blob/master/patchwork/views/patch.py#L157)

Since it is, in fact, always utf-8:

https://github.com/getpatchwork/patchwork/blob/master/patchwork/views/utils.py#L117

I don't have time to submit a proper patch, so I'm just cowardly cc'ing 
the patchwork list in hopes that either someone does it for me, or that 
I remember to do it at some later point. :)

-K
___
Patchwork mailing list
Patchwork@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/patchwork

Cleaning out archived patches

[Konstantin Ryabitsev]

Hi, all:

One way to slim down the DB size would be to turn archived patches into 
a simple mapping of patchid/msgid to the archival URL. A request to an 
archived patch would then be a simple redirect to the archival site, 
e.g.:

https://patchwork.kernel.org/patch/11865643 
 -> 
https://lore.kernel.org/r/a90452e4fe7c6064026edc8377debd456f54137d.1603944740.git.pavana.sha...@digi.com

Similarly,

https://patchwork.kernel.org/project/netdevbpf/patch/a90452e4fe7c6064026edc8377debd456f54137d.1603944740.git.pavana.sha...@digi.com/
 -> 
https://lore.kernel.org/r/a90452e4fe7c6064026edc8377debd456f54137d.1603944740.git.pavana.sha...@digi.com

The table of archived patches would simply be:

patch_id | proj_name | msgid| archive_url
11865643 | netdevbpf | a9045... | https://lore.kernel.org/...

None of the above values would be foreign keys, which would allow 
cleaning up old patches and old projects without breaking links.

This should help dramatically reduce the number of rows involved in most 
operations.

What do you think?

-K
___
Patchwork mailing list
Patchwork@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/patchwork

Re: Parsing mail on a different system

[Konstantin Ryabitsev]

On Wed, Oct 28, 2020 at 11:31:49AM +1100, Daniel Axtens wrote:
> > I think we may be oversetimating how many people out there run
> > patchwork. :) My general concern was to make sure that parsemail doesn't
> > do things like clear django data. If, to your knowledge, all it does is
> > write to the database without interacting with the running process, then
> > there's no reason why the feeder and the consumer can't live on
> > different systems.
> 
> Yeah, Patchwork sits in this really weird space where there are not many
> deployments, but still too many for us to deeply integrate with any
> one site's infrastructure/setup. It also means that we don't have direct
> access to a copy of any site's data, which regularly bites everyone when
> we write inefficient migrations.

I can make available the data from https://lore.kernel.org/patchwork -- 
that's a "read-only" patchwork with LKML sources that is only modified 
by the patchwork-bot and no actual developers. It uses postgresql as the 
backend, so it would be easy for me to provide a full DB dump for tests 
without any privacy concerns.

-K
___
Patchwork mailing list
Patchwork@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/patchwork

Re: Parsing mail on a different system

[Konstantin Ryabitsev]

On Wed, Oct 21, 2020 at 07:40:07PM +0100, Stephen Finucane wrote:
> On Mon, 2020-10-05 at 16:13 -0400, Konstantin Ryabitsev wrote:
> > Hello:
> > 
> > Quick question -- does parsemail need to run on the same system where 
> > the frontend runs, or can this be done on another system that just 
> > writes to the same database?
> 
> I was hoping someone that has deployed Patchwork in production would
> answer this more authoritatively than I can, however, I can't see any
> reason you couldn't run these on separate hosts. Whether this has
> actually been done is a different matter. I guess anyone containerizing
> the thing will have encountered and resolved this issue?

I think we may be oversetimating how many people out there run
patchwork. :) My general concern was to make sure that parsemail doesn't
do things like clear django data. If, to your knowledge, all it does is
write to the database without interacting with the running process, then
there's no reason why the feeder and the consumer can't live on
different systems.

-K
___
Patchwork mailing list
Patchwork@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/patchwork

Re: 0039_unique_series_references migration needs optimization

[Konstantin Ryabitsev]

On Tue, 13 Oct 2020 at 13:15, Konstantin Ryabitsev
 wrote:
> I may be venting some frustration, but I'm now 4 hours into a
> 2.1 -> 2.2 upgrade. The following query from 0039 has now been running
> over 3 hours:
>
>   COMMAND: Query
>  TIME: 10921
> STATE: Sending data
>  INFO: DELETE a FROM patchwork_seriesreference a
> INNER JOIN patchwork_seriesreference b
> WHERE
>   a.id < b.id
>   AND a.project_id = b.project_id
>   AND a.msgid = b.msgid
>
> That table has 220,000 rows, but it still seems strange that it's taking
> so long to execute. As there is no way to know if it's about to be
> finished or will take another few hours to run, I'm not willing to kill
> it yet.
>
> Would adding an index across (id, project_id, msgid) make this query
> faster to run?

I can confirm that adding a temporary index on (project_id, msgid) and
then rerunning the above query makes it complete in less than 2
seconds. So, the migration should do the following:

add the project field to the model
create temporary non-unique index across msgid, project_id
run the duplicate-removal query above
delete temporary index
create the unique index

-K


-- 
Konstantin Ryabitsev
Director, IT Core Projects
The Linux Foundation
Montréal, Québec
___
Patchwork mailing list
Patchwork@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/patchwork

0039_unique_series_references migration needs optimization

[Konstantin Ryabitsev]

Hello:

I may be venting some frustration, but I'm now 4 hours into a 
2.1 -> 2.2 upgrade. The following query from 0039 has now been running 
over 3 hours:

  COMMAND: Query
 TIME: 10921
STATE: Sending data
 INFO: DELETE a FROM patchwork_seriesreference a
INNER JOIN patchwork_seriesreference b
WHERE
  a.id < b.id
  AND a.project_id = b.project_id
  AND a.msgid = b.msgid

That table has 220,000 rows, but it still seems strange that it's taking 
so long to execute. As there is no way to know if it's about to be 
finished or will take another few hours to run, I'm not willing to kill 
it yet.

Would adding an index across (id, project_id, msgid) make this query 
faster to run?

-K
___
Patchwork mailing list
Patchwork@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/patchwork

Parsing mail on a different system

[Konstantin Ryabitsev]

Hello:

Quick question -- does parsemail need to run on the same system where 
the frontend runs, or can this be done on another system that just 
writes to the same database?

-K
___
Patchwork mailing list
Patchwork@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/patchwork

Re: Patch [2/5] not part of the rest of the series

[Konstantin Ryabitsev]

On Thu, Mar 19, 2020 at 03:28:32PM +, Stephen Finucane wrote:
> > I have a rather strange case here:
> > 
> > https://patchwork.kernel.org/project/linux-rdma/list/
> > 
> > If you look for "258273" in the page, you will find an oddball situation 
> > where a [v2,2/5] is randomly recorded as part of "Untitled series", even 
> > though the rest of the patches in that series are processed properly.
> > 
> > I looked through the headers and all the threading appears to be 
> > correct. It would be easy to sketch this up to random failure, but the 
> > exact same thing happened with the v1 of that series:
> > 
> > https://patchwork.kernel.org/patch/11442933/
> > https://patchwork.kernel.org/patch/11442939/
> > 
> > Again, it's 2/5 that is oddly out of place.
> > 
> > Any guesses?
> 
> You're looking for #241 [1]. tl;dr: We race when creating a series and
> the database constraints that should have protected us were not
> correct. That's been fixed on master [2] and should form part of
> v2.2.0.

Thanks for the pointer! I'll plan to upgrade sooner than later, then. :)

-K
___
Patchwork mailing list
Patchwork@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/patchwork

Patch [2/5] not part of the rest of the series

[Konstantin Ryabitsev]

Hello:

I have a rather strange case here:

https://patchwork.kernel.org/project/linux-rdma/list/

If you look for "258273" in the page, you will find an oddball situation 
where a [v2,2/5] is randomly recorded as part of "Untitled series", even 
though the rest of the patches in that series are processed properly.

I looked through the headers and all the threading appears to be 
correct. It would be easy to sketch this up to random failure, but the 
exact same thing happened with the v1 of that series:

https://patchwork.kernel.org/patch/11442933/
https://patchwork.kernel.org/patch/11442939/

Again, it's 2/5 that is oddly out of place.

Any guesses?

-K
___
Patchwork mailing list
Patchwork@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/patchwork

[PATCH] Handle pull requests with random trailing space

[Konstantin Ryabitsev]

Another fix for copy-pasted pull requests, this time for cases
when something is copy-pasted from a terminal and retains all
the bogus trailing whitespace.

Example:
https://lore.kernel.org/r/043eb5b2-a302-4de6-a3e8-8238e4948...@ti.com

Signed-off-by: Konstantin Ryabitsev 
---
 patchwork/parser.py   |  2 +-
 .../0024-git-pull-request-trailing-space.mbox | 60 +++
 patchwork/tests/test_parser.py|  9 +++
 3 files changed, 70 insertions(+), 1 deletion(-)
 create mode 100644 
patchwork/tests/mail/0024-git-pull-request-trailing-space.mbox

diff --git a/patchwork/parser.py b/patchwork/parser.py
index 45dd8db..5d86426 100644
--- a/patchwork/parser.py
+++ b/patchwork/parser.py
@@ -937,7 +937,7 @@ def parse_patch(content):
 def parse_pull_request(content):
 git_re = re.compile(
 r'^The following changes since commit.*'
-r'^are available in the git repository at:\n'
+r'^are available in the git repository at:\s*\n'
 r'^\s*([\w+-]+(?:://|@)[\w/.@:~-]+[\s\\]*[\w/._-]*)\s*$',
 re.DOTALL | re.MULTILINE | re.IGNORECASE)
 match = git_re.search(content)
diff --git a/patchwork/tests/mail/0024-git-pull-request-trailing-space.mbox 
b/patchwork/tests/mail/0024-git-pull-request-trailing-space.mbox
new file mode 100644
index 000..d62d070
--- /dev/null
+++ b/patchwork/tests/mail/0024-git-pull-request-trailing-space.mbox
@@ -0,0 +1,60 @@
+From mboxrd@z Thu Jan  1 00:00:00 1970
+To: Linux ARM Kernel List 
+From: XXX XXX 
+Subject: [GIT PULL] DaVinci SoC updates for v5.6
+Message-ID: <043eb5b2-a302-4de6-a3e8-8238e4948...@ti.com>
+Date: Tue, 14 Jan 2020 23:48:54 +0530
+Content-Type: text/plain; charset="us-ascii"
+Content-Transfer-Encoding: 7bit
+
+The following changes since commit e42617b825f8073569da76dc4510bfa019b1c35a:   
 
+   
 
+  Linux 5.5-rc1 (2019-12-08 14:57:55 -0800)
 
+   
 
+are available in the Git repository at:
 
+   
 
+  git://git.kernel.org/pub/scm/linux/kernel/git/nsekhar/linux-davinci.git 
tags/davinci-for-v5.6/soc
+   
 
+for you to fetch changes up to 5e06d19694a463a012c2589e29078196eb209448:   
 
+   
 
+  ARM: davinci: dm644x-evm: Add Fixed regulators needed for tlv320aic33 
(2020-01-13 17:36:26 +0530)
+   
 
+   
 
+DaVinci SoC updates for v5.6 include migrating DM365 SoC to use
 
+drivers/clocksource based driver for timer. This leads to removal  
 
+of machine specific timer driver.  
 
+   
 
+There are two patches adding missing fixed regulators for audio codecs 
 
+on DM365 and DM644x EVMs.  
 
+   
 
+   
 
+Bartosz Golaszewski (3):   
 
+  clocksource: davinci: only enable clockevents once tim34 is initialized  
 
+  ARM: davinci: dm365: switch to using the clocksource driver  
 
+  ARM: davinci: remove legacy timer support
 
+   
 
+Peter Ujfalusi (2):
 
+  ARM: davinci: dm365-evm: Add Fixed regulators needed for tlv320aic3101   
 
+  ARM: davinci: dm644x-evm: Add Fixed regulators needed for tlv320aic33
 
+   
 
+ arch/arm/mach-davinci/Makefile  |   3 +-  
 
+ arch/arm/mach-davinci/board-dm365-evm.c |  20 ++  
 
+ arch/arm/mach-davinci/board-dm644x-evm.c|  20 ++  
 
+ arch/arm/mach-davinci/devices-da8xx.c   |   1 -   
 
+ arch/arm/mach-davinci/devices.c |  19 --  
 
+ arch/arm/mach-davinci/dm365.c   |  22 +-  
 
+ arch/arm/mach-davinci/include/mach/common.h |  17 --  
 
+ arch/arm/mach-davinci/include/mach/time.h   |  33 ---   

[PATCH v2] Improve pull request URL matching regex

[Konstantin Ryabitsev]

When git-request-pull output is pasted into a mail client instead of
mailed directly, the ref part of the pull URL may end up wrapped to the
next line.

Example: 
https://lore.kernel.org/r/294422a4-37b2-def5-5d32-8988f27c3...@gmail.com/

This change properly parses URLs both with and without newlines.

Signed-off-by: Konstantin Ryabitsev 
---
 patchwork/parser.py   |  4 +-
 .../0023-git-pull-request-newline-in-url.mbox | 48 +++
 patchwork/tests/test_parser.py|  9 
 3 files changed, 59 insertions(+), 2 deletions(-)
 create mode 100644 
patchwork/tests/mail/0023-git-pull-request-newline-in-url.mbox

diff --git a/patchwork/parser.py b/patchwork/parser.py
index c794f09..d25c0df 100644
--- a/patchwork/parser.py
+++ b/patchwork/parser.py
@@ -939,11 +939,11 @@ def parse_patch(content):
 def parse_pull_request(content):
 git_re = re.compile(r'^The following changes since commit.*'
 r'^are available in the git repository at:\n'
-r'^\s*([\S]+://[^\n]+)$',
+
r'^\s*([\w+-]+(?:://|@)[\w/.@:~-]+[\s\\]*[\w/._-]*)\s*$',
 re.DOTALL | re.MULTILINE | re.IGNORECASE)
 match = git_re.search(content)
 if match:
-return match.group(1)
+return re.sub('\s+', ' ', match.group(1)).strip()
 return None
 
 
diff --git a/patchwork/tests/mail/0023-git-pull-request-newline-in-url.mbox 
b/patchwork/tests/mail/0023-git-pull-request-newline-in-url.mbox
new file mode 100644
index 000..74c29ce
--- /dev/null
+++ b/patchwork/tests/mail/0023-git-pull-request-newline-in-url.mbox
@@ -0,0 +1,48 @@
+From mboxrd@z Thu Jan  1 00:00:00 1970
+To: s...@kernel.org
+From: Matthias Brugger 
+Subject: [GIT PULL] soc: updates for v5.5
+Message-ID: <294422a4-37b2-def5-5d32-8988f27c3...@gmail.com>
+Date: Mon, 11 Nov 2019 13:23:51 +0100
+
+Hi Olof and Arnd,
+
+Please have a look on the following updates of drivers/soc for v5.5
+
+Thanks a lot,
+Matthias
+
+---
+
+The following changes since commit 54ecb8f7028c5eb3d740bb82b0f1d90f2df63c5c:
+
+  Linux 5.4-rc1 (2019-09-30 10:35:40 -0700)
+
+are available in the Git repository at:
+
+  https://git.kernel.org/pub/scm/linux/kernel/git/matthias.bgg/linux.git/
+tags/v5.4-next-soc
+
+for you to fetch changes up to 662c9d55c5ccb37f3920ecab9720f2ebf2a6ca18:
+
+  soc: mediatek: Refactor bus protection control (2019-11-07 10:11:04 +0100)
+
+
+refactor code of mtk-scpsys
+
+
+Weiyi Lu (5):
+  soc: mediatek: Refactor polling timeout and documentation
+  soc: mediatek: Refactor regulator control
+  soc: mediatek: Refactor clock control
+  soc: mediatek: Refactor sram control
+  soc: mediatek: Refactor bus protection control
+
+ drivers/soc/mediatek/mtk-scpsys.c | 214 ++
+ 1 file changed, 146 insertions(+), 68 deletions(-)
+
+___
+linux-arm-kernel mailing list
+linux-arm-ker...@lists.infradead.org
+http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
+
diff --git a/patchwork/tests/test_parser.py b/patchwork/tests/test_parser.py
index 85c6c52..0bf7158 100644
--- a/patchwork/tests/test_parser.py
+++ b/patchwork/tests/test_parser.py
@@ -630,6 +630,15 @@ class PatchParseTest(PatchTest):
 diff.startswith('diff --git a/arch/x86/include/asm/smp.h'),
 diff)
 
+def test_git_pull_newline_in_url(self):
+diff, message = self._find_content(
+'0023-git-pull-request-newline-in-url.mbox')
+pull_url = parse_pull_request(message)
+self.assertEqual(
+'https://git.kernel.org/pub/scm/linux/kernel/git/matthias.bgg/'
+'linux.git/ tags/v5.4-next-soc',
+pull_url)
+
 def test_git_rename(self):
 diff, _ = self._find_content('0008-git-rename.mbox')
 self.assertTrue(diff is not None)

base-commit: 239fbd2ca1bf140bc61fdee922944624b23c812c
-- 
2.17.1

___
Patchwork mailing list
Patchwork@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/patchwork

Re: DB-murdering API query (index suggestions needed)

[Konstantin Ryabitsev]

On Sat, Nov 16, 2019 at 12:48:33AM +1100, Daniel Axtens wrote:
> > GET 
> > /api/1.1/patches/?project=62&before=2019-11-01T00:00:00&per_page=100&page=6150
> >  
> >
> > The query behind this takes about 1 minute to run on a 20-core HT Xeon 
> > system and requires creating a huge temporary file (there are 18375 
> > patches in that project).
> 
> Ouch, I'm sorry to hear that.

Well, it's true that some of kernel.org's projects are large beyond what 
would be considered "sane". :)

(Another annoyance of mine is when someone requests a project but 
doesn't end up using it. We end up accumulating huge amounts of dead 
data over the years with no clear way of determining "is this useful to 
anyone?" Now that we have mailing list archives available in a 
public-inbox format, it's easier to justify deleting projects that are 
clearly not being used.)

> We have a bunch of db magic contributed by Stewart that will hit 2.2.
> 
> Stewart, do you happen to know if any of your magic will affect API
> queries? They're advertised as affecting the general listing of patches
> in the UI, I'm not sure if they also affect this.
> 
> If not, we can definitely have a look at getting an index or rate
> limiting/authentication thingy in for 2.2.

I'm not sure rate-limiting would have helped much in this particular 
case, because whoever was doing it was not trying to be malicious. There 
were only 2-3 queries running in parallel, but they were so heavy that 
it was enough to affect DB performance. Per-second rate-limiting is also 
easy enough to do on the upstream proxy, so I doubt it's worth the 
effort to develop this feature in the code.

In fact, I can probably do the same for unauthenticated API access, but 
it might be nicer if there's a simple toggle available in Patchwork's 
UI. I wouldn't consider this a high-priority feature.

Best regards,
-K
___
Patchwork mailing list
Patchwork@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/patchwork

DB-murdering API query (index suggestions needed)

[Konstantin Ryabitsev]

Hi, all:

Today, the DB behind patchwork.kernel.org was in a semi-permanent state 
of suffering due to someone trying to suck down all patches in the 
linux-arm-kernel project. This is what the API request looked like:

GET 
/api/1.1/patches/?project=62&before=2019-11-01T00:00:00&per_page=100&page=6150 

The query behind this takes about 1 minute to run on a 20-core HT Xeon 
system and requires creating a huge temporary file (there are 18375 
patches in that project).

So, two questions, really:

1. Any indexes we can put in place to make this query perform better?
2. Is there a way to disable anonymous API access?

-K
___
Patchwork mailing list
Patchwork@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/patchwork

[PATCH] Improve pull request URL matching regex

[Konstantin Ryabitsev]

Existing regex was missing several important use cases, such as:

- tag/branch info wrapping to the next line, e.g.:


are available in the Git repository at:

  https://git.kernel.org/pub/scm/linux/kernel/git/matthias.bgg/linux.git/
tags/v5.4-next-soc


(see example: https://patchwork.kernel.org/patch/11236893/)

- tag/branch info being wrapped to the next line with a backslash, e.g.:


are available in the Git repository at:

  https://git.kernel.org/pub/scm/linux/kernel/git/matthias.bgg/linux.git/ \
  tags/v5.4-next-soc


(no example, but I've seen this before)

The proposed change deals with these edge-cases.

Signed-off-by: Konstantin Ryabitsev 
---
 patchwork/parser.py | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/patchwork/parser.py b/patchwork/parser.py
index c794f09..d25c0df 100644
--- a/patchwork/parser.py
+++ b/patchwork/parser.py
@@ -939,11 +939,11 @@ def parse_patch(content):
 def parse_pull_request(content):
 git_re = re.compile(r'^The following changes since commit.*'
 r'^are available in the git repository at:\n'
-r'^\s*([\S]+://[^\n]+)$',
+
r'^\s*([\w+-]+(?:://|@)[\w/.@:~-]+[\s\\]*[\w/._-]*)\s*$',
 re.DOTALL | re.MULTILINE | re.IGNORECASE)
 match = git_re.search(content)
 if match:
-return match.group(1)
+return re.sub('\s+', ' ', match.group(1)).strip()
 return None
 
 

base-commit: 239fbd2ca1bf140bc61fdee922944624b23c812c
-- 
2.23.0

___
Patchwork mailing list
Patchwork@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/patchwork

Re: RFE: use patchwork to submit a patch

[Konstantin Ryabitsev]

On Fri, Nov 08, 2019 at 03:25:00PM +0100, Dmitry Vyukov wrote:
> > Would you like a mail sending account so you can use mail.kernel.org 
> > as
> > your outgoing smtp? We promise we won't do anything that nasty. :)
> >
> > -K
> 
> You know that we also serve Fuchsia, Akaros, FreeBSD, NetBSD, OpenBSD
> and potentially Windows and Darwin in the future, right? :)
> 
> How does that work? Currently we just call mail.Send:
> https://cloud.google.com/appengine/docs/standard/go/mail/reference
> and it magically does the thing. I can't even setup my own email
> client, so don't expect lots from me :)

I didn't realize you were using appengine, so I have no idea if using an 
external SMTP submission server is even feasible. I'm sure it can be 
done, but since you're serving all those other communities as well, that 
makes the offer of acting as a submission point a bit more tricky. :) 
Perhaps we can revisit this idea once you start publishing a feed?

-K
___
Patchwork mailing list
Patchwork@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/patchwork

Re: RFE: use patchwork to submit a patch

[Konstantin Ryabitsev]

On Fri, Nov 08, 2019 at 03:12:51PM +0100, Dmitry Vyukov wrote:
> Just to clarify: syzbot does not do this on purpose. It generates and
> hands off unwrapped lines. The "format=flowed; delsp=yes" part was
> done by somebody else.

Looks like vger received it already mangled:
https://lore.kernel.org/linux-usb/c460630596c1d...@google.com/raw

Would you like a mail sending account so you can use mail.kernel.org as 
your outgoing smtp? We promise we won't do anything that nasty. :)

-K
___
Patchwork mailing list
Patchwork@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/patchwork

Re: Deduplication of patchwork mail content?

[Konstantin Ryabitsev]

On Wed, Oct 09, 2019 at 05:35:03PM +1100, Daniel Axtens wrote:
> Hi sfr, jk, Konstantin and any other admins lurking,
> 
> I'm in the process of reworking the patchwork db schema to avoid one of
> our very big and very annoying (and slow) JOINs.
> 
> While I'm at it, it occurred to me that for both the ozlabs and
> kernel.org instances, there are a lot of mails that are sent across
> multiple projects. ATM the entire contents of the mail - content,
> headers, diff, what have you, will be stored in full for each project.
> 
> Would it be of value for your deployments if I used this opportunity to
> normalise the database and deduplicate emails? I was thinking of
> splitting the big raw text fields (diff, content, headers) into their
> own table and then indexing into that by message-id.

Daniel:

I think space is pretty cheap, and it's going to be a lot of work for
little savings. Adding some indexes would be a much more effective way
of improving performance in my view.

Best,
-K
___
Patchwork mailing list
Patchwork@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/patchwork

Re: [PATCH] api: support filtering patches by hash

[Konstantin Ryabitsev]

On Thu, Oct 24, 2019 at 01:33:42AM +1100, Daniel Axtens wrote:
> This is a feature that the XML-RPC API has, and which is used in
> the wild [1], so support it in the REST API.

Nice! Thank you for this.

> I tried to version the new filter field, but it's not at all clear
> how to do this with django-filters. The best way I could find
> requires manually manipulating request.GET, which seems to defeat
> the point of django-filters. So document it for 1.2, and have it
> work on older versions as an undocumented feature.
> 
> [1] 
> https://git.kernel.org/pub/scm/linux/kernel/git/mricon/korg-helpers.git/tree/git-patchwork-bot.py?id=104e7374e1be8458e6d2e82478625a7bf8c822ff
> 
> Cc: Konstantin Ryabitsev 
> Signed-off-by: Daniel Axtens 

Acked-by: Konstantin Ryabitsev 

-K
___
Patchwork mailing list
Patchwork@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/patchwork

Re: [PATCH 3/3] [PW3] Remove XML-RPC API

[Konstantin Ryabitsev]

On Fri, Oct 18, 2019 at 01:41:49PM +1100, Daniel Axtens wrote:

It's been deprecated since 2.0 with the new REST API. That API is
now pretty solid, and git-pw is good. Drop the old API.


I believe the only thing I was not able to do with the REST API is look 
up patches by patch hash. Is this limitation fixed with the 
current/upcoming REST API?


It's kind of a central feature of git-patchwork-bot, so I really want it 
preserved. :) I asked about it a year ago:

https://lists.ozlabs.org/pipermail/patchwork/2018-November/005699.html

-K
___
Patchwork mailing list
Patchwork@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/patchwork

Re: RFE: use patchwork to submit a patch

[Konstantin Ryabitsev]

On Tue, Oct 15, 2019 at 07:32:41PM +0300, Laurent Pinchart wrote:

Well, this is largely what GitGitGadget does
(https://gitgitgadget.github.io), and we could go that route, sure. I'm
reluctant only because, quoth:

  GitGitGadget itself is a GitHub App that is backed by an Azure
  Function written in pure Javascript which in turn triggers an Azure
  Pipeline written in Typescript (which is really easy to understand and
  write for everybody who knows even just a little Javascript),
  maintained at https://github.com/gitgitgadget/gitgitgadget.

I have zero familiarity with any of the above. That said, we do have a
bunch of CI engineers working at the LF, and I can probably avail myself
of their expertise if we decide to set this up.


I certainly wouldn't recommend a solution based on a proprietary
closed-source stack :-) But as we're talking about performing new
development for patchwork, I wanted to point out that we could also
consider a different technical approach that would involve new
development for a different open-source project. For instance, is the
above idea something that could be developed on top of gitolite ? Or
possibly even as a tiny standalone git server ?


I wouldn't do it on top of gitolite, because the administrative overhead 
would be too large. It certainly can be done as a separate service -- 
after all, any relation between this tool and patchwork is pretty 
tenuous.


That said, I'm leaning towards shelving this idea for the moment, at 
least as an official service provided by kernel.org -- it is easier to 
limit the scope at first and target maintainer tools and communication 
frameworks. It seems that sr.ht folks are already making something like 
this available, so we can just sit on our hands for a bit and see where 
this takes us.


-K
___
Patchwork mailing list
Patchwork@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/patchwork

Re: RFE: use patchwork to submit a patch

[Konstantin Ryabitsev]

On Sat, Oct 12, 2019 at 09:19:11AM +0200, Greg KH wrote:
This is basically why SMTP sucks in my view -- and it's worthless 
trying to

pick fights with IT departments, because they are told to do so by lawyers.
So, I want to take SMTP out of the equation:

1. provide a way for someone to submit a patch using a web interface   (but
still in a way that From: is their corporate ID)


If you do this, what happens when a maintainer/reviewer responds to that
patch and says "looks good, but can you change X and resend it?"

How will they get that message if it didn't go through their email
system?  How will they be able to respond to it?


The magic of git and email headers. Only the initial submission will go 
out of this web service -- the follow-up is expected to arrive into 
their mailbox.



2. use individual git feeds as a way to send out patches instead of   always
being secondary to SMTP


Sending patches that way is one thing, the interaction based on those
patches is another.

Everyone needs to remember that only 1/3 of the patches submitted are
applied.  The "normal" path of development is at least a review/resend
cycle for submissions (2/3 of patches).  So that 2/3 can't be ignored as
the "new/drive-by submissions" are probably more in that category than
not.


Right, the idea is that the imaginary tool that is backed by 
public-inbox will use multiple sources of content:


- the mailing list repository
- individual developer feeds
- the person's imap folder

Anything that shows up in the individual feeds should have a 
corresponding entry in the mailing list repository, but the latter is 
also cryptographically signed and therefore end-to-end attestable.  
Mailing lists and SMTP continue to be the fallback delivery method for 
the vast majority of the content.


I expect to lay this out in more detail as I prepare the "kthul MVP" 
roadplan.


-K
___
Patchwork mailing list
Patchwork@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/patchwork

Re: RFE: use patchwork to submit a patch

[Konstantin Ryabitsev]

On Mon, Oct 14, 2019 at 06:33:33PM +0300, Laurent Pinchart wrote:

If the goal is to work around SMTP-related technical issues, is a web UI
really the best way to go ? Wouldn't it be better to do the same through
a git push ? We could setup a git server that requires authentication,
and implement a push-to-email bridge. The information that would need to
be entered in a web UI could be put in a tag message, and we could have
a CLI to create the tag from a list of questions.


Well, this is largely what GitGitGadget does 
(https://gitgitgadget.github.io), and we could go that route, sure. I'm 
reluctant only because, quoth:


 GitGitGadget itself is a GitHub App that is backed by an Azure 
 Function written in pure Javascript which in turn triggers an Azure 
 Pipeline written in Typescript (which is really easy to understand and 
 write for everybody who knows even just a little Javascript), 
 maintained at https://github.com/gitgitgadget/gitgitgadget.


I have zero familiarity with any of the above. That said, we do have a 
bunch of CI engineers working at the LF, and I can probably avail myself 
of their expertise if we decide to set this up.


-K
___
Patchwork mailing list
Patchwork@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/patchwork

Re: RFE: use patchwork to submit a patch

[Konstantin Ryabitsev]

On Sun, Oct 13, 2019 at 11:39:30PM +, Eric Wong wrote:
Well, this is the bit where I say that it may not be allowed by 
corporate

rules. I see this all the time in CAF/Android world where companies
*require* that all email goes through their SMTP server so that it can be
properly logged (often for legal reasons). And it is often equally required
that any code submissions come from per...@corporate.com and not
per...@free-email-provider.com for License/CLA reasons, so setting up a
webmail server is not a solution either.


Aren't they still allowed to submit stuff via forges the same way
they'd use a potential hacker-oriented webmail/SMTP/IMAP solution?


You're trying to apply logic to legal requirements. :) It's actually 
fine for someone to submit a Github pull request using their corporate 
"From:", because it's not an action that sends email from their 
corporate address. Email is considered official company correspondence 
and therefore must be preserved in case it needs to be turned over 
during lawsuits (for fact-finding).



We can also find creative ways to subvert corporate policies:
For example; if their policy specifically prevents outgoing SMTP,
"git imap-send" could be used.


I'm generally against recommending solutions that can get people into 
legal trouble. If their company requires that all work correspondence is 
sent via the company's SMTP server, then we shouldn't be putting forth 
technical solutions defeating this.


-K
___
Patchwork mailing list
Patchwork@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/patchwork

Re: RFE: use patchwork to submit a patch

[Konstantin Ryabitsev]

On Tue, Oct 15, 2019 at 09:35:27AM -0400, Theodore Y. Ts'o wrote:

On Tue, Oct 15, 2019 at 08:37:41AM -0400, Steven Rostedt wrote:

Now, if we had a way to send and receive upstream patches via a web
site, that would actually make things easier.


Alas, for those corporations who choose to enable DMARC in hard-fail
mode (which includes google.com BTW, although thankfully, not
gmail.com) sending (from DMARC's perspective, "forging") e-mail from
someone at that particular domain won't work unless it comes from an
authorized e-mail server.


This is not a huge deal for web services mailing patches, because they 
just need to specify the `From:` header in the body of the patch. From 
`man git-am`:


 "From: " and "Subject: " lines starting the body override the 
 respective commit author name and title values taken from the headers.


So, a web service emailing patches can have:

 From j...@webserviceaddr.example.org
 From: J.Doe 
 Reply-to: J.Doe 
 Cc: j...@corpaddress.example.com
 Subject: [PATCH] Fix foo in bar

 From: J.Doe 

 Signed-off-by: J.Doe 
 ---

This *should* be kosher with legal requirements in most companies:

- the email traverses the corporate server when the cc is received, so 
 there is a log of it in their legally required auto-cc inbox

- all replies will be sent to the corporate server
- when the patch is applied, the commit will be properly listing the 
 corporate address



Of course, we do have a workaround for that, which is to use a
kernel.org address.


Only a very small subset of people have these, and it may still be not 
allowed by individual company's legal departments even when this is 
available.


-K
___
Patchwork mailing list
Patchwork@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/patchwork

Re: RFE: use patchwork to submit a patch

[Konstantin Ryabitsev]

On Mon, Oct 14, 2019 at 11:28:59AM -0300, Mauro Carvalho Chehab wrote:

Yeah, our current security model is based at the maintainer for
him to do his duties, properly reviewing the patch.

Yet, at the example that Daniel gave:

Instead of:

if ((permissions == allowed) && other_stuff) {
do_things();
}
do_more_stuff(permissions);

Patch was maliciously modified to:

if ((permission == allowed) && other_stuff) {
do_things();
}
do_more_stuff(permissions);

I suspect that a change like that might sleep though the maintainer's
review.


I submitted a proposal to the git list that would address this, but it 
saw little uptake in the discussion:


https://public-inbox.org/git/20190910121324.GA6867@pure.paranoia.local/t/#u

To summarize it in brief, I suggested using minisign signatures and 
git-tracked TOFU (trust on first use) databases. TOFU is not perfect, 
but it's better than nothing at all, which is what we have right now.


-K
___
Patchwork mailing list
Patchwork@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/patchwork

Re: RFE: use patchwork to submit a patch

[Konstantin Ryabitsev]

On Fri, Oct 11, 2019 at 09:23:08PM +, Eric Wong wrote:

(This is the same reason I generally disagree with Eric Wong about
preserving SMTP as the primary transmission protocol -- I've heard lots of
complaints both from kernel developers and especially from people trying to
contribute to CAF about corporate policies actually making it impossible to
submit patches -- and no, using a different mail server is not a possibility
for them because it can be a firing offense under their IT AUP rules.)


I'm not opposed to a webmail interface tailored to kernel hacking
which does stuff like checkpatch.pl and get_maintainer.pl before
sending (similar to your patchwork proposal and
gitgadgetgadget).  That would get around security appliances
but SMTP would still be used in the background.

Or offer full-blown HTTPS webmail + IMAP + SMTP access like any
other webmail provider + checkpatch + get_maintainer helpers.


Well, this is the bit where I say that it may not be allowed by 
corporate rules. I see this all the time in CAF/Android world where 
companies *require* that all email goes through their SMTP server so 
that it can be properly logged (often for legal reasons). And it is 
often equally required that any code submissions come from 
per...@corporate.com and not per...@free-email-provider.com for 
License/CLA reasons, so setting up a webmail server is not a solution 
either.


This is basically why SMTP sucks in my view -- and it's worthless trying 
to pick fights with IT departments, because they are told to do so by 
lawyers. So, I want to take SMTP out of the equation:


1. provide a way for someone to submit a patch using a web interface 
  (but still in a way that From: is their corporate ID)
2. use individual git feeds as a way to send out patches instead of 
  always being secondary to SMTP


-K
___
Patchwork mailing list
Patchwork@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/patchwork

Re: RFE: use patchwork to submit a patch

[Konstantin Ryabitsev]

On Fri, Oct 11, 2019 at 10:57:02AM +0200, Greg KH wrote:

So other than that minor thing, sounds interesting.  It's hard to
determine just how difficult the whole "set up git and send a patch out"
process is for people these days given the _huge_ numbers of new
contributions we keep getting, and the numerous good tutorials we have
created that spell out exactly how to do this.

So you might be "solving" a problem that we don't really have.  It's
hard to tell :(


It is interesting that there are split views on this. The main reason 
why I was thinking about it was because the topic came up a few times 
already. For example, in a conversation last year on ksummit-discuss:


https://lore.kernel.org/ksummit-discuss/ecadff3fd767c149ad96a924e7ea6eaf7c1ea...@usculxmsg01.am.sony.com/

Tim Bird mentioned that Sony developers couldn't send/receive patches 
because their corporate mail server rewrote all links to go through some 
kind of security appliance verification. If you read that thread, what 
we are discussing now is what I suggested we did then -- a web tool that 
could take corporate SMTP servers out of the equation.


(This is the same reason I generally disagree with Eric Wong about 
preserving SMTP as the primary transmission protocol -- I've heard lots 
of complaints both from kernel developers and especially from people 
trying to contribute to CAF about corporate policies actually making it 
impossible to submit patches -- and no, using a different mail server is 
not a possibility for them because it can be a firing offense under 
their IT AUP rules.)



I know this is a pretty big RFE, and I would like to hear your thoughts
about this. If there is general agreement that this is doable/good idea, I
may be able to come up with funding for this development as part of the
overall tooling improvement proposal.


The workflow seems sane, and matches what most people do today, with the
exception that it "solves" the git send-email issue, right?  Is that our
biggest barrier?


Well, I can't really speak from my extensive experience as a kernel 
developer, but I *have* submitted patches to Documentation/* before.  
This happens infrequently enough that I basically have to relearn the 
whole process from scratch, and it *is* a lot of steps. I can't fault 
people who are only familiar with the GitHub way of doing things when

they complain that this process is a challenge for them.

Not everyone submitting changes to the kernel are going to be highly 
skilled and comfortable with the terminal and command-line tools. They 
may be submitting a documentation fix, or it can be a driver developer 
who never leaves Visual Studio submitting a small bugfix so their driver 
works better in Linux.



I would recommend interviewing some of the recent kernel mentor project
and outreachy applicants first, to try to determine exactly what their
problems, if any, were with our development process.  If they say that
this type of tool/workflow would have saved them hours of time and
energy, then that's a great indication that we should try to do this.


I don't disagree and Shuah's comments are very valuable here. However, I 
would argue that these folks don't necessarily represent the target 
audience for this tool. They may be newbies, but they join these 
initiatives with the goal of spending significant time with the kernel 
and its code, so they don't mind the effort of learning the proper way 
of submitting patches.


I'm thinking of someone who needs to submit an occasional contribution 
once every six months and to whom this document is both long and 
daunting: 
https://www.kernel.org/doc/html/v4.17/process/submitting-patches.html.


-K



___
Patchwork mailing list
Patchwork@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/patchwork

Re: RFE: use patchwork to submit a patch

[Konstantin Ryabitsev]

On Fri, Oct 11, 2019 at 08:38:49AM +1100, Daniel Axtens wrote:

Hi Konstantin,

tl;dr: I think a git-to-email bridge is a good step. I'm not sure why
patchwork would be the thing to build it on top of, and I'm worried that
it would slow us both down. I'm very open to being convinced though.


In very broad terms, I chose patchwork because:

1. in people's minds patchwork.kernel.org already deals with patches and 
  mailing lists, so it seems to be a logical place for such tool to 
  live
2. it's already built on top of a powerful web system (django), and we 
  already have it up and running, so this wouldn't require setting up 
  Yet Another Web Framework Service



I will readily admit that both of these assertions are pretty tenuous.


If I understand correctly, you're using Patchwork as:


- user creates an account (which requires a mail confirmation)


a) an identity provider, and


Well, rather as a tool that already has account management which 
includes email confirmation at some point.



b) a way to integrate with existing concepts of a project and keep
   metadata about them in 1 place


Yes.


c) a handy tool for getting previous series by a given user.


Yes, it's convenient to already have that user's previously submitted 
series readily available.



d) a 'trusted' source of email.


Well, this part isn't really that important. Rather, it's a tool where, 
to have an account, one must confirm email delivery.



Is that right? I just ask because this idea seems a long way from what
Patchwork traditionally does. That's not necessarily bad, I just want to
make sure I understand, and that if you get funding you're not tying
yourself to a platform that doesn't suit your needs.


Well, in my mind patchwork:

1. already deals with patches and series, including knowing how to do 
  diff highlights and all the fancy stuff
2. will hopefully gain ability to do interdiffs in the future, so if 
  someone submits a series revision, they can see what actually changed 
  before their previous submission and their new attempt

3. already has a lot of knowledge around git, mboxes, formats, etc.

This, of course, is not to say that patchwork is where this *must* 
happen, but I think it would be *nice* if this is where this happens. :)



I'm particularly curious about Patchwork as (a) an identity
provider. You wrote:


- user creates an account (which requires a mail confirmation)


This seems like "optional centralising" on Patchwork - it becomes a
central identity provider but it's optional in that you can just send
email directly if you prefer.


Right, this is a tool to help people allergic to CLI (or who do this 
infrequently enough that they can never remember all the steps, and oh 
my god, why isn't there a web tool to hold my hand?).



If you're going to do 'lightweight' centralisation like that, why not do
it on a platform that already understands git? It's really easy to
extract the information you describe in (c) just by querying the
patchwork API. You don't need to actually integrate into patchwork, or
be logged in, to do that. You lose the ability to load any git remote,
but if you have a git remote that isn't github or gitlab, you probably
already have a good email flow (e.g. if you repo is on kernel.org).

If you really want to use Patchwork as an identity provider, rather than
a forge, could we just teach Patchwork how to be an identity broker, and
then build things separately, authenticating through Patchwork to
confirm a user's identity? That means you could build in whatever
language you like and, critically, run on whatever deployment schedule
you want. You could also get much better isolation that way, which would
be good - I don't want an RCE in the git library to allow someone to
wipe out all patchwork data, for example.


Well, ve hawe vays of prewenting that (e.g. by transitioning git calls 
into their own selinux domain which cannot talk to databases). 


I know this is a pretty big RFE, and I would like to hear your thoughts
about this. If there is general agreement that this is doable/good idea,
I may be able to come up with funding for this development as part of
the overall tooling improvement proposal.


As I said up top, I'm not opposed to this per se. I think a git-to-email
bridge is a good step. I'm just confused as to why patchwork would be
the thing to build it on top of, and I'm worried about how you'd deploy
and update this extended Patchwork. I'm very open to being convinced
though.


Generally, I think patchwork, as a web application that already deals 
with patches and series is a convenient place for this tool to live, 
that's basically the extent of my thinking. For sure, it can exist as a 
separate tool, but then I'd have to set up and maintain that separate 
tool in addition to patchwork, as opposed to just patchwork.


Best,
-K
___
Patchwork mailing list
Patchwork@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/patchwo

Re: RFE: use patchwork to submit a patch

[Konstantin Ryabitsev]

On Thu, Oct 10, 2019 at 03:07:29PM -0300, Mauro Carvalho Chehab wrote:

- the patch submission screen has a succession of screens:

  1. a screen with a single field allowing a user to paste a URL to
 their fork of the git repository.


This will raise the bar, as it will force all developers to have a
public site to host the tree. I guess only a fraction of the 4k kernel
devs have it... In special, the ones that just want to send us a patch
fixing a bug may have serious troubles implementing that.


I don't think this will raise the bar, as Github/Gitlab allow for very 
easy forking of https://github.com/torvalds/linux. This is also not at 
all aimed at "all developers" -- only those that don't want to use the 
current CLI workflow and are more comfortable with web tools like 
Github.


  2. next screen asks the user to select the ref to work from using 
 the

 list obtained from the remote. Once submitted, patchwork performs a
 `git clone --reference` to clone the repository locally using a
 local fork of the same repo to minimize object transfer. This part
 requires that:
   a. patchwork project is configured with a path to a local fork,
  if this feature is enabled for a project
   b. that fork is kept current via some mechanism outside of
  patchwork (e.g. with grokmirror)
   c. there is some sanity-checking during the clone process to
  avoid abuse (e.g. a sane timeout, a tmpdir with limited size,
  etc -- other suggestions welcome)


That would require a high bandwidth at the machine with as patchwork.
Also, doesn't sound a good idea to me, as the server may end by having
tons of open sections, most waiting for tens of minutes, in order to
complete git clone.


This is actually really fast if you already have a local copy of the 
repository with most objects. Try this yourself if you have 
torvalds/linux.git locally:


git clone --bare -s torvalds/linux.git test
cd test
git remote add arm-soc 
https://git.kernel.org/pub/scm/linux/kernel/git/arm/arm-soc
git fetch arm-soc for-next

The whole process takes a second or so and the resulting repo is 328K in 
size.


Of course, this assumes that the remote repository isn't trying to do 
something nasty, which is why I suggest anti-abuse precautions.


I know this is a pretty big RFE, and I would like to hear your 
thoughts

about this. If there is general agreement that this is doable/good idea,
I may be able to come up with funding for this development as part of
the overall tooling improvement proposal.


The procedure itself is not bad, but, if implemented, IMHO, it should,
instead, be something that will run at the machine of the one submitting
the patch. For instance, this could be a perl or python script inside
Kernel's ./script directory that would handle everything locally, and
then submit the patch via patchwork's REST API.


I think I didn't make clear that this isn't supposed to go straight to 
patchwork. Patchwork is merely a convenient tool where this happens -- 
the resulting patch gets mailed out to the mailing list just as the user 
would have done.


-K
___
Patchwork mailing list
Patchwork@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/patchwork

RFE: use patchwork to submit a patch

[Konstantin Ryabitsev]

Hi, all:

I would like to propose a new (large) feature to patchwork with the goal 
to make the process of submitting a patch easier for newbies and people 
generally less familiar with patch-based development. This was discussed 
previously on the workflows list: 
https://lore.kernel.org/workflows/20190930202451.GA14403@pure.paranoia.local/


How I envision this would work:

- user creates an account (which requires a mail confirmation)
- they choose a "submit patch" option from the menu
- the patch submission screen has a succession of screens:

 1. a screen with a single field allowing a user to paste a URL to 
their fork of the git repository. Once submitted, patchwork does a 
"git ls-remote" to attempt to get a list of refs and to verify that 
this is indeed a valid git repository


 2. next screen asks the user to select the ref to work from using the 
list obtained from the remote. Once submitted, patchwork performs a 
`git clone --reference` to clone the repository locally using a 
local fork of the same repo to minimize object transfer. This part 
requires that:
  a. patchwork project is configured with a path to a local fork, 
 if this feature is enabled for a project
  b. that fork is kept current via some mechanism outside of 
 patchwork (e.g. with grokmirror)
  c. there is some sanity-checking during the clone process to 
 avoid abuse (e.g. a sane timeout, a tmpdir with limited size, 
 etc -- other suggestions welcome)


 3. next screen asks the user to pick a starting commit from the log.  
Once submitted, patchwork generates the patch from the commit 
provided to the tip of the branch selected by the user earlier,

using git format-patch.

 4. next screen asks the user to review the patch to make sure this is 
what they want to submit. Once confirmed, patchwork performs two 
admin-defined optional hooks:


  a. a hook to generate a list of cc's (e.g. get_maintainer.pl)
  b. a sanity check hook (e.g. checkpatch.pl)

 5. if sanity checking is defined, next screen shows the output of the 
sanity check hook, asking confirmation to proceed.


 6. next screen shows the user three fields:

  a. title of the patch/series
  b. cover letter for the patch/series
  c. message-id of the previous patch revision (can be picked from 
 the list of previously submitted series by this user -- 
 patchwork should have them already)
  d. number of the revision (can be auto-filled if previous field 
 is provided) and other tags to include in []


 7. next screen shows final review of what would be sent out to the 
list (and cc's, if the hook to get cc's is defined and returned any 
results). Once submitted, patchwork sends the patch/series using 
patchwork's envelope-from and the user's own email in the From: 
header.


 8. once sent successfully, cleanups are performed (also needs to be 
done as part of the regular cron job, for any aborted attempts)


I know this is a pretty big RFE, and I would like to hear your thoughts 
about this. If there is general agreement that this is doable/good idea, 
I may be able to come up with funding for this development as part of 
the overall tooling improvement proposal.


Best regards,
-K
___
Patchwork mailing list
Patchwork@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/patchwork

Re: [PATCH] Fix issue with delegation of patch via REST API

[Konstantin Ryabitsev]

On Wed, 9 Oct 2019 at 12:58, Bjorn Helgaas  wrote:
>
> On Thu, Oct 3, 2019 at 4:22 PM Bjorn Helgaas  wrote:
> >
> > I can't get patchwork delegation via git-pw to work either on ozlabs
> > or kernel.org.  Any hints on where to look or more data to collect?
>
> This magically started working on patchwork.kernel.org.  I don't know
> what changed, but thank you to whoever fixed it!

The 2.1.5 upgrade was the magical change. :)

-K
___
Patchwork mailing list
Patchwork@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/patchwork

Re: Debugging dropped mails

[Konstantin Ryabitsev]

On Tue, Apr 09, 2019 at 08:20:19PM +0200, Petr Vorel wrote:
> any idea how can I debug dropped mails in patchwork instances?
> Few weeks ago I started to be unable to get patches to patchwork.ozlabs.org 
> and
> patchwork.kernel.org. Sometimes only gmail.com address is having troubles,
> sometimes both my email addresses (I cc my gmail address).
> 
> Of course, I'm registered to mailing lists, I see my mail in mailing list
> archives. Problems are on some kernel mailing lists, but on others as well
> (buildroot, LTP).

Can you give me a link to one of the messages that show up in mailing
list archives but aren't on patchwork.kernel.org?

-K
___
Patchwork mailing list
Patchwork@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/patchwork

RFE: Ability to retrieve only non-superseded series

[Konstantin Ryabitsev]

Hello:

It would be handy to have an ability on the REST API side of things to 
retrieve series containing patches matching only certain states, such 
that if all patches in series v1 are marked as "superseded", then it's 
filtered out of the returned results when querying /series/.


Unless I'm not reading the code right (not familiar with Django at all, 
unfortunately), I don't appear to have a way to apply any filtering to 
series at the moment.


Best,
-K
___
Patchwork mailing list
Patchwork@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/patchwork

Re: [PATCH v3] Move to msgid based URLs

[Konstantin Ryabitsev]

On Wed, Oct 31, 2018 at 01:06:56AM +1100, Daniel Axtens wrote:

Migrate our URL schema as follows:

Patches:   /project//patch//
Cover Letters: /project//cover//

The usual sub-resources (mbox, raw) hang off those URLs.
The old style URLs (/patch/NNN/*, /cover/NNN/*) redirect appropriately.

Also add /project//comment// which redirects you
in the same way /comment/NNN/ does.

Add /message/ as well. This does not require a project,
so if a mail has been sent to multiple projects, the project that you
will be redirected to is arbitrary.

This patch doesn't expose the new /message/ URL in the UI anywhere,
we can address that in a follow-up.

I also haven't attempted to do anything meaningful with series.

Our database still stores message ids as with angle brackets; we
just work around that rather than trying to migrate. That too can
come later if we think the pain is justified.

Partially-closes: #106
Reported-by: Konstantin Ryabitsev 
Reported-by: Linus Torvalds 
Reported-by: Stephen Finucane 
Signed-off-by: Daniel Axtens 

---
v1->v2: Move to msgid by default
   Add release note
   Add tests

Rebase: rebased on top of master. Tested bundles.
I'll apply it in the next couple of days in the absence of objection.


I'm very anxious to see it in place, so please consider me a captive 
cheerleader.


I'd also be very excited to see a new release with this and the database 
performance patches included -- perhaps a 2.2? Performance on projects 
with a long list of paginated patches is still rather slow on MySQL.


Best,
-K


signature.asc
Description: PGP signature
___
Patchwork mailing list
Patchwork@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/patchwork

RFC: allow rest api to filter patches by hash

[Konstantin Ryabitsev]

Hi, all:

I was looking into integrating git with some of the projects so that
applied patches can be auto-accepted and archived and I found out that
while the XMLRPC API knows how to look up a patch by its hash, the REST
API is not able to filter patches by hash. The main reason I wanted to
stick to the REST API is because it allows me to use an API token
instead of a username/password.

So, two questions, really -- is the goal to retain both APIs, or is the
REST API supposed to eventually fully replace the other? And if that is
the case, would it be possible to teach the REST API how to filter by
hash?

Thanks,
-K
___
Patchwork mailing list
Patchwork@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/patchwork

Re: [RFC PATCH] 3x improvement in patch listing

[Konstantin Ryabitsev]

On Thu, Aug 09, 2018 at 11:55:16AM +1000, Stewart Smith wrote:

Out of interest, does the kernel.org instance back onto PostgreSQL or
MySQL? 


We have one of each. The original instance patchwork.kernel.org is on 
MySQL (well, MariaDB), because that's the database cluster we have in 
our PDX DC. When we split the LKML project off to a VM in AWS 
(lore.kernel.org/patchwork), we set up a separate instance that's backed 
by the AWS PostgreSQL instance. However, that instance is for archival 
purposes only, and the one that's actually used for patch tracking and 
management is patchwork.kernel.org.



I have a bunch of ideas that would quite dramatically improve
performance on MySQL (properly using primary keys to force disk layout
to be sane, rather than the current screw-locality method that Django
enforces). On PostgreSQL, things are a bit different, but it's possible
an occasional ALTER TABLE CLUSTER BY (or whatever the syntax is) could
help a *lot*.


If you want to recreate a huge project, you can feed 20 years of LKML 
archives into it. :) 


I'm not sure that archiving the patches would help query performance at
all, but I'd have to check it out a bit. The query that Django generates
is certainly "interesting".


I know for a fact that it does, because I have empirical evidence to 
prove it. Go to lore.kernel.org/patchwork and try the patch listing with 
Archived=No and without it. There will be a few seconds difference.


-K


signature.asc
Description: PGP signature
___
Patchwork mailing list
Patchwork@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/patchwork

Re: Database consistency issues

[Konstantin Ryabitsev]

On Thu, Aug 09, 2018 at 02:01:11AM +1000, Daniel Axtens wrote:

In the case of Patchwork 1.1.3, I am more confused - the Comment model
has a foreign key to Patch here. So if you try to delete Patch you
should have got a referential integrity error like I did with
e.g. Events. (Patchwork pre 2.0 is a pain to test as it uses Vagrant
VMs, which use VirtualBox by default, which breaks SecureBoot, so I'm
not going to try to get it going tonight.)

Was there anything you deleted after or during the migration?


Daniel:

Hmm... Well, the migration was a bit hairy, so it's possible something 
did get missed. We migrated from 1.0, though, not from 1.1.3, so it's 
possible something got missed? If it helps, I can send the database 
schema that we currently have.


Best,
-K


signature.asc
Description: PGP signature
___
Patchwork mailing list
Patchwork@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/patchwork

Re: Database consistency issues

[Konstantin Ryabitsev]

On Thu, Aug 09, 2018 at 01:16:25AM +1000, Daniel Axtens wrote:

SELECT id, submission_id FROM patchwork_comment WHERE

submission_id=9896319;
+--+---+
| id   | submission_id |
+--+---+
| 20810589 |   9896319 |
+--+---+
1 row in set (0.00 sec)

MariaDB [KORG_PATCHWORK]> SELECT * FROM patchwork_submission WHERE
id=9896319;
Empty set (0.00 sec)


Right, well this is definitely the most terrifying start to any email I
have received since starting as one of the patchwork maintainers!


Haha, sorry, I should have prefixed this with "I'm not seeing any data 
loss." :)



I will prepare a patch that adds the foreign keys.


Thanks! I know they are in place for the pgsql side of things, since the 
lore.kernel.org/patchwork instance is using postgresql. So, it was 
either a difference between db implementations, or a side-effect of 
migration.


Best,
-K


signature.asc
Description: PGP signature
___
Patchwork mailing list
Patchwork@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/patchwork

Re: [RFC PATCH] 3x improvement in patch listing

[Konstantin Ryabitsev]

On Wed, Aug 08, 2018 at 05:40:06PM +1000, Stewart Smith wrote:

There's two main bits that are really expensive when composing the list
of patches for a project: the query getting the list, and the query
finding the series for each patch.


Stewart:

Thanks for working on this! Do you think this would help with pagination 
as well? I find that in semi-abandoned projects like 
https://patchwork.kernel.org/project/qemu-devel/list/ it takes a few 
seconds to load the list view due to 800+ pages of unprocessed patches.  
I am currently considering an automated script that would auto-archive 
patches older than 6 months, but if simply improving pagination times 
would fix the issue, then I wouldn't need to bother.


Best,
-K


signature.asc
Description: PGP signature
___
Patchwork mailing list
Patchwork@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/patchwork

Database consistency issues

[Konstantin Ryabitsev]

Hi, all:

I'm trying to figure out how this can possibly happen:

> SELECT id, submission_id FROM patchwork_comment WHERE
submission_id=9896319;
+--+---+
| id   | submission_id |
+--+---+
| 20810589 |   9896319 |
+--+---+
1 row in set (0.00 sec)

MariaDB [KORG_PATCHWORK]> SELECT * FROM patchwork_submission WHERE
id=9896319;
Empty set (0.00 sec)

There don't appear to be any foreign key constraints on the
patchwork_comment table -- would that be the side-effect of migration?


Regards,
-- 
Konstantin Ryabitsev
Director, IT Infrastructure Security
The Linux Foundation



signature.asc
Description: OpenPGP digital signature
___
Patchwork mailing list
Patchwork@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/patchwork

RFE: permalink to submissions using list-id/message-id (GH issue 106)

[Konstantin Ryabitsev]

Hi, all:

I'm largely duplicating what I wrote in the GitHub issue 106 
(https://github.com/getpatchwork/patchwork/issues/106) just to reiterate 
a personal plea. :)


One of the side-effects of moving the LKML project to a dedicated 
instance was the fact that I discovered just how many people were using 
patchwork.kernel.org links in various places. Arguably, this is the 
wrong way to use patchwork, since it's supposed to remain entirely 
behind the scenes, but this is obviously a losing battle, considering 
how many hits the following query returns:


https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/log/?qt=grep&q=patchwork.kernel.org

Android and ChromiumOS folks are also using it:

https://chromium-review.googlesource.com/c/chromiumos/third_party/kernel/+/1092792

Since all patches have different IDs on the new instance 
(lore.kernel.org/patchwork), all these URLs were now broken. I fixed 
this for the moment by creating a simple dict matching of oldid -> newid 
mapped using the message-id of each submission, and added the following 
nasty hack to the running code:


diff --git a/patchwork/views/patch.py b/patchwork/views/patch.py
index eccb544..242da7b 100644
--- a/patchwork/views/patch.py
+++ b/patchwork/views/patch.py
@@ -37,6 +37,13 @@ from patchwork.views import generic_list
from patchwork.views.utils import patch_to_mbox
from patchwork.views.utils import series_patch_to_mbox

+try:
+with open('/opt/patchwork/idmap.pickle', 'rb') as idp:
+import pickle
+lkml_idmap = pickle.load(idp)
+idp.close()
+except:
+lkml_idmap = {}

def patch_list(request, project_id):
project = get_object_or_404(Project, linkname=project_id)
@@ -58,6 +65,9 @@ def patch_detail(request, patch_id):
if submissions:
return HttpResponseRedirect(
reverse('cover-detail', kwargs={'cover_id': patch_id}))
+elif int(patch_id) in lkml_idmap:
+newurl = 'https://lore.kernel.org/patchwork/patch/%s/' % 
lkml_idmap[int(patch_id)]
+return HttpResponseRedirect(newurl)
raise exc

editable = patch.is_editable(request.user)


This does the job for the moment, though I probably shouldn't be keeping 
the mapping dict in RAM at all times, even if it *is* just a bunch of 
ints.


Anyway, my request is that, since people can't be convinced to stop 
using patchwork links, that we provide them with a convenient way to 
obtain permalinks in a format that doesn't rely on patch-id. I suggest:


sitename/p/list-id/message-id

e.g.:

patchwork.kernel.org/p/linux-kernel.vger.kernel.org/20170619152033.893467...@linuxfoundation.org

Perhaps add a [permalink] button to the same row currently showing 
[patch-id | diff | mbox | series ].


This should contain all the necessary info for patchwork to rewrite this 
into a patch view. If the list-id changes, as they sometimes do, the 
site administrator can easily add a one-line rewrite rule that would 
preserve link validity.


This has two added benefits over the status quo:

1. Moving projects between databases becomes possible
2. More importantly, this allows future code historians to still be able 
to get context about a commit even if the patchwork service is no longer 
operating. If they can see both the list-id and the message-id, that 
should give them enough information to locate the conversation in 
mailing list archives.


-K


signature.asc
Description: PGP signature
___
Patchwork mailing list
Patchwork@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/patchwork

Postmortem upgrading patchwork.kernel.org 1.0->2.1

[Konstantin Ryabitsev]

Hello, patchwork list!

About a week ago I performed the upgrade of patchwork.kernel.org from 
version 1.0 to version 2.1. This is my war story, since it didn't go 
nearly as smooth as I was hoping.


First, general notes to describe the infra:

- patchwork.kernel.org consists of 74 projects and, prior to the 
 migration, 1,755,019 patches (according to count(*) in the 
 patchwork_patch table).

- of these, 750,081 patches are from one single project (LKML).
- the database is MySQL, in a master-slave failover configuration, 
 accessed using haproxy


The LKML project was mostly dead weight, since nobody was actually using 
it for tracking patches. We ended up creating a separate patchwork 
instance just for LKML, located here: https://lore.kernel.org/patchwork.  
The migration, therefore, included two overall steps:


1. delete the LKML project from patchwork.kernel.org, dumping nearly 
half of db entries.

2. migrate the remainder to patchwork 2.1

# Problems during the first stage

Attempt to delete the LKML project via the admin interface failed.  
Clicking the "Delete" button on the project settings page basically 
consumed all the RAM on the patchwork system and OOM-ed in most horrible 
ways, requiring a system reboot. In an attempt to solve it, I manually 
deleted all patches from patchwork_patch that belonged to the LKML 
project. This allowed me to access the "Delete" page and delete the 
project, though this also resulted in a corrupted session, because my 
admin profile ended up corrupted. The uwsgi log showed this error:


Traceback (most recent call last):
 File 
"/opt/patchwork/venv/lib/python2.7/site-packages/django/core/handlers/base.py", 
line 132, in get_response
   response = wrapped_callback(request, *callback_args, **callback_kwargs)
 File "./patchwork/views/patch.py", line 106, in list
   view_args = {'project_id': project.linkname})
 File "./patchwork/views/__init__.py", line 59, in generic_list
   if project.is_editable(user):
 File "./patchwork/models.py", line 69, in is_editable
   return self in user.profile.maintainer_projects.all()
 File 
"/opt/patchwork/venv/lib/python2.7/site-packages/django/utils/functional.py", 
line 226, in inner
   return func(self._wrapped, *args)
 File 
"/opt/patchwork/venv/lib/python2.7/site-packages/django/db/models/fields/related.py",
 line 483, in __get__
   self.related.get_accessor_name()
RelatedObjectDoesNotExist: User has no profile.

I was able to create another admin user and continue.

# Problems during the second stage

At this stage, I started the migration process using manage.py migrate.  
Immediately, this resulted in a problem due to haproxy inactivity 
timeouts. As I mentioned, our setup uses a master-slave setup, and tcp 
connections are configured to time out after 5 minutes of inactivity.  
The migration script was doing some serious database modifications 
operating on tables with about a million or more rows, which took MUCH 
longer than our 5-minute timeout setting.


After setting things up to connect directly to the master, bypassing 
haproxy, I was able to proceed to the next step. Unfortunately, I didn't 
get very far, since at this point migration routines were failing 
because they were trying to lock millions of rows and running out of 
database resources. Unfortunately, I could not easily fix this because 
raising maximum locks would have required restarting the database server 
(an operation that affects multiple projects using it). I had to look in 
the django migration scripts and run mysql queries manually, adding 
WHERE clauses so that they would operate on subsets of rows (limiting by 
id ranges). This took a few hours -- to some degree because all 
operations had to be replicated to the slave server. Some of the tables 
it operated on were tens of gigabytes in size, so shipping all these 
replication logs to the slave server also took a lot of resources and 
resulted in lots of network and disk IO.


In the end, it mostly worked out, despite the somewhat gruelling 
process. I do have a somewhat mysterious side-effect of deleting the 
LKML project in that some people lost maintainer status in other 
projects. I'm not sure how this came to be, but at least it's an easy 
fix -- probably the same reason my admin profile got purged requiring me 
to create a new one.


Needless to say, I hope future upgrades are a lot more smooth. Did I 
test the migration before starting it? Yes, but on a very small subset 
of test data, which is why I didn't hit any of the above conditions, 
which stemmed from a) deleting a project and b) copying, merging and 
deleting huge datasets as the backend db format got rejigged between 
version 1.0 and 2.1.


My suggestion for anyone performing similar migrations from 1.0 to 2.1 
-- if you have more than a few thousand patches -- is to perform this 
migration on a standalone database and then dump and re-import the 
result into production instead of performing this live on multi-master 
or mas

Re: RFC: Access patch by message-ID

[Konstantin Ryabitsev]

On Tue, May 29, 2018 at 12:37:11PM +1000, Daniel Axtens wrote:

Seems like it would be easy to implement a way to access a patch by
Message-ID, but there is no such option currently. I would very much
like to have something like that, which would allow us to link from a
public-inbox archive directly to the patch within patchwork, e.g.:

https://patchwork.kernel.org/mid/foobarbaz-message...@gmail.com
->
https://patchwork.kernel.org/patch/12345/


This sounds good to me. You'd also have to specify a project, because we
allow the same MID to be present in multiple projects. I hope that won't
be too much of a problem?


It isn't in my particular use-case, but you can also default to a more
general query if project id is not specified, e.g.:

SELECT id FROM patchwork_patch WHERE msgid=? ORDER BY project LIMIT 1

This will select the project with the lowest ID, which usually
corresponds to the earliest defined project.

Best,
-K
___
Patchwork mailing list
Patchwork@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/patchwork

RFC: Access patch by message-ID

[Konstantin Ryabitsev]

Hello, all:

Seems like it would be easy to implement a way to access a patch by
Message-ID, but there is no such option currently. I would very much
like to have something like that, which would allow us to link from a
public-inbox archive directly to the patch within patchwork, e.g.:

https://patchwork.kernel.org/mid/foobarbaz-message...@gmail.com
->
https://patchwork.kernel.org/patch/12345/

I recall doing that at some point at lkml.kernel.org using this SQL query:

SELECT id FROM patchwork_patch WHERE msgid=?

So, I imagine it would be 3 lines in patchwork to support this feature. :)

Best,
-- 
Konstantin Ryabitsev
Director, IT Infrastructure Security
The Linux Foundation



signature.asc
Description: OpenPGP digital signature
___
Patchwork mailing list
Patchwork@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/patchwork

Re: Is it possible to change Patchwork username?

[Konstantin Ryabitsev]

On 2017-12-14 04:20 PM, Cengiz Can wrote:
> Hello
> 
> I've registered an account in patchwork.kernel.org with a different
> spelling of my username.
> 
> I would like to know if we can request a rename from someone with privileges.
> 
> Deletion and recreation of the account is also acceptable since I
> don't have any valuable
> data within my account yet.

Please email helpd...@kernel.org.

Best,
-- 
Konstantin Ryabitsev
Director, IT Infrastructure Security
The Linux Foundation
___
Patchwork mailing list
Patchwork@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/patchwork

Patchwork for (legal) public patch archive

[Konstantin Ryabitsev]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256

Hi, all:

I have a slightly off-topic question -- one of our projects requires
that all patches sent via a mailing list are archived on a public
website (just specifically patches, not any other discussions). Before
we set up patchwork with no project admins (basically) -- is there any
other tools you may be aware of that are able to receive mailing list
traffic and create a static archive of any patches seen?

Best regards,
- -- 
Konstantin Ryabitsev
Linux Foundation Collab Projects
Montréal, Québec
-BEGIN PGP SIGNATURE-
Version: GnuPG v2

iQIcBAEBCAAGBQJWnnBGAAoJEDS6uAr58ke4yHsP/AxynDwQ8qZ+57hqR0YVVPQI
iyU0Zt4DoZqqjcRs+VDksjYEjbFqQotlhsVUT2I0k1YlbfdIsxfBzKCzvdUoAHrZ
sfoja1iLM5rN7mFymiu2PbIMP+NGH13M/1UYr9qBHGbTB+m/hyL/QRPh7dNp2nbn
mOJP7OK7ctioIEurQzsSkKWIFO8D9w3ev+6Pbe6wxsZZ+UpA4t4UiqZjMcRJUVED
oMrPJbQOxY1vWMTwnLoYrfMd7dYsLcVk+nlLTBLTJypxXgajdnYsN+7qWi00TDCf
SkSK+GTQ7tsf21Ja4SIDj+dAT+urMtHqCA55CJby53jhYVp7yAqv22iCVTw19ud0
Q4CVEPQap8NlNQrRWh6jSd42hR1xbV7Z+y06vjkyuGFG/uiYzhL3xDzPYtHB5Q/j
iFRIxcTyv2yowH0x3J9MF8OLv/SvFkGakE4O6PaD/oC41RqVR9vBilQ4vWLtMGoI
KX/2NkU/UO/wK0k5hyRC2VDj0hTd8w4Z6AsNimFqXM8QaR64aeZqqZX8gIG8zp37
fY38Scz2gNdOQ3ZINJbx3F28haBBGZ6GeXvPf1CH7GO/Cw1//wC2YDK8zSvb4RTM
BY01Ku7RyTcddBnN8S5nj7ifCS4mIxlEd32yr5qKTPXwm6WzYx3X0V8wzVH5Gcyx
xctaPfn48vXd58JZPgHZ
=MhRk
-END PGP SIGNATURE-
___
Patchwork mailing list
Patchwork@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/patchwork

[PATCH] Add a config option to FORCE_HTTPS_LINKS

[Konstantin Ryabitsev]

In situations where SSL is terminated at the load-balancer, we cannot
rely on guessing the scheme based on whether patchwork itself was
accessed via http or https, since the last-leg is always going to be
done over http.

Unfortunately, wrongly using http:// URLs results in unusable
.pwclientrc files, since xmlrpc does not handle http->https redirects
and instead displays a traceback.

This change introduces a FORCE_HTTPS_LINKS option, which forces
pwclientrc links to always return "https" regardless of how the project
itself is accessed.

It appears that the http/https check is currently only used for
generating pwclientrc -- a lot of other places seem to hardcode
"http://"; and rely on the server to transparently upgrade the
connection. This is not a secure approach (it allows for MITM and
SSL-Strip attacks) and therefore all places currently hardcoding
http://{{site.domain}} and similar should be switched to using the
"sheme" variable, the same as done for generating pwclientrc files.
---
 apps/patchwork/views/base.py | 2 +-
 apps/settings.py | 5 +
 2 files changed, 6 insertions(+), 1 deletion(-)

diff --git a/apps/patchwork/views/base.py b/apps/patchwork/views/base.py
index 82c0368..6b54a9b 100644
--- a/apps/patchwork/views/base.py
+++ b/apps/patchwork/views/base.py
@@ -42,7 +42,7 @@ def pwclientrc(request, project_id):
 project = get_object_or_404(Project, linkname = project_id)
 context = PatchworkRequestContext(request)
 context.project = project
-if request.is_secure():
+if settings.FORCE_HTTPS_LINKS or request.is_secure():
 context['scheme'] = 'https'
 else:
 context['scheme'] = 'http'
diff --git a/apps/settings.py b/apps/settings.py
index 537c380..43a37d8 100644
--- a/apps/settings.py
+++ b/apps/settings.py
@@ -114,6 +114,11 @@ ENABLE_XMLRPC = False
 # of patchwork
 COMPAT_REDIR = True
 
+# Set to True to always generate https:// links instead of guessing
+# the scheme based on current access. This is useful if SSL protocol
+# is terminated upstream of the server (e.g. at the load balancer)
+FORCE_HTTPS_LINKS = False
+
 try:
 from local_settings import *
 except ImportError, ex:
-- 
1.8.3.1

___
Patchwork mailing list
Patchwork@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/patchwork

[PATCH] Don't use total_seconds for python < 2.7

[Konstantin Ryabitsev]

The total_seconds function was added to datetime in python-2.7. For
compatibility with previous versions of python, use its suggested
equivalent (except drop microseconds, since we don't care about them in
this context).

Signed-off-by: Konstantin Ryabitsev 
---
 apps/patchwork/views/__init__.py | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/apps/patchwork/views/__init__.py b/apps/patchwork/views/__init__.py
index 681532a..a823388 100644
--- a/apps/patchwork/views/__init__.py
+++ b/apps/patchwork/views/__init__.py
@@ -190,8 +190,8 @@ def patch_to_mbox(patch):
 if patch.content:
 body += '\n' + patch.content
 
-utc_timestamp = (patch.date -
-datetime.datetime.utcfromtimestamp(0)).total_seconds()
+delta = patch.date - datetime.datetime.utcfromtimestamp(0)
+utc_timestamp = delta.seconds + delta.days*24*3600
 
 mail = PatchMbox(body)
 mail['Subject'] = patch.name
-- 
1.8.1.4

___
Patchwork mailing list
Patchwork@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/patchwork

Re: [git pull] Please pull powerpc.git merge branch

[Konstantin Ryabitsev]

On 10/06/13 12:20 AM, Jeremy Kerr wrote:
> Hi Linus,
> 
>> No. The date from the email was
>>
>> Date: Fri, 7 Jun 2013 15:42:54 +1000
>>
>> and we want *that* date.
> 
> Ah, gotchya.
> 
> So, we now use the original date header (if present) in the mbox views:
> 
> $ wget -qO - http://patchwork.ozlabs.org/patch/249598/mbox/ | grep ^Date
> Date: Fri, 7 Jun 2013 15:42:54 +1000
> 
> ... for all your data-mining needs.

Looks like this functionality uses datetime.total_seconds(), which
doesn't exist on python < 2.7. Can this be changed to make it work with
python-2.6, so patchwork can be served on RHEL 6?

> 67181f5c (Jeremy Kerr   2013-06-10 11:37:25 +0800 193) utc_timestamp 
> = (patch.date -
> 67181f5c (Jeremy Kerr   2013-06-10 11:37:25 +0800 194)     
> datetime.datetime.utcfromtimestamp(0)).total_seconds()


Best,
-- 
Konstantin Ryabitsev
Senior Systems Administrator
Linux Foundation Collab Projects
Montréal, Québec



signature.asc
Description: OpenPGP digital signature
___
Patchwork mailing list
Patchwork@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/patchwork