Re: Why Operating Systems don't always upgrade GnuPG [was: Re: How can we utilize latest GPG from RPM repository?]

2018-02-20 Thread Daniel Kahn Gillmor 
On Tue 2018-02-20 13:18:40 +0100, Dashamir Hoxha wrote:
> One solution to this situation may be to install the latest GnuPG
> in a Docker container, where it can have all the required libraries
> and dependencies that it needs, without disturbing the host OS.

I think this misses the point that it's not just *what does gnupg depend
on* but it's also *what depends on gnupg*.  The dependencies work in
both directions.

> Another solution may be to use a "snap", which is a kind of new
> software packaging invented by Ubuntu:

The basic idea behind "snap" and "flatpak" and other similar tools is
what many people call "bundling" or "vendoring" -- you ship the program
together with all its dependencies, regardless of what dependencies are
on the host system.  it's not a new idea at all, and is quite common on
many platforms, including in some flavors of cowboy web development.

As with docker containsers, this approach doesn't address the other
direction of the dependency graph.  In addition, all of these approaches
have maintenance costs and open questions about responsibility.  if
every app ships with its own bundled copy of libfoo, and a flaw is found
in libfoo, then it needs to be fixed.  can you be sure you've found and
fixed all copies?  Who is responsible for fixing each specific copy?  Do
those maintainers have enough time/attention/living expenses to make
sure vulerabilities and software flaws get patched in all of their
dependencies?  are they willing to re-ship the entire bundle/snap/docker
image for each dependency that needs an upgrade?

I recently heard bundling/vendoring/snaps/docker containers
characterized in the following way, which resonated with me:

Hm, maintaining a complex operating system is hard.  I know, we can
fix that by trying to maintain 100 complex operating systems
instead!

To be clear, i believe that there are contexts where bundling is
actually the right approach.  But it is not an obvious win to me in most
cases.

Regards,

--dkg


signature.asc
Description: PGP signature
___
Gnupg-users mailing list
Gnupg-users@gnupg.org
http://lists.gnupg.org/mailman/listinfo/gnupg-users

Re: Why Operating Systems don't always upgrade GnuPG [was: Re: How can we utilize latest GPG from RPM repository?]

2018-02-20 Thread Ben McGinnes 
On Mon, Feb 19, 2018 at 10:45:52AM -0800, Daniel Kahn Gillmor wrote:
> 
> How can GnuPG contribute to fixing this problem?  The traditional way
> that many other projects have taken is to define their core programmatic
> functionality into a library with a strict interface guarantees, and
> have explicitly deprecated other use.  The closest that GnuPG comes to
> this technique is GPGME, which is not feature-complete (as compared to
> the gpg executable), and has its own history of both difficult upgrades
> and unclear/surprising semantics.

True, but what it does have available still covers a *lot* of what
people do mainly want to access programmatically.  My trusty key
counter is now done that way (it takes a while to run, but that's due
to the size of the keybox, not due to GPGME).

> It also doesn't have bindings in many popular programming languages.

There is a cunning plan which may provide a bridge to at least in part
address that.

> When programmers in those language want to use GnuPG, their shortest
> path to "something that works" often involves shelling out to gpg,
> rather than binding to GPGME. :/

Yes

> Another thing that would help would be to explicitly and succinctly
> document the preferred ways of interacting with GnuPG in ways that other
> developers find useful.

That would be brilliant, but it requires convincing developers to not
simply document their work, but to actually step back from their
focussed POV and abstract what they're intending to do so we can
determine what will actually be useful to procide all developers.
Otherwise we'll just end up with a bunch of feature requests specific
to the problem each developer happens to be working on at the time
they notify us.

If you can work out a way to get engineers to understand why that's
important ... you'll become very rich shortly thereafter.  ;)

> Perhaps GnuPG could also itself try to detect when it is being used
> programmatically in an unstable way and produce warnings?

Most of the worst of those examples are in bash, how're you going to
tell the difference between that and a user?

> Yet another complementary approach might be to aggressively police
> the ecosystem by finding other software that deends on GnuPG in any
> of the aforementioned brittle ways, and either ask those developers
> to stop using GnuPG entirely, or to provide them with stable,
> well-supported ways to do what they're looking to do.

And people accuse me of being combative!  Wow ... you're brave ...

Moreseriously, though, let's besure we've got an alternative method to
replace whatever their pet project's method is first.

Arguably gpgme-tool might already do that, but we'll see.

> I welcome discussion/suggestions on how we can improve this situation,
> and i *definitely* welcome help in doing the kind of
> ecosystem-perspective work necessary to make it easier to maintain an
> up-to-date branch of GnuPG.

I'm already laying the groundwork on the API side of things, as well
as working on doumentation.

> But shrugging and suggesting it's uncontroversial to upgrade arbitrary
> machines to the latest version of GnuPG doesn't appreciate the scope of
> the problem involved with software maintenance in an active and
> interdependent ecosystem.

Indeed.  I didn't check your buglist ... but I did recall that one of
the dependencies of GMIME is GPGME.  Pretty sure that'll affect a few
packages and libgcrypt & libgpg-error are a dependencies for lots things.

It's exactly this sort of thing which led to the popularity of
installing things the users wanted in /usr/local or /opt or wherever
specifically to prevent the end user use case adversely affecting the
system's use case.


Regards,
Ben


signature.asc
Description: PGP signature
___
Gnupg-users mailing list
Gnupg-users@gnupg.org
http://lists.gnupg.org/mailman/listinfo/gnupg-users

Re: Why Operating Systems don't always upgrade GnuPG [was: Re: How can we utilize latest GPG from RPM repository?]

2018-02-20 Thread Kristian Fiskerstrand 
On 02/20/2018 01:18 PM, Dashamir Hoxha wrote:
> If anybody is willing to give a try to any of these solutions I would
> like to help.

I would be generally cautious for both approaches without proper support
in the surrounding infrastructure. In particular an upgrade to a
depending library would need to automatically cause a rebuild of the
container in the case of a security upgrade when such embedding happens,
which is generally a bad thing unless you have a large scale deployment
and defined QA processes for terminating and replacing containers with
new deployments regularly.

-- 

Kristian Fiskerstrand
Blog: https://blog.sumptuouscapital.com
Twitter: @krifisk

Public OpenPGP keyblock at hkp://pool.sks-keyservers.net
fpr:94CB AFDD 3034 5109 5618 35AA 0B7F 8B60 E3ED FAE3

Manus manum lavat
One hand washes the other



signature.asc
Description: OpenPGP digital signature
___
Gnupg-users mailing list
Gnupg-users@gnupg.org
http://lists.gnupg.org/mailman/listinfo/gnupg-users

Re: Why Operating Systems don't always upgrade GnuPG [was: Re: How can we utilize latest GPG from RPM repository?]

2018-02-20 Thread Dashamir Hoxha 
On Mon, Feb 19, 2018 at 7:45 PM, Daniel Kahn Gillmor 
wrote:

> On Sat 2018-02-17 17:06:54 -0600, helices wrote:
> > I will probably never understand why wanting to run the most current
> > version of gnupg on a plethora of servers is controversial.
>
> Here's one last try to explain the situation.
>
> GnuPG (and the libraries it depends on) are used by (aka "depended on
> by") other libraries and tools, both those integrated into the operating
> system itself, and those that might be externally installed.  Some of
> these dependencies are "brittle".
>

One solution to this situation may be to install the latest GnuPG
in a Docker container, where it can have all the required libraries
and dependencies that it needs, without disturbing the host OS.

But I am aware that this may present some challenges for  normal
usage and may not be suitable except for testing.

Another solution may be to use a "snap", which is a kind of new
software packaging invented by Ubuntu:
 - https://snapcraft.io/
 - https://docs.snapcraft.io/snaps/intro
The idea is that a software is shipped with all the dependencies,
so it does not matter in which OS it is installed, it will always work.

I don't know the details of snaps. Since it is a "containerized software
package" maybe it is not much different from the docker solution
above and maybe has the same challenges/problems.

If anybody is willing to give a try to any of these solutions I would like
to help.

Regards,
Dashamir
___
Gnupg-users mailing list
Gnupg-users@gnupg.org
http://lists.gnupg.org/mailman/listinfo/gnupg-users

Re: Why Operating Systems don't always upgrade GnuPG [was: Re: How can we utilize latest GPG from RPM repository?]

2018-02-19 Thread MFPA 
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA512

Hi


On Monday 19 February 2018 at 8:51:08 PM, in a message with no id,
ed...@pettijohn-web.com wrote:-


> I think gpgme is the answer here as well. If you mean
> specifically
> a python interface to gpgme then it's probably up to
> a python developer to get that rolling.

Python bindings are among those maintained within the GPGME
repository.

See https://wiki.gnupg.org/APIs



- --
Best regards

MFPA  

After all is said and done, a lot more will be said than done.
-BEGIN PGP SIGNATURE-

iNUEARYKAH0WIQSWDIYo1ZL/jN6LsL/g4t7h1sju+gUCWotpwF8UgAAuAChp
c3N1ZXItZnByQG5vdGF0aW9ucy5vcGVucGdwLmZpZnRoaG9yc2VtYW4ubmV0OTYw
Qzg2MjhENTkyRkY4Q0RFOEJCMEJGRTBFMkRFRTFENkM4RUVGQQAKCRDg4t7h1sju
+rFDAP9aIDxhmkvC8xFKh2TI/JWefO9UNIDUh+0iYom167DTKQD+OQ/iUn5MW6fu
PfJfdRiAMR5fMB7NpFvzcrRbkoXzQQCJApMEAQEKAH0WIQRSX6konxd5jbM7JygT
DfUWES/A/wUCWotpwF8UgAAuAChpc3N1ZXItZnByQG5vdGF0aW9ucy5vcGVu
cGdwLmZpZnRoaG9yc2VtYW4ubmV0NTI1RkE5Mjg5RjE3Nzk4REIzM0IyNzI4MTMw
REY1MTYxMTJGQzBGRgAKCRATDfUWES/A/0IVD/0ZWM6fz12is1z2FrZq9S9RX/dE
pz4UedrGjWcufMzv39R2V8TS03u0dx+vAzRo8KT05DGtvIna+3FhosnXSRBu3JSZ
jfT3pfA8hIPY2p2lJK0s2haeL98S2Yu3wJ2Yt+KgS9MmyFOSL/022fVDvYFwi+9Y
/oJlQaE5BXBXvS3n/0WsMrZzDcY3nLf5S0eSK9MSd9rM/M4z09KxS2dngj1Sifgv
GIiBYD7MwnZaaIi8WLGYTj6YeXSVMtlHzXa83fNiZtNOmLRI6p8VkMQUTh5Tjt3F
qlDd+zTSkn801IGiivEYfaOQcRylfTqORhSEFZqIUXI4W/uzJ6NL165SMNgfUa9m
UAinMeKWuC8SdJwKU3BcaqhqhdwoI5HjNq4RQm5K833dkjvn8XoBZ/Tk+Xy918oW
5UCsF3w+AZ9pCG1li0ZnCRkyK7BL0LE8qmotRfknyd9pbGkmljbQYYrxBFKtQdjx
QFvV5OymnxeJcFLqa/cWv7Mz2jPNkA5dSIdPEp0Gsq1Of9hfwS8sddJ6Z/bFONa9
WfSam337dgrZv9aa6I7sO9AP28gkKbB+W9TRVi5WW1m3tHqTm9WOXv30HKLuuyeO
f+K9v0NbKbhoUu1KzYPmXbzBzqeRlYgtGac85I/UvVWV4XzN80G8sxoHsg/fPyd5
sMEr+A2cUaWR7F0DUQ==
=VNMJ
-END PGP SIGNATURE-


___
Gnupg-users mailing list
Gnupg-users@gnupg.org
http://lists.gnupg.org/mailman/listinfo/gnupg-users

Re: Why Operating Systems don't always upgrade GnuPG [was: Re: How can we utilize latest GPG from RPM repository?]

2018-02-19 Thread edgar 

On Feb 19, 2018 12:45 PM, Daniel Kahn Gillmor  wrote:
>
> On Sat 2018-02-17 17:06:54 -0600, helices wrote:
> > I will probably never understand why wanting to run the most current
> > version of gnupg on a plethora of servers is controversial.
>
> Here's one last try to explain the situation.
>
> GnuPG (and the libraries it depends on) are used by (aka "depended on
> by") other libraries and tools, both those integrated into the operating
> system itself, and those that might be externally installed.  Some of
> these dependencies are "brittle".
>
> Brittle software dependencies
> -
>
> GnuPG is under active development, and it has never had a fully-featured
> stable API (Application Programming Interface).  What i mean is, there
> are some capabilities that are only available from the user interface
> (UI), and are not in the stable API.  GnuPG's UI has been constantly
> improving; sometimes that means that older (worse) UI gets deprecated,
> removed, or has its semantics change.
>
> For historical reasons, there are a number of tools that were built
> around some element of the GnuPG UI that was current at the time the
> tool was built.  Even worse, there are a number of tools that assume
> certain behaviors and features of GnuPG's internal storage (e.g. what
> goes on in ~/.gnupg/), which has never been explicitly part of its API
> (confusingly, there are some exceptions where GnuPG upstream *has*
> encouraged other tools to programmatically interact with some elements
> within its internal storage).  Newer versions of GnuPG do different
> things with its internal storage (and as users we get benefits from
> those improvements).
>
> Simply upgrading GnuPG to the latest available version on a server that
> also ships other complex software is likely to lead to breakage
> elsewhere in the OS because of these brittle assumptions and
> dependencies around GnuPG's UI and internal storage.
>
> A case study
> 
>
> For example, the current stable version of the Debian operating system
> is Debian 9 ("stretch"), and it ships a version of the "modern" branch
> of GnuPG.
>
> As one of the GnuPG maintainers for Debian, i was hoping at one point to
> backport the "modern" version of GnuPG to the previous version of Debian
> (Debian 8, "jessie"), which some people still run on servers.  I found
> that such an upgrade would break at least a half-dozen other packages in
> Debian jessie *that i knew of* [0] -- and their breakage would in turn
> likely affect some number of other packages.  This was not an exhaustive
> survey of all possible bugs, just the most visible ones. :/
>
> I have personally given up on the project of backporting modern GnuPG to
> "jessie", because i think what time i can devote to GnuPG maintenance is
> better-spent elsewhere.  I don't have the bandwidth to cope with the
> resultant bug reports in other packages that such a backport would
> produce.  Generally, i encourage users of "jessie" to uprade their
> entire OS to the current version of debian stable, and to take advantage
> of the improvements to GnuPG that way.
>
> What can we do?
> ---
>
> The problems described above point to problems in the ecosystem *around*
> GnuPG, but it also points to concerns about GnuPG's presentation of its
> capabilities *to* the rest of the ecosystem.  To the extent that GnuPG
> offers features that other tools might want to use, when those features
> are not part of an explicit, documented API, the ecosystem apparently
> *does* try to manipulate them anyway, with all the attendant brittleness
> that you can imagine.
>
> How can GnuPG contribute to fixing this problem?  The traditional way
> that many other projects have taken is to define their core programmatic
> functionality into a library with a strict interface guarantees, and
> have explicitly deprecated other use.  The closest that GnuPG comes to
> this technique is GPGME, which is not feature-complete (as compared to
> the gpg executable), and has its own history of both difficult upgrades
> and unclear/surprising semantics.  It also doesn't have bindings in many
> popular programming languages.  When programmers in those language want
> to use GnuPG, their shortest path to "something that works" often
> involves shelling out to gpg, rather than binding to GPGME. :/ 

That doesn't sound secure. Probably shouldn't use such programs on a server. 
Gpgme looks to do everything that it needs to do. Can you specify a function it 
is missing? 

>
> Another thing that would help would be to explicitly and succinctly
> document the preferred ways of interacting with GnuPG in ways that other
> developers find useful.  Perhaps GnuPG could also itself try to detect
> when it is being used programmatically in an unstable way and produce
> warnings?
>
> Yet another complementary approach might be to aggressively police the
> ecosystem by finding other software that deends on GnuPG in

Why Operating Systems don't always upgrade GnuPG [was: Re: How can we utilize latest GPG from RPM repository?]

2018-02-19 Thread Daniel Kahn Gillmor 
On Sat 2018-02-17 17:06:54 -0600, helices wrote:
> I will probably never understand why wanting to run the most current
> version of gnupg on a plethora of servers is controversial.

Here's one last try to explain the situation.

GnuPG (and the libraries it depends on) are used by (aka "depended on
by") other libraries and tools, both those integrated into the operating
system itself, and those that might be externally installed.  Some of
these dependencies are "brittle".

Brittle software dependencies
-

GnuPG is under active development, and it has never had a fully-featured
stable API (Application Programming Interface).  What i mean is, there
are some capabilities that are only available from the user interface
(UI), and are not in the stable API.  GnuPG's UI has been constantly
improving; sometimes that means that older (worse) UI gets deprecated,
removed, or has its semantics change.

For historical reasons, there are a number of tools that were built
around some element of the GnuPG UI that was current at the time the
tool was built.  Even worse, there are a number of tools that assume
certain behaviors and features of GnuPG's internal storage (e.g. what
goes on in ~/.gnupg/), which has never been explicitly part of its API
(confusingly, there are some exceptions where GnuPG upstream *has*
encouraged other tools to programmatically interact with some elements
within its internal storage).  Newer versions of GnuPG do different
things with its internal storage (and as users we get benefits from
those improvements).

Simply upgrading GnuPG to the latest available version on a server that
also ships other complex software is likely to lead to breakage
elsewhere in the OS because of these brittle assumptions and
dependencies around GnuPG's UI and internal storage.

A case study


For example, the current stable version of the Debian operating system
is Debian 9 ("stretch"), and it ships a version of the "modern" branch
of GnuPG.

As one of the GnuPG maintainers for Debian, i was hoping at one point to
backport the "modern" version of GnuPG to the previous version of Debian
(Debian 8, "jessie"), which some people still run on servers.  I found
that such an upgrade would break at least a half-dozen other packages in
Debian jessie *that i knew of* [0] -- and their breakage would in turn
likely affect some number of other packages.  This was not an exhaustive
survey of all possible bugs, just the most visible ones. :/

I have personally given up on the project of backporting modern GnuPG to
"jessie", because i think what time i can devote to GnuPG maintenance is
better-spent elsewhere.  I don't have the bandwidth to cope with the
resultant bug reports in other packages that such a backport would
produce.  Generally, i encourage users of "jessie" to uprade their
entire OS to the current version of debian stable, and to take advantage
of the improvements to GnuPG that way.

What can we do?
---

The problems described above point to problems in the ecosystem *around*
GnuPG, but it also points to concerns about GnuPG's presentation of its
capabilities *to* the rest of the ecosystem.  To the extent that GnuPG
offers features that other tools might want to use, when those features
are not part of an explicit, documented API, the ecosystem apparently
*does* try to manipulate them anyway, with all the attendant brittleness
that you can imagine.

How can GnuPG contribute to fixing this problem?  The traditional way
that many other projects have taken is to define their core programmatic
functionality into a library with a strict interface guarantees, and
have explicitly deprecated other use.  The closest that GnuPG comes to
this technique is GPGME, which is not feature-complete (as compared to
the gpg executable), and has its own history of both difficult upgrades
and unclear/surprising semantics.  It also doesn't have bindings in many
popular programming languages.  When programmers in those language want
to use GnuPG, their shortest path to "something that works" often
involves shelling out to gpg, rather than binding to GPGME. :/ 

Another thing that would help would be to explicitly and succinctly
document the preferred ways of interacting with GnuPG in ways that other
developers find useful.  Perhaps GnuPG could also itself try to detect
when it is being used programmatically in an unstable way and produce
warnings?

Yet another complementary approach might be to aggressively police the
ecosystem by finding other software that deends on GnuPG in any of the
aforementioned brittle ways, and either ask those developers to stop
using GnuPG entirely, or to provide them with stable, well-supported
ways to do what they're looking to do.

I welcome discussion/suggestions on how we can improve this situation,
and i *definitely* welcome help in doing the kind of
ecosystem-perspective work necessary to make it easier to maintain an
up-to-date branch of GnuPG.

But