Re: WebDAV module ported from serf to curl; curl using openssl and zlib (was: Re: Openssl, serf and curl)

[Damjan Jovanovic]

On Sat, May 28, 2022 at 7:20 AM Arrigo Marchiori 
wrote:

>
> Apparently, we are not. If we force CURLOPT_CAINFO and CURLOPT_CAPATH
> to be NULL, we get a bit further but eventually Curl aborts because
> validateServerX509Certificate fails.
>
>
Sorry to resurrect an old thread, but I think I now understand how this
crash happens.

So I always thought it was OpenSSL doing something wrong with the
certificates, but then in Curl's configure.ac, close to the end of the
file, there is the following:
---snip---
  ca cert bundle:   ${ca}${ca_warning}
  ca cert path: ${capath}${capath_warning}
  ca fallback:  ${with_ca_fallback}
---snip---

which apparently comes from:

---snip---
dnl **
dnl Check for the CA bundle
dnl **

if test -n "$check_for_ca_bundle"; then
  CURL_CHECK_CA_BUNDLE
fi
---snip---

and in Curl's acinclude.m4 we see that the CURL_CHECK_CA_BUNDLE function
uses a number of options, distro-specific search paths, etc.

This would also explain why system Curl always worked perfectly for me on
FreeBSD - the Port already built it with
--with-ca-bundle="${LOCALBASE}/share/certs/ca-root-nss.crt".

Now presumably things go wrong when Curl is using a bad path to the CA
bundle, because either it detected the path wrong during ./configure, or
it's running on a different distro.

So we should either build our Curl with the --without-ca-bundle
--without-ca-path --without-ca-fallback options, or disable use of those
settings at runtime. I prefer doing it at runtime, to be on the safe side.

I have a patch ready, but I am now unsure about other issues. If we are
going to use our own certificate validation instead of OpenSSL's, shouldn't
we use SSL_CTX_set_cert_verify_callback() which completely replaces
OpenSSL's verification, instead of  SSL_CTX_set_verify() which just allows
us to override its result on each certificate? Also what steps does OpenSSL
follow? We have a page on our Wiki (
https://wiki.openoffice.org/wiki/Certificate_Path_Validation) with proposed
certificate validation requirements, and also need to confirm what OpenSSL
does, and whether it does the more elaborate checks like connecting to OCSP
and CRL servers to verify revocation status. If not, we'd have to develop
that ourselves.

Regards
Damjan

Re: WebDAV module ported from serf to curl; curl using openssl and zlib (was: Re: Openssl, serf and curl)

[Damjan Jovanovic]

On Sat, May 28, 2022 at 6:54 PM Arrigo Marchiori 
wrote:

> Hello Damjan, All,
>
> On Sat, May 28, 2022 at 03:22:53PM +0200, Damjan Jovanovic wrote:
>
> > On Sat, May 28, 2022 at 11:48 AM Arrigo Marchiori
> 
> > wrote:
> >
> > > Hello Damjan,
> > >
> > > On Sat, May 28, 2022 at 11:12:45AM +0200, Damjan Jovanovic wrote:
> > >
> > > > On Sat, May 28, 2022 at 9:20 AM Arrigo Marchiori
>  > > >
> > > > wrote:
> > > >
> > > > > Hello Damjan, all,
> > > > >
> > > > > On Fri, May 27, 2022 at 11:27:11PM +0200, Arrigo Marchiori wrote:
> > > > >
> > > > > > Hello,
> > > > > >
> > > > > > On Fri, May 27, 2022 at 09:46:51PM +0200, Arrigo Marchiori wrote:
> > > > > >
> > > > > > > Hello Damjan,
> > > > > > >
> > > > > > > On Sun, May 22, 2022 at 06:10:46PM +0200, Damjan Jovanovic
> wrote:
> > > > > > >
> > > > > > > > On Sun, May 22, 2022 at 2:43 PM Arrigo Marchiori
> > > > > 
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hello Damjan, all,
> > > > > > > > >
> > > > > > > > > On Tue, Apr 26, 2022 at 07:56:22PM +0200, Damjan Jovanovic
> > > wrote:
> > > > > > > > >
> > > > > > > > > > On Mon, Nov 15, 2021 at 9:57 PM Jim Jagielski <
> > > j...@jagunet.com>
> > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > I'm gonna look into the serf->(lib)curl option...
> Since we
> > > > > don't use
> > > > > > > > > any
> > > > > > > > > > > of the fancy features of serf, I'm thinking that the
> easy
> > > > > option might
> > > > > > > > > be
> > > > > > > > > > > best
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > Hi
> > > > > > > > > >
> > > > > > > > > > I've ported our WebDAV content provider module from Serf
> to
> > > Curl.
> > > > > > > > >
> > > > > > > > > I just enhanced the error reporting a bit; I am finding a
> > > problem
> > > > > > > > > under Linux and I do not really know how to assess it.
> > > > > > > > >
> > > > > > > > > The problem: if we build AOO on CentOS (that is our
> reference
> > > > > > > > > platform) then Curl will look for CA certificates in
> > > > > > > > > /etc/pki/tls/certs/ca-bundle.crt
> > > > > > > > >
> > > > > > > > > This will fail on openSUSE and probably on Ubuntu as well.
> > > > > > > > >
> > > > > > > > > It seems that the above path is set at configure time and
> > > embedded
> > > > > > > > > into Curl's code as #define macros.
> > > > > > > > >
> > > > > > > > > Is there an ``official'' way to assess this? Like, can we
> > > depend on
> > > > > > > > > NSS' certificate store as you wrote (quoted below)?
> > > > > > > > >
> > > > > > > >
> > > > > > > > Curl/OpenSSL have an enormous number of options and I am
> pretty
> > > sure
> > > > > it can
> > > > > > > > be fixed, but first I need to understand where and how it's
> > > failing.
> > > > > > > >
> > > > > > > > We currently allow it to run with the default CA certificate
> > > path, do
> > > > > > > > pre-verification on the server's certificate using those CA
> > > > > certificates,
> > > > > > > > then call our SSL_VERIFY_PEER function where we override the
> > > > > verification
> > > > > > > > result with the certificates from NSS.
> > > > > > >
> > > > > > > Apparently, it is failing before calling our SSL_VERIFY_PEER
> > > function.
> > > > > > >
> > > > > > > > If it's failing before reaching our SSL_VERIFY_PEER
> function, we
> > > > > should be
> > > > > > > > able to use Curl's CURLOPT_CAINFO or CURLOPT_CAINFO_BLOB
> > > functions
> > > > > to set a
> > > > > > > > custom CA certificate path (or in-memory buffer), maybe even
> an
> > > empty
> > > > > > > > buffer, so that it proceeds further. ("man CURLOPT_CAINFO",
> "man
> > > > > > > > CURLOPT_CAINFO_BLOB", or "man curl_easy_setopt" and read
> under
> > > the
> > > > > "SSL and
> > > > > > > > SECURITY OPTIONS" section.)
> > > > > > >
> > > > > > > So we would need to hard-code and try all possible paths to
> the CA
> > > > > > > bundle on Unix systems?
> > > > > > >
> > > > > > > > With the CURLOPT_CAINFO_BLOB option it might even be
> possible to
> > > > > skip the
> > > > > > > > custom certificate verification we do later, and pre-populate
> > > > > Curl/OpenSSL
> > > > > > > > with NSS certificates from the beginning, I just don't know
> > > enough
> > > > > about
> > > > > > > > NSS to rely on that (eg. if you are using a cryptographic
> device
> > > or
> > > > > smart
> > > > > > > > card in NSS, how does that work?). If that option is ok,
> then we
> > > > > might not
> > > > > > > > even need the NSS libraries: recent versions of NSS store
> all the
> > > > > > > > certificates in an SQLite database, which can be accessed
> with
> > > > > SQLite APIs
> > > > > > > > directly, no need to build with or ship the NSS libraries at
> all.
> > > > > > >
> > > > > > > If I understood correctly [1], a NSS-linked Curl would query
> NSS by
> > > > > > > itself... are we not in this condition?
> > > > > [...]
> > > > > >   1: https://curl.se/libcurl/c/CURLOPT_CAINFO.html
> > > > >
> > > > > Apparently, we are not. If 

Re: WebDAV module ported from serf to curl; curl using openssl and zlib (was: Re: Openssl, serf and curl)

[Arrigo Marchiori]

Hello Damjan, All,

On Sat, May 28, 2022 at 03:22:53PM +0200, Damjan Jovanovic wrote:

> On Sat, May 28, 2022 at 11:48 AM Arrigo Marchiori 
> wrote:
> 
> > Hello Damjan,
> >
> > On Sat, May 28, 2022 at 11:12:45AM +0200, Damjan Jovanovic wrote:
> >
> > > On Sat, May 28, 2022 at 9:20 AM Arrigo Marchiori  > >
> > > wrote:
> > >
> > > > Hello Damjan, all,
> > > >
> > > > On Fri, May 27, 2022 at 11:27:11PM +0200, Arrigo Marchiori wrote:
> > > >
> > > > > Hello,
> > > > >
> > > > > On Fri, May 27, 2022 at 09:46:51PM +0200, Arrigo Marchiori wrote:
> > > > >
> > > > > > Hello Damjan,
> > > > > >
> > > > > > On Sun, May 22, 2022 at 06:10:46PM +0200, Damjan Jovanovic wrote:
> > > > > >
> > > > > > > On Sun, May 22, 2022 at 2:43 PM Arrigo Marchiori
> > > > 
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hello Damjan, all,
> > > > > > > >
> > > > > > > > On Tue, Apr 26, 2022 at 07:56:22PM +0200, Damjan Jovanovic
> > wrote:
> > > > > > > >
> > > > > > > > > On Mon, Nov 15, 2021 at 9:57 PM Jim Jagielski <
> > j...@jagunet.com>
> > > > wrote:
> > > > > > > > >
> > > > > > > > > > I'm gonna look into the serf->(lib)curl option... Since we
> > > > don't use
> > > > > > > > any
> > > > > > > > > > of the fancy features of serf, I'm thinking that the easy
> > > > option might
> > > > > > > > be
> > > > > > > > > > best
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > Hi
> > > > > > > > >
> > > > > > > > > I've ported our WebDAV content provider module from Serf to
> > Curl.
> > > > > > > >
> > > > > > > > I just enhanced the error reporting a bit; I am finding a
> > problem
> > > > > > > > under Linux and I do not really know how to assess it.
> > > > > > > >
> > > > > > > > The problem: if we build AOO on CentOS (that is our reference
> > > > > > > > platform) then Curl will look for CA certificates in
> > > > > > > > /etc/pki/tls/certs/ca-bundle.crt
> > > > > > > >
> > > > > > > > This will fail on openSUSE and probably on Ubuntu as well.
> > > > > > > >
> > > > > > > > It seems that the above path is set at configure time and
> > embedded
> > > > > > > > into Curl's code as #define macros.
> > > > > > > >
> > > > > > > > Is there an ``official'' way to assess this? Like, can we
> > depend on
> > > > > > > > NSS' certificate store as you wrote (quoted below)?
> > > > > > > >
> > > > > > >
> > > > > > > Curl/OpenSSL have an enormous number of options and I am pretty
> > sure
> > > > it can
> > > > > > > be fixed, but first I need to understand where and how it's
> > failing.
> > > > > > >
> > > > > > > We currently allow it to run with the default CA certificate
> > path, do
> > > > > > > pre-verification on the server's certificate using those CA
> > > > certificates,
> > > > > > > then call our SSL_VERIFY_PEER function where we override the
> > > > verification
> > > > > > > result with the certificates from NSS.
> > > > > >
> > > > > > Apparently, it is failing before calling our SSL_VERIFY_PEER
> > function.
> > > > > >
> > > > > > > If it's failing before reaching our SSL_VERIFY_PEER function, we
> > > > should be
> > > > > > > able to use Curl's CURLOPT_CAINFO or CURLOPT_CAINFO_BLOB
> > functions
> > > > to set a
> > > > > > > custom CA certificate path (or in-memory buffer), maybe even an
> > empty
> > > > > > > buffer, so that it proceeds further. ("man CURLOPT_CAINFO", "man
> > > > > > > CURLOPT_CAINFO_BLOB", or "man curl_easy_setopt" and read under
> > the
> > > > "SSL and
> > > > > > > SECURITY OPTIONS" section.)
> > > > > >
> > > > > > So we would need to hard-code and try all possible paths to the CA
> > > > > > bundle on Unix systems?
> > > > > >
> > > > > > > With the CURLOPT_CAINFO_BLOB option it might even be possible to
> > > > skip the
> > > > > > > custom certificate verification we do later, and pre-populate
> > > > Curl/OpenSSL
> > > > > > > with NSS certificates from the beginning, I just don't know
> > enough
> > > > about
> > > > > > > NSS to rely on that (eg. if you are using a cryptographic device
> > or
> > > > smart
> > > > > > > card in NSS, how does that work?). If that option is ok, then we
> > > > might not
> > > > > > > even need the NSS libraries: recent versions of NSS store all the
> > > > > > > certificates in an SQLite database, which can be accessed with
> > > > SQLite APIs
> > > > > > > directly, no need to build with or ship the NSS libraries at all.
> > > > > >
> > > > > > If I understood correctly [1], a NSS-linked Curl would query NSS by
> > > > > > itself... are we not in this condition?
> > > > [...]
> > > > >   1: https://curl.se/libcurl/c/CURLOPT_CAINFO.html
> > > >
> > > > Apparently, we are not. If we force CURLOPT_CAINFO and CURLOPT_CAPATH
> > > > to be NULL, we get a bit further but eventually Curl aborts because
> > > > validateServerX509Certificate fails.
> > > >
> > > > Here is the log for checking AOO updates (please excuse the long
> > > > lines).  I added an extra bit of logging when entering and leaving
> > > > OPENSSL_ValidateServerCerti

Re: WebDAV module ported from serf to curl; curl using openssl and zlib (was: Re: Openssl, serf and curl)

[Damjan Jovanovic]

On Sat, May 28, 2022 at 11:48 AM Arrigo Marchiori 
wrote:

> Hello Damjan,
>
> On Sat, May 28, 2022 at 11:12:45AM +0200, Damjan Jovanovic wrote:
>
> > On Sat, May 28, 2022 at 9:20 AM Arrigo Marchiori  >
> > wrote:
> >
> > > Hello Damjan, all,
> > >
> > > On Fri, May 27, 2022 at 11:27:11PM +0200, Arrigo Marchiori wrote:
> > >
> > > > Hello,
> > > >
> > > > On Fri, May 27, 2022 at 09:46:51PM +0200, Arrigo Marchiori wrote:
> > > >
> > > > > Hello Damjan,
> > > > >
> > > > > On Sun, May 22, 2022 at 06:10:46PM +0200, Damjan Jovanovic wrote:
> > > > >
> > > > > > On Sun, May 22, 2022 at 2:43 PM Arrigo Marchiori
> > > 
> > > > > > wrote:
> > > > > >
> > > > > > > Hello Damjan, all,
> > > > > > >
> > > > > > > On Tue, Apr 26, 2022 at 07:56:22PM +0200, Damjan Jovanovic
> wrote:
> > > > > > >
> > > > > > > > On Mon, Nov 15, 2021 at 9:57 PM Jim Jagielski <
> j...@jagunet.com>
> > > wrote:
> > > > > > > >
> > > > > > > > > I'm gonna look into the serf->(lib)curl option... Since we
> > > don't use
> > > > > > > any
> > > > > > > > > of the fancy features of serf, I'm thinking that the easy
> > > option might
> > > > > > > be
> > > > > > > > > best
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > Hi
> > > > > > > >
> > > > > > > > I've ported our WebDAV content provider module from Serf to
> Curl.
> > > > > > >
> > > > > > > I just enhanced the error reporting a bit; I am finding a
> problem
> > > > > > > under Linux and I do not really know how to assess it.
> > > > > > >
> > > > > > > The problem: if we build AOO on CentOS (that is our reference
> > > > > > > platform) then Curl will look for CA certificates in
> > > > > > > /etc/pki/tls/certs/ca-bundle.crt
> > > > > > >
> > > > > > > This will fail on openSUSE and probably on Ubuntu as well.
> > > > > > >
> > > > > > > It seems that the above path is set at configure time and
> embedded
> > > > > > > into Curl's code as #define macros.
> > > > > > >
> > > > > > > Is there an ``official'' way to assess this? Like, can we
> depend on
> > > > > > > NSS' certificate store as you wrote (quoted below)?
> > > > > > >
> > > > > >
> > > > > > Curl/OpenSSL have an enormous number of options and I am pretty
> sure
> > > it can
> > > > > > be fixed, but first I need to understand where and how it's
> failing.
> > > > > >
> > > > > > We currently allow it to run with the default CA certificate
> path, do
> > > > > > pre-verification on the server's certificate using those CA
> > > certificates,
> > > > > > then call our SSL_VERIFY_PEER function where we override the
> > > verification
> > > > > > result with the certificates from NSS.
> > > > >
> > > > > Apparently, it is failing before calling our SSL_VERIFY_PEER
> function.
> > > > >
> > > > > > If it's failing before reaching our SSL_VERIFY_PEER function, we
> > > should be
> > > > > > able to use Curl's CURLOPT_CAINFO or CURLOPT_CAINFO_BLOB
> functions
> > > to set a
> > > > > > custom CA certificate path (or in-memory buffer), maybe even an
> empty
> > > > > > buffer, so that it proceeds further. ("man CURLOPT_CAINFO", "man
> > > > > > CURLOPT_CAINFO_BLOB", or "man curl_easy_setopt" and read under
> the
> > > "SSL and
> > > > > > SECURITY OPTIONS" section.)
> > > > >
> > > > > So we would need to hard-code and try all possible paths to the CA
> > > > > bundle on Unix systems?
> > > > >
> > > > > > With the CURLOPT_CAINFO_BLOB option it might even be possible to
> > > skip the
> > > > > > custom certificate verification we do later, and pre-populate
> > > Curl/OpenSSL
> > > > > > with NSS certificates from the beginning, I just don't know
> enough
> > > about
> > > > > > NSS to rely on that (eg. if you are using a cryptographic device
> or
> > > smart
> > > > > > card in NSS, how does that work?). If that option is ok, then we
> > > might not
> > > > > > even need the NSS libraries: recent versions of NSS store all the
> > > > > > certificates in an SQLite database, which can be accessed with
> > > SQLite APIs
> > > > > > directly, no need to build with or ship the NSS libraries at all.
> > > > >
> > > > > If I understood correctly [1], a NSS-linked Curl would query NSS by
> > > > > itself... are we not in this condition?
> > > [...]
> > > >   1: https://curl.se/libcurl/c/CURLOPT_CAINFO.html
> > >
> > > Apparently, we are not. If we force CURLOPT_CAINFO and CURLOPT_CAPATH
> > > to be NULL, we get a bit further but eventually Curl aborts because
> > > validateServerX509Certificate fails.
> > >
> > > Here is the log for checking AOO updates (please excuse the long
> > > lines).  I added an extra bit of logging when entering and leaving
> > > OPENSSL_ValidateServerCertificate, that is ``our SSL_VERIFY_PEER
> > > function''.
> > >
> > >
> 8<8<8<8<8<8<8<-
> > >
> > >  19 2022-05-28 07:13:52.61 CurlSession::CurlSession
> with
> > > URL
> https://ooo-updates.apache.org/aoonext/check.Update?pkgfmt=installed
> > >  29 2022-05-28 

Re: WebDAV module ported from serf to curl; curl using openssl and zlib (was: Re: Openssl, serf and curl)

[Arrigo Marchiori]

Hello Damjan,

On Sat, May 28, 2022 at 11:12:45AM +0200, Damjan Jovanovic wrote:

> On Sat, May 28, 2022 at 9:20 AM Arrigo Marchiori 
> wrote:
> 
> > Hello Damjan, all,
> >
> > On Fri, May 27, 2022 at 11:27:11PM +0200, Arrigo Marchiori wrote:
> >
> > > Hello,
> > >
> > > On Fri, May 27, 2022 at 09:46:51PM +0200, Arrigo Marchiori wrote:
> > >
> > > > Hello Damjan,
> > > >
> > > > On Sun, May 22, 2022 at 06:10:46PM +0200, Damjan Jovanovic wrote:
> > > >
> > > > > On Sun, May 22, 2022 at 2:43 PM Arrigo Marchiori
> > 
> > > > > wrote:
> > > > >
> > > > > > Hello Damjan, all,
> > > > > >
> > > > > > On Tue, Apr 26, 2022 at 07:56:22PM +0200, Damjan Jovanovic wrote:
> > > > > >
> > > > > > > On Mon, Nov 15, 2021 at 9:57 PM Jim Jagielski 
> > wrote:
> > > > > > >
> > > > > > > > I'm gonna look into the serf->(lib)curl option... Since we
> > don't use
> > > > > > any
> > > > > > > > of the fancy features of serf, I'm thinking that the easy
> > option might
> > > > > > be
> > > > > > > > best
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > Hi
> > > > > > >
> > > > > > > I've ported our WebDAV content provider module from Serf to Curl.
> > > > > >
> > > > > > I just enhanced the error reporting a bit; I am finding a problem
> > > > > > under Linux and I do not really know how to assess it.
> > > > > >
> > > > > > The problem: if we build AOO on CentOS (that is our reference
> > > > > > platform) then Curl will look for CA certificates in
> > > > > > /etc/pki/tls/certs/ca-bundle.crt
> > > > > >
> > > > > > This will fail on openSUSE and probably on Ubuntu as well.
> > > > > >
> > > > > > It seems that the above path is set at configure time and embedded
> > > > > > into Curl's code as #define macros.
> > > > > >
> > > > > > Is there an ``official'' way to assess this? Like, can we depend on
> > > > > > NSS' certificate store as you wrote (quoted below)?
> > > > > >
> > > > >
> > > > > Curl/OpenSSL have an enormous number of options and I am pretty sure
> > it can
> > > > > be fixed, but first I need to understand where and how it's failing.
> > > > >
> > > > > We currently allow it to run with the default CA certificate path, do
> > > > > pre-verification on the server's certificate using those CA
> > certificates,
> > > > > then call our SSL_VERIFY_PEER function where we override the
> > verification
> > > > > result with the certificates from NSS.
> > > >
> > > > Apparently, it is failing before calling our SSL_VERIFY_PEER function.
> > > >
> > > > > If it's failing before reaching our SSL_VERIFY_PEER function, we
> > should be
> > > > > able to use Curl's CURLOPT_CAINFO or CURLOPT_CAINFO_BLOB functions
> > to set a
> > > > > custom CA certificate path (or in-memory buffer), maybe even an empty
> > > > > buffer, so that it proceeds further. ("man CURLOPT_CAINFO", "man
> > > > > CURLOPT_CAINFO_BLOB", or "man curl_easy_setopt" and read under the
> > "SSL and
> > > > > SECURITY OPTIONS" section.)
> > > >
> > > > So we would need to hard-code and try all possible paths to the CA
> > > > bundle on Unix systems?
> > > >
> > > > > With the CURLOPT_CAINFO_BLOB option it might even be possible to
> > skip the
> > > > > custom certificate verification we do later, and pre-populate
> > Curl/OpenSSL
> > > > > with NSS certificates from the beginning, I just don't know enough
> > about
> > > > > NSS to rely on that (eg. if you are using a cryptographic device or
> > smart
> > > > > card in NSS, how does that work?). If that option is ok, then we
> > might not
> > > > > even need the NSS libraries: recent versions of NSS store all the
> > > > > certificates in an SQLite database, which can be accessed with
> > SQLite APIs
> > > > > directly, no need to build with or ship the NSS libraries at all.
> > > >
> > > > If I understood correctly [1], a NSS-linked Curl would query NSS by
> > > > itself... are we not in this condition?
> > [...]
> > >   1: https://curl.se/libcurl/c/CURLOPT_CAINFO.html
> >
> > Apparently, we are not. If we force CURLOPT_CAINFO and CURLOPT_CAPATH
> > to be NULL, we get a bit further but eventually Curl aborts because
> > validateServerX509Certificate fails.
> >
> > Here is the log for checking AOO updates (please excuse the long
> > lines).  I added an extra bit of logging when entering and leaving
> > OPENSSL_ValidateServerCertificate, that is ``our SSL_VERIFY_PEER
> > function''.
> >
> > 8<8<8<8<8<8<8<-
> >
> >  19 2022-05-28 07:13:52.61 CurlSession::CurlSession with
> > URL https://ooo-updates.apache.org/aoonext/check.Update?pkgfmt=installed
> >  29 2022-05-28 07:13:52.61 Not using a proxy server
> >  39 2022-05-28 07:13:52.61 GET line 1093
> >  49 2022-05-28 07:13:52.61 [CurlINFO  ]   Trying
> > 151.101.2.132:443...
> >  59 2022-05-28 07:13:52.63 [CurlINFO  ] Connected to
> > ooo-updates.apache.org (151.101.2.132) port 443 (#0)
> >  69 2022-05

Re: WebDAV module ported from serf to curl; curl using openssl and zlib (was: Re: Openssl, serf and curl)

[Damjan Jovanovic]

On Sat, May 28, 2022 at 9:20 AM Arrigo Marchiori 
wrote:

> Hello Damjan, all,
>
> On Fri, May 27, 2022 at 11:27:11PM +0200, Arrigo Marchiori wrote:
>
> > Hello,
> >
> > On Fri, May 27, 2022 at 09:46:51PM +0200, Arrigo Marchiori wrote:
> >
> > > Hello Damjan,
> > >
> > > On Sun, May 22, 2022 at 06:10:46PM +0200, Damjan Jovanovic wrote:
> > >
> > > > On Sun, May 22, 2022 at 2:43 PM Arrigo Marchiori
> 
> > > > wrote:
> > > >
> > > > > Hello Damjan, all,
> > > > >
> > > > > On Tue, Apr 26, 2022 at 07:56:22PM +0200, Damjan Jovanovic wrote:
> > > > >
> > > > > > On Mon, Nov 15, 2021 at 9:57 PM Jim Jagielski 
> wrote:
> > > > > >
> > > > > > > I'm gonna look into the serf->(lib)curl option... Since we
> don't use
> > > > > any
> > > > > > > of the fancy features of serf, I'm thinking that the easy
> option might
> > > > > be
> > > > > > > best
> > > > > >
> > > > > >
> > > > > >
> > > > > > Hi
> > > > > >
> > > > > > I've ported our WebDAV content provider module from Serf to Curl.
> > > > >
> > > > > I just enhanced the error reporting a bit; I am finding a problem
> > > > > under Linux and I do not really know how to assess it.
> > > > >
> > > > > The problem: if we build AOO on CentOS (that is our reference
> > > > > platform) then Curl will look for CA certificates in
> > > > > /etc/pki/tls/certs/ca-bundle.crt
> > > > >
> > > > > This will fail on openSUSE and probably on Ubuntu as well.
> > > > >
> > > > > It seems that the above path is set at configure time and embedded
> > > > > into Curl's code as #define macros.
> > > > >
> > > > > Is there an ``official'' way to assess this? Like, can we depend on
> > > > > NSS' certificate store as you wrote (quoted below)?
> > > > >
> > > >
> > > > Curl/OpenSSL have an enormous number of options and I am pretty sure
> it can
> > > > be fixed, but first I need to understand where and how it's failing.
> > > >
> > > > We currently allow it to run with the default CA certificate path, do
> > > > pre-verification on the server's certificate using those CA
> certificates,
> > > > then call our SSL_VERIFY_PEER function where we override the
> verification
> > > > result with the certificates from NSS.
> > >
> > > Apparently, it is failing before calling our SSL_VERIFY_PEER function.
> > >
> > > > If it's failing before reaching our SSL_VERIFY_PEER function, we
> should be
> > > > able to use Curl's CURLOPT_CAINFO or CURLOPT_CAINFO_BLOB functions
> to set a
> > > > custom CA certificate path (or in-memory buffer), maybe even an empty
> > > > buffer, so that it proceeds further. ("man CURLOPT_CAINFO", "man
> > > > CURLOPT_CAINFO_BLOB", or "man curl_easy_setopt" and read under the
> "SSL and
> > > > SECURITY OPTIONS" section.)
> > >
> > > So we would need to hard-code and try all possible paths to the CA
> > > bundle on Unix systems?
> > >
> > > > With the CURLOPT_CAINFO_BLOB option it might even be possible to
> skip the
> > > > custom certificate verification we do later, and pre-populate
> Curl/OpenSSL
> > > > with NSS certificates from the beginning, I just don't know enough
> about
> > > > NSS to rely on that (eg. if you are using a cryptographic device or
> smart
> > > > card in NSS, how does that work?). If that option is ok, then we
> might not
> > > > even need the NSS libraries: recent versions of NSS store all the
> > > > certificates in an SQLite database, which can be accessed with
> SQLite APIs
> > > > directly, no need to build with or ship the NSS libraries at all.
> > >
> > > If I understood correctly [1], a NSS-linked Curl would query NSS by
> > > itself... are we not in this condition?
> [...]
> >   1: https://curl.se/libcurl/c/CURLOPT_CAINFO.html
>
> Apparently, we are not. If we force CURLOPT_CAINFO and CURLOPT_CAPATH
> to be NULL, we get a bit further but eventually Curl aborts because
> validateServerX509Certificate fails.
>
> Here is the log for checking AOO updates (please excuse the long
> lines).  I added an extra bit of logging when entering and leaving
> OPENSSL_ValidateServerCertificate, that is ``our SSL_VERIFY_PEER
> function''.
>
> 8<8<8<8<8<8<8<-
>
>  19 2022-05-28 07:13:52.61 CurlSession::CurlSession with
> URL https://ooo-updates.apache.org/aoonext/check.Update?pkgfmt=installed
>  29 2022-05-28 07:13:52.61 Not using a proxy server
>  39 2022-05-28 07:13:52.61 GET line 1093
>  49 2022-05-28 07:13:52.61 [CurlINFO  ]   Trying
> 151.101.2.132:443...
>  59 2022-05-28 07:13:52.63 [CurlINFO  ] Connected to
> ooo-updates.apache.org (151.101.2.132) port 443 (#0)
>  69 2022-05-28 07:13:52.63 [CurlINFO  ] ALPN, offering
> http/1.1
>  79 2022-05-28 07:13:52.63 [CurlINFO  ] Cipher selection:
> ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
>  89 2022-05-28 07:13:52.63 [CurlINFO  ] TLSv1.2 (OUT), TLS
> handshake, Client hello (1):
>  99 2022

Re: WebDAV module ported from serf to curl; curl using openssl and zlib (was: Re: Openssl, serf and curl)

[Damjan Jovanovic]

On Fri, May 27, 2022 at 9:47 PM Arrigo Marchiori 
wrote:

> Hello Damjan,
>
> On Sun, May 22, 2022 at 06:10:46PM +0200, Damjan Jovanovic wrote:
>
> > On Sun, May 22, 2022 at 2:43 PM Arrigo Marchiori  >
> > wrote:
> >
> > > Hello Damjan, all,
> > >
> > > On Tue, Apr 26, 2022 at 07:56:22PM +0200, Damjan Jovanovic wrote:
> > >
> > > > On Mon, Nov 15, 2021 at 9:57 PM Jim Jagielski 
> wrote:
> > > >
> > > > > I'm gonna look into the serf->(lib)curl option... Since we don't
> use
> > > any
> > > > > of the fancy features of serf, I'm thinking that the easy option
> might
> > > be
> > > > > best
> > > >
> > > >
> > > >
> > > > Hi
> > > >
> > > > I've ported our WebDAV content provider module from Serf to Curl.
> > >
> > > I just enhanced the error reporting a bit; I am finding a problem
> > > under Linux and I do not really know how to assess it.
> > >
> > > The problem: if we build AOO on CentOS (that is our reference
> > > platform) then Curl will look for CA certificates in
> > > /etc/pki/tls/certs/ca-bundle.crt
> > >
> > > This will fail on openSUSE and probably on Ubuntu as well.
> > >
> > > It seems that the above path is set at configure time and embedded
> > > into Curl's code as #define macros.
> > >
> > > Is there an ``official'' way to assess this? Like, can we depend on
> > > NSS' certificate store as you wrote (quoted below)?
> > >
> >
> > Curl/OpenSSL have an enormous number of options and I am pretty sure it
> can
> > be fixed, but first I need to understand where and how it's failing.
> >
> > We currently allow it to run with the default CA certificate path, do
> > pre-verification on the server's certificate using those CA certificates,
> > then call our SSL_VERIFY_PEER function where we override the verification
> > result with the certificates from NSS.
>
> Apparently, it is failing before calling our SSL_VERIFY_PEER function.
>
> > If it's failing before reaching our SSL_VERIFY_PEER function, we should
> be
> > able to use Curl's CURLOPT_CAINFO or CURLOPT_CAINFO_BLOB functions to
> set a
> > custom CA certificate path (or in-memory buffer), maybe even an empty
> > buffer, so that it proceeds further. ("man CURLOPT_CAINFO", "man
> > CURLOPT_CAINFO_BLOB", or "man curl_easy_setopt" and read under the "SSL
> and
> > SECURITY OPTIONS" section.)
>
> So we would need to hard-code and try all possible paths to the CA
> bundle on Unix systems?
>
> > With the CURLOPT_CAINFO_BLOB option it might even be possible to skip the
> > custom certificate verification we do later, and pre-populate
> Curl/OpenSSL
> > with NSS certificates from the beginning, I just don't know enough about
> > NSS to rely on that (eg. if you are using a cryptographic device or smart
> > card in NSS, how does that work?). If that option is ok, then we might
> not
> > even need the NSS libraries: recent versions of NSS store all the
> > certificates in an SQLite database, which can be accessed with SQLite
> APIs
> > directly, no need to build with or ship the NSS libraries at all.
>
> If I understood correctly [1], a NSS-linked Curl would query NSS by
> itself... are we not in this condition?
>

Yes but NSS is category B, shouldn't we be avoiding it?

Re: WebDAV module ported from serf to curl; curl using openssl and zlib (was: Re: Openssl, serf and curl)

[Arrigo Marchiori]

Hello Damjan, all,

On Fri, May 27, 2022 at 11:27:11PM +0200, Arrigo Marchiori wrote:

> Hello,
> 
> On Fri, May 27, 2022 at 09:46:51PM +0200, Arrigo Marchiori wrote:
> 
> > Hello Damjan,
> > 
> > On Sun, May 22, 2022 at 06:10:46PM +0200, Damjan Jovanovic wrote:
> > 
> > > On Sun, May 22, 2022 at 2:43 PM Arrigo Marchiori 
> > > wrote:
> > > 
> > > > Hello Damjan, all,
> > > >
> > > > On Tue, Apr 26, 2022 at 07:56:22PM +0200, Damjan Jovanovic wrote:
> > > >
> > > > > On Mon, Nov 15, 2021 at 9:57 PM Jim Jagielski  
> > > > > wrote:
> > > > >
> > > > > > I'm gonna look into the serf->(lib)curl option... Since we don't use
> > > > any
> > > > > > of the fancy features of serf, I'm thinking that the easy option 
> > > > > > might
> > > > be
> > > > > > best
> > > > >
> > > > >
> > > > >
> > > > > Hi
> > > > >
> > > > > I've ported our WebDAV content provider module from Serf to Curl.
> > > >
> > > > I just enhanced the error reporting a bit; I am finding a problem
> > > > under Linux and I do not really know how to assess it.
> > > >
> > > > The problem: if we build AOO on CentOS (that is our reference
> > > > platform) then Curl will look for CA certificates in
> > > > /etc/pki/tls/certs/ca-bundle.crt
> > > >
> > > > This will fail on openSUSE and probably on Ubuntu as well.
> > > >
> > > > It seems that the above path is set at configure time and embedded
> > > > into Curl's code as #define macros.
> > > >
> > > > Is there an ``official'' way to assess this? Like, can we depend on
> > > > NSS' certificate store as you wrote (quoted below)?
> > > >
> > > 
> > > Curl/OpenSSL have an enormous number of options and I am pretty sure it 
> > > can
> > > be fixed, but first I need to understand where and how it's failing.
> > > 
> > > We currently allow it to run with the default CA certificate path, do
> > > pre-verification on the server's certificate using those CA certificates,
> > > then call our SSL_VERIFY_PEER function where we override the verification
> > > result with the certificates from NSS.
> > 
> > Apparently, it is failing before calling our SSL_VERIFY_PEER function.
> > 
> > > If it's failing before reaching our SSL_VERIFY_PEER function, we should be
> > > able to use Curl's CURLOPT_CAINFO or CURLOPT_CAINFO_BLOB functions to set 
> > > a
> > > custom CA certificate path (or in-memory buffer), maybe even an empty
> > > buffer, so that it proceeds further. ("man CURLOPT_CAINFO", "man
> > > CURLOPT_CAINFO_BLOB", or "man curl_easy_setopt" and read under the "SSL 
> > > and
> > > SECURITY OPTIONS" section.)
> > 
> > So we would need to hard-code and try all possible paths to the CA
> > bundle on Unix systems?
> >  
> > > With the CURLOPT_CAINFO_BLOB option it might even be possible to skip the
> > > custom certificate verification we do later, and pre-populate Curl/OpenSSL
> > > with NSS certificates from the beginning, I just don't know enough about
> > > NSS to rely on that (eg. if you are using a cryptographic device or smart
> > > card in NSS, how does that work?). If that option is ok, then we might not
> > > even need the NSS libraries: recent versions of NSS store all the
> > > certificates in an SQLite database, which can be accessed with SQLite APIs
> > > directly, no need to build with or ship the NSS libraries at all.
> > 
> > If I understood correctly [1], a NSS-linked Curl would query NSS by
> > itself... are we not in this condition?
[...]
>   1: https://curl.se/libcurl/c/CURLOPT_CAINFO.html

Apparently, we are not. If we force CURLOPT_CAINFO and CURLOPT_CAPATH
to be NULL, we get a bit further but eventually Curl aborts because
validateServerX509Certificate fails.

Here is the log for checking AOO updates (please excuse the long
lines).  I added an extra bit of logging when entering and leaving
OPENSSL_ValidateServerCertificate, that is ``our SSL_VERIFY_PEER
function''.

8<8<8<8<8<8<8<-

 19 2022-05-28 07:13:52.61 CurlSession::CurlSession with URL 
https://ooo-updates.apache.org/aoonext/check.Update?pkgfmt=installed
 29 2022-05-28 07:13:52.61 Not using a proxy server
 39 2022-05-28 07:13:52.61 GET line 1093
 49 2022-05-28 07:13:52.61 [CurlINFO  ]   Trying 
151.101.2.132:443...
 59 2022-05-28 07:13:52.63 [CurlINFO  ] Connected to 
ooo-updates.apache.org (151.101.2.132) port 443 (#0)
 69 2022-05-28 07:13:52.63 [CurlINFO  ] ALPN, offering http/1.1
 79 2022-05-28 07:13:52.63 [CurlINFO  ] Cipher selection: 
ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
 89 2022-05-28 07:13:52.63 [CurlINFO  ] TLSv1.2 (OUT), TLS 
handshake, Client hello (1):
 99 2022-05-28 07:13:52.64 [CurlINFO  ] TLSv1.2 (IN), TLS 
handshake, Server hello (2):
109 2022-05-28 07:13:52.64 [CurlINFO  ] TLSv1.2 (IN), TLS 
handshake, Certificate (11):
 --here we are inside OPENSSL_ValidateServerCertificate

Re: WebDAV module ported from serf to curl; curl using openssl and zlib (was: Re: Openssl, serf and curl)

[Arrigo Marchiori]

Hello,

On Fri, May 27, 2022 at 09:46:51PM +0200, Arrigo Marchiori wrote:

> Hello Damjan,
> 
> On Sun, May 22, 2022 at 06:10:46PM +0200, Damjan Jovanovic wrote:
> 
> > On Sun, May 22, 2022 at 2:43 PM Arrigo Marchiori 
> > wrote:
> > 
> > > Hello Damjan, all,
> > >
> > > On Tue, Apr 26, 2022 at 07:56:22PM +0200, Damjan Jovanovic wrote:
> > >
> > > > On Mon, Nov 15, 2021 at 9:57 PM Jim Jagielski  wrote:
> > > >
> > > > > I'm gonna look into the serf->(lib)curl option... Since we don't use
> > > any
> > > > > of the fancy features of serf, I'm thinking that the easy option might
> > > be
> > > > > best
> > > >
> > > >
> > > >
> > > > Hi
> > > >
> > > > I've ported our WebDAV content provider module from Serf to Curl.
> > >
> > > I just enhanced the error reporting a bit; I am finding a problem
> > > under Linux and I do not really know how to assess it.
> > >
> > > The problem: if we build AOO on CentOS (that is our reference
> > > platform) then Curl will look for CA certificates in
> > > /etc/pki/tls/certs/ca-bundle.crt
> > >
> > > This will fail on openSUSE and probably on Ubuntu as well.
> > >
> > > It seems that the above path is set at configure time and embedded
> > > into Curl's code as #define macros.
> > >
> > > Is there an ``official'' way to assess this? Like, can we depend on
> > > NSS' certificate store as you wrote (quoted below)?
> > >
> > 
> > Curl/OpenSSL have an enormous number of options and I am pretty sure it can
> > be fixed, but first I need to understand where and how it's failing.
> > 
> > We currently allow it to run with the default CA certificate path, do
> > pre-verification on the server's certificate using those CA certificates,
> > then call our SSL_VERIFY_PEER function where we override the verification
> > result with the certificates from NSS.
> 
> Apparently, it is failing before calling our SSL_VERIFY_PEER function.
> 
> > If it's failing before reaching our SSL_VERIFY_PEER function, we should be
> > able to use Curl's CURLOPT_CAINFO or CURLOPT_CAINFO_BLOB functions to set a
> > custom CA certificate path (or in-memory buffer), maybe even an empty
> > buffer, so that it proceeds further. ("man CURLOPT_CAINFO", "man
> > CURLOPT_CAINFO_BLOB", or "man curl_easy_setopt" and read under the "SSL and
> > SECURITY OPTIONS" section.)
> 
> So we would need to hard-code and try all possible paths to the CA
> bundle on Unix systems?
>  
> > With the CURLOPT_CAINFO_BLOB option it might even be possible to skip the
> > custom certificate verification we do later, and pre-populate Curl/OpenSSL
> > with NSS certificates from the beginning, I just don't know enough about
> > NSS to rely on that (eg. if you are using a cryptographic device or smart
> > card in NSS, how does that work?). If that option is ok, then we might not
> > even need the NSS libraries: recent versions of NSS store all the
> > certificates in an SQLite database, which can be accessed with SQLite APIs
> > directly, no need to build with or ship the NSS libraries at all.
> 
> If I understood correctly [1], a NSS-linked Curl would query NSS by
> itself... are we not in this condition?

Sorry, I forgot the link!
Here it is:

  1: https://curl.se/libcurl/c/CURLOPT_CAINFO.html

> Thank you in advance and best regards,

-- 
Arrigo

-
To unsubscribe, e-mail: dev-unsubscr...@openoffice.apache.org
For additional commands, e-mail: dev-h...@openoffice.apache.org

Re: WebDAV module ported from serf to curl; curl using openssl and zlib (was: Re: Openssl, serf and curl)

[Arrigo Marchiori]

Hello Damjan,

On Sun, May 22, 2022 at 06:10:46PM +0200, Damjan Jovanovic wrote:

> On Sun, May 22, 2022 at 2:43 PM Arrigo Marchiori 
> wrote:
> 
> > Hello Damjan, all,
> >
> > On Tue, Apr 26, 2022 at 07:56:22PM +0200, Damjan Jovanovic wrote:
> >
> > > On Mon, Nov 15, 2021 at 9:57 PM Jim Jagielski  wrote:
> > >
> > > > I'm gonna look into the serf->(lib)curl option... Since we don't use
> > any
> > > > of the fancy features of serf, I'm thinking that the easy option might
> > be
> > > > best
> > >
> > >
> > >
> > > Hi
> > >
> > > I've ported our WebDAV content provider module from Serf to Curl.
> >
> > I just enhanced the error reporting a bit; I am finding a problem
> > under Linux and I do not really know how to assess it.
> >
> > The problem: if we build AOO on CentOS (that is our reference
> > platform) then Curl will look for CA certificates in
> > /etc/pki/tls/certs/ca-bundle.crt
> >
> > This will fail on openSUSE and probably on Ubuntu as well.
> >
> > It seems that the above path is set at configure time and embedded
> > into Curl's code as #define macros.
> >
> > Is there an ``official'' way to assess this? Like, can we depend on
> > NSS' certificate store as you wrote (quoted below)?
> >
> 
> Curl/OpenSSL have an enormous number of options and I am pretty sure it can
> be fixed, but first I need to understand where and how it's failing.
> 
> We currently allow it to run with the default CA certificate path, do
> pre-verification on the server's certificate using those CA certificates,
> then call our SSL_VERIFY_PEER function where we override the verification
> result with the certificates from NSS.

Apparently, it is failing before calling our SSL_VERIFY_PEER function.

> If it's failing before reaching our SSL_VERIFY_PEER function, we should be
> able to use Curl's CURLOPT_CAINFO or CURLOPT_CAINFO_BLOB functions to set a
> custom CA certificate path (or in-memory buffer), maybe even an empty
> buffer, so that it proceeds further. ("man CURLOPT_CAINFO", "man
> CURLOPT_CAINFO_BLOB", or "man curl_easy_setopt" and read under the "SSL and
> SECURITY OPTIONS" section.)

So we would need to hard-code and try all possible paths to the CA
bundle on Unix systems?
 
> With the CURLOPT_CAINFO_BLOB option it might even be possible to skip the
> custom certificate verification we do later, and pre-populate Curl/OpenSSL
> with NSS certificates from the beginning, I just don't know enough about
> NSS to rely on that (eg. if you are using a cryptographic device or smart
> card in NSS, how does that work?). If that option is ok, then we might not
> even need the NSS libraries: recent versions of NSS store all the
> certificates in an SQLite database, which can be accessed with SQLite APIs
> directly, no need to build with or ship the NSS libraries at all.

If I understood correctly [1], a NSS-linked Curl would query NSS by
itself... are we not in this condition?

Thank you in advance and best regards,
-- 
Arrigo

-
To unsubscribe, e-mail: dev-unsubscr...@openoffice.apache.org
For additional commands, e-mail: dev-h...@openoffice.apache.org

Re: WebDAV module ported from serf to curl; curl using openssl and zlib (was: Re: Openssl, serf and curl)

[Damjan Jovanovic]

On Sun, May 22, 2022 at 2:43 PM Arrigo Marchiori 
wrote:

> Hello Damjan, all,
>
> On Tue, Apr 26, 2022 at 07:56:22PM +0200, Damjan Jovanovic wrote:
>
> > On Mon, Nov 15, 2021 at 9:57 PM Jim Jagielski  wrote:
> >
> > > I'm gonna look into the serf->(lib)curl option... Since we don't use
> any
> > > of the fancy features of serf, I'm thinking that the easy option might
> be
> > > best
> >
> >
> >
> > Hi
> >
> > I've ported our WebDAV content provider module from Serf to Curl.
>
> I just enhanced the error reporting a bit; I am finding a problem
> under Linux and I do not really know how to assess it.
>
> The problem: if we build AOO on CentOS (that is our reference
> platform) then Curl will look for CA certificates in
> /etc/pki/tls/certs/ca-bundle.crt
>
> This will fail on openSUSE and probably on Ubuntu as well.
>
> It seems that the above path is set at configure time and embedded
> into Curl's code as #define macros.
>
> Is there an ``official'' way to assess this? Like, can we depend on
> NSS' certificate store as you wrote (quoted below)?
>

Curl/OpenSSL have an enormous number of options and I am pretty sure it can
be fixed, but first I need to understand where and how it's failing.

We currently allow it to run with the default CA certificate path, do
pre-verification on the server's certificate using those CA certificates,
then call our SSL_VERIFY_PEER function where we override the verification
result with the certificates from NSS.

If it's failing before reaching our SSL_VERIFY_PEER function, we should be
able to use Curl's CURLOPT_CAINFO or CURLOPT_CAINFO_BLOB functions to set a
custom CA certificate path (or in-memory buffer), maybe even an empty
buffer, so that it proceeds further. ("man CURLOPT_CAINFO", "man
CURLOPT_CAINFO_BLOB", or "man curl_easy_setopt" and read under the "SSL and
SECURITY OPTIONS" section.)

With the CURLOPT_CAINFO_BLOB option it might even be possible to skip the
custom certificate verification we do later, and pre-populate Curl/OpenSSL
with NSS certificates from the beginning, I just don't know enough about
NSS to rely on that (eg. if you are using a cryptographic device or smart
card in NSS, how does that work?). If that option is ok, then we might not
even need the NSS libraries: recent versions of NSS store all the
certificates in an SQLite database, which can be accessed with SQLite APIs
directly, no need to build with or ship the NSS libraries at all.

I am planning to write a separate email, when I get a chance, about the
cryptography libraries and certificates story.

Regards
Damjan

Re: WebDAV module ported from serf to curl; curl using openssl and zlib (was: Re: Openssl, serf and curl)

[Arrigo Marchiori]

Hello Damjan, all,

On Tue, Apr 26, 2022 at 07:56:22PM +0200, Damjan Jovanovic wrote:

> On Mon, Nov 15, 2021 at 9:57 PM Jim Jagielski  wrote:
> 
> > I'm gonna look into the serf->(lib)curl option... Since we don't use any
> > of the fancy features of serf, I'm thinking that the easy option might be
> > best
> 
> 
> 
> Hi
> 
> I've ported our WebDAV content provider module from Serf to Curl.

I just enhanced the error reporting a bit; I am finding a problem
under Linux and I do not really know how to assess it.

The problem: if we build AOO on CentOS (that is our reference
platform) then Curl will look for CA certificates in
/etc/pki/tls/certs/ca-bundle.crt

This will fail on openSUSE and probably on Ubuntu as well.

It seems that the above path is set at configure time and embedded
into Curl's code as #define macros.

Is there an ``official'' way to assess this? Like, can we depend on
NSS' certificate store as you wrote (quoted below)?

[...]

> HTTPS works, with a custom certificate verification function, using our own
> certificate store from NSS and its API (like the Serf code used). A bug was
> discovered (which is in the Serf implementation too) where self-signed
> certificates were being unconditionally rejected; apparently NSS wants to
> see that a copy of that certificate  in its certificate chain parameter
> too. Now they work, and the user gets prompted to allow access.

Thank you in advance and best regards,
-- 
Arrigo

-
To unsubscribe, e-mail: dev-unsubscr...@openoffice.apache.org
For additional commands, e-mail: dev-h...@openoffice.apache.org

Re: WebDAV module ported from serf to curl; curl using openssl and zlib (was: Re: Openssl, serf and curl)

[Arrigo Marchiori]

Hello Damjan, all,

On Sun, May 15, 2022 at 04:12:14PM +0200, Damjan Jovanovic wrote:

> On Sun, May 15, 2022 at 3:57 PM Arrigo Marchiori 
> wrote:
> 
> > Dear All,
> >
> > On Thu, May 12, 2022 at 07:36:49PM +0200, Arrigo Marchiori wrote:
> >
> > > Dear all,
> > >
> > > if nobody objects, I will merge this PR during the week-end.
> > >
> > > Reference: https://github.com/apache/openoffice/pull/146
> >
> > Done.
> >
> > Thank you again, Damjan!
> >
> 
> It's a pleasure :).
> 
> The next library to update could be zlib, because this version of curl
> > does _not_ compile with the bundled zlib.
> >
> 
> I distinctly recall that zlib and curl compiled for me on Windows. Where
> and how is it failing for you?

It's failing on Linux.

If we want to use the bundled zlib, we have to tell the configure
script --with-zlib=$(OUTDIR) .

The configure script would look for:
 1- the header file in $(OUTDIR)/include
 2- the library in $(OUTDIR)/lib/
But the bundled zlib module installs:
 1a- the header file into $(OUTDIR)/inc
 2a- the static library as file $(OUTDIR)/lib/libzlib.a

So we have to patch it in a similar way as we did for the openssl paths
(i.e. replicate bb956dfd85428dd08d991a01ee6a6c4e966bf431)

Even in that case, it is not easy to understand if we are using the
bundled zlib or the system-level one. The main cause, IMHO, is that
system-level zlib is surely installed (because of other build
dependencies) and is auto-detected by configure scripts and default
Makefile settings.

To a quick glance, also the "openssl" module seems to link to the
system zlib on Linux.

As the bundled zlib is linked statically, I wonder if invoking ldd on
the built executables and .so files is the only way to check for
proper linking.

I am sorry I do not have more time to get into details by now. I hope
the above makes sense.

Best regards,
-- 
Arrigo

-
To unsubscribe, e-mail: dev-unsubscr...@openoffice.apache.org
For additional commands, e-mail: dev-h...@openoffice.apache.org

Re: WebDAV module ported from serf to curl; curl using openssl and zlib (was: Re: Openssl, serf and curl)

[Damjan Jovanovic]

On Sun, May 15, 2022 at 3:57 PM Arrigo Marchiori 
wrote:

> Dear All,
>
> On Thu, May 12, 2022 at 07:36:49PM +0200, Arrigo Marchiori wrote:
>
> > Dear all,
> >
> > if nobody objects, I will merge this PR during the week-end.
> >
> > Reference: https://github.com/apache/openoffice/pull/146
>
> Done.
>
> Thank you again, Damjan!
>

It's a pleasure :).

The next library to update could be zlib, because this version of curl
> does _not_ compile with the bundled zlib.
>

I distinctly recall that zlib and curl compiled for me on Windows. Where
and how is it failing for you?

Re: WebDAV module ported from serf to curl; curl using openssl and zlib (was: Re: Openssl, serf and curl)

[Arrigo Marchiori]

Dear All,

On Thu, May 12, 2022 at 07:36:49PM +0200, Arrigo Marchiori wrote:

> Dear all,
> 
> if nobody objects, I will merge this PR during the week-end.
> 
> Reference: https://github.com/apache/openoffice/pull/146

Done.

Thank you again, Damjan!

The next library to update could be zlib, because this version of curl
does _not_ compile with the bundled zlib.

But I guess it is only one more... among many.

Best regards,
-- 
Arrigo

-
To unsubscribe, e-mail: dev-unsubscr...@openoffice.apache.org
For additional commands, e-mail: dev-h...@openoffice.apache.org

Re: WebDAV module ported from serf to curl; curl using openssl and zlib (was: Re: Openssl, serf and curl)

[Arrigo Marchiori]

Dear all,

if nobody objects, I will merge this PR during the week-end.

Reference: https://github.com/apache/openoffice/pull/146

Best regards,
-- 
Arrigo

-
To unsubscribe, e-mail: dev-unsubscr...@openoffice.apache.org
For additional commands, e-mail: dev-h...@openoffice.apache.org

Re: WebDAV module ported from serf to curl; curl using openssl and zlib (was: Re: Openssl, serf and curl)

[Dave Fisher]

> On May 6, 2022, at 12:56 PM, Arrigo Marchiori  wrote:
> 
> Hello Dave,
> 
> On Mon, May 02, 2022 at 05:53:19PM -0700, Dave Fisher wrote:
> 
>> Just so that you aren’t the only one here.
> 
> I am sure I am not alone... but thank you anyway ;-)
> 
>> It will be important to write good release notes about these changes.
>> 
>> Users will need to understand clearly that updates to cURL and
>> OpenSSL for CVEs won’t happen immediately unless they build with 2
>> on their own.
> 
> I am not sure I understand this. We are just updating/changing bundled
> libraries... Can you please explain better what you mean?

Some users might hear about a CVE on cURL and/or OpenSSL and update their 
systems thinking that they have also fixed OpenOffice when due to our now 
bundling of these they would need to wait for an OpenOffice release.

ATB,
Dave

> 
> Best regards,
> -- 
> Arrigo
> 
> -
> To unsubscribe, e-mail: dev-unsubscr...@openoffice.apache.org
> For additional commands, e-mail: dev-h...@openoffice.apache.org
> 


-
To unsubscribe, e-mail: dev-unsubscr...@openoffice.apache.org
For additional commands, e-mail: dev-h...@openoffice.apache.org

Re: WebDAV module ported from serf to curl; curl using openssl and zlib (was: Re: Openssl, serf and curl)

[Arrigo Marchiori]

Hello All,

replying to myself now...

On Mon, May 02, 2022 at 10:28:51PM +0200, Arrigo Marchiori wrote:

> > > > On Fri, Apr 29, 2022 at 04:53:59AM +0200, Damjan Jovanovic wrote:
> > > [...]
> > > > > When I run your Linux binaries, I also get that error.
> > > > > 
> > > > > One problem is libcurl -> openssl.
[...]
> I suggest:

[...]

>  2- we patch curl's configure script to use the bundled OpenSSL
>  library, when requested.

PR here: https://github.com/DamjanJovanovic/openoffice/pull/1

This should fix the crash.

Best regards,
-- 
Arrigo

-
To unsubscribe, e-mail: dev-unsubscr...@openoffice.apache.org
For additional commands, e-mail: dev-h...@openoffice.apache.org

Re: WebDAV module ported from serf to curl; curl using openssl and zlib (was: Re: Openssl, serf and curl)

[Arrigo Marchiori]

Hello Dave,

On Mon, May 02, 2022 at 05:53:19PM -0700, Dave Fisher wrote:

> Just so that you aren’t the only one here.

I am sure I am not alone... but thank you anyway ;-)

> It will be important to write good release notes about these changes.
> 
> Users will need to understand clearly that updates to cURL and
> OpenSSL for CVEs won’t happen immediately unless they build with 2
> on their own.

I am not sure I understand this. We are just updating/changing bundled
libraries... Can you please explain better what you mean?

Best regards,
-- 
Arrigo

-
To unsubscribe, e-mail: dev-unsubscr...@openoffice.apache.org
For additional commands, e-mail: dev-h...@openoffice.apache.org

Re: WebDAV module ported from serf to curl; curl using openssl and zlib (was: Re: Openssl, serf and curl)

[Dave Fisher]

> On May 2, 2022, at 1:28 PM, Arrigo Marchiori  wrote:
> 
> Hello All,
> 
> On Sat, Apr 30, 2022 at 10:31:33PM +0200, Arrigo Marchiori wrote:
> 
>> Replying to myself again...
>> 
>> On Sat, Apr 30, 2022 at 10:03:54PM +0200, Arrigo Marchiori wrote:
>> 
>>> Replying to myself...
>>> 
>>> On Sat, Apr 30, 2022 at 09:41:04PM +0200, Arrigo Marchiori wrote:
>>> 
 Hello Damjan,
 
 On Fri, Apr 29, 2022 at 04:53:59AM +0200, Damjan Jovanovic wrote:
>>> [...]
> When I run your Linux binaries, I also get that error.
> 
> One problem is libcurl -> openssl.
> 
> "ldd ./libcurl.so" shows:
>libssl.so.10 => not found
>libcrypto.so.10 => not found
>>> [...]
> 
> It connects and seems to begin the SSL negotiations, and then probably
> crashes while doing the verify_callback we set with SSL_CTX_set_verify().
> 
> I suspect the Curl and/or OpenSSL headers used to build AOO, and their
> libraries available at runtime, are incompatible.
> 
> What happens if you run AOO on the same machine it was built on?
>>> [...]
 As I wrote above, according to your suspects, it may be worthwile to
 bundle curl and openssl binaries. I will try to follow this path. It
 should be a matter of parameters to the configure script.
>>> 
>>> Curl and openssl are already bundled.
>>> 
>>> But Curl is not using the openssl from AOO, but rather from the
>>> system. In other words, it does not honor the environment variable
>>> SYSTEM_OPENSSL=NO set by the AOO build system.
>>> 
>>> Curl's configure script looks for the SSL headers and libraries either
>>> from pkgconfig or at a path provided with parameter --with-ssl
>>> 
>>> We do not support pkgconfig inside the solver, so we have to rely on
>>> the path.
>>> 
>>> But Curl's configure script expects to find a directory structure such as:
>>> - $PREFIX/include/openssl
>>> - $PREFIX/lib[64]
>>> whereas our $OUTDIR has openssl installed into different paths:
>>> - $OUTDIR/inc/external/openssl
>>> - $OUTDIR/lib
>>> 
>>> I believe this means we have to patch Curl's configure script.
>> 
>> Not only. We also have to actually bundle libssl.so and libcrypto.so
>> because they are not!
>> 
>> This means we have to patch:
>> 
>> - main/openssl/prj/d.lst to have them included into the solver;
>> 
>> - scp2/source/ooo/file_library_ooo.scp to have their Unix names
>> included.
>> 
>> This is a separate bug and will be worth its own pull request.
> 
> Here it is:
> https://github.com/apache/openoffice/pull/147
> 
> I suggest:
> 
> 1- we merge the above PR (I will on Friday, if no one vetoes and no
> issues are raised);
> 
> 2- we patch curl's configure script to use the bundled OpenSSL
> library, when requested.
> 
> Comments are always welcome.

Just so that you aren’t the only one here.

It will be important to write good release notes about these changes.

Users will need to understand clearly that updates to cURL and OpenSSL for CVEs 
won’t happen immediately unless they build with 2 on their own.

Regards,
Dave


> 
> Best regards,
> -- 
> Arrigo
> 
> -
> To unsubscribe, e-mail: dev-unsubscr...@openoffice.apache.org
> For additional commands, e-mail: dev-h...@openoffice.apache.org
> 


-
To unsubscribe, e-mail: dev-unsubscr...@openoffice.apache.org
For additional commands, e-mail: dev-h...@openoffice.apache.org

Re: WebDAV module ported from serf to curl; curl using openssl and zlib (was: Re: Openssl, serf and curl)

[Arrigo Marchiori]

Hello All,

On Sat, Apr 30, 2022 at 10:31:33PM +0200, Arrigo Marchiori wrote:

> Replying to myself again...
> 
> On Sat, Apr 30, 2022 at 10:03:54PM +0200, Arrigo Marchiori wrote:
> 
> > Replying to myself...
> > 
> > On Sat, Apr 30, 2022 at 09:41:04PM +0200, Arrigo Marchiori wrote:
> > 
> > > Hello Damjan,
> > > 
> > > On Fri, Apr 29, 2022 at 04:53:59AM +0200, Damjan Jovanovic wrote:
> > [...]
> > > > When I run your Linux binaries, I also get that error.
> > > > 
> > > > One problem is libcurl -> openssl.
> > > > 
> > > > "ldd ./libcurl.so" shows:
> > > > libssl.so.10 => not found
> > > > libcrypto.so.10 => not found
> > [...]
> > > > 
> > > > It connects and seems to begin the SSL negotiations, and then probably
> > > > crashes while doing the verify_callback we set with 
> > > > SSL_CTX_set_verify().
> > > > 
> > > > I suspect the Curl and/or OpenSSL headers used to build AOO, and their
> > > > libraries available at runtime, are incompatible.
> > > > 
> > > > What happens if you run AOO on the same machine it was built on?
> > [...]
> > > As I wrote above, according to your suspects, it may be worthwile to
> > > bundle curl and openssl binaries. I will try to follow this path. It
> > > should be a matter of parameters to the configure script.
> > 
> > Curl and openssl are already bundled.
> > 
> > But Curl is not using the openssl from AOO, but rather from the
> > system. In other words, it does not honor the environment variable
> > SYSTEM_OPENSSL=NO set by the AOO build system.
> > 
> > Curl's configure script looks for the SSL headers and libraries either
> > from pkgconfig or at a path provided with parameter --with-ssl
> > 
> > We do not support pkgconfig inside the solver, so we have to rely on
> > the path.
> > 
> > But Curl's configure script expects to find a directory structure such as:
> >  - $PREFIX/include/openssl
> >  - $PREFIX/lib[64]
> > whereas our $OUTDIR has openssl installed into different paths:
> >  - $OUTDIR/inc/external/openssl
> >  - $OUTDIR/lib
> > 
> > I believe this means we have to patch Curl's configure script.
> 
> Not only. We also have to actually bundle libssl.so and libcrypto.so
> because they are not!
> 
> This means we have to patch:
> 
>  - main/openssl/prj/d.lst to have them included into the solver;
> 
>  - scp2/source/ooo/file_library_ooo.scp to have their Unix names
>  included.
> 
> This is a separate bug and will be worth its own pull request.

Here it is:
https://github.com/apache/openoffice/pull/147

I suggest:

 1- we merge the above PR (I will on Friday, if no one vetoes and no
 issues are raised);

 2- we patch curl's configure script to use the bundled OpenSSL
 library, when requested.

Comments are always welcome.
 
Best regards,
-- 
Arrigo

-
To unsubscribe, e-mail: dev-unsubscr...@openoffice.apache.org
For additional commands, e-mail: dev-h...@openoffice.apache.org

Re: WebDAV module ported from serf to curl; curl using openssl and zlib (was: Re: Openssl, serf and curl)

[Arrigo Marchiori]

Replying to myself again...

On Sat, Apr 30, 2022 at 10:03:54PM +0200, Arrigo Marchiori wrote:

> Replying to myself...
> 
> On Sat, Apr 30, 2022 at 09:41:04PM +0200, Arrigo Marchiori wrote:
> 
> > Hello Damjan,
> > 
> > On Fri, Apr 29, 2022 at 04:53:59AM +0200, Damjan Jovanovic wrote:
> [...]
> > > When I run your Linux binaries, I also get that error.
> > > 
> > > One problem is libcurl -> openssl.
> > > 
> > > "ldd ./libcurl.so" shows:
> > > libssl.so.10 => not found
> > > libcrypto.so.10 => not found
> [...]
> > > 
> > > It connects and seems to begin the SSL negotiations, and then probably
> > > crashes while doing the verify_callback we set with SSL_CTX_set_verify().
> > > 
> > > I suspect the Curl and/or OpenSSL headers used to build AOO, and their
> > > libraries available at runtime, are incompatible.
> > > 
> > > What happens if you run AOO on the same machine it was built on?
> [...]
> > As I wrote above, according to your suspects, it may be worthwile to
> > bundle curl and openssl binaries. I will try to follow this path. It
> > should be a matter of parameters to the configure script.
> 
> Curl and openssl are already bundled.
> 
> But Curl is not using the openssl from AOO, but rather from the
> system. In other words, it does not honor the environment variable
> SYSTEM_OPENSSL=NO set by the AOO build system.
> 
> Curl's configure script looks for the SSL headers and libraries either
> from pkgconfig or at a path provided with parameter --with-ssl
> 
> We do not support pkgconfig inside the solver, so we have to rely on
> the path.
> 
> But Curl's configure script expects to find a directory structure such as:
>  - $PREFIX/include/openssl
>  - $PREFIX/lib[64]
> whereas our $OUTDIR has openssl installed into different paths:
>  - $OUTDIR/inc/external/openssl
>  - $OUTDIR/lib
> 
> I believe this means we have to patch Curl's configure script.

Not only. We also have to actually bundle libssl.so and libcrypto.so
because they are not!

This means we have to patch:

 - main/openssl/prj/d.lst to have them included into the solver;

 - scp2/source/ooo/file_library_ooo.scp to have their Unix names
 included.

This is a separate bug and will be worth its own pull request.

Comments are welcome.

Best regards,
-- 
Arrigo

-
To unsubscribe, e-mail: dev-unsubscr...@openoffice.apache.org
For additional commands, e-mail: dev-h...@openoffice.apache.org

Re: WebDAV module ported from serf to curl; curl using openssl and zlib (was: Re: Openssl, serf and curl)

[Arrigo Marchiori]

Replying to myself...

On Sat, Apr 30, 2022 at 09:41:04PM +0200, Arrigo Marchiori wrote:

> Hello Damjan,
> 
> On Fri, Apr 29, 2022 at 04:53:59AM +0200, Damjan Jovanovic wrote:
[...]
> > When I run your Linux binaries, I also get that error.
> > 
> > One problem is libcurl -> openssl.
> > 
> > "ldd ./libcurl.so" shows:
> > libssl.so.10 => not found
> > libcrypto.so.10 => not found
[...]
> > 
> > It connects and seems to begin the SSL negotiations, and then probably
> > crashes while doing the verify_callback we set with SSL_CTX_set_verify().
> > 
> > I suspect the Curl and/or OpenSSL headers used to build AOO, and their
> > libraries available at runtime, are incompatible.
> > 
> > What happens if you run AOO on the same machine it was built on?
[...]
> As I wrote above, according to your suspects, it may be worthwile to
> bundle curl and openssl binaries. I will try to follow this path. It
> should be a matter of parameters to the configure script.

Curl and openssl are already bundled.

But Curl is not using the openssl from AOO, but rather from the
system. In other words, it does not honor the environment variable
SYSTEM_OPENSSL=NO set by the AOO build system.

Curl's configure script looks for the SSL headers and libraries either
from pkgconfig or at a path provided with parameter --with-ssl

We do not support pkgconfig inside the solver, so we have to rely on
the path.

But Curl's configure script expects to find a directory structure such as:
 - $PREFIX/include/openssl
 - $PREFIX/lib[64]
whereas our $OUTDIR has openssl installed into different paths:
 - $OUTDIR/inc/external/openssl
 - $OUTDIR/lib

I believe this means we have to patch Curl's configure script.

Best regards,
-- 
Arrigo

-
To unsubscribe, e-mail: dev-unsubscr...@openoffice.apache.org
For additional commands, e-mail: dev-h...@openoffice.apache.org

Re: WebDAV module ported from serf to curl; curl using openssl and zlib (was: Re: Openssl, serf and curl)

[Arrigo Marchiori]

Hello Damjan,

On Fri, Apr 29, 2022 at 04:53:59AM +0200, Damjan Jovanovic wrote:

> On Thu, Apr 28, 2022 at 8:46 PM Arrigo Marchiori 
> wrote:
> 
> > Hello Damjan,
> >
> > On Thu, Apr 28, 2022 at 06:53:43PM +0200, Damjan Jovanovic wrote:
> >
> > > On Thu, Apr 28, 2022 at 6:12 PM Arrigo Marchiori  > >
> > > wrote:
> > >
> > > > Dear Damjan, All,
> > > >
> > > > On Tue, Apr 26, 2022 at 07:56:22PM +0200, Damjan Jovanovic wrote:
> > > >
> > > > > On Mon, Nov 15, 2021 at 9:57 PM Jim Jagielski 
> > wrote:
> > > > >
> > > > > > I'm gonna look into the serf->(lib)curl option... Since we don't
> > use
> > > > any
> > > > > > of the fancy features of serf, I'm thinking that the easy option
> > might
> > > > be
> > > > > > best
> > > > >
> > > > >
> > > > >
> > > > > Hi
> > > > >
> > > > > I've ported our WebDAV content provider module from Serf to Curl.
> > > >
> > > > Binary for Linux available here for download:
> > > >
> > > >
> > https://home.apache.org/~ardovm/openoffice/linux/openoffice4-serf2curl-2022-04-28-installed.tar.bz2
> > > >
> > > > I understand from your previous message that you don't really like
> > > > GitHub, so I am writing here.
> > > >
> > > > [...]
> > > > > STATUS
> > > > >
> > > > > It builds and works well on FreeBSD and Windows.
> > > > >
> > > > > Most of the code was reused, and all the operations and semantics
> > > > > previously present with Serf, should have been preserved.
> > > > >
> > > > > Browsing WebDAV files and directories, loading files, overwriting
> > them
> > > > > ("Save"), creating them ("Save As"), renaming and deleting them, all
> > > > works.
> > > >
> > > > I am testing the binary for Linux linked above.
> > > >
> > > > I tried "Open" and entered a https address, that I know is password
> > > > protected.
> > > >
> > > > The current trunk would ask for the password. I got an error message
> > > > instead:
> > > >
> > > > > > Nonexistent object.
> > > > > > Nonexistent file.
> > > >
> > > > The address I tried to open is in the form https://host.domain:port/
> > > >
> > > > I tried to substitute "https" with "davs" and I got the same error.
> > > >
> > > > Maybe something is going wrong in the Linux build?
> > > >
> > > > I will now begin recompiling with debugging symbols enabled. Please
> > > > let me know how I can help.
> > > >
> > >
> > > That's not good :(.
> > >
> > > Set your macro security to "Medium", open the spreadsheet I've attached,
> > > and run the "RunMe" Basic macro. That should enable logging to the
> > console
> > > at the finest level of detail. Then exit and re-run AOO like this:
> > > soffice 2>&1 | tee /tmp/log.txt
> > > and examine the console output interactively as you use AOO, and/or check
> > > the log file afterwards. It should have everything, our logging, HTTP
> > > request and response headers and bodies, Curl's internal logging.
> >
> > I can't get to that point.
> >
> > I fired gdb and it seems that I end up into the blind
> >   catch (Exception const &)
> > block in file ucb/source/core/provprox.cxx:361
> >
> > Method UcbContentProviderProxy::getContentProvider() in fact is called
> > many times, but it only fails when I enter the https url in the "Open"
> > dialog box and press ENTER.
> >
> > Sorry this is all the debugging I can do for today. I hope it helps.
> >
> >
> When I run your Linux binaries, I also get that error.
> 
> One problem is libcurl -> openssl.
> 
> "ldd ./libcurl.so" shows:
> libssl.so.10 => not found
> libcrypto.so.10 => not found

This probably means that we have to ship our own binaries of openssl
on Linux.

> When I rename AOO's libcurl.so so that it's forced to use my system libcurl
> instead, it proceeds further, but runs into another problem:
> 
> Thread 1 "soffice.bin" received signal SIGSEGV, Segmentation fault.
> 0x0001 in ?? ()
[...snip of stack trace with no debugging symbols...]
> 
> It gets far enough to log and produces some output before the crash:
> 
>  11 2022-04-29 02:24:24.03 CurlSession::CurlSession with
> URL https://10.0.2.2:82/files/
>  21 2022-04-29 02:24:24.03 Not using a proxy server
>  31 2022-04-29 02:24:24.03 PROPFIND line 914
>  41 2022-04-29 02:24:24.03 [CurlINFO  ]   Trying 10.0.2.2:82
> ...
>  51 2022-04-29 02:24:24.03 [CurlINFO  ] TCP_NODELAY set
>  61 2022-04-29 02:24:24.03 [CurlINFO  ] Connected to
> 10.0.2.2 (10.0.2.2) port 82 (#0)
>  71 2022-04-29 02:24:24.03 [CurlINFO  ] ALPN, offering h2
>  81 2022-04-29 02:24:24.03 [CurlINFO  ] ALPN, offering
> http/1.1
>  91 2022-04-29 02:24:24.04 [CurlINFO  ] successfully set
> certificate verify locations:
> 101 2022-04-29 02:24:24.04 [CurlINFO  ]   CAfile:
> /etc/ssl/certs/ca-certificates.crt
>   CApath: /etc/ssl/certs
> 
> It connects and seems to begin the SSL negotiations, and then probably
> crashes while doing the verify_callback we set with SSL_CTX_

Re: WebDAV module ported from serf to curl; curl using openssl and zlib (was: Re: Openssl, serf and curl)

[Don Lewis]

On 26 Apr, Damjan Jovanovic wrote:
> On Mon, Nov 15, 2021 at 9:57 PM Jim Jagielski  wrote:
> 
>> I'm gonna look into the serf->(lib)curl option... Since we don't use any
>> of the fancy features of serf, I'm thinking that the easy option might be
>> best
> 
> 
> 
> Hi
> 
> I've ported our WebDAV content provider module from Serf to Curl.
> 
> While it ended well, and several other bugs were found and fixed, it
> definitely wasn't the "easy option" Jim ;). Starting with conservative
> changes, it ended up needing total restructuring, and became more of a
> rewrite. The crashes were frequent and hung connections many, and I had to
> read up on the HTTP protocol, and read Curl and Serf's source code, but
> eventually I prevailed, and a clean elegant stable Curl WebDAV module
> emerged.
> 
> The huge patch is attached for anyone wishing to review and test. Unless
> there are major objections, I'll push it in a couple of days.
> 
> STATUS
> 
> It builds and works well on FreeBSD and Windows.
> 
> Most of the code was reused, and all the operations and semantics
> previously present with Serf, should have been preserved.
> 
> Browsing WebDAV files and directories, loading files, overwriting them
> ("Save"), creating them ("Save As"), renaming and deleting them, all works.
> 
> HTTP and HTTPS proxies work. Unlike Serf, Curl could also support SOCKS
> proxies (with minimal further changes), but AOO lacks that setting in the
> proxy UI and configuration.
> 
> Authentication works, both to the destination server and to the proxy
> server. I've successfully tested Basic and Digest authentication. Curl
> supports every authentication method Serf does and more.
> 
> HTTPS works, with a custom certificate verification function, using our own
> certificate store from NSS and its API (like the Serf code used). A bug was
> discovered (which is in the Serf implementation too) where self-signed
> certificates were being unconditionally rejected; apparently NSS wants to
> see that a copy of that certificate  in its certificate chain parameter
> too. Now they work, and the user gets prompted to allow access.
> 
> HTTPS and authentication can be used together on the same connection and
> work well, both bringing up their UI dialogs as needed.
> 
> A bug was fixed where when username and password were both present in the
> URL (dav://user:pass@host/path), the code was trying to split them at the
> "@" instead of ":".
> 
> Unnecessary base64 encoding and decoding was removed, when importing the
> TLS connection's certificates into our XSecurityEnvironment. They now come
> in directly as ASN.1 DER, and still work.
> 
> The code was greatly restructured and cleaned up, as Curl's API is
> synchronous and blocking, with parameters set in advance instead of through
> many callbacks, which has allowed using short clear methods, and clean
> separation between the session and request classes. The WebDAV content
> provider has shrunk from 35 to 21 C++ files, 43 to 29 header files, and
> 19129 to 15991 lines of code. With WebDAV methods centralized and
> refactored into only 10-20 lines of code each, instead of scattered across
> 4 files, it is much more understandable and maintainable now.
> 
> Curl is vastly more portable than Serf. We should build easily now even on
> OS/2. We can remain with existing build tools instead of needing scons or
> cmake just to build Serf.
> 
> 3 now unused dependencies were removed: apr, apr-util, and serf. Serf isn't
> so bad. APR's pool idea is an ingenious way of doing resource management in
> C. However Curl has excellent documentation, guides, examples, and detailed
> explanations and even example code for each setting, while Serf has no
> documentation. Serf might be worth it in a project that already uses APR a
> lot, but we don't.
> 
> Instead of the historical, crippled forms of logging like OSL_TRACE(),
> which don't appear in release builds, I've made it use the newer
> com.sun.star.logging UNO component (wrapped in comphelper::EventLogger),
> which was inspired by java.util.logging, with configurable verbosity
> levels, handlers (file and console) and output formats (plain, csv), and
> importantly, which produces output in release builds too. I've also made it
> so that on LogLevel::FINEST, Curl's verbose mode is enabled and Curl's
> debug output is also logged through us, with descriptions of what Curl is
> doing, and logs of all HTTP traffic including headers and bodies, before
> encryption and after decryption in the case of HTTPS, giving us tremendous
> detail that can be used for troubleshooting problems.
> 
> CURL CHANGED TO USE OPENSSL AND ZLIB
> 
> Curl only supports the custom TLS certificate verification function we use
> (the CURLOPT_SSL_CTX_FUNCTION option) when built with OpenSSL, wolfSSL or
> mbedTLS providers. We currently use schannel on Windows instead, which had
> to be changed. I also made it use zlib, which generally helps, and WebDAV
> uses XML which is very verbose and bene

Re: WebDAV module ported from serf to curl; curl using openssl and zlib (was: Re: Openssl, serf and curl)

[Damjan Jovanovic]

On Thu, Apr 28, 2022 at 8:46 PM Arrigo Marchiori 
wrote:

> Hello Damjan,
>
> On Thu, Apr 28, 2022 at 06:53:43PM +0200, Damjan Jovanovic wrote:
>
> > On Thu, Apr 28, 2022 at 6:12 PM Arrigo Marchiori  >
> > wrote:
> >
> > > Dear Damjan, All,
> > >
> > > On Tue, Apr 26, 2022 at 07:56:22PM +0200, Damjan Jovanovic wrote:
> > >
> > > > On Mon, Nov 15, 2021 at 9:57 PM Jim Jagielski 
> wrote:
> > > >
> > > > > I'm gonna look into the serf->(lib)curl option... Since we don't
> use
> > > any
> > > > > of the fancy features of serf, I'm thinking that the easy option
> might
> > > be
> > > > > best
> > > >
> > > >
> > > >
> > > > Hi
> > > >
> > > > I've ported our WebDAV content provider module from Serf to Curl.
> > >
> > > Binary for Linux available here for download:
> > >
> > >
> https://home.apache.org/~ardovm/openoffice/linux/openoffice4-serf2curl-2022-04-28-installed.tar.bz2
> > >
> > > I understand from your previous message that you don't really like
> > > GitHub, so I am writing here.
> > >
> > > [...]
> > > > STATUS
> > > >
> > > > It builds and works well on FreeBSD and Windows.
> > > >
> > > > Most of the code was reused, and all the operations and semantics
> > > > previously present with Serf, should have been preserved.
> > > >
> > > > Browsing WebDAV files and directories, loading files, overwriting
> them
> > > > ("Save"), creating them ("Save As"), renaming and deleting them, all
> > > works.
> > >
> > > I am testing the binary for Linux linked above.
> > >
> > > I tried "Open" and entered a https address, that I know is password
> > > protected.
> > >
> > > The current trunk would ask for the password. I got an error message
> > > instead:
> > >
> > > > > Nonexistent object.
> > > > > Nonexistent file.
> > >
> > > The address I tried to open is in the form https://host.domain:port/
> > >
> > > I tried to substitute "https" with "davs" and I got the same error.
> > >
> > > Maybe something is going wrong in the Linux build?
> > >
> > > I will now begin recompiling with debugging symbols enabled. Please
> > > let me know how I can help.
> > >
> >
> > That's not good :(.
> >
> > Set your macro security to "Medium", open the spreadsheet I've attached,
> > and run the "RunMe" Basic macro. That should enable logging to the
> console
> > at the finest level of detail. Then exit and re-run AOO like this:
> > soffice 2>&1 | tee /tmp/log.txt
> > and examine the console output interactively as you use AOO, and/or check
> > the log file afterwards. It should have everything, our logging, HTTP
> > request and response headers and bodies, Curl's internal logging.
>
> I can't get to that point.
>
> I fired gdb and it seems that I end up into the blind
>   catch (Exception const &)
> block in file ucb/source/core/provprox.cxx:361
>
> Method UcbContentProviderProxy::getContentProvider() in fact is called
> many times, but it only fails when I enter the https url in the "Open"
> dialog box and press ENTER.
>
> Sorry this is all the debugging I can do for today. I hope it helps.
>
>
When I run your Linux binaries, I also get that error.

One problem is libcurl -> openssl.

"ldd ./libcurl.so" shows:
libssl.so.10 => not found
libcrypto.so.10 => not found

When I rename AOO's libcurl.so so that it's forced to use my system libcurl
instead, it proceeds further, but runs into another problem:

Thread 1 "soffice.bin" received signal SIGSEGV, Segmentation fault.
0x0001 in ?? ()
(gdb) bt
#0  0x0001 in ?? ()
#1  0x7fffd6d52a76 in ?? () from /lib/x86_64-linux-gnu/libssl.so.1.1
#2  0x7fffd6d52f75 in ?? () from /lib/x86_64-linux-gnu/libssl.so.1.1
#3  0x7fffd6d8505d in ?? () from /lib/x86_64-linux-gnu/libssl.so.1.1
#4  0x7fffd6d7c37e in ?? () from /lib/x86_64-linux-gnu/libssl.so.1.1
#5  0x7fffd6d67d98 in SSL_do_handshake () from
/lib/x86_64-linux-gnu/libssl.so.1.1
#6  0x7fffe4154e40 in ?? () from /lib/x86_64-linux-gnu/libcurl.so.4
#7  0x7fffe41571a2 in ?? () from /lib/x86_64-linux-gnu/libcurl.so.4
#8  0x7fffe415814f in ?? () from /lib/x86_64-linux-gnu/libcurl.so.4
#9  0x7fffe4103296 in ?? () from /lib/x86_64-linux-gnu/libcurl.so.4
#10 0x7fffe4104d13 in ?? () from /lib/x86_64-linux-gnu/libcurl.so.4
#11 0x7fffe412597d in ?? () from /lib/x86_64-linux-gnu/libcurl.so.4
#12 0x7fffe4126a11 in curl_multi_perform () from
/lib/x86_64-linux-gnu/libcurl.so.4
#13 0x7fffe411ce8b in curl_easy_perform () from
/lib/x86_64-linux-gnu/libcurl.so.4
#14 0x7fffc3c1c972 in ?? () from
/home/user/openoffice4/program/../program/libucpdav1.so
#15 0x7fffc3c1d02a in ?? () from
/home/user/openoffice4/program/../program/libucpdav1.so
#16 0x7fffc3c13e37 in ?? () from
/home/user/openoffice4/program/../program/libucpdav1.so
#17 0x7fffc3c18c70 in ?? () from
/home/user/openoffice4/program/../program/libucpdav1.so
#18 0x7fffc3c08586 in ?? () from
/home/user/openoffice4/program/../program/libucpdav1.so
#19 0x7fffc3bf25c2 in ?? () from
/home/user/openoffice4/prog

Re: WebDAV module ported from serf to curl; curl using openssl and zlib (was: Re: Openssl, serf and curl)

[Arrigo Marchiori]

Hello Damjan,

On Thu, Apr 28, 2022 at 06:53:43PM +0200, Damjan Jovanovic wrote:

> On Thu, Apr 28, 2022 at 6:12 PM Arrigo Marchiori 
> wrote:
> 
> > Dear Damjan, All,
> >
> > On Tue, Apr 26, 2022 at 07:56:22PM +0200, Damjan Jovanovic wrote:
> >
> > > On Mon, Nov 15, 2021 at 9:57 PM Jim Jagielski  wrote:
> > >
> > > > I'm gonna look into the serf->(lib)curl option... Since we don't use
> > any
> > > > of the fancy features of serf, I'm thinking that the easy option might
> > be
> > > > best
> > >
> > >
> > >
> > > Hi
> > >
> > > I've ported our WebDAV content provider module from Serf to Curl.
> >
> > Binary for Linux available here for download:
> >
> > https://home.apache.org/~ardovm/openoffice/linux/openoffice4-serf2curl-2022-04-28-installed.tar.bz2
> >
> > I understand from your previous message that you don't really like
> > GitHub, so I am writing here.
> >
> > [...]
> > > STATUS
> > >
> > > It builds and works well on FreeBSD and Windows.
> > >
> > > Most of the code was reused, and all the operations and semantics
> > > previously present with Serf, should have been preserved.
> > >
> > > Browsing WebDAV files and directories, loading files, overwriting them
> > > ("Save"), creating them ("Save As"), renaming and deleting them, all
> > works.
> >
> > I am testing the binary for Linux linked above.
> >
> > I tried "Open" and entered a https address, that I know is password
> > protected.
> >
> > The current trunk would ask for the password. I got an error message
> > instead:
> >
> > > > Nonexistent object.
> > > > Nonexistent file.
> >
> > The address I tried to open is in the form https://host.domain:port/
> >
> > I tried to substitute "https" with "davs" and I got the same error.
> >
> > Maybe something is going wrong in the Linux build?
> >
> > I will now begin recompiling with debugging symbols enabled. Please
> > let me know how I can help.
> >
> 
> That's not good :(.
> 
> Set your macro security to "Medium", open the spreadsheet I've attached,
> and run the "RunMe" Basic macro. That should enable logging to the console
> at the finest level of detail. Then exit and re-run AOO like this:
> soffice 2>&1 | tee /tmp/log.txt
> and examine the console output interactively as you use AOO, and/or check
> the log file afterwards. It should have everything, our logging, HTTP
> request and response headers and bodies, Curl's internal logging.

I can't get to that point.

I fired gdb and it seems that I end up into the blind
  catch (Exception const &)
block in file ucb/source/core/provprox.cxx:361

Method UcbContentProviderProxy::getContentProvider() in fact is called
many times, but it only fails when I enter the https url in the "Open"
dialog box and press ENTER.

Sorry this is all the debugging I can do for today. I hope it helps.

Best regards,
-- 
Arrigo

-
To unsubscribe, e-mail: dev-unsubscr...@openoffice.apache.org
For additional commands, e-mail: dev-h...@openoffice.apache.org

Re: WebDAV module ported from serf to curl; curl using openssl and zlib (was: Re: Openssl, serf and curl)

[Matthias Seidel]

Hi Pedro,

Am 28.04.22 um 18:49 schrieb Pedro Lino:
> Hi Damjan, Arrigo, all
>
>> On 04/28/2022 5:11 PM Arrigo Marchiori  wrote:
>> Binary for Linux available here for download:
>> https://home.apache.org/~ardovm/openoffice/linux/openoffice4-serf2curl-2022-04-28-installed.tar.bz2
>> I tried "Open" and entered a https address, that I know is password
>> protected.
>>
>> The current trunk would ask for the password. I got an error message
>> instead:
>>
 Nonexistent object.
 Nonexistent file.
> Same problem under Linux. Maybe it only works on 42X?
>
> I could not test under Windows because there aren't any Windows Dev builds 
> (42X or any other) available at
> https://nightlies.apache.org/openoffice/install/
> so I can't replace the DLLs

Yes, the Windows buildbot is broken for some quite time (looks like
Cygwin was updated without modifying awk/gawk).

But I didn't have the time to open a JIRA ticket and apparently nobody
else is interested in the Windows buildbots.

Regards,

   Matthias

>
> Best,
> Pedro
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@openoffice.apache.org
> For additional commands, e-mail: dev-h...@openoffice.apache.org
>



smime.p7s
Description: S/MIME Cryptographic Signature

Re: WebDAV module ported from serf to curl; curl using openssl and zlib (was: Re: Openssl, serf and curl)

[Damjan Jovanovic]

On Thu, Apr 28, 2022 at 6:12 PM Arrigo Marchiori 
wrote:

> Dear Damjan, All,
>
> On Tue, Apr 26, 2022 at 07:56:22PM +0200, Damjan Jovanovic wrote:
>
> > On Mon, Nov 15, 2021 at 9:57 PM Jim Jagielski  wrote:
> >
> > > I'm gonna look into the serf->(lib)curl option... Since we don't use
> any
> > > of the fancy features of serf, I'm thinking that the easy option might
> be
> > > best
> >
> >
> >
> > Hi
> >
> > I've ported our WebDAV content provider module from Serf to Curl.
>
> Binary for Linux available here for download:
>
> https://home.apache.org/~ardovm/openoffice/linux/openoffice4-serf2curl-2022-04-28-installed.tar.bz2
>
> I understand from your previous message that you don't really like
> GitHub, so I am writing here.
>
> [...]
> > STATUS
> >
> > It builds and works well on FreeBSD and Windows.
> >
> > Most of the code was reused, and all the operations and semantics
> > previously present with Serf, should have been preserved.
> >
> > Browsing WebDAV files and directories, loading files, overwriting them
> > ("Save"), creating them ("Save As"), renaming and deleting them, all
> works.
>
> I am testing the binary for Linux linked above.
>
> I tried "Open" and entered a https address, that I know is password
> protected.
>
> The current trunk would ask for the password. I got an error message
> instead:
>
> > > Nonexistent object.
> > > Nonexistent file.
>
> The address I tried to open is in the form https://host.domain:port/
>
> I tried to substitute "https" with "davs" and I got the same error.
>
> Maybe something is going wrong in the Linux build?
>
> I will now begin recompiling with debugging symbols enabled. Please
> let me know how I can help.
>

That's not good :(.

Set your macro security to "Medium", open the spreadsheet I've attached,
and run the "RunMe" Basic macro. That should enable logging to the console
at the finest level of detail. Then exit and re-run AOO like this:
soffice 2>&1 | tee /tmp/log.txt
and examine the console output interactively as you use AOO, and/or check
the log file afterwards. It should have everything, our logging, HTTP
request and response headers and bodies, Curl's internal logging.

If you can't see anything obviously wrong, send me that log file, but audit
it carefully first, it will probably contain your password in plaintext!


>
> [...]
> > P.S. APACHE 2.4 SETUP FOR TESTING
> [...]
>
> I still have to try this. Thank you for this tutorial!!
>
>
Pleasure :).


Macros.ods
Description: application/vnd.oasis.opendocument.spreadsheet

-
To unsubscribe, e-mail: dev-unsubscr...@openoffice.apache.org
For additional commands, e-mail: dev-h...@openoffice.apache.org

Re: WebDAV module ported from serf to curl; curl using openssl and zlib (was: Re: Openssl, serf and curl)

[Pedro Lino]

Hi Damjan, Arrigo, all

> On 04/28/2022 5:11 PM Arrigo Marchiori  wrote:

> Binary for Linux available here for download:
> https://home.apache.org/~ardovm/openoffice/linux/openoffice4-serf2curl-2022-04-28-installed.tar.bz2

> I tried "Open" and entered a https address, that I know is password
> protected.
> 
> The current trunk would ask for the password. I got an error message
> instead:
> 
> > > Nonexistent object.
> > > Nonexistent file.

Same problem under Linux. Maybe it only works on 42X?

I could not test under Windows because there aren't any Windows Dev builds (42X 
or any other) available at
https://nightlies.apache.org/openoffice/install/
so I can't replace the DLLs

Best,
Pedro

-
To unsubscribe, e-mail: dev-unsubscr...@openoffice.apache.org
For additional commands, e-mail: dev-h...@openoffice.apache.org

Re: WebDAV module ported from serf to curl; curl using openssl and zlib (was: Re: Openssl, serf and curl)

[Arrigo Marchiori]

Dear Damjan, All,

On Tue, Apr 26, 2022 at 07:56:22PM +0200, Damjan Jovanovic wrote:

> On Mon, Nov 15, 2021 at 9:57 PM Jim Jagielski  wrote:
> 
> > I'm gonna look into the serf->(lib)curl option... Since we don't use any
> > of the fancy features of serf, I'm thinking that the easy option might be
> > best
> 
> 
> 
> Hi
> 
> I've ported our WebDAV content provider module from Serf to Curl.

Binary for Linux available here for download:
https://home.apache.org/~ardovm/openoffice/linux/openoffice4-serf2curl-2022-04-28-installed.tar.bz2

I understand from your previous message that you don't really like
GitHub, so I am writing here.

[...]
> STATUS
> 
> It builds and works well on FreeBSD and Windows.
> 
> Most of the code was reused, and all the operations and semantics
> previously present with Serf, should have been preserved.
> 
> Browsing WebDAV files and directories, loading files, overwriting them
> ("Save"), creating them ("Save As"), renaming and deleting them, all works.

I am testing the binary for Linux linked above.

I tried "Open" and entered a https address, that I know is password
protected.

The current trunk would ask for the password. I got an error message
instead:

> > Nonexistent object.
> > Nonexistent file.

The address I tried to open is in the form https://host.domain:port/

I tried to substitute "https" with "davs" and I got the same error.

Maybe something is going wrong in the Linux build?

I will now begin recompiling with debugging symbols enabled. Please
let me know how I can help.

[...]
> P.S. APACHE 2.4 SETUP FOR TESTING
[...]

I still have to try this. Thank you for this tutorial!!

Best regards,
-- 
Arrigo

-
To unsubscribe, e-mail: dev-unsubscr...@openoffice.apache.org
For additional commands, e-mail: dev-h...@openoffice.apache.org

Re: WebDAV module ported from serf to curl; curl using openssl and zlib (was: Re: Openssl, serf and curl)

[Arrigo Marchiori]

Dear All,

first of all thank you Damjan, for this huge work!
It looks... wonderful.

On Wed, Apr 27, 2022 at 07:11:14PM +0200, Damjan Jovanovic wrote:

> Thank you Matthias.
> 
> Not too happy to use Github, but here is the pull request:
> https://github.com/apache/openoffice/pull/146

This is much helpful, thanks.

I see that my review was requested. I need a couple of days to check
it builds fine under Linux. Matthias, will you please check under
Windows?

Jim, could you please check MacOS? I cannot build trunk on that
platform yet.

I am not asking because I don't trust Damjan, but just to suggest we
honestly hunt for ``works on my machine'' effects ;-)
The commit is... big and IMHO we shall not hurry.

Best regards.

> > On Mon, Nov 15, 2021 at 9:57 PM Jim Jagielski  wrote:
> >
> >> I'm gonna look into the serf->(lib)curl option... Since we don't use any
> >> of the fancy features of serf, I'm thinking that the easy option might be
> >> best
> >
> >
> >
> > Hi
> >
> > I've ported our WebDAV content provider module from Serf to Curl.

[...]

-- 
Arrigo

-
To unsubscribe, e-mail: dev-unsubscr...@openoffice.apache.org
For additional commands, e-mail: dev-h...@openoffice.apache.org

Re: WebDAV module ported from serf to curl; curl using openssl and zlib (was: Re: Openssl, serf and curl)

[Damjan Jovanovic]

On Wed, Apr 27, 2022 at 9:22 PM Matthias Seidel 
wrote:

> Hi,
>
> Am 27.04.22 um 21:09 schrieb Damjan Jovanovic:
> > On Wed, Apr 27, 2022 at 8:07 PM Matthias Seidel <
> matthias.sei...@hamburg.de>
> > wrote:
> >
> >> Hi Damjan,
> >>
> >> Am 27.04.22 um 19:11 schrieb Damjan Jovanovic:
> >>> Thank you Matthias.
> >>>
> >>> Not too happy to use Github, but here is the pull request:
> >>> https://github.com/apache/openoffice/pull/146
> >> Thanks, Git makes it a lot easier for me to build...
> >>
> >> BTW: Is it OK when I cherry-pick your other two commits to AOO42X?
> >>
> >>
> > Which ones? The Python 3.8 and the syslog commits?
> Yes
> >
>
>
Sure, cherry-pick them. The IDL file for the syslog patch needs to be
changed to say "since 4.2" though, instead of the current 4.5.

Re: WebDAV module ported from serf to curl; curl using openssl and zlib (was: Re: Openssl, serf and curl)

[Matthias Seidel]

Hi,

Am 27.04.22 um 21:09 schrieb Damjan Jovanovic:
> On Wed, Apr 27, 2022 at 8:07 PM Matthias Seidel 
> wrote:
>
>> Hi Damjan,
>>
>> Am 27.04.22 um 19:11 schrieb Damjan Jovanovic:
>>> Thank you Matthias.
>>>
>>> Not too happy to use Github, but here is the pull request:
>>> https://github.com/apache/openoffice/pull/146
>> Thanks, Git makes it a lot easier for me to build...
>>
>> BTW: Is it OK when I cherry-pick your other two commits to AOO42X?
>>
>>
> Which ones? The Python 3.8 and the syslog commits?
Yes
>



smime.p7s
Description: S/MIME Cryptographic Signature

Re: WebDAV module ported from serf to curl; curl using openssl and zlib (was: Re: Openssl, serf and curl)

[Damjan Jovanovic]

On Wed, Apr 27, 2022 at 8:07 PM Matthias Seidel 
wrote:

> Hi Damjan,
>
> Am 27.04.22 um 19:11 schrieb Damjan Jovanovic:
> > Thank you Matthias.
> >
> > Not too happy to use Github, but here is the pull request:
> > https://github.com/apache/openoffice/pull/146
>
> Thanks, Git makes it a lot easier for me to build...
>
> BTW: Is it OK when I cherry-pick your other two commits to AOO42X?
>
>
Which ones? The Python 3.8 and the syslog commits?

Re: WebDAV module ported from serf to curl; curl using openssl and zlib (was: Re: Openssl, serf and curl)

[Matthias Seidel]

Hi Damjan,

Am 27.04.22 um 19:11 schrieb Damjan Jovanovic:
> Thank you Matthias.
>
> Not too happy to use Github, but here is the pull request:
> https://github.com/apache/openoffice/pull/146

Thanks, Git makes it a lot easier for me to build...

BTW: Is it OK when I cherry-pick your other two commits to AOO42X?

Regards,

   Matthias

>
>
>
> On Wed, Apr 27, 2022 at 5:44 PM Matthias Seidel 
> wrote:
>
>> Hi Damjan,
>>
>> Sounds great!
>>
>> Can we have that as a Pull Request?
>>
>> Regards,
>>
>>Matthias
>> Am 26.04.22 um 19:56 schrieb Damjan Jovanovic:
>>
>> On Mon, Nov 15, 2021 at 9:57 PM Jim Jagielski  wrote:
>>
>>> I'm gonna look into the serf->(lib)curl option... Since we don't use any
>>> of the fancy features of serf, I'm thinking that the easy option might be
>>> best
>>
>>
>> Hi
>>
>> I've ported our WebDAV content provider module from Serf to Curl.
>>
>> While it ended well, and several other bugs were found and fixed, it
>> definitely wasn't the "easy option" Jim ;). Starting with conservative
>> changes, it ended up needing total restructuring, and became more of a
>> rewrite. The crashes were frequent and hung connections many, and I had to
>> read up on the HTTP protocol, and read Curl and Serf's source code, but
>> eventually I prevailed, and a clean elegant stable Curl WebDAV module
>> emerged.
>>
>> The huge patch is attached for anyone wishing to review and test. Unless
>> there are major objections, I'll push it in a couple of days.
>>
>> STATUS
>>
>> It builds and works well on FreeBSD and Windows.
>>
>> Most of the code was reused, and all the operations and semantics
>> previously present with Serf, should have been preserved.
>>
>> Browsing WebDAV files and directories, loading files, overwriting them
>> ("Save"), creating them ("Save As"), renaming and deleting them, all works.
>>
>> HTTP and HTTPS proxies work. Unlike Serf, Curl could also support SOCKS
>> proxies (with minimal further changes), but AOO lacks that setting in the
>> proxy UI and configuration.
>>
>> Authentication works, both to the destination server and to the proxy
>> server. I've successfully tested Basic and Digest authentication. Curl
>> supports every authentication method Serf does and more.
>>
>> HTTPS works, with a custom certificate verification function, using our
>> own certificate store from NSS and its API (like the Serf code used). A bug
>> was discovered (which is in the Serf implementation too) where self-signed
>> certificates were being unconditionally rejected; apparently NSS wants to
>> see that a copy of that certificate  in its certificate chain parameter
>> too. Now they work, and the user gets prompted to allow access.
>>
>> HTTPS and authentication can be used together on the same connection and
>> work well, both bringing up their UI dialogs as needed.
>>
>> A bug was fixed where when username and password were both present in the
>> URL (dav://user:pass@host/path), the code was trying to split them at the
>> "@" instead of ":".
>>
>> Unnecessary base64 encoding and decoding was removed, when importing the
>> TLS connection's certificates into our XSecurityEnvironment. They now come
>> in directly as ASN.1 DER, and still work.
>>
>> The code was greatly restructured and cleaned up, as Curl's API is
>> synchronous and blocking, with parameters set in advance instead of through
>> many callbacks, which has allowed using short clear methods, and clean
>> separation between the session and request classes. The WebDAV content
>> provider has shrunk from 35 to 21 C++ files, 43 to 29 header files, and
>> 19129 to 15991 lines of code. With WebDAV methods centralized and
>> refactored into only 10-20 lines of code each, instead of scattered across
>> 4 files, it is much more understandable and maintainable now.
>>
>> Curl is vastly more portable than Serf. We should build easily now even on
>> OS/2. We can remain with existing build tools instead of needing scons or
>> cmake just to build Serf.
>>
>> 3 now unused dependencies were removed: apr, apr-util, and serf. Serf
>> isn't so bad. APR's pool idea is an ingenious way of doing resource
>> management in C. However Curl has excellent documentation, guides,
>> examples, and detailed explanations and even example code for each setting,
>> while Serf has no documentation. Serf might be worth it in a project that
>> already uses APR a lot, but we don't.
>>
>> Instead of the historical, crippled forms of logging like OSL_TRACE(),
>> which don't appear in release builds, I've made it use the newer
>> com.sun.star.logging UNO component (wrapped in comphelper::EventLogger),
>> which was inspired by java.util.logging, with configurable verbosity
>> levels, handlers (file and console) and output formats (plain, csv), and
>> importantly, which produces output in release builds too. I've also made it
>> so that on LogLevel::FINEST, Curl's verbose mode is enabled and Curl's
>> debug output is also logged through us, with descriptions of what Curl is
>> d

Re: WebDAV module ported from serf to curl; curl using openssl and zlib (was: Re: Openssl, serf and curl)

[Marcus]

Am 27.04.22 um 17:44 schrieb Matthias Seidel:

Sounds great!


I also think that it's a great progress.

As none-developer I cannot really judge how good and valuable the 
changes are. But all in all I'm happy to see this refactored to have 
less problems.


Well done. :-)

Marcus




Am 26.04.22 um 19:56 schrieb Damjan Jovanovic:
On Mon, Nov 15, 2021 at 9:57 PM Jim Jagielski > wrote:


I'm gonna look into the serf->(lib)curl option... Since we don't
use any of the fancy features of serf, I'm thinking that the easy
option might be best



Hi

I've ported our WebDAV content provider module from Serf to Curl.

While it ended well, and several other bugs were found and fixed, it 
definitely wasn't the "easy option" Jim ;). Starting with conservative 
changes, it ended up needing total restructuring, and became more of a 
rewrite. The crashes were frequent and hung connections many, and I 
had to read up on the HTTP protocol, and read Curl and Serf's source 
code, but eventually I prevailed, and a clean elegant stable Curl 
WebDAV module emerged.


The huge patch is attached for anyone wishing to review and test. 
Unless there are major objections, I'll push it in a couple of days.


STATUS

It builds and works well on FreeBSD and Windows.

Most of the code was reused, and all the operations and semantics 
previously present with Serf, should have been preserved.


Browsing WebDAV files and directories, loading files, overwriting them 
("Save"), creating them ("Save As"), renaming and deleting them, all 
works.


HTTP and HTTPS proxies work. Unlike Serf, Curl could also support 
SOCKS proxies (with minimal further changes), but AOO lacks that 
setting in the proxy UI and configuration.


Authentication works, both to the destination server and to the proxy 
server. I've successfully tested Basic and Digest authentication. Curl 
supports every authentication method Serf does and more.


HTTPS works, with a custom certificate verification function, using 
our own certificate store from NSS and its API (like the Serf code 
used). A bug was discovered (which is in the Serf implementation too) 
where self-signed certificates were being unconditionally rejected; 
apparently NSS wants to see that a copy of that certificate  in its 
certificate chain parameter too. Now they work, and the user gets 
prompted to allow access.


HTTPS and authentication can be used together on the same connection 
and work well, both bringing up their UI dialogs as needed.


A bug was fixed where when username and password were both present in 
the URL (dav://user:pass@host/path), the code was trying to split them 
at the "@" instead of ":".


Unnecessary base64 encoding and decoding was removed, when importing 
the TLS connection's certificates into our XSecurityEnvironment. They 
now come in directly as ASN.1 DER, and still work.


The code was greatly restructured and cleaned up, as Curl's API is 
synchronous and blocking, with parameters set in advance instead of 
through many callbacks, which has allowed using short clear methods, 
and clean separation between the session and request classes. The 
WebDAV content provider has shrunk from 35 to 21 C++ files, 43 to 29 
header files, and 19129 to 15991 lines of code. With WebDAV methods 
centralized and refactored into only 10-20 lines of code each, instead 
of scattered across 4 files, it is much more understandable and 
maintainable now.


Curl is vastly more portable than Serf. We should build easily now 
even on OS/2. We can remain with existing build tools instead of 
needing scons or cmake just to build Serf.


3 now unused dependencies were removed: apr, apr-util, and serf. Serf 
isn't so bad. APR's pool idea is an ingenious way of doing resource 
management in C. However Curl has excellent documentation, guides, 
examples, and detailed explanations and even example code for each 
setting, while Serf has no documentation. Serf might be worth it in a 
project that already uses APR a lot, but we don't.


Instead of the historical, crippled forms of logging like OSL_TRACE(), 
which don't appear in release builds, I've made it use the newer 
com.sun.star.logging UNO component (wrapped in 
comphelper::EventLogger), which was inspired by java.util.logging, 
with configurable verbosity levels, handlers (file and console) and 
output formats (plain, csv), and importantly, which produces output in 
release builds too. I've also made it so that on LogLevel::FINEST, 
Curl's verbose mode is enabled and Curl's debug output is also logged 
through us, with descriptions of what Curl is doing, and logs of all 
HTTP traffic including headers and bodies, before encryption and after 
decryption in the case of HTTPS, giving us tremendous detail that can 
be used for troubleshooting problems.


CURL CHANGED TO USE OPENSSL AND ZLIB

Curl only supports the custom TLS certificate verification function we 
use (the CURLOPT_SSL_CTX_FUNCTION option) when built with Ope

Re: WebDAV module ported from serf to curl; curl using openssl and zlib (was: Re: Openssl, serf and curl)

[Damjan Jovanovic]

Thank you Matthias.

Not too happy to use Github, but here is the pull request:
https://github.com/apache/openoffice/pull/146



On Wed, Apr 27, 2022 at 5:44 PM Matthias Seidel 
wrote:

> Hi Damjan,
>
> Sounds great!
>
> Can we have that as a Pull Request?
>
> Regards,
>
>Matthias
> Am 26.04.22 um 19:56 schrieb Damjan Jovanovic:
>
> On Mon, Nov 15, 2021 at 9:57 PM Jim Jagielski  wrote:
>
>> I'm gonna look into the serf->(lib)curl option... Since we don't use any
>> of the fancy features of serf, I'm thinking that the easy option might be
>> best
>
>
>
> Hi
>
> I've ported our WebDAV content provider module from Serf to Curl.
>
> While it ended well, and several other bugs were found and fixed, it
> definitely wasn't the "easy option" Jim ;). Starting with conservative
> changes, it ended up needing total restructuring, and became more of a
> rewrite. The crashes were frequent and hung connections many, and I had to
> read up on the HTTP protocol, and read Curl and Serf's source code, but
> eventually I prevailed, and a clean elegant stable Curl WebDAV module
> emerged.
>
> The huge patch is attached for anyone wishing to review and test. Unless
> there are major objections, I'll push it in a couple of days.
>
> STATUS
>
> It builds and works well on FreeBSD and Windows.
>
> Most of the code was reused, and all the operations and semantics
> previously present with Serf, should have been preserved.
>
> Browsing WebDAV files and directories, loading files, overwriting them
> ("Save"), creating them ("Save As"), renaming and deleting them, all works.
>
> HTTP and HTTPS proxies work. Unlike Serf, Curl could also support SOCKS
> proxies (with minimal further changes), but AOO lacks that setting in the
> proxy UI and configuration.
>
> Authentication works, both to the destination server and to the proxy
> server. I've successfully tested Basic and Digest authentication. Curl
> supports every authentication method Serf does and more.
>
> HTTPS works, with a custom certificate verification function, using our
> own certificate store from NSS and its API (like the Serf code used). A bug
> was discovered (which is in the Serf implementation too) where self-signed
> certificates were being unconditionally rejected; apparently NSS wants to
> see that a copy of that certificate  in its certificate chain parameter
> too. Now they work, and the user gets prompted to allow access.
>
> HTTPS and authentication can be used together on the same connection and
> work well, both bringing up their UI dialogs as needed.
>
> A bug was fixed where when username and password were both present in the
> URL (dav://user:pass@host/path), the code was trying to split them at the
> "@" instead of ":".
>
> Unnecessary base64 encoding and decoding was removed, when importing the
> TLS connection's certificates into our XSecurityEnvironment. They now come
> in directly as ASN.1 DER, and still work.
>
> The code was greatly restructured and cleaned up, as Curl's API is
> synchronous and blocking, with parameters set in advance instead of through
> many callbacks, which has allowed using short clear methods, and clean
> separation between the session and request classes. The WebDAV content
> provider has shrunk from 35 to 21 C++ files, 43 to 29 header files, and
> 19129 to 15991 lines of code. With WebDAV methods centralized and
> refactored into only 10-20 lines of code each, instead of scattered across
> 4 files, it is much more understandable and maintainable now.
>
> Curl is vastly more portable than Serf. We should build easily now even on
> OS/2. We can remain with existing build tools instead of needing scons or
> cmake just to build Serf.
>
> 3 now unused dependencies were removed: apr, apr-util, and serf. Serf
> isn't so bad. APR's pool idea is an ingenious way of doing resource
> management in C. However Curl has excellent documentation, guides,
> examples, and detailed explanations and even example code for each setting,
> while Serf has no documentation. Serf might be worth it in a project that
> already uses APR a lot, but we don't.
>
> Instead of the historical, crippled forms of logging like OSL_TRACE(),
> which don't appear in release builds, I've made it use the newer
> com.sun.star.logging UNO component (wrapped in comphelper::EventLogger),
> which was inspired by java.util.logging, with configurable verbosity
> levels, handlers (file and console) and output formats (plain, csv), and
> importantly, which produces output in release builds too. I've also made it
> so that on LogLevel::FINEST, Curl's verbose mode is enabled and Curl's
> debug output is also logged through us, with descriptions of what Curl is
> doing, and logs of all HTTP traffic including headers and bodies, before
> encryption and after decryption in the case of HTTPS, giving us tremendous
> detail that can be used for troubleshooting problems.
>
> CURL CHANGED TO USE OPENSSL AND ZLIB
>
> Curl only supports the custom TLS certificate verification functio

Re: WebDAV module ported from serf to curl; curl using openssl and zlib (was: Re: Openssl, serf and curl)

[Matthias Seidel]

Hi Damjan,

Sounds great!

Can we have that as a Pull Request?

Regards,

   Matthias

Am 26.04.22 um 19:56 schrieb Damjan Jovanovic:
> On Mon, Nov 15, 2021 at 9:57 PM Jim Jagielski  > wrote:
>
> I'm gonna look into the serf->(lib)curl option... Since we don't
> use any of the fancy features of serf, I'm thinking that the easy
> option might be best
>
>
>
> Hi
>
> I've ported our WebDAV content provider module from Serf to Curl.
>
> While it ended well, and several other bugs were found and fixed, it
> definitely wasn't the "easy option" Jim ;). Starting with conservative
> changes, it ended up needing total restructuring, and became more of a
> rewrite. The crashes were frequent and hung connections many, and I
> had to read up on the HTTP protocol, and read Curl and Serf's source
> code, but eventually I prevailed, and a clean elegant stable Curl
> WebDAV module emerged.
>
> The huge patch is attached for anyone wishing to review and test.
> Unless there are major objections, I'll push it in a couple of days.
>
> STATUS
>
> It builds and works well on FreeBSD and Windows.
>
> Most of the code was reused, and all the operations and semantics
> previously present with Serf, should have been preserved.
>
> Browsing WebDAV files and directories, loading files, overwriting them
> ("Save"), creating them ("Save As"), renaming and deleting them, all
> works.
>
> HTTP and HTTPS proxies work. Unlike Serf, Curl could also support
> SOCKS proxies (with minimal further changes), but AOO lacks that
> setting in the proxy UI and configuration.
>
> Authentication works, both to the destination server and to the proxy
> server. I've successfully tested Basic and Digest authentication. Curl
> supports every authentication method Serf does and more.
>
> HTTPS works, with a custom certificate verification function, using
> our own certificate store from NSS and its API (like the Serf code
> used). A bug was discovered (which is in the Serf implementation too)
> where self-signed certificates were being unconditionally rejected;
> apparently NSS wants to see that a copy of that certificate  in its
> certificate chain parameter too. Now they work, and the user gets
> prompted to allow access.
>
> HTTPS and authentication can be used together on the same connection
> and work well, both bringing up their UI dialogs as needed.
>
> A bug was fixed where when username and password were both present in
> the URL (dav://user:pass@host/path), the code was trying to split them
> at the "@" instead of ":".
>
> Unnecessary base64 encoding and decoding was removed, when importing
> the TLS connection's certificates into our XSecurityEnvironment. They
> now come in directly as ASN.1 DER, and still work.
>
> The code was greatly restructured and cleaned up, as Curl's API is
> synchronous and blocking, with parameters set in advance instead of
> through many callbacks, which has allowed using short clear methods,
> and clean separation between the session and request classes. The
> WebDAV content provider has shrunk from 35 to 21 C++ files, 43 to 29
> header files, and 19129 to 15991 lines of code. With WebDAV methods
> centralized and refactored into only 10-20 lines of code each, instead
> of scattered across 4 files, it is much more understandable and
> maintainable now.
>
> Curl is vastly more portable than Serf. We should build easily now
> even on OS/2. We can remain with existing build tools instead of
> needing scons or cmake just to build Serf.
>
> 3 now unused dependencies were removed: apr, apr-util, and serf. Serf
> isn't so bad. APR's pool idea is an ingenious way of doing resource
> management in C. However Curl has excellent documentation, guides,
> examples, and detailed explanations and even example code for each
> setting, while Serf has no documentation. Serf might be worth it in a
> project that already uses APR a lot, but we don't.
>
> Instead of the historical, crippled forms of logging like OSL_TRACE(),
> which don't appear in release builds, I've made it use the newer
> com.sun.star.logging UNO component (wrapped in
> comphelper::EventLogger), which was inspired by java.util.logging,
> with configurable verbosity levels, handlers (file and console) and
> output formats (plain, csv), and importantly, which produces output in
> release builds too. I've also made it so that on LogLevel::FINEST,
> Curl's verbose mode is enabled and Curl's debug output is also logged
> through us, with descriptions of what Curl is doing, and logs of all
> HTTP traffic including headers and bodies, before encryption and after
> decryption in the case of HTTPS, giving us tremendous detail that can
> be used for troubleshooting problems.
>
> CURL CHANGED TO USE OPENSSL AND ZLIB
>
> Curl only supports the custom TLS certificate verification function we
> use (the CURLOPT_SSL_CTX_FUNCTION option) when built with OpenSSL,
> wolfSSL or mbedTLS providers. We currently use schannel on Windows
> instead, which

Re: WebDAV module ported from serf to curl; curl using openssl and zlib (was: Re: Openssl, serf and curl)

[Damjan Jovanovic]

Hi Pedro

That's great.

I think all you need to do to test on Windows is replace libcurl.dll and
ucpdav1.dll with the copies in the ZIP file you download from here:
https://drive.google.com/file/d/1YnpCr4woe8laah4nxtgzGfuEfGpVrf9L/view?usp=sharing

Regards
Damjan


On Tue, Apr 26, 2022 at 8:20 PM Pedro Lino 
wrote:

>
> Hi Damian
>
>
>
> I use Webdav almost on a daily basis.
> Looking forward to testing a new build under Linux as soon as you push it
> to 42X
>
> Can you share your working binaries for Windows?
>
>
>
> Thanks!
>
> Pedro
> > On 04/26/2022 6:56 PM Damjan Jovanovic  wrote:
> >
> >
> >
> >
> >
> >
> > On Mon, Nov 15, 2021 at 9:57 PM Jim Jagielski  j...@jagunet.com>> wrote:
> > > I'm gonna look into the serf->(lib)curl option... Since we don't use
> any of the fancy features of serf, I'm thinking that the easy option might
> be best
> > >
> > >
> > > Hi
> > >
> > > I've ported our WebDAV content provider module from Serf to Curl.
> > >
> > > While it ended well, and several other bugs were found and fixed, it
> definitely wasn't the "easy option" Jim ;). Starting with conservative
> changes, it ended up needing total restructuring, and became more of a
> rewrite. The crashes were frequent and hung connections many, and I had to
> read up on the HTTP protocol, and read Curl and Serf's source code, but
> eventually I prevailed, and a clean elegant stable Curl WebDAV module
> emerged.
> > >
> > > The huge patch is attached for anyone wishing to review and test.
> Unless there are major objections, I'll push it in a couple of days.
> > >
> > > STATUS
> > >
> > > It builds and works well on FreeBSD and Windows.
> > >
> > > Most of the code was reused, and all the operations and semantics
> previously present with Serf, should have been preserved.
> > >
> > > Browsing WebDAV files and directories, loading files, overwriting them
> ("Save"), creating them ("Save As"), renaming and deleting them, all works.
> > >
> > > HTTP and HTTPS proxies work. Unlike Serf, Curl could also support
> SOCKS proxies (with minimal further changes), but AOO lacks that setting in
> the proxy UI and configuration.
> > >
> > > Authentication works, both to the destination server and to the proxy
> server. I've successfully tested Basic and Digest authentication. Curl
> supports every authentication method Serf does and more.
> > >
> > > HTTPS works, with a custom certificate verification function, using
> our own certificate store from NSS and its API (like the Serf code used). A
> bug was discovered (which is in the Serf implementation too) where
> self-signed certificates were being unconditionally rejected; apparently
> NSS wants to see that a copy of that certificate  in its certificate chain
> parameter too. Now they work, and the user gets prompted to allow access.
> > >
> > >
> > > HTTPS and authentication can be used together on the same connection
> and work well, both bringing up their UI dialogs as needed.
> > >
> > > A bug was fixed where when username and password were both present in
> the URL (dav://user:pass@host/path), the code was trying to split them at
> the "@" instead of ":".
> > >
> > > Unnecessary base64 encoding and decoding was removed, when importing
> the TLS connection's certificates into our XSecurityEnvironment. They now
> come in directly as ASN.1 DER, and still work.
> > >
> > > The code was greatly restructured and cleaned up, as Curl's API is
> synchronous and blocking, with parameters set in advance instead of through
> many callbacks, which has allowed using short clear methods, and clean
> separation between the session and request classes. The WebDAV content
> provider has shrunk from 35 to 21 C++ files, 43 to 29 header files, and
> 19129 to 15991 lines of code. With WebDAV methods centralized and
> refactored into only 10-20 lines of code each, instead of scattered across
> 4 files, it is much more understandable and maintainable now.
> > >
> > > Curl is vastly more portable than Serf. We should build easily now
> even on OS/2. We can remain with existing build tools instead of needing
> scons or cmake just to build Serf.
> > >
> > > 3 now unused dependencies were removed: apr, apr-util, and serf. Serf
> isn't so bad. APR's pool idea is an ingenious way of doing resource
> management in C. However Curl has excellent documentation, guides,
> examples, and detailed explanations and even example code for each setting,
> while Serf has no documentation. Serf might be worth it in a project that
> already uses APR a lot, but we don't.
> > >
> > > Instead of the historical, crippled forms of logging like OSL_TRACE(),
> which don't appear in release builds, I've made it use the newer
> com.sun.star.logging UNO component (wrapped in comphelper::EventLogger),
> which was inspired by java.util.logging, with configurable verbosity
> levels, handlers (file and console) and output formats (plain, csv), and
> importantly, which produces output in release builds too. I've also made it
> so that on Lo

Re: WebDAV module ported from serf to curl; curl using openssl and zlib (was: Re: Openssl, serf and curl)

[Pedro Lino]

Hi Damian



I use Webdav almost on a daily basis.
Looking forward to testing a new build under Linux as soon as you push it to 42X

Can you share your working binaries for Windows?



Thanks!

Pedro
> On 04/26/2022 6:56 PM Damjan Jovanovic  wrote:
> 
> 
> 
> 
> 
> 
> On Mon, Nov 15, 2021 at 9:57 PM Jim Jagielski  > wrote:
> > I'm gonna look into the serf->(lib)curl option... Since we don't use any of 
> > the fancy features of serf, I'm thinking that the easy option might be best 
> > 
> > 
> > Hi
> > 
> > I've ported our WebDAV content provider module from Serf to Curl.
> > 
> > While it ended well, and several other bugs were found and fixed, it 
> > definitely wasn't the "easy option" Jim ;). Starting with conservative 
> > changes, it ended up needing total restructuring, and became more of a 
> > rewrite. The crashes were frequent and hung connections many, and I had to 
> > read up on the HTTP protocol, and read Curl and Serf's source code, but 
> > eventually I prevailed, and a clean elegant stable Curl WebDAV module 
> > emerged.
> > 
> > The huge patch is attached for anyone wishing to review and test. Unless 
> > there are major objections, I'll push it in a couple of days.
> > 
> > STATUS
> > 
> > It builds and works well on FreeBSD and Windows.
> > 
> > Most of the code was reused, and all the operations and semantics 
> > previously present with Serf, should have been preserved.
> > 
> > Browsing WebDAV files and directories, loading files, overwriting them 
> > ("Save"), creating them ("Save As"), renaming and deleting them, all works.
> > 
> > HTTP and HTTPS proxies work. Unlike Serf, Curl could also support SOCKS 
> > proxies (with minimal further changes), but AOO lacks that setting in the 
> > proxy UI and configuration.
> > 
> > Authentication works, both to the destination server and to the proxy 
> > server. I've successfully tested Basic and Digest authentication. Curl 
> > supports every authentication method Serf does and more.
> > 
> > HTTPS works, with a custom certificate verification function, using our own 
> > certificate store from NSS and its API (like the Serf code used). A bug was 
> > discovered (which is in the Serf implementation too) where self-signed 
> > certificates were being unconditionally rejected; apparently NSS wants to 
> > see that a copy of that certificate  in its certificate chain parameter 
> > too. Now they work, and the user gets prompted to allow access.
> > 
> > 
> > HTTPS and authentication can be used together on the same connection and 
> > work well, both bringing up their UI dialogs as needed.
> > 
> > A bug was fixed where when username and password were both present in the 
> > URL (dav://user:pass@host/path), the code was trying to split them at the 
> > "@" instead of ":".
> > 
> > Unnecessary base64 encoding and decoding was removed, when importing the 
> > TLS connection's certificates into our XSecurityEnvironment. They now come 
> > in directly as ASN.1 DER, and still work.
> > 
> > The code was greatly restructured and cleaned up, as Curl's API is 
> > synchronous and blocking, with parameters set in advance instead of through 
> > many callbacks, which has allowed using short clear methods, and clean 
> > separation between the session and request classes. The WebDAV content 
> > provider has shrunk from 35 to 21 C++ files, 43 to 29 header files, and 
> > 19129 to 15991 lines of code. With WebDAV methods centralized and 
> > refactored into only 10-20 lines of code each, instead of scattered across 
> > 4 files, it is much more understandable and maintainable now.
> > 
> > Curl is vastly more portable than Serf. We should build easily now even on 
> > OS/2. We can remain with existing build tools instead of needing scons or 
> > cmake just to build Serf.
> > 
> > 3 now unused dependencies were removed: apr, apr-util, and serf. Serf isn't 
> > so bad. APR's pool idea is an ingenious way of doing resource management in 
> > C. However Curl has excellent documentation, guides, examples, and detailed 
> > explanations and even example code for each setting, while Serf has no 
> > documentation. Serf might be worth it in a project that already uses APR a 
> > lot, but we don't.
> > 
> > Instead of the historical, crippled forms of logging like OSL_TRACE(), 
> > which don't appear in release builds, I've made it use the newer 
> > com.sun.star.logging UNO component (wrapped in comphelper::EventLogger), 
> > which was inspired by java.util.logging, with configurable verbosity 
> > levels, handlers (file and console) and output formats (plain, csv), and 
> > importantly, which produces output in release builds too. I've also made it 
> > so that on LogLevel::FINEST, Curl's verbose mode is enabled and Curl's 
> > debug output is also logged through us, with descriptions of what Curl is 
> > doing, and logs of all HTTP traffic including headers and bodies, before 
> > encryption and after decryption in the case of HTTP