I'm upgrading an Apache HTTP server of our SVN repos on RedHat Enterprise Linux
8. Using Subversion 1.14.1, svn checkout of even a small, simple repo with
about 150 files hangs about 90% of the time, crashes 5%, and succeeds 5%.
Given enough time, the hangs eventually time out after checking out much of the
repo. A debugger shows the following stack during the hang.
#0 epoll_wait /usr/lib64/libc.so.6
#1 impl_pollset_poll /usr/lib64/libapr-1.so.0
#2 serf_context_run /usr/lib64/libserf-1.so.0
#3 svn_ra_serf.context_run /usr/lib64/libsvn_ra_serf-1.so.0
#4 finish_report /usr/lib64/libsvn_ra_serf-1.so.0
#5 svn_wc_crawl_revisions5 /usr/lib64/libsvn_wc-1.so.0
#6 update_internal.isra /usr/lib64/libsvn_client-1.so.0
#7 svn_client.update_internal /usr/lib64/libsvn_client-1.so.0
#8 svn_client.checkout_internal /usr/lib64/libsvn_client-1.so.0
#9 svn_client_checkout3 /usr/lib64/libsvn_client-1.so.0
#10 svn_cl.checkout
#11 sub_main
#12 main
strace shows repeated calls to epoll_wait about 1 sec apart.
When the checkout crashes, it's a SIGSEGV with this stack,
#0 apr_pool_create_ex (libapr-1.so.0)
#1 svn_pool_create_ex (libsvn_subr-1.so.0)
#2 update_opened (libsvn_ra_serf-1.so.0)
#3 expat_start (libsvn_ra_serf-1.so.0)
#4 expat_start_handler (libsvn_subr-1.so.0)
#5 doContent (libexpat.so.1)
#6 contentProcessor (libexpat.so.1)
#7 XML_ParseBuffer (libexpat.so.1)
#8 svn_xml_parse (libsvn_subr-1.so.0)
#9 expat_response_handler (libsvn_ra_serf-1.so.0)
#10 process_buffer.isra.9 (libsvn_ra_serf-1.so.0)
#11 finish_report (libsvn_ra_serf-1.so.0)
#12 svn_wc_crawl_revisions5 (libsvn_wc-1.so.0)
#13 update_internal.isra.0 (libsvn_client-1.so.0)
#14 svn_client__update_internal (libsvn_client-1.so.0)
#15 svn_client__checkout_internal (libsvn_client-1.so.0)
#16 svn_client_checkout3 (libsvn_client-1.so.0)
#17 svn_cl__checkout (svn)
#18 sub_main (svn)
#19 main (svn)
#20 __libc_start_main (libc.so.6)
#21 _start (svn)
or this one,
#0 apr_allocator_alloc (libapr-1.so.0)
#1 serf_bucket_mem_alloc (libserf-1.so.0)
#2 serf_bucket_response_create (libserf-1.so.0)
#3 serf.process_connection (libserf-1.so.0)
#4 serf_event_trigger (libserf-1.so.0)
#5 serf_context_run (libserf-1.so.0)
#6 svn_ra_serf.context_run (libsvn_ra_serf-1.so.0)
#7 finish_report (libsvn_ra_serf-1.so.0)
#8 svn_wc_crawl_revisions5 (libsvn_wc-1.so.0)
#9 update_internal.isra (libsvn_client-1.so.0)
#10 svn_client.update_internal (libsvn_client-1.so.0)
#11 svn_client.checkout_internal (libsvn_client-1.so.0)
#12 svn_client_checkout3 (libsvn_client-1.so.0)
#13 svn_cl.checkout (svn)
#14 sub_main (svn)
#15 main (svn)
After a failure, I'm left with a half-checked out working copy with many locks.
I can complete it with svn cleanup and another svn checkout, but that's not
realistic for our CI/CD or general use. Server logs show no indication of a
problem; the server appears healthy.
I've tried a million things before submitting this bug report, read half a
million posts and searches, but haven't been able to get past this. I'd sure
appreciate any ideas you have on the way forward. Here's a bit more about my
system.
* svn, version 1.14.1 (r1886195)
* ra_svn : Module for accessing a repository using the svn network
protocol.
- with Cyrus SASL authentication
- handles 'svn' scheme
* ra_local : Module for accessing a repository on local disk.
- handles 'file' scheme
* ra_serf : Module for accessing a repository via WebDAV protocol using
serf.
- using serf 1.3.9 (compiled with 1.3.9)
- handles 'http' scheme
- handles 'https' scheme
The following authentication credential caches are available:
* Plaintext cache in /home/me/.subversion
* Gnome Keyring
* GPG-Agent
* svn 1.10.2 was failing the same way before we upgraded to 1.14.1 as a
possible fix.
* Checking out to a local disk succeeds more often, but still hangs and
crashes. Checking out to an NFS drive just makes it worse.
And here's more about our Apache.
* Server version: Apache/2.4.37 (Red Hat Enterprise Linux)
Server built: Aug 30 2023 11:01:53
Server's Module Magic Number: 20120211:83
Server loaded: APR 1.6.3, APR-UTIL 1.6.1
Compiled using: APR 1.6.3, APR-UTIL 1.6.1
Architecture: 64-bit
Server MPM: worker
threaded: yes (fixed thread count)
forked: yes (variable process count)
Server compiled with....
-D APR_HAS_SENDFILE
-D APR_HAS_MMAP
-D APR_HAVE_IPV6 (IPv4-mapped addresses enabled)
-D APR_USE_SYSVSEM_SERIALIZE
-D APR_USE_PTHREAD_SERIALIZE
-D SINGLE_LISTEN_UNSERIALIZED_ACCEPT
-D APR_HAS_OTHER_CHILD
-D AP_HAVE_RELIABLE_PIPED_LOGS
-D DYNAMIC_MODULE_LIMIT=256
-D HTTPD_ROOT="/etc/httpd"
-D SUEXEC_BIN="/usr/sbin/suexec"
-D DEFAULT_PIDLOG="run/httpd.pid"
-D DEFAULT_SCOREBOARD="logs/apache_runtime_status"
-D DEFAULT_ERRORLOG="logs/error_log"
-D AP_TYPES_CONFIG_FILE="conf/mime.types"
-D SERVER_CONFIG_FILE="conf/httpd.conf"
* Access to this server from a browser with SVN and ViewVC pages seems to
work.
* Authentication is over Kerberos with mod_auth_gssapi.
* Authorization uses AuthzSVNAccessFile and an access file.
* SSL is used with SSLCryptoDevice set to builtin, based on
this<https://lists.apache.org/thread/zysfq4cb0jkz59p0wkhfm49xwr8lj5to>.
* I've tried all three MPMs with no change, based on another post:
prefork, worker, and event.
* We've had Apache running on RedHat 6 with these repos for many years.
I'd be glad to provide additional details or run more tests. Thanks for any
ideas you have, and for supporting this software.
Jim
