Hi VPP community,
I'm running a custom VPP build with SSLO plugin and Nginx as reverse proxy
behind VCL/LDP, and I'm encountering several interrelated stability issues.
I'd appreciate any insight from the community on whether these are known
limitations or if there are workarounds I'm missing.
Architecture
Client ──TLS──→ [VPP SSLO plugin] ──→ loop0 ──→ Nginx-decrypt (VCL
namespace v10)
│
HTTP (cleartext)
│
↓
loop1 ──→
Nginx-encrypt (VCL namespace v20)
│
TLS re-encrypt
│
↓
[VPP SSLO plugin]
──TLS──→ Backend :443
- VPP: 23.06 custom build with SSLO plugin, DPDK 4-worker, 8G main heap
- Nginx-decrypt: TLS termination, bind to loop0 (172.16.102.213)
- Nginx-encrypt: receives cleartext HTTP on loop1 (172.16.102.214:9003),
proxy_pass https:// to backends
- Both Nginx instances: master_process off, worker_processes 1,
LD_PRELOAD libvcl_ldpreload.so
- VPP session: preallocated-sessions 500000, 4 worker threads
Issue 1: VCL epoll event notification delay (5+ seconds)
Symptom: Nginx-decrypt returns 504 Gateway Timeout to clients despite
backend responding promptly.
Evidence from SSLO-decrypted packet capture between decrypt and encrypt
Nginx:
- At T=42.058s: Backend sends ~100KB of HTTP response data (30+
segments, 1518 bytes each) within 50ms
- TCP ACKs are returned immediately at the VPP/TCP layer
(sub-millisecond)
- At T=47.110s: Backend sends FIN, ACK (5 seconds after the last data
segment)
- During this 5-second gap, no VCL events were delivered to
Nginx-encrypt's epoll
The backend clearly responded within ~8 seconds total, but the VCL layer
introduced a 5-second delay in delivering the readable-event to
Nginx-encrypt's epoll. This pushed the total round-trip past
Nginx-decrypt's proxy_read_timeout (8s), causing the 504.
VCL config for the affected Nginx-encrypt instance:
vcl {
huge_page
heapsize 512M
segment-size 268435456
add-segment-size 268435456
rx-fifo-size 524288
tx-fifo-size 524288
event-queue-size 100000
app-timeout 10.0
session-timeout 30.0
use-mq-eventfd
app-socket-api /run/vpp/app_ns_sockets/v20
namespace-id v20
}
The Nginx-decrypt instance uses event-queue-size 500000 and does not
exhibit the same delay.
Questions:
- Is event-queue-size 100000 too small for a proxy handling large HTTP
responses (30+ TCP segments per response)?
- Are there known VCL epoll delivery latency issues when the event queue
is near capacity?
- Would increasing rx-fifo-size help buffer bursty response data while
waiting for epoll events?
Issue 2: epoll_wait() returns no events without timeout (Nginx crashes)
Symptom: At least one Nginx instance crashes with:
[alert] PID#0: epoll_wait() returned no events without timeout
ldp_destructor:2913: LDP<PID>: LDP destructor: done!
This occurs under moderate load (estimated ~2000 QPS) after variable uptime
(minutes to hours). Both Nginx instances are affected, and the crash can
cascade when one side goes down.
This happens with both use-mq-eventfd enabled and disabled across different
test iterations.
Questions:
- Is this a known race condition in VCL's epoll simulation layer?
- Are there recommended VCL parameters (event-queue-size,
session-timeout, app-timeout) to mitigate this?
- Could the number of active sessions (TIME-WAIT from short-lived proxy
connections) correlate with this failure?
Issue 3: CLOSE-WAIT accumulation from missing EPOLLRDHUP
Symptom: vppctl show tcp stats shows thousands of timeout close-wait events,
with corresponding segments retransmitted in the hundreds of thousands.
After disabling upstream keepalive in Nginx (forcing short-lived
connections), close-wait timeouts dropped to near zero, but Nginx still
experienced epoll_wait failures (Issue 2).
Root cause appears to be: VCL does not reliably deliver EPOLLRDHUP to
Nginx's epoll queue when the remote peer sends FIN, causing Nginx to reuse
dead connections from the keepalive pool and VPP to accumulate CLOSE-WAIT
sessions that eventually timeout.
Questions:
- Is EPOLLRDHUP support in VCL considered production-ready?
- What is the recommended approach for Nginx + VCL with upstream
keepalive?
Issue 4: optname 31 (TCP_ULP / kTLS) unsupported in VCL setsockopt
Symptom: VCL debug log flooded with:
ERROR: fd XX: setsockopt() SOL_TCP: vlsh X optname 31 unsupported!
This is OpenSSL 3.x attempting setsockopt(fd, SOL_TCP, TCP_ULP, "tls", 3) to
enable kernel TLS (kTLS). VCL's setsockopt implementation has case
statements for TCP_KEEPIDLE, TCP_KEEPINTVL, TCP_CONGESTION, and TCP_CORK,
but not TCP_ULP (31), so it falls to the default branch which logs an error.
While setsockopt(TCP_ULP) failure is non-fatal (OpenSSL falls back to
user-space TLS), the high-frequency debug logging under load contributes to
unnecessary I/O. The fix would be to add case TCP_ULP: return
-EOPNOTSUPP; without
the debug log.
Environment
- VPP: 23.06 custom build with SSLO plugin
- Nginx: 1.28.0 (custom build with setsockopt SO_VCL_SET_PEER_INFO patch)
- OpenSSL: 3.x (linked with Nginx)
- OS: Ubuntu 22.04 LTS
- Kernel: 5.10.134
Summary
The common thread across all issues is VCL's epoll event delivery
reliability under load. Data arrives at VPP's TCP layer correctly (verified
by vppctl pcap trace showing clean ACK sequences and no retransmissions on
the VPP side), but the events are either delayed (Issue 1), lost entirely
(Issue 2), or missing specific flags (Issue 3) when delivered to the
application.
Are these known issues with specific VPP versions? Are there configuration
guidelines for tuning VCL for high-throughput Nginx reverse proxy workloads?
Thank you for any guidance.
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#27089): https://lists.fd.io/g/vpp-dev/message/27089
Mute This Topic: https://lists.fd.io/mt/120027879/21656
Group Owner: [email protected]
Unsubscribe: https://lists.fd.io/g/vpp-dev/leave/14379924/21656/631435203/xyzzy
[[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-
