Inline ...
On Tue, Nov 20, 2018 at 9:26 AM Silvio Bierman <[email protected]>
wrote:
> Hello Joakim,
>
> Firstly, this is the setup (Scala) code for the SslContextFactory etc.:
>
> /***** start code *****/
> val sslContextFactory = new SslContextFactory
>
> val ciphers = Array(
> "TLS_AES_128_GCM_SHA256", /* TLS 1.3 */
> "TLS_AES_256_GCM_SHA384", /* TLS 1.3 */
> "TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384",
> "TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256",
> "TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384",
> "TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256",
> "TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384",
> "TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256",
> "TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA", /* TLS 1.0 / 1.1 -
> IE7/8/9 - Android <= 4.3 */
> "TLS_DHE_DSS_WITH_AES_256_GCM_SHA384",
> "TLS_DHE_DSS_WITH_AES_128_GCM_SHA256",
> "TLS_DHE_DSS_WITH_AES_256_CBC_SHA256")
>
> sslContextFactory.setCipherComparator(HTTP2Cipher.COMPARATOR)
>
sslContextFactory.setUseCipherSuitesOrder(true)
>
Not relevant, anywhere anymore.
You can get an A+ rating on ssllabs without attempting to control cipher
suite order.
(Note: A+ rating is for all HTTP/1.x and HTTP/2 behaviors)
> sslContextFactory.setIncludeCipherSuites(ciphers:_*)
>
Don't include, exclude only.
You eliminate a ton of valid Cipher Suites doing it this way.
>
>
> sslContextFactory.setExcludeCipherSuites("SSL_RSA_WITH_DES_CBC_SHA",
> "SSL_DHE_RSA_WITH_DES_CBC_SHA",
> "SSL_DHE_DSS_WITH_DES_CBC_SHA",
> "SSL_RSA_EXPORT_WITH_RC4_40_MD5",
> "SSL_RSA_EXPORT_WITH_DES40_CBC_SHA",
> "SSL_DHE_RSA_EXPORT_WITH_DES40_CBC_SHA",
> "SSL_DHE_DSS_EXPORT_WITH_DES40_CBC_SHA")
>
Use defaults, none of those Cipher suites would be selected anyway due to
default Cipher Suite exclusions in SslContextFactory AND the JVM disabled
cipher suites.
https://github.com/eclipse/jetty.project/blob/jetty-9.4.14.v20181114/jetty-util/src/main/java/org/eclipse/jetty/util/ssl/SslContextFactory.java#L135-L148
/** Default Excluded Cipher Suite List */
private static final String[] DEFAULT_EXCLUDED_CIPHER_SUITES = {
// Exclude weak / insecure ciphers
"^.*_(MD5|SHA|SHA1)$",
// Exclude ciphers that don't support forward secrecy
"^TLS_RSA_.*$",
// The following exclusions are present to cleanup known bad
cipher
// suites that may be accidentally included via include
patterns.
// The default enabled cipher list in Java will not include
these
// (but they are available in the supported list).
"^SSL_.*$",
"^.*_NULL_.*$",
"^.*_anon_.*$"
};
>
> sslContextFactory.addExcludeProtocols("SSLv3")
>
This is the in the default excluded protocols values, and also
excluded/disabled at the JVM level, don't set it.
https://github.com/eclipse/jetty.project/blob/jetty-9.4.14.v20181114/jetty-util/src/main/java/org/eclipse/jetty/util/ssl/SslContextFactory.java#L132-L133
/** Default Excluded Protocols List */
private static final String[] DEFAULT_EXCLUDED_PROTOCOLS = {"SSL",
"SSLv2", "SSLv2Hello", "SSLv3"};
> sslContextFactory.setRenegotiationAllowed(false)
>
sslContextFactory.setEndpointIdentificationAlgorithm("HTTPS")
>
This is only relevant for a client initiating a connection to a destination
machine with TLS.
Comment out all of the above.
It screams old-school behaviors and techniques.
Use defaults for SslContextFactory when it comes to Cipher Suites and
Protocols, and don't attempt to support TLS/1.0 or TLS/1.1 Those protocols
are disabled in so many places (User-Agents, Browsers, OS level, Java VM
level, Server level, various hardware intermediates) now its pointless to
attempt to support them. If this is still important to you, make a
different connector for legacy behaviors.
> sslContextFactory.setKeyStorePath(keyStore)
> sslContextFactory.setKeyStorePassword(keyPassword)
> sslContextFactory.setKeyManagerPassword(keyPassword)
> sslContextFactory.setKeyStoreType(keyStoreType)
>
> val httpsConfig = new HttpConfiguration
> httpsConfig.setSecureScheme("https")
> httpsConfig.setSecurePort(port)
> httpsConfig.setSendXPoweredBy(false)
> httpsConfig.setSendServerVersion(false)
> httpsConfig.setOutputBufferSize(32768)
> httpsConfig.addCustomizer(new SecureRequestCustomizer)
> val http1 = new HttpConnectionFactory(httpsConfig)
> val connector =
> {
> if (httpVersion != "http/2")
> {
> val ssl = new
> SslConnectionFactory(sslContextFactory,"http/1.1")
> val http = new HttpConnectionFactory(httpsConfig)
> new ServerConnector(server,ssl,http)
> }
> else
> {
> val http2 = new HTTP2ServerConnectionFactory(httpsConfig)
> val alpn = new ALPNServerConnectionFactory
> alpn.setDefaultProtocol(http1.getProtocol)
>
Use alpn.setDefaultProtocol(http1.getDefaultProtocol())
val ssl = new
> SslConnectionFactory(sslContextFactory,alpn.getProtocol)
> new ServerConnector(server,ssl,alpn,http2,http1)
> }
> }
> if (address != null) connector.setHost(address)
> connector.setPort(port)
> connector.setIdleTimeout(1000L * idleTimeout)
> server.addConnector(connector)
> /***** end code *****/
>
> Then for the ThreadPool part:
>
> /***** start code *****/
> val pool = new QueuedThreadPool(maxThreads,minThreads,idleTimeout,new
> ArrayBlockingQueue(queueSize))
> val server = new Server(pool)
> /***** end code *****/
>
HTTP/2 will seriously stress out your thread pool.
Don't specify the queue, use defaults.
Don't use less then defaults for min/max threads with HTTP/2.
It's good that you are using the QueuedThreadPool, it behaves much better
then a raw Executor with regards to HTTP/2.
>
> Threads are 10-100, idletimeouts I have tried are from 1000 to 20000,
> queueSize is now 500 but I have tried both much smaller and larger values.
>
> I have no idea what values would be sensible. The number of concurrent
> users is relatively low: up to a couple of hundred, rarely more. Users can
> do requests that take longish to complete (sometimes up to 30 or even 60
> seconds) but those are very rare. The majority is relatively quick (<
> 50ms). Current load on the problematic servers is very low (perhaps 10
> users) and as said: low memory and CPU close to zero.
> No problems with HTTP/1.1 with this code/settings, problematic with
> HTTP/2. Strangely enough some servers seem fine (or users just did not
> report any problems yet).
>
Start by using a default QueuedThreadPool, no configuration, no min thread,
no max threads, all strictly defaults.
Then monitor your usages and behaviors with HTTP/2 enabled over at least a
week.
Then you can tune your maxThreads.
As to thread idle timeouts, don't bother setting them yet.
Your connector.setIdleTimeout() is far more useful to you.
>
> Any help would be welcome.
>
> Kind regards,
>
> Silvio
>
Many folks have gotten HTTP/2 working just fine in embedded-jetty.
The two most common issues:
1. Not having alpn support setup properly
2. Too restrictive of an SslContextFactory that is incompatible with HTTP/2
When it comes to the ALPN suppport, you have to worry about what JVM
runtime you are on first.
If you are using Java 8 then you have to align your alpn-boot version to
the specific Java 8 JVM you are using.
https://github.com/jetty-project/jetty-alpn/blob/master/docs/version_mapping.properties
Under Java 8, you MUST use the -Xbootclasspath on JVM startup to have alpn
work properly.
However, if you are using Java 11 then you can use the
jetty-alpn-java-server artifact instead (no bootclasspath requirement) to
get ALPN behaviors required for HTTP/2.
You also have a conscrypt option for running ALPN + HTTP/2 on Jetty, which
will work the same regardless of Java 8 or Java 11.
When it comes to too restrictive of an SslContextFactory we have the
following advice:
* If you are specifying Include Cipher Suites you are too restrictive.
* Your certificates need to be RSA (not DES) and at a higher bit size (like
2048+)
* If you are attempting to get old Protocols (TLS/1.0 TLS/1.1) working,
then you are introducing incompatible Cipher Suites too.
* If you are resetting the Exclude Cipher Suites you are introducing
incompatible Cipher Suites.
* Avoid using Include Cipher Suites (this exists for people stuck in the
past and not ever wanting to use HTTP/2)
* Avoid using Include Protocols (this exists for people stuck supporting
old clients and not wanting to use HTTP/2)
* Add to the existing Exclude Cipher Suites (if needed)
* Add to the existing Exclude Protocols (if needed)
* Understand the OpenJDK JVM Crypto Roadmap and its implications -
https://java.com/en/jre-jdk-cryptoroadmap.html
* Always use an Up to date (not-expired) JVM
* Learn how to use the SslContextFactory.dump() methods at runtime to know
what the current state of your SslContextFactory is (including all selected
cipher suites and protocols)
* Test your running server on https://www.ssllabs.com/ssltest/ to see where
you are still having configuration issues with SslContextFactory
We have an example codebase that shows how to start an HTTP/2 server
connector.
https://github.com/eclipse/jetty.project/blob/jetty-9.4.x/examples/embedded/src/main/java/org/eclipse/jetty/embedded/Http2Server.java
You can check the Jenkins project for their usage of HTTP/2 on Embedded
Jetty as well. (their self executing WAR file does just this)
You could also run h2spec against your server and see if its behaving
properly.
https://github.com/summerwind/h2spec
- Joakim
>
>
> On 20-11-18 16:06, Joakim Erdfelt wrote:
>
> How do you enable HTTP/2 in your embedded application?
>
> Also, If you have configured your SslContextFactory, what does this
> configuration look like?
>
> Lastly, what does your ThreadPool / Executor configuration look like?
>
> Joakim Erdfelt / [email protected]
>
>
> On Tue, Nov 20, 2018 at 9:01 AM Silvio Bierman <
> [email protected]> wrote:
>
>> Hello all,
>>
>> We are running Jetty 9.4.14 embedded in an application that runs on a
>> number of Ubuntu 18.10 servers. Last night we switched from HTTP1.1 only
>> to HTTP2. Today we are experiencing serious problems. Users can not
>> access the application or are experiencing extremely long latencies. At
>> the same time the server is almost idle.
>> We have configured the Ubuntu servers as almost default setups.
>> OpenJDK11 with Jetty using pure Java HTTPS. We have been doing this for
>> years now and each time we switched from HTTP1.1 to HTTP2 things started
>> falling apart. In the past we had reports of people not being able to
>> connect at all receiving spurious messages from their browsers (some of
>> which are undoubtedly outdated). We expected things to have improved by
>> now and decided to try again, with the above result.
>>
>> Are there additional things we need to take care of when using HTTP2
>> when compared to HTTP1.1? More threads, different queue size, more file
>> handles etc?
>>
>> Any pointers would be very much appreciated.
>>
>> Kind regards,
>>
>> Silvio
>>
>> _______________________________________________
>> jetty-users mailing list
>> [email protected]
>> To change your delivery options, retrieve your password, or unsubscribe
>> from this list, visit
>> https://www.eclipse.org/mailman/listinfo/jetty-users
>>
>
> _______________________________________________
> jetty-users mailing [email protected]
> To change your delivery options, retrieve your password, or unsubscribe from
> this list, visithttps://www.eclipse.org/mailman/listinfo/jetty-users
>
>
>
_______________________________________________
jetty-users mailing list
[email protected]
To change your delivery options, retrieve your password, or unsubscribe from
this list, visit
https://www.eclipse.org/mailman/listinfo/jetty-users
