Issue with SSL handshake implementation

[chengjingwei (A)]

Hi,



There may be some issue with the SSL handshake implementation.



In sun.security.x509.X500Name.java, there's a static hashmap field named 
*internedOIDs*, used for caching X.500 attributes.

Each time a new oid is encountered, jdk will cache them into the static hashmap.



With a purposely crafted cert on the client side, it's possible to create some 
long oids, and letting the server

save them permanently during SSL handshake, which will eat server memory, and 
cause OOM & DoS in the end.



Is it necessary to make some changes to this?





BTW, The issue was reported by one of our customers, with the following 
stacktrace.

Although they got this on jdk8u212, I believe the same issue exists in jdk11 
too.



```

http-bio-172.24.18.21-443-exec-4

at java.lang.OutOfMemoryError.<init>()V (OutOfMemoryError.java:48)

at java.util.Arrays.copyOf([Ljava/lang/Object;I)[Ljava/lang/Object; 
(Arrays.java:3181)

at java.util.Vector.grow(I)V (Vector.java:269)

at java.util.Vector.ensureCapacityHelper(I)V (Vector.java:249)

at java.util.Vector.addElement(Ljava/lang/Object;)V (Vector.java:623)

at sun.security.util.DerInputStream.readVector(I)[Lsun/security/util/DerValue; 
(DerInputStream.java:425)

at sun.security.util.DerInputStream.getSequence(I)[Lsun/security/util/DerValue; 
(DerInputStream.java:331)

at sun.security.x509.X500Name.parseDER(Lsun/security/util/DerInputStream;)V 
(X500Name.java:793)

at sun.security.x509.X500Name.<init>(Lsun/security/util/DerInputStream;)V 
(X500Name.java:306)

at sun.security.x509.X509CertInfo.parse(Lsun/security/util/DerValue;)V 
(X509CertInfo.java:649)

at sun.security.x509.X509CertInfo.<init>(Lsun/security/util/DerValue;)V 
(X509CertInfo.java:167)

at sun.security.x509.X509CertImpl.parse(Lsun/security/util/DerValue;)V 
(X509CertImpl.java:1804)

at sun.security.x509.X509CertImpl.<init>([B)V (X509CertImpl.java:195)

at 
sun.security.provider.X509Factory.engineGenerateCertificate(Ljava/io/InputStream;)Ljava/security/cert/Certificate;
 (X509Factory.java:102)

at 
java.security.cert.CertificateFactory.generateCertificate(Ljava/io/InputStream;)Ljava/security/cert/Certificate;
 (CertificateFactory.java:339)

at 
sun.security.ssl.HandshakeMessage$CertificateMsg.<init>(Lsun/security/ssl/HandshakeInStream;)V
 (HandshakeMessage.java:455)

at sun.security.ssl.ServerHandshaker.processMessage(BI)V 
(ServerHandshaker.java:230)

at sun.security.ssl.Handshaker.processLoop()V (Handshaker.java:1037)

at sun.security.ssl.Handshaker.process_record(Lsun/security/ssl/InputRecord;Z)V 
(Handshaker.java:965)

at sun.security.ssl.SSLSocketImpl.readRecord(Lsun/security/ssl/InputRecord;Z)V 
(SSLSocketImpl.java:1064)

at sun.security.ssl.SSLSocketImpl.performInitialHandshake()V 
(SSLSocketImpl.java:1367)

at sun.security.ssl.SSLSocketImpl.startHandshake(Z)V (SSLSocketImpl.java:1395)

at sun.security.ssl.SSLSocketImpl.getSession()Ljavax/net/ssl/SSLSession; 
(SSLSocketImpl.java:2288)

at 
org.apache.tomcat.util.net.jsse.JSSESocketFactory.handshake(Ljava/net/Socket;)V 
(JSSESocketFactory.java:293)

at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run()V 
(JIoEndpoint.java:343)

at 
java.util.concurrent.ThreadPoolExecutor.runWorker(Ljava/util/concurrent/ThreadPoolExecutor$Worker;)V
 (ThreadPoolExecutor.java:1149)

at java.util.concurrent.ThreadPoolExecutor$Worker.run()V 
(ThreadPoolExecutor.java:624)

at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run()V 
(TaskThread.java:61)

at java.lang.Thread.run()V (Thread.java:748)

```