[ https://issues.apache.org/jira/browse/NETBEANS-58?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16449908#comment-16449908 ]
Jean-Marc Borer commented on NETBEANS-58: ----------------------------------------- @Geertjan, no it did not happen before. Now, I don't know from when exactly it started, I mean which JDK/NB version combination. Actually it doesn't matter. Now it no longer works and the reason has been quite clearly identified: dead locking while querying the Keyring. If you look at the extract from my block IDE thread dump you will see that: 1) "Thread-8" takes the lock {color:red}0x00000000c0d64370 {color}in sun.net.www.protocol.http.AuthenticationHeader.parse(AuthenticationHeader.java:200) Then further down the call stack, it creates a task that will wait forever reading the a value from Keyring at org.netbeans.api.keyring.Keyring.read(Keyring.java:144) 2) This task is posted on thraed "*org.netbeans.api.keyring.Keyring*" where it tries to get the lock on {color:red}0x00000000c0d64370{color}. Boom deadlock... This blocks all threads on the ModuleManager$SystemClassLoader lock on {color:red}0x00000000c0d64370 {color}for every other thread including the AWT Event one which leads to the HMI freeze. "org.netbeans.api.keyring.Keyring" #26 daemon prio=1 os_prio=-2 tid=0x0000000019593000 nid=0x1640 waiting for monitor entry [0x000000002655e000] java.lang.Thread.State: BLOCKED (on object monitor) at org.netbeans.ModuleManager$SystemClassLoader.getResourcesImpl(ModuleManager.java:708) - waiting to lock <{color:#FF0000}0x00000000c0d64370{color}> (a org.netbeans.ModuleManager$SystemClassLoader) at org.netbeans.ProxyClassLoader.getResources(ProxyClassLoader.java:390) at org.openide.util.lookup.MetaInfServicesLookup.search(MetaInfServicesLookup.java:205) at org.openide.util.lookup.MetaInfServicesLookup.beforeLookup(MetaInfServicesLookup.java:156) at org.openide.util.lookup.MetaInfServicesLookup.beforeLookupResult(MetaInfServicesLookup.java:135) at org.openide.util.lookup.AbstractLookup.lookup(AbstractLookup.java:483) at org.openide.util.lookup.ProxyLookup$R.initResults(ProxyLookup.java:390) at org.openide.util.lookup.ProxyLookup$R.myBeforeLookup(ProxyLookup.java:673) at org.openide.util.lookup.ProxyLookup$R.computeResult(ProxyLookup.java:553) at org.openide.util.lookup.ProxyLookup$R.allInstances(ProxyLookup.java:513) at org.openide.util.lookup.ProxyLookup$R.allInstances(ProxyLookup.java:509) at org.openide.util.Lookup.lookupAll(Lookup.java:312) at org.netbeans.api.keyring.Keyring.provider(Keyring.java:89) - locked <0x00000000f0525f68> (a java.lang.Class for org.netbeans.api.keyring.Keyring) at org.netbeans.api.keyring.Keyring.readImpl(Keyring.java:105) - locked <0x00000000f0525f68> (a java.lang.Class for org.netbeans.api.keyring.Keyring) at org.netbeans.api.keyring.Keyring.access$100(Keyring.java:75) at org.netbeans.api.keyring.Keyring$1.call(Keyring.java:128) at org.netbeans.api.keyring.Keyring$1.call(Keyring.java:125) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at org.openide.util.RequestProcessor$Task.run(RequestProcessor.java:1443) at org.netbeans.modules.openide.util.GlobalLookup.execute(GlobalLookup.java:68) at org.openide.util.lookup.Lookups.executeWith(Lookups.java:303) at org.openide.util.RequestProcessor$Processor.run(RequestProcessor.java:2058) "Thread-8" #49 prio=6 os_prio=0 tid=0x00000000256a5000 nid=0xdb8 waiting on condition [0x000000002cd7a000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00000000f05d6c10> (a org.openide.util.RequestProcessor$RPFutureTask) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.FutureTask.awaitDone(FutureTask.java:429) at java.util.concurrent.FutureTask.get(FutureTask.java:191) at org.netbeans.api.keyring.Keyring.read(Keyring.java:144) at org.netbeans.core.ProxySettings.getAuthenticationPassword(ProxySettings.java:230) at org.netbeans.core.NbAuthenticator.getPasswordAuthentication(NbAuthenticator.java:87) at java.net.Authenticator.requestPasswordAuthentication(Authenticator.java:317) - locked <0x00000000c2c66008> (a org.netbeans.core.NbAuthenticator) at sun.net.www.protocol.http.spnego.NegotiateCallbackHandler.getAnswer(NegotiateCallbackHandler.java:65) at sun.net.www.protocol.http.spnego.NegotiateCallbackHandler.handle(NegotiateCallbackHandler.java:86) at com.sun.security.auth.module.Krb5LoginModule.promptForName(Krb5LoginModule.java:858) at com.sun.security.auth.module.Krb5LoginModule.attemptAuthentication(Krb5LoginModule.java:704) at com.sun.security.auth.module.Krb5LoginModule.login(Krb5LoginModule.java:617) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at javax.security.auth.login.LoginContext.invoke(LoginContext.java:755) at javax.security.auth.login.LoginContext.access$000(LoginContext.java:195) at javax.security.auth.login.LoginContext$4.run(LoginContext.java:682) at javax.security.auth.login.LoginContext$4.run(LoginContext.java:680) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:680) at javax.security.auth.login.LoginContext.login(LoginContext.java:587) at sun.security.jgss.GSSUtil.login(GSSUtil.java:258) at sun.security.jgss.krb5.Krb5Util.getTicket(Krb5Util.java:158) at sun.security.jgss.krb5.Krb5InitCredential$1.run(Krb5InitCredential.java:335) at sun.security.jgss.krb5.Krb5InitCredential$1.run(Krb5InitCredential.java:331) at java.security.AccessController.doPrivileged(Native Method) at sun.security.jgss.krb5.Krb5InitCredential.getTgt(Krb5InitCredential.java:330) at sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:145) at sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:122) at sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(Krb5MechFactory.java:187) at sun.security.jgss.GSSManagerImpl.getMechanismContext(GSSManagerImpl.java:224) at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:212) at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179) at sun.security.jgss.spnego.SpNegoContext.GSS_initSecContext(SpNegoContext.java:882) at sun.security.jgss.spnego.SpNegoContext.initSecContext(SpNegoContext.java:317) at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:248) at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179) at sun.net.www.protocol.http.spnego.NegotiatorImpl.init(NegotiatorImpl.java:108) at sun.net.www.protocol.http.spnego.NegotiatorImpl.<init>(NegotiatorImpl.java:117) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at sun.net.www.protocol.http.Negotiator.getNegotiator(Negotiator.java:63) at sun.net.www.protocol.http.NegotiateAuthentication.isSupportedImpl(NegotiateAuthentication.java:130) - locked <0x00000000f0e74ed8> (a java.lang.Class for sun.net.www.protocol.http.NegotiateAuthentication) at sun.net.www.protocol.http.NegotiateAuthentication.isSupported(NegotiateAuthentication.java:102) - locked <{color:#FF0000}0x00000000c0d64370{color}> (a org.netbeans.ModuleManager$SystemClassLoader) at sun.net.www.protocol.http.AuthenticationHeader.parse(AuthenticationHeader.java:200) at sun.net.www.protocol.http.AuthenticationHeader.<init>(AuthenticationHeader.java:143) at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1626) - locked <0x00000000f0e75008> (a sun.net.www.protocol.http.HttpURLConnection) at sun.net.www.protocol.http.HttpURLConnection.access$200(HttpURLConnection.java:91) at sun.net.www.protocol.http.HttpURLConnection$9.run(HttpURLConnection.java:1466) at sun.net.www.protocol.http.HttpURLConnection$9.run(HttpURLConnection.java:1464) at java.security.AccessController.doPrivileged(Native Method) at java.security.AccessController.doPrivilegedWithCombiner(AccessController.java:782) at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1463) - locked <0x00000000f0e75008> (a sun.net.www.protocol.http.HttpURLConnection) at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:480) at org.netbeans.modules.welcome.content.RSSFeed.findInputSource(RSSFeed.java:233) at org.netbeans.modules.welcome.content.RSSFeed.buildItemList(RSSFeed.java:196) at org.netbeans.modules.welcome.content.RSSFeed$Reload.run(RSSFeed.java:308) The authenticator plugin did not help because as far as I understand it improves SPNEGO support not the way the Keyring is invoked. I am sitting behind a NTLMv2 proxy. Authentication works (and worked) but it dead locks... Maybe a solution would be not to request the Keyring value on a separate thread (line org.netbeans.api.keyring.Keyring.read(Keyring.java:144))but do it synchronously. As the call context already holds the org.netbeans.ModuleManager$SystemClassLoader lock it would not block? > NB IDE or NB Platform freeze on startup (proxy with Negotiate auth) > ------------------------------------------------------------------- > > Key: NETBEANS-58 > URL: https://issues.apache.org/jira/browse/NETBEANS-58 > Project: NetBeans > Issue Type: Bug > Components: platform - Proxy > Affects Versions: 8.2, 9.0 > Environment: Primarily Windows. > Reporter: phansson > Priority: Critical > Attachments: NETBEANS-58-workaround1.diff, nb-freeze-dump.txt, > netbeans.txt > > > When any network operation is performed, such as attempting to contact > NetBeans Update Center, the application (IDE or Platform) may freeze. Users > will typically experience this on startup. It was reported in old bug tracker > as [bug 248308|https://netbeans.org/bugzilla/show_bug.cgi?id=248308]. > The problem arises because of the fix JDK folks applied as a consequence of > the reported [JDK-8032832 > bug|https://bugs.openjdk.java.net/browse/JDK-8032832]. This fix wasn't very > clever IMO: it puts a lock on the classloader, thus introducing a range of > other problems, one of them being that NetNeans IDE or NetBeans Platform will > likely freeze on startup when it attempts a network operation. The fact that > their fix made things worse (while no doubt fixing the original issue) has > been reported as > [JDK-8068184|https://bugs.openjdk.java.net/browse/JDK-8068184]. > h3. WHEN DOES IT HAPPEN? > As the lock is introduced for authentication of type 'Negotiate' it of course > only happens if there's a network proxy on the path which uses this type of > authentication. Also known as SPNEGO. This form of authentication is in my > experience very common in corporate networks, in particular those that base > themselves on the Microsoft stack. But a person on Oracle's own internal > network, such as a JDK developer, is most likely not exposed to it. :-) > There's another condition for it to happen: The JRE runtime must be unable to > provide 'credentials' (a Kerberos token) to the network proxy on its own. > SPNEGO is really designed to be seamless and promptless. Support for it was > added in Java 6. But later on Microsoft tightened the desktop security around > obtaining the so-called 'session token' and the JDK folks were never able to > work around this (unlike the makers of Chrome, FF, Opera, etc). Therefore, in > real-life, SPNEGO in the JRE on Windows is no longer promptless: it will be > forced to ask the user for credentials, thus negating the idea of SPNEGO. It > is the prompting which causes the freeze. SPNEGO on Mac OS X and Linux is > most likely working just fine and the bug will never be experienced. > h3. HOW DO I KNOW IF I'M AFFECTED BY EXACTLY THIS BUG? > This bug in this ticket is characterized by the fact that you'll always be > able to find the following in your thread dump: > {noformat} > at > sun.net.www.protocol.http.NegotiateAuthentication.isSupported(NegotiateAuthentication.java:<lineno>) > - locked <OBJECTID> (a org.netbeans.ModuleManager$SystemClassLoader) > {noformat} > Note that the [Ctrl-Break > method|https://docs.oracle.com/javase/8/docs/technotes/guides/troubleshoot/tooldescr019.html] > of obtaining a thread dump is favoured over jstack and other methods. > h3. WHY DOES IT HAPPEN? > There will be a lock held on the classloader object when the JRE's registered > Authenticator is invoked. If the Authenticator does work on another thread, > that other thread has a need for some classloading and the current thread > needs to wait for the result of that thread, then bum!, there's a deadlock > between the two threads. This means the lock on the classloader will never be > released and it will ultimately affect other threads, such as the AWT > dispatch thread (aka Swing EDT) which will then also lock. Then you have what > the user experiences as a freeze. > The NB Platform's own Authenticator, {{NbAuthenticator}}, does exactly what I > described and will thus be triggering the deadlock. More precisely it will > happen when NbAuthenticator calls Keyring. Does this mean the NbAuthenticator > does something wrong? No, of course it doesn't. The real problem is the lock > on the classloader. It is actually virtually impossible to design an > Authenticator which doesn't trigger this problem. You cannot predict when > classloading is needed. In fact it is very likely to be needed when > application is still not "warm", i.e. during startup. > h3. WORKAROUNDS > *#1* > If on Windows: Setting the following registry key: > {{HKLM\System\CurrentControlSet\Control\Lsa\Kerberos\Parameters\allowtgtsessionkey}} > to {{true}} will allow the JRE to obtain the session key, thus the JDK's HTTP > classes will have no need to invoke Authenticator, thus deadlock will not > happen. But this registry key only has effect for users who are _not_ local > admins and since it is a HKLM such user will need to ask his administrator to > do this change. There's probably zero chance in a million that a corporate > network administrator will allow this change. After all, Microsoft introduced > the tightened security for a reason and the admin will rightly ask why only > the JRE needs this and not Chrome, IE, FF, Opera and so on? > *#2* > Convice the JDK folks that they've made a mess of it with this lock. I've > pursued this avenue too. I've done that on the JDK security-dev mailing list. > I've pointed to similar bug tickets for Eclipse IDE and IDEA and I got the > attention of Weijun Wang ("Max") of Oracle who promised he would look at it. > But to be honest the JDK people have a lot of other things on their plate and > a fix will take some time. > UPDATE 29-OCT-2017: To be fair to the JDK folks to issue only occurs with > NB's own classloaders. So the chance they'll fix it at the JDK end is > probably slim. > Link: > http://mail.openjdk.java.net/pipermail/security-dev/2017-August/016267.html > *#3* > Re-design the Authenticator in the Platform. As I cannot change the code in > NB itself, I've created a workaround as a plugin. The recipe is described in > [Comment 44|https://netbeans.org/bugzilla/show_bug.cgi?id=248308#c44] on the > original NB bug ticket. This is really a workaround, not a fix. It will > simply give up on attempting to obtain credentials if it discovers that it is > likely to be in this deadlock scenario. Thus it leaves the application with > no outbound network connectivity but it is still better than the freeze. It > also alerts the user to the situation using a bubble notification. > Links: > https://bitbucket.org/phansson/netbeansnetworkauthenticator > https://bitbucket.org/phansson/netbeansnetworkauthenticator/wiki/JDK-8068184%20Workaround > *#4* > It may help to set proxy username/password _explicitly_ in the NB's Options > panel. But this is not a solution I recommend. It means you must put your AD > password into NetBeans IDE Options. It will of course be static so once your > AD password changes then you must remember to change there as well. And for > the user to configure this, it requires that he can actually start the IDE or > Platform app in the first place ... which is often not the case because of > the freeze. > *#5* > Use a JRE prior to 8u20 or prior to 7u76. To most people this workaround is > unacceptable. > h3. CONCLUSION > IMHO #3 is the most attractive solution for now. It doesn't exclude the user > still doing #1 or #4 to get full benefit. The real solution is of course #2. > .. and the very best solution long term is if JDK would have same support for > SPNEGO on Windows as does 'regular' applications such as Chrome, FF, Opera, > IE, Edge, etc. Then we wouldn't have the problem in the first place. This has > been discussed intensively over the last 7-8 years but there's a somewhat > religious tug of war between Sun/Oracle and Microsoft on the matter. The > problem can be solved if the JDK would base itself on the Win32 SSPI api in > this area, rather than the long-time deprecated Win32 > LsaCallAuthenticationPackage API. We are getting slightly off topic here and > not appropriate for a bug ticket. :-) -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@netbeans.apache.org For additional commands, e-mail: commits-h...@netbeans.apache.org For further information about the NetBeans mailing lists, visit: https://cwiki.apache.org/confluence/display/NETBEANS/Mailing+lists