So it's can be that neither... Thanks for the links. I have restarted the application. I have added the debug level for RequestAddCookies and asked the application to stop as soon as there is an exception. I just need to let it run for some time. Last time I saw the exception it was running for few days. So it might take time before it happens again.
2012/8/2, sebb <[email protected]>: > On 2 August 2012 19:36, Jean-Marc Spaggiari <[email protected]> > wrote: >> I'm using 1.7.0_05 from Sun. >> >> I have enabled the RequestAddCookies debug logs and will see when I >> will get the next NPE. >> >> One the line 193 (I don't have the source) is version an int? Or an >> Integer? > > int, see: > > http://hc.apache.org/httpcomponents-client-ga/httpclient/xref/org/apache/http/impl/cookie/BestMatchSpec.html#193 > and > http://hc.apache.org/httpcomponents-client-ga/httpclient/xref/org/apache/http/cookie/Cookie.html#126 > >> I agree with you, cookie can't be null, else we will have got an NPE >> at "if (!matchedCookies.isEmpty())" >> >> I will restart my tool with the debug mode enabled and keep you posted. >> >> 2012/8/2, sebb <[email protected]>: >>> On 2 August 2012 19:03, Jean-Marc Spaggiari <[email protected]> >>> wrote: >>>> I'm using version 4.2.1 binary distribution. >>> >>> In which case the NPE is very odd. >>> >>> The stack dump shows: >>> >>> java.lang.NullPointerException >>> at >>> org.apache.http.impl.cookie.BestMatchSpec.formatCookies(BestMatchSpec.java:193) >>> at >>> org.apache.http.client.protocol.RequestAddCookies.process(RequestAddCookies.java:196) >>> >>> Now the code in question is: >>> >>> BestMatchSpec.java:193: if (cookie.getVersion() < version) { >>> >>> AFAICT, this can only cause NPE if "cookie" is null. >>> >>> The cookie variable is derived from the list created in >>> RequestAddCookies.java: >>> >>> List<Cookie> cookies = new >>> ArrayList<Cookie>(cookieStore.getCookies()); >>> // Find cookies matching the given origin >>> List<Cookie> matchedCookies = new ArrayList<Cookie>(); >>> Date now = new Date(); >>> for (Cookie cookie : cookies) { >>> if (!cookie.isExpired(now)) { // <== cookie cannot be null >>> here >>> if (cookieSpec.match(cookie, cookieOrigin)) { >>> if (this.log.isDebugEnabled()) { >>> this.log.debug("Cookie " + cookie + " match " >>> + cookieOrigin); >>> } >>> matchedCookies.add(cookie); // <== so cannot be null >>> here >>> } >>> } else { >>> if (this.log.isDebugEnabled()) { >>> this.log.debug("Cookie " + cookie + " expired"); >>> } >>> } >>> } >>> // Generate Cookie request headers >>> if (!matchedCookies.isEmpty()) { >>> List<Header> headers = >>> cookieSpec.formatCookies(matchedCookies); // <== 196 >>> for (Header header : headers) { >>> request.addHeader(header); >>> } >>> } >>> >>> I cannot see how a null cookie can be added to the list, nor how a >>> list entry could be set to null later. >>> >>> It would be worth enabling Debug logging for the RequestAddCookies >>> class. >>> >>> What JVM are you using? >>> >>>> 2012/8/2, sebb <[email protected]>: >>>>> On 1 August 2012 12:08, Jean-Marc Spaggiari <[email protected]> >>>>> wrote: >>>>>> Hi Oleg, >>>>>> >>>>>> My application is reading many other URLs before this one. So maybe >>>>>> it's a combination of multiple loads. >>>>>> >>>>>> I will try to dump more when I will get the issue again. >>>>>> >>>>>> In the meantime, here is how I'm initializing the HttpClient: >>>>>> >>>>>> // Creation of the HTTP Client >>>>>> SchemeRegistry schemeRegistry = new SchemeRegistry(); >>>>>> schemeRegistry.register(new Scheme("http", 80, >>>>>> PlainSocketFactory.getSocketFactory())); >>>>>> schemeRegistry.register(new Scheme("https", 443, >>>>>> SSLSocketFactory.getSocketFactory())); >>>>>> PoolingClientConnectionManager cm = new >>>>>> PoolingClientConnectionManager(schemeRegistry); >>>>>> cm.setMaxTotal(200); >>>>>> cm.setDefaultMaxPerRoute(20); >>>>>> >>>>>> HttpParams params = new BasicHttpParams(); >>>>>> if (useProxy) >>>>>> params.setParameter(ConnRoutePNames.DEFAULT_PROXY, new >>>>>> HttpHost("proxy", 80)); >>>>>> client = new DefaultHttpClient(cm, params); >>>>>> client.getParams().setParameter(CoreConnectionPNames.SO_TIMEOUT, new >>>>>> Integer(30000)); >>>>>> client.getParams().setParameter(CoreConnectionPNames.CONNECTION_TIMEOUT, >>>>>> new Integer(30000)); >>>>>> client.getParams().setParameter(CoreConnectionPNames.TCP_NODELAY, >>>>>> false); >>>>>> >>>>>> It's not the first time I'm getting this exception, so I might be >>>>>> able >>>>>> to reproduce that in a near futur. >>>>> >>>>> What version of HttpClient are you using? >>>>> >>>>>> JM >>>>>> >>>>>> 2012/8/1, Oleg Kalnichevski <[email protected]>: >>>>>>> On Tue, 2012-07-31 at 17:37 -0400, Jean-Marc Spaggiari wrote: >>>>>>>> Hi, >>>>>>>> >>>>>>>> I'm getting the exception below when I'm trying to load >>>>>>>> http://www.expedia.ca/Sunset-District-Hotels.d163838.Travel-Guide-Hotels >>>>>>>> ... It's not critical since I'm catching everything, but I'm >>>>>>>> wondering >>>>>>>> if this is normal or if this is something which need to be fixed... >>>>>>>> >>>>>>>> java.lang.NullPointerException >>>>>>>> at >>>>>>>> org.apache.http.impl.cookie.BestMatchSpec.formatCookies(BestMatchSpec.java:193) >>>>>>>> at >>>>>>>> org.apache.http.client.protocol.RequestAddCookies.process(RequestAddCookies.java:196) >>>>>>>> at >>>>>>>> org.apache.http.protocol.ImmutableHttpProcessor.process(ImmutableHttpProcessor.java:109) >>>>>>>> at >>>>>>>> org.apache.http.protocol.HttpRequestExecutor.preProcess(HttpRequestExecutor.java:176) >>>>>>>> at >>>>>>>> org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:515) >>>>>>>> at >>>>>>>> org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:906) >>>>>>>> at >>>>>>>> org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:805) >>>>>>>> at >>>>>>>> org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:784) >>>>>>>> at >>>>>>>> org.spaggiari.distparser.CrawlerThread$1.run(CrawlerThread.java:267) >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> JM >>>>>>>> >>>>>>> >>>>>>> Jean-Marc >>>>>>> >>>>>>> I am not able to reproduce the problem just by executing HTTP GET to >>>>>>> the >>>>>>> URL in question. All looks perfectly normal. >>>>>>> >>>>>>> [DEBUG] BasicClientConnectionManager - Get connection for route >>>>>>> {}->http://www.expedia.ca >>>>>>> [DEBUG] DefaultClientConnectionOperator - Connecting to >>>>>>> www.expedia.ca:80 >>>>>>> [DEBUG] RequestAddCookies - CookieSpec selected: best-match >>>>>>> [DEBUG] RequestAuthCache - Auth cache not set in the context >>>>>>> [DEBUG] RequestTargetAuthentication - Target auth state: >>>>>>> UNCHALLENGED >>>>>>> [DEBUG] RequestProxyAuthentication - Proxy auth state: UNCHALLENGED >>>>>>> [DEBUG] DefaultHttpClient - Attempt 1 to execute request >>>>>>> [DEBUG] DefaultClientConnection - Sending request: >>>>>>> GET /Sunset-District-Hotels.d163838.Travel-Guide-Hotels HTTP/1.1 >>>>>>> [DEBUG] headers - >> >>>>>>> GET /Sunset-District-Hotels.d163838.Travel-Guide-Hotels HTTP/1.1 >>>>>>> [DEBUG] headers - >> Host: www.expedia.ca >>>>>>> [DEBUG] headers - >> Connection: Keep-Alive >>>>>>> [DEBUG] headers - >> User-Agent: Apache-HttpClient/4.2.1 (java 1.5) >>>>>>> [DEBUG] DefaultClientConnection - Receiving response: HTTP/1.1 200 >>>>>>> OK >>>>>>> [DEBUG] headers - << HTTP/1.1 200 OK >>>>>>> [DEBUG] headers - << Content-Type: text/html;charset=UTF-8 >>>>>>> [DEBUG] headers - << Content-Language: en-CA >>>>>>> [DEBUG] headers - << P3P: policyref="/w3c/p3p.xml", CP="CAO DSP IND >>>>>>> COR >>>>>>> ADM CONo CUR CUSi DEV PSA PSD DELi OUR COM NAV PHY ONL PUR UNI" >>>>>>> [DEBUG] headers - << RTSS: 1 >>>>>>> [DEBUG] headers - << Date: Wed, 01 Aug 2012 10:49:28 GMT >>>>>>> [DEBUG] headers - << Transfer-Encoding: chunked >>>>>>> [DEBUG] headers - << Connection: keep-alive >>>>>>> [DEBUG] headers - << Connection: Transfer-Encoding >>>>>>> [DEBUG] headers - << Set-Cookie: SSLB=1; path=/; domain=.expedia.ca >>>>>>> [DEBUG] headers - << Set-Cookie: >>>>>>> MC1=GUID=c55cfa1393ce4bfca1ac5b6532085717; Domain=.expedia.ca; >>>>>>> Expires=Fri, 04-Aug-2017 03:44:09 GMT; Path=/ >>>>>>> [DEBUG] headers - << Set-Cookie: iEAPID=000000,; Domain=.expedia.ca; >>>>>>> Path=/ >>>>>>> [DEBUG] headers - << Set-Cookie: >>>>>>> JSESSION=13201604-6f17-4f6e-8b03-570fd433ad4e; Domain=.expedia.ca; >>>>>>> Path=/ >>>>>>> [DEBUG] headers - << Set-Cookie: s1=`0; Domain=.expedia.ca; Path=/ >>>>>>> [DEBUG] headers - << Set-Cookie: >>>>>>> p1=`tpid=v.1,4`linfo=v.4,|0|0|255|1| >>>>>>> 0||||||||4105|0|0||0|0|0|-1|-1`63; Domain=.expedia.ca; Expires=Tue, >>>>>>> 01-Aug-2017 15:54:40 GMT; Path=/ >>>>>>> [DEBUG] ResponseProcessCookies - Cookie accepted: "[version: >>>>>>> 0][name: >>>>>>> SSLB][value: 1][domain: .expedia.ca][path: /][expiry: null]". >>>>>>> [DEBUG] ResponseProcessCookies - Cookie accepted: "[version: >>>>>>> 0][name: >>>>>>> MC1][value: >>>>>>> GUID=c55cfa1393ce4bfca1ac5b6532085717][domain: .expedia.ca][path: >>>>>>> /][expiry: >>>>>>> Fri Aug 04 04:44:09 GMT+01:00 2017]". >>>>>>> [DEBUG] ResponseProcessCookies - Cookie accepted: "[version: >>>>>>> 0][name: >>>>>>> iEAPID][value: 000000,][domain: .expedia.ca][path: /][expiry: >>>>>>> null]". >>>>>>> [DEBUG] ResponseProcessCookies - Cookie accepted: "[version: >>>>>>> 0][name: >>>>>>> JSESSION][value: >>>>>>> 13201604-6f17-4f6e-8b03-570fd433ad4e][domain: .expedia.ca][path: >>>>>>> /][expiry: >>>>>>> null]". >>>>>>> [DEBUG] ResponseProcessCookies - Cookie accepted: "[version: >>>>>>> 0][name: >>>>>>> s1][value: `0][domain: .expedia.ca][path: /][expiry: null]". >>>>>>> [DEBUG] ResponseProcessCookies - Cookie accepted: "[version: >>>>>>> 0][name: >>>>>>> p1][value: `tpid=v.1,4`linfo=v.4,|0|0|255|1|0||||||||4105|0|0||0|0| >>>>>>> 0|-1|-1`63][domain: .expedia.ca][path: /][expiry: Tue Aug 01 >>>>>>> 16:54:40 >>>>>>> GMT+01:00 2017]". >>>>>>> [DEBUG] DefaultHttpClient - Connection can be kept alive >>>>>>> indefinitely >>>>>>> >>>>>>> Can you put together a small test app that could be used to >>>>>>> reproduce >>>>>>> the NPE? >>>>>>> >>>>>>> Oleg >>>>>>> >>>>>>> >>>>>>> >>>>>>> --------------------------------------------------------------------- >>>>>>> To unsubscribe, e-mail: [email protected] >>>>>>> For additional commands, e-mail: [email protected] >>>>>>> >>>>>>> >>>>>> >>>>>> --------------------------------------------------------------------- >>>>>> To unsubscribe, e-mail: [email protected] >>>>>> For additional commands, e-mail: [email protected] >>>>>> >>>>> >>>>> --------------------------------------------------------------------- >>>>> To unsubscribe, e-mail: [email protected] >>>>> For additional commands, e-mail: [email protected] >>>>> >>>>> >>>> >>>> --------------------------------------------------------------------- >>>> To unsubscribe, e-mail: [email protected] >>>> For additional commands, e-mail: [email protected] >>>> >>> >>> --------------------------------------------------------------------- >>> To unsubscribe, e-mail: [email protected] >>> For additional commands, e-mail: [email protected] >>> >>> >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: [email protected] >> For additional commands, e-mail: [email protected] >> > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [email protected] > For additional commands, e-mail: [email protected] > > --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
