Re: svn update via HTTPS works 95% of the time, then randomly shanks, issuer not trusted
Or to state the below (pardon the top-post) much more simply: the --trust-server-cert flag does not work. It fails to perform its singular function, which is...to force trust of the server cert, right? (Though I still think something else is going on, I can start there). On Mon, Jul 25, 2011 at 1:12 PM, Dan Yost yod...@gmail.com wrote: Greetings, I'm at my wit's end trying to debug a troubling, highly irregular issue connecting to a SVN server via HTTPS. I am very confident this is a client issue, not a server issue. The server is CollabNet's Subversion Edge server, which has 1.6.15 built in. The client is 1.6.15 running under Cygwin. I have many, many clients, most of which are running Windows Server 2003 (if that matters)--all Cygwin, all this same package. Lots of traffic every day. Almost all of it works just fine. Now, a very important fact: the svn operations WORK 95% of the time. They just work. Everything is absolutely fine. This is an unattended, fully-automated environment, launched from cron (well, Task Scheduler when running on these Windows boxes). 95% of the time, the job launches, runs my svn update command from within one of its working copies: svn update --username=some --password=thing --non-interactive and it works beautifully. Every once in awhile, for absolutely no reason I can find (have looked at logs, logs, logs, timing, the stars and planets, everything), with absolutely nothing that I can find having changed, it dies with this: svn: OPTIONS of 'https://myserver:/svn/some/path': Server certificate verification failed: certificate issued for a different hostname, issuer is not trusted (https://myserver:) (I have changed the paths/ports to protect the innocent). Now for most of my clients, the issued for a different hostname part is omitted--they still have the issuer not trusted part. That's because my one host that I'm really digging in to debug this has to use a different hostname for certain reasons. So, focus on the issuer not trusted part because that's what we see across all clients. But keep in mind: it works. It works 95% of the time. I change absolutely nothing. Having found no indication of why it will *randomly* fail with the above error, I decided to add the --trust-cerver-cert flag, thinking surely this will eliminate all woes. Just trust it and go (trust it like you already do 95% of the time anyway)! svn update --username=some --password=thing --non-interactive --trust-server-cert But, nope, I randomly get the very same error. These key facts: 1) It works 95% of the time (well, maybe 85%) 2) I can find NOTHING changing (on client or server) between a working op and a breaking op--it's all automated, and just randomly shanks are leading me to believe that this error message is actually a complete wild goose chase, and that the problem is not actually cert-based but is something else (with a wrong error message). In looking at the server HTTP logs, it does indeed appear that the client bails out before continuing the connection--there's no log, meaning it does appear that the client can't handshake (or refuses to), and then quits. I have thought about adding options to config (on the command line) such as ssl-trust-default-ca, but again, I already told svn to --trust-server-cert and it randomly still dies, so I don't think that will help. You'll be tempted to focus on SSL cert issues on the server or something, but remember that we don't change anything and it works fine almost all the time! (I'm still open to any thoughts of course). Can anybody point me to some other debug options, some other flag, or any other way to determine why, 15% of the time, it dies, yet 85% of the time it updates just fine? Thanks, Dan
Re: svn update via HTTPS works 95% of the time, then randomly shanks, issuer not trusted
On Tue, Jul 26, 2011 at 2:19 PM, Daniel Shahaf d...@daniel.shahaf.name wrote: Dan Yost wrote on Tue, Jul 26, 2011 at 12:57:29 -0500: Or to state the below (pardon the top-post) much more simply: the --trust-server-cert flag does not work. It fails to perform its singular function, which is...to force trust of the server cert, right? Its function is to accept certificates signed by unknown CA's without prompting. In your case you have two failures, one of them being the mismatching subject name (hostname), so you do get prompted. Ah, so indeed perhaps this will reveal a key difference between my test client and the rest of the live clients (still gathering info on them to see if they continue to fail). The rest of the clients don't have the hostname issue, I believe, so perhaps they will indeed be improved by the flag. But, any idea why it works one minute, then doesn't the next, when nobody touches anything? Or, a good way to dig deeper? Perhaps I need a very verbose dump of the whole SSL handshake, I guess, to find out why it works fine one minute then doesn't the next, but I didn't find an option (particularly on the update subcommand). Thanks, Dan
Re: svn update via HTTPS works 95% of the time, then randomly shanks, issuer not trusted
On Tue, Jul 26, 2011 at 2:32 PM, Daniel Shahaf d...@daniel.shahaf.name wrote: Daniel Shahaf wrote on Tue, Jul 26, 2011 at 22:19:13 +0300: Dan Yost wrote on Tue, Jul 26, 2011 at 12:57:29 -0500: Or to state the below (pardon the top-post) much more simply: the --trust-server-cert flag does not work. It fails to perform its singular function, which is...to force trust of the server cert, right? Its function is to accept certificates signed by unknown CA's without prompting. In your case you have two failures, one of them being the mismatching subject name (hostname), so you do get prompted. See ssl_trust_unknown_server_cert() in subversion/libsvn_subr/cmdline.c. There were discussions about extending this to, for example, --trust-server-cert2=comma,separated,list,of,failures,to,ignore , and I think someone may have started working on a patch, but they never submitted it to us. A possibility, indeed. What's still very odd is that there's actually no error--nothing to ignore. That is, 95% of the time, then randomly (5%) it shanks, and nobody touched anything (on either server side or client side). That's what is making me crazy. Dan
Re: svn update via HTTPS works 95% of the time, then randomly shanks, issuer not trusted
On Tue, Jul 26, 2011 at 3:11 PM, Geoff Hoffman ghoff...@cardinalpath.com wrote: Long shot here... this is probably off base, as I am not that experienced with lower-level SSL problems, but are you by chance using an issuer that provides an intermediary certificate? For example, to install an SSL cert from GoDaddy, you have to also include the gd_bundle.crt. The Wikipedia article below makes me wonder if there is just some network hiccups sometimes, trying to validate your certificate chain authority. From http://en.wikipedia.org/wiki/Intermediate_certificate_authorities If the certificate was not issued by a trusted CA, the connecting device (e.g., a web browser) will then check to see if the issuing CA of the certificate was issued by a trusted CA, and so on until either a trusted CA is found (at which point a trusted, secure connection will be established) or no trusted CA can be found (at which point the device will usually display an error). Yes, and indeed this is a GoDaddy cert, with the bundle installed to keep the chain intact, so thus it does work that 95% of the time. I was thinking that the chain is all presented from the server to client in one fell swoop, with no need to go fetch anything else out there (not that you're suggesting that is what it needs to do--go outside to fetch something). But indeed, I suppose it could complicate the handshake in such a way as to cause this intermittent failure--would really like to be able to watch that happen via some kind of verbose log. Dan
svn update via HTTPS works 95% of the time, then randomly shanks, issuer not trusted
Greetings, I'm at my wit's end trying to debug a troubling, highly irregular issue connecting to a SVN server via HTTPS. I am very confident this is a client issue, not a server issue. The server is CollabNet's Subversion Edge server, which has 1.6.15 built in. The client is 1.6.15 running under Cygwin. I have many, many clients, most of which are running Windows Server 2003 (if that matters)--all Cygwin, all this same package. Lots of traffic every day. Almost all of it works just fine. Now, a very important fact: the svn operations WORK 95% of the time. They just work. Everything is absolutely fine. This is an unattended, fully-automated environment, launched from cron (well, Task Scheduler when running on these Windows boxes). 95% of the time, the job launches, runs my svn update command from within one of its working copies: svn update --username=some --password=thing --non-interactive and it works beautifully. Every once in awhile, for absolutely no reason I can find (have looked at logs, logs, logs, timing, the stars and planets, everything), with absolutely nothing that I can find having changed, it dies with this: svn: OPTIONS of 'https://myserver:/svn/some/path': Server certificate verification failed: certificate issued for a different hostname, issuer is not trusted (https://myserver:) (I have changed the paths/ports to protect the innocent). Now for most of my clients, the issued for a different hostname part is omitted--they still have the issuer not trusted part. That's because my one host that I'm really digging in to debug this has to use a different hostname for certain reasons. So, focus on the issuer not trusted part because that's what we see across all clients. But keep in mind: it works. It works 95% of the time. I change absolutely nothing. Having found no indication of why it will *randomly* fail with the above error, I decided to add the --trust-cerver-cert flag, thinking surely this will eliminate all woes. Just trust it and go (trust it like you already do 95% of the time anyway)! svn update --username=some --password=thing --non-interactive --trust-server-cert But, nope, I randomly get the very same error. These key facts: 1) It works 95% of the time (well, maybe 85%) 2) I can find NOTHING changing (on client or server) between a working op and a breaking op--it's all automated, and just randomly shanks are leading me to believe that this error message is actually a complete wild goose chase, and that the problem is not actually cert-based but is something else (with a wrong error message). In looking at the server HTTP logs, it does indeed appear that the client bails out before continuing the connection--there's no log, meaning it does appear that the client can't handshake (or refuses to), and then quits. I have thought about adding options to config (on the command line) such as ssl-trust-default-ca, but again, I already told svn to --trust-server-cert and it randomly still dies, so I don't think that will help. You'll be tempted to focus on SSL cert issues on the server or something, but remember that we don't change anything and it works fine almost all the time! (I'm still open to any thoughts of course). Can anybody point me to some other debug options, some other flag, or any other way to determine why, 15% of the time, it dies, yet 85% of the time it updates just fine? Thanks, Dan
