Has there been any news? I'd love to get this tied up so that you're able to proceed. Karl
On Thu, Jan 16, 2020 at 12:08 PM Jörn Franke <jornfra...@gmail.com> wrote: > Ok I understand. I will try and let you know. Thanks again very much for > your fast and detailed answer. Really appreciated. I hope I can give back > with the solution to fetch WSDLs from https and maybe a solution to this > problem (maybe other will face this as well). > > About the connector: the WSDL is successfully fetched via https (not file > - no clue why) - after the modification I made. The only problem I see now > is that the xsd to which the WSDL is referring are not fetched. The bizarre > thing is that the https url that it mention for the xsd is absolutely > correct. So I assume it does not understand an http url, maybe that is > related to configuration. > > Am 16.01.2020 um 14:53 schrieb Karl Wright <daddy...@gmail.com>: > > > The WSDLS are bundled with the jar. We intended this to be the ONLY way > the wsdls were accessed, and made lots of changes to the wsdls accordingly, > so that they referenced other wsdls via the "file system". The wsdls are > the fixed up ones that are used to build the java stubs locally, plus a > config file that's supposed to tell CXF how to resolve referenced wsdls. > That config file may or may not be correct, because we never were able to > get CXF to use the local resource wsdls during actual connection. > > Except now they seem to be both fetched via https AND locally sourced. I > have no idea how that can be. I had assumed it was done one way or the > other but not both. > > Perhaps the problem is that the configuration file is being read but the > resource wsdls are not being found? Removing the meta-inf from the jar > would then force everything to go through https. Ideally I'd love it if > that wasn't needed and we could get the resource fetch working everywhere. > > > Karl > > > On Thu, Jan 16, 2020 at 8:20 AM Jörn Franke <jornfra...@gmail.com> wrote: > >> Well i am not sure how they solved it - I will share a tested solution in >> Jira and everyone can check. Maybe their wsdl is accessible through http? >> >> What works is doing call through https, but thee fetching of WSDL did >> not - as this is through another mechanism. >> >> I don’t think that the open text is different, the WSDL look very similar >> to the repository. >> >> The strange thing is that for this error message it tries to access the >> xsd through a https url (which is perfectly accessible for the server). >> Could it be that the connector restrict itself somehow to local file >> system only or similar? >> Have you faced this issue before? >> >> >> >> Am 16.01.2020 um 12:56 schrieb Karl Wright <daddy...@gmail.com>: >> >> >> I should say that we have (AFAICT) at least two independent installations >> of the csws connector working in the field, at least one of them using >> secure connections. >> >> Karl >> >> >> On Thu, Jan 16, 2020 at 6:54 AM Karl Wright <daddy...@gmail.com> wrote: >> >>> We solved the WSDL fetching through HTTPS, or thought we had, by >>> restructuring the code according to a number of articles we found. This >>> was supposedly tested and worked in one installation. Nobody has ever >>> reported issues with the wsdls being fetched however; I worry that you may >>> have a different version of OpenText that is incompatible with the one we >>> developed against. That's the problem with this kind of architecture; >>> unless the wsdls are included in the jar there can be issues. We tried to >>> do that too but were unable to get it to work. >>> >>> Karl >>> >>> >>> On Thu, Jan 16, 2020 at 5:34 AM Jörn Franke <jornfra...@gmail.com> >>> wrote: >>> >>>> Ok i fixed the source to fetch WSDL from https (it is not perfect yet >>>> as it does not use the truststore yet but this I can fix) - I will share >>>> later in a Jira. >>>> It is however now unable to locate the imported document >>>> /Authentication?xsd=2 relative to Authenticaton?wsdl#types1 >>>> >>>> I will look into this, but if someone has come cross it then please let >>>> me know. >>>> >>>> Am 16.01.2020 um 10:22 schrieb Jörn Franke <jornfra...@gmail.com>: >>>> >>>> >>>> Coming back to the original topic. I believe SSL was never fully solved >>>> from what i read in the corresponding issue. Apparently, the fetching of >>>> the WSDL itself through https was not possible. Do you remember still some >>>> insights beyond what is written in the issue ? >>>> >>>> Am 16.01.2020 um 00:37 schrieb Karl Wright <daddy...@gmail.com>: >>>> >>>> >>>> Let me think about that option. >>>> >>>> Karl >>>> >>>> >>>> On Wed, Jan 15, 2020 at 5:38 PM Jörn Franke <jornfra...@gmail.com> >>>> wrote: >>>> >>>>> We could make it configurable, e.g. in properties.xml. Here people >>>>> could set it to SSL, TLS, TLSv1.2 (to restrict it to TLS1.2 => some >>>>> companies may want that!). Is this a viable option? That would be also >>>>> future proof. We can leave it by default to SSL, but we should put in the >>>>> example config files TLS by default (so new starters do not get even the >>>>> idea to use an outdated protocol) AND put a comment with recommendation to >>>>> use/enforce always newest protocols for security reasons. Of course, the >>>>> choice is then with the people using the software. >>>>> Could that be something sensible from your point of view? >>>>> >>>>> On Wed, Jan 15, 2020 at 11:14 PM Karl Wright <daddy...@gmail.com> >>>>> wrote: >>>>> >>>>>> It's rather immaterial what browsers do here. What's important is >>>>>> what *existing servers* support, since that is what we're connecting >>>>>> with. >>>>>> >>>>>> I tend to agree that *most* people have probably upgraded to web >>>>>> servers that support TLS. But we can't guarantee it, nor can we assume >>>>>> that people have upgraded to the most modern version of TLS exclusively. >>>>>> In fact I think we can assume they have *not*. When the SSL issues were >>>>>> discovered a couple of years back, the standard recommendation was simply >>>>>> to *disable* SSLv1 and SSLv2, not to upgrade to Java 11 or some such. We >>>>>> still support (and have people using!!) early forms of NTLM (v1 to be >>>>>> specific), for instance. We're not going to be able to wag the dog here. >>>>>> Breaking changes of this kind usually mean we go to a whole new major >>>>>> version of MCF. >>>>>> >>>>>> However, if you can show that SSLContext.getSSLFactory("TLS") >>>>>> produces a SSLSocketFactory that works with all versions of TLS and SSL >>>>>> that do not have known security holes, I would support changing over to >>>>>> that. If it turns out we need much more specificity about the kind of >>>>>> SSLSocketFactory we produce, then we need a better solution anyhow for >>>>>> handling multiple protocols in one socket factory. >>>>>> >>>>>> Karl >>>>>> >>>>>> >>>>>> On Wed, Jan 15, 2020 at 5:17 AM Jörn Franke <jornfra...@gmail.com> >>>>>> wrote: >>>>>> >>>>>>> Hi Karl, >>>>>>> >>>>>>> No it does not. I can look into that further, but Current browsers >>>>>>> stop supporting anything below TLSv1.2 in March 2020. >>>>>>> Then TLS exists since more than ten years. I expect any server >>>>>>> running nowadays will always have tls support. >>>>>>> SSL itself is not supported since some time now. From a security >>>>>>> perspective it should even break servers that run only SSL as they are >>>>>>> inherently insecure and also clients that only support SSL are adding to >>>>>>> this. >>>>>>> However if you have an idea how this should be made configurable >>>>>>> then I can look into this. >>>>>>> >>>>>>> Best regards >>>>>>> >>>>>>> Am 15.01.2020 um 10:52 schrieb Karl Wright <daddy...@gmail.com>: >>>>>>> >>>>>>> >>>>>>> Hi, >>>>>>> >>>>>>> Mcf currently requires jdk8. Jdk11 is non trivial to support >>>>>>> because of the removal of many jdk classes connectors need. It will be >>>>>>> ported at some point but not lightly. >>>>>>> >>>>>>> Similarly, disabling SSL would certainly break many installations >>>>>>> upon upgrade and we do not do that lightly. >>>>>>> >>>>>>> The core methods that mcf supplies its connectors should therefore >>>>>>> be updated to support but not mandate tls. The protocol specification >>>>>>> one >>>>>>> gives to sslcontext is not a detailed one but rather a major version. >>>>>>> What >>>>>>> I don't know is whether"tlsv1" also allows for older protocols etc. >>>>>>> >>>>>>> Karl >>>>>>> >>>>>>> On Wed, Jan 15, 2020, 1:19 AM Jörn Franke <jornfra...@gmail.com> >>>>>>> wrote: >>>>>>> >>>>>>>> Yes I am doing that but I will need to rebuild. >>>>>>>> I don’t recommend TLSv1 - this is already outphased and will lock >>>>>>>> out TLSv1.2. I try TLS only as it includes all TLS protocols (depends >>>>>>>> on >>>>>>>> JDK). >>>>>>>> >>>>>>>> SSL will not be supported by this (however as I said there are >>>>>>>> other parts of the code where there is a getInstance(TLS). And some >>>>>>>> caveats: On JDK6+7 TLS only means TLSv1 (and newer TLS Protocols are >>>>>>>> deactivated) on JDK8 it means also that newer TLS protocols are >>>>>>>> enabled. >>>>>>>> To be honest in my opinion - a SSL only one is a significant >>>>>>>> security hole and given how old TLS support is JDK i would be >>>>>>>> surprised if >>>>>>>> there is someone using such a server (most Organisations should switch >>>>>>>> to >>>>>>>> TLSv1.2 in any case as all protocols below have been broken). >>>>>>>> While it works for all JDKs - probably JDK8 should be recommended >>>>>>>> as it seems to have all TLS protocols activated when using „TLS“. Older >>>>>>>> JDKs seem to deactivate TLSv1.1 and TLSv1.2 when using TLS. I will >>>>>>>> write >>>>>>>> more about this in the JIRA, once I verified that this solves the >>>>>>>> problem. >>>>>>>> Then TLSv1.3 is JDK11 only - I will investigate what that implies. >>>>>>>> Does ManifoldCf supports JDK11? >>>>>>>> >>>>>>>> Am 15.01.2020 um 00:08 schrieb Karl Wright <daddy...@gmail.com>: >>>>>>>> >>>>>>>> >>>>>>>> I think you can just change the code to read as follows when it >>>>>>>> creates the SSLContext: >>>>>>>> >>>>>>>> SSLContext ctx = SSLContext.getInstance("TLSv1"); >>>>>>>> I don't know if TLS will downgrade to SSL if that's all that's >>>>>>>> available. >>>>>>>> >>>>>>>> >>>>>>>> Karl >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On Tue, Jan 14, 2020 at 6:02 PM Jörn Franke <jornfra...@gmail.com> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> Yes it you do not change this setting as what I suspect happens >>>>>>>>> here. See my previous mail for details. >>>>>>>>> >>>>>>>>> Am 14.01.2020 um 23:51 schrieb Karl Wright <daddy...@gmail.com>: >>>>>>>>> >>>>>>>>> >>>>>>>>> It looks looks TLS is actually enabled in the SSLSocketFactory >>>>>>>>> framework based on how you create the SSLSocketContext. See: >>>>>>>>> >>>>>>>>> >>>>>>>>> https://docs.oracle.com/cd/E19698-01/816-7609/security-83/index.html >>>>>>>>> >>>>>>>>> >>>>>>>>> Karl >>>>>>>>> >>>>>>>>> >>>>>>>>> On Tue, Jan 14, 2020 at 5:48 PM Karl Wright <daddy...@gmail.com> >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> The design of ManifoldCF deliberately manages keystores on a >>>>>>>>>> connection by connection basis, not globally. If you think the only >>>>>>>>>> way to >>>>>>>>>> implement TLS is via global keystore I very much doubt it. >>>>>>>>>> >>>>>>>>>> I am on the road until late tomorrow but somewhere along the line >>>>>>>>>> I can do some research into why TLS won't work as we are currently >>>>>>>>>> doing it. >>>>>>>>>> >>>>>>>>>> Karl >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Tue, Jan 14, 2020 at 12:56 PM Jörn Franke < >>>>>>>>>> jornfra...@gmail.com> wrote: >>>>>>>>>> >>>>>>>>>>> These are TLS only. So maybe you have other servers where tls >>>>>>>>>>> and ssl are possible and it downgrades to ssl.however, this is >>>>>>>>>>> speculation >>>>>>>>>>> and I need to verify it. I have to rebuilt manifold for that. >>>>>>>>>>> Probably I >>>>>>>>>>> have to reinstall everything as the keystorefactory is a dependency >>>>>>>>>>> in the >>>>>>>>>>> connector. >>>>>>>>>>> >>>>>>>>>>> Am 14.01.2020 um 18:34 schrieb Karl Wright <daddy...@gmail.com>: >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> If you can recommend changes to support TLS, that would be >>>>>>>>>>> great. The basic infrastructure should still work; it is just a >>>>>>>>>>> custom >>>>>>>>>>> keystone and associated SSLSocketFactory, which I think also is >>>>>>>>>>> used for >>>>>>>>>>> TLS connections, unless I am missing something. >>>>>>>>>>> >>>>>>>>>>> On Tue, Jan 14, 2020, 9:38 AM Jörn Franke <jornfra...@gmail.com> >>>>>>>>>>> wrote: >>>>>>>>>>> >>>>>>>>>>>> Yes this works fine. I believe the error comes from the fact >>>>>>>>>>>> that TLS connections are not supported. >>>>>>>>>>>> >>>>>>>>>>>> Am 14.01.2020 um 15:31 schrieb Michael Cizmar < >>>>>>>>>>>> michael.ciz...@mcplusa.com>: >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> If you want to test the url and the ssl, I would recommend >>>>>>>>>>>> attempting using SSLPoke to confirm that they keystore is setup >>>>>>>>>>>> properly: >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> https://github.com/MichalHecko/SSLPoke >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Michael >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> *From: *Karl Wright <daddy...@gmail.com> >>>>>>>>>>>> *Reply-To: *"user@manifoldcf.apache.org" < >>>>>>>>>>>> user@manifoldcf.apache.org> >>>>>>>>>>>> *Date: *Tuesday, January 14, 2020 at 7:21 AM >>>>>>>>>>>> *To: *"user@manifoldcf.apache.org" <user@manifoldcf.apache.org> >>>>>>>>>>>> *Subject: *Re: CSWS Connector : ServiceConstructionException: >>>>>>>>>>>> Failed to create service >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Hmm, others have succeeded setting up SSL connections with the >>>>>>>>>>>> current code. Hoping they chime in here. >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Karl >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> On Tue, Jan 14, 2020, 8:19 AM Jörn Franke <jornfra...@gmail.com> >>>>>>>>>>>> wrote: >>>>>>>>>>>> >>>>>>>>>>>> It seems that it has indeed a certificate issue as it cannot >>>>>>>>>>>> find a valid certification path to the target. The thing is: I >>>>>>>>>>>> added those >>>>>>>>>>>> certificates in the UI should it should not happen. >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Am 10.01.2020 um 20:51 schrieb Jörn Franke < >>>>>>>>>>>> jornfra...@gmail.com>: >>>>>>>>>>>> >>>>>>>>>>>> 2.15 ... >>>>>>>>>>>> >>>>>>>>>>>> I will try on the weekend to see if I can get some logs out of >>>>>>>>>>>> it. >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Am 10.01.2020 um 19:02 schrieb Karl Wright <daddy...@gmail.com >>>>>>>>>>>> >: >>>>>>>>>>>> >>>>>>>>>>>> Can I ask what version of MCF you are using? There were issues >>>>>>>>>>>> with SSL in the first release of the csws connector if I recall >>>>>>>>>>>> correctly, >>>>>>>>>>>> that were fixed for the second release. >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Karl >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> On Fri, Jan 10, 2020 at 11:42 AM Jörn Franke < >>>>>>>>>>>> jornfra...@gmail.com> wrote: >>>>>>>>>>>> >>>>>>>>>>>> I added root, intermediate and server certificate (in base64 >>>>>>>>>>>> cer, it seems to be recognized by manifoldcf), but I still get the >>>>>>>>>>>> same >>>>>>>>>>>> message. I will try to get somehow the full stacktrace >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Am 10.01.2020 um 17:21 schrieb Karl Wright <daddy...@gmail.com >>>>>>>>>>>> >: >>>>>>>>>>>> >>>>>>>>>>>> If you are using SSL you need to have the proper certificate >>>>>>>>>>>> saved in the connection's keystore. >>>>>>>>>>>> >>>>>>>>>>>> Karl >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> On Fri, Jan 10, 2020 at 11:20 AM Jörn Franke < >>>>>>>>>>>> jornfra...@gmail.com> wrote: >>>>>>>>>>>> >>>>>>>>>>>> It is actually a server using configuration of the command - >>>>>>>>>>>> driven multi-process model (but the agents executed as a service >>>>>>>>>>>> and the >>>>>>>>>>>> war on a tomcat executed as a service) under Linux. >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> I thought as well that it cannot reach the webservices, the >>>>>>>>>>>> question is why. On the same server I can reach the webservices >>>>>>>>>>>> and fetch >>>>>>>>>>>> the WSDL without issues. >>>>>>>>>>>> >>>>>>>>>>>> Maybe sth related to ssl ? >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Am 10.01.2020 um 14:59 schrieb Karl Wright <daddy...@gmail.com >>>>>>>>>>>> >: >>>>>>>>>>>> >>>>>>>>>>>> How are you running manifoldcf? Single process example, or a >>>>>>>>>>>> custom setup of some kind? >>>>>>>>>>>> >>>>>>>>>>>> This exception is a "catch all" exception generated far below >>>>>>>>>>>> anything in ManifoldCF, but usually means it cannot download the >>>>>>>>>>>> WSDLs from >>>>>>>>>>>> the service. Getting the full exception dumped in the log >>>>>>>>>>>> requires a >>>>>>>>>>>> "hack" to the check() method of the connector, but I'm pretty sure >>>>>>>>>>>> that's >>>>>>>>>>>> what's happening anyway. >>>>>>>>>>>> >>>>>>>>>>>> Karl >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> On Fri, Jan 10, 2020 at 8:50 AM Jörn Franke < >>>>>>>>>>>> jornfra...@gmail.com> wrote: >>>>>>>>>>>> >>>>>>>>>>>> Hi, >>>>>>>>>>>> >>>>>>>>>>>> I tried to use the CSWS connector, but already for the >>>>>>>>>>>> Authority connection I receive a >>>>>>>>>>>> org.apache.cxf.service.factory.ServiceConstructionException: >>>>>>>>>>>> Failed to >>>>>>>>>>>> create service. >>>>>>>>>>>> >>>>>>>>>>>> Unfortunately I don’t see more details , also not in the log >>>>>>>>>>>> (debug is activated). I try to get a little bit more output by >>>>>>>>>>>> modifying >>>>>>>>>>>> the connector, but maybe someone has already an idea why this can >>>>>>>>>>>> happen? >>>>>>>>>>>> >>>>>>>>>>>> Are there some special instructions to use it? The pointers to >>>>>>>>>>>> the webservices are correct, I tested via Curl and SOAPUI. >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Thank you. >>>>>>>>>>>> Best regards >>>>>>>>>>>> >>>>>>>>>>>>