This is an automated email from the ASF dual-hosted git repository.
hansva pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/hop.git
The following commit(s) were added to refs/heads/main by this push:
new 3b5de5d841 fix wdsl file load, fixes #4813 (#8552)
3b5de5d841 is described below
commit 3b5de5d841a0f332023b56899b96fbc3919cbc49
Author: Hans Van Akelyen <[email protected]>
AuthorDate: Thu Sep 24 10:07:58 2026 +0200
fix wdsl file load, fixes #4813 (#8552)
* fix wdsl file load, fixes #4813
* fix remarks
---
.../pages/pipeline/transforms/webservices.adoc | 37 +-
.../http/0013-webservice-local-wsdl.hpl | 225 +++++++++++++
integration-tests/http/main-0013-webservice.hwf | 35 ++
integration-tests/http/wsdl/BookService.wsdl | 179 ++++++++++
.../transforms/webservices/WebService.java | 162 +++++----
.../transforms/webservices/WebServiceDialog.java | 30 +-
.../webservices/wsdl/HopVfsWsdlLocator.java | 227 +++++++++++++
.../pipeline/transforms/webservices/wsdl/Wsdl.java | 141 +++++---
.../webservices/messages/messages_en_US.properties | 4 +-
.../transforms/webservices/WebServiceTest.java | 163 +++++++++
.../transforms/webservices/wsdl/WsdlTest.java | 374 +++++++++++++++++++++
.../webservices/wsdl/abstract/hello-abstract.wsdl | 34 ++
.../transforms/webservices/wsdl/hello-service.wsdl | 38 +++
.../transforms/webservices/wsdl/hello-typed.wsdl | 61 ++++
.../transforms/webservices/wsdl/hello.wsdl | 48 +++
.../webservices/wsdl/schema/hello-types.xsd | 36 ++
16 files changed, 1647 insertions(+), 147 deletions(-)
diff --git
a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/webservices.adoc
b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/webservices.adoc
index 1c8591c1dd..7ff5567511 100644
---
a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/webservices.adoc
+++
b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/webservices.adoc
@@ -31,9 +31,18 @@ The Web Services Lookup transform performs a Web Services
lookup using the Web S
|===
|Option|Description
|Transform Name|Name of the transform this name has to be unique in a single
pipeline.
-|URL|The base URL pointing to the WSDL document that will be retrieved
+|URL|The location of the WSDL document.
+This is either an `http://` or `https://` URL, or a file, for example
`${PROJECT_HOME}/wsdl/service.wsdl`.
+A file can be on any file system Hop can read: a local folder, a `file://`
URL, S3, Azure, a named VFS connection and so on.
+Variables are resolved, both in the dialog and when the pipeline runs.
+|Browse... (button)|Select a WSDL file.
+Its location is put in the URL field and the WSDL is loaded, as with the Load
button.
+A file in your project folder is kept relative to `${PROJECT_HOME}`, so the
pipeline keeps working when the project moves.
|Load (button)|The load button loads the WSDL at the specified URL and tries
to automatically populate the input and output tabs and fields.
NOTE: If this doesn't work, you can still try to manually specify the input
and output fields using the "Add Input" and "Add Output" buttons.
+|Operation|The operation of the web service to call.
+|Operation request name (optional)|On occasion the WSDL gives back the wrong
operation request name to put in the body.
+Put in the correct one (as expected by the service) here.
|The number of rows per call|
The number of rows to send with each WSDL call
|Pass input data to output|
@@ -42,6 +51,9 @@ v2.x/3.x compatibility mode We improved the parsing engine of
the 3.x engine dra
For compatibility reasons, we kept the old engine around to make sure older
transforms would still work OK.
|Repeating element name|
Specify the name of the repeating element in the output XML (if any).
+|Return the complete reply from the service as a String|
+Don't parse the reply of the service: pass the complete SOAP reply on as a
String.
+You need to specify a single String output field to receive it.
|HTTP authentication|
Here you can put the username and password if these are required for the web
service.
|Proxy to use|
@@ -50,3 +62,26 @@ Here you can optionally put the proxy host and port
information.
These buttons will allow you to manually specify the input and output
specifications of the WSDL service.
Note: The data of the input fields can be escaped by the Calculator transform
and the function "Mask XML content from string A"
|===
+
+== WSDL files and imports
+
+A WSDL can import other WSDL documents (`<import location="...">`) and XML
schemas (`<xsd:import schemaLocation="...">`, `<xsd:include>`).
+A relative location is looked up next to the document that names it, on the
same file system or web server as that document: a WSDL on S3 finds its imports
on S3.
+An imported document can import others in turn.
+
+HTTP authentication is used to fetch the WSDL over `http://` or `https://`,
its imports from the same host, and to call the service itself.
+It is not sent to any other host a WSDL imports from.
+
+== Input and output fields
+
+After loading the WSDL and selecting an operation, the input and output tabs
list the fields of the request and the reply.
+When a WSDL can't be loaded, you can fill them in yourself with "Add Input"
and "Add Output", for example from a request and reply you captured with a tool
like SoapUI.
+
+[options="header"]
+|===
+|Column|Description
+|Name|The name of the field in the Hop pipeline: the input field that provides
the value, or the output field that receives it.
+|WS Name|The name of the XML element in the SOAP request or reply, without its
namespace prefix.
+|WS Type|The XML schema type of the element: `string`, `int`, `integer`,
`short`, `boolean`, `date`, `time`, `datetime`, `double`, `float`,
`base64Binary` or `decimal`.
+Dates, times and booleans are formatted to match this type in the request.
+|===
diff --git a/integration-tests/http/0013-webservice-local-wsdl.hpl
b/integration-tests/http/0013-webservice-local-wsdl.hpl
new file mode 100644
index 0000000000..99f69232c6
--- /dev/null
+++ b/integration-tests/http/0013-webservice-local-wsdl.hpl
@@ -0,0 +1,225 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to You under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+
+-->
+<pipeline>
+ <info>
+ <name>0013-webservice-local-wsdl</name>
+ <name_sync_with_filename>Y</name_sync_with_filename>
+ <description>Issue #4813: the WSDL is a file in the project, addressed
through ${PROJECT_HOME}. It used to be read as an http URL only, so a plain
path failed. The service itself is still the castlemock mock the WSDL points
to.</description>
+ <extended_description/>
+ <pipeline_version/>
+ <pipeline_type>Normal</pipeline_type>
+ <parameters>
+ </parameters>
+ <capture_transform_performance>N</capture_transform_performance>
+
<transform_performance_capturing_delay>1000</transform_performance_capturing_delay>
+
<transform_performance_capturing_size_limit>100</transform_performance_capturing_size_limit>
+ <created_user>-</created_user>
+ <created_date>2026/09/23 10:00:00.000</created_date>
+ <modified_user>-</modified_user>
+ <modified_date>2026/09/23 10:00:00.000</modified_date>
+ <key_for_session_key>H4sIAAAAAAAAAAMAAAAAAAAAAAA=</key_for_session_key>
+ <is_key_private>N</is_key_private>
+ </info>
+ <notepads>
+ </notepads>
+ <order>
+ <hop>
+ <from>Generate rows</from>
+ <to>Web services lookup</to>
+ <enabled>Y</enabled>
+ </hop>
+ <hop>
+ <from>Web services lookup</from>
+ <to>check reply</to>
+ <enabled>Y</enabled>
+ </hop>
+ <hop>
+ <from>Web services lookup</from>
+ <to>no reply</to>
+ <enabled>Y</enabled>
+ </hop>
+ <hop>
+ <from>check reply</from>
+ <to>Abort</to>
+ <enabled>Y</enabled>
+ </hop>
+ <hop>
+ <from>no reply</from>
+ <to>Abort</to>
+ <enabled>Y</enabled>
+ </hop>
+ </order>
+ <transform>
+ <name>Generate rows</name>
+ <type>RowGenerator</type>
+ <description/>
+ <distribute>Y</distribute>
+ <custom_distribution/>
+ <copies>1</copies>
+ <partitioning>
+ <method>none</method>
+ <schema_name/>
+ </partitioning>
+ <fields>
+ <field>
+ <length>-1</length>
+ <name>body</name>
+ <precision>-1</precision>
+ <set_empty_string>N</set_empty_string>
+ <type>String</type>
+ <nullif>this is a body</nullif>
+ </field>
+ </fields>
+ <interval_in_ms>5000</interval_in_ms>
+ <last_time_field>FiveSecondsAgo</last_time_field>
+ <never_ending>N</never_ending>
+ <limit>1</limit>
+ <row_time_field>now</row_time_field>
+ <attributes/>
+ <GUI>
+ <xloc>112</xloc>
+ <yloc>64</yloc>
+ </GUI>
+ </transform>
+ <transform>
+ <name>Web services lookup</name>
+ <type>WebServiceLookup</type>
+ <description/>
+ <distribute>N</distribute>
+ <custom_distribution/>
+ <copies>1</copies>
+ <partitioning>
+ <method>none</method>
+ <schema_name/>
+ </partitioning>
+ <wsURL>${PROJECT_HOME}/wsdl/BookService.wsdl</wsURL>
+ <wsOperation>AddBook</wsOperation>
+ <wsOperationRequest/>
+
<wsOperationNamespace>http://www.cleverbuilder.com/BookService/</wsOperationNamespace>
+ <wsInFieldContainer/>
+ <wsInFieldArgument/>
+ <wsOutFieldContainer>Book</wsOutFieldContainer>
+ <wsOutFieldArgument>Book</wsOutFieldArgument>
+ <proxyHost/>
+ <proxyPort/>
+ <httpLogin>admin</httpLogin>
+ <httpPassword>admin</httpPassword>
+ <callTransform>1</callTransform>
+ <passingInputData>Y</passingInputData>
+ <compatible>N</compatible>
+ <repeating_element/>
+ <reply_as_string>Y</reply_as_string>
+ <fieldsIn>
+ <field>
+ <name>body</name>
+ <wsName>Book</wsName>
+ <xsdType>Book</xsdType>
+ </field>
+ </fieldsIn>
+ <fieldsOut>
+ <field>
+ <name>reply</name>
+ <wsName>AddBookResponse</wsName>
+ <xsdType>string</xsdType>
+ </field>
+ </fieldsOut>
+ <attributes/>
+ <GUI>
+ <xloc>304</xloc>
+ <yloc>64</yloc>
+ </GUI>
+ </transform>
+ <transform>
+ <name>check reply</name>
+ <type>FilterRows</type>
+ <description>Fails unless the reply of the mocked AddBook operation came
back.</description>
+ <distribute>Y</distribute>
+ <custom_distribution/>
+ <copies>1</copies>
+ <partitioning>
+ <method>none</method>
+ <schema_name/>
+ </partitioning>
+ <send_true_to/>
+ <send_false_to/>
+ <compare>
+ <condition>
+ <negated>Y</negated>
+ <leftvalue>reply</leftvalue>
+ <function>CONTAINS</function>
+ <rightvalue/>
+ <value>
+ <name>constant</name>
+ <type>String</type>
+ <text>AddBookResponse</text>
+ <length>-1</length>
+ <precision>-1</precision>
+ <isnull>N</isnull>
+ <mask></mask>
+ </value>
+ </condition>
+ </compare>
+ <attributes/>
+ <GUI>
+ <xloc>496</xloc>
+ <yloc>64</yloc>
+ </GUI>
+ </transform>
+ <transform>
+ <name>no reply</name>
+ <type>DetectEmptyStream</type>
+ <description>Fails when the web service gave no row at all.</description>
+ <distribute>Y</distribute>
+ <custom_distribution/>
+ <copies>1</copies>
+ <partitioning>
+ <method>none</method>
+ <schema_name/>
+ </partitioning>
+ <attributes/>
+ <GUI>
+ <xloc>496</xloc>
+ <yloc>176</yloc>
+ </GUI>
+ </transform>
+ <transform>
+ <name>Abort</name>
+ <type>Abort</type>
+ <description/>
+ <distribute>Y</distribute>
+ <custom_distribution/>
+ <copies>1</copies>
+ <partitioning>
+ <method>none</method>
+ <schema_name/>
+ </partitioning>
+ <abort_option>ABORT_WITH_ERROR</abort_option>
+ <always_log_rows>Y</always_log_rows>
+ <row_threshold>0</row_threshold>
+ <attributes/>
+ <GUI>
+ <xloc>688</xloc>
+ <yloc>64</yloc>
+ </GUI>
+ </transform>
+ <transform_error_handling>
+ </transform_error_handling>
+ <attributes/>
+</pipeline>
diff --git a/integration-tests/http/main-0013-webservice.hwf
b/integration-tests/http/main-0013-webservice.hwf
index b04d753d8d..c888d79506 100644
--- a/integration-tests/http/main-0013-webservice.hwf
+++ b/integration-tests/http/main-0013-webservice.hwf
@@ -110,6 +110,34 @@ limitations under the License.
<yloc>48</yloc>
<attributes_hac/>
</action>
+ <action>
+ <name>0013-webservice-local-wsdl.hpl</name>
+ <description/>
+ <type>PIPELINE</type>
+ <attributes/>
+ <filename>${PROJECT_HOME}/0013-webservice-local-wsdl.hpl</filename>
+ <params_from_previous>N</params_from_previous>
+ <exec_per_row>N</exec_per_row>
+ <clear_rows>N</clear_rows>
+ <clear_files>N</clear_files>
+ <set_logfile>N</set_logfile>
+ <logfile/>
+ <logext/>
+ <add_date>N</add_date>
+ <add_time>N</add_time>
+ <loglevel>Basic</loglevel>
+ <set_append_logfile>N</set_append_logfile>
+ <wait_until_finished>Y</wait_until_finished>
+ <create_parent_folder>N</create_parent_folder>
+ <run_configuration>local</run_configuration>
+ <parameters>
+ <pass_all_parameters>Y</pass_all_parameters>
+ </parameters>
+ <parallel>N</parallel>
+ <xloc>656</xloc>
+ <yloc>160</yloc>
+ <attributes_hac/>
+ </action>
</actions>
<hops>
<hop>
@@ -126,6 +154,13 @@ limitations under the License.
<evaluation>Y</evaluation>
<unconditional>N</unconditional>
</hop>
+ <hop>
+ <from>0013-webservice.hpl</from>
+ <to>0013-webservice-local-wsdl.hpl</to>
+ <enabled>Y</enabled>
+ <evaluation>Y</evaluation>
+ <unconditional>N</unconditional>
+ </hop>
</hops>
<notepads>
</notepads>
diff --git a/integration-tests/http/wsdl/BookService.wsdl
b/integration-tests/http/wsdl/BookService.wsdl
new file mode 100644
index 0000000000..91464ba564
--- /dev/null
+++ b/integration-tests/http/wsdl/BookService.wsdl
@@ -0,0 +1,179 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to You under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+
+-->
+<!--
+ The BookService WSDL that the castlemock container serves (see
+ docker/integration-tests/resource/http/CastleMockProject.xml), kept as a
project file so
+ 0013-webservice-local-wsdl.hpl can load it from ${PROJECT_HOME} (issue
#4813). The service
+ address points to that castlemock mock.
+-->
+<wsdl:definitions xmlns:soap="http://schemas.xmlsoap.org/wsdl/soap/";
xmlns:tns="http://www.cleverbuilder.com/BookService/";
xmlns:wsdl="http://schemas.xmlsoap.org/wsdl/";
xmlns:xsd="http://www.w3.org/2001/XMLSchema"; name="BookService"
targetNamespace="http://www.cleverbuilder.com/BookService/";>
+ <wsdl:documentation>Definition for a web service called BookService,
+ which can be used to add or retrieve books from a collection.
+ </wsdl:documentation>
+ <!--
+ The `types` element defines the data types (XML elements)
+ that are used by the web service.
+ -->
+ <wsdl:types>
+ <xsd:schema
targetNamespace="http://www.cleverbuilder.com/BookService/";>
+ <xsd:element name="Book">
+ <xsd:complexType>
+ <xsd:sequence>
+ <xsd:element minOccurs="0" name="ID"
type="xsd:string"/>
+ <xsd:element name="Title" type="xsd:string"/>
+ <xsd:element name="Author" type="xsd:string"/>
+ </xsd:sequence>
+ </xsd:complexType>
+ </xsd:element>
+ <xsd:element name="Books">
+ <xsd:complexType>
+ <xsd:sequence>
+ <xsd:element maxOccurs="unbounded" minOccurs="0"
ref="tns:Book"/>
+ </xsd:sequence>
+ </xsd:complexType>
+ </xsd:element>
+ <xsd:element name="GetBook">
+ <xsd:complexType>
+ <xsd:sequence>
+ <xsd:element name="ID" type="xsd:string"/>
+ </xsd:sequence>
+ </xsd:complexType>
+ </xsd:element>
+ <xsd:element name="GetBookResponse">
+ <xsd:complexType>
+ <xsd:sequence>
+ <xsd:element maxOccurs="1" minOccurs="0"
ref="tns:Book"/>
+ </xsd:sequence>
+ </xsd:complexType>
+ </xsd:element>
+ <xsd:element name="AddBook">
+ <xsd:complexType>
+ <xsd:sequence>
+ <xsd:element maxOccurs="1" minOccurs="1"
ref="tns:Book"/>
+ </xsd:sequence>
+ </xsd:complexType>
+ </xsd:element>
+ <xsd:element name="AddBookResponse">
+ <xsd:complexType>
+ <xsd:sequence>
+ <xsd:element maxOccurs="1" minOccurs="0"
ref="tns:Book"/>
+ </xsd:sequence>
+ </xsd:complexType>
+ </xsd:element>
+ <xsd:element name="GetAllBooks">
+ <xsd:complexType/>
+ </xsd:element>
+ <xsd:element name="GetAllBooksResponse">
+ <xsd:complexType>
+ <xsd:sequence>
+ <xsd:element maxOccurs="unbounded" minOccurs="0"
ref="tns:Book"/>
+ </xsd:sequence>
+ </xsd:complexType>
+ </xsd:element>
+ </xsd:schema>
+ </wsdl:types>
+ <!--
+ A wsdl `message` element is used to define a message
+ exchanged between a web service, consisting of zero
+ or more `part`s.
+ -->
+ <wsdl:message name="GetBookRequest">
+ <wsdl:part element="tns:GetBook" name="parameters"/>
+ </wsdl:message>
+ <wsdl:message name="GetBookResponse">
+ <wsdl:part element="tns:GetBookResponse" name="parameters"/>
+ </wsdl:message>
+ <wsdl:message name="AddBookRequest">
+ <wsdl:part element="tns:AddBook" name="parameters"/>
+ </wsdl:message>
+ <wsdl:message name="AddBookResponse">
+ <wsdl:part element="tns:AddBookResponse" name="parameters"/>
+ </wsdl:message>
+ <wsdl:message name="GetAllBooksRequest">
+ <wsdl:part element="tns:GetAllBooks" name="parameters"/>
+ </wsdl:message>
+ <wsdl:message name="GetAllBooksResponse">
+ <wsdl:part element="tns:GetAllBooksResponse" name="parameters"/>
+ </wsdl:message>
+ <!--
+ A WSDL `portType` is used to combine multiple `message`s
+ (e.g. input, output) into a single operation.
+ Here we define three synchronous (input/output) operations
+ and the `message`s that must be used for each.
+ -->
+ <wsdl:portType name="BookService">
+ <wsdl:operation name="GetBook">
+ <wsdl:input message="tns:GetBookRequest"/>
+ <wsdl:output message="tns:GetBookResponse"/>
+ </wsdl:operation>
+ <wsdl:operation name="AddBook">
+ <wsdl:input message="tns:AddBookRequest"/>
+ <wsdl:output message="tns:AddBookResponse"/>
+ </wsdl:operation>
+ <wsdl:operation name="GetAllBooks">
+ <wsdl:input message="tns:GetAllBooksRequest"/>
+ <wsdl:output message="tns:GetAllBooksResponse"/>
+ </wsdl:operation>
+ </wsdl:portType>
+ <!--
+ The `binding` element defines exactly how each
+ `operation` will take place over the network.
+ In this case, we are using SOAP.
+ -->
+ <wsdl:binding name="BookServiceSOAP" type="tns:BookService">
+ <soap:binding style="document"
transport="http://schemas.xmlsoap.org/soap/http"/>
+ <wsdl:operation name="GetBook">
+ <soap:operation
soapAction="http://www.cleverbuilder.com/BookService/GetBook"/>
+ <wsdl:input>
+ <soap:body use="literal"/>
+ </wsdl:input>
+ <wsdl:output>
+ <soap:body use="literal"/>
+ </wsdl:output>
+ </wsdl:operation>
+ <wsdl:operation name="AddBook">
+ <soap:operation
soapAction="http://www.cleverbuilder.com/BookService/AddBook"/>
+ <wsdl:input>
+ <soap:body use="literal"/>
+ </wsdl:input>
+ <wsdl:output>
+ <soap:body use="literal"/>
+ </wsdl:output>
+ </wsdl:operation>
+ <wsdl:operation name="GetAllBooks">
+ <soap:operation
soapAction="http://www.cleverbuilder.com/BookService/GetAllBooks"/>
+ <wsdl:input>
+ <soap:body use="literal"/>
+ </wsdl:input>
+ <wsdl:output>
+ <soap:body use="literal"/>
+ </wsdl:output>
+ </wsdl:operation>
+ </wsdl:binding>
+ <!--
+ The `service` element finally says where the service
+ can be accessed from - in other words, its endpoint.
+ -->
+ <wsdl:service name="BookService">
+ <wsdl:port binding="tns:BookServiceSOAP" name="BookServiceSOAP">
+ <soap:address
location="http://castlemock:8080/castlemock/mock/soap/project/17hLFE/BookServiceSOAP"/>
+ </wsdl:port>
+ </wsdl:service>
+</wsdl:definitions>
diff --git
a/plugins/transforms/webservices/src/main/java/org/apache/hop/pipeline/transforms/webservices/WebService.java
b/plugins/transforms/webservices/src/main/java/org/apache/hop/pipeline/transforms/webservices/WebService.java
index 3ce7a7ec68..5789dd09c6 100644
---
a/plugins/transforms/webservices/src/main/java/org/apache/hop/pipeline/transforms/webservices/WebService.java
+++
b/plugins/transforms/webservices/src/main/java/org/apache/hop/pipeline/transforms/webservices/WebService.java
@@ -497,7 +497,8 @@ public class WebService extends
BaseTransform<WebServiceMeta, WebServiceData> {
try {
cachedWsdl =
new Wsdl(
- new java.net.URI(data.realUrl),
+ data.realUrl,
+ this,
null,
null,
resolve(meta.getHttpLogin()),
@@ -624,7 +625,7 @@ public class WebService extends
BaseTransform<WebServiceMeta, WebServiceData> {
}
}
- private void processRows(
+ void processRows(
InputStream anXml,
Object[] rowData,
IRowMeta rowMeta,
@@ -709,15 +710,9 @@ public class WebService extends
BaseTransform<WebServiceMeta, WebServiceData> {
if (meta.isReturningReplyAsString()) {
- // Just return the body node as an XML string...
+ // Just return the complete reply as an XML string...
//
- StringWriter nodeXML = new StringWriter();
- transformer.transform(new DOMSource(bodyNode), new
StreamResult(nodeXML));
- String xml = response;
- Object[] outputRowData = createNewRow(rowData);
- int index = rowData == null ? 0 : getInputRowMeta().size();
- outputRowData[index++] = xml;
- putRow(data.outputRowMeta, outputRowData);
+ putWholeReplyRow(rowData, response);
} else {
@@ -743,12 +738,7 @@ public class WebService extends
BaseTransform<WebServiceMeta, WebServiceData> {
//
StringWriter nodeXML = new StringWriter();
transformer.transform(new DOMSource(responseNode), new
StreamResult(nodeXML));
- String xml = nodeXML.toString();
-
- Object[] outputRowData = createNewRow(rowData);
- int index = rowData == null ? 0 : getInputRowMeta().size();
- outputRowData[index++] = xml;
- putRow(data.outputRowMeta, outputRowData);
+ putWholeReplyRow(rowData, nodeXML.toString());
} else {
if (responseNode != null) {
@@ -781,83 +771,68 @@ public class WebService extends
BaseTransform<WebServiceMeta, WebServiceData> {
for (int i = 0; i < nodeList.getLength(); i++) {
Node node = nodeList.item(i);
- if (meta.isReturningReplyAsString()) {
-
- // Just return the body node as an XML string...
- //
- StringWriter nodeXML = new StringWriter();
- transformer.transform(new DOMSource(bodyNode), new
StreamResult(nodeXML));
- String xml = nodeXML.toString();
- outputRowData = createNewRow(rowData);
- int index = rowData == null ? 0 : getInputRowMeta().size();
- outputRowData[index++] = xml;
- putRow(data.outputRowMeta, outputRowData);
-
+ // This node either contains the data for a single row or it contains
the first element of
+ // a single result
+ // response
+ // If we find the node name in out output result fields list, we are
going to consider it
+ // a single row result.
+ //
+ WebServiceField field =
+ meta.getFieldOutFromWsName(node.getNodeName(),
ignoreNamespacePrefix);
+ if (field != null) {
+ if (getNodeValue(outputRowData, node, field, transformer, true)) {
+ // We found a match.
+ // This means that we are dealing with a single row
+ // It also means that we need to update the output index pointer
+ //
+ singleRow = true;
+ fieldsFound++;
+ }
} else {
-
- // This node either contains the data for a single row or it
contains the first element of
- // a single result
- // response
- // If we find the node name in out output result fields list, we are
going to consider it
- // a single row result.
+ // If we didn't already get data in the previous block we'll assume
multiple rows coming
+ // back.
//
- WebServiceField field =
- meta.getFieldOutFromWsName(node.getNodeName(),
ignoreNamespacePrefix);
- if (field != null) {
- if (getNodeValue(outputRowData, node, field, transformer, true)) {
- // We found a match.
- // This means that we are dealing with a single row
- // It also means that we need to update the output index pointer
- //
- singleRow = true;
- fieldsFound++;
- }
- } else {
- // If we didn't already get data in the previous block we'll
assume multiple rows coming
- // back.
+ if (!singleRow) {
+ // Sticking with the multiple-results scenario...
//
- if (!singleRow) {
- // Sticking with the multiple-results scenario...
- //
- // TODO: remove next 2 lines, added for debug reasons.
- //
- if (isDetailed()) {
- StringWriter nodeXML = new StringWriter();
- transformer.transform(new DOMSource(node), new
StreamResult(nodeXML));
- logDetailed(
- BaseMessages.getString(
- PKG, "WebServices.Log.ResultRowDataFound",
nodeXML.toString()));
- }
+ // TODO: remove next 2 lines, added for debug reasons.
+ //
+ if (isDetailed()) {
+ StringWriter nodeXML = new StringWriter();
+ transformer.transform(new DOMSource(node), new
StreamResult(nodeXML));
+ logDetailed(
+ BaseMessages.getString(
+ PKG, "WebServices.Log.ResultRowDataFound",
nodeXML.toString()));
+ }
- // Allocate a new row...
- //
- outputRowData = createNewRow(rowData);
+ // Allocate a new row...
+ //
+ outputRowData = createNewRow(rowData);
- // Let's see what's in there...
- //
- NodeList childNodes = node.getChildNodes();
- for (int j = 0; j < childNodes.getLength(); j++) {
- Node childNode = childNodes.item(j);
-
- field = meta.getFieldOutFromWsName(childNode.getNodeName(),
ignoreNamespacePrefix);
- if (field != null
- && getNodeValue(outputRowData, childNode, field,
transformer, false)) {
- // We found a match.
- // This means that we are dealing with a single row
- // It also means that we need to update the output index
pointer
- //
- fieldsFound++;
- }
+ // Let's see what's in there...
+ //
+ NodeList childNodes = node.getChildNodes();
+ for (int j = 0; j < childNodes.getLength(); j++) {
+ Node childNode = childNodes.item(j);
+
+ field = meta.getFieldOutFromWsName(childNode.getNodeName(),
ignoreNamespacePrefix);
+ if (field != null
+ && getNodeValue(outputRowData, childNode, field,
transformer, false)) {
+ // We found a match.
+ // This means that we are dealing with a single row
+ // It also means that we need to update the output index
pointer
+ //
+ fieldsFound++;
}
+ }
- // Prevent empty rows from being sent out.
+ // Prevent empty rows from being sent out.
+ //
+ if (fieldsFound > 0) {
+ // Send a row in a series of rows on its way.
//
- if (fieldsFound > 0) {
- // Send a row in a series of rows on its way.
- //
- putRow(data.outputRowMeta, outputRowData);
- }
+ putRow(data.outputRowMeta, outputRowData);
}
}
}
@@ -874,12 +849,29 @@ public class WebService extends
BaseTransform<WebServiceMeta, WebServiceData> {
}
}
+ /**
+ * Allocate an output row. It starts with the input fields only when we pass
those along:
+ * otherwise the input values would end up in the output fields the reply
doesn't fill.
+ */
private Object[] createNewRow(Object[] inputRowData) {
- return inputRowData == null
+ return inputRowData == null || !meta.isPassingInputData()
? RowDataUtil.allocateRowData(data.outputRowMeta.size())
: RowDataUtil.createResizedCopy(inputRowData,
data.outputRowMeta.size());
}
+ /**
+ * Send a row with a complete piece of the reply, as XML, in the first
output field. The output
+ * fields come after the input fields when those are passed along, and first
otherwise.
+ */
+ private void putWholeReplyRow(Object[] rowData, String xml) throws
HopTransformException {
+ Object[] outputRowData = createNewRow(rowData);
+ int index = data.outputRowMeta.size() - meta.getFieldsOut().size();
+ if (index < data.outputRowMeta.size()) {
+ outputRowData[index] = xml;
+ }
+ putRow(data.outputRowMeta, outputRowData);
+ }
+
private void compatibleProcessRows(
InputStream anXml,
Object[] rowData,
diff --git
a/plugins/transforms/webservices/src/main/java/org/apache/hop/pipeline/transforms/webservices/WebServiceDialog.java
b/plugins/transforms/webservices/src/main/java/org/apache/hop/pipeline/transforms/webservices/WebServiceDialog.java
index a299f360d1..0a17d8962a 100644
---
a/plugins/transforms/webservices/src/main/java/org/apache/hop/pipeline/transforms/webservices/WebServiceDialog.java
+++
b/plugins/transforms/webservices/src/main/java/org/apache/hop/pipeline/transforms/webservices/WebServiceDialog.java
@@ -17,8 +17,6 @@
package org.apache.hop.pipeline.transforms.webservices;
-import java.io.File;
-import java.net.URI;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.Collections;
@@ -28,6 +26,7 @@ import java.util.Properties;
import javax.xml.namespace.QName;
import org.apache.hop.core.Const;
import org.apache.hop.core.Props;
+import org.apache.hop.core.encryption.Encr;
import org.apache.hop.core.exception.HopException;
import org.apache.hop.core.exception.HopPluginException;
import org.apache.hop.core.exception.HopTransformException;
@@ -156,7 +155,14 @@ public class WebServiceDialog extends BaseTransformDialog {
systemProperties.setProperty("http.proxyHost",
variables.resolve(wProxyHost.getText()));
systemProperties.setProperty("http.proxyPort",
variables.resolve(wProxyPort.getText()));
}
- wsdl = new Wsdl(new URI(anURI), null, null, wHttpLogin.getText(),
wHttpPassword.getText());
+ wsdl =
+ new Wsdl(
+ anURI,
+ variables,
+ null,
+ null,
+ variables.resolve(wHttpLogin.getText()),
+
Encr.decryptPasswordOptionallyEncrypted(variables.resolve(wHttpPassword.getText())));
} catch (Exception e) {
wsdl = null;
new ErrorDialog(
@@ -494,7 +500,7 @@ public class WebServiceDialog extends BaseTransformDialog {
if (tabItemFieldOut == null) {
tabItemFieldOut = new CTabItem(wTabFolder, SWT.NONE);
- tabItemFieldIn.setFont(GuiResource.getInstance().getFontDefault());
+ tabItemFieldOut.setFont(GuiResource.getInstance().getFontDefault());
}
ColumnInfo[] colinf =
new ColumnInfo[] {
@@ -883,6 +889,7 @@ public class WebServiceDialog extends BaseTransformDialog {
Button wbFile = new Button(compositeTabWebService, SWT.PUSH | SWT.CENTER);
PropsUi.setLook(wbFile);
wbFile.setText(BaseMessages.getString(PKG, "WebServiceDialog.File.Load"));
+ wbFile.setToolTipText(BaseMessages.getString(PKG,
"WebServiceDialog.File.Tooltip"));
FormData fdbFile = new FormData();
fdbFile.right = new FormAttachment(wbURL, 0);
fdbFile.top = new FormAttachment(0, 0);
@@ -891,21 +898,27 @@ public class WebServiceDialog extends BaseTransformDialog
{
wbFile.addListener(
SWT.Selection,
e -> {
+ // The file dialog can hand back a name relative to a variable like
${PROJECT_HOME}.
+ // Keep it that way in the URL field so the transform stays
portable: loading resolves it.
+ //
String filename =
BaseDialog.presentFileDialog(
+ false,
shell,
+ null,
+ variables,
+ null,
new String[] {"*.wsdl;*.WSDL", "*.*"},
new String[] {
BaseMessages.getString(PKG,
"WebServiceDialog.FileType.WsdlFiles"),
- BaseMessages.getString(PKG, "System.FileType.CSVFiles"),
- BaseMessages.getString(PKG, "System.FileType.TextFiles"),
BaseMessages.getString(PKG, "System.FileType.AllFiles")
},
- true);
+ false);
if (filename != null) {
+ wURL.setText(filename);
try {
- initTreeTabWebService(new
File(filename).toURI().toASCIIString());
+ initTreeTabWebService(filename);
} catch (Throwable throwable) {
new ErrorDialog(
shell,
@@ -919,6 +932,7 @@ public class WebServiceDialog extends BaseTransformDialog {
});
wURL = new TextVar(variables, compositeTabWebService, SWT.SINGLE |
SWT.LEFT | SWT.BORDER);
+ wURL.setToolTipText(BaseMessages.getString(PKG,
"WebServiceDialog.URL.Tooltip"));
wURL.addModifyListener(lsMod);
PropsUi.setLook(wURL);
FormData fdURL = new FormData();
diff --git
a/plugins/transforms/webservices/src/main/java/org/apache/hop/pipeline/transforms/webservices/wsdl/HopVfsWsdlLocator.java
b/plugins/transforms/webservices/src/main/java/org/apache/hop/pipeline/transforms/webservices/wsdl/HopVfsWsdlLocator.java
new file mode 100644
index 0000000000..c0acf737b8
--- /dev/null
+++
b/plugins/transforms/webservices/src/main/java/org/apache/hop/pipeline/transforms/webservices/wsdl/HopVfsWsdlLocator.java
@@ -0,0 +1,227 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hop.pipeline.transforms.webservices.wsdl;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.net.URI;
+import java.net.URISyntaxException;
+import java.net.URL;
+import java.net.URLConnection;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Base64;
+import java.util.List;
+import java.util.Objects;
+import javax.wsdl.xml.WSDLLocator;
+import org.apache.commons.lang3.Strings;
+import org.apache.commons.vfs2.FileObject;
+import org.apache.hop.core.exception.HopException;
+import org.apache.hop.core.exception.HopRuntimeException;
+import org.apache.hop.core.variables.IVariables;
+import org.apache.hop.core.vfs.HopVfs;
+import org.xml.sax.InputSource;
+
+/**
+ * Finds a WSDL and everything it imports, the WSDL imports and the XML
schemas it includes, through
+ * Hop VFS or over http(s).
+ *
+ * <p>Without a locator WSDL4J opens every import with {@link java.net.URL},
which knows file: and
+ * http(s) but none of the other file systems Hop reads from: a WSDL on S3,
Azure or HDFS loaded
+ * fine and then failed on its first relative import.
+ *
+ * <p>A relative location is resolved against the document that names it, so
an import can import in
+ * turn. The user name and password go along with http(s) requests to the host
the WSDL itself came
+ * from, and to no other.
+ *
+ * <p>Everything this opens stays open until {@link #close()}, which the
caller must call once the
+ * WSDL has been read: WSDL4J reads an import after asking for it, not while.
+ */
+final class HopVfsWsdlLocator implements WSDLLocator, AutoCloseable {
+
+ private final String baseUri;
+ private final IVariables variables;
+ private final String username;
+ private final String password;
+ private final List<AutoCloseable> opened = new ArrayList<>();
+ private String latestImportUri;
+
+ /**
+ * @param baseUri the absolute location of the WSDL: an http(s) URL or a Hop
VFS URI
+ * @param variables to find the named VFS connections with, can be null
+ * @param username for HTTP authentication with the WSDL's own host, can be
null
+ * @param password for HTTP authentication with the WSDL's own host, can be
null
+ */
+ HopVfsWsdlLocator(String baseUri, IVariables variables, String username,
String password) {
+ this.baseUri = baseUri;
+ this.variables = variables;
+ this.username = username;
+ this.password = password;
+ }
+
+ @Override
+ public InputSource getBaseInputSource() {
+ try {
+ return inputSource(baseUri);
+ } catch (HopException e) {
+ throw new HopRuntimeException(e.getMessage(), e);
+ }
+ }
+
+ /**
+ * The stream of the WSDL itself, closed by {@link #close()} at the latest.
+ *
+ * @throws HopException when the WSDL does not exist or cannot be read
+ */
+ InputStream openBase() throws HopException {
+ return open(baseUri);
+ }
+
+ @Override
+ public InputSource getImportInputSource(String parentLocation, String
importLocation) {
+ String location = resolve(parentLocation == null ? baseUri :
parentLocation, importLocation);
+ latestImportUri = location;
+ try {
+ return inputSource(location);
+ } catch (HopException e) {
+ throw new HopRuntimeException(
+ "Unable to read " + importLocation + ", imported by " +
parentLocation, e);
+ }
+ }
+
+ @Override
+ public String getBaseURI() {
+ return baseUri;
+ }
+
+ @Override
+ public String getLatestImportURI() {
+ return latestImportUri;
+ }
+
+ @Override
+ public void close() {
+ for (AutoCloseable closeable : opened) {
+ try {
+ closeable.close();
+ } catch (Exception e) {
+ // Nothing left to read from it; closing is best effort.
+ }
+ }
+ opened.clear();
+ }
+
+ private InputSource inputSource(String location) throws HopException {
+ InputSource source = new InputSource(open(location));
+ source.setSystemId(location);
+ return source;
+ }
+
+ private InputStream open(String location) throws HopException {
+ if (Wsdl.isHttpLocation(location)) {
+ return remember(openHttp(location));
+ }
+ FileObject file = remember(fileObject(location));
+ try {
+ if (!file.exists()) {
+ throw new HopException("WSDL file " + location + " does not exist");
+ }
+ return remember(HopVfs.getInputStream(file));
+ } catch (IOException e) {
+ throw new HopException("Unable to read WSDL file " + location, e);
+ }
+ }
+
+ private InputStream openHttp(String url) throws HopException {
+ try {
+ URLConnection connection = new URL(url).openConnection();
+ if (username != null && !username.isEmpty() && sameOrigin(url, baseUri))
{
+ String raw = username + ":" + (password == null ? "" : password);
+ String encoded =
Base64.getEncoder().encodeToString(raw.getBytes(StandardCharsets.UTF_8));
+ connection.setRequestProperty("Authorization", "Basic " + encoded);
+ }
+ return connection.getInputStream();
+ } catch (IOException e) {
+ throw new HopException("Unable to read WSDL from " + url, e);
+ }
+ }
+
+ /**
+ * An absolute location as it stands; a relative one next to the document
that names it: over
+ * http(s) by URL resolution, anywhere else through Hop VFS, which knows how
the file system of
+ * the parent spells a path.
+ */
+ private String resolve(String parentLocation, String location) {
+ if (isAbsolute(location)) {
+ return location;
+ }
+ try {
+ if (Wsdl.isHttpLocation(parentLocation)) {
+ return URI.create(parentLocation).resolve(location).toString();
+ }
+ try (FileObject parent = fileObject(parentLocation);
+ FileObject folder = parent.getParent();
+ FileObject resolved = folder.resolveFile(location)) {
+ return resolved.getName().getURI();
+ }
+ } catch (Exception e) {
+ throw new HopRuntimeException(
+ "Unable to resolve " + location + " relative to " + parentLocation,
e);
+ }
+ }
+
+ /** True for a location with a scheme of its own: http:, file:, s3:, a named
VFS connection. */
+ private static boolean isAbsolute(String location) {
+ try {
+ // A single letter is a Windows drive, not a scheme.
+ String scheme = new URI(location).getScheme();
+ return scheme != null && scheme.length() > 1;
+ } catch (URISyntaxException e) {
+ return false;
+ }
+ }
+
+ private static boolean sameOrigin(String url, String other) {
+ if (!Wsdl.isHttpLocation(other)) {
+ return false;
+ }
+ URI a = URI.create(url);
+ URI b = URI.create(other);
+ return Strings.CI.equals(a.getScheme(), b.getScheme())
+ && Strings.CI.equals(a.getHost(), b.getHost())
+ && port(a) == port(b);
+ }
+
+ private static int port(URI uri) {
+ if (uri.getPort() != -1) {
+ return uri.getPort();
+ }
+ return "https".equalsIgnoreCase(uri.getScheme()) ? 443 : 80;
+ }
+
+ private FileObject fileObject(String location) throws HopException {
+ return variables == null
+ ? HopVfs.getFileObject(location)
+ : HopVfs.getFileObject(location, variables);
+ }
+
+ private <T extends AutoCloseable> T remember(T closeable) {
+ opened.add(Objects.requireNonNull(closeable));
+ return closeable;
+ }
+}
diff --git
a/plugins/transforms/webservices/src/main/java/org/apache/hop/pipeline/transforms/webservices/wsdl/Wsdl.java
b/plugins/transforms/webservices/src/main/java/org/apache/hop/pipeline/transforms/webservices/wsdl/Wsdl.java
index 7b789d52ba..090d86de28 100644
---
a/plugins/transforms/webservices/src/main/java/org/apache/hop/pipeline/transforms/webservices/wsdl/Wsdl.java
+++
b/plugins/transforms/webservices/src/main/java/org/apache/hop/pipeline/transforms/webservices/wsdl/Wsdl.java
@@ -20,13 +20,8 @@ package org.apache.hop.pipeline.transforms.webservices.wsdl;
import java.io.IOException;
import java.io.InputStream;
import java.io.Serial;
-import java.net.MalformedURLException;
import java.net.URI;
-import java.net.URL;
-import java.net.URLConnection;
-import java.nio.charset.StandardCharsets;
import java.util.ArrayList;
-import java.util.Base64;
import java.util.Collection;
import java.util.HashMap;
import java.util.Iterator;
@@ -43,10 +38,17 @@ import javax.wsdl.factory.WSDLFactory;
import javax.wsdl.xml.WSDLLocator;
import javax.wsdl.xml.WSDLReader;
import javax.xml.namespace.QName;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.commons.lang3.Strings;
+import org.apache.commons.vfs2.FileObject;
+import org.apache.commons.vfs2.FileSystemException;
+import org.apache.hop.core.Const;
import org.apache.hop.core.exception.HopException;
import org.apache.hop.core.exception.HopRuntimeException;
import org.apache.hop.core.exception.HopTransformException;
import org.apache.hop.core.logging.LogChannel;
+import org.apache.hop.core.variables.IVariables;
+import org.apache.hop.core.vfs.HopVfs;
import org.apache.hop.core.xml.XmlHandler;
import org.w3c.dom.Document;
@@ -59,7 +61,6 @@ public final class Wsdl implements java.io.Serializable {
private final Service service;
private final WsdlTypes wsdlTypes;
private HashMap<String, WsdlOperation> operationHashMap;
- private URI wsdlURI = null;
/**
* Loads and parses the specified WSDL file.
@@ -73,12 +74,38 @@ public final class Wsdl implements java.io.Serializable {
}
public Wsdl(URI wsdlURI, QName serviceQName, String portName, String
username, String password) {
+ this(wsdlURI.toString(), null, serviceQName, portName, username, password);
+ }
+
+ /**
+ * Loads and parses the WSDL file at the specified location.
+ *
+ * @param wsdlLocation An http(s) URL, or any file name Hop VFS understands:
a plain path, a file:
+ * URL or a VFS location like s3://.
+ * @param variables The variables of the caller, used to find the named VFS
connections. Can be
+ * null.
+ * @param serviceQName Name of the service in the WSDL, if null default to
first service in WSDL.
+ * @param portName The service port name, if null default to first port in
service.
+ * @param username to use for HTTP authentication
+ * @param password to use for HTTP authentication
+ */
+ public Wsdl(
+ String wsdlLocation,
+ IVariables variables,
+ QName serviceQName,
+ String portName,
+ String username,
+ String password) {
- this.wsdlURI = wsdlURI;
try {
- wsdlDefinition = parse(wsdlURI, username, password);
+ wsdlDefinition = parse(wsdlLocation, variables, username, password);
} catch (WSDLException | HopException e) {
- throw new HopRuntimeException(CONST_COULD_NOT_LOAD_WSDL_FILE +
e.getMessage(), e);
+ throw new HopRuntimeException(
+ CONST_COULD_NOT_LOAD_WSDL_FILE
+ + wsdlLocation
+ + " : "
+ + Const.NVL(e.getMessage(), "").trim(),
+ e);
}
if (serviceQName == null) {
service = (Service)
wsdlDefinition.getServices().values().iterator().next();
@@ -86,7 +113,7 @@ public final class Wsdl implements java.io.Serializable {
service = wsdlDefinition.getService(serviceQName);
if (service == null) {
throw new IllegalArgumentException(
- "Service: " + serviceQName + " is not defined in the WSDL file " +
wsdlURI);
+ "Service: " + serviceQName + " is not defined in the WSDL file " +
wsdlLocation);
}
}
@@ -145,8 +172,8 @@ public final class Wsdl implements java.io.Serializable {
// load and parse the WSDL
try {
- wsdlDefinition = parse(wsdlLocator, username, password);
- } catch (WSDLException | HopException e) {
+ wsdlDefinition = parse(wsdlLocator);
+ } catch (WSDLException e) {
throw new HopRuntimeException(CONST_COULD_NOT_LOAD_WSDL_FILE +
e.getMessage(), e);
}
@@ -315,65 +342,75 @@ public final class Wsdl implements java.io.Serializable {
* Load and parse the WSDL file using the wsdlLocator.
*
* @param wsdlLocator A WSDLLocator instance.
- * @param username to use for authentication
- * @param password to use for authentication
* @return wsdl Definition.
* @throws WSDLException on error.
*/
- private Definition parse(WSDLLocator wsdlLocator, String username, String
password)
- throws WSDLException, HopException {
-
- WSDLReader wsdlReader = getReader();
- try {
-
- return wsdlReader.readWSDL(wsdlLocator);
- } catch (WSDLException we) {
- readWsdl(wsdlReader, wsdlURI.toString(), username, password);
- return null;
- }
+ private Definition parse(WSDLLocator wsdlLocator) throws WSDLException {
+ return getReader().readWSDL(wsdlLocator);
}
/**
- * Load and parse the WSDL file at the specified URI.
+ * Load and parse the WSDL file at the specified location.
*
- * @param wsdlURI URI of the WSDL file.
- * @param username to use for authentication
- * @param password to use for authentication
+ * @param wsdlLocation http(s) URL or Hop VFS file name of the WSDL file.
+ * @param variables to find the named VFS connections with, can be null
+ * @param username to use for HTTP authentication
+ * @param password to use for HTTP authentication
* @return wsdl Definition
* @throws WSDLException on error.
*/
- private Definition parse(URI wsdlURI, String username, String password)
+ private Definition parse(
+ String wsdlLocation, IVariables variables, String username, String
password)
throws WSDLException, HopException {
+ if (StringUtils.isBlank(wsdlLocation)) {
+ throw new HopException("No WSDL location was specified");
+ }
+ String location = wsdlLocation.trim();
WSDLReader wsdlReader = getReader();
- return readWsdl(wsdlReader, wsdlURI.toString(), username, password);
- }
-
- private Definition readWsdl(WSDLReader wsdlReader, String uri, String
username, String password)
- throws WSDLException, HopException {
- try (InputStream wsdlStream = openWsdlStream(uri, username, password)) {
- Document doc = XmlHandler.loadXmlFile(wsdlStream, uri, false, true);
- if (doc != null) {
- return wsdlReader.readWSDL(uri, doc);
- } else {
+ // Imports are resolved relative to the WSDL's own URI and read through
the same locator, so
+ // they come from wherever the WSDL came from: over http(s), or from any
file system Hop VFS
+ // knows about. Everything it opens is closed once the WSDL has been read,
failure or not.
+ //
+ try (HopVfsWsdlLocator locator =
+ new HopVfsWsdlLocator(baseUri(location, variables), variables,
username, password)) {
+ Document doc;
+ try (InputStream wsdlStream = locator.openBase()) {
+ doc = XmlHandler.loadXmlFile(wsdlStream, locator.getBaseURI(), false,
true);
+ } catch (IOException e) {
+ throw new HopException(e);
+ }
+ if (doc == null) {
throw new HopException("Unable to get document.");
}
- } catch (MalformedURLException mue) {
- throw new HopException(mue);
- } catch (IOException ioe) {
- throw new HopException(ioe);
+ return wsdlReader.readWSDL(locator, doc.getDocumentElement());
+ } catch (HopRuntimeException e) {
+ // An import the locator could not read. Unwrapped, so the message says
which.
+ throw new HopException(e.getMessage(), e.getCause() == null ? e :
e.getCause());
}
}
- private InputStream openWsdlStream(String uri, String username, String
password)
- throws IOException {
- URLConnection connection = new URL(uri).openConnection();
- if (username != null && !username.isEmpty()) {
- String raw = username + ":" + (password == null ? "" : password);
- String encoded =
Base64.getEncoder().encodeToString(raw.getBytes(StandardCharsets.UTF_8));
- connection.setRequestProperty("Authorization", "Basic " + encoded);
+ /**
+ * The absolute URI of the WSDL: an http(s) URL as it stands, anything else,
a plain path like the
+ * file dialog hands out, a file: URL or any other location Hop VFS knows
about, as Hop VFS names
+ * it.
+ */
+ private static String baseUri(String location, IVariables variables) throws
HopException {
+ if (isHttpLocation(location)) {
+ return location;
}
- return connection.getInputStream();
+ try (FileObject wsdlFile =
+ variables == null
+ ? HopVfs.getFileObject(location)
+ : HopVfs.getFileObject(location, variables)) {
+ return wsdlFile.getName().getURI();
+ } catch (FileSystemException e) {
+ throw new HopException("Unable to read WSDL file " + location, e);
+ }
+ }
+
+ static boolean isHttpLocation(String wsdlLocation) {
+ return Strings.CI.startsWithAny(wsdlLocation, "http://";, "https://";);
}
/**
diff --git
a/plugins/transforms/webservices/src/main/resources/org/apache/hop/pipeline/transforms/webservices/messages/messages_en_US.properties
b/plugins/transforms/webservices/src/main/resources/org/apache/hop/pipeline/transforms/webservices/messages/messages_en_US.properties
index 481130fba2..2ab07c0ae1 100644
---
a/plugins/transforms/webservices/src/main/resources/org/apache/hop/pipeline/transforms/webservices/messages/messages_en_US.properties
+++
b/plugins/transforms/webservices/src/main/resources/org/apache/hop/pipeline/transforms/webservices/messages/messages_en_US.properties
@@ -29,7 +29,8 @@ WebServiceDialog.ERROR0009.UnreachableURI=Unreachable URI
WebServiceDialog.ErrorDialog.Title=Unable to load
WebServiceDialog.Exception.UnableToLoadWebService.Message=Unexpected error
while trying to load web service details
WebServiceDialog.Exception.UnableToLoadWebService.Title=Error
-WebServiceDialog.File.Load=Test file
+WebServiceDialog.File.Load=Browse...
+WebServiceDialog.File.Tooltip=Select a WSDL file, put its location in the URL
field and load it.
WebServiceDialog.FileType.WsdlFiles=WSDL files
WebServiceDialog.HttpLogin.Label=HTTP Login
WebServiceDialog.HttpLogin.Tooltip=HTTP login used to authenticate the call
@@ -57,6 +58,7 @@ WebServiceDialog.Transform.Label=The number of rows per call
WebServiceDialog.Transform.Tooltip=Number of rows to send in one web service
call
WebServiceDialog.TypeColumn.Column=WS Type
WebServiceDialog.URL.Label=URL
+WebServiceDialog.URL.Tooltip=Location of the WSDL document: an http(s) URL, or
a file like '${PROJECT_HOME}/wsdl/service.wsdl'.\nFiles can be on any file
system Hop can read (local, S3, Azure, ...).
WebServiceDialog.URL.Load=Load
WebServiceDialog.WsNameColumn.Column=WS Name
WebServiceLookup.Description=Look up information using web services (WSDL)
diff --git
a/plugins/transforms/webservices/src/test/java/org/apache/hop/pipeline/transforms/webservices/WebServiceTest.java
b/plugins/transforms/webservices/src/test/java/org/apache/hop/pipeline/transforms/webservices/WebServiceTest.java
index 61a58051bf..ec6e6c9cba 100644
---
a/plugins/transforms/webservices/src/test/java/org/apache/hop/pipeline/transforms/webservices/WebServiceTest.java
+++
b/plugins/transforms/webservices/src/test/java/org/apache/hop/pipeline/transforms/webservices/WebServiceTest.java
@@ -18,20 +18,34 @@
package org.apache.hop.pipeline.transforms.webservices;
import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertNull;
import static org.junit.jupiter.api.Assertions.assertThrows;
+import static org.junit.jupiter.api.Assertions.assertTrue;
import static org.mockito.ArgumentMatchers.any;
+import static org.mockito.Mockito.doAnswer;
import static org.mockito.Mockito.doReturn;
import static org.mockito.Mockito.mock;
import static org.mockito.Mockito.spy;
import static org.mockito.Mockito.when;
+import java.io.ByteArrayInputStream;
import java.net.URISyntaxException;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.List;
import org.apache.hc.client5.http.classic.methods.HttpPost;
import org.apache.hc.core5.http.Header;
import org.apache.hc.core5.http.message.BasicHeader;
+import org.apache.hop.core.HopClientEnvironment;
import org.apache.hop.core.logging.ILoggingObject;
+import org.apache.hop.core.row.IRowMeta;
+import org.apache.hop.core.row.RowDataUtil;
+import org.apache.hop.core.row.RowMeta;
+import org.apache.hop.core.row.value.ValueMetaString;
+import org.apache.hop.core.variables.Variables;
import org.apache.hop.pipeline.transforms.mock.TransformMockHelper;
import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeAll;
import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Test;
@@ -47,6 +61,11 @@ class WebServiceTest {
private WebService webServiceTransform;
+ @BeforeAll
+ static void initEnvironment() throws Exception {
+ HopClientEnvironment.init();
+ }
+
@BeforeEach
void setUpBefore() {
mockHelper =
@@ -84,4 +103,148 @@ class WebServiceTest {
assertEquals(TEST_URL, WebService.getLocationFrom(postMethod));
}
+
+ /** A SOAP reply of the sayHello operation. */
+ private static final String REPLY =
+ """
+ <soapenv:Envelope
xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/"; \
+ xmlns:tns="http://example.com/hello";>
+ <soapenv:Body>
+ <tns:sayHelloResponse>
+ <greeting>Hello, Hop</greeting>
+ </tns:sayHelloResponse>
+ </soapenv:Body>
+ </soapenv:Envelope>
+ """;
+
+ /** An input row, over-allocated like the rows the transforms before this
one send. */
+ private static Object[] inputRow() {
+ Object[] row = RowDataUtil.allocateRowData(2);
+ row[0] = "Hop";
+ row[1] = "some other input";
+ return row;
+ }
+
+ private static IRowMeta inputRowMeta() {
+ IRowMeta rowMeta = new RowMeta();
+ rowMeta.addValueMeta(new ValueMetaString("firstName"));
+ rowMeta.addValueMeta(new ValueMetaString("other"));
+ return rowMeta;
+ }
+
+ private static WebServiceField outputField(String name, String wsName) {
+ WebServiceField field = new WebServiceField();
+ field.setName(name);
+ field.setWsName(wsName);
+ field.setXsdType("string");
+ return field;
+ }
+
+ /**
+ * Feed {@link #REPLY} to the reply parsing of a transform with this
configuration, the way
+ * processRow() sets it up, and return the rows it sends on.
+ */
+ private List<Object[]> processReply(WebServiceMeta meta) throws Exception {
+ meta.setCompatible(false);
+ WebServiceData data = new WebServiceData();
+ IRowMeta outputRowMeta = inputRowMeta().clone();
+ meta.getFields(outputRowMeta, "Web services lookup", null, null, new
Variables(), null);
+ data.outputRowMeta = outputRowMeta;
+
+ WebService transform =
+ spy(
+ new WebService(
+ mockHelper.transformMeta,
+ meta,
+ data,
+ 0,
+ mockHelper.pipelineMeta,
+ mockHelper.pipeline));
+ transform.setInputRowMeta(inputRowMeta());
+ List<Object[]> rows = new ArrayList<>();
+ doAnswer(
+ invocation -> {
+ rows.add(invocation.getArgument(1));
+ return null;
+ })
+ .when(transform)
+ .putRow(any(), any());
+
+ transform.processRows(
+ new ByteArrayInputStream(REPLY.getBytes(StandardCharsets.UTF_8)),
+ inputRow(),
+ inputRowMeta(),
+ false,
+ StandardCharsets.UTF_8.name());
+ return rows;
+ }
+
+ @Test
+ void replyAsStringWithoutInputDataGoesInTheOutputField() throws Exception {
+ WebServiceMeta meta = new WebServiceMeta();
+ meta.setReturningReplyAsString(true);
+ meta.setPassingInputData(false);
+ meta.getFieldsOut().add(outputField("reply", "sayHelloResponse"));
+
+ List<Object[]> rows = processReply(meta);
+
+ assertEquals(1, rows.size());
+ assertEquals(REPLY, rows.get(0)[0]);
+ }
+
+ @Test
+ void replyAsStringWithInputDataGoesAfterTheInputFields() throws Exception {
+ WebServiceMeta meta = new WebServiceMeta();
+ meta.setReturningReplyAsString(true);
+ meta.setPassingInputData(true);
+ meta.getFieldsOut().add(outputField("reply", "sayHelloResponse"));
+
+ List<Object[]> rows = processReply(meta);
+
+ assertEquals(1, rows.size());
+ assertEquals("Hop", rows.get(0)[0]);
+ assertEquals("some other input", rows.get(0)[1]);
+ assertEquals(REPLY, rows.get(0)[2]);
+ }
+
+ @Test
+ void replyAsStringWithoutInputDataLeaksNoInputValues() throws Exception {
+ WebServiceMeta meta = new WebServiceMeta();
+ meta.setReturningReplyAsString(true);
+ meta.setPassingInputData(false);
+ meta.getFieldsOut().add(outputField("reply", "sayHelloResponse"));
+ meta.getFieldsOut().add(outputField("unused", "unused"));
+
+ List<Object[]> rows = processReply(meta);
+
+ assertEquals(1, rows.size());
+ assertEquals(REPLY, rows.get(0)[0]);
+ assertNull(rows.get(0)[1]);
+ }
+
+ @Test
+ void replyAsStringWithoutOutputFieldStillSendsTheRow() throws Exception {
+ WebServiceMeta meta = new WebServiceMeta();
+ meta.setReturningReplyAsString(true);
+ meta.setPassingInputData(true);
+
+ List<Object[]> rows = processReply(meta);
+
+ assertEquals(1, rows.size());
+ assertEquals("Hop", rows.get(0)[0]);
+ assertEquals("some other input", rows.get(0)[1]);
+ }
+
+ @Test
+ void responseNodeAsXmlWithoutInputDataGoesInTheOutputField() throws
Exception {
+ WebServiceMeta meta = new WebServiceMeta();
+ meta.setPassingInputData(false);
+ meta.getFieldsOut().add(outputField("response", "tns:sayHelloResponse"));
+
+ List<Object[]> rows = processReply(meta);
+
+ assertEquals(1, rows.size());
+ String xml = (String) rows.get(0)[0];
+ assertTrue(xml.contains("sayHelloResponse") && xml.contains("Hello, Hop"),
xml);
+ }
}
diff --git
a/plugins/transforms/webservices/src/test/java/org/apache/hop/pipeline/transforms/webservices/wsdl/WsdlTest.java
b/plugins/transforms/webservices/src/test/java/org/apache/hop/pipeline/transforms/webservices/wsdl/WsdlTest.java
new file mode 100644
index 0000000000..0a4593afee
--- /dev/null
+++
b/plugins/transforms/webservices/src/test/java/org/apache/hop/pipeline/transforms/webservices/wsdl/WsdlTest.java
@@ -0,0 +1,374 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hop.pipeline.transforms.webservices.wsdl;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertNotNull;
+import static org.junit.jupiter.api.Assertions.assertNull;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+import com.sun.net.httpserver.HttpServer;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.net.InetAddress;
+import java.net.InetSocketAddress;
+import java.net.URI;
+import java.nio.charset.StandardCharsets;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.Base64;
+import java.util.List;
+import java.util.Map;
+import java.util.UUID;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.atomic.AtomicReference;
+import javax.wsdl.extensions.schema.Schema;
+import javax.wsdl.extensions.schema.SchemaImport;
+import org.apache.hop.core.exception.HopRuntimeException;
+import org.apache.hop.core.variables.Variables;
+import org.apache.hop.core.vfs.HopVfs;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.io.TempDir;
+
+/**
+ * Loading a WSDL from the places a user can point the Web services lookup
transform at: an http(s)
+ * URL, a plain path as the file dialog hands it out, a file: URL, or any
other Hop VFS location.
+ */
+class WsdlTest {
+
+ private static final String RESOURCES =
"/org/apache/hop/pipeline/transforms/webservices/wsdl/";
+
+ @TempDir Path tempDir;
+
+ /** A folder with a space in its name: a plain path like that is not a valid
URI. */
+ private Path wsdlFolder;
+
+ @BeforeEach
+ void copyWsdlFiles() throws IOException {
+ wsdlFolder = tempDir.resolve("my wsdl folder");
+ Files.createDirectories(wsdlFolder.resolve("abstract"));
+ copyResource("hello.wsdl");
+ copyResource("hello-service.wsdl");
+ copyResource("abstract/hello-abstract.wsdl");
+ Files.createDirectories(wsdlFolder.resolve("schema"));
+ copyResource("hello-typed.wsdl");
+ copyResource("schema/hello-types.xsd");
+ }
+
+ /**
+ * Copies the WSDL files to an in-memory Hop VFS file system: one Java's URL
class cannot open,
+ * like s3:// or azure://, so every import has to go through Hop VFS to be
found.
+ */
+ private String copyToRam() throws Exception {
+ String folder = "ram:///wsdl-" + UUID.randomUUID();
+ for (String name :
+ List.of(
+ "hello-service.wsdl",
+ "abstract/hello-abstract.wsdl",
+ "hello-typed.wsdl",
+ "schema/hello-types.xsd")) {
+ HopVfs.getFileObject(folder + "/" + name).getParent().createFolder();
+ try (OutputStream out = HopVfs.getOutputStream(folder + "/" + name,
false)) {
+ Files.copy(wsdlFolder.resolve(name), out);
+ }
+ }
+ return folder;
+ }
+
+ /** The XML schema the typed WSDL imports was found and read, not just
named. */
+ private static void assertTypesSchemaWasImported(Wsdl wsdl) {
+ Schema schema = (Schema) wsdl.getWsdlTypes().getSchemas().get(0);
+ List<?> imports = (List<?>)
schema.getImports().get("http://example.com/hello/types";);
+ assertNotNull(imports, "the schema import is missing");
+ SchemaImport schemaImport = (SchemaImport) imports.get(0);
+ assertNotNull(schemaImport.getReferencedSchema(), "the imported schema was
not read");
+ }
+
+ private void copyResource(String name) throws IOException {
+ try (InputStream in = WsdlTest.class.getResourceAsStream(RESOURCES +
name)) {
+ assertNotNull(in, "Missing test resource " + name);
+ Files.copy(in, wsdlFolder.resolve(name));
+ }
+ }
+
+ private static Wsdl load(String location) {
+ return new Wsdl(location, null, null, null, null, null);
+ }
+
+ private static void assertHelloService(Wsdl wsdl) throws Exception {
+ assertEquals("Hello_Service", wsdl.getServiceName());
+ assertEquals("Hello_Port", wsdl.getPortName());
+ assertEquals("http://localhost:8080/hello";, wsdl.getServiceEndpoint());
+ assertEquals(1, wsdl.getOperations().size());
+ assertNotNull(wsdl.getOperation("sayHello"));
+ }
+
+ @Test
+ void loadsFromPlainPath() throws Exception {
+ // What the file dialog hands out, once a variable like ${PROJECT_HOME} is
resolved (#4813)
+ assertHelloService(load(wsdlFolder.resolve("hello.wsdl").toString()));
+ }
+
+ @Test
+ void ignoresSurroundingWhitespace() throws Exception {
+ assertHelloService(load(" " + wsdlFolder.resolve("hello.wsdl") + " "));
+ }
+
+ @Test
+ void loadsFromFileUrl() throws Exception {
+
assertHelloService(load(wsdlFolder.resolve("hello.wsdl").toUri().toString()));
+ }
+
+ @Test
+ void loadsFromFileUri() throws Exception {
+ Wsdl wsdl = new Wsdl(wsdlFolder.resolve("hello.wsdl").toUri(), null, null);
+ assertHelloService(wsdl);
+ }
+
+ @Test
+ void loadsWithVariables() throws Exception {
+ Wsdl wsdl =
+ new Wsdl(
+ wsdlFolder.resolve("hello.wsdl").toString(), new Variables(),
null, null, null, null);
+ assertHelloService(wsdl);
+ }
+
+ @Test
+ void resolvesImportsRelativeToTheFile() throws Exception {
+ // The messages and port type live in abstract/hello-abstract.wsdl, next
to this file
+
assertHelloService(load(wsdlFolder.resolve("hello-service.wsdl").toString()));
+ }
+
+ @Test
+ void resolvesSchemaImportsRelativeToTheFile() throws Exception {
+ Wsdl wsdl = load(wsdlFolder.resolve("hello-typed.wsdl").toString());
+ assertEquals("Hello_Service", wsdl.getServiceName());
+ assertTypesSchemaWasImported(wsdl);
+ }
+
+ @Test
+ void resolvesImportsOnAnyHopVfsFileSystem() throws Exception {
+ assertHelloService(load(copyToRam() + "/hello-service.wsdl"));
+ }
+
+ @Test
+ void resolvesSchemaImportsOnAnyHopVfsFileSystem() throws Exception {
+ Wsdl wsdl = load(copyToRam() + "/hello-typed.wsdl");
+ assertEquals("Hello_Service", wsdl.getServiceName());
+ assertTypesSchemaWasImported(wsdl);
+ }
+
+ @Test
+ void aMissingImportNamesTheImport() throws Exception {
+ String folder = copyToRam();
+ HopVfs.getFileObject(folder + "/abstract/hello-abstract.wsdl").delete();
+ HopRuntimeException e =
+ assertThrows(HopRuntimeException.class, () -> load(folder +
"/hello-service.wsdl"));
+ assertTrue(e.getMessage().contains("abstract/hello-abstract.wsdl"),
e.getMessage());
+ }
+
+ @Test
+ void relativeImportsOverHttpAreAuthenticatedWithTheWsdlHost() throws
Exception {
+ Map<String, String> authorizations = new ConcurrentHashMap<>();
+ HttpServer server = serveFolder(authorizations, true);
+ try {
+ String url = baseUrlOf(server) + "/hello-service.wsdl";
+ assertHelloService(new Wsdl(url, null, null, null, "user", "secret"));
+ String expected =
+ "Basic "
+ +
Base64.getEncoder().encodeToString("user:secret".getBytes(StandardCharsets.UTF_8));
+ assertEquals(expected, authorizations.get("/hello-service.wsdl"));
+ assertEquals(expected,
authorizations.get("/abstract/hello-abstract.wsdl"));
+ } finally {
+ server.stop(0);
+ }
+ }
+
+ @Test
+ void importsFromAnotherHostGetNoCredentials() throws Exception {
+ Map<String, String> wsdlHost = new ConcurrentHashMap<>();
+ Map<String, String> otherHost = new ConcurrentHashMap<>();
+ HttpServer other = serveFolder(otherHost, false);
+ // The WSDL names its import with an absolute URL on another server
+ String service =
+ Files.readString(wsdlFolder.resolve("hello-service.wsdl"))
+ .replace(
+ "location=\"abstract/hello-abstract.wsdl\"",
+ "location=\"" + baseUrlOf(other) +
"/abstract/hello-abstract.wsdl\"");
+ Files.writeString(wsdlFolder.resolve("hello-service.wsdl"), service);
+ HttpServer server = serveFolder(wsdlHost, false);
+ try {
+ assertHelloService(
+ new Wsdl(baseUrlOf(server) + "/hello-service.wsdl", null, null,
null, "user", "secret"));
+ assertNotNull(wsdlHost.get("/hello-service.wsdl"));
+ assertTrue(otherHost.containsKey("/abstract/hello-abstract.wsdl"));
+ assertEquals(
+ "", otherHost.get("/abstract/hello-abstract.wsdl"), "credentials
went to another host");
+ } finally {
+ server.stop(0);
+ other.stop(0);
+ }
+ }
+
+ @Test
+ void missingFileNamesTheLocation() {
+ String location = wsdlFolder.resolve("does-not-exist.wsdl").toString();
+ HopRuntimeException e = assertThrows(HopRuntimeException.class, () ->
load(location));
+ assertTrue(e.getMessage().startsWith(Wsdl.CONST_COULD_NOT_LOAD_WSDL_FILE +
location));
+ assertTrue(e.getMessage().contains("does not exist"), e.getMessage());
+ }
+
+ @Test
+ void blankLocationIsRejected() {
+ HopRuntimeException e = assertThrows(HopRuntimeException.class, () ->
load(" "));
+ assertTrue(e.getMessage().contains("No WSDL location"), e.getMessage());
+ }
+
+ @Test
+ void recognizesHttpLocations() {
+ assertTrue(Wsdl.isHttpLocation("http://host/service?wsdl";));
+ assertTrue(Wsdl.isHttpLocation("HTTPS://host/service?wsdl"));
+ assertFalse(Wsdl.isHttpLocation("/work/wsdl/service.wsdl"));
+ assertFalse(Wsdl.isHttpLocation("file:///work/wsdl/service.wsdl"));
+ assertFalse(Wsdl.isHttpLocation("C:\\work\\wsdl\\service.wsdl"));
+ assertFalse(Wsdl.isHttpLocation("s3://bucket/service.wsdl"));
+ }
+
+ @Test
+ void loadsOverHttpWithBasicAuthentication() throws Exception {
+ AtomicReference<String> authorization = new AtomicReference<>();
+ HttpServer server = serveWsdl(authorization, 200);
+ try {
+ Wsdl wsdl = new Wsdl(urlOf(server), null, null, null, "user", "secret");
+ assertHelloService(wsdl);
+ String expected =
+
Base64.getEncoder().encodeToString("user:secret".getBytes(StandardCharsets.UTF_8));
+ assertEquals("Basic " + expected, authorization.get());
+ } finally {
+ server.stop(0);
+ }
+ }
+
+ @Test
+ void loadsOverHttpWithoutAuthentication() throws Exception {
+ AtomicReference<String> authorization = new AtomicReference<>();
+ HttpServer server = serveWsdl(authorization, 200);
+ try {
+ assertHelloService(new Wsdl(new URI(urlOf(server)), null, null));
+ assertNull(authorization.get());
+ } finally {
+ server.stop(0);
+ }
+ }
+
+ @Test
+ void loadsOverHttpWithSurroundingWhitespace() throws Exception {
+ HttpServer server = serveWsdl(new AtomicReference<>(), 200);
+ try {
+ assertHelloService(load(" " + urlOf(server) + " "));
+ } finally {
+ server.stop(0);
+ }
+ }
+
+ @Test
+ void httpErrorNamesTheUrl() throws Exception {
+ HttpServer server = serveWsdl(new AtomicReference<>(), 404);
+ try {
+ String url = urlOf(server);
+ HopRuntimeException e = assertThrows(HopRuntimeException.class, () ->
load(url));
+ assertTrue(e.getMessage().contains("Unable to read WSDL from " + url),
e.getMessage());
+ } finally {
+ server.stop(0);
+ }
+ }
+
+ private HttpServer serveWsdl(AtomicReference<String> authorization, int
status)
+ throws IOException {
+ byte[] body = Files.readAllBytes(wsdlFolder.resolve("hello.wsdl"));
+ HttpServer server =
+ HttpServer.create(new
InetSocketAddress(InetAddress.getLoopbackAddress(), 0), 0);
+ server.createContext(
+ "/hello",
+ exchange -> {
+
authorization.set(exchange.getRequestHeaders().getFirst("Authorization"));
+ exchange.getResponseHeaders().add("Content-Type", "text/xml");
+ if (status == 200) {
+ exchange.sendResponseHeaders(200, body.length);
+ try (OutputStream out = exchange.getResponseBody()) {
+ out.write(body);
+ }
+ } else {
+ exchange.sendResponseHeaders(status, -1);
+ exchange.close();
+ }
+ });
+ server.start();
+ return server;
+ }
+
+ /**
+ * Serves the WSDL folder, recording the Authorization header of each
request by path ("" when
+ * there was none). With requireAuthentication a request without one gets a
401.
+ */
+ private HttpServer serveFolder(Map<String, String> authorizations, boolean
requireAuthentication)
+ throws IOException {
+ HttpServer server =
+ HttpServer.create(new
InetSocketAddress(InetAddress.getLoopbackAddress(), 0), 0);
+ server.createContext(
+ "/",
+ exchange -> {
+ String path = exchange.getRequestURI().getPath();
+ String authorization =
exchange.getRequestHeaders().getFirst("Authorization");
+ authorizations.put(path, authorization == null ? "" : authorization);
+ Path file = wsdlFolder.resolve(path.substring(1));
+ if ((requireAuthentication && authorization == null) ||
!Files.isRegularFile(file)) {
+ exchange.sendResponseHeaders(authorization == null ? 401 : 404,
-1);
+ exchange.close();
+ return;
+ }
+ byte[] body = Files.readAllBytes(file);
+ exchange.getResponseHeaders().add("Content-Type", "text/xml");
+ exchange.sendResponseHeaders(200, body.length);
+ try (OutputStream out = exchange.getResponseBody()) {
+ out.write(body);
+ }
+ });
+ server.start();
+ return server;
+ }
+
+ private static String baseUrlOf(HttpServer server) {
+ return "http://";
+ + InetAddress.getLoopbackAddress().getHostAddress()
+ + ":"
+ + server.getAddress().getPort();
+ }
+
+ private static String urlOf(HttpServer server) {
+ return "http://";
+ + InetAddress.getLoopbackAddress().getHostAddress()
+ + ":"
+ + server.getAddress().getPort()
+ + "/hello?wsdl";
+ }
+}
diff --git
a/plugins/transforms/webservices/src/test/resources/org/apache/hop/pipeline/transforms/webservices/wsdl/abstract/hello-abstract.wsdl
b/plugins/transforms/webservices/src/test/resources/org/apache/hop/pipeline/transforms/webservices/wsdl/abstract/hello-abstract.wsdl
new file mode 100644
index 0000000000..a41b8b9dd3
--- /dev/null
+++
b/plugins/transforms/webservices/src/test/resources/org/apache/hop/pipeline/transforms/webservices/wsdl/abstract/hello-abstract.wsdl
@@ -0,0 +1,34 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one or more
+ ~ contributor license agreements. See the NOTICE file distributed with
+ ~ this work for additional information regarding copyright ownership.
+ ~ The ASF licenses this file to You under the Apache License, Version 2.0
+ ~ (the "License"); you may not use this file except in compliance with
+ ~ the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing, software
+ ~ distributed under the License is distributed on an "AS IS" BASIS,
+ ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ ~ See the License for the specific language governing permissions and
+ ~ limitations under the License.
+ -->
+<definitions name="HelloAbstract" targetNamespace="http://example.com/hello";
+ xmlns="http://schemas.xmlsoap.org/wsdl/";
+ xmlns:tns="http://example.com/hello";
+ xmlns:xsd="http://www.w3.org/2001/XMLSchema";>
+ <message name="SayHelloRequest">
+ <part name="firstName" type="xsd:string"/>
+ </message>
+ <message name="SayHelloResponse">
+ <part name="greeting" type="xsd:string"/>
+ </message>
+ <portType name="Hello_PortType">
+ <operation name="sayHello">
+ <input message="tns:SayHelloRequest"/>
+ <output message="tns:SayHelloResponse"/>
+ </operation>
+ </portType>
+</definitions>
diff --git
a/plugins/transforms/webservices/src/test/resources/org/apache/hop/pipeline/transforms/webservices/wsdl/hello-service.wsdl
b/plugins/transforms/webservices/src/test/resources/org/apache/hop/pipeline/transforms/webservices/wsdl/hello-service.wsdl
new file mode 100644
index 0000000000..9ba90523a8
--- /dev/null
+++
b/plugins/transforms/webservices/src/test/resources/org/apache/hop/pipeline/transforms/webservices/wsdl/hello-service.wsdl
@@ -0,0 +1,38 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one or more
+ ~ contributor license agreements. See the NOTICE file distributed with
+ ~ this work for additional information regarding copyright ownership.
+ ~ The ASF licenses this file to You under the Apache License, Version 2.0
+ ~ (the "License"); you may not use this file except in compliance with
+ ~ the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing, software
+ ~ distributed under the License is distributed on an "AS IS" BASIS,
+ ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ ~ See the License for the specific language governing permissions and
+ ~ limitations under the License.
+ -->
+<!-- The binding and service of the Hello service. The messages and the port
type are imported
+ from a file next to this one, by a relative location. -->
+<definitions name="HelloService" targetNamespace="http://example.com/hello";
+ xmlns="http://schemas.xmlsoap.org/wsdl/";
+ xmlns:soap="http://schemas.xmlsoap.org/wsdl/soap/";
+ xmlns:tns="http://example.com/hello";>
+ <import namespace="http://example.com/hello";
location="abstract/hello-abstract.wsdl"/>
+ <binding name="Hello_Binding" type="tns:Hello_PortType">
+ <soap:binding style="rpc"
transport="http://schemas.xmlsoap.org/soap/http"/>
+ <operation name="sayHello">
+ <soap:operation soapAction="sayHello"/>
+ <input><soap:body use="literal"
namespace="http://example.com/hello"/></input>
+ <output><soap:body use="literal"
namespace="http://example.com/hello"/></output>
+ </operation>
+ </binding>
+ <service name="Hello_Service">
+ <port binding="tns:Hello_Binding" name="Hello_Port">
+ <soap:address location="http://localhost:8080/hello"/>
+ </port>
+ </service>
+</definitions>
diff --git
a/plugins/transforms/webservices/src/test/resources/org/apache/hop/pipeline/transforms/webservices/wsdl/hello-typed.wsdl
b/plugins/transforms/webservices/src/test/resources/org/apache/hop/pipeline/transforms/webservices/wsdl/hello-typed.wsdl
new file mode 100644
index 0000000000..e7dbee130e
--- /dev/null
+++
b/plugins/transforms/webservices/src/test/resources/org/apache/hop/pipeline/transforms/webservices/wsdl/hello-typed.wsdl
@@ -0,0 +1,61 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one or more
+ ~ contributor license agreements. See the NOTICE file distributed with
+ ~ this work for additional information regarding copyright ownership.
+ ~ The ASF licenses this file to You under the Apache License, Version 2.0
+ ~ (the "License"); you may not use this file except in compliance with
+ ~ the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing, software
+ ~ distributed under the License is distributed on an "AS IS" BASIS,
+ ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ ~ See the License for the specific language governing permissions and
+ ~ limitations under the License.
+ -->
+<!-- The Hello service with document/literal messages. Its types are in an XML
schema imported
+ from schema/hello-types.xsd, next to this file. -->
+<definitions name="HelloTyped" targetNamespace="http://example.com/hello/typed";
+ xmlns="http://schemas.xmlsoap.org/wsdl/";
+ xmlns:soap="http://schemas.xmlsoap.org/wsdl/soap/";
+ xmlns:tns="http://example.com/hello/typed";
+ xmlns:types="http://example.com/hello/types";
+ xmlns:xsd="http://www.w3.org/2001/XMLSchema";>
+ <types>
+ <xsd:schema targetNamespace="http://example.com/hello/typed";>
+ <xsd:import namespace="http://example.com/hello/types";
+ schemaLocation="schema/hello-types.xsd"/>
+ </xsd:schema>
+ </types>
+ <message name="SayHelloRequest">
+ <part name="parameters" element="types:sayHello"/>
+ </message>
+ <message name="SayHelloResponse">
+ <part name="parameters" element="types:sayHelloResponse"/>
+ </message>
+ <portType name="Hello_PortType">
+ <operation name="sayHello">
+ <input message="tns:SayHelloRequest"/>
+ <output message="tns:SayHelloResponse"/>
+ </operation>
+ </portType>
+ <binding name="Hello_Binding" type="tns:Hello_PortType">
+ <soap:binding style="document"
transport="http://schemas.xmlsoap.org/soap/http"/>
+ <operation name="sayHello">
+ <soap:operation soapAction="sayHello"/>
+ <input>
+ <soap:body use="literal"/>
+ </input>
+ <output>
+ <soap:body use="literal"/>
+ </output>
+ </operation>
+ </binding>
+ <service name="Hello_Service">
+ <port name="Hello_Port" binding="tns:Hello_Binding">
+ <soap:address location="http://localhost:8080/hello"/>
+ </port>
+ </service>
+</definitions>
diff --git
a/plugins/transforms/webservices/src/test/resources/org/apache/hop/pipeline/transforms/webservices/wsdl/hello.wsdl
b/plugins/transforms/webservices/src/test/resources/org/apache/hop/pipeline/transforms/webservices/wsdl/hello.wsdl
new file mode 100644
index 0000000000..a491ab9dd3
--- /dev/null
+++
b/plugins/transforms/webservices/src/test/resources/org/apache/hop/pipeline/transforms/webservices/wsdl/hello.wsdl
@@ -0,0 +1,48 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one or more
+ ~ contributor license agreements. See the NOTICE file distributed with
+ ~ this work for additional information regarding copyright ownership.
+ ~ The ASF licenses this file to You under the Apache License, Version 2.0
+ ~ (the "License"); you may not use this file except in compliance with
+ ~ the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing, software
+ ~ distributed under the License is distributed on an "AS IS" BASIS,
+ ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ ~ See the License for the specific language governing permissions and
+ ~ limitations under the License.
+ -->
+<definitions name="Hello" targetNamespace="http://example.com/hello";
+ xmlns="http://schemas.xmlsoap.org/wsdl/";
+ xmlns:soap="http://schemas.xmlsoap.org/wsdl/soap/";
+ xmlns:tns="http://example.com/hello";
+ xmlns:xsd="http://www.w3.org/2001/XMLSchema";>
+ <message name="SayHelloRequest">
+ <part name="firstName" type="xsd:string"/>
+ </message>
+ <message name="SayHelloResponse">
+ <part name="greeting" type="xsd:string"/>
+ </message>
+ <portType name="Hello_PortType">
+ <operation name="sayHello">
+ <input message="tns:SayHelloRequest"/>
+ <output message="tns:SayHelloResponse"/>
+ </operation>
+ </portType>
+ <binding name="Hello_Binding" type="tns:Hello_PortType">
+ <soap:binding style="rpc"
transport="http://schemas.xmlsoap.org/soap/http"/>
+ <operation name="sayHello">
+ <soap:operation soapAction="sayHello"/>
+ <input><soap:body use="literal"
namespace="http://example.com/hello"/></input>
+ <output><soap:body use="literal"
namespace="http://example.com/hello"/></output>
+ </operation>
+ </binding>
+ <service name="Hello_Service">
+ <port binding="tns:Hello_Binding" name="Hello_Port">
+ <soap:address location="http://localhost:8080/hello"/>
+ </port>
+ </service>
+</definitions>
diff --git
a/plugins/transforms/webservices/src/test/resources/org/apache/hop/pipeline/transforms/webservices/wsdl/schema/hello-types.xsd
b/plugins/transforms/webservices/src/test/resources/org/apache/hop/pipeline/transforms/webservices/wsdl/schema/hello-types.xsd
new file mode 100644
index 0000000000..631e74f4aa
--- /dev/null
+++
b/plugins/transforms/webservices/src/test/resources/org/apache/hop/pipeline/transforms/webservices/wsdl/schema/hello-types.xsd
@@ -0,0 +1,36 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one or more
+ ~ contributor license agreements. See the NOTICE file distributed with
+ ~ this work for additional information regarding copyright ownership.
+ ~ The ASF licenses this file to You under the Apache License, Version 2.0
+ ~ (the "License"); you may not use this file except in compliance with
+ ~ the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing, software
+ ~ distributed under the License is distributed on an "AS IS" BASIS,
+ ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ ~ See the License for the specific language governing permissions and
+ ~ limitations under the License.
+ -->
+<!-- The message types of hello-typed.wsdl, imported from it by a relative
schemaLocation. -->
+<xsd:schema targetNamespace="http://example.com/hello/types";
+ xmlns:xsd="http://www.w3.org/2001/XMLSchema";
+ elementFormDefault="qualified">
+ <xsd:element name="sayHello">
+ <xsd:complexType>
+ <xsd:sequence>
+ <xsd:element name="firstName" type="xsd:string"/>
+ </xsd:sequence>
+ </xsd:complexType>
+ </xsd:element>
+ <xsd:element name="sayHelloResponse">
+ <xsd:complexType>
+ <xsd:sequence>
+ <xsd:element name="greeting" type="xsd:string"/>
+ </xsd:sequence>
+ </xsd:complexType>
+ </xsd:element>
+</xsd:schema>
