This is an automated email from the ASF dual-hosted git repository. github-merge-queue[bot] pushed a commit to branch gh-readonly-queue/dev/pr-13222-5d8937dd970ba12f2e29b9b17d4dc1ffcebb4e69 in repository https://gitbox.apache.org/repos/asf/seatunnel.git
commit 09a95daf02831f03ff0e4b6de0122315c4734624 Author: Sepuri Sai Krishna <[email protected]> AuthorDate: Tue Nov 17 15:53:16 2026 +0000 [Fix][Connector-V2][GraphQL] Fail a request that the server rejects (#13222) --- docs/en/connectors/sink/GraphQL.md | 5 + docs/en/connectors/source/GraphQL.md | 2 + .../introduction/concepts/incompatible-changes.md | 7 + docs/zh/connectors/sink/GraphQL.md | 5 + docs/zh/connectors/source/GraphQL.md | 2 + .../introduction/concepts/incompatible-changes.md | 7 + .../graphql/config/GraphQLSinkParameter.java | 3 +- .../graphql/config/GraphQLSourceParameter.java | 3 +- .../seatunnel/graphql/sink/GraphQLSinkWriter.java | 48 ++++- .../source/reader/GraphQLSourceHttpReader.java | 5 + .../seatunnel/graphql/util/GraphQLUtil.java | 30 +++ .../graphql/sink/GraphQLSinkWriterTest.java | 221 +++++++++++++++++++++ .../source/GraphQLSourceHttpReaderTest.java | 169 ++++++++++++++++ .../seatunnel/e2e/connector/graphql/GraphQLIT.java | 117 ++++++++++- .../src/test/resources/fake_to_graphql.conf | 2 +- .../test/resources/fake_to_graphql_rejected.conf | 58 ++++++ .../graphql_rejected_query_to_assert.conf | 64 ++++++ .../src/test/resources/graphql_to_assert.conf | 10 + 18 files changed, 746 insertions(+), 12 deletions(-) diff --git a/docs/en/connectors/sink/GraphQL.md b/docs/en/connectors/sink/GraphQL.md index d9afc0addd..ab5ace631d 100644 --- a/docs/en/connectors/sink/GraphQL.md +++ b/docs/en/connectors/sink/GraphQL.md @@ -16,6 +16,11 @@ The GraphQL sink connector writes SeaTunnel rows to a GraphQL service by sending `mutation` over HTTP POST. For each input row, the sink adds row fields to GraphQL `variables` with the same field names, then sends the configured mutation. +A write fails when the server answers with a status other than 200, or with a non-empty `errors` +array. GraphQL servers usually report a rejected mutation with HTTP 200 and an `errors` array, so +the sink checks both. Date and time fields are sent as ISO 8601 strings, for example +`2020-02-02T02:02:00`. When `variables` is not set, the row fields are sent in a new object. + ## Key Features - [ ] [exactly-once](../../introduction/concepts/connector-v2-features.md) diff --git a/docs/en/connectors/source/GraphQL.md b/docs/en/connectors/source/GraphQL.md index a829ec54f7..4506020302 100644 --- a/docs/en/connectors/source/GraphQL.md +++ b/docs/en/connectors/source/GraphQL.md @@ -68,6 +68,8 @@ It can be downloaded via install-plugin.sh or from Maven central repository. - When `schema.fields` is configured, set `format = "json"` and point `content_field` to the array or object that should become SeaTunnel rows. - For streaming query mode over HTTP, configure `poll_interval_millis` to control how often SeaTunnel sends the same query again. - For WebSocket subscription mode, `max_retries` and `retry_delay_ms` only control reconnect behavior after a subscription connection fails. +- In HTTP query mode, the read fails when the response contains a non-empty `errors` array, even with HTTP 200 and partial `data`. Subscription messages are not checked. +- When `variables` is not set, the request sends an empty `variables` object. ## Task Example diff --git a/docs/en/introduction/concepts/incompatible-changes.md b/docs/en/introduction/concepts/incompatible-changes.md index 53c0bf9ce7..302affb445 100644 --- a/docs/en/introduction/concepts/incompatible-changes.md +++ b/docs/en/introduction/concepts/incompatible-changes.md @@ -42,6 +42,13 @@ You need to check this document before you upgrade to related version. restart instead of retrying a batch locally or continuing after a row error. Auto-commit connections, jobs without checkpointing, and XA writers retain their existing behavior. +### GraphQL connector: a response with GraphQL errors fails the job + +- **Behavior change: the GraphQL sink and the HTTP query mode of the GraphQL source fail when the server reports errors** + - **Description**: GraphQL servers usually answer a rejected query or mutation with HTTP 200 and an `errors` array. The sink only checked the status code, and for a status other than 200 it only logged the response, so a rejected mutation was dropped and the job finished. The source read such a response as if it were data, so a rejected query produced no rows. Both now fail with the server's errors in the message, and the sink also fails on a status other than 200. Date and time fields [...] + - **Impact**: A job whose requests the server rejects now fails instead of finishing with missing rows. This includes a source response with both `data` and a non-empty `errors` array, which was read as rows with `null` in the fields that failed: it now fails the job, and there is no option to keep the old behavior. In the streaming HTTP query mode, one poll answered with errors now fails the task, where before that poll produced no rows and polling went on. Subscription mode is unchanged. + - **Migration Guide**: No configuration change is needed. If a job starts failing, the error message shows what the server rejected. Fix the query, the mutation, the variable types or the permissions on the server; for example, Hasura expects `float8` for a PostgreSQL `double precision` column, not `Float`. + ### Lance sink: TIMESTAMP values keep their microseconds - **Behavior change: a `TIMESTAMP` written by the Lance sink keeps the digits below the millisecond** diff --git a/docs/zh/connectors/sink/GraphQL.md b/docs/zh/connectors/sink/GraphQL.md index b449f00984..d9c15e961a 100644 --- a/docs/zh/connectors/sink/GraphQL.md +++ b/docs/zh/connectors/sink/GraphQL.md @@ -16,6 +16,11 @@ GraphQL Sink 连接器通过 HTTP POST 向 GraphQL 服务写入数据。它会 GraphQL `mutation`:输入行里的字段会按同名字段放入 GraphQL `variables`,然后随配置的 mutation 一起发送。 +当服务端返回的状态码不是 200,或者响应中的 `errors` 数组不为空时,写入会失败。GraphQL +服务通常用 HTTP 200 加 `errors` 数组来表示 mutation 被拒绝,所以 sink 会同时检查这两点。 +日期和时间字段会以 ISO 8601 字符串发送,例如 `2020-02-02T02:02:00`。未配置 `variables` +时,行字段会放入一个新的对象发送。 + ## 主要特性 - [ ] [精确一次](../../introduction/concepts/connector-v2-features.md) diff --git a/docs/zh/connectors/source/GraphQL.md b/docs/zh/connectors/source/GraphQL.md index 805f3ee620..956c2e5269 100644 --- a/docs/zh/connectors/source/GraphQL.md +++ b/docs/zh/connectors/source/GraphQL.md @@ -67,6 +67,8 @@ GraphQL 源连接器用于从 GraphQL 服务读取数据。它支持: - 配置 `schema.fields` 读取结构化数据时,通常需要设置 `format = "json"`,并让 `content_field` 指向要转换为 SeaTunnel 行的数组或对象。 - 使用 HTTP 流式轮询读取时,可以通过 `poll_interval_millis` 控制重复发送同一条查询的间隔。 - 使用 WebSocket 订阅读取时,`max_retries` 和 `retry_delay_ms` 只控制订阅连接失败后的重连行为。 +- HTTP 查询模式下,只要响应中的 `errors` 数组不为空,读取就会失败,即使状态码是 200 且带有部分 `data`。订阅消息不做这项检查。 +- 未配置 `variables` 时,请求会发送一个空的 `variables` 对象。 ## 任务示例 diff --git a/docs/zh/introduction/concepts/incompatible-changes.md b/docs/zh/introduction/concepts/incompatible-changes.md index 1e9e3bd0dc..ef2f3208d1 100644 --- a/docs/zh/introduction/concepts/incompatible-changes.md +++ b/docs/zh/introduction/concepts/incompatible-changes.md @@ -37,6 +37,13 @@ - **升级说明**:无需迁移配置或 checkpoint 格式。受影响的作业可能由批次原地重试或忽略行级错误继续运行, 转为失败恢复。自动提交连接、未开启 checkpoint 的作业和 XA Writer 保持原有行为。 +### GraphQL 连接器:响应中带有 GraphQL errors 时作业会失败 + +- **行为变更:服务端报告错误时,GraphQL Sink 和 GraphQL Source 的 HTTP 查询模式会失败** + - **描述**:GraphQL 服务通常用 HTTP 200 加 `errors` 数组来表示查询或 mutation 被拒绝。Sink 此前只检查状态码,状态码不是 200 时也只记录日志,因此被拒绝的 mutation 会被丢弃,作业照常结束。Source 会把这样的响应当作数据读取,因此被拒绝的查询不会产生任何行。现在两者都会失败,并在错误信息中带上服务端返回的错误;状态码不是 200 时 Sink 也会失败。日期和时间字段现在以 ISO 8601 字符串发送;此前在 Java 11 上会被发送为嵌套的 JSON 对象,在 Java 17 上写入会失败。未配置 `variables` 时,现在发送空对象,而不是服务端会拒绝的字符串 `"{}"`;此前未配置 `variables` 的 Sink 每一行都会失败。 + - **影响**:请求被服务端拒绝的作业现在会失败,而不是在缺少数据的情况下正常结束。这也包括同时带有 `data` 和非空 `errors` 数组的 Source 响应:此前会被读取为数据行,出错的字段为 `null`;现在作业会失败,且没有选项可以恢复以前的行为。在流式 HTTP 查询模式下,只要有一次轮询的响应带有错误,任务现在就会失败;此前这次轮询不产生数据,轮询继续进行。订阅模式不受影响。 + - **迁移指南**:无需修改配置。如果作业开始失败,请根据错误信息查看服务端拒绝的内容,并修正查询、mutation、变量类型或服务端的权限。例如,对于 PostgreSQL 的 `double precision` 列,Hasura 期望的类型是 `float8` 而不是 `Float`。 + ### Lance Sink:TIMESTAMP 值保留微秒 - **行为变更:Lance Sink 写入的 `TIMESTAMP` 保留毫秒以下的数字** diff --git a/seatunnel-connectors-v2/connector-graphql/src/main/java/org/apache/seatunnel/connectors/seatunnel/graphql/config/GraphQLSinkParameter.java b/seatunnel-connectors-v2/connector-graphql/src/main/java/org/apache/seatunnel/connectors/seatunnel/graphql/config/GraphQLSinkParameter.java index 6125dbee9a..3699f84074 100644 --- a/seatunnel-connectors-v2/connector-graphql/src/main/java/org/apache/seatunnel/connectors/seatunnel/graphql/config/GraphQLSinkParameter.java +++ b/seatunnel-connectors-v2/connector-graphql/src/main/java/org/apache/seatunnel/connectors/seatunnel/graphql/config/GraphQLSinkParameter.java @@ -47,7 +47,8 @@ public class GraphQLSinkParameter implements Serializable { GraphQLSinkOptions.VARIABLES.key(), pluginConfig.get(GraphQLSinkOptions.VARIABLES)); } else { - bodymap.put(GraphQLSinkOptions.VARIABLES.key(), "{}"); + // GraphQL servers expect variables as a JSON object, never as a string. + bodymap.put(GraphQLSinkOptions.VARIABLES.key(), new HashMap<>()); } bodymap.put(GraphQLSinkOptions.QUERY.key(), query); this.httpParameter.setBody(JsonUtils.toJsonString(bodymap)); diff --git a/seatunnel-connectors-v2/connector-graphql/src/main/java/org/apache/seatunnel/connectors/seatunnel/graphql/config/GraphQLSourceParameter.java b/seatunnel-connectors-v2/connector-graphql/src/main/java/org/apache/seatunnel/connectors/seatunnel/graphql/config/GraphQLSourceParameter.java index fef75079a0..0c529e84b1 100644 --- a/seatunnel-connectors-v2/connector-graphql/src/main/java/org/apache/seatunnel/connectors/seatunnel/graphql/config/GraphQLSourceParameter.java +++ b/seatunnel-connectors-v2/connector-graphql/src/main/java/org/apache/seatunnel/connectors/seatunnel/graphql/config/GraphQLSourceParameter.java @@ -54,7 +54,8 @@ public class GraphQLSourceParameter implements Serializable { pluginConfig.get(GraphQLSourceOptions.VARIABLES)); } else { - bodymap.put(GraphQLSourceOptions.VARIABLES.key(), "{}"); + // GraphQL servers expect variables as a JSON object, never as a string. + bodymap.put(GraphQLSourceOptions.VARIABLES.key(), new HashMap<>()); } bodymap.put(GraphQLSourceOptions.QUERY.key(), query); this.httpParameter.setBody(JsonUtils.toJsonString(bodymap)); diff --git a/seatunnel-connectors-v2/connector-graphql/src/main/java/org/apache/seatunnel/connectors/seatunnel/graphql/sink/GraphQLSinkWriter.java b/seatunnel-connectors-v2/connector-graphql/src/main/java/org/apache/seatunnel/connectors/seatunnel/graphql/sink/GraphQLSinkWriter.java index b69783f489..2f69a93a9b 100644 --- a/seatunnel-connectors-v2/connector-graphql/src/main/java/org/apache/seatunnel/connectors/seatunnel/graphql/sink/GraphQLSinkWriter.java +++ b/seatunnel-connectors-v2/connector-graphql/src/main/java/org/apache/seatunnel/connectors/seatunnel/graphql/sink/GraphQLSinkWriter.java @@ -21,14 +21,24 @@ import org.apache.seatunnel.api.table.type.SeaTunnelRowType; import org.apache.seatunnel.common.exception.CommonError; import org.apache.seatunnel.common.utils.JsonUtils; import org.apache.seatunnel.connectors.seatunnel.graphql.config.GraphQLSinkParameter; +import org.apache.seatunnel.connectors.seatunnel.graphql.exception.GraphQLConnectorErrorCode; +import org.apache.seatunnel.connectors.seatunnel.graphql.exception.GraphQLConnectorException; +import org.apache.seatunnel.connectors.seatunnel.graphql.util.GraphQLUtil; import org.apache.seatunnel.connectors.seatunnel.http.client.HttpClientProvider; import org.apache.seatunnel.connectors.seatunnel.http.client.HttpResponse; +import org.apache.seatunnel.connectors.seatunnel.http.exception.HttpConnectorErrorCode; +import org.apache.seatunnel.connectors.seatunnel.http.exception.HttpConnectorException; import org.apache.seatunnel.connectors.seatunnel.http.sink.HttpSinkWriter; import com.google.gson.Gson; import lombok.extern.slf4j.Slf4j; import java.io.IOException; +import java.time.LocalDate; +import java.time.LocalDateTime; +import java.time.LocalTime; +import java.time.OffsetDateTime; +import java.time.format.DateTimeFormatter; import java.util.Collections; import java.util.HashMap; import java.util.Map; @@ -70,7 +80,7 @@ public class GraphQLSinkWriter extends HttpSinkWriter { continue; } - variablesTemplate.put(fieldName, fieldValue); + variablesTemplate.put(fieldName, toVariableValue(fieldValue)); } } @@ -83,19 +93,43 @@ public class GraphQLSinkWriter extends HttpSinkWriter { try { HttpResponse response = httpClient.doPost(httpParameter.getUrl(), httpParameter.getHeaders(), body); - if (HttpResponse.STATUS_OK == response.getCode()) { - return; + if (HttpResponse.STATUS_OK != response.getCode()) { + String message = + String.format( + "http client execute exception, http response status code:[%s], content:[%s]", + response.getCode(), response.getContent()); + throw new HttpConnectorException(HttpConnectorErrorCode.REQUEST_FAILED, message); } - log.error( - "http client execute exception, http response status code:[{}], content:[{}]", - response.getCode(), - response.getContent()); + GraphQLUtil.checkNoErrors( + response.getContent(), GraphQLConnectorErrorCode.GRAPHQL_SINK_OPERATION_ERROR); + } catch (HttpConnectorException | GraphQLConnectorException e) { + throw e; } catch (Exception e) { log.error(e.getMessage(), e); throw CommonError.jsonOperationError("GraphQLSinkWriter", body, e); } } + /** + * Gson cannot serialize java.time values: it writes them as nested objects on Java 11 and fails + * on Java 17. GraphQL date and time scalars take ISO 8601 strings. + */ + static Object toVariableValue(Object value) { + if (value instanceof LocalDateTime) { + return ((LocalDateTime) value).format(DateTimeFormatter.ISO_LOCAL_DATE_TIME); + } + if (value instanceof LocalDate) { + return ((LocalDate) value).format(DateTimeFormatter.ISO_LOCAL_DATE); + } + if (value instanceof LocalTime) { + return ((LocalTime) value).format(DateTimeFormatter.ISO_LOCAL_TIME); + } + if (value instanceof OffsetDateTime) { + return ((OffsetDateTime) value).format(DateTimeFormatter.ISO_OFFSET_DATE_TIME); + } + return value; + } + @Override public void close() throws IOException { if (Objects.nonNull(httpClient)) { diff --git a/seatunnel-connectors-v2/connector-graphql/src/main/java/org/apache/seatunnel/connectors/seatunnel/graphql/source/reader/GraphQLSourceHttpReader.java b/seatunnel-connectors-v2/connector-graphql/src/main/java/org/apache/seatunnel/connectors/seatunnel/graphql/source/reader/GraphQLSourceHttpReader.java index 69c98e2ca4..8918c306e5 100644 --- a/seatunnel-connectors-v2/connector-graphql/src/main/java/org/apache/seatunnel/connectors/seatunnel/graphql/source/reader/GraphQLSourceHttpReader.java +++ b/seatunnel-connectors-v2/connector-graphql/src/main/java/org/apache/seatunnel/connectors/seatunnel/graphql/source/reader/GraphQLSourceHttpReader.java @@ -26,6 +26,7 @@ import org.apache.seatunnel.api.table.type.SeaTunnelRow; import org.apache.seatunnel.connectors.seatunnel.common.source.AbstractSingleSplitReader; import org.apache.seatunnel.connectors.seatunnel.common.source.SingleSplitReaderContext; import org.apache.seatunnel.connectors.seatunnel.graphql.config.GraphQLSourceParameter; +import org.apache.seatunnel.connectors.seatunnel.graphql.exception.GraphQLConnectorErrorCode; import org.apache.seatunnel.connectors.seatunnel.graphql.util.GraphQLUtil; import org.apache.seatunnel.connectors.seatunnel.http.client.HttpClientProvider; import org.apache.seatunnel.connectors.seatunnel.http.client.HttpResponse; @@ -122,9 +123,13 @@ public class GraphQLSourceHttpReader extends AbstractSingleSplitReader<SeaTunnel BufferedReader bufferedReader = new BufferedReader(stringReader); String lineStr; while ((lineStr = bufferedReader.readLine()) != null) { + GraphQLUtil.checkNoErrors( + lineStr, GraphQLConnectorErrorCode.GRAPHQL_SOURCE_OPERATION_ERROR); GraphQLUtil.collect(deserializationCollector, lineStr, contentJson, output); } } else { + GraphQLUtil.checkNoErrors( + content, GraphQLConnectorErrorCode.GRAPHQL_SOURCE_OPERATION_ERROR); GraphQLUtil.collect(deserializationCollector, content, contentJson, output); } } diff --git a/seatunnel-connectors-v2/connector-graphql/src/main/java/org/apache/seatunnel/connectors/seatunnel/graphql/util/GraphQLUtil.java b/seatunnel-connectors-v2/connector-graphql/src/main/java/org/apache/seatunnel/connectors/seatunnel/graphql/util/GraphQLUtil.java index 5d5c78b25e..79280d615c 100644 --- a/seatunnel-connectors-v2/connector-graphql/src/main/java/org/apache/seatunnel/connectors/seatunnel/graphql/util/GraphQLUtil.java +++ b/seatunnel-connectors-v2/connector-graphql/src/main/java/org/apache/seatunnel/connectors/seatunnel/graphql/util/GraphQLUtil.java @@ -17,6 +17,8 @@ package org.apache.seatunnel.connectors.seatunnel.graphql.util; +import org.apache.seatunnel.shade.com.fasterxml.jackson.databind.JsonNode; + import org.apache.seatunnel.api.source.Collector; import org.apache.seatunnel.api.table.type.SeaTunnelRow; import org.apache.seatunnel.common.utils.JsonUtils; @@ -158,4 +160,32 @@ public class GraphQLUtil { } } } + + /** + * Fails when a GraphQL response reports errors. GraphQL servers usually answer a rejected query + * or mutation with HTTP 200 and a non-empty {@code errors} array, so the status code alone does + * not show that the request failed. A body that is not a JSON object is left to the caller. + * + * @param content the response body + * @param errorCode the error code to fail with + */ + public static void checkNoErrors(String content, GraphQLConnectorErrorCode errorCode) { + if (content == null || content.isEmpty()) { + return; + } + JsonNode response; + try { + response = JsonUtils.stringToJsonNode(content); + } catch (Exception e) { + return; + } + if (response == null || !response.isObject()) { + return; + } + JsonNode errors = response.get("errors"); + if (errors != null && errors.isArray() && errors.size() > 0) { + throw new GraphQLConnectorException( + errorCode, "The GraphQL server returned errors: " + errors); + } + } } diff --git a/seatunnel-connectors-v2/connector-graphql/src/test/java/org/apache/seatunnel/connectors/seatunnel/graphql/sink/GraphQLSinkWriterTest.java b/seatunnel-connectors-v2/connector-graphql/src/test/java/org/apache/seatunnel/connectors/seatunnel/graphql/sink/GraphQLSinkWriterTest.java new file mode 100644 index 0000000000..fd54566ae3 --- /dev/null +++ b/seatunnel-connectors-v2/connector-graphql/src/test/java/org/apache/seatunnel/connectors/seatunnel/graphql/sink/GraphQLSinkWriterTest.java @@ -0,0 +1,221 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.seatunnel.connectors.seatunnel.graphql.sink; + +import org.apache.seatunnel.shade.com.fasterxml.jackson.databind.JsonNode; + +import org.apache.seatunnel.api.configuration.ReadonlyConfig; +import org.apache.seatunnel.api.table.type.BasicType; +import org.apache.seatunnel.api.table.type.LocalTimeType; +import org.apache.seatunnel.api.table.type.SeaTunnelDataType; +import org.apache.seatunnel.api.table.type.SeaTunnelRow; +import org.apache.seatunnel.api.table.type.SeaTunnelRowType; +import org.apache.seatunnel.common.utils.JsonUtils; +import org.apache.seatunnel.connectors.seatunnel.graphql.config.GraphQLSinkParameter; +import org.apache.seatunnel.connectors.seatunnel.graphql.exception.GraphQLConnectorErrorCode; +import org.apache.seatunnel.connectors.seatunnel.graphql.exception.GraphQLConnectorException; +import org.apache.seatunnel.connectors.seatunnel.http.exception.HttpConnectorErrorCode; +import org.apache.seatunnel.connectors.seatunnel.http.exception.HttpConnectorException; + +import org.junit.jupiter.api.AfterEach; +import org.junit.jupiter.api.Assertions; +import org.junit.jupiter.api.BeforeEach; +import org.junit.jupiter.api.Test; + +import com.sun.net.httpserver.HttpServer; + +import java.io.ByteArrayOutputStream; +import java.io.InputStream; +import java.io.OutputStream; +import java.net.InetAddress; +import java.net.InetSocketAddress; +import java.nio.charset.StandardCharsets; +import java.time.LocalDate; +import java.time.LocalDateTime; +import java.time.LocalTime; +import java.time.OffsetDateTime; +import java.time.ZoneOffset; +import java.util.HashMap; +import java.util.Map; + +/** Checks that the GraphQL sink fails a write the server rejects, and how it sends values. */ +public class GraphQLSinkWriterTest { + + private static final SeaTunnelRowType ROW_TYPE = + new SeaTunnelRowType( + new String[] {"id", "created_at", "day", "at_time"}, + new SeaTunnelDataType<?>[] { + BasicType.INT_TYPE, + LocalTimeType.LOCAL_DATE_TIME_TYPE, + LocalTimeType.LOCAL_DATE_TYPE, + LocalTimeType.LOCAL_TIME_TYPE + }); + + private static final SeaTunnelRow ROW = + new SeaTunnelRow( + new Object[] { + 1, + LocalDateTime.of(2020, 2, 2, 2, 2), + LocalDate.of(2020, 2, 2), + LocalTime.of(10, 15, 30, 123_000_000) + }); + + private HttpServer server; + private volatile int status; + private volatile String responseBody; + private volatile String requestBody; + + @BeforeEach + void startServer() throws Exception { + server = HttpServer.create(new InetSocketAddress(InetAddress.getLoopbackAddress(), 0), 0); + server.createContext( + "/graphql", + exchange -> { + requestBody = read(exchange.getRequestBody()); + byte[] bytes = responseBody.getBytes(StandardCharsets.UTF_8); + exchange.getResponseHeaders().add("Content-Type", "application/json"); + exchange.sendResponseHeaders(status, bytes.length); + try (OutputStream out = exchange.getResponseBody()) { + out.write(bytes); + } + }); + server.start(); + } + + @AfterEach + void stopServer() { + server.stop(0); + } + + @Test + void graphQLErrorsInAnOkResponseFailTheWrite() throws Exception { + respond( + 200, + "{\"errors\":[{\"extensions\":{\"code\":\"validation-failed\"}," + + "\"message\":\"no mutations exist\"}]}"); + GraphQLSinkWriter writer = newWriter(); + try { + GraphQLConnectorException e = + Assertions.assertThrows( + GraphQLConnectorException.class, () -> writer.write(ROW)); + Assertions.assertEquals( + GraphQLConnectorErrorCode.GRAPHQL_SINK_OPERATION_ERROR, + e.getSeaTunnelErrorCode()); + Assertions.assertTrue(e.getMessage().contains("no mutations exist"), e.getMessage()); + } finally { + writer.close(); + } + } + + @Test + void aNonOkResponseFailsTheWrite() throws Exception { + respond(503, "service unavailable"); + GraphQLSinkWriter writer = newWriter(); + try { + HttpConnectorException e = + Assertions.assertThrows(HttpConnectorException.class, () -> writer.write(ROW)); + Assertions.assertEquals( + HttpConnectorErrorCode.REQUEST_FAILED, e.getSeaTunnelErrorCode()); + Assertions.assertTrue(e.getMessage().contains("503"), e.getMessage()); + } finally { + writer.close(); + } + } + + @Test + void anAcceptedMutationSendsDateAndTimeValuesAsIsoStrings() throws Exception { + respond(200, "{\"data\":{\"insert_sink\":{\"affected_rows\":1}}}"); + GraphQLSinkWriter writer = newWriter(); + try { + writer.write(ROW); + } finally { + writer.close(); + } + JsonNode variables = JsonUtils.stringToJsonNode(requestBody).get("variables"); + Assertions.assertEquals(1, variables.get("id").asInt()); + Assertions.assertEquals("2020-02-02T02:02:00", variables.get("created_at").asText()); + Assertions.assertEquals("2020-02-02", variables.get("day").asText()); + Assertions.assertEquals("10:15:30.123", variables.get("at_time").asText()); + } + + @Test + void anEmptyErrorsArrayIsNotAFailure() throws Exception { + respond(200, "{\"data\":{\"insert_sink\":{\"affected_rows\":1}},\"errors\":[]}"); + GraphQLSinkWriter writer = newWriter(); + try { + writer.write(ROW); + } finally { + writer.close(); + } + } + + @Test + void anOffsetDateTimeKeepsItsOffset() { + Assertions.assertEquals( + "2020-02-02T02:02:00+08:00", + GraphQLSinkWriter.toVariableValue( + OffsetDateTime.of(2020, 2, 2, 2, 2, 0, 0, ZoneOffset.ofHours(8)))); + Assertions.assertEquals(7, GraphQLSinkWriter.toVariableValue(7)); + } + + @Test + void aSinkWithoutVariablesSendsTheRowAsAnObject() throws Exception { + respond(200, "{\"data\":{\"insert_sink\":{\"affected_rows\":1}}}"); + GraphQLSinkWriter writer = newWriter(false); + try { + writer.write(ROW); + } finally { + writer.close(); + } + JsonNode variables = JsonUtils.stringToJsonNode(requestBody).get("variables"); + Assertions.assertTrue(variables.isObject(), requestBody); + Assertions.assertEquals(1, variables.get("id").asInt()); + } + + private void respond(int status, String body) { + this.status = status; + this.responseBody = body; + } + + private GraphQLSinkWriter newWriter() { + return newWriter(true); + } + + private GraphQLSinkWriter newWriter(boolean withVariables) { + Map<String, Object> config = new HashMap<>(); + config.put("url", "http://127.0.0.1:" + server.getAddress().getPort() + "/graphql"); + config.put( + "query", + "mutation M($id: Int!) { insert_sink(objects: {id: $id}) { affected_rows } }"); + if (withVariables) { + config.put("variables", new HashMap<>()); + } + return new GraphQLSinkWriter( + ROW_TYPE, new GraphQLSinkParameter(ReadonlyConfig.fromMap(config))); + } + + private static String read(InputStream in) throws java.io.IOException { + ByteArrayOutputStream out = new ByteArrayOutputStream(); + byte[] buffer = new byte[4096]; + int n; + while ((n = in.read(buffer)) > 0) { + out.write(buffer, 0, n); + } + return new String(out.toByteArray(), StandardCharsets.UTF_8); + } +} diff --git a/seatunnel-connectors-v2/connector-graphql/src/test/java/org/apache/seatunnel/connectors/seatunnel/graphql/source/GraphQLSourceHttpReaderTest.java b/seatunnel-connectors-v2/connector-graphql/src/test/java/org/apache/seatunnel/connectors/seatunnel/graphql/source/GraphQLSourceHttpReaderTest.java new file mode 100644 index 0000000000..e65f03f7bf --- /dev/null +++ b/seatunnel-connectors-v2/connector-graphql/src/test/java/org/apache/seatunnel/connectors/seatunnel/graphql/source/GraphQLSourceHttpReaderTest.java @@ -0,0 +1,169 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.seatunnel.connectors.seatunnel.graphql.source; + +import org.apache.seatunnel.shade.com.fasterxml.jackson.databind.JsonNode; + +import org.apache.seatunnel.api.configuration.ReadonlyConfig; +import org.apache.seatunnel.api.source.Boundedness; +import org.apache.seatunnel.api.source.Collector; +import org.apache.seatunnel.api.table.type.SeaTunnelRow; +import org.apache.seatunnel.common.utils.JsonUtils; +import org.apache.seatunnel.connectors.seatunnel.common.source.AbstractSingleSplitReader; +import org.apache.seatunnel.connectors.seatunnel.common.source.SingleSplitReaderContext; +import org.apache.seatunnel.connectors.seatunnel.graphql.exception.GraphQLConnectorErrorCode; +import org.apache.seatunnel.connectors.seatunnel.graphql.exception.GraphQLConnectorException; + +import org.junit.jupiter.api.AfterEach; +import org.junit.jupiter.api.Assertions; +import org.junit.jupiter.api.BeforeEach; +import org.junit.jupiter.api.Test; +import org.mockito.Mockito; + +import com.sun.net.httpserver.HttpServer; + +import java.io.ByteArrayOutputStream; +import java.io.IOException; +import java.io.InputStream; +import java.io.OutputStream; +import java.net.InetAddress; +import java.net.InetSocketAddress; +import java.nio.charset.StandardCharsets; +import java.util.ArrayList; +import java.util.HashMap; +import java.util.LinkedHashMap; +import java.util.List; +import java.util.Map; + +/** Checks that the GraphQL source fails when the server reports errors with HTTP 200. */ +public class GraphQLSourceHttpReaderTest { + + private HttpServer server; + private volatile String responseBody; + private volatile String requestBody; + + @BeforeEach + void startServer() throws Exception { + server = HttpServer.create(new InetSocketAddress(InetAddress.getLoopbackAddress(), 0), 0); + server.createContext( + "/graphql", + exchange -> { + requestBody = readAll(exchange.getRequestBody()); + byte[] bytes = responseBody.getBytes(StandardCharsets.UTF_8); + exchange.getResponseHeaders().add("Content-Type", "application/json"); + exchange.sendResponseHeaders(200, bytes.length); + try (OutputStream out = exchange.getResponseBody()) { + out.write(bytes); + } + }); + server.start(); + } + + @AfterEach + void stopServer() { + server.stop(0); + } + + @Test + void graphQLErrorsInAnOkResponseFailTheRead() throws Exception { + responseBody = + "{\"errors\":[{\"extensions\":{\"code\":\"validation-failed\"}," + + "\"message\":\"field 'source' not found in type: 'query_root'\"}]}"; + GraphQLConnectorException e = + Assertions.assertThrows(GraphQLConnectorException.class, () -> read()); + Assertions.assertEquals( + GraphQLConnectorErrorCode.GRAPHQL_SOURCE_OPERATION_ERROR, + e.getSeaTunnelErrorCode()); + Assertions.assertTrue(e.getMessage().contains("not found in type"), e.getMessage()); + } + + @Test + void partialDataWithErrorsFailsTheRead() { + responseBody = + "{\"data\":{\"source\":[{\"id\":1,\"name\":\"a\"}]}," + + "\"errors\":[{\"message\":\"cannot read field name\"}]}"; + Assertions.assertThrows(GraphQLConnectorException.class, () -> read()); + } + + @Test + void anOkResponseWithoutErrorsIsRead() throws Exception { + responseBody = + "{\"data\":{\"source\":[{\"id\":1,\"name\":\"a\"},{\"id\":2,\"name\":\"b\"}]}}"; + List<SeaTunnelRow> rows = read(); + Assertions.assertEquals(2, rows.size()); + Assertions.assertEquals(1, rows.get(0).getField(0)); + Assertions.assertEquals("b", rows.get(1).getField(1)); + } + + @Test + void aQueryWithoutVariablesSendsAnEmptyObject() throws Exception { + responseBody = "{\"data\":{\"source\":[{\"id\":1,\"name\":\"a\"}]}}"; + read(); + JsonNode variables = JsonUtils.stringToJsonNode(requestBody).get("variables"); + Assertions.assertTrue(variables.isObject(), requestBody); + Assertions.assertEquals(0, variables.size(), requestBody); + } + + private static String readAll(InputStream in) throws IOException { + ByteArrayOutputStream out = new ByteArrayOutputStream(); + byte[] buffer = new byte[4096]; + int n; + while ((n = in.read(buffer)) > 0) { + out.write(buffer, 0, n); + } + return new String(out.toByteArray(), StandardCharsets.UTF_8); + } + + private List<SeaTunnelRow> read() throws Exception { + Map<String, Object> fields = new LinkedHashMap<>(); + fields.put("id", "int"); + fields.put("name", "string"); + Map<String, Object> schema = new HashMap<>(); + schema.put("fields", fields); + Map<String, Object> config = new HashMap<>(); + config.put("url", "http://127.0.0.1:" + server.getAddress().getPort() + "/graphql"); + config.put("query", "query Q { source { id name } }"); + config.put("content_field", "$.data.source"); + config.put("schema", schema); + + SingleSplitReaderContext context = Mockito.mock(SingleSplitReaderContext.class); + Mockito.when(context.getBoundedness()).thenReturn(Boundedness.BOUNDED); + AbstractSingleSplitReader<SeaTunnelRow> reader = + new GraphQLSource(ReadonlyConfig.fromMap(config)).createReader(context); + List<SeaTunnelRow> rows = new ArrayList<>(); + Object lock = new Object(); + reader.open(); + try { + reader.pollNext( + new Collector<SeaTunnelRow>() { + @Override + public void collect(SeaTunnelRow record) { + rows.add(record); + } + + @Override + public Object getCheckpointLock() { + return lock; + } + }); + } finally { + reader.close(); + } + return rows; + } +} diff --git a/seatunnel-e2e/seatunnel-connector-v2-e2e/connector-graphql-e2e/src/test/java/org/apache/seatunnel/e2e/connector/graphql/GraphQLIT.java b/seatunnel-e2e/seatunnel-connector-v2-e2e/connector-graphql-e2e/src/test/java/org/apache/seatunnel/e2e/connector/graphql/GraphQLIT.java index 220f885d55..e6e45e5d3c 100644 --- a/seatunnel-e2e/seatunnel-connector-v2-e2e/connector-graphql-e2e/src/test/java/org/apache/seatunnel/e2e/connector/graphql/GraphQLIT.java +++ b/seatunnel-e2e/seatunnel-connector-v2-e2e/connector-graphql-e2e/src/test/java/org/apache/seatunnel/e2e/connector/graphql/GraphQLIT.java @@ -35,7 +35,10 @@ import org.testcontainers.utility.DockerLoggerFactory; import lombok.extern.slf4j.Slf4j; import java.io.IOException; +import java.io.OutputStream; +import java.net.HttpURLConnection; import java.net.URL; +import java.nio.charset.StandardCharsets; import java.nio.file.Files; import java.nio.file.Paths; import java.sql.Connection; @@ -43,6 +46,7 @@ import java.sql.DriverManager; import java.sql.ResultSet; import java.sql.SQLException; import java.sql.Statement; +import java.util.ArrayList; import java.util.Arrays; import java.util.List; import java.util.concurrent.TimeUnit; @@ -93,6 +97,17 @@ public class GraphQLIT extends TestSuiteBase implements TestResource { .withLogConsumer( new Slf4jLogConsumer(DockerLoggerFactory.getLogger(IMAGE))); Startables.deepStart(Stream.of(genericContainer)).join(); + // Hasura serves only tracked tables. Without this, every query and mutation of this + // test is rejected with HTTP 200 and an errors array. Each table is retried on its own: + // Hasura rejects tracking a table twice, so retrying both after one of them was tracked + // could never pass. + for (String table : Arrays.asList("source", "sink")) { + given().ignoreExceptions() + .await() + .pollInterval(1, TimeUnit.SECONDS) + .atMost(2, TimeUnit.MINUTES) + .untilAsserted(() -> trackTable(table)); + } } public void checkTableData() { @@ -166,9 +181,107 @@ public class GraphQLIT extends TestSuiteBase implements TestResource { throws IOException, InterruptedException { checkTableData(); Container.ExecResult execResult1 = container.executeJob("/graphql_to_assert.conf"); - Assertions.assertEquals(0, execResult1.getExitCode()); + Assertions.assertEquals(0, execResult1.getExitCode(), execResult1.getStderr()); + executeSql("TRUNCATE sink"); Container.ExecResult execResult = container.executeJob("/fake_to_graphql.conf"); - Assertions.assertEquals(0, execResult.getExitCode()); + Assertions.assertEquals(0, execResult.getExitCode(), execResult.getStderr()); + Assertions.assertEquals( + Arrays.asList( + "1|true|NEW|6.3|2020-02-02 02:02:02.0", + "2|true|NEW|6.3|2020-02-02 02:02:02.0"), + querySinkRows()); + } + + @TestTemplate + public void testRejectedMutationFailsTheJob(TestContainer container) + throws IOException, InterruptedException { + Container.ExecResult execResult = container.executeJob("/fake_to_graphql_rejected.conf"); + Assertions.assertNotEquals(0, execResult.getExitCode()); + String output = jobOutputAndServerLogs(container, execResult); + Assertions.assertTrue(output.contains("The GraphQL server returned errors")); + Assertions.assertTrue(output.contains("'insert_missing_table' not found")); + } + + @TestTemplate + public void testRejectedQueryFailsTheJob(TestContainer container) + throws IOException, InterruptedException { + Container.ExecResult execResult = + container.executeJob("/graphql_rejected_query_to_assert.conf"); + Assertions.assertNotEquals(0, execResult.getExitCode()); + String output = jobOutputAndServerLogs(container, execResult); + Assertions.assertTrue(output.contains("The GraphQL server returned errors")); + // This job sets no variables, so it also checks that they are sent as an object. + Assertions.assertTrue(output.contains("'missing_table' not found")); + } + + /** + * On Flink the client prints only one failure cause, and when the failed task also loses an + * operator event, that one can be reported instead of the connector error. The connector error + * is always in the server logs. + */ + private static String jobOutputAndServerLogs( + TestContainer container, Container.ExecResult execResult) { + return execResult.getStdout() + execResult.getStderr() + container.getServerLogs(); + } + + private void trackTable(String table) throws IOException { + URL url = + new URL( + "http://" + + genericContainer.getHost() + + ":" + + genericContainer.getMappedPort(8080) + + "/v1/metadata"); + HttpURLConnection connection = (HttpURLConnection) url.openConnection(); + connection.setRequestMethod("POST"); + connection.setRequestProperty("Content-Type", "application/json"); + connection.setDoOutput(true); + String body = + "{\"type\":\"pg_track_table\",\"args\":{\"source\":\"default\"," + + "\"table\":{\"schema\":\"public\",\"name\":\"" + + table + + "\"}}}"; + try (OutputStream out = connection.getOutputStream()) { + out.write(body.getBytes(StandardCharsets.UTF_8)); + } + int code = connection.getResponseCode(); + connection.disconnect(); + Assertions.assertEquals(200, code, "tracking table " + table); + } + + private void executeSql(String sql) { + try (Connection connection = getJdbcConnection(); + Statement statement = connection.createStatement()) { + statement.execute(sql); + } catch (SQLException e) { + throw new RuntimeException("Failed to execute " + sql, e); + } + } + + private List<String> querySinkRows() { + List<String> rows = new ArrayList<>(); + try (Connection connection = getJdbcConnection(); + Statement statement = connection.createStatement(); + ResultSet resultSet = + statement.executeQuery( + "SELECT id, val_bool, val_string, val_decimal," + + " val_unixtime_micros FROM sink ORDER BY id")) { + while (resultSet.next()) { + rows.add( + resultSet.getInt(1) + + "|" + + resultSet.getBoolean(2) + + "|" + + resultSet.getString(3) + + "|" + + resultSet.getBigDecimal(4) + + "|" + + resultSet.getTimestamp(5)); + } + } catch (SQLException e) { + throw new RuntimeException("Failed to read the sink table", e); + } + return rows; } private Connection getJdbcConnection() throws SQLException { diff --git a/seatunnel-e2e/seatunnel-connector-v2-e2e/connector-graphql-e2e/src/test/resources/fake_to_graphql.conf b/seatunnel-e2e/seatunnel-connector-v2-e2e/connector-graphql-e2e/src/test/resources/fake_to_graphql.conf index 3293e557be..40e5b5bb86 100644 --- a/seatunnel-e2e/seatunnel-connector-v2-e2e/connector-graphql-e2e/src/test/resources/fake_to_graphql.conf +++ b/seatunnel-e2e/seatunnel-connector-v2-e2e/connector-graphql-e2e/src/test/resources/fake_to_graphql.conf @@ -89,7 +89,7 @@ sink { $val_int32: Int! $val_int64: bigint! $val_float: Float! - $val_double: Float! + $val_double: float8! $val_decimal: numeric! $val_string: String! $val_unixtime_micros: timestamp! diff --git a/seatunnel-e2e/seatunnel-connector-v2-e2e/connector-graphql-e2e/src/test/resources/fake_to_graphql_rejected.conf b/seatunnel-e2e/seatunnel-connector-v2-e2e/connector-graphql-e2e/src/test/resources/fake_to_graphql_rejected.conf new file mode 100644 index 0000000000..9735d9defc --- /dev/null +++ b/seatunnel-e2e/seatunnel-connector-v2-e2e/connector-graphql-e2e/src/test/resources/fake_to_graphql_rejected.conf @@ -0,0 +1,58 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# +# Writes one row with a mutation the GraphQL server rejects. The server answers HTTP 200 with +# an errors array, and the job must fail instead of dropping the row. + +env { + parallelism = 1 + job.mode = "BATCH" + # Zeta restores a failed pipeline 3 times by default. The rejection fails every attempt, so + # restoring only delays the result. + job.retry.times = 0 +} + +source { + FakeSource { + plugin_output = "fake" + schema = { + fields { + id = int + } + } + rows = [ + { + kind = INSERT + fields = [1] + } + ] + } +} + +sink { + GraphQL { + url = "http://graphql:8080/v1/graphql" + plugin_input = "fake" + query = """ + mutation Rejected($id: Int!) { + insert_missing_table(objects: {id: $id}) { + affected_rows + } + } + """ + variables = {} + } +} diff --git a/seatunnel-e2e/seatunnel-connector-v2-e2e/connector-graphql-e2e/src/test/resources/graphql_rejected_query_to_assert.conf b/seatunnel-e2e/seatunnel-connector-v2-e2e/connector-graphql-e2e/src/test/resources/graphql_rejected_query_to_assert.conf new file mode 100644 index 0000000000..b4474c50bf --- /dev/null +++ b/seatunnel-e2e/seatunnel-connector-v2-e2e/connector-graphql-e2e/src/test/resources/graphql_rejected_query_to_assert.conf @@ -0,0 +1,64 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# +# Reads a field the GraphQL server does not have. The server answers HTTP 200 with an errors +# array, and the job must fail instead of reading no rows. + +env { + parallelism = 1 + job.mode = "BATCH" + # Report the first failure. After a Flink restart, the single split enumerator fails with + # its own error, which replaces the GraphQL error in the job result. + execution.restart.strategy = "no" + # Zeta restores a failed pipeline 3 times by default. The rejection fails every attempt, so + # restoring only delays the result. + job.retry.times = 0 +} + +source { + GraphQL { + plugin_output = "http" + url = "http://graphql:8080/v1/graphql" + format = "json" + content_field = "$.data.missing_table" + query = """ + query Rejected { + missing_table { + id + } + } + """ + schema = { + fields { + id = "int" + } + } + } +} + +sink { + Assert { + plugin_input = http + rules { + row_rules = [ + { + rule_type = MIN_ROW + rule_value = 0 + } + ] + } + } +} diff --git a/seatunnel-e2e/seatunnel-connector-v2-e2e/connector-graphql-e2e/src/test/resources/graphql_to_assert.conf b/seatunnel-e2e/seatunnel-connector-v2-e2e/connector-graphql-e2e/src/test/resources/graphql_to_assert.conf index fc0f69ee3b..8bd047af95 100644 --- a/seatunnel-e2e/seatunnel-connector-v2-e2e/connector-graphql-e2e/src/test/resources/graphql_to_assert.conf +++ b/seatunnel-e2e/seatunnel-connector-v2-e2e/connector-graphql-e2e/src/test/resources/graphql_to_assert.conf @@ -57,6 +57,16 @@ sink { Assert { plugin_input = http rules { + row_rules = [ + { + rule_type = MAX_ROW + rule_value = 2 + }, + { + rule_type = MIN_ROW + rule_value = 2 + } + ], field_rules = [ { field_name = id
