Hisoka-X commented on code in PR #9138:
URL: https://github.com/apache/seatunnel/pull/9138#discussion_r2043382793
##########
docs/en/connector-v2/source/Http.md:
##########
@@ -44,33 +44,37 @@ They can be downloaded via install-plugin.sh or from the
Maven central repositor
## Source Options
-| Name | Type | Required | Default | Description
|
-|-----------------------------|---------|----------|---------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
-| url | String | Yes | - | Http request
url.
|
-| schema | Config | No | - | Http and
seatunnel data structure mapping
|
-| schema.fields | Config | No | - | The schema
fields of upstream data
|
-| json_field | Config | No | - | This parameter
helps you configure the schema,so this parameter must be used with schema.
|
-| pageing | Config | No | - | This parameter
is used for paging queries
|
-| pageing.page_field | String | No | - | This parameter
is used to specify the page field name in the request parameter
|
-| pageing.total_page_size | Int | No | - | This parameter
is used to control the total number of pages
|
-| pageing.batch_size | Int | No | - | The batch size
returned per request is used to determine whether to continue when the total
number of pages is unknown
|
-| pageing.start_page_number | Int | No | 1 | Specify the
page number from which synchronization starts
|
-| content_json | String | No | - | This parameter
can get some json data.If you only need the data in the 'book' section,
configure `content_field = "$.store.book.*"`.
|
-| format | String | No | text | The format of
upstream data, now only support `json` `text`, default `text`.
|
-| method | String | No | get | Http request
method, only supports GET, POST method.
|
-| headers | Map | No | - | Http headers.
|
-| params | Map | No | - | Http params.
|
-| body | String | No | - | Http body,the
program will automatically add http header application/json,body is jsonbody.
|
-| poll_interval_millis | Int | No | - | Request http
api interval(millis) in stream mode.
|
-| retry | Int | No | - | The max retry
times if request http return to `IOException`.
|
-| retry_backoff_multiplier_ms | Int | No | 100 | The
retry-backoff times(millis) multiplier if request http failed.
|
-| retry_backoff_max_ms | Int | No | 10000 | The maximum
retry-backoff times(millis) if request http failed
|
-| enable_multi_lines | Boolean | No | false |
|
-| connect_timeout_ms | Int | No | 12000 | Connection
timeout setting, default 12s.
|
-| socket_timeout_ms | Int | No | 60000 | Socket timeout
setting, default 60s.
|
-| common-options | | No | - | Source plugin
common parameters, please refer to [Source Common
Options](../source-common-options.md) for details
|
-| keep_params_as_form | Boolean | No | false |
Whether the params are submitted according to the form, used for compatibility
with legacy behaviors. When true, the value of the params parameter is
submitted through the form. |
-| keep_page_param_as_http_param | Boolean | No | false
| Whether to set the paging parameters to params. For compatibility with
legacy behaviors.
|
+| Name | Type | Required | Default |
Description
|
+|-------------------------------|---------|----------|----------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| url | String | Yes | - |
Http request url.
|
+| schema | Config | No | - |
Http and seatunnel data structure mapping
|
+| schema.fields | Config | No | - |
The schema fields of upstream data
|
+| json_field | Config | No | - |
This parameter helps you configure the schema,so this parameter must be used
with schema.
|
+| pageing | Config | No | - |
This parameter is used for paging queries
|
+| pageing.page_field | String | No | - |
This parameter is used to specify the page field name in the request parameter
|
+| pageing.total_page_size | Int | No | - |
This parameter is used to control the total number of pages
|
+| pageing.batch_size | Int | No | - |
The batch size returned per request is used to determine whether to continue
when the total number of pages is unknown
|
+| pageing.start_page_number | Int | No | 1 |
Specify the page number from which synchronization starts
|
+| pageing.page_type | String | No | PageNumberPagination |
this parameter is used to specify the page type ,or PageNumberPagination if not
set, only support `PageNumberPagination` and `CursorPagination`.
|
Review Comment:
```suggestion
| pageing.page_type | String | No | PageNumber | this
parameter is used to specify the page type ,or PageNumber if not set, only
support `PageNumber` and `Cursor`. |
```
##########
docs/en/connector-v2/source/Http.md:
##########
@@ -393,6 +404,42 @@ source {
}
+```
+
+#### 2. CursorPagination
Review Comment:
```suggestion
#### Cursor
```
##########
docs/en/connector-v2/source/Http.md:
##########
@@ -362,10 +366,15 @@ source {
- See this link for task configuration
[http_jsonpath_to_assert.conf](../../../../seatunnel-e2e/seatunnel-connector-v2-e2e/connector-http-e2e/src/test/resources/http_jsonpath_to_assert.conf).
### pageing
-When you need to concatenate page param in the URL,then add params.
+The current supported pagination type are `PageNumberPagination` and
`CursorPagination`.
Review Comment:
```suggestion
The current supported pagination type are `PageNumber` and `Cursor`.
```
##########
docs/en/connector-v2/source/Http.md:
##########
@@ -362,10 +366,15 @@ source {
- See this link for task configuration
[http_jsonpath_to_assert.conf](../../../../seatunnel-e2e/seatunnel-connector-v2-e2e/connector-http-e2e/src/test/resources/http_jsonpath_to_assert.conf).
### pageing
-When you need to concatenate page param in the URL,then add params.
+The current supported pagination type are `PageNumberPagination` and
`CursorPagination`.
+if you need to use pagination, you need to configure `pageing`. the default
pagination type is `PageNumberPagination`.
+
+#### 1. PageNumberPagination
Review Comment:
```suggestion
#### PageNumber
```
##########
docs/en/connector-v2/source/Http.md:
##########
@@ -362,10 +366,15 @@ source {
- See this link for task configuration
[http_jsonpath_to_assert.conf](../../../../seatunnel-e2e/seatunnel-connector-v2-e2e/connector-http-e2e/src/test/resources/http_jsonpath_to_assert.conf).
### pageing
-When you need to concatenate page param in the URL,then add params.
+The current supported pagination type are `PageNumberPagination` and
`CursorPagination`.
+if you need to use pagination, you need to configure `pageing`. the default
pagination type is `PageNumberPagination`.
Review Comment:
```suggestion
if you need to use pagination, you need to configure `pageing`. the default
pagination type is `PageNumber`.
```
##########
seatunnel-connectors-v2/connector-http/connector-http-base/src/main/java/org/apache/seatunnel/connectors/seatunnel/http/config/HttpPaginationType.java:
##########
@@ -0,0 +1,49 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.seatunnel.connectors.seatunnel.http.config;
+
+public enum HttpPaginationType {
+ /** Page number based pagination */
+ PAGE_NUMBER(
+ "PageNumberPagination",
+ "traditional page-number-based pagination,uses a page number and
page size to retrieve a specific slice of data"),
+ /** Cursor based pagination */
+ CURSOR(
+ "CursorPagination",
+ "token-based cursor pagination,uses a cursor/token to fetch the
next set of data based on a specific point or marker");
Review Comment:
```suggestion
/** Page number based pagination */
PAGE_NUMBER(
"PageNumber",
"traditional page-number-based pagination,uses a page number and
page size to retrieve a specific slice of data"),
/** Cursor based pagination */
CURSOR(
"Cursor",
"token-based cursor pagination,uses a cursor/token to fetch the
next set of data based on a specific point or marker");
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
