yzeng1618 commented on PR #9273:
URL: https://github.com/apache/seatunnel/pull/9273#issuecomment-2861673340
> > Complexity in Scenarios with Numerous Fields
> > Complexity in Multi-Table/Full Database Synchronization Scenarios
>
> In fact, seatunnel supports using one transform to solve the case
conversion problem for all tables and all fields. For example
>
> ```hocon
> FieldRename {
> plugin_input = "transform1"
> plugin_output = "transform2"
>
> table_match_regex = ".*"
> convert_case = "LOWER"
> }
> ```
>
> Please refer
https://seatunnel.apache.org/docs/2.3.10/transform-v2/transform-multi-table
>
> > Enables automatic table creation functionality when case insensitivity
parameters are set
>
> This shouldn't be a problem since the field case is according to upstream.
Thank you for suggesting the use of transform-multi-table. We acknowledge
that we hadn't fully considered this solution before, and it is indeed an
excellent option.
However, we still believe that adding a case sensitive parameter to the
Doris connector has its unique value and necessity:
1. Simplified User Configuration Experience : While transform-multi-table is
powerful, providing a parameter directly in the connector allows users to solve
the issue with just one line of configuration, without needing to learn and
configure additional transform plugins, significantly lowering the usage
barrier.
2. Performance Optimization : Handling case sensitivity at the connector
level is more efficient than at the transform level, as it avoids the overhead
of data transfer and processing between different plugins. This optimization is
particularly noticeable when processing large volumes of data.
3. Consistency Across Connectors : As you mentioned, the Iceberg connector
already provides similar parameters. Adding the same functionality to the Doris
connector maintains consistency across SeaTunnel connectors, providing users
with a consistent experience when switching between different connectors.
Therefore, for simple scenarios, using connector parameters would be ideal;
for more complex transformation requirements, the transform-multi-table
functionality would be more appropriate. This approach provides users with
maximum flexibility while maintaining configuration simplicity. We kindly
request your evaluation of the necessity of adding the case-sensitive parameter
to the Doris connector.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
