Excellent, this is exactly what I needed.
I created a task with the following where_condition:
where service_name=${serviceParam:'HDFS'}
It fails on the engine side with the error: "check the manual that corresponds
to your MySQL server version for the right syntax to use near
'{serviceParam:'HDFS'}'"
I will investigate this; it might be a bug in the execution logic.
It's good that the option to customize the where condition is available.
Thanks, Jia Fan, for your help.
Thanks & Regards
Mohammad Arshad
From: Jia Fan <[email protected]>
Date: Friday, 30 August 2024 at 3:09 PM
To: [email protected] <[email protected]>
Subject: Re: How to create a seatunnel job in seatunnel-web with conditional
query
PS: we supported `where_condition` in jdbc. Please refer
https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fseatunnel.apache.org%2Fdocs%2F2.3.7%2Fconnector-v2%2Fsource%2FJdbc&data=05%7C02%7Cmohammad.arshad%40visa.com%7C87f608946d6145f3bfe408dcc8d7a07c%7C38305e12e15d4ee888b9c4db1c477d76%7C0%7C0%7C638606075656686744%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=PWiWdc2Z7Z%2FB3%2BUHp5RgRUd5RHV1ZzKpV1gooYE4RBY%3D&reserved=0<https://seatunnel.apache.org/docs/2.3.7/connector-v2/source/Jdbc>
Jia Fan <[email protected]> 于2024年8月30日周五 14:49写道：
> I have two ideas.
> 1. Why not create a new virtual table type named `SQL`?
> We can define `SQL` not only containing the `Where` clause, but also
> can support the `Join` statement.
> 2. Add a new option in the source step only in the SeaTunnel Web named
> `Where` clause.
>
> Arshad, Mohammad <[email protected]> 于2024年8月30日周五 14:41写道：
>
>>
>> https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fseatunnel-web%2Fblob%2F30ed42fdac21a0edc6a9cb06ed00f81967b64717%2Fseatunnel-server%2Fseatunnel-app%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fseatunnel%2Fapp%2Fthirdparty%2Fdatasource%2Fimpl%2FBaseJdbcDataSourceConfigSwitcher.java%23L217&data=05%7C02%7Cmohammad.arshad%40visa.com%7C87f608946d6145f3bfe408dcc8d7a07c%7C38305e12e15d4ee888b9c4db1c477d76%7C0%7C0%7C638606075656696367%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=CskdX3OG72%2F%2Fmjcv9yHNMu625cuMbnothjXEoQMoIAQ%3D&reserved=0<https://github.com/apache/seatunnel-web/blob/30ed42fdac21a0edc6a9cb06ed00f81967b64717/seatunnel-server/seatunnel-app/src/main/java/org/apache/seatunnel/app/thirdparty/datasource/impl/BaseJdbcDataSourceConfigSwitcher.java#L217>
>> Above generateSql method generates the query for all JDBC data sources.
>> It does not support adding a WHERE clause because through UI there is not
>> enough information collected to generate WHERE clause.
>> Then, How does seatunnel-web support the WHERE clause for JDBC data
>> sources?
>>
>>
>> Thanks & Regards
>> Mohammad Arshad
>>
>>
>> From: Jia Fan <[email protected]>
>> Date: Friday, 30 August 2024 at 8:00 AM
>> To: [email protected] <[email protected]>
>> Subject: Re: How to create a seatunnel job in seatunnel-web with
>> conditional query
>> Hi Mohammad:
>> We can support default value in SeaTunnel Web just like we did in
>>
>> https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fseatunnel.apache.org%2Fdocs%2Fconcept%2Fsink-options-placeholders%23placeholder&data=05%7C02%7Cmohammad.arshad%40visa.com%7C87f608946d6145f3bfe408dcc8d7a07c%7C38305e12e15d4ee888b9c4db1c477d76%7C0%7C0%7C638606075656703311%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=eIX3iOZodUPyAz30WdN2P9IuUZ7I3F%2FEfTXSyWiv7v0%3D&reserved=0<https://seatunnel.apache.org/docs/concept/sink-options-placeholders#placeholder>
>> <
>> https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fseatunnel.apache.org%2Fdocs%2Fconcept%2Fsink-options-placeholders%23placeholder&data=05%7C02%7Cmohammad.arshad%40visa.com%7C87f608946d6145f3bfe408dcc8d7a07c%7C38305e12e15d4ee888b9c4db1c477d76%7C0%7C0%7C638606075656708291%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=rYFR%2B3pcdpUD4K30qsJn61%2FN6W9pyzayi%2FcXMiaauVI%3D&reserved=0<https://seatunnel.apache.org/docs/concept/sink-options-placeholders#placeholder>
>> >
>> eg:
>> SELECT name, age FROM test.test_table WHERE age > ${ageParam:25}. Users
>> should define the default value when creating a source.
>> WDYT?
>>
>> Mohammad Arshad <[email protected]> 于2024年8月30日周五 00:22写道：
>>
>> > Thank you Jia Fan for your reply
>> >
>> > Yes, the variable substitution feature is available in SeaTunnel
>> Engine. If
>> > the job were created manually, there would be no issues. However, the
>> > problem arises when a job is created through SeaTunnel Web.
>> >
>> > In SeaTunnel Web, how can I create a source step with a query like
>> "SELECT
>> > name, age FROM test.test_table WHERE age > ${ageParam}"? Do you think
>> this
>> > is possible through the SeaTunnel Web?
>> >
>> > Thanks & Regards
>> > Mohammad Arshad
>> >
>> >
>> >
>> > On Thu, Aug 29, 2024 at 7:32 AM Jia Fan <[email protected]> wrote:
>> >
>> > > One way is to use our config variable feature in
>> > >
>> > >
>> >
>> https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fseatunnel.apache.org%2Fdocs%2F2.3.7%2Fconcept%2Fconfig%23config-variable-substitution&data=05%7C02%7Cmohammad.arshad%40visa.com%7C87f608946d6145f3bfe408dcc8d7a07c%7C38305e12e15d4ee888b9c4db1c477d76%7C0%7C0%7C638606075656712700%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=cjUryK1ikdbSG%2FUlAA2gQOXd91hR1pITx01fYtBkzlY%3D&reserved=0<https://seatunnel.apache.org/docs/2.3.7/concept/config#config-variable-substitution>
>> <
>> https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fseatunnel.apache.org%2Fdocs%2F2.3.7%2Fconcept%2Fconfig%23config-variable-substitution&data=05%7C02%7Cmohammad.arshad%40visa.com%7C87f608946d6145f3bfe408dcc8d7a07c%7C38305e12e15d4ee888b9c4db1c477d76%7C0%7C0%7C638606075656717067%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=%2BexZ6s1kl7HHgqnMQzZ0hgQlGNiomXNSeTzSKCjOl8I%3D&reserved=0<https://seatunnel.apache.org/docs/2.3.7/concept/config#config-variable-substitution>
>> >
>> > > .
>> > >
>> > > Arshad, Mohammad <[email protected]> 于2024年8月28日周三
>> > 19:27写道：
>> > >
>> > > > Hi All,
>> > > > I'm interested in creating a job that includes a conditional query,
>> yet
>> > > it
>> > > > appears that this option is currently unavailable in SeaTunnel-Web.
>> > > > To illustrate, suppose we have a test_table with fields name and
>> age.
>> > My
>> > > > objective is to transfer all records from test_table to another
>> table,
>> > > but
>> > > > only for those entries where age is greater than 25.
>> > > >
>> > > > At present, SeaTunnel-Web enables the creation of jobs with queries
>> > such
>> > > > as “SELECT `name`, `age` FROM `test`.`test_table`”. However, it does
>> > not
>> > > > provide an option for creating queries like
>> > > > “SELECT `name`, `age` FROM `test`.`test_table` where `age` > 25” or
>> > > > “SELECT `name`, `age` FROM `test`.`test_table` where `age` >$
>> > > {ageParam}”
>> > > >
>> > > > Could there be something I'm overlooking? Are there any alternative
>> > > > methods to accomplish this?
>> > > >
>> > > > An attempt is made to add support for this feature at execution
>> time as
>> > > > part of the pull request at
>> > > >
>> https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fseatunnel-web%2Fpull%2F193&data=05%7C02%7Cmohammad.arshad%40visa.com%7C87f608946d6145f3bfe408dcc8d7a07c%7C38305e12e15d4ee888b9c4db1c477d76%7C0%7C0%7C638606075656721392%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=isiWUB7k4v6y4DEkHCNbeX7Ba%2FNz3sPYsste%2BLdmC%2Bw%3D&reserved=0<https://github.com/apache/seatunnel-web/pull/193>
>> <https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fseatunnel-web%2Fpull%2F193&data=05%7C02%7Cmohammad.arshad%40visa.com%7C87f608946d6145f3bfe408dcc8d7a07c%7C38305e12e15d4ee888b9c4db1c477d76%7C0%7C0%7C638606075656725653%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=hFJxIoBIWEJ3e3hSyq8RpAVzt4TbH49kOq6Gat%2BxXTQ%3D&reserved=0<https://github.com/apache/seatunnel-web/pull/193>>.
>> > > > However, it might be more efficient if the conditional query could
>> be
>> > > > supported directly at the job creation stage. Only some parameters
>> to
>> > be
>> > > > changed at execution time, not the whole query.
>> > > >
>> > > > Does anyone have any ideas or suggestions on how to implement this
>> > > feature?
>> > > >
>> > > > Thanks & Regards
>> > > > Mohammad Arshad
>> > > >
>> > > >
>> > >
>> >
>>
>
