At present, it is normal to report an error. To support parameters and default values in the where condition, the engine side needs to be extended to process all el expressions containing default values in the config file. Currently, only the sink side is supported. I created an issue for this https://github.com/apache/seatunnel/issues/7535.
Arshad, Mohammad <[email protected]> 于2024年8月30日周五 19:29写道： > Excellent, this is exactly what I needed. > I created a task with the following where_condition: > where service_name=${serviceParam:'HDFS'} > It fails on the engine side with the error: "check the manual that > corresponds to your MySQL server version for the right syntax to use near > '{serviceParam:'HDFS'}'" > I will investigate this; it might be a bug in the execution logic. > It's good that the option to customize the where condition is available. > Thanks, Jia Fan, for your help. > > Thanks & Regards > Mohammad Arshad > > > From: Jia Fan <[email protected]> > Date: Friday, 30 August 2024 at 3:09 PM > To: [email protected] <[email protected]> > Subject: Re: How to create a seatunnel job in seatunnel-web with > conditional query > PS: we supported `where_condition` in jdbc. Please refer > > https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fseatunnel.apache.org%2Fdocs%2F2.3.7%2Fconnector-v2%2Fsource%2FJdbc&data=05%7C02%7Cmohammad.arshad%40visa.com%7C87f608946d6145f3bfe408dcc8d7a07c%7C38305e12e15d4ee888b9c4db1c477d76%7C0%7C0%7C638606075656686744%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=PWiWdc2Z7Z%2FB3%2BUHp5RgRUd5RHV1ZzKpV1gooYE4RBY%3D&reserved=0 > <https://seatunnel.apache.org/docs/2.3.7/connector-v2/source/Jdbc> > > Jia Fan <[email protected]> 于2024年8月30日周五 14:49写道： > > > I have two ideas. > > 1. Why not create a new virtual table type named `SQL`? > > We can define `SQL` not only containing the `Where` clause, but also > > can support the `Join` statement. > > 2. Add a new option in the source step only in the SeaTunnel Web named > > `Where` clause. > > > > Arshad, Mohammad <[email protected]> 于2024年8月30日周五 > 14:41写道： > > > >> > >> > https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fseatunnel-web%2Fblob%2F30ed42fdac21a0edc6a9cb06ed00f81967b64717%2Fseatunnel-server%2Fseatunnel-app%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fseatunnel%2Fapp%2Fthirdparty%2Fdatasource%2Fimpl%2FBaseJdbcDataSourceConfigSwitcher.java%23L217&data=05%7C02%7Cmohammad.arshad%40visa.com%7C87f608946d6145f3bfe408dcc8d7a07c%7C38305e12e15d4ee888b9c4db1c477d76%7C0%7C0%7C638606075656696367%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=CskdX3OG72%2F%2Fmjcv9yHNMu625cuMbnothjXEoQMoIAQ%3D&reserved=0 > < > https://github.com/apache/seatunnel-web/blob/30ed42fdac21a0edc6a9cb06ed00f81967b64717/seatunnel-server/seatunnel-app/src/main/java/org/apache/seatunnel/app/thirdparty/datasource/impl/BaseJdbcDataSourceConfigSwitcher.java#L217 > > > >> Above generateSql method generates the query for all JDBC data sources. > >> It does not support adding a WHERE clause because through UI there is > not > >> enough information collected to generate WHERE clause. > >> Then, How does seatunnel-web support the WHERE clause for JDBC data > >> sources? > >> > >> > >> Thanks & Regards > >> Mohammad Arshad > >> > >> > >> From: Jia Fan <[email protected]> > >> Date: Friday, 30 August 2024 at 8:00 AM > >> To: [email protected] <[email protected]> > >> Subject: Re: How to create a seatunnel job in seatunnel-web with > >> conditional query > >> Hi Mohammad: > >> We can support default value in SeaTunnel Web just like we did in > >> > >> > https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fseatunnel.apache.org%2Fdocs%2Fconcept%2Fsink-options-placeholders%23placeholder&data=05%7C02%7Cmohammad.arshad%40visa.com%7C87f608946d6145f3bfe408dcc8d7a07c%7C38305e12e15d4ee888b9c4db1c477d76%7C0%7C0%7C638606075656703311%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=eIX3iOZodUPyAz30WdN2P9IuUZ7I3F%2FEfTXSyWiv7v0%3D&reserved=0 > < > https://seatunnel.apache.org/docs/concept/sink-options-placeholders#placeholder > > > >> < > >> > https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fseatunnel.apache.org%2Fdocs%2Fconcept%2Fsink-options-placeholders%23placeholder&data=05%7C02%7Cmohammad.arshad%40visa.com%7C87f608946d6145f3bfe408dcc8d7a07c%7C38305e12e15d4ee888b9c4db1c477d76%7C0%7C0%7C638606075656708291%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=rYFR%2B3pcdpUD4K30qsJn61%2FN6W9pyzayi%2FcXMiaauVI%3D&reserved=0 > < > https://seatunnel.apache.org/docs/concept/sink-options-placeholders#placeholder > > > >> > > >> eg: > >> SELECT name, age FROM test.test_table WHERE age > ${ageParam:25}. Users > >> should define the default value when creating a source. > >> WDYT? > >> > >> Mohammad Arshad <[email protected]> 于2024年8月30日周五 00:22写道： > >> > >> > Thank you Jia Fan for your reply > >> > > >> > Yes, the variable substitution feature is available in SeaTunnel > >> Engine. If > >> > the job were created manually, there would be no issues. However, the > >> > problem arises when a job is created through SeaTunnel Web. > >> > > >> > In SeaTunnel Web, how can I create a source step with a query like > >> "SELECT > >> > name, age FROM test.test_table WHERE age > ${ageParam}"? Do you think > >> this > >> > is possible through the SeaTunnel Web? > >> > > >> > Thanks & Regards > >> > Mohammad Arshad > >> > > >> > > >> > > >> > On Thu, Aug 29, 2024 at 7:32 AM Jia Fan <[email protected]> wrote: > >> > > >> > > One way is to use our config variable feature in > >> > > > >> > > > >> > > >> > https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fseatunnel.apache.org%2Fdocs%2F2.3.7%2Fconcept%2Fconfig%23config-variable-substitution&data=05%7C02%7Cmohammad.arshad%40visa.com%7C87f608946d6145f3bfe408dcc8d7a07c%7C38305e12e15d4ee888b9c4db1c477d76%7C0%7C0%7C638606075656712700%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=cjUryK1ikdbSG%2FUlAA2gQOXd91hR1pITx01fYtBkzlY%3D&reserved=0 > < > https://seatunnel.apache.org/docs/2.3.7/concept/config#config-variable-substitution > > > >> < > >> > https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fseatunnel.apache.org%2Fdocs%2F2.3.7%2Fconcept%2Fconfig%23config-variable-substitution&data=05%7C02%7Cmohammad.arshad%40visa.com%7C87f608946d6145f3bfe408dcc8d7a07c%7C38305e12e15d4ee888b9c4db1c477d76%7C0%7C0%7C638606075656717067%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=%2BexZ6s1kl7HHgqnMQzZ0hgQlGNiomXNSeTzSKCjOl8I%3D&reserved=0 > < > https://seatunnel.apache.org/docs/2.3.7/concept/config#config-variable-substitution > > > >> > > >> > > . > >> > > > >> > > Arshad, Mohammad <[email protected]> 于2024年8月28日周三 > >> > 19:27写道： > >> > > > >> > > > Hi All, > >> > > > I'm interested in creating a job that includes a conditional > query, > >> yet > >> > > it > >> > > > appears that this option is currently unavailable in > SeaTunnel-Web. > >> > > > To illustrate, suppose we have a test_table with fields name and > >> age. > >> > My > >> > > > objective is to transfer all records from test_table to another > >> table, > >> > > but > >> > > > only for those entries where age is greater than 25. > >> > > > > >> > > > At present, SeaTunnel-Web enables the creation of jobs with > queries > >> > such > >> > > > as “SELECT `name`, `age` FROM `test`.`test_table`”. However, it > does > >> > not > >> > > > provide an option for creating queries like > >> > > > “SELECT `name`, `age` FROM `test`.`test_table` where `age` > 25” > or > >> > > > “SELECT `name`, `age` FROM `test`.`test_table` where `age` >$ > >> > > {ageParam}” > >> > > > > >> > > > Could there be something I'm overlooking? Are there any > alternative > >> > > > methods to accomplish this? > >> > > > > >> > > > An attempt is made to add support for this feature at execution > >> time as > >> > > > part of the pull request at > >> > > > > >> > https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fseatunnel-web%2Fpull%2F193&data=05%7C02%7Cmohammad.arshad%40visa.com%7C87f608946d6145f3bfe408dcc8d7a07c%7C38305e12e15d4ee888b9c4db1c477d76%7C0%7C0%7C638606075656721392%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=isiWUB7k4v6y4DEkHCNbeX7Ba%2FNz3sPYsste%2BLdmC%2Bw%3D&reserved=0 > <https://github.com/apache/seatunnel-web/pull/193> > >> < > https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fseatunnel-web%2Fpull%2F193&data=05%7C02%7Cmohammad.arshad%40visa.com%7C87f608946d6145f3bfe408dcc8d7a07c%7C38305e12e15d4ee888b9c4db1c477d76%7C0%7C0%7C638606075656725653%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=hFJxIoBIWEJ3e3hSyq8RpAVzt4TbH49kOq6Gat%2BxXTQ%3D&reserved=0 > <https://github.com/apache/seatunnel-web/pull/193>>. > >> > > > However, it might be more efficient if the conditional query could > >> be > >> > > > supported directly at the job creation stage. Only some parameters > >> to > >> > be > >> > > > changed at execution time, not the whole query. > >> > > > > >> > > > Does anyone have any ideas or suggestions on how to implement this > >> > > feature? > >> > > > > >> > > > Thanks & Regards > >> > > > Mohammad Arshad > >> > > > > >> > > > > >> > > > >> > > >> > > >
