Re: Is it possible to change 'connector.startup-mode' option in the flink job
Hi Jingsong, Cool, Thanks for your reply. Best wishes. From: Jingsong Li Sent: Tuesday, May 19, 2020 10:46 To: Thomas Huang Cc: Flink Subject: Re: Is it possible to change 'connector.startup-mode' option in the flink job Hi Thomas, Good to hear from you. This is a very common problem. In 1.11, we have two FLIP to solve your problem. [1][2] You can take a look. I think dynamic table options (table hints) is enough for your requirement. [1]https://cwiki.apache.org/confluence/display/FLINK/FLIP-113%3A+Supports+Dynamic+Table+Options+for+Flink+SQL [2]https://cwiki.apache.org/confluence/display/FLINK/FLIP-110%3A+Support+LIKE+clause+in+CREATE+TABLE Best, Jingsong Lee On Tue, May 19, 2020 at 10:37 AM Thomas Huang mailto:lyang...@hotmail.com>> wrote: Hi guys, I'm using hive to store kafka topic metadata as follows:: CREATE TABLE orders ( user_idBIGINT, productSTRING, order_time TIMESTAMP(3), WATERMARK FOR order_time AS order_time - '5' SECONDS ) WITH ( 'connector.type' = 'kafka', 'connector.version' = 'universal', 'connector.topic'= 'orders', 'connector.startup-mode' = 'earliest-offset', 'connector.properties.bootstrap.servers' = 'localhost:9092', 'format.type' = 'json' ); However, sometimes for incident handle, I have to set 'connector.startup-mode' to a specific timestamp, but I don't want to change the option setting in the hive table. Is there any way (api or ddl) to change 'connector.startup-mode' option in the flink jobs, but not impact the option stores in hive? Best Wishes. -- Best, Jingsong Lee
Re: Is it possible to change 'connector.startup-mode' option in the flink job
Hi Thomas, Good to hear from you. This is a very common problem. In 1.11, we have two FLIP to solve your problem. [1][2] You can take a look. I think dynamic table options (table hints) is enough for your requirement. [1] https://cwiki.apache.org/confluence/display/FLINK/FLIP-113%3A+Supports+Dynamic+Table+Options+for+Flink+SQL [2] https://cwiki.apache.org/confluence/display/FLINK/FLIP-110%3A+Support+LIKE+clause+in+CREATE+TABLE Best, Jingsong Lee On Tue, May 19, 2020 at 10:37 AM Thomas Huang wrote: > Hi guys, > > I'm using hive to store kafka topic metadata as follows:: > > CREATE TABLE orders ( > user_idBIGINT, > productSTRING, > order_time TIMESTAMP(3), > WATERMARK FOR order_time AS order_time - '5' SECONDS) WITH ( > 'connector.type' = 'kafka', > 'connector.version' = 'universal', > 'connector.topic'= 'orders', > 'connector.startup-mode' = 'earliest-offset', > 'connector.properties.bootstrap.servers' = 'localhost:9092', > 'format.type' = 'json' ); > > However, sometimes for incident handle, I have to set > 'connector.startup-mode' to a specific timestamp, but I don't want to > change the option setting in the hive table. Is there any way (api or ddl) > to change 'connector.startup-mode' option in the flink jobs, but not > impact the option stores in hive? > > > Best Wishes. > -- Best, Jingsong Lee
Is it possible to change 'connector.startup-mode' option in the flink job
Hi guys, I'm using hive to store kafka topic metadata as follows:: CREATE TABLE orders ( user_idBIGINT, productSTRING, order_time TIMESTAMP(3), WATERMARK FOR order_time AS order_time - '5' SECONDS ) WITH ( 'connector.type' = 'kafka', 'connector.version' = 'universal', 'connector.topic'= 'orders', 'connector.startup-mode' = 'earliest-offset', 'connector.properties.bootstrap.servers' = 'localhost:9092', 'format.type' = 'json' ); However, sometimes for incident handle, I have to set 'connector.startup-mode' to a specific timestamp, but I don't want to change the option setting in the hive table. Is there any way (api or ddl) to change 'connector.startup-mode' option in the flink jobs, but not impact the option stores in hive? Best Wishes.
