Re: Bug report for reading Hive table as streaming source.

2024-04-15 Thread Xiaolong Wang
Reported. JIRA link: https://issues.apache.org/jira/browse/FLINK-35118 On Mon, Apr 15, 2024 at 12:03 PM Xiaolong Wang wrote: > Sure > > On Mon, Apr 1, 2024 at 9:28 AM yuxia wrote: > >> Thanks for reporting. Could you please help create a jira about it? >> &

Re: Bug report for reading Hive table as streaming source.

2024-04-14 Thread Xiaolong Wang
Sure On Mon, Apr 1, 2024 at 9:28 AM yuxia wrote: > Thanks for reporting. Could you please help create a jira about it? > > Best regards, > Yuxia > > - 原始邮件 - > 发件人: "Xiaolong Wang" > 收件人: "dev" > 发送时间: 星期四, 2024年 3 月 28日 下午 5:11

Re: Bug report for reading Hive table as streaming source.

2024-03-28 Thread Xiaolong Wang
I think it worth mentioning in the documentation of Hive read that it cannot read a table that has more than 32,767 partitions. On Thu, Mar 28, 2024 at 5:10 PM Xiaolong Wang wrote: > Found out the reason: > > It turned out that in Flink, it uses hive’s IMetaStoreClient to fetch >

Re: Bug report for reading Hive table as streaming source.

2024-03-28 Thread Xiaolong Wang
partition. On Tue, Mar 26, 2024 at 5:00 PM Xiaolong Wang wrote: > Hi, > > I found a weird bug when reading a Hive table as a streaming source. > > In summary, if the first partition is not time related, then the Hive > table cannot be read as a streaming source. > > e.

Bug report for reading Hive table as streaming source.

2024-03-26 Thread Xiaolong Wang
Hi, I found a weird bug when reading a Hive table as a streaming source. In summary, if the first partition is not time related, then the Hive table cannot be read as a streaming source. e.g. I've a Hive table in the definition of ``` CREATE TABLE article ( id BIGINT, edition STRING, dt

Re: Flink kinesis connector v4.2.0-1.18 has issue when consuming from EFO consumer

2024-02-29 Thread Xiaolong Wang
sdk.auth.credentials. > *DefaultCredentialsProvider* > .resolveCredentials(DefaultCredentialsProvider.java: > 128) ~[?:?] > > On Thu, 29 Feb 2024 at 02:24, Xiaolong Wang > wrote: > > > Sorry, I just attached a wrong file. Let me paste the error log: > > > > java.l

Re: Flink kinesis connector v4.2.0-1.18 has issue when consuming from EFO consumer

2024-02-28 Thread Xiaolong Wang
gt; Attached file does not have anything related to Kinesis or any errors. > > Best, > Aleksandr > > On Wed, 28 Feb 2024 at 02:28, Xiaolong Wang > wrote: > > > Hi, > > > > I used the flink-connector-kinesis (4.0.2-1.18) to consume from Kinesis. > > The

Flink kinesis connector v4.2.0-1.18 has issue when consuming from EFO consumer

2024-02-27 Thread Xiaolong Wang
Hi, I used the flink-connector-kinesis (4.0.2-1.18) to consume from Kinesis. The job can start but will fail within 1 hour. Detailed error log is attached. When I changed the version of the flink-connector-kinesis to `1.15.2` , everything settled. Any idea to fix it ? create table

Re: [Discuss] CRD for flink sql gateway in the flink k8s operator

2023-09-14 Thread Xiaolong Wang
e added a separate container to the > existing JobManager pod and started the sql gateway using the > "sql-gateway.sh start-foreground" command. > I haven't explored deploying the sql gateway as an independent deployment > yet, but that's something I'm considering after modi

Re: [Discuss] CRD for flink sql gateway in the flink k8s operator

2023-09-14 Thread Xiaolong Wang
Hi, I've experiment this feature on K8S recently, here is some of my trial: 1. Create a new kubernetes-jobmanager.sh script with the following content #!/usr/bin/env bash $FLINK_HOME/bin/sql-gateway.sh start $FLINK_HOME/bin/kubernetes-jobmanager1.sh kubernetes-session 2. Build your own Flink