flinksql写出到kafka

[leiyanrui]

flink1.10通过sql的形式连接kafka，处理数据写出到kafka，然后我里面大概是10个这样的逻辑，insert into sink
select
...，10个sql都是直接sink的，运行几分钟就报错，org.apache.flink.kafka.shaded.org.apache.kafka.common.errors.TimeoutException:
Topic boss.nlp.hbase.feature-sync-60 not present in metadata after 6 ms.
但是我吧这个10个select语句union后在sink到kafka，没有这个问题，这个有什么合理的解释吗



--
Sent from: http://apache-flink.147419.n8.nabble.com/

flinksql连接kafka，数据里面有个字段名字是time

[leiyanrui]

使用flinksql连接kafka，kafka的数据格式内部有个字段叫time，我在create
table的时候将time字段加了反单引号还是不行，报错，有什么别的方法吗




--
Sent from: http://apache-flink.147419.n8.nabble.com/

Re: flinksql连接kafka，数据里面有个字段名字是time

[leiyanrui]

CREATE TABLE table1 (
bg BIGINT,
user_source BIGINT,
bossid   BIGINT,
geekid BIGINT,
qq_intent BIGINT,
phone_intent BIGINT,
wechat_intent BIGINT,
`time` BIGINT,
t as to_timestamp(from_unixtime(__ts,'-MM-dd HH:mm:ss')),
watermark for t as t - interval '5' second
) WITH (
...
);



--
Sent from: http://apache-flink.147419.n8.nabble.com/

Re: flinksql连接kafka，数据里面有个字段名字是time

[leiyanrui]

1.10
 



--
Sent from: http://apache-flink.147419.n8.nabble.com/

Re: flinksql连接kafka，数据里面有个字段名字是time

[leiyanrui]

 



--
Sent from: http://apache-flink.147419.n8.nabble.com/

Re: flinksql连接kafka，数据里面有个字段名字是time

[leiyanrui]

好的 我去看下 谢谢哈



--
Sent from: http://apache-flink.147419.n8.nabble.com/

flink sink redis

[leiyanrui]

flink将结果输出到redis，并发数大于1的情况下，在查询的时候会出现延迟吗，采取的是flink提供的redisMapper



--
Sent from: http://apache-flink.147419.n8.nabble.com/

blob server相关，文件找不到

[leiyanrui]

java.util.concurrent.CompletionException:
org.apache.flink.util.FlinkException: Could not retrieve file from transient
blob store.
at
org.apache.flink.runtime.rest.handler.taskmanager.AbstractTaskManagerFileHandler.lambda$respondToRequest$0(AbstractTaskManagerFileHandler.java:135)
at
java.util.concurrent.CompletableFuture.uniAccept(CompletableFuture.java:656)
at
java.util.concurrent.CompletableFuture$UniAccept.tryFire(CompletableFuture.java:632)
at
java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:442)
at
org.apache.flink.shaded.netty4.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163)
at
org.apache.flink.shaded.netty4.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:404)
at
org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:474)
at
org.apache.flink.shaded.netty4.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:909)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.flink.util.FlinkException: Could not retrieve file
from transient blob store.
... 9 more
Caused by: java.io.FileNotFoundException: Local file
/data/yarn/nm/usercache/nlp_realtime/appcache/application_1571815761034_7982/blobStore-ed462e9a-7468-4d9d-8295-c126c84fb5ef/no_job/blob_t-bd919682fabe41b123dbd280e435de85410e3ad5-bd65dcaa30ecab21dfe21ff3c434d906
does not exist and failed to copy from blob store.
at
org.apache.flink.runtime.blob.BlobServer.getFileInternal(BlobServer.java:516)
at
org.apache.flink.runtime.blob.BlobServer.getFileInternal(BlobServer.java:444)
at org.apache.flink.runtime.blob.BlobServer.getFile(BlobServer.java:369)
at
org.apache.flink.runtime.rest.handler.taskmanager.AbstractTaskManagerFileHandler.lambda$respondToRequest$0(AbstractTaskManagerFileHandler.java:133)
... 8 more



--
Sent from: http://apache-flink.147419.n8.nabble.com/

Re: Demo：从埋点日志中，统计实时的 PV 遇到的问题

[leiyanrui]

keyby(_behavior) 你看到的四个结果应该是每种behavior的pv 不是整个窗口的pv



--
Sent from: http://apache-flink.147419.n8.nabble.com/

Re: 回复： Demo：从埋点日志中，统计实时的 PV 遇到的问题

[leiyanrui]

进一步KeyedProcessFunction的处理是按照window的end时间，这样就会只有一个key进而将聚合后的数据再次聚合



--
Sent from: http://apache-flink.147419.n8.nabble.com/

flinksql 输出到kafka用的fixed的方式 结果都输出到一个topic分区了，这个是什么原因

[leiyanrui]

--
Sent from: http://apache-flink.147419.n8.nabble.com/

Re: Re:flinksql 输出到kafka用的fixed的方式 结果都输出到一个topic分区了，这个是什么原因

[leiyanrui]

我的topic分区数是10个，sink的并发是25个 ，按照取余计算的话 也不应该只输出到一个partition的



--
Sent from: http://apache-flink.147419.n8.nabble.com/

Re: Re:Re: Re:flinksql 输出到kafka用的fixed的方式 结果都输出到一个topic分区了，这个是什么原因

[leiyanrui]

好的 谢谢哈 



--
Sent from: http://apache-flink.147419.n8.nabble.com/

Re: flink本地编译源码失败

[leiyanrui]

修改为这个源之后可以编译下来吗 有人测过吗



--
Sent from: http://apache-flink.147419.n8.nabble.com/

flinksql维表join

[leiyanrui]

flinksql维表join之后不能做些过滤这样的操作吗



--
Sent from: http://apache-flink.147419.n8.nabble.com/