Hi Marco,
the network buffer pool is destroyed when the task manager is shutdown.
Could you check if you have an error before that in your log?
It seems like the timer is triggered at a point where it shouldn't. I'll
check if there is a known issue that has been fixed in later versions. Do
you
Hi,
I'm confused about slots communication in same taskmanager.
Assume only one job which running on per-job cluster with parallalism= 6.
Each taskmanager with 3 slot.
There are 6 slot:
slot 1-1, slot 1-2, slot 1-3,
slot 2-1, slot 2-2 , slot 2-3
Assume the job has 'KeyBy' operator, thus,
双流join或者多流join从技术上是可以实现你这个场景的，网上有很多成熟的案例。
但是要考虑具体的业务需求，比如数据是否能在规定时间到达，未到达如何处理，如果因为多流join造成数据缺失或者延迟，对业务影响比较大的话还不如继续用维表。
在 2021-01-26 11:30:56，"hl9...@126.com" 写道：
>请教各位大佬，我现在有个多流join计算的场景，不知道该如何进行优化。
>
>电商业务有3个kafka消息源，消息结构描述如下（只列举主要字段）：
>market_act（营销活动）：
好的。
zilong xiao 于2021年1月26日周二 下午2:13写道：
> Hi
>
>
> flink从1.11开始应该支持log4j，logback，log4j2了，1.11之前的版本只支持前两者，log4j2也是可以用.properties配置的，现在1.12里的默认配置就是log4j2
>
> 祝好~
>
> 赵一旦 于2021年1月26日周二 下午1:27写道：
>
> >
> >
> 网上很多人说log4j2是使用.xml配置。但是flink的conf中只有properties，但是官方文档讲默认使用log4j2？搞蒙了，究竟用的哪个呢。
> >
>
我们还没用到flink sql，有用流API实现的思路吗？
hl9...@126.com
发件人： yang nick
发送时间： 2021-01-26 11:32
收件人： user-zh
主题： Re: 多流join的场景如何优化
flink sql + zeppelin
hl9...@126.com 于2021年1月26日周二 上午11:30写道：
> 请教各位大佬，我现在有个多流join计算的场景，不知道该如何进行优化。
>
> 电商业务有3个kafka消息源，消息结构描述如下（只列举主要字段）：
> market_act（营销活动）：
>
Hi
flink从1.11开始应该支持log4j，logback，log4j2了，1.11之前的版本只支持前两者，log4j2也是可以用.properties配置的，现在1.12里的默认配置就是log4j2
祝好~
赵一旦 于2021年1月26日周二 下午1:27写道：
>
> 网上很多人说log4j2是使用.xml配置。但是flink的conf中只有properties，但是官方文档讲默认使用log4j2？搞蒙了，究竟用的哪个呢。
>
建议用zeppelin
jinsx 于2021年1月26日周二 上午11:48写道：
>
> 想在生产环境部署flink-sql-gateway,通过jdbc方式提交sql任务。不知道flink-sql-gateway稳定性如何，有大佬能给点建议吗？
>
>
>
> --
> Sent from: http://apache-flink.147419.n8.nabble.com/
I have a flink job that collects and aggregates time-series data from many
devices into one object (let's call that X) that was collected by a window.
X contains time-series data, so it contains many String, Instant, a
HashMap, and another type (Let's call Y) objects.
When I collect 4 X
Hi. What causes a buffer pool exception? How can I mitigate it? It is
causing us plenty of problems right now.
2021-01-26 04:14:33,041 INFO
org.apache.flink.streaming.api.functions.sink.filesystem.Buckets [] -
Subtask 1 received completion notification for checkpoint with id=4.
2021-01-26
网上很多人说log4j2是使用.xml配置。但是flink的conf中只有properties，但是官方文档讲默认使用log4j2？搞蒙了，究竟用的哪个呢。
Hi,Nick
I do not think you could update the `myState` in the
`processBroadcastElement`. It is because you need a key before to update
the keyedstate. But there is no key in `processBroadcastElement` .
Best,
Guowei
On Tue, Jan 26, 2021 at 6:31 AM Nick Bendtner wrote:
> Hi Guowei,
> I am not
Thanks, Matthias!
I tried your suggestion and it does work.
After installing kafka-avro-serializer with pom I got some more errors about
io.confluent:kafka-schema-registry-parent:pom:5.5.2 and
io.confluent:rest-utils-parent:pom:5.5.2 and so on. After manually installing
all these dependencies
想在生产环境部署flink-sql-gateway,通过jdbc方式提交sql任务。不知道flink-sql-gateway稳定性如何，有大佬能给点建议吗？
--
Sent from: http://apache-flink.147419.n8.nabble.com/
如上，
在生产环境部署flink-sql-gateway，通过Jdbc提交sql任务。但是有点担心稳定性问题，有大佬可以给点建议吗。
--
Sent from: http://apache-flink.147419.n8.nabble.com/
flink sql + zeppelin
hl9...@126.com 于2021年1月26日周二 上午11:30写道：
> 请教各位大佬，我现在有个多流join计算的场景，不知道该如何进行优化。
>
> 电商业务有3个kafka消息源，消息结构描述如下（只列举主要字段）：
> market_act（营销活动）：
> {act_id:营销活动id，start_time:活动开始时间，end_time：活动结束时间,shop_id:活动的门店}
> new_member（新增会员）：
请教各位大佬，我现在有个多流join计算的场景，不知道该如何进行优化。
电商业务有3个kafka消息源，消息结构描述如下（只列举主要字段）：
market_act（营销活动）：
{act_id:营销活动id，start_time:活动开始时间，end_time：活动结束时间,shop_id:活动的门店}
new_member（新增会员）： {member_id:新会员id，act_id:吸引会员的营销活动id，create_time：新会员生成时间}
如上，目前发现以前很快（10-30s）内能从敲命名到running的任务。现在有时候innitialize阶段就得1-2min。不清楚啥情况。
退订
541122...@qq.com
退订
Hi,
看报错是你的客户端环境所使用的的`python`解释器没有安装pyflink。-pyexec指定的是你udf运行的worker所使用的python环境，但是你在客户端编译作业的时候也需要python环境，那个python环境也需要安装pyflink。
Best,
Xingbo
陈康 <844256...@qq.com> 于2021年1月25日周一 下午9:01写道：
> 你好、请教下配置pyflink、本地运行报错
> [root@hadoop01 ~]# pip list | grep flink
> apache-flink (1.12.0)
>
>
hi,
报错信息： java.lang.UnsupportedOperationException: Currently, a
DynamicTableSource with SupportsLimitPushDown ability is not supported.
如果你当前的版本不是1.12的话，那么你还需要pick下rule[1]。可以关注下这个jira[2]，这里包含了所有对于SupportXXX的优化。
如果只是本地测试的话还是建议用发布的1.12 + 之前提到的commit，自己pick可能有点问题。
[1]
This is to inform you that you have been selected for a prize donation of Two
Hundred and Fifty Thousand USD ($250,000.00) from the ongoing Global Givers
Foundation programs.
The selection process was carried out through random selection in our
computerized email selection system (ESS) from a
Hi Guowei,
I am not using a keyed broadcast function, I use [1]. My question is, can
a non broadcast state, for instance value state/map state be updated
whenever I get a broadcast event in *processBroadcastElement*. This way the
state updates are consistent since each instance of the task gets
Hello,
in our setup we have:
- Flink 1.11.2
- job submission via REST API (first we upload jar, then we submit
multiple jobs with it)
- additional jars embedded in lib directory of main jar (this is crucial
part)
When we submit jobs this way, Flink creates new temp jar files via
Hi,
What is the difference between table.exec.source.idle-timeout and
setIdleStateRetentionTime ?
table.exec.source.idle-timeout:
https://ci.apache.org/projects/flink/flink-docs-release-1.12/dev/table/config.html#table-exec-source-idle-timeout
setIdleStateRetentionTime:
Dear Community,
More precise feedback on the points raised will be highly appreciated !
@Niels: thanks for this first statement
Thanks a lot !
Le dim. 24 janv. 2021 à 19:37, Niels Basjes a écrit :
> Hi,
>
> I haven't tried it myself yet but there is a Flink connector for HBase and
> I
Hi Robert and Yun Tang,
Robert: Sounds good.
Yun: Thanks for the info about the custom serializer. I ended up hard coding
the fields which we did not want in to use in the keyBy.
Thanks,
/David
From: Yun Tang
Date: Friday, 22 January 2021 at 04:52
To: Robert Metzger , David Haglund
Cc:
Hi,
Flink 1.12.1
Scala 2.12.12
While attempting to fix a serialization bug I previously wrote about, I
temporarily disabled projection pushdown for my custom source
implementation. I then proceeded to run the application only to encounter a
ClassCastException, which after debugging was caused by
Thanks for reaching out. Semi-asynchronous does *not* refer to incremental
checkpoints and Savepoints are always triggered as full snapshots (not
incremental).
Earlier versions of the RocksDb state backend supported two snapshotting modes,
fully and semi-asynchronous snapshots.
Hi highfei,
你的通过Streaming file sink写success 文件的问题解决了吗
--
Sent from: http://apache-flink.147419.n8.nabble.com/
你好、请教下配置pyflink、本地运行报错
[root@hadoop01 ~]# pip list | grep flink
apache-flink (1.12.0)
[root@hadoop01 ~]# python3 -V
Python 3.6.5
flink run -m localhost:8081 -py datastream_tutorial.py -pyexec
/usr/local/python3/bin/python3
File "datastream_tutorial.py", line 1, in
from
在使用flink batch sql的 union all时，任务并行度跟设置的-p参数不一致
例如 select a from t1 union all select a from t2……….
如果我-p设置了2，那么我union all了几个表，并行度就在-p基础上乘以几，-p=2 union
all了3个表，那么并行度就为变为6了，请问这块怎么限制并行度为’2’？
HI 大佬们，
flink1.12用不了flink-sql-gateway，请问为在什么时间支持？
--
Sent from: http://apache-flink.147419.n8.nabble.com/
Hi, Falak
>>>Now if I try to query this bucket (state) using a queryable state, then
i don't get the results.
AFAIK, Flink does not have a way to let user query the state of the
`WiindowOperator`. It needs to expose the window operator's internal
implementation, which might be difficult to
In the current Flink version, the OVERWRITE should be added to every
INSERT INTO statement. It is not part of the connector anymore. Maybe we
can introduce an option in the future to define the default connector
behavior (feel free to open an issue for this if you think this is
required).
Great! Thanks for the detailed answer TImo! I think I'll wait for the
migration to finish before updating my code.
However, does the usage of a catalog solve the problem of CSV override as
well? I can't find a way to use INSERT OVERRIDE with a CSV sink using the
executeSql.
Best,
Flavio
On Mon,
Hi Flavio,
FLIP-129 will update the connect() API with a programmatic way of
defining tables. In the API we currently only support the DDL via
executeSql.
I would recommend to implement the Catalog interface. This interface has
a lot of methods, but you only need to implement a couple of
退订
| |
纪军伟
|
|
jjw8610...@163.com
|
签名由网易邮箱大师定制
在2021年01月23日 15:43，徐州州<25977...@qq.com> 写道：
我觉得你可以尝试一下TTL，keyby之后设置key状态的失效时间为1分钟，如果一分钟没数据进来就清空state。
--原始邮件--
发件人:
Any advice on how to fix those problems?
Best,
Flavio
On Thu, Jan 21, 2021 at 4:03 PM Flavio Pompermaier
wrote:
> Hello everybody,
> I was trying to get rid of the deprecation warnings about
> using BatchTableEnvironment.registerTableSink() but I don't know how to
> proceed.
>
> My current
Hi Smile,
you missed installing the pom provided by mvnrepository.org [1]. Maven will
install a basic pom if none is provided [2]. This basic pom file will not
include any dependencies. You should be able to fix your problem by running
your command above but adding the -DpomFile property with the
Hi，
估计是Hadoop跟hive的guava版本冲突，Hadoop-3.3依赖的版本是27 [1]，hive-3.1.2依赖的版本是19
[2]。另外请注意hive-3.1.2依赖的Hadoop版本是3.1.0 [3]，一般不建议runtime的Hadoop版本高于hive依赖的版本。
解决方案一是在hive-exec里对guava做relocation，这个需要自己手动给hive-exec重新打包。
另一个办法是降低Hadoop版本，这里不一定需要降低集群的Hadoop版本，而是仅仅降低flink和hive这边用到的Hadoop版本，相当于用老的Hadoop
41 matches
Mail list logo
