Re: flink权限问题

hi hdfs有严格的权限管理系统，这个问题似乎绕不过去吧，flink该有的权限就应该给它开了。 On Thu, Oct 17, 2019 at 7:58 PM jinhai wang wrote: > 背景： > > 实时flink任务从kafka同步数据到hdfs。 > > > > 问题： > > 比如flink的job使用A用户提交，但写hdfs的文件用户只能是B，那这个除了给hdfs添加A的操作权限外，flink > 这块儿可以优雅解决这类问题吗？checkpoint的写权限应该是A用户吧 >

Re: 文件重命名

May you want to rename them in HDFS with FileSystem.rename method? on 2019/10/8 17:39, yanggang_it_job wrote: 写入hdfs的文件都是 part-{parallel-task}-{count} 这种格式 如何重命名啊？

Re: Could not forward element to next operator

Hello, May this article match your issue? https://blog.csdn.net/qq_41910230/article/details/90411237 regards. On Sun, Sep 29, 2019 at 10:33 AM allan <18612537...@163.com> wrote: > Hi， > > 最近发现作业一直在报错，我的窗口是一分钟的窗口。这是什么原因，谁能帮助一下？flink版本1.6 ,错误如下： > > > > TimerException{org.apache.flink.streaming.r

Re: 订阅邮件

Hello You should not pose across multiple groups. One thread to one group is more graceful. 杨利君 于2019年9月26日 周四下午3:42写道： > 订阅flink社区邮件

Re: Split a stream into any number of streams

Hi on 2019/9/17 10:28, 王佩 wrote: * // How should I do it?* splitStream.select("productID1").print(); If I understand for that correctly, you want somewhat the dynamic number of Sinks? regards

Re: Split a stream into any number of streams

on 2019/9/17 9:55, 王佩 wrote: I want to split a stream into any number of streams according to a field, and then process the split stream one by one. I think that should be easy done. refer to: https://stackoverflow.com/questions/53588554/apache-flink-using-filter-or-split-to-split-a-stream

Re: Flink大state读取磁盘，磁盘IO打满，任务相互影响的问题探讨

on 2019/9/10 13:47, 蒋涛涛 wrote: 尝试手段： 1. 手动迁移IO比较高的任务到其他机器，但是yarn任务提交比较随机，只能偶尔为之 2. 目前没有SSD，只能用普通STATA盘，目前加了两块盘提示磁盘IO能力，但是单盘对单任务的磁盘IO瓶颈还在 还有哪些策略可以解决或者缓解么？ It seems the tricks to improve RocksDB's throughput might be helpfu. With writes and reads accessing mostly the recent data, our goal

Re: FLINK WEEKLY 2019/36

on 2019/9/9 11:23, Zili Chen wrote: FLINK WEEKLY 2019/36 很高兴和大家分享上周 FLINK 社区的发展。在过去的一周里，更多 FLINK 1.10 的特性被提出和讨论，包括新的 FLIP，来自 Apache Pulsar 社区的 Connector 贡献等等。专门讨论 FLINK 1.10 将要实现什么特性的议程也在进行。 用户问题 A nice work. thank you. regards.

Re: 希望获得一份编译后的1.6版本以上flink源码

Helo guaishushu1...@163.com wrote: 因windows系统搭载虚拟机环境，总是不能成功编译flink源码，所以希望能得到一份编译后的1.6版本以上的flink源码 there is docker image for flink 1.9. since you are using a virtual machine, then docker might be used. regards

Re: Flink Savepoint 超时

SJMSTER wrote: Checkpoints一直都是成功的。 今天重新尝试了一下cancle job with savepoint又成功了.. 不知道之前为什么试了几次都是超时的.. are there any log items for diagnosis? regards.

Re: flink1.9.0对DDL的支持

Hi on 2019/9/5 11:23, pengcheng...@bonc.com.cn wrote: 请教一下， 1.flink1.9.0的table API/sql是不是还没有支持Create view？ from the official documentation of flink 1.9: Views can also be created within a CLI session using the CREATE VIEW statement: CREATE VIEW MyNewView AS SELECT MyField2 FROM MyTableSour

Re: FLink WEB 怎么加登录验证？

Hi on 2019/9/5 10:46, wanghongquan.sh wrote: FLink WEB 控制台中，没有找到登录验证的配置，请问这个WEB怎么加登录验证？ Flink does not directly support authenticating access to the web UI, but you can always put something like Apache's basic_auth in front of it.

Re: flink集群模式下如何动态添加taskmanager

Hi on 2019/9/4 10:27, 如影随形 wrote: 在flink集群部署时，有1个jobmanager,有2个taskmanager(slave1,slave2)， 现在想添加slave3作为 taskmnanager。如何在不停止集群的情况下，可以像spark一样动态添加吗？ AFAIK the answer is NO for now. However, community tells that this has been under consideration from the FLIP-6 Flink Development and Proce

Re: 回复： flink使用StateBackend问题

on 2019/9/3 15:38, 守护 wrote: org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Received late message for now expired checkpoint attempt 3 from 24674987621178ed1a363901acc5b128 of job fd5010cbf20501339f1136600f0709c3. 请问这个是什么问题呢？ 可以根据这些失败的task的id去查询这些任务落在哪一个taskmanager上，经过排查发现

Re: flink使用StateBackend问题

Hi on 2019/9/3 12:14, 々守护々 wrote: 现象：我是在代码中配置StateBackend的，代码是这样写的env.setStateBackend(new RocksDBStateBackend("hdfs://host51:9000/flink/checkpoints",true))，但是我发现在下面的SQLwindow中始终在checkpoint那就过不去，总是报n/a， 下面我就把代码中的这句给注释掉了，在flink-conf.xml文件中配置 state.checkpoints.dir: hdfs://host51:9000/flink/flink-

Re: FlinkKafkaProducer 开启Excatly Once之后 初始化事务状态超时的问题

Hi on 2019/9/2 11:49, 陈赋赟 wrote: 2019-09-02 10:24:28,599 INFO org.apache.flink.runtime.taskmanager.Task - Interval Join -> Sink: Unnamed (1/4) (e8b85b6f144879efbb0b4209f226c69b) switched from RUNNING to FAILED. org.apache.kafka.common.errors.TimeoutException: Timeout expir

Re: 关于flink 写于kafka时的transactionId 生成问题

Hi on 2019/8/29 17:50, ddwcg wrote: broker就一个，flink集群的时钟确实和broker的不一样，是不同的机房，能自己指定transactionalId吗，两个机房的调整成一样怕影响其他的应用 AFAIK the transID is generated by systems. regards.

Re: 关于flink 写于kafka时的transactionId 生成问题

Hi on 2019/8/29 17:13, ddwcg wrote: 作业我已经停止了，但是看kafka的日志还是不断的在刷Initialized transactionalId………. ，而且该程序再此启动就会报： Caused by: org.apache.kafka.common.errors.ProducerFencedException: Producer attempted an operation with an old epoch. Either there is a newer producer with the same transactionalId, or