Re:flink sql jdbc connector是否支持多流拼接？
这种情况下有两种方式可以处理 1> 注册表-使用join方式直接拼接成大宽表写入 2> 每个任务-直接写入下游数据 ，每个任务只更新自己的字段即可(因为主键相同) 在 2023-03-02 20:59:59，"casel.chen" 写道： >flink sql jdbc connector是否支持多流拼接？ >业务上存在基于主键打宽场景，即多个流有相同的主键字段，除此之前其他字段没有重叠，现在想打成一张大宽表写入mysql/oracle这些关系数据库。 >每条流更新大宽表的一部分字段。
Pyflink提交
在使用pyflink提交任务时，部署模式onyarn 1 在不使用Map等算子下如下参数 能够提交成功 并且运行 .flink run -ynm pytest -m yarn-cluster -pyclientexec ***/python3 -pyexec ***/python3 -pyarch *** /python3.6.8.zip -py demo.py 2 在使用到map算子时 提交没有问题，但是运行报错，报错日志如下： .flink run -ynm pytest -m yarn-cluster -pyclientexec ***/python3 -pyexec ***/python3 -pyarch *** /python3.6.8.zip -py demo.py 报错内容： Caused by: java.io.IOException: Failed to execute the command: ./python3.6.8.zip/bin/python3 -c import pyflink;import os;print(os.path.join(os.path.abspath(os.path.dirname(pyflink.__file__)), 'bin')) output: Traceback (most recent call last): File "", line 1, in ModuleNotFoundError: No module named 'pyflink' at org.apache.flink.python.util.PythonEnvironmentManagerUtils.execute(PythonEnvironmentManagerUtils.java:211) at org.apache.flink.python.util.PythonEnvironmentManagerUtils.getPythonUdfRunnerScript(PythonEnvironmentManagerUtils.java:154) at org.apache.flink.python.env.beam.ProcessPythonEnvironmentManager.createEnvironment(ProcessPythonEnvironmentManager.java:156) at org.apache.flink.streaming.api.runners.python.beam.BeamPythonFunctionRunner.createPythonExecutionEnvironment(BeamPythonFunctionRunner.java:398) at org.apache.flink.streaming.api.runners.python.beam.BeamPythonFunctionRunner.lambda$open$0(BeamPythonFunctionRunner.java:246) at org.apache.flink.runtime.memory.MemoryManager.lambda$getSharedMemoryResourceForManagedMemory$5(MemoryManager.java:539) 请问环境变量如何设置才能正常运行
Re:Re: pyflink1.11.1连接hive问题
完整日志乳如下：
Traceback (most recent call last):
File
"/Users/bjhl/PycharmProjects/flink-example/com.baijiahulian/connecthive/HiveTest.py",
line 26, in
t_env.register_catalog("default", hive_catalog)
File
"/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/pyflink/table/table_environment.py",
line 133, in register_catalog
self._j_tenv.registerCatalog(catalog_name, catalog._j_catalog)
File
"/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/py4j/java_gateway.py",
line 1286, in __call__
answer, self.gateway_client, self.target_id, self.name)
File
"/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/pyflink/util/exceptions.py",
line 147, in deco
return f(*a, **kw)
File
"/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/py4j/protocol.py",
line 332, in get_return_value
format(target_id, ".", name), value)
py4j.protocol.Py4JJavaError: An error occurred while calling
o10.registerCatalog.
: java.lang.NullPointerException
at org.apache.hadoop.util.StringUtils.stringifyException(StringUtils.java:90)
at
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.open(HiveMetaStoreClient.java:512)
at
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.(HiveMetaStoreClient.java:244)
at
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.(HiveMetaStoreClient.java:187)
at
org.apache.flink.table.catalog.hive.client.HiveShimV100.getHiveMetastoreClient(HiveShimV100.java:97)
at
org.apache.flink.table.catalog.hive.client.HiveMetastoreClientWrapper.createMetastoreClient(HiveMetastoreClientWrapper.java:240)
at
org.apache.flink.table.catalog.hive.client.HiveMetastoreClientWrapper.(HiveMetastoreClientWrapper.java:71)
at
org.apache.flink.table.catalog.hive.client.HiveMetastoreClientFactory.create(HiveMetastoreClientFactory.java:35)
at org.apache.flink.table.catalog.hive.HiveCatalog.open(HiveCatalog.java:223)
at
org.apache.flink.table.catalog.CatalogManager.registerCatalog(CatalogManager.java:191)
at
org.apache.flink.table.api.internal.TableEnvironmentImpl.registerCatalog(TableEnvironmentImpl.java:337)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.flink.api.python.shaded.py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
at
org.apache.flink.api.python.shaded.py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
at org.apache.flink.api.python.shaded.py4j.Gateway.invoke(Gateway.java:282)
at
org.apache.flink.api.python.shaded.py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
at
org.apache.flink.api.python.shaded.py4j.commands.CallCommand.execute(CallCommand.java:79)
at
org.apache.flink.api.python.shaded.py4j.GatewayConnection.run(GatewayConnection.java:238)
at java.lang.Thread.run(Thread.java:748)
在 2020-09-03 16:15:30，"Dian Fu" 写道：
>有更完整的log吗？
>
>> 在 2020年9月3日，下午4:12，程龙 <13162790...@163.com> 写道：
>>
>> 使用hivecatalog连接 抱一下错误
>>
>>
>>
>>
>> flink py4j.protocol.Py4JJavaError: An error occurred while calling
>> o10.registerCatalog. : java.lang.NullPointerException
pyflink1.11.1连接hive问题
使用hivecatalog连接 抱一下错误 flink py4j.protocol.Py4JJavaError: An error occurred while calling o10.registerCatalog. : java.lang.NullPointerException
Re:无法从checkpoint中恢复state
再启动服务的时候 需要指定checkpoint回复地址，你这里只是指定了做checkpint地址
在 2020-09-03 16:03:41，"sun" <1392427...@qq.com> 写道：
>你好，我有2个问题
>
>1：每次重启服务，checkpoint的目录中chk- 总是从chk-1开始，chk-2 ,没有从上次的编号开始
>
>2：重启服务后，没有从checkpoint中恢复state的数据
>
>下面是我的配置，我是在本地调试的，单机
>
>
>
>final StreamExecutionEnvironment streamExecutionEnvironment =
>StreamExecutionEnvironment.createLocalEnvironmentWithWebUI(conf);
>
>//StateBackend stateBackend = new
>RocksDBStateBackend("hdfs://10.100.51.101:9000/flink/checkpoints",true);
>StateBackend stateBackend = new
> FsStateBackend("file:///flink/checkpoints");
>//StateBackend stateBackend = new MemoryStateBackend();
>streamExecutionEnvironment.setStateBackend(stateBackend);
>
>streamExecutionEnvironment.enableCheckpointing(1000);
>
> streamExecutionEnvironment.getCheckpointConfig().setCheckpointingMode(CheckpointingMode.EXACTLY_ONCE);
>
> streamExecutionEnvironment.getCheckpointConfig().setMinPauseBetweenCheckpoints(500);
>
> streamExecutionEnvironment.getCheckpointConfig().setCheckpointTimeout(6);
>
> streamExecutionEnvironment.getCheckpointConfig().setMaxConcurrentCheckpoints(1);
>streamExecutionEnvironment.getCheckpointConfig()
>
> .enableExternalizedCheckpoints(CheckpointConfig.ExternalizedCheckpointCleanup.RETAIN_ON_CANCELLATION);
Re:Re: Re: Re: 回复：flink1.10 checkpoint 运行一段时间空指针异常
ok 我过两天试一下 最近有点忙 过两天测试一下1.11 到时候结果同步给大家 ，另外之前测试jdk使用jdk1.8.0_231 也是不行的 会报同样的错误 在 2020-07-13 20:06:01，"Congxian Qiu" 写道： >Hi 程龙 > >如果可以的话，也麻烦使用 1.11.0 测试下看问题是否还存在。 > >Best, >Congxian > > >程龙 <13162790...@163.com> 于2020年7月13日周一 上午10:45写道： > >> >> >> >> >> >> >> 问题不是很常见 ，但是同一个任务，提交在flink1.10 和 flink1.10.1上都会复现， 准备尝试一下升级一下jdk试试 >> >> >> >> >> >> >> >> >> >> >> >> 在 2020-07-06 16:11:17，"Congxian Qiu" 写道： >> >@chenkaibit 多谢你的回复~ >> > >> >Best, >> >Congxian >> > >> > >> >chenkaibit 于2020年7月6日周一 下午3:53写道： >> > >> >> hi，Congxian。我在发现这个问题时也很奇怪，但是在打印了一些日志后，确实验证了我的想法。因为 <低版本jdk+flink1.9> 和 >> >> <高版本jdk+1.10> 都不会抛 NPE(见 FLINK-17479)，我猜测和 lambda 表达式中外部变量的垃圾回收机制以及 1.10 >> >> 引入的 MailBox 模型有关，外部 checkpointMetaData 实例被意外回收了。所以在修复的 patch 中我在 lambda >> >> 表达式内部实例化了一个新的 checkpointMetaData，目前看这个方法是有效的，没有再发现过 >> >> NPE。这是个临时的修复方法，根本原因可能还需要进一步分析。 >> >> >> >> >> >> -- >> >> Best, yuchuan >> >> >> >> >> >> >> >> 在 2020-07-06 14:04:58，"Congxian Qiu" 写道： >> >> >@陈凯 感谢你分享的这个方法，比较好奇这两个的区别是什么？修改后的 patch 在 closure 中一开始 copy 了一份 >> >> >CheckpointMeta，也就是说 845 - 867 行之间，之前的 checkpointMeta 会变为 null，这个比较奇怪。 >> >> > >> >> >Best, >> >> >Congxian >> >> > >> >> > >> >> >陈凯 于2020年7月6日周一 上午9:53写道： >> >> > >> >> >> >> >> >> Hi,zhisheng 程龙.我们也遇到这个问题了，jdk版本jdk8_40，低版本 jdk 确实有大概率会NPE。 >> >> >> 我之前提了个jira 描述了这个问题 >> >> >> https://issues.apache.org/jira/browse/FLINK-18196 >> >> >> >> >> >> 修改了Checkpoint 相关代码后，在低版本 jdk 上也没有再发现过过NPE。如果实在不能升级 jdk >> 版本，可以参考下面的patch： >> >> >> >> >> >> >> https://github.com/yuchuanchen/flink/commit/e5122d9787be1fee9bce141887e0d70c9b0a4f19 >> >> >> >> >> >> >> >> >> >> >> >> -邮件原件- >> >> >> 发件人: zhisheng >> >> >> 发送时间: 2020年7月5日 15:01 >> >> >> 收件人: user-zh >> >> >> 主题: Re: 回复：flink1.10 checkpoint 运行一段时间空指针异常 >> >> >> >> >> >> 生产集群 JDK 使用的地方比较多，不敢轻易换版本，后面再观察一下，如果频繁出现这种问题再考虑更换版本，感谢 Congxian >> >> >> >> >> >> Best！ >> >> >> zhisheng >> >> >> >> >> >> Congxian Qiu 于2020年7月4日周六 下午3:21写道： >> >> >> >> >> >> > @zhisheng 你们有尝试过更换 jdk 版本吗？更换版本是否能解决这个问题呢？ >> >> >> > >> >> >> > Best, >> >> >> > Congxian >> >> >> > >> >> >> > >> >> >> > zhisheng 于2020年7月4日周六 下午12:27写道： >> >> >> > >> >> >> > > 我们也有遇到过这个异常，但是不是很常见 >> >> >> > > >> >> >> > > Congxian Qiu 于2020年7月3日周五 下午2:08写道： >> >> >> > > >> >> >> > > > 你可以看看是否 FLINK-17479[1] 和你的问题一样，是的话，可以尝试修改一下 jdk 版本试试 >> >> >> > > > [1] https://issues.apache.org/jira/browse/FLINK-17479 >> >> >> > > > Best, >> >> >> > > > Congxian >> >> >> > > > >> >> >> > > > >> >> >> > > > 程龙 <13162790...@163.com> 于2020年7月1日周三 下午9:09写道： >> >> >> > > > >> >> >> > > > > >> >> >> > > > > >> >> >> > > > > >> >> >> > > > > >> >> >> > > > > >> >> >> > > > > >> >> >> > > > > >> >> >> > > > > >> >> >> > > > > >> >> >> > > > > >> >> >> > > > > >> >> >> > > > > >> >> >> > > > > 都是分配不到资源(slot)的错误，应该还是checkpoint 为空导致的,不知道为啥为空 >> >> >> > > > > >> >> >> > > > > >> >> >> > > > > >> >> >> > >
Re:Re: Re: flink on yarn日志问题
运行的日志会越来越多 导致查看日志比较慢 大多采用elk这种方式 除了这个有没有比较好的方案推荐一下 在 2020-07-14 12:35:06，"zhisheng" 写道： >知道 YARN 的 applicationId，应该也可以去 HDFS 找对应的 taskmanager 的日志（可以拼出路径），然后复制到本地去查看 > >Yangze Guo 于2020年7月14日周二 上午11:58写道： > >> Hi, 王松 >> >> 我理解拼接url就可以了，不用实际去登陆机器然后进到对应目录。 >> >> Best, >> Yangze Guo >> >> On Tue, Jul 14, 2020 at 8:26 AM 王松 wrote: >> > >> > 我们也有问题 1，和 Yangze Guo 说的一样，每次都要去对应的tm目录中去找日志，很麻烦，不知道有没有更简单的办法。 >> > >> > Yangze Guo 于2020年7月13日周一 下午5:03写道： >> > >> > > 1. >> > > >> 我验证了一下，如果开启了日志收集，那tm的日志是会保存的，但是你整个application结束前可能看不到，有一个trick的方法，首先在jm日志中找到tm分配到了哪个NodeManager上，通过拼接url的方式来获取container的日志 >> > > 2. 你是否需要调整一下重启策略[1]? 如果开启了ck，默认情况下就会一直尝试重启job >> > > >> > > [1] >> > > >> https://ci.apache.org/projects/flink/flink-docs-master/zh/dev/task_failure_recovery.html >> > > >> > > Best, >> > > Yangze Guo >> > > >> > > >> > > On Mon, Jul 13, 2020 at 2:40 PM 程龙 <13162790...@163.com> wrote: >> > > > >> > > > 不好意思 怪我灭有描述清楚 >> > > > 1 目前开启日志收集功能 >> > > > 2 目前已是 per-job模式 >> > > > 3 集群使用cdh flink.1.10 >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > 在 2020-07-13 11:18:46，"Yangze Guo" 写道： >> > > > >Hi, >> > > > > >> > > > >第一个问题，您可以尝试开启Yarn的日志收集功能[1] >> > > > > >> > > > >第二个问题，您可以尝试一下per-job mode [2][3] >> > > > > >> > > > >[1] >> > > >> https://ci.apache.org/projects/flink/flink-docs-master/zh/ops/deployment/yarn_setup.html#log-files >> > > > >[2] >> > > >> https://ci.apache.org/projects/flink/flink-docs-master/zh/ops/deployment/#per-job-mode >> > > > >[3] >> > > >> https://ci.apache.org/projects/flink/flink-docs-master/zh/ops/deployment/yarn_setup.html#run-a-single-flink-job-on-yarn >> > > > > >> > > > > >> > > > >Best, >> > > > >Yangze Guo >> > > > > >> > > > >On Mon, Jul 13, 2020 at 10:49 AM 程龙 <13162790...@163.com> wrote: >> > > > >> >> > > > >> 请问一下两个问题 >> > > > >> 1 flink on yarn的时候 taskmanager 挂掉的时候 上面的日志会被删除掉 无法查看 >> > > ，除了使用es收集日志的这种方案， 还有没有可以使taskmanager 挂掉，相关日志仍然可以保留。 >> > > > >> 2 flink on yarn模式 当由于错误导致taskmanager 挂掉，但是jobmanager 却一直存在， >> > > 有没有好的方式或者策略 ， 可以是当task失败 达到重试次数之后 taskmanager挂掉，jobmanager也挂掉 >> > > > >> >> > > >>
flink on yarn日志问题
请问一下两个问题 1 flink on yarn的时候 taskmanager 挂掉的时候 上面的日志会被删除掉 无法查看 ，除了使用es收集日志的这种方案， 还有没有可以使taskmanager 挂掉，相关日志仍然可以保留。 2 flink on yarn模式 当由于错误导致taskmanager 挂掉，但是jobmanager 却一直存在， 有没有好的方式或者策略 ， 可以是当task失败 达到重试次数之后 taskmanager挂掉，jobmanager也挂掉
Re:Re: Re: 回复：flink1.10 checkpoint 运行一段时间空指针异常
问题不是很常见 ，但是同一个任务，提交在flink1.10 和 flink1.10.1上都会复现， 准备尝试一下升级一下jdk试试 在 2020-07-06 16:11:17，"Congxian Qiu" 写道： >@chenkaibit 多谢你的回复~ > >Best, >Congxian > > >chenkaibit 于2020年7月6日周一 下午3:53写道： > >> hi，Congxian。我在发现这个问题时也很奇怪，但是在打印了一些日志后，确实验证了我的想法。因为 <低版本jdk+flink1.9> 和 >> <高版本jdk+1.10> 都不会抛 NPE(见 FLINK-17479)，我猜测和 lambda 表达式中外部变量的垃圾回收机制以及 1.10 >> 引入的 MailBox 模型有关，外部 checkpointMetaData 实例被意外回收了。所以在修复的 patch 中我在 lambda >> 表达式内部实例化了一个新的 checkpointMetaData，目前看这个方法是有效的，没有再发现过 >> NPE。这是个临时的修复方法，根本原因可能还需要进一步分析。 >> >> >> -- >> Best, yuchuan >> >> >> >> 在 2020-07-06 14:04:58，"Congxian Qiu" 写道： >> >@陈凯 感谢你分享的这个方法，比较好奇这两个的区别是什么？修改后的 patch 在 closure 中一开始 copy 了一份 >> >CheckpointMeta，也就是说 845 - 867 行之间，之前的 checkpointMeta 会变为 null，这个比较奇怪。 >> > >> >Best, >> >Congxian >> > >> > >> >陈凯 于2020年7月6日周一 上午9:53写道： >> > >> >> >> >> Hi,zhisheng 程龙.我们也遇到这个问题了，jdk版本jdk8_40，低版本 jdk 确实有大概率会NPE。 >> >> 我之前提了个jira 描述了这个问题 >> >> https://issues.apache.org/jira/browse/FLINK-18196 >> >> >> >> 修改了Checkpoint 相关代码后，在低版本 jdk 上也没有再发现过过NPE。如果实在不能升级 jdk 版本，可以参考下面的patch： >> >> >> >> https://github.com/yuchuanchen/flink/commit/e5122d9787be1fee9bce141887e0d70c9b0a4f19 >> >> >> >> >> >> >> >> -邮件原件- >> >> 发件人: zhisheng >> >> 发送时间: 2020年7月5日 15:01 >> >> 收件人: user-zh >> >> 主题: Re: 回复：flink1.10 checkpoint 运行一段时间空指针异常 >> >> >> >> 生产集群 JDK 使用的地方比较多，不敢轻易换版本，后面再观察一下，如果频繁出现这种问题再考虑更换版本，感谢 Congxian >> >> >> >> Best！ >> >> zhisheng >> >> >> >> Congxian Qiu 于2020年7月4日周六 下午3:21写道： >> >> >> >> > @zhisheng 你们有尝试过更换 jdk 版本吗？更换版本是否能解决这个问题呢？ >> >> > >> >> > Best, >> >> > Congxian >> >> > >> >> > >> >> > zhisheng 于2020年7月4日周六 下午12:27写道： >> >> > >> >> > > 我们也有遇到过这个异常，但是不是很常见 >> >> > > >> >> > > Congxian Qiu 于2020年7月3日周五 下午2:08写道： >> >> > > >> >> > > > 你可以看看是否 FLINK-17479[1] 和你的问题一样，是的话，可以尝试修改一下 jdk 版本试试 >> >> > > > [1] https://issues.apache.org/jira/browse/FLINK-17479 >> >> > > > Best, >> >> > > > Congxian >> >> > > > >> >> > > > >> >> > > > 程龙 <13162790...@163.com> 于2020年7月1日周三 下午9:09写道： >> >> > > > >> >> > > > > >> >> > > > > >> >> > > > > >> >> > > > > >> >> > > > > >> >> > > > > >> >> > > > > >> >> > > > > >> >> > > > > >> >> > > > > >> >> > > > > >> >> > > > > >> >> > > > > 都是分配不到资源(slot)的错误，应该还是checkpoint 为空导致的,不知道为啥为空 >> >> > > > > >> >> > > > > >> >> > > > > >> >> > > > > >> >> > > > > >> >> > > > > 在 2020-07-01 20:51:34，"JasonLee" <17610775...@163.com> 写道： >> >> > > > > >你到具体的tm上找到相关的operator看看是不是有异常信息 >> >> > > > > > >> >> > > > > > >> >> > > > > >| | >> >> > > > > >JasonLee >> >> > > > > >| >> >> > > > > >| >> >> > > > > >邮箱：17610775...@163.com >> >> > > > > >| >> >> > > > > > >> >> > > > > >Signature is customized by Netease Mail Master >> >> > > > > > >> >> > > > > >在2020年07月01日 20:43，程龙 写道： >> >> > > > > >flink1.10上 程序运行几个小时后就会报不能执行checkpoint 空指针异常 具体如下： >> >> > > > > > >> >> > > > > > >> >> > > > > >java.lang.Exception: Could not perform checkpoint 3201 for >> >> operator >> >> > > > > Filter -> Map (2/8). >> >> > > > > > at >> >> > > > > >> >> > > > >&
Re:回复：flink1.10 checkpoint 运行一段时间空指针异常
都是分配不到资源(slot)的错误，应该还是checkpoint 为空导致的,不知道为啥为空 在 2020-07-01 20:51:34，"JasonLee" <17610775...@163.com> 写道： >你到具体的tm上找到相关的operator看看是不是有异常信息 > > >| | >JasonLee >| >| >邮箱：17610775...@163.com >| > >Signature is customized by Netease Mail Master > >在2020年07月01日 20:43，程龙 写道： >flink1.10上 程序运行几个小时后就会报不能执行checkpoint 空指针异常 具体如下： > > >java.lang.Exception: Could not perform checkpoint 3201 for operator Filter -> >Map (2/8). > at > org.apache.flink.streaming.runtime.tasks.StreamTask.triggerCheckpointOnBarrier(StreamTask.java:816) > at > org.apache.flink.streaming.runtime.io.CheckpointBarrierHandler.notifyCheckpoint(CheckpointBarrierHandler.java:86) > at > org.apache.flink.streaming.runtime.io.CheckpointBarrierAligner.processBarrier(CheckpointBarrierAligner.java:177) > at > org.apache.flink.streaming.runtime.io.CheckpointedInputGate.pollNext(CheckpointedInputGate.java:155) > at > org.apache.flink.streaming.runtime.io.StreamTaskNetworkInput.emitNext(StreamTaskNetworkInput.java:133) > at > org.apache.flink.streaming.runtime.io.StreamOneInputProcessor.processInput(StreamOneInputProcessor.java:69) > at > org.apache.flink.streaming.runtime.tasks.StreamTask.processInput(StreamTask.java:310) > at > org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:187) > at > org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:485) > at > org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:469) > at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:708) > at org.apache.flink.runtime.taskmanager.Task.run(Task.java:533) > at java.lang.Thread.run(Thread.java:745) >Caused by: java.lang.NullPointerException > at > org.apache.flink.streaming.runtime.tasks.StreamTask$CheckpointingOperation.executeCheckpointing(StreamTask.java:1382) > at > org.apache.flink.streaming.runtime.tasks.StreamTask.checkpointState(StreamTask.java:974) > at > org.apache.flink.streaming.runtime.tasks.StreamTask.lambda$performCheckpoint$5(StreamTask.java:870) > at > org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$SynchronizedStreamTaskActionExecutor.runThrowing(StreamTaskActionExecutor.java:94) > at > org.apache.flink.streaming.runtime.tasks.StreamTask.performCheckpoint(StreamTask.java:843) > at > org.apache.flink.streaming.runtime.tasks.StreamTask.triggerCheckpointOnBarrier(StreamTask.java:803)
flink1.10 checkpoint 运行一段时间空指针异常
flink1.10上 程序运行几个小时后就会报不能执行checkpoint 空指针异常 具体如下： java.lang.Exception: Could not perform checkpoint 3201 for operator Filter -> Map (2/8). at org.apache.flink.streaming.runtime.tasks.StreamTask.triggerCheckpointOnBarrier(StreamTask.java:816) at org.apache.flink.streaming.runtime.io.CheckpointBarrierHandler.notifyCheckpoint(CheckpointBarrierHandler.java:86) at org.apache.flink.streaming.runtime.io.CheckpointBarrierAligner.processBarrier(CheckpointBarrierAligner.java:177) at org.apache.flink.streaming.runtime.io.CheckpointedInputGate.pollNext(CheckpointedInputGate.java:155) at org.apache.flink.streaming.runtime.io.StreamTaskNetworkInput.emitNext(StreamTaskNetworkInput.java:133) at org.apache.flink.streaming.runtime.io.StreamOneInputProcessor.processInput(StreamOneInputProcessor.java:69) at org.apache.flink.streaming.runtime.tasks.StreamTask.processInput(StreamTask.java:310) at org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:187) at org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:485) at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:469) at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:708) at org.apache.flink.runtime.taskmanager.Task.run(Task.java:533) at java.lang.Thread.run(Thread.java:745) Caused by: java.lang.NullPointerException at org.apache.flink.streaming.runtime.tasks.StreamTask$CheckpointingOperation.executeCheckpointing(StreamTask.java:1382) at org.apache.flink.streaming.runtime.tasks.StreamTask.checkpointState(StreamTask.java:974) at org.apache.flink.streaming.runtime.tasks.StreamTask.lambda$performCheckpoint$5(StreamTask.java:870) at org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$SynchronizedStreamTaskActionExecutor.runThrowing(StreamTaskActionExecutor.java:94) at org.apache.flink.streaming.runtime.tasks.StreamTask.performCheckpoint(StreamTask.java:843) at org.apache.flink.streaming.runtime.tasks.StreamTask.triggerCheckpointOnBarrier(StreamTask.java:803)
Re:flink open 时候 transient使用问问题
1 首先transient 是对修饰的变量不进行序列化 2 你使用transient的目的需要明确 使用来干啥的 3 状态都是可以读取并且使用的 不进行序列化 在 2020-06-24 11:37:09，"kcz" <573693...@qq.com> 写道： >请教大佬一个代码问题，当在open初始化一些mysql的client或者 >初始化state时候，用了transient是不是会对代码有优化作用，这里不是太理解。
Re:flink任务失败重启时， flink last checkpoint 失败但任务仍然正常重启，导致 state 重启前后不一致
可以自己改一下源码中的消费者 判断偏移量 ，如果是原先的正常启动 如果不是则不进行启动 在 2020-06-22 20:09:11，"莫失莫忘" 写道： >如题，可以要求flink失败重启时 必须正常从checkpoint恢复，否则就重启失败吗？
Re:Flink 1.10中是否有接口或方法获取批任务执行进度
可以试试自定义listener 在 2020-06-24 09:12:05，"faaron zheng" 写道： >Flink 1.10中是否有接口或方法获取批任务执行进度，百分比？ faaron zheng 邮箱：faaronzh...@gmail.com 签名由 >网易邮箱大师 定制
Re:Re: flink 通过 --classpath 传入https-xxx.xxx.jar 在1.8上正常运行 在flink1.10 就会报传入的jar包找不到
在同一套集群离 安装有连个版本的flink 使用-C http://xxx.jar的方式 1.8能正常运行 说明都能访问 使用1.10 就不行
At 2020-06-22 17:46:56, "Yang Wang" wrote:
> -C,--classpath Adds a URL to each user code
> classloader on all nodes in the
> cluster. The paths must specify a
> protocol (e.g. file://) and be
> accessible on all nodes (e.g. by
>means
> of a NFS share). You can use this
> option multiple times for
>specifying
> more than one URL. The protocol
>must
> be supported by the {@link
> java.net.URLClassLoader}.
>
>
>--classpath这个参数需要提前将jar部署在各个节点上或者使用NFS，确认jar是存在的吗
>
>
>Best,
>Yang
>
>程龙 <13162790...@163.com> 于2020年6月22日周一 上午11:43写道：
>
>> 2020-06-22 10:16:34,379 INFO
>> org.apache.flink.runtime.executiongraph.ExecutionGraph- Sink:
>> Unnamed (6/6) (5dd98ec92e0d5e53597cb7520643c7f5) switched from SCHEDULED to
>> DEPLOYING.
>> 2020-06-22 10:16:34,379 INFO
>> org.apache.flink.runtime.executiongraph.ExecutionGraph- Deploying
>> Sink: Unnamed (6/6) (attempt #0) to container_1590655249980_0081_01_02
>> @ al-bj-bigdata-inf-research-flink04 (dataPort=34781)
>> 2020-06-22 10:16:34,456 INFO
>> org.apache.flink.runtime.executiongraph.ExecutionGraph- Source:
>> Custom Source (5/6) (aca298438b9eb6fcf295cb8af6eebcd8) switched from
>> DEPLOYING to RUNNING.
>> 2020-06-22 10:16:34,481 INFO
>> org.apache.flink.runtime.executiongraph.ExecutionGraph- Filter ->
>> Map (3/8) (0daed15d107c3031891f0c9e84093068) switched from DEPLOYING to
>> RUNNING.
>> 2020-06-22 10:16:34,492 INFO
>> org.apache.flink.runtime.executiongraph.ExecutionGraph- Sink:
>> Unnamed (1/6) (44ca248aba351026452ba4abdb5f33a6) switched from DEPLOYING to
>> RUNNING.
>> 2020-06-22 10:16:34,497 INFO
>> org.apache.flink.runtime.executiongraph.ExecutionGraph- Filter ->
>> Map (1/8) (70612735eb755269fe9590e8ab51d3e2) switched from DEPLOYING to
>> RUNNING.
>> 2020-06-22 10:16:34,512 INFO
>> org.apache.flink.runtime.executiongraph.ExecutionGraph- Filter ->
>> Map (2/8) (baba331cd6bcde1f5a6024eac0b953b4) switched from DEPLOYING to
>> RUNNING.
>> 2020-06-22 10:16:34,524 INFO
>> org.apache.flink.runtime.executiongraph.ExecutionGraph- Source:
>> Custom Source (4/6) (a34992362c2cf3634a29bd5a9c188754) switched from
>> DEPLOYING to RUNNING.
>> 2020-06-22 10:16:34,531 INFO
>> org.apache.flink.runtime.executiongraph.ExecutionGraph- Source:
>> Custom Source (2/6) (7683c15e3ebb3e718c2439c6e32f0d7d) switched from
>> DEPLOYING to RUNNING.
>> 2020-06-22 10:16:34,564 INFO
>> org.apache.flink.runtime.executiongraph.ExecutionGraph- Source:
>> Custom Source (6/6) (6ab1aaa5e1811c79c702197f984e9bb6) switched from
>> DEPLOYING to RUNNING.
>> 2020-06-22 10:16:34,609 INFO
>> org.apache.flink.runtime.executiongraph.ExecutionGraph- Source:
>> Custom Source (1/6) (6245b508b8e0494f06ef71c6ad4954b6) switched from
>> DEPLOYING to RUNNING.
>> 2020-06-22 10:16:34,616 INFO
>> org.apache.flink.runtime.executiongraph.ExecutionGraph- Filter ->
>> Map (4/8) (2ea049476f0b48fcd85dcd9084091e9f) switched from DEPLOYING to
>> RUNNING.
>> 2020-06-22 10:16:34,650 INFO
>> org.apache.flink.runtime.executiongraph.ExecutionGraph- Sink:
>> Unnamed (2/6) (e785e44e4212dcc5279bcde761b28292) switched from DEPLOYING to
>> RUNNING.
>> 2020-06-22 10:16:34,656 INFO
>> org.apache.flink.runtime.executiongraph.ExecutionGraph- Sink:
>> Unnamed (5/6) (97a5e72f731d14aa97d93253b71b6aeb) switched from DEPLOYING to
>> RUNNING.
>> 2020-06-22 10:16:34,662 INFO
>> org.apache.flink.runtime.executiongraph.ExecutionGraph- Filter ->
>> Map (7/8) (59babdbc2d7bea362a2794a966fe59ef) switched from DEPLOYING to
>> RUNNING.
>> 2020-06-22 10:16:34,664 INFO
>> org.apache.flink.runtime.executiongraph.ExecutionGraph- Source:
>> Custom Source (3/6) (aed33b78cc847561908ea43164e0311a) switched from
>> DEPLOYING to RUNNING.
>> 2020-06-22 10:16:34,669 INFO
>> org.apache.flink.runtime.executiongraph.ExecutionGraph- Filter ->
>> Map (6/8) (fd87d0111b10e9f9027068a72e9ce209) switched from DEPLOYING to
>> RUNNING.
>> 2020-06-22 10:16:34,726 INF
Re:Re:flink 通过 --classpath 传入https-xxx.xxx.jar 在1.8上正常运行 在flink1.10 就会报传入的jar包找不到
嗯 确认都能访问 使用http的方式 在 2020-06-23 10:04:55，"Weixubin" <18925434...@163.com> 写道： >和版本应该没什么关系。如果是多节点部署的情况下，-C 所指定的URL 需要各个节点都能访问得到。 确认下该URL能被所有节点访问到吗 > Best, > Bin > > > > > > > > > > > > > > >At 2020-06-22 11:43:11, "程龙" <13162790...@163.com> wrote: >>2020-06-22 10:16:34,379 INFO >>org.apache.flink.runtime.executiongraph.ExecutionGraph- Sink: Unnamed >>(6/6) (5dd98ec92e0d5e53597cb7520643c7f5) switched from SCHEDULED to DEPLOYING. >>2020-06-22 10:16:34,379 INFO >>org.apache.flink.runtime.executiongraph.ExecutionGraph- Deploying >>Sink: Unnamed (6/6) (attempt #0) to container_1590655249980_0081_01_02 @ >>al-bj-bigdata-inf-research-flink04 (dataPort=34781) >>2020-06-22 10:16:34,456 INFO >>org.apache.flink.runtime.executiongraph.ExecutionGraph- Source: >>Custom Source (5/6) (aca298438b9eb6fcf295cb8af6eebcd8) switched from >>DEPLOYING to RUNNING. >>2020-06-22 10:16:34,481 INFO >>org.apache.flink.runtime.executiongraph.ExecutionGraph- Filter -> Map >>(3/8) (0daed15d107c3031891f0c9e84093068) switched from DEPLOYING to RUNNING. >>2020-06-22 10:16:34,492 INFO >>org.apache.flink.runtime.executiongraph.ExecutionGraph- Sink: Unnamed >>(1/6) (44ca248aba351026452ba4abdb5f33a6) switched from DEPLOYING to RUNNING. >>2020-06-22 10:16:34,497 INFO >>org.apache.flink.runtime.executiongraph.ExecutionGraph- Filter -> Map >>(1/8) (70612735eb755269fe9590e8ab51d3e2) switched from DEPLOYING to RUNNING. >>2020-06-22 10:16:34,512 INFO >>org.apache.flink.runtime.executiongraph.ExecutionGraph- Filter -> Map >>(2/8) (baba331cd6bcde1f5a6024eac0b953b4) switched from DEPLOYING to RUNNING. >>2020-06-22 10:16:34,524 INFO >>org.apache.flink.runtime.executiongraph.ExecutionGraph- Source: >>Custom Source (4/6) (a34992362c2cf3634a29bd5a9c188754) switched from >>DEPLOYING to RUNNING. >>2020-06-22 10:16:34,531 INFO >>org.apache.flink.runtime.executiongraph.ExecutionGraph- Source: >>Custom Source (2/6) (7683c15e3ebb3e718c2439c6e32f0d7d) switched from >>DEPLOYING to RUNNING. >>2020-06-22 10:16:34,564 INFO >>org.apache.flink.runtime.executiongraph.ExecutionGraph- Source: >>Custom Source (6/6) (6ab1aaa5e1811c79c702197f984e9bb6) switched from >>DEPLOYING to RUNNING. >>2020-06-22 10:16:34,609 INFO >>org.apache.flink.runtime.executiongraph.ExecutionGraph- Source: >>Custom Source (1/6) (6245b508b8e0494f06ef71c6ad4954b6) switched from >>DEPLOYING to RUNNING. >>2020-06-22 10:16:34,616 INFO >>org.apache.flink.runtime.executiongraph.ExecutionGraph- Filter -> Map >>(4/8) (2ea049476f0b48fcd85dcd9084091e9f) switched from DEPLOYING to RUNNING. >>2020-06-22 10:16:34,650 INFO >>org.apache.flink.runtime.executiongraph.ExecutionGraph- Sink: Unnamed >>(2/6) (e785e44e4212dcc5279bcde761b28292) switched from DEPLOYING to RUNNING. >>2020-06-22 10:16:34,656 INFO >>org.apache.flink.runtime.executiongraph.ExecutionGraph- Sink: Unnamed >>(5/6) (97a5e72f731d14aa97d93253b71b6aeb) switched from DEPLOYING to RUNNING. >>2020-06-22 10:16:34,662 INFO >>org.apache.flink.runtime.executiongraph.ExecutionGraph- Filter -> Map >>(7/8) (59babdbc2d7bea362a2794a966fe59ef) switched from DEPLOYING to RUNNING. >>2020-06-22 10:16:34,664 INFO >>org.apache.flink.runtime.executiongraph.ExecutionGraph- Source: >>Custom Source (3/6) (aed33b78cc847561908ea43164e0311a) switched from >>DEPLOYING to RUNNING. >>2020-06-22 10:16:34,669 INFO >>org.apache.flink.runtime.executiongraph.ExecutionGraph- Filter -> Map >>(6/8) (fd87d0111b10e9f9027068a72e9ce209) switched from DEPLOYING to RUNNING. >>2020-06-22 10:16:34,726 INFO >>org.apache.flink.runtime.executiongraph.ExecutionGraph- Sink: Unnamed >>(6/6) (5dd98ec92e0d5e53597cb7520643c7f5) switched from DEPLOYING to RUNNING. >>2020-06-22 10:16:34,729 INFO >>org.apache.flink.runtime.executiongraph.ExecutionGraph- Filter -> Map >>(8/8) (22b73fa7f7435a405e1102a3480c09c1) switched from DEPLOYING to RUNNING. >>2020-06-22 10:16:34,760 INFO >>org.apache.flink.runtime.executiongraph.ExecutionGraph- Sink: Unnamed >>(4/6) (4c677579ef44cf394618af38a75497da) switched from DEPLOYING to RUNNING. >>2020-06-22 10:16:37,081 INFO >>org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Triggering >>checkpoint 1 @ 1592792197072 for job 1797e2f64b7b1caeb6356608290263cc. >>2020-06-22 10:16:45,065 INFO
Re:在线数据质量
一般都会的 在 2020-06-22 10:52:17，"wangxiangyan" 写道： >hi，大家的在线任务会在晚上重跑一次做校正吗？
Re:Re: 【Flink在sink端的Exactly once语义】
需要自己实现比如幂等操作 比如通过表示为操作 在 2020-06-22 10:04:43，"Benchao Li" 写道： >看起来现在只有Kafka实现了TwoPhaseCommitSinkFunction，所以目前应该也只有Kafka支持exactly once。 > >不过像Mysql、ES这种，可以根据主键来更新的，只要能做到at least once应该就可以了。 > >忝忝向仧 <153488...@qq.com> 于2020年6月21日周日 下午11:27写道： > >> Hi,all: >> >> >> Flink连接器这块，如果是sink到mysql,ES等，有对应的实现exactly once语义么? >> 比如kafka的连接有sink的exactly once语义,sink时候指定即可. >> 那么,如果是mysql后者其他的有么? >> 谢谢. >> return new FlinkKafkaProducer011<( >> "topic", >> new KeyedSerializationSchemaWrapper<(new SimpleStringSchema()), >> producerProperties, >> FlinkKafkaProducer011.Semantic.EXACTLY_ONCE); > > > >-- > >Best, >Benchao Li
flink 通过 --classpath 传入https-xxx.xxx.jar 在1.8上正常运行 在flink1.10 就会报传入的jar包找不到
2020-06-22 10:16:34,379 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph- Sink: Unnamed (6/6) (5dd98ec92e0d5e53597cb7520643c7f5) switched from SCHEDULED to DEPLOYING. 2020-06-22 10:16:34,379 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph- Deploying Sink: Unnamed (6/6) (attempt #0) to container_1590655249980_0081_01_02 @ al-bj-bigdata-inf-research-flink04 (dataPort=34781) 2020-06-22 10:16:34,456 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph- Source: Custom Source (5/6) (aca298438b9eb6fcf295cb8af6eebcd8) switched from DEPLOYING to RUNNING. 2020-06-22 10:16:34,481 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph- Filter -> Map (3/8) (0daed15d107c3031891f0c9e84093068) switched from DEPLOYING to RUNNING. 2020-06-22 10:16:34,492 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph- Sink: Unnamed (1/6) (44ca248aba351026452ba4abdb5f33a6) switched from DEPLOYING to RUNNING. 2020-06-22 10:16:34,497 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph- Filter -> Map (1/8) (70612735eb755269fe9590e8ab51d3e2) switched from DEPLOYING to RUNNING. 2020-06-22 10:16:34,512 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph- Filter -> Map (2/8) (baba331cd6bcde1f5a6024eac0b953b4) switched from DEPLOYING to RUNNING. 2020-06-22 10:16:34,524 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph- Source: Custom Source (4/6) (a34992362c2cf3634a29bd5a9c188754) switched from DEPLOYING to RUNNING. 2020-06-22 10:16:34,531 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph- Source: Custom Source (2/6) (7683c15e3ebb3e718c2439c6e32f0d7d) switched from DEPLOYING to RUNNING. 2020-06-22 10:16:34,564 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph- Source: Custom Source (6/6) (6ab1aaa5e1811c79c702197f984e9bb6) switched from DEPLOYING to RUNNING. 2020-06-22 10:16:34,609 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph- Source: Custom Source (1/6) (6245b508b8e0494f06ef71c6ad4954b6) switched from DEPLOYING to RUNNING. 2020-06-22 10:16:34,616 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph- Filter -> Map (4/8) (2ea049476f0b48fcd85dcd9084091e9f) switched from DEPLOYING to RUNNING. 2020-06-22 10:16:34,650 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph- Sink: Unnamed (2/6) (e785e44e4212dcc5279bcde761b28292) switched from DEPLOYING to RUNNING. 2020-06-22 10:16:34,656 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph- Sink: Unnamed (5/6) (97a5e72f731d14aa97d93253b71b6aeb) switched from DEPLOYING to RUNNING. 2020-06-22 10:16:34,662 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph- Filter -> Map (7/8) (59babdbc2d7bea362a2794a966fe59ef) switched from DEPLOYING to RUNNING. 2020-06-22 10:16:34,664 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph- Source: Custom Source (3/6) (aed33b78cc847561908ea43164e0311a) switched from DEPLOYING to RUNNING. 2020-06-22 10:16:34,669 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph- Filter -> Map (6/8) (fd87d0111b10e9f9027068a72e9ce209) switched from DEPLOYING to RUNNING. 2020-06-22 10:16:34,726 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph- Sink: Unnamed (6/6) (5dd98ec92e0d5e53597cb7520643c7f5) switched from DEPLOYING to RUNNING. 2020-06-22 10:16:34,729 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph- Filter -> Map (8/8) (22b73fa7f7435a405e1102a3480c09c1) switched from DEPLOYING to RUNNING. 2020-06-22 10:16:34,760 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph- Sink: Unnamed (4/6) (4c677579ef44cf394618af38a75497da) switched from DEPLOYING to RUNNING. 2020-06-22 10:16:37,081 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Triggering checkpoint 1 @ 1592792197072 for job 1797e2f64b7b1caeb6356608290263cc. 2020-06-22 10:16:45,065 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph- Sink: Unnamed (3/6) (d9a9c913dcbf782bd933b0adae157b38) switched from DEPLOYING to RUNNING. 2020-06-22 10:16:45,066 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph- Filter -> Map (5/8) (39dbdd04e2066d1d93be1641c0ab7add) switched from DEPLOYING to RUNNING. 2020-06-22 10:16:48,512 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph- Filter -> Map (3/8) (0daed15d107c3031891f0c9e84093068) switched from RUNNING to FAILED on org.apache.flink.runtime.jobmaster.slotpool.SingleLogicalSlot@25d7d2c5. java.lang.NoClassDefFoundError: core 看日志是在部分task里面报错
Re:re:提交flink代码 无法创建taskmanager flink web界面一致停留在created状态 ，日志报错如下
是用代码提交的jobmanager 是可以加载的 就是启动taskmanager 这个目录就没有创建， 界面如下 ，错误日志就是我下面贴出来的那个 在 2020-05-30 19:16:57，"462329521" <462329...@qq.com> 写道： >你的提交命令是什么呢看样子是加载不到配置文件 > > >-- 原始邮件 -- >发件人: "程龙"<13162790...@163.com; >发件时间: 2020-05-30 19:13 >收件人: "user-zh"主题: 提交flink代码 无法创建taskmanager flink web界面一致停留在created状态 ，日志报错如下 > > > >2020-05-30 19:07:31,418 INFO org.apache.flink.yarn.YarnTaskExecutorRunner > - > > 2020-05-30 19:07:31,418 INFO org.apache.flink.yarn.YarnTaskExecutorRunner > - Registered UNIX signal handlers for [TERM, HUP, INT] >2020-05-30 19:07:31,420 INFO org.apache.flink.yarn.YarnTaskExecutorRunner > - Current working Directory: >/tmp/hadoop-bjhl/nm-local-dir/usercache/bjhl/appcache/application_1590820026922_0020/container_1590820026922_0020_01_08 > 2020-05-30 19:07:31,427 ERROR org.apache.flink.yarn.YarnTaskExecutorRunner > - YARN TaskManager initialization failed. >org.apache.flink.configuration.IllegalConfigurationException: The Flink config >file >'/tmp/hadoop-bjhl/nm-local-dir/usercache/bjhl/appcache/application_1590820026922_0020/container_1590820026922_0020_01_08/flink-conf.yaml' > >(/tmp/hadoop-bjhl/nm-local-dir/usercache/bjhl/appcache/application_1590820026922_0020/container_1590820026922_0020_01_08) > does not exist. at >org.apache.flink.configuration.GlobalConfiguration.loadConfiguration(GlobalConfiguration.java:112) > at >org.apache.flink.yarn.YarnTaskExecutorRunner.run(YarnTaskExecutorRunner.java:114) > at >org.apache.flink.yarn.YarnTaskExecutorRunner.main(YarnTaskExecutorRunner.java:82)
提交flink代码 无法创建taskmanager flink web界面一致停留在created状态 ，日志报错如下
2020-05-30 19:07:31,418 INFO org.apache.flink.yarn.YarnTaskExecutorRunner - 2020-05-30 19:07:31,418 INFO org.apache.flink.yarn.YarnTaskExecutorRunner - Registered UNIX signal handlers for [TERM, HUP, INT] 2020-05-30 19:07:31,420 INFO org.apache.flink.yarn.YarnTaskExecutorRunner - Current working Directory: /tmp/hadoop-bjhl/nm-local-dir/usercache/bjhl/appcache/application_1590820026922_0020/container_1590820026922_0020_01_08 2020-05-30 19:07:31,427 ERROR org.apache.flink.yarn.YarnTaskExecutorRunner - YARN TaskManager initialization failed. org.apache.flink.configuration.IllegalConfigurationException: The Flink config file '/tmp/hadoop-bjhl/nm-local-dir/usercache/bjhl/appcache/application_1590820026922_0020/container_1590820026922_0020_01_08/flink-conf.yaml' (/tmp/hadoop-bjhl/nm-local-dir/usercache/bjhl/appcache/application_1590820026922_0020/container_1590820026922_0020_01_08) does not exist. at org.apache.flink.configuration.GlobalConfiguration.loadConfiguration(GlobalConfiguration.java:112) at org.apache.flink.yarn.YarnTaskExecutorRunner.run(YarnTaskExecutorRunner.java:114) at org.apache.flink.yarn.YarnTaskExecutorRunner.main(YarnTaskExecutorRunner.java:82)
Re:flink 访问hadoop集群问题
下面的代码是你本地运行的是吗 如果是本地需要最简单的方式 就是把hdfs-site.xml 和core-site.xml 配置文件放到资源目录下
在 2020-05-29 15:06:21，"了不起的盖茨比" <573693...@qq.com> 写道：
>请教大家一个问题 ，
>hadoop服务TestHACluster，可是我用api访问时候，填写了path
>hdfs://TestHACluster/user/flink/test
>就会去访问TestHACluster:8020， 但是我是没有那个端口的，针对这种情况怎么处理下。
>因为这个问题，我后面去操作hive时候也是会提示问题，无法连接TestHACluster:8020
>StreamExecutionEnvironment env =
>StreamExecutionEnvironment.getExecutionEnvironment();
>DataStream
>StreamingFileSinknew Path("hdfs://TestHACluster/user/flink/test"),
>new SimpleStringEncoder.withBucketAssigner(new DateTimeBucketAssigner<())
>.build();
>input.addSink(sink);
>env.execute();
Re:flink写入hbase 报错如下 还会导致反压 任务变慢
这不是flink的问题，之前遇到过相关问题 hbase region分裂的时候 会出现此类问题 你可以看看日志是否hbase region当时正在分裂， 在 2020-05-28 16:57:35，"air23" 写道： 2020-05-28 16:54:23,867 INFO org.apache.hadoop.hbase.client.AsyncRequestFutureImpl - id=2, table=GC_SCHEM:mon1, attempt=7/16, failureCount=427ops, last exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Above memstore limit, regionName=GC_SCHEM:mon1,,1590655288228.89343281db3d32d630482b536933121c., server=zongteng75,60020,1590565532547, memstoreSize=575455635, blockingMemStoreSize=536870912 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:3777) at org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2935) at org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2886) at org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:765) at org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:716) at org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2146) at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:33656) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2191) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:112) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:183) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:163) on zongteng75,60020,1590565532547, tracking started null, retrying after=4037ms, operationsToReplay=427
Re:Re: flink-sql watermark问题
可以先在之前对于long类型的字段转换成Timestmap 类型 再生成watermark 在 2020-05-28 17:00:53，"Benchao Li" 写道： >Hi， > >没太看明白你的问题是什么。目前的确是只支持Timestmap(3)作为事件时间列。 >之所以还不支持long作为事件时间列，主要考虑的是时区问题。但是这个社区也在考虑，可以参考[1] > >[1] https://issues.apache.org/jira/browse/FLINK-16938 > >guaishushu1...@163.com 于2020年5月28日周四 下午4:22写道： > >> flink-1.10 sql只支持 timestamp（3） 类型字段生成watermark >> 但是long这样转换后也可以生成watermark很奇怪？ >> CREATE TABLE user_log ( >> response_size int, >> rowtime BIGINT, >> w_ts as TO_TIMESTAMP(FROM_UNIXTIME(rowtime/1000),'-MM-dd HH:mm:ss'), >> WATERMARK FOR w_ts AS w_ts - INTERVAL '5' SECOND --5秒的延迟 >> ) >> >> >> >> guaishushu1...@163.com >> > > >-- > >Best, >Benchao Li
