Re: flink sql 使用自定义函数 ，返回嵌套行， 查询报错 scala.MatchError

2021-04-27 文章 Shengkai Fang 
 hi, gen.

近期内应该就会发布，应该是五一左右就会发布1.13的版本。

Best,
Shengkai

gen  于2021年4月27日周二 下午8:57写道：

> hi, Shengkai
>   非常感谢你的解答， 解决了困扰我几天的问题。
>   按照你的建议 ，我使用 今天（2021-4-27） 主干版本，运行正常，发现确实是已经修复的。
>   我之前使用的版本是 1.12.2。
>
>
>   目前最新的release版本是1.12.2 ，应该还没有包含这个修复。不知道你是否了解 1.13的发布计划。
>
>
>
> --
> Sent from: http://apache-flink.147419.n8.nabble.com/

回复：Fwd: flink1.12.2 CLI连接hive出现异常

2021-04-27 文章 guoyb 
我的也有这种问题，没解决，kerberos认证的hive导致的。



---原始邮件---
发件人: "张锴"

Fwd: flink1.12.2 CLI连接hive出现异常

2021-04-27 文章 张锴 
-- Forwarded message -
发件人： 张锴 
Date: 2021年4月27日周二 下午1:59
Subject: flink1.12.2 CLI连接hive出现异常
To: 


*使用flink1.12.2 CLI连接hive时，可以查看到catalog，也能看到指定的catalog下的tables，但是执行select
语句时就出现异常。*
[ERROR] Could not execute SQL statement. Reason:
org.apache.hadoop.ipc.RemoteException: Application with id
'application_1605840182730_29292' doesn't exist in RM. Please check that
the job submission was suc
at
org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.getApplicationReport(ClientRMService.java:382)
at
org.apache.hadoop.yarn.api.impl.pb.service.ApplicationClientProtocolPBServiceImpl.getApplicationReport(ApplicationClientProtocolPBServiceImpl.java:
at
org.apache.hadoop.yarn.proto.ApplicationClientProtocol$ApplicationClientProtocolService$2.callBlockingMethod(ApplicationClientProtocol.java:561)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:524)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1025)
at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:876)
at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:822)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2682)

*使用yarn logs -applicationId  application_1605840182730_29292
查看日志时，并没有给出具体的错误，以下是打出的日志。日志几乎看不出啥问题。*
INFO client.RMProxy: Connecting to ResourceManager at
hadoop01.xxx.xxx.xxx/xx.xx.x.xx:8050
Unable to get ApplicationState. Attempting to fetch logs directly from the
filesystem.
Can not find the appOwner. Please specify the correct appOwner
Could not locate application logs for application_1605840182730_29292

这个如何排查呢，有遇到类似的问题的小伙伴吗

回复：解析kafka 非标准JSON问题

2021-04-27 文章 guoyb 
好的，谢谢！我都试试看



---原始邮件---
发件人: "JasonLee"<17610775...@163.com
发送时间: 2021年4月27日(周二) 晚上7:14
收件人: "user-zh"http://apache-flink.147419.n8.nabble.com/

Re: flink sql 使用自定义函数 ，返回嵌套行， 查询报错 scala.MatchError

2021-04-27 文章 gen 
hi, Shengkai
  非常感谢你的解答， 解决了困扰我几天的问题。
  按照你的建议 ，我使用 今天（2021-4-27） 主干版本，运行正常，发现确实是已经修复的。
  我之前使用的版本是 1.12.2。
  

  目前最新的release版本是1.12.2 ，应该还没有包含这个修复。不知道你是否了解 1.13的发布计划。



--
Sent from: http://apache-flink.147419.n8.nabble.com/

Re: 回复：解析kafka 非标准JSON问题

2021-04-27 文章 JasonLee 
hi

SQL 可以定义一个字段然后分隔再去获取 JSON 数据 或者可以自定义 UDF 去处理



-
Best Wishes
JasonLee
--
Sent from: http://apache-flink.147419.n8.nabble.com/

回复：解析kafka 非标准JSON问题

2021-04-27 文章 guoyb 
SQL能实现吗？



---原始邮件---
发件人: "19971028...@163.com"<19971028...@163.com
发送时间: 2021年4月27日(周二) 晚上6:51
收件人: "user-zh"

Re: 解析kafka 非标准JSON问题

2021-04-27 文章 19971028...@163.com 
可以尝试在kafka 写个拦截器处理json



19971028...@163.com
 
发件人： guoyb
发送时间： 2021-04-27 17:55
收件人： user-zh
主题： 解析kafka 非标准JSON问题
您好！请问遇到如下格式的该如何用SQL解析。
string {name=string} {id : 1, name : abram}
eg:
2021-04-03x{name=aa} {id : 1, name : abram}
最重要为第三个字段 JSON，里面包含了data
第一和第二个字段可要可不要
 
 
请问，有思路可以参考吗？或者demo

解析kafka 非标准JSON问题

2021-04-27 文章 guoyb 
您好！请问遇到如下格式的该如何用SQL解析。
string {name=string} {id : 1, name : abram}
eg:
2021-04-03x{name=aa} {id : 1, name : abram}
最重要为第三个字段 JSON，里面包含了data
第一和第二个字段可要可不要


请问，有思路可以参考吗？或者demo

recovery from savepoint appear java.lang.NullPointerException

2021-04-27 文章 张美玲 
2021-04-2716:19:34
java.lang.Exception: Exceptionwhile creating StreamOperatorStateContext.
at 
org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.streamOperatorStateContext(StreamTaskStateInitializerImpl.java:254)
at 
org.apache.flink.streaming.api.operators.AbstractStreamOperator.initializeState(AbstractStreamOperator.java:272)
at 
org.apache.flink.streaming.runtime.tasks.OperatorChain.initializeStateAndOpenOperators(OperatorChain.java:427)
at 
org.apache.flink.streaming.runtime.tasks.StreamTask.lambda$beforeInvoke$2(StreamTask.java:543)
at 
org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$1.runThrowing(StreamTaskActionExecutor.java:50)
at 
org.apache.flink.streaming.runtime.tasks.StreamTask.beforeInvoke(StreamTask.java:533)
at 
org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:573)
at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:755)
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:570)
at java.base/java.lang.Thread.run(Thread.java:832)
Causedby: org.apache.flink.util.FlinkException: Couldnot restore keyed state 
backend forEvictingWindowOperator_cc588d0c8a499f01bdca964487231660_(1/1) 
fromanyof the 1 provided restore options.
at 
org.apache.flink.streaming.api.operators.BackendRestorerProcedure.createAndRestore(BackendRestorerProcedure.java:160)
at 
org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.keyedStatedBackend(StreamTaskStateInitializerImpl.java:345)
at 
org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.streamOperatorStateContext(StreamTaskStateInitializerImpl.java:163)
... 9 more
Causedby: org.apache.flink.runtime.state.BackendBuildingException: Failedwhen 
trying to restore heap backend
at 
org.apache.flink.runtime.state.heap.HeapKeyedStateBackendBuilder.build(HeapKeyedStateBackendBuilder.java:115)
at 
org.apache.flink.runtime.state.memory.MemoryStateBackend.createKeyedStateBackend(MemoryStateBackend.java:357)
at 
org.apache.flink.runtime.state.memory.MemoryStateBackend.createKeyedStateBackend(MemoryStateBackend.java:104)
at 
org.apache.flink.runtime.state.StateBackend.createKeyedStateBackend(StateBackend.java:181)
at 
org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.lambda$keyedStatedBackend$1(StreamTaskStateInitializerImpl.java:328)
at 
org.apache.flink.streaming.api.operators.BackendRestorerProcedure.attemptCreateAndRestore(BackendRestorerProcedure.java:168)
at 
org.apache.flink.streaming.api.operators.BackendRestorerProcedure.createAndRestore(BackendRestorerProcedure.java:135)
... 11 more
Causedby: java.lang.NullPointerException
at 
org.apache.flink.runtime.state.heap.HeapRestoreOperation.readKeyGroupStateData(HeapRestoreOperation.java:307)
at 
org.apache.flink.runtime.state.heap.HeapRestoreOperation.readStateHandleStateData(HeapRestoreOperation.java:285)
at 
org.apache.flink.runtime.state.heap.HeapRestoreOperation.restore(HeapRestoreOperation.java:172)
at 
org.apache.flink.runtime.state.heap.HeapKeyedStateBackendBuilder.build(HeapKeyedStateBackendBuilder.java:112)
... 17 more

使用processor api生成savepoints，运行后报的错误

2021-04-27 文章 张美玲 
2021-04-2715:59:18
java.lang.Exception: Exceptionwhile creating StreamOperatorStateContext.
at 
org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.streamOperatorStateContext(StreamTaskStateInitializerImpl.java:254)
at 
org.apache.flink.streaming.api.operators.AbstractStreamOperator.initializeState(AbstractStreamOperator.java:272)
at 
org.apache.flink.streaming.runtime.tasks.OperatorChain.initializeStateAndOpenOperators(OperatorChain.java:427)
at 
org.apache.flink.streaming.runtime.tasks.StreamTask.lambda$beforeInvoke$2(StreamTask.java:543)
at 
org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$1.runThrowing(StreamTaskActionExecutor.java:50)
at 
org.apache.flink.streaming.runtime.tasks.StreamTask.beforeInvoke(StreamTask.java:533)
at 
org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:573)
at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:755)
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:570)
at java.base/java.lang.Thread.run(Thread.java:832)
Causedby: org.apache.flink.util.FlinkException: Couldnot restore keyed state 
backend forEvictingWindowOperator_cc588d0c8a499f01bdca964487231660_(1/1) 
fromanyof the 1 provided restore options.
at 
org.apache.flink.streaming.api.operators.BackendRestorerProcedure.createAndRestore(BackendRestorerProcedure.java:160)
at 
org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.keyedStatedBackend(StreamTaskStateInitializerImpl.java:345)
at 
org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.streamOperatorStateContext(StreamTaskStateInitializerImpl.java:163)
... 9 more
Causedby: org.apache.flink.runtime.state.BackendBuildingException: Failedwhen 
trying to restore heap backend
at 
org.apache.flink.runtime.state.heap.HeapKeyedStateBackendBuilder.build(HeapKeyedStateBackendBuilder.java:115)
at 
org.apache.flink.runtime.state.memory.MemoryStateBackend.createKeyedStateBackend(MemoryStateBackend.java:357)
at 
org.apache.flink.runtime.state.memory.MemoryStateBackend.createKeyedStateBackend(MemoryStateBackend.java:104)
at 
org.apache.flink.runtime.state.StateBackend.createKeyedStateBackend(StateBackend.java:181)
at 
org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.lambda$keyedStatedBackend$1(StreamTaskStateInitializerImpl.java:328)
at 
org.apache.flink.streaming.api.operators.BackendRestorerProcedure.attemptCreateAndRestore(BackendRestorerProcedure.java:168)
at 
org.apache.flink.streaming.api.operators.BackendRestorerProcedure.createAndRestore(BackendRestorerProcedure.java:135)
... 11 more
Causedby: java.lang.IllegalStateException: Unexpected key-groupin restore.
at org.apache.flink.util.Preconditions.checkState(Preconditions.java:193)
at 
org.apache.flink.runtime.state.heap.HeapRestoreOperation.readStateHandleStateData(HeapRestoreOperation.java:279)
at 
org.apache.flink.runtime.state.heap.HeapRestoreOperation.restore(HeapRestoreOperation.java:172)
at 
org.apache.flink.runtime.state.heap.HeapKeyedStateBackendBuilder.build(HeapKeyedStateBackendBuilder.java:112)
... 17 more

??????flink cdc ????mysql binlog?????? streaming????????????????????reload????????????????????

2021-04-27 文章 ?????????? 



----
??: 
   "user-zh"

<18814118...@163.com;
:2021??4??27??(??) 3:15
??:"user-zh@flink.apache.org"

??????flink cdc ????mysql binlog?????? streaming????????????????????reload????????????????????

2021-04-27 文章 MOBIN 
datastreaming 
API??debeziumdebezium??
public static Properties debeziumProperties(){
Properties properties = new Properties();
properties.setProperty(????,??");
return properties;
}


SourceFunction sourceFunction = MySQLSource.builder()
. . .
.debeziumProperties(debeziumProperties())
.build();


| |
MOBIN
|
|
18814118...@163.com
|
??


??2021??04??27?? 14:46<1764232...@qq.com> ??
hi all

??flink cdc??streaming 
mode??binlog??mysql??RELOADsqlcdc
debezium.snap.shot.locking.mode = none


streaming 
mode??reload??flink 
cdc??
Properties properties = new Properties();
properties.setProperty("debezium.snapshot.locking.mode", "none");
SourceFunction

flink cdc ????mysql binlog?????? streaming????????????????????reload????????????????????

2021-04-27 文章 ?????????? 
hi all

??flink cdc??streaming 
mode??binlog??mysql??RELOADsqlcdc
debezium.snap.shot.locking.mode = none


streaming 
mode??reload??flink 
cdc??
Properties properties = new Properties();
properties.setProperty("debezium.snapshot.locking.mode", "none");
SourceFunction