[ https://issues.apache.org/jira/browse/FLINK-20505?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17247626#comment-17247626 ]
Zhenqiu Huang edited comment on FLINK-20505 at 12/11/20, 4:41 AM: ------------------------------------------------------------------ [~xintongsong] Sorry for the late response. We use the HTTP filesystem for remote udf for a while. I think the error is probably due to the negative size of file can't be parsed. Should we just change "size=([\\d])" to "size=([\\-?d])" in line 44? private static final Pattern LOCAL_RESOURCE_DESC_FORMAT = Pattern.compile("YarnLocalResourceDescriptor\\{" + "key=(\\S+), path=(\\S+), {color:#DE350B}size=([\\d]+){color}, modificationTime=([\\d]+), visibility=(\\S+), type=(\\S+)}"); was (Author: zhenqiuhuang): [~xintongsong] Sorry for the late response. We use HTTP filesystem for remote udf for a while. I think the error is probably due to the negative size of file can't be parsed. Should we just change "size=([\\d])" to "size=([\\-?d])" in line 44? private static final Pattern LOCAL_RESOURCE_DESC_FORMAT = Pattern.compile("YarnLocalResourceDescriptor\\{" + "key=(\\S+), path=(\\S+), {color:#DE350B}size=([\\d]+){color}, modificationTime=([\\d]+), visibility=(\\S+), type=(\\S+)}"); > Yarn provided lib does not work with http paths. > ------------------------------------------------ > > Key: FLINK-20505 > URL: https://issues.apache.org/jira/browse/FLINK-20505 > Project: Flink > Issue Type: Bug > Components: Deployment / YARN > Affects Versions: 1.12.0, 1.11.2 > Reporter: Xintong Song > Assignee: Xintong Song > Priority: Major > > If an http path is used for provided lib, the following exception will be > thrown on the resource manager side: > {code:java} > 2020-12-04 17:01:28.955 ERROR org.apache.flink.yarn.YarnResourceManager - > Could not start TaskManager in container containerXXXXXX. > org.apache.flink.util.FlinkException: Error to parse > YarnLocalResourceDescriptor from YarnLocalResourceDescriptor{key=XXXXX.jar, > path=https://XXXXXXX.jar, size=-1, modificationTime=0, visibility=APPLICATION} > at > org.apache.flink.yarn.YarnLocalResourceDescriptor.fromString(YarnLocalResourceDescriptor.java:99) > at > org.apache.flink.yarn.Utils.decodeYarnLocalResourceDescriptorListFromString(Utils.java:721) > at org.apache.flink.yarn.Utils.createTaskExecutorContext(Utils.java:626) > at > org.apache.flink.yarn.YarnResourceManager.getOrCreateContainerLaunchContext(YarnResourceManager.java:746) > at > org.apache.flink.yarn.YarnResourceManager.createTaskExecutorLaunchContext(YarnResourceManager.java:726) > at > org.apache.flink.yarn.YarnResourceManager.startTaskExecutorInContainer(YarnResourceManager.java:500) > at > org.apache.flink.yarn.YarnResourceManager.onContainersOfResourceAllocated(YarnResourceManager.java:455) > at > org.apache.flink.yarn.YarnResourceManager.lambda$onContainersAllocated$1(YarnResourceManager.java:415) > {code} > The problem is that, `HttpFileSystem#getFilsStatus` returns file status with > length `-1`, while `YarnLocalResourceDescriptor` does not recognize the > negative file length. -- This message was sent by Atlassian Jira (v8.3.4#803005)