[jira] [Updated] (FLINK-20462) MailboxOperatorTest.testAvoidTaskStarvation
[
https://issues.apache.org/jira/browse/FLINK-20462?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Huang Xingbo updated FLINK-20462:
-
Component/s: (was: Tests)
Runtime / Task
> MailboxOperatorTest.testAvoidTaskStarvation
> ---
>
> Key: FLINK-20462
> URL: https://issues.apache.org/jira/browse/FLINK-20462
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Task
>Affects Versions: 1.12.0
>Reporter: Huang Xingbo
>Priority: Major
> Labels: test-stability
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=10450&view=logs&j=f0ac5c25-1168-55a5-07ff-0e88223afed9&t=0dbaca5d-7c38-52e6-f4fe-2fb69ccb3ada
> {code:java}
> [ERROR]
> testAvoidTaskStarvation(org.apache.flink.streaming.runtime.operators.MailboxOperatorTest)
> Time elapsed: 1.142 s <<< FAILURE!
> java.lang.AssertionError:
>
> Expected: is <[0, 2, 4]>
> but: was <[0, 2, 516]>
> at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
> at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:8)
> at
> org.apache.flink.streaming.runtime.operators.MailboxOperatorTest.testAvoidTaskStarvation(MailboxOperatorTest.java:85)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-20462) MailboxOperatorTest.testAvoidTaskStarvation
[
https://issues.apache.org/jira/browse/FLINK-20462?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Huang Xingbo updated FLINK-20462:
-
Labels: test-stability (was: )
> MailboxOperatorTest.testAvoidTaskStarvation
> ---
>
> Key: FLINK-20462
> URL: https://issues.apache.org/jira/browse/FLINK-20462
> Project: Flink
> Issue Type: Bug
> Components: Tests
>Affects Versions: 1.12.0
>Reporter: Huang Xingbo
>Priority: Major
> Labels: test-stability
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=10450&view=logs&j=f0ac5c25-1168-55a5-07ff-0e88223afed9&t=0dbaca5d-7c38-52e6-f4fe-2fb69ccb3ada
> {code:java}
> [ERROR]
> testAvoidTaskStarvation(org.apache.flink.streaming.runtime.operators.MailboxOperatorTest)
> Time elapsed: 1.142 s <<< FAILURE!
> java.lang.AssertionError:
>
> Expected: is <[0, 2, 4]>
> but: was <[0, 2, 516]>
> at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
> at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:8)
> at
> org.apache.flink.streaming.runtime.operators.MailboxOperatorTest.testAvoidTaskStarvation(MailboxOperatorTest.java:85)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink-statefun] tzulitai edited a comment on pull request #178: [FLINK-20303][test] Add a SmokeE2E test
tzulitai edited a comment on pull request #178: URL: https://github.com/apache/flink-statefun/pull/178#issuecomment-737726878 I had tried running this in a loop for 10 times in my local fork's Github Actions CI, and all attempts were successful. So it seems to be stable, and has proven to be useful by catching a few bugs already! Thanks for working on this @igalshilman, I'll proceed to merging this. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink-statefun] tzulitai commented on pull request #178: [FLINK-20303][test] Add a SmokeE2E test
tzulitai commented on pull request #178: URL: https://github.com/apache/flink-statefun/pull/178#issuecomment-737726878 I had tried running this in a loop for 10 times in my local fork's Github Actions CI, and all attempts were successful. This looks good to me, and has proven to be useful by catching a few bugs already! Thanks for working on this @igalshilman, I'll proceed to merging this. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14244: [FLINK-20299][docs-zh] Update Chinese table overview
flinkbot edited a comment on pull request #14244: URL: https://github.com/apache/flink/pull/14244#issuecomment-734665945 ## CI report: * e2d25f123282470094c6ebe4122e04921304cc57 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10227) * 2c6bd4bafadb83252e0fa01b777b44539e320396 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink-statefun] tzulitai commented on pull request #183: [hotfix][sdk] Change variable names to comply with camel case naming rules and correct spelling of wrong words.
tzulitai commented on pull request #183: URL: https://github.com/apache/flink-statefun/pull/183#issuecomment-737726297 Thanks @RocMarshal, merging ... This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink-statefun] tzulitai commented on pull request #182: [FLINK-20334] Introduce module YAML format version 3.0 and function endpoint templating
tzulitai commented on pull request #182: URL: https://github.com/apache/flink-statefun/pull/182#issuecomment-737725858 Thanks for the comments @igalshilman. I addressed your comments with one minor difference: instead of calling it `FunctionTypeNamespaceSelector`, I found that `FunctionTypeNamespaceMatcher` made more sense. Basically, it conveys that if a message's target function matches the provided matcher, then that function provider will be used. The same for a potential `FunctionTypeNamespacePrefixMatcher` in the future. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-15906) physical memory exceeded causing being killed by yarn
[
https://issues.apache.org/jira/browse/FLINK-15906?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17242974#comment-17242974
]
Xintong Song commented on FLINK-15906:
--
The exception suggests that the task manager is consuming more memory than
expected.
A java program may consume various types of memory: heap, direct, native,
metaspace. For all the types, except for native memory, Flink sets explicit
upper limits via JVM parameters, so that an `OutOfMemoryError` will be thrown
if the process tries to use more memory than the limit. Since there's no OOM
thrown, the only possibility is that Flink uses more native memory than it
planned.
Increasing JVM overhead, Flink will reserve more native memory in the
container. The extra memory may not be actually used by JVM as its overhead,
but should help with your problem.
BTW, did it solves your problem?
> physical memory exceeded causing being killed by yarn
> -
>
> Key: FLINK-15906
> URL: https://issues.apache.org/jira/browse/FLINK-15906
> Project: Flink
> Issue Type: Bug
> Components: Deployment / YARN
>Reporter: liupengcheng
>Priority: Major
>
> Recently, we encoutered this issue when testing TPCDS query with 100g data.
> I first meet this issue when I only set the
> `taskmanager.memory.total-process.size` to `4g` with `-tm` option. Then I try
> to increase the jvmOverhead size with following arguments, but still failed.
> {code:java}
> taskmanager.memory.jvm-overhead.min: 640m
> taskmanager.memory.jvm-metaspace: 128m
> taskmanager.memory.task.heap.size: 1408m
> taskmanager.memory.framework.heap.size: 128m
> taskmanager.memory.framework.off-heap.size: 128m
> taskmanager.memory.managed.size: 1408m
> taskmanager.memory.shuffle.max: 256m
> {code}
> {code:java}
> java.lang.Exception: [2020-02-05 11:31:32.345]Container
> [pid=101677,containerID=container_e08_1578903621081_4785_01_51] is
> running 46342144B beyond the 'PHYSICAL' memory limit. Current usage: 4.04 GB
> of 4 GB physical memory used; 17.68 GB of 40 GB virtual memory used. Killing
> container.java.lang.Exception: [2020-02-05 11:31:32.345]Container
> [pid=101677,containerID=container_e08_1578903621081_4785_01_51] is
> running 46342144B beyond the 'PHYSICAL' memory limit. Current usage: 4.04 GB
> of 4 GB physical memory used; 17.68 GB of 40 GB virtual memory used. Killing
> container.Dump of the process-tree for
> container_e08_1578903621081_4785_01_51 : |- PID PPID PGRPID SESSID
> CMD_NAME USER_MODE_TIME(MILLIS) SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES)
> RSSMEM_USAGE(PAGES) FULL_CMD_LINE |- 101938 101677 101677 101677 (java) 25762
> 3571 18867417088 1059157 /opt/soft/openjdk1.8.0/bin/java
> -Dhadoop.root.logfile=syslog -Xmx1610612736 -Xms1610612736
> -XX:MaxDirectMemorySize=402653184 -XX:MaxMetaspaceSize=134217728
> -Dlog.file=/home/work/hdd5/yarn/zjyprc-analysis/nodemanager/application_1578903621081_4785/container_e08_1578903621081_4785_01_51/taskmanager.log
> -Dlog4j.configuration=file:./log4j.properties
> org.apache.flink.yarn.YarnTaskExecutorRunner -D
> taskmanager.memory.shuffle.max=268435456b -D
> taskmanager.memory.framework.off-heap.size=134217728b -D
> taskmanager.memory.framework.heap.size=134217728b -D
> taskmanager.memory.managed.size=1476395008b -D taskmanager.cpu.cores=1.0 -D
> taskmanager.memory.task.heap.size=1476395008b -D
> taskmanager.memory.task.off-heap.size=0b -D
> taskmanager.memory.shuffle.min=268435456b --configDir .
> -Djobmanager.rpc.address=zjy-hadoop-prc-st2805.bj -Dweb.port=0
> -Dweb.tmpdir=/tmp/flink-web-4bf6cd3a-a6e1-4b46-b140-b8ac7bdffbeb
> -Djobmanager.rpc.port=36769 -Dtaskmanager.memory.managed.size=1476395008b
> -Drest.address=zjy-hadoop-prc-st2805.bj |- 101677 101671 101677 101677 (bash)
> 1 1 118030336 733 /bin/bash -c /opt/soft/openjdk1.8.0/bin/java
> -Dhadoop.root.logfile=syslog -Xmx1610612736 -Xms1610612736
> -XX:MaxDirectMemorySize=402653184 -XX:MaxMetaspaceSize=134217728
> -Dlog.file=/home/work/hdd5/yarn/zjyprc-analysis/nodemanager/application_1578903621081_4785/container_e08_1578903621081_4785_01_51/taskmanager.log
> -Dlog4j.configuration=file:./log4j.properties
> org.apache.flink.yarn.YarnTaskExecutorRunner -D
> taskmanager.memory.shuffle.max=268435456b -D
> taskmanager.memory.framework.off-heap.size=134217728b -D
> taskmanager.memory.framework.heap.size=134217728b -D
> taskmanager.memory.managed.size=1476395008b -D taskmanager.cpu.cores=1.0 -D
> taskmanager.memory.task.heap.size=1476395008b -D
> taskmanager.memory.task.off-heap.size=0b -D
> taskmanager.memory.shuffle.min=268435456b --configDir .
> -Djobmanager.rpc.address=zjy-hadoop-prc-st2805.bj -Dweb.port=0
> -Dweb.tmpdir=/tmp/flink-web-4bf6cd3a-a6e1-4b46-b140-b8ac7bdffbeb
> -Djobmanager.rpc.port=36769 -Dtaskmana
[jira] [Updated] (FLINK-20461) YARNFileReplicationITCase.testPerJobModeWithDefaultFileReplication
[
https://issues.apache.org/jira/browse/FLINK-20461?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matthias updated FLINK-20461:
-
Component/s: (was: Tests)
> YARNFileReplicationITCase.testPerJobModeWithDefaultFileReplication
> --
>
> Key: FLINK-20461
> URL: https://issues.apache.org/jira/browse/FLINK-20461
> Project: Flink
> Issue Type: Bug
> Components: Deployment / YARN
>Affects Versions: 1.12.0
>Reporter: Huang Xingbo
>Priority: Major
> Labels: testability
>
> [https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=10450&view=logs&j=fc5181b0-e452-5c8f-68de-1097947f6483&t=62110053-334f-5295-a0ab-80dd7e2babbf]
> {code:java}
> [ERROR]
> testPerJobModeWithDefaultFileReplication(org.apache.flink.yarn.YARNFileReplicationITCase)
> Time elapsed: 32.501 s <<< ERROR! java.io.FileNotFoundException: File does
> not exist:
> hdfs://localhost:46072/user/agent04_azpcontainer/.flink/application_1606950278664_0001/flink-dist_2.11-1.12-SNAPSHOT.jar
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$27.doCall(DistributedFileSystem.java:1441)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$27.doCall(DistributedFileSystem.java:1434)
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1434)
> at
> org.apache.flink.yarn.YARNFileReplicationITCase.extraVerification(YARNFileReplicationITCase.java:148)
> at
> org.apache.flink.yarn.YARNFileReplicationITCase.deployPerJob(YARNFileReplicationITCase.java:113)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-20461) YARNFileReplicationITCase.testPerJobModeWithDefaultFileReplication
[
https://issues.apache.org/jira/browse/FLINK-20461?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matthias updated FLINK-20461:
-
Component/s: Deployment / YARN
> YARNFileReplicationITCase.testPerJobModeWithDefaultFileReplication
> --
>
> Key: FLINK-20461
> URL: https://issues.apache.org/jira/browse/FLINK-20461
> Project: Flink
> Issue Type: Bug
> Components: Deployment / YARN, Tests
>Affects Versions: 1.12.0
>Reporter: Huang Xingbo
>Priority: Major
>
> [https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=10450&view=logs&j=fc5181b0-e452-5c8f-68de-1097947f6483&t=62110053-334f-5295-a0ab-80dd7e2babbf]
> {code:java}
> [ERROR]
> testPerJobModeWithDefaultFileReplication(org.apache.flink.yarn.YARNFileReplicationITCase)
> Time elapsed: 32.501 s <<< ERROR! java.io.FileNotFoundException: File does
> not exist:
> hdfs://localhost:46072/user/agent04_azpcontainer/.flink/application_1606950278664_0001/flink-dist_2.11-1.12-SNAPSHOT.jar
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$27.doCall(DistributedFileSystem.java:1441)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$27.doCall(DistributedFileSystem.java:1434)
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1434)
> at
> org.apache.flink.yarn.YARNFileReplicationITCase.extraVerification(YARNFileReplicationITCase.java:148)
> at
> org.apache.flink.yarn.YARNFileReplicationITCase.deployPerJob(YARNFileReplicationITCase.java:113)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-20461) YARNFileReplicationITCase.testPerJobModeWithDefaultFileReplication
[
https://issues.apache.org/jira/browse/FLINK-20461?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matthias updated FLINK-20461:
-
Labels: testability (was: )
> YARNFileReplicationITCase.testPerJobModeWithDefaultFileReplication
> --
>
> Key: FLINK-20461
> URL: https://issues.apache.org/jira/browse/FLINK-20461
> Project: Flink
> Issue Type: Bug
> Components: Deployment / YARN, Tests
>Affects Versions: 1.12.0
>Reporter: Huang Xingbo
>Priority: Major
> Labels: testability
>
> [https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=10450&view=logs&j=fc5181b0-e452-5c8f-68de-1097947f6483&t=62110053-334f-5295-a0ab-80dd7e2babbf]
> {code:java}
> [ERROR]
> testPerJobModeWithDefaultFileReplication(org.apache.flink.yarn.YARNFileReplicationITCase)
> Time elapsed: 32.501 s <<< ERROR! java.io.FileNotFoundException: File does
> not exist:
> hdfs://localhost:46072/user/agent04_azpcontainer/.flink/application_1606950278664_0001/flink-dist_2.11-1.12-SNAPSHOT.jar
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$27.doCall(DistributedFileSystem.java:1441)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$27.doCall(DistributedFileSystem.java:1434)
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1434)
> at
> org.apache.flink.yarn.YARNFileReplicationITCase.extraVerification(YARNFileReplicationITCase.java:148)
> at
> org.apache.flink.yarn.YARNFileReplicationITCase.deployPerJob(YARNFileReplicationITCase.java:113)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] T0UGH commented on pull request #14244: [FLINK-20299][docs-zh] Update Chinese table overview
T0UGH commented on pull request #14244: URL: https://github.com/apache/flink/pull/14244#issuecomment-737721392 Hi, @HuangXingBo, I rebased the master branch, and fixed the conflict. Would you review it in your free time? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14028: [FLINK-20020][client] Make UnsuccessfulExecutionException part of the JobClient.getJobExecutionResult() contract
flinkbot edited a comment on pull request #14028: URL: https://github.com/apache/flink/pull/14028#issuecomment-725222918 ## CI report: * ab8acbd972d3b45ab254da9d3229bcdacddc7c05 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9869) * a711ce5736230806169b800574dccd387d7b66eb Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10465) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-15906) physical memory exceeded causing being killed by yarn
[
https://issues.apache.org/jira/browse/FLINK-15906?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17242968#comment-17242968
]
yang gang commented on FLINK-15906:
---
Hi [~xintongsong],
please teach me,What is the relationship between this exception
and this configuration option?
thanks
> physical memory exceeded causing being killed by yarn
> -
>
> Key: FLINK-15906
> URL: https://issues.apache.org/jira/browse/FLINK-15906
> Project: Flink
> Issue Type: Bug
> Components: Deployment / YARN
>Reporter: liupengcheng
>Priority: Major
>
> Recently, we encoutered this issue when testing TPCDS query with 100g data.
> I first meet this issue when I only set the
> `taskmanager.memory.total-process.size` to `4g` with `-tm` option. Then I try
> to increase the jvmOverhead size with following arguments, but still failed.
> {code:java}
> taskmanager.memory.jvm-overhead.min: 640m
> taskmanager.memory.jvm-metaspace: 128m
> taskmanager.memory.task.heap.size: 1408m
> taskmanager.memory.framework.heap.size: 128m
> taskmanager.memory.framework.off-heap.size: 128m
> taskmanager.memory.managed.size: 1408m
> taskmanager.memory.shuffle.max: 256m
> {code}
> {code:java}
> java.lang.Exception: [2020-02-05 11:31:32.345]Container
> [pid=101677,containerID=container_e08_1578903621081_4785_01_51] is
> running 46342144B beyond the 'PHYSICAL' memory limit. Current usage: 4.04 GB
> of 4 GB physical memory used; 17.68 GB of 40 GB virtual memory used. Killing
> container.java.lang.Exception: [2020-02-05 11:31:32.345]Container
> [pid=101677,containerID=container_e08_1578903621081_4785_01_51] is
> running 46342144B beyond the 'PHYSICAL' memory limit. Current usage: 4.04 GB
> of 4 GB physical memory used; 17.68 GB of 40 GB virtual memory used. Killing
> container.Dump of the process-tree for
> container_e08_1578903621081_4785_01_51 : |- PID PPID PGRPID SESSID
> CMD_NAME USER_MODE_TIME(MILLIS) SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES)
> RSSMEM_USAGE(PAGES) FULL_CMD_LINE |- 101938 101677 101677 101677 (java) 25762
> 3571 18867417088 1059157 /opt/soft/openjdk1.8.0/bin/java
> -Dhadoop.root.logfile=syslog -Xmx1610612736 -Xms1610612736
> -XX:MaxDirectMemorySize=402653184 -XX:MaxMetaspaceSize=134217728
> -Dlog.file=/home/work/hdd5/yarn/zjyprc-analysis/nodemanager/application_1578903621081_4785/container_e08_1578903621081_4785_01_51/taskmanager.log
> -Dlog4j.configuration=file:./log4j.properties
> org.apache.flink.yarn.YarnTaskExecutorRunner -D
> taskmanager.memory.shuffle.max=268435456b -D
> taskmanager.memory.framework.off-heap.size=134217728b -D
> taskmanager.memory.framework.heap.size=134217728b -D
> taskmanager.memory.managed.size=1476395008b -D taskmanager.cpu.cores=1.0 -D
> taskmanager.memory.task.heap.size=1476395008b -D
> taskmanager.memory.task.off-heap.size=0b -D
> taskmanager.memory.shuffle.min=268435456b --configDir .
> -Djobmanager.rpc.address=zjy-hadoop-prc-st2805.bj -Dweb.port=0
> -Dweb.tmpdir=/tmp/flink-web-4bf6cd3a-a6e1-4b46-b140-b8ac7bdffbeb
> -Djobmanager.rpc.port=36769 -Dtaskmanager.memory.managed.size=1476395008b
> -Drest.address=zjy-hadoop-prc-st2805.bj |- 101677 101671 101677 101677 (bash)
> 1 1 118030336 733 /bin/bash -c /opt/soft/openjdk1.8.0/bin/java
> -Dhadoop.root.logfile=syslog -Xmx1610612736 -Xms1610612736
> -XX:MaxDirectMemorySize=402653184 -XX:MaxMetaspaceSize=134217728
> -Dlog.file=/home/work/hdd5/yarn/zjyprc-analysis/nodemanager/application_1578903621081_4785/container_e08_1578903621081_4785_01_51/taskmanager.log
> -Dlog4j.configuration=file:./log4j.properties
> org.apache.flink.yarn.YarnTaskExecutorRunner -D
> taskmanager.memory.shuffle.max=268435456b -D
> taskmanager.memory.framework.off-heap.size=134217728b -D
> taskmanager.memory.framework.heap.size=134217728b -D
> taskmanager.memory.managed.size=1476395008b -D taskmanager.cpu.cores=1.0 -D
> taskmanager.memory.task.heap.size=1476395008b -D
> taskmanager.memory.task.off-heap.size=0b -D
> taskmanager.memory.shuffle.min=268435456b --configDir .
> -Djobmanager.rpc.address=zjy-hadoop-prc-st2805.bj -Dweb.port=0
> -Dweb.tmpdir=/tmp/flink-web-4bf6cd3a-a6e1-4b46-b140-b8ac7bdffbeb
> -Djobmanager.rpc.port=36769 -Dtaskmanager.memory.managed.size=1476395008b
> -Drest.address=zjy-hadoop-prc-st2805.bj 1>
> /home/work/hdd5/yarn/zjyprc-analysis/nodemanager/application_1578903621081_4785/container_e08_1578903621081_4785_01_51/taskmanager.out
> 2>
> /home/work/hdd5/yarn/zjyprc-analysis/nodemanager/application_1578903621081_4785/container_e08_1578903621081_4785_01_51/taskmanager.err
> {code}
> I suspect there are some leaks or unexpected offheap memory usage.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-20463) flink-1.11.2 -sql cannot ignore exception record
谢波 created FLINK-20463:
--
Summary: flink-1.11.2 -sql cannot ignore exception record
Key: FLINK-20463
URL: https://issues.apache.org/jira/browse/FLINK-20463
Project: Flink
Issue Type: Bug
Components: Table SQL / Runtime
Affects Versions: 1.11.2
Environment: 1.11.2
2.11
Reporter: 谢波
can Flink SQL provide an option to ignore exception record？
I have a table that maps kafka data in json format.
When parsing the exception data, an exception is thrown, but the data is valid
JSON, not a valid record.
{color:#FF}exception data：{"SHEET":[""]}{color}
{color:#FF}my table：{color}
CREATE TABLE offline
(
SHEET ROW (
HEADER MAP < STRING, STRING >,
ITEM ROW (
AMOUNT STRING,
COST STRING,
GOODSID STRING,
SALEVALUE STRING,
SAP_RTMATNR STRING,
SAP_RTPLU STRING,
SERIALID STRING,
SHEETID STRING
) ARRAY,
ITEM5 MAP < STRING, STRING > ARRAY,
ITEM1 MAP < STRING, STRING > ARRAY,
TENDER MAP < STRING, STRING > ARRAY
) ARRAY
)
WITH (
'connector' = 'kafka',
'properties.bootstrap.servers' = 'xxx:9092',
'properties.group.id' = 'realtime.sales.offline.group',
'topic' = 'bms133',
'format' = 'json',
{color:#FF}'json.ignore-parse-errors' = 'true',{color}
'scan.startup.mode' = 'earliest-offset'
);
{color:#FF}exception：{color}
Caused by: java.lang.NullPointerExceptionCaused by:
java.lang.NullPointerException at
org.apache.flink.table.runtime.typeutils.RowDataSerializer.copy(RowDataSerializer.java:116)
at
org.apache.flink.table.runtime.typeutils.RowDataSerializer.copy(RowDataSerializer.java:50)
at
org.apache.flink.table.runtime.typeutils.ArrayDataSerializer.copyGenericArray(ArrayDataSerializer.java:129)
at
org.apache.flink.table.runtime.typeutils.ArrayDataSerializer.copy(ArrayDataSerializer.java:90)
at
org.apache.flink.table.runtime.typeutils.ArrayDataSerializer.copy(ArrayDataSerializer.java:51)
at
org.apache.flink.table.runtime.typeutils.RowDataSerializer.copyRowData(RowDataSerializer.java:156)
at
org.apache.flink.table.runtime.typeutils.RowDataSerializer.copy(RowDataSerializer.java:123)
at
org.apache.flink.table.runtime.typeutils.RowDataSerializer.copy(RowDataSerializer.java:50)
at
org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.pushToOperator(OperatorChain.java:715)
at
org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:692)
at
org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:672)
at
org.apache.flink.streaming.api.operators.CountingOutput.collect(CountingOutput.java:52)
at
org.apache.flink.streaming.api.operators.CountingOutput.collect(CountingOutput.java:30)
at
org.apache.flink.streaming.api.operators.StreamSourceContexts$NonTimestampContext.collect(StreamSourceContexts.java:104)
at
org.apache.flink.streaming.api.operators.StreamSourceContexts$NonTimestampContext.collectWithTimestamp(StreamSourceContexts.java:111)
at
org.apache.flink.streaming.connectors.kafka.internals.AbstractFetcher.emitRecordsWithTimestamps(AbstractFetcher.java:352)
at
org.apache.flink.streaming.connectors.kafka.internal.KafkaFetcher.partitionConsumerRecordsHandler(KafkaFetcher.java:185)
at
org.apache.flink.streaming.connectors.kafka.internal.KafkaFetcher.runFetchLoop(KafkaFetcher.java:141)
at
org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumerBase.run(FlinkKafkaConsumerBase.java:755)
at
org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:100)
at
org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:63)
at
org.apache.flink.streaming.runtime.tasks.SourceStreamTask$LegacySourceFunctionThread.run(SourceStreamTask.java:213)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on pull request #14028: [FLINK-20020][client] Make UnsuccessfulExecutionException part of the JobClient.getJobExecutionResult() contract
flinkbot edited a comment on pull request #14028: URL: https://github.com/apache/flink/pull/14028#issuecomment-725222918 ## CI report: * ab8acbd972d3b45ab254da9d3229bcdacddc7c05 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9869) * a711ce5736230806169b800574dccd387d7b66eb UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] SteNicholas edited a comment on pull request #14028: [FLINK-20020][client] Make UnsuccessfulExecutionException part of the JobClient.getJobExecutionResult() contract
SteNicholas edited a comment on pull request #14028: URL: https://github.com/apache/flink/pull/14028#issuecomment-737624706 > Although the `ProgramInvocationException` is not annotated as `Public` or `PublicEvolving`, due to the fact that it has been here since forever, it seems that in some places we may be handling it specially and even expose it to the users. For example, the `ClientUtils.executeProgram()` is public and in the mailing list in the past there were people saying that they use it in other projects (I think Beam used to use it and I do not know if they still do). > > On a separate note, given that this PR although small it has taken more than 3 weeks and a lot of discussions, I am starting to doubt if we should move forward with it. For example, the question that @tillrohrmann mentioned to have a separate exception for the runtime components and a separate for the client so that we can handle them differently, seems a deeper issue that requires more thought. > > I may be wrong on that, but the discussions seem to be disproportionately long for the size and also the added value of this PR. What do you think? > > BTW sorry for your time @SteNicholas if this does not get merged in the end but when I opened the JIRA I did not have a clear view of all the related changes that would pop up. @kl0u , I have followed with the `KafkaConsumerTestBase` comment from @tillrohrmann and pushed a commit for this. And I really want to follow the remaining comments from @tillrohrmann , and continue to update the commit according to Till's above comments. It's better for me to merge this pull request in the end. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-20451) Taskmanager can't allocate resources when submit job with configs `env.java.opts.taskmanager: -javaagent:`
[
https://issues.apache.org/jira/browse/FLINK-20451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17242957#comment-17242957
]
Xintong Song commented on FLINK-20451:
--
For public visibility:
[~xiaozilong] reached out to me offline. We discovered that the problem is
caused by improper path in the `-javaagent` parameter, leading to TM start-up
failure.
> Taskmanager can't allocate resources when submit job with configs
> `env.java.opts.taskmanager: -javaagent:`
> --
>
> Key: FLINK-20451
> URL: https://issues.apache.org/jira/browse/FLINK-20451
> Project: Flink
> Issue Type: Bug
> Components: Deployment / YARN
>Affects Versions: 1.11.0
>Reporter: xiaozilong
>Priority: Major
> Attachments: image-2020-12-02-20-24-57-809.png
>
>
> Hi, I meet an unexpected problem that the taskmanager can't apply resources
> when submit job with configs `env.java.opts.taskmanager: -javaagent` in
> flink-conf.yaml.
> This my configuration:
> {code:java}
> env.java.opts: -Djava.net.preferIPv4Stack=true
> -Djob.name=monitoring_demo_64648b66-1923-4b2e-825b-a970a2b6ecf8
> env.java.opts.taskmanager:
> -javaagent:/data/opt/file_tmp/jvm-profiler-1.0.0.jar=sampleInterval=50{code}
> Partial log:
> !image-2020-12-02-20-24-57-809.png!
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] wenlong88 commented on a change in pull request #14293: [Flink 20437][table-planner-blink] Move the utility methods in ExecNode into ExecNodeUtil & Port ExecNode to Java
wenlong88 commented on a change in pull request #14293:
URL: https://github.com/apache/flink/pull/14293#discussion_r534735056
##
File path:
flink-table/flink-table-planner-blink/src/main/java/org/apache/flink/table/planner/plan/nodes/exec/utils/ExecNodeUtil.java
##
@@ -0,0 +1,89 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.planner.plan.nodes.exec.utils;
+
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.dag.Transformation;
+import org.apache.flink.core.memory.ManagedMemoryUseCase;
+import org.apache.flink.streaming.api.operators.StreamOperatorFactory;
+import org.apache.flink.streaming.api.transformations.OneInputTransformation;
+import org.apache.flink.streaming.api.transformations.TwoInputTransformation;
+import org.apache.flink.table.api.TableException;
+import org.apache.flink.table.planner.plan.nodes.exec.ExecNode;
+
+import java.util.Optional;
+
+/**
+ * An Utility class that helps translating {@link ExecNode} to {@link
Transformation}.
+ */
+public class ExecNodeUtil {
Review comment:
use TransformationUtil? I don't see any relation with ExecNode.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Closed] (FLINK-20451) Taskmanager can't allocate resources when submit job with configs `env.java.opts.taskmanager: -javaagent:`
[
https://issues.apache.org/jira/browse/FLINK-20451?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
xiaozilong closed FLINK-20451.
--
Resolution: Not A Problem
> Taskmanager can't allocate resources when submit job with configs
> `env.java.opts.taskmanager: -javaagent:`
> --
>
> Key: FLINK-20451
> URL: https://issues.apache.org/jira/browse/FLINK-20451
> Project: Flink
> Issue Type: Bug
> Components: Deployment / YARN
>Affects Versions: 1.11.0
>Reporter: xiaozilong
>Priority: Major
> Attachments: image-2020-12-02-20-24-57-809.png
>
>
> Hi, I meet an unexpected problem that the taskmanager can't apply resources
> when submit job with configs `env.java.opts.taskmanager: -javaagent` in
> flink-conf.yaml.
> This my configuration:
> {code:java}
> env.java.opts: -Djava.net.preferIPv4Stack=true
> -Djob.name=monitoring_demo_64648b66-1923-4b2e-825b-a970a2b6ecf8
> env.java.opts.taskmanager:
> -javaagent:/data/opt/file_tmp/jvm-profiler-1.0.0.jar=sampleInterval=50{code}
> Partial log:
> !image-2020-12-02-20-24-57-809.png!
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on pull request #14291: [FLINK-20431][connector/kafka] Fix the offsets commit in KafkaSourceReader when the SplitFetcher shuts down.
flinkbot edited a comment on pull request #14291: URL: https://github.com/apache/flink/pull/14291#issuecomment-737574351 ## CI report: * 5893b7cc51aae55b09105724aa5be69d92f15266 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10455) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14293: [Flink 20437][table-planner-blink] Move the utility methods in ExecNode into ExecNodeUtil & Port ExecNode to Java
flinkbot edited a comment on pull request #14293: URL: https://github.com/apache/flink/pull/14293#issuecomment-737686417 ## CI report: * a7b28dca113de5b753a4b1d57af6c3846f1f4e9e Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10464) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #14293: [Flink 20437][table-planner-blink] Move the utility methods in ExecNode into ExecNodeUtil & Port ExecNode to Java
flinkbot commented on pull request #14293: URL: https://github.com/apache/flink/pull/14293#issuecomment-737686417 ## CI report: * a7b28dca113de5b753a4b1d57af6c3846f1f4e9e UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14229: [FLINK-20292][doc] Improve the document about table formats overlap in user fat jar
flinkbot edited a comment on pull request #14229: URL: https://github.com/apache/flink/pull/14229#issuecomment-734175831 ## CI report: * b6cb320ef6f35f097a3ddf721d68929802a46a0f Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10463) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13964: [FLINK-19314][coordination] Add DeclarativeSlotPoolBridge
flinkbot edited a comment on pull request #13964: URL: https://github.com/apache/flink/pull/13964#issuecomment-723100396 ## CI report: * b2fc4902b25f60f403bf5839ca717b465f09b67a Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10452) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #14293: [Flink 20437][table-planner-blink] Move the utility methods in ExecNode into ExecNodeUtil & Port ExecNode to Java
flinkbot commented on pull request #14293: URL: https://github.com/apache/flink/pull/14293#issuecomment-737679257 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit a7b28dca113de5b753a4b1d57af6c3846f1f4e9e (Thu Dec 03 05:40:50 UTC 2020) **Warnings:** * No documentation files were touched! Remember to keep the Flink docs up to date! * **Invalid pull request title: No valid Jira ID provided** Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] godfreyhe opened a new pull request #14293: [Flink 20437][table-planner-blink] Move the utility methods in ExecNode into ExecNodeUtil & Port ExecNode to Java
godfreyhe opened a new pull request #14293: URL: https://github.com/apache/flink/pull/14293 ## What is the purpose of the change *Move the utility methods in ExecNode into ExecNodeUtil & Port ExecNode to Java* ## Brief change log - *Move the utility methods in ExecNode into ExecNodeUtil* - *Port ExecNode to Java* ## Verifying this change *(Please pick either of the following options)* This change is a refactor rework with existing test coverage. ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (yes / **no**) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (yes / **no**) - The serializers: (yes / **no** / don't know) - The runtime per-record code paths (performance sensitive): (yes / **no** / don't know) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn/Mesos, ZooKeeper: (yes / **no** / don't know) - The S3 file system connector: (yes / **no** / don't know) ## Documentation - Does this pull request introduce a new feature? (yes / **no**) - If yes, how is the feature documented? (not applicable / docs / JavaDocs / **not documented**) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Assigned] (FLINK-20437) Port ExecNode to Java
[ https://issues.apache.org/jira/browse/FLINK-20437?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] godfrey he reassigned FLINK-20437: -- Assignee: godfrey he > Port ExecNode to Java > - > > Key: FLINK-20437 > URL: https://issues.apache.org/jira/browse/FLINK-20437 > Project: Flink > Issue Type: Sub-task > Components: Table SQL / Planner >Reporter: godfrey he >Assignee: godfrey he >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-20462) MailboxOperatorTest.testAvoidTaskStarvation
Huang Xingbo created FLINK-20462:
Summary: MailboxOperatorTest.testAvoidTaskStarvation
Key: FLINK-20462
URL: https://issues.apache.org/jira/browse/FLINK-20462
Project: Flink
Issue Type: Bug
Components: Tests
Affects Versions: 1.12.0
Reporter: Huang Xingbo
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=10450&view=logs&j=f0ac5c25-1168-55a5-07ff-0e88223afed9&t=0dbaca5d-7c38-52e6-f4fe-2fb69ccb3ada
{code:java}
[ERROR]
testAvoidTaskStarvation(org.apache.flink.streaming.runtime.operators.MailboxOperatorTest)
Time elapsed: 1.142 s <<< FAILURE!
java.lang.AssertionError:
Expected: is <[0, 2, 4]>
but: was <[0, 2, 516]>
at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:8)
at
org.apache.flink.streaming.runtime.operators.MailboxOperatorTest.testAvoidTaskStarvation(MailboxOperatorTest.java:85)
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-20461) YARNFileReplicationITCase.testPerJobModeWithDefaultFileReplication
Huang Xingbo created FLINK-20461:
Summary:
YARNFileReplicationITCase.testPerJobModeWithDefaultFileReplication
Key: FLINK-20461
URL: https://issues.apache.org/jira/browse/FLINK-20461
Project: Flink
Issue Type: Bug
Components: Tests
Affects Versions: 1.12.0
Reporter: Huang Xingbo
[https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=10450&view=logs&j=fc5181b0-e452-5c8f-68de-1097947f6483&t=62110053-334f-5295-a0ab-80dd7e2babbf]
{code:java}
[ERROR]
testPerJobModeWithDefaultFileReplication(org.apache.flink.yarn.YARNFileReplicationITCase)
Time elapsed: 32.501 s <<< ERROR! java.io.FileNotFoundException: File does not
exist:
hdfs://localhost:46072/user/agent04_azpcontainer/.flink/application_1606950278664_0001/flink-dist_2.11-1.12-SNAPSHOT.jar
at
org.apache.hadoop.hdfs.DistributedFileSystem$27.doCall(DistributedFileSystem.java:1441)
at
org.apache.hadoop.hdfs.DistributedFileSystem$27.doCall(DistributedFileSystem.java:1434)
at
org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at
org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1434)
at
org.apache.flink.yarn.YARNFileReplicationITCase.extraVerification(YARNFileReplicationITCase.java:148)
at
org.apache.flink.yarn.YARNFileReplicationITCase.deployPerJob(YARNFileReplicationITCase.java:113)
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-19970) State leak in CEP Operators (expired events/keys not removed from state)
[
https://issues.apache.org/jira/browse/FLINK-19970?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17242914#comment-17242914
]
Jiayi Liao commented on FLINK-19970:
[~dwysakowicz] Does this bug affect all the patterns？
> State leak in CEP Operators (expired events/keys not removed from state)
>
>
> Key: FLINK-19970
> URL: https://issues.apache.org/jira/browse/FLINK-19970
> Project: Flink
> Issue Type: Bug
> Components: Library / CEP
>Affects Versions: 1.11.2
> Environment: Flink 1.11.2 run using the official docker containers in
> AWS ECS Fargate.
> 1 Job Manager, 1 Taskmanager with 2vCPUs and 8GB memory
>Reporter: Thomas Wozniakowski
>Priority: Critical
> Attachments: image-2020-11-04-11-35-12-126.png
>
>
> We have been observing instability in our production environment recently,
> seemingly related to state backends. We ended up building a load testing
> environment to isolate factors and have discovered that the CEP library
> appears to have some serious problems with state expiry.
> h2. Job Topology
> Source: Kinesis (standard connector) -> keyBy() and forward to...
> CEP: Array of simple Keyed CEP Pattern operators (details below) -> forward
> output to...
> Sink: SQS (custom connector)
> The CEP Patterns in the test look like this:
> {code:java}
> Pattern.begin(SCANS_SEQUENCE, AfterMatchSkipStrategy.skipPastLastEvent())
> .times(20)
> .subtype(ScanEvent.class)
> .within(Duration.minutes(30));
> {code}
> h2. Taskmanager Config
> {code:java}
> taskmanager.numberOfTaskSlots: $numberOfTaskSlots
> taskmanager.data.port: 6121
> taskmanager.rpc.port: 6122
> taskmanager.exit-on-fatal-akka-error: true
> taskmanager.memory.process.size: $memoryProcessSize
> taskmanager.memory.jvm-metaspace.size: 256m
> taskmanager.memory.managed.size: 0m
> jobmanager.rpc.port: 6123
> blob.server.port: 6130
> rest.port: 8081
> web.submit.enable: true
> fs.s3a.connection.maximum: 50
> fs.s3a.threads.max: 50
> akka.framesize: 250m
> akka.watch.threshold: 14
> state.checkpoints.dir: s3://$savepointBucketName/checkpoints
> state.savepoints.dir: s3://$savepointBucketName/savepoints
> state.backend: filesystem
> state.backend.async: true
> s3.access-key: $s3AccessKey
> s3.secret-key: $s3SecretKey
> {code}
> (the substitutions are controlled by terraform).
> h2. Tests
> h4. Test 1 (No key rotation)
> 8192 actors (different keys) emitting 1 Scan Event every 10 minutes
> indefinitely. Actors (keys) never rotate in or out.
> h4. Test 2 (Constant key rotation)
> 8192 actors that produce 2 Scan events 10 minutes apart, then retire and
> never emit again. The setup creates new actors (keys) as soon as one finishes
> so we always have 8192. This test basically constantly rotates the key space.
> h2. Results
> For both tests, the state size (checkpoint size) grows unbounded and linearly
> well past the 30 minute threshold that should have caused old keys or events
> to be discard from the state. In the chart below, the left (steep) half is
> the 24 hours we ran Test 1, the right (shallow) half is Test 2. My
> understanding is that the checkpoint size should level off after ~45 minutes
> or so then stay constant.
> !image-2020-11-04-11-35-12-126.png!
> Could someone please assist us with this? Unless we have dramatically
> misunderstood how the CEP library is supposed to function this seems like a
> pretty severe bug.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on pull request #14292: [FLINK-20456][docs] Make streaming SQL concepts more approachable
flinkbot edited a comment on pull request #14292: URL: https://github.com/apache/flink/pull/14292#issuecomment-737644985 ## CI report: * dd2905f71cb565e5fd513217a056fa2d757f84e7 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10461) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14229: [FLINK-20292][doc] Improve the document about table formats overlap in user fat jar
flinkbot edited a comment on pull request #14229: URL: https://github.com/apache/flink/pull/14229#issuecomment-734175831 ## CI report: * 1fb988b0bded589b2092ec6664d995c5016e9ba6 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10398) * b6cb320ef6f35f097a3ddf721d68929802a46a0f Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10463) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13722: [FLINK-19636][coordination] Add DeclarativeSlotPool
flinkbot edited a comment on pull request #13722: URL: https://github.com/apache/flink/pull/13722#issuecomment-713509789 ## CI report: * 7cdd555313da89f3b6be3da396e6782460a482d8 UNKNOWN * 948c92c9a9ea9f6f31b0e0b005e9373b9a897a71 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10451) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14229: [FLINK-20292][doc] Improve the document about table formats overlap in user fat jar
flinkbot edited a comment on pull request #14229: URL: https://github.com/apache/flink/pull/14229#issuecomment-734175831 ## CI report: * 1fb988b0bded589b2092ec6664d995c5016e9ba6 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10398) * b6cb320ef6f35f097a3ddf721d68929802a46a0f UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14120: [FLINK-19984][core] Add TypeSerializerTestCoverageTest to check whether tests based on SerializerTestBase and TypeSerializerUpgradeTe
flinkbot edited a comment on pull request #14120: URL: https://github.com/apache/flink/pull/14120#issuecomment-729644511 ## CI report: * 0d7a9c59ca7752c8b8da941da0095139fe587d3a Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10174) * 340a6c872bbf65b85c1eaaaf7399b1cec764152a Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10462) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (FLINK-20460) Support async lookup for HBase connector
Jark Wu created FLINK-20460:
---
Summary: Support async lookup for HBase connector
Key: FLINK-20460
URL: https://issues.apache.org/jira/browse/FLINK-20460
Project: Flink
Issue Type: New Feature
Components: Connectors / HBase, Table SQL / Ecosystem
Reporter: Jark Wu
Currenlty, {{HBaseRowDataLookupFunction}} implements {{TableFunction}} which is
a sync operation. Would be better to have an {{AsyncTableFunction}}
implementation which has better performance.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on pull request #14120: [FLINK-19984][core] Add TypeSerializerTestCoverageTest to check whether tests based on SerializerTestBase and TypeSerializerUpgradeTe
flinkbot edited a comment on pull request #14120: URL: https://github.com/apache/flink/pull/14120#issuecomment-729644511 ## CI report: * 0d7a9c59ca7752c8b8da941da0095139fe587d3a Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10174) * 340a6c872bbf65b85c1eaaaf7399b1cec764152a UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] leonardBang commented on pull request #14229: [FLINK-20292][doc] Improve the document about table formats overlap in user fat jar
leonardBang commented on pull request #14229: URL: https://github.com/apache/flink/pull/14229#issuecomment-737655202 @gaoyunhaii @JingsongLi Updated as discussed This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Closed] (FLINK-20459) Handle null message correctly in JSON RowData deserializer
[ https://issues.apache.org/jira/browse/FLINK-20459?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jark Wu closed FLINK-20459. --- Resolution: Duplicate Let's fix this in FLINK-20321. > Handle null message correctly in JSON RowData deserializer > -- > > Key: FLINK-20459 > URL: https://issues.apache.org/jira/browse/FLINK-20459 > Project: Flink > Issue Type: Bug > Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile) >Reporter: Qingsheng Ren >Priority: Major > > Currently if the input message of the `deserialize()` method is null and the > deserializer is not ignoring parse errors, `new String(message)` in the > exception part will lead to an NPE. > > A possible solution is returning null directly if the input message is null, > but I'm not sure if this is a correct behavior since a null message can be > treated as a "corrupted" message which cannot be deserialized as JSON and > should lead to an exception. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-20451) Taskmanager can't allocate resources when submit job with configs `env.java.opts.taskmanager: -javaagent:`
[
https://issues.apache.org/jira/browse/FLINK-20451?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
xiaozilong updated FLINK-20451:
---
Summary: Taskmanager can't allocate resources when submit job with configs
`env.java.opts.taskmanager: -javaagent:` (was: Taskmanager can't apply
resources when submit job with configs `env.java.opts.taskmanager: -javaagent:`)
> Taskmanager can't allocate resources when submit job with configs
> `env.java.opts.taskmanager: -javaagent:`
> --
>
> Key: FLINK-20451
> URL: https://issues.apache.org/jira/browse/FLINK-20451
> Project: Flink
> Issue Type: Bug
> Components: Deployment / YARN
>Affects Versions: 1.11.0
>Reporter: xiaozilong
>Priority: Major
> Attachments: image-2020-12-02-20-24-57-809.png
>
>
> Hi, I meet an unexpected problem that the taskmanager can't apply resources
> when submit job with configs `env.java.opts.taskmanager: -javaagent` in
> flink-conf.yaml.
> This my configuration:
> {code:java}
> env.java.opts: -Djava.net.preferIPv4Stack=true
> -Djob.name=monitoring_demo_64648b66-1923-4b2e-825b-a970a2b6ecf8
> env.java.opts.taskmanager:
> -javaagent:/data/opt/file_tmp/jvm-profiler-1.0.0.jar=sampleInterval=50{code}
> Partial log:
> !image-2020-12-02-20-24-57-809.png!
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-20451) Taskmanager can't apply resources when submit job with configs `env.java.opts.taskmanager: -javaagent:`
[
https://issues.apache.org/jira/browse/FLINK-20451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17242892#comment-17242892
]
xiaozilong commented on FLINK-20451:
Hi [~xintongsong],
The problem is that TM cannot allocate resources and the job is always in the
scheduled state. If remove this configuration, the job runs fine. The
screenshot is the last log of jobmanager, it looks like applying for resources
but unsuccessful. Could you reproduce it?
> Taskmanager can't apply resources when submit job with configs
> `env.java.opts.taskmanager: -javaagent:`
> ---
>
> Key: FLINK-20451
> URL: https://issues.apache.org/jira/browse/FLINK-20451
> Project: Flink
> Issue Type: Bug
> Components: Deployment / YARN
>Affects Versions: 1.11.0
>Reporter: xiaozilong
>Priority: Major
> Attachments: image-2020-12-02-20-24-57-809.png
>
>
> Hi, I meet an unexpected problem that the taskmanager can't apply resources
> when submit job with configs `env.java.opts.taskmanager: -javaagent` in
> flink-conf.yaml.
> This my configuration:
> {code:java}
> env.java.opts: -Djava.net.preferIPv4Stack=true
> -Djob.name=monitoring_demo_64648b66-1923-4b2e-825b-a970a2b6ecf8
> env.java.opts.taskmanager:
> -javaagent:/data/opt/file_tmp/jvm-profiler-1.0.0.jar=sampleInterval=50{code}
> Partial log:
> !image-2020-12-02-20-24-57-809.png!
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-20459) Handle null message correctly in JSON RowData deserializer
Qingsheng Ren created FLINK-20459: - Summary: Handle null message correctly in JSON RowData deserializer Key: FLINK-20459 URL: https://issues.apache.org/jira/browse/FLINK-20459 Project: Flink Issue Type: Bug Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile) Reporter: Qingsheng Ren Currently if the input message of the `deserialize()` method is null and the deserializer is not ignoring parse errors, `new String(message)` in the exception part will lead to an NPE. A possible solution is returning null directly if the input message is null, but I'm not sure if this is a correct behavior since a null message can be treated as a "corrupted" message which cannot be deserialized as JSON and should lead to an exception. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-20424) The percent of acknowledged checkpoint seems incorrect
[ https://issues.apache.org/jira/browse/FLINK-20424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17242891#comment-17242891 ] Andrew.D.lin commented on FLINK-20424: -- I discovered this problem before, and I think it is more appropriate to keep the percentage to two decimal places. Can i take it？ > The percent of acknowledged checkpoint seems incorrect > -- > > Key: FLINK-20424 > URL: https://issues.apache.org/jira/browse/FLINK-20424 > Project: Flink > Issue Type: Improvement > Components: Runtime / Web Frontend >Reporter: zlzhang0122 >Priority: Minor > Attachments: 2020-11-30 14-18-34 的屏幕截图.png > > > As the picture below, the percent of acknowledged checkpoint seems > incorrect.I think the number must not be 100% because one of the checkpoint > acknowledge was failed. > !2020-11-30 14-18-34 的屏幕截图.png! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on pull request #14292: [FLINK-20456][docs] Make streaming SQL concepts more approachable
flinkbot edited a comment on pull request #14292: URL: https://github.com/apache/flink/pull/14292#issuecomment-737644985 ## CI report: * dd2905f71cb565e5fd513217a056fa2d757f84e7 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10461) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] gaoyunhaii edited a comment on pull request #14229: [FLINK-20292][doc] Improve the document about table formats overlap in user fat jar
gaoyunhaii edited a comment on pull request #14229: URL: https://github.com/apache/flink/pull/14229#issuecomment-737646572 Ok, got it, if building an uber-jar is not necessary for SQL user, then I think it should be ok to not add the note. Very thanks for the information! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] gaoyunhaii edited a comment on pull request #14229: [FLINK-20292][doc] Improve the document about table formats overlap in user fat jar
gaoyunhaii edited a comment on pull request #14229: URL: https://github.com/apache/flink/pull/14229#issuecomment-737646572 Ok, got it, if building an uber-jar is not necessary for SQL user, then I think it might also be ok to not add the note. Very thanks for the information! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] gaoyunhaii commented on pull request #14229: [FLINK-20292][doc] Improve the document about table formats overlap in user fat jar
gaoyunhaii commented on pull request #14229: URL: https://github.com/apache/flink/pull/14229#issuecomment-737646572 Ok, got it, if building an uber-jar is not necessary or not commonly used for SQL user, then I think it might also be ok to not add the note. Very thanks for the information! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #14292: [FLINK-20456][docs] Make streaming SQL concepts more approachable
flinkbot commented on pull request #14292: URL: https://github.com/apache/flink/pull/14292#issuecomment-737644985 ## CI report: * dd2905f71cb565e5fd513217a056fa2d757f84e7 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Comment Edited] (FLINK-20447) Querying grouy by PK does not work
[
https://issues.apache.org/jira/browse/FLINK-20447?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17242886#comment-17242886
]
Zhenwei Feng edited comment on FLINK-20447 at 12/3/20, 3:38 AM:
Hi Jark,
After I reconsidered the problem, I agree with you. SQL like "select * from
table group by pk" is meaningless to some extent. Actually what I'd like to
achieve is to create a table in which the data would be updated by PK
continously(a snapshot table).
I've found the new feature 'Upsert Kafka' in the doc v1.12, which may solve my
problem.
was (Author: bulterman):
Hi Jark,
After I reconsidered the problem, I agree with you. SQL like "select * from
table group by pk" is meaningless to some extent. Actually what I'd like to
achieve is to create a table in which the data would be updated by PK
continously(a snapshot table).
I've found the new pattern 'Upsert Kafka' in the doc v1.12, which may solve my
problem.
> Querying grouy by PK does not work
> --
>
> Key: FLINK-20447
> URL: https://issues.apache.org/jira/browse/FLINK-20447
> Project: Flink
> Issue Type: Improvement
> Components: Table SQL / API
>Affects Versions: 1.11.2
>Reporter: Zhenwei Feng
>Priority: Major
>
> Since PRIMARY KEY is unique, it should be feasible to screen columns by PK.
> The problem could be reproduced by creating a simple table:
> {code:java}
> CREATE TABLE test_table(
> Code STRING,
> Name STRING,
> ...,
> PRIMARY KEY (Code) NOT ENFORCED
> )WITH (...)
> {code}
> then parsing a SQL statement `SELECT *FROM test_table GROUP BY Code`. An
> exception as below will be thrown:
>
> {code:java}
> org.apache.calcite.sql.validate.SqlValidatorException: Expression
> 'test_table.Name' is not being grouped
> {code}
>
>
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] leonardBang commented on pull request #14229: [FLINK-20292][doc] Improve the document about table formats overlap in user fat jar
leonardBang commented on pull request #14229: URL: https://github.com/apache/flink/pull/14229#issuecomment-737644485 I think we need add a section for connector/index page. And do we to add NOTE for every connector and format ? I tend to do not, building an uber-jar is not necessary for SQL user, many user case is downloading the connector jar and format jar directly, if user's project contains multiple connectors/formats, I think user will read connectors/formats matrix in index page. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #14292: [FLINK-20456][docs] Make streaming SQL concepts more approachable
flinkbot commented on pull request #14292: URL: https://github.com/apache/flink/pull/14292#issuecomment-737642096 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit dd2905f71cb565e5fd513217a056fa2d757f84e7 (Thu Dec 03 03:28:45 UTC 2020) ✅no warnings Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] sjwiesman commented on pull request #14292: [FLINK-20456][docs] Make streaming SQL concepts more approachable
sjwiesman commented on pull request #14292: URL: https://github.com/apache/flink/pull/14292#issuecomment-737641638 @twalthr @wuchong @morsapaes @alpinegizmo @knaufk I think all of you would have good opinions on these changes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-20456) Make streaming SQL concepts more approachable
[ https://issues.apache.org/jira/browse/FLINK-20456?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-20456: --- Labels: pull-request-available (was: ) > Make streaming SQL concepts more approachable > - > > Key: FLINK-20456 > URL: https://issues.apache.org/jira/browse/FLINK-20456 > Project: Flink > Issue Type: Improvement > Components: Documentation, Table SQL / Ecosystem >Reporter: Seth Wiesman >Assignee: Seth Wiesman >Priority: Major > Labels: pull-request-available > Fix For: 1.12.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] sjwiesman opened a new pull request #14292: [FLINK-20456][docs] Make streaming SQL concepts more approachable
sjwiesman opened a new pull request #14292: URL: https://github.com/apache/flink/pull/14292 ## What is the purpose of the change The streaming concepts section of the documentation is very negative and focuses on the difficulties of SQL. While we don't want to lie to users about edge cases, or things that may be tricky, I think we can make the experience of getting started more approachable. ## Brief change log See commit messages. ## Verifying this change This change is a trivial rework / code cleanup without any test coverage. *(or)* This change is already covered by existing tests, such as *(please describe tests)*. ## Does this pull request potentially affect one of the following parts: No ## Documentation - Does this pull request introduce a new feature? (yes / **no**) - If yes, how is the feature documented? (not applicable / **docs** / JavaDocs / not documented) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-20447) Querying grouy by PK does not work
[
https://issues.apache.org/jira/browse/FLINK-20447?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17242886#comment-17242886
]
Zhenwei Feng commented on FLINK-20447:
--
Hi Jark,
After I reconsidered the problem, I agree with you. SQL like "select * from
table group by pk" is meaningless to some extent. Actually what I'd like to
achieve is to create a table in which the data would be updated by PK
continously(a snapshot table).
I've found the new pattern 'Upsert Kafka' in the doc v1.12, which may solve my
problem.
> Querying grouy by PK does not work
> --
>
> Key: FLINK-20447
> URL: https://issues.apache.org/jira/browse/FLINK-20447
> Project: Flink
> Issue Type: Improvement
> Components: Table SQL / API
>Affects Versions: 1.11.2
>Reporter: Zhenwei Feng
>Priority: Major
>
> Since PRIMARY KEY is unique, it should be feasible to screen columns by PK.
> The problem could be reproduced by creating a simple table:
> {code:java}
> CREATE TABLE test_table(
> Code STRING,
> Name STRING,
> ...,
> PRIMARY KEY (Code) NOT ENFORCED
> )WITH (...)
> {code}
> then parsing a SQL statement `SELECT *FROM test_table GROUP BY Code`. An
> exception as below will be thrown:
>
> {code:java}
> org.apache.calcite.sql.validate.SqlValidatorException: Expression
> 'test_table.Name' is not being grouped
> {code}
>
>
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-20458) Translate page 'SQL-gettingStarted' into Chinese
[ https://issues.apache.org/jira/browse/FLINK-20458?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jark Wu updated FLINK-20458: Component/s: Table SQL / Ecosystem > Translate page 'SQL-gettingStarted' into Chinese > > > Key: FLINK-20458 > URL: https://issues.apache.org/jira/browse/FLINK-20458 > Project: Flink > Issue Type: Sub-task > Components: chinese-translation, Documentation, Table SQL / Ecosystem >Reporter: CaoZhen >Priority: Minor > > Translate the doc located in "docs/dev/table/sql/gettingStarted.zh.md" -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-20456) Make streaming SQL concepts more approachable
[ https://issues.apache.org/jira/browse/FLINK-20456?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jark Wu updated FLINK-20456: Component/s: Table SQL / Ecosystem > Make streaming SQL concepts more approachable > - > > Key: FLINK-20456 > URL: https://issues.apache.org/jira/browse/FLINK-20456 > Project: Flink > Issue Type: Improvement > Components: Documentation, Table SQL / Ecosystem >Reporter: Seth Wiesman >Assignee: Seth Wiesman >Priority: Major > Fix For: 1.12.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-20454) Allow to read metadata for debezium-avro-confluent format
[ https://issues.apache.org/jira/browse/FLINK-20454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17242884#comment-17242884 ] Jark Wu commented on FLINK-20454: - Sure [~hailong wang]. You can take a look the implementation of debezium json metadata. > Allow to read metadata for debezium-avro-confluent format > - > > Key: FLINK-20454 > URL: https://issues.apache.org/jira/browse/FLINK-20454 > Project: Flink > Issue Type: Sub-task > Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile), Table > SQL / Ecosystem >Reporter: Jark Wu >Assignee: hailong wang >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (FLINK-20454) Allow to read metadata for debezium-avro-confluent format
[ https://issues.apache.org/jira/browse/FLINK-20454?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jark Wu reassigned FLINK-20454: --- Assignee: hailong wang > Allow to read metadata for debezium-avro-confluent format > - > > Key: FLINK-20454 > URL: https://issues.apache.org/jira/browse/FLINK-20454 > Project: Flink > Issue Type: Sub-task > Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile), Table > SQL / Ecosystem >Reporter: Jark Wu >Assignee: hailong wang >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-20458) Translate page 'SQL-gettingStarted' into Chinese
[ https://issues.apache.org/jira/browse/FLINK-20458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17242878#comment-17242878 ] CaoZhen commented on FLINK-20458: - Hi, [~sjwiesman] I want to translate this document. Can you assign it to me? > Translate page 'SQL-gettingStarted' into Chinese > > > Key: FLINK-20458 > URL: https://issues.apache.org/jira/browse/FLINK-20458 > Project: Flink > Issue Type: Sub-task > Components: chinese-translation, Documentation >Reporter: CaoZhen >Priority: Minor > > Translate the doc located in "docs/dev/table/sql/gettingStarted.zh.md" -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-20458) Translate page 'SQL-gettingStarted' into Chinese
CaoZhen created FLINK-20458: --- Summary: Translate page 'SQL-gettingStarted' into Chinese Key: FLINK-20458 URL: https://issues.apache.org/jira/browse/FLINK-20458 Project: Flink Issue Type: Sub-task Components: chinese-translation, Documentation Reporter: CaoZhen Translate the doc located in "docs/dev/table/sql/gettingStarted.zh.md" -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] gaoyunhaii commented on pull request #14229: [FLINK-20292][doc] Improve the document about table formats overlap in user fat jar
gaoyunhaii commented on pull request #14229: URL: https://github.com/apache/flink/pull/14229#issuecomment-737632520 From my side of view I still tend to keep the links since that if users want to add the dependency of one format, he would tend to jump to the corresponding page to see how to add the dependency, without read the index page first. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-20451) Taskmanager can't apply resources when submit job with configs `env.java.opts.taskmanager: -javaagent:`
[
https://issues.apache.org/jira/browse/FLINK-20451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17242872#comment-17242872
]
Xintong Song commented on FLINK-20451:
--
Hi [~xiaozilong],
I don't understand the problem. What do you mean by "taskmanager can't apply
resources"? And which part of the screenshot is different from what you
expected?
> Taskmanager can't apply resources when submit job with configs
> `env.java.opts.taskmanager: -javaagent:`
> ---
>
> Key: FLINK-20451
> URL: https://issues.apache.org/jira/browse/FLINK-20451
> Project: Flink
> Issue Type: Bug
> Components: Deployment / YARN
>Affects Versions: 1.11.0
>Reporter: xiaozilong
>Priority: Major
> Attachments: image-2020-12-02-20-24-57-809.png
>
>
> Hi, I meet an unexpected problem that the taskmanager can't apply resources
> when submit job with configs `env.java.opts.taskmanager: -javaagent` in
> flink-conf.yaml.
> This my configuration:
> {code:java}
> env.java.opts: -Djava.net.preferIPv4Stack=true
> -Djob.name=monitoring_demo_64648b66-1923-4b2e-825b-a970a2b6ecf8
> env.java.opts.taskmanager:
> -javaagent:/data/opt/file_tmp/jvm-profiler-1.0.0.jar=sampleInterval=50{code}
> Partial log:
> !image-2020-12-02-20-24-57-809.png!
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-20457) Fix the handling of timestamp in DataStream.from_collection
Dian Fu created FLINK-20457: --- Summary: Fix the handling of timestamp in DataStream.from_collection Key: FLINK-20457 URL: https://issues.apache.org/jira/browse/FLINK-20457 Project: Flink Issue Type: Bug Components: API / Python Affects Versions: 1.12.0 Reporter: Dian Fu Fix For: 1.13.0, 1.12.1 Currently, DataStream.from_collection firstly converts date/time/dateTime objects to int at Python side and then construct the corresponding Date/Time/Timestamp object at Java side. It will lose the timezone information. Pickle could handle date/time/datetime properly and the conversion could be avoided. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] SteNicholas commented on pull request #14028: [FLINK-20020][client] Make UnsuccessfulExecutionException part of the JobClient.getJobExecutionResult() contract
SteNicholas commented on pull request #14028: URL: https://github.com/apache/flink/pull/14028#issuecomment-737624706 > Although the `ProgramInvocationException` is not annotated as `Public` or `PublicEvolving`, due to the fact that it has been here since forever, it seems that in some places we may be handling it specially and even expose it to the users. For example, the `ClientUtils.executeProgram()` is public and in the mailing list in the past there were people saying that they use it in other projects (I think Beam used to use it and I do not know if they still do). > > On a separate note, given that this PR although small it has taken more than 3 weeks and a lot of discussions, I am starting to doubt if we should move forward with it. For example, the question that @tillrohrmann mentioned to have a separate exception for the runtime components and a separate for the client so that we can handle them differently, seems a deeper issue that requires more thought. > > I may be wrong on that, but the discussions seem to be disproportionately long for the size and also the added value of this PR. What do you think? > > BTW sorry for your time @SteNicholas if this does not get merged in the end but when I opened the JIRA I did not have a clear view of all the related changes that would pop up. @kl0u , IMO, I really want to follow the remaining comments from @tillrohrmann , and continue to update the commit according to Till's above comments. It's better for me to merge this pull request in the end. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] SteNicholas removed a comment on pull request #14028: [FLINK-20020][client] Make UnsuccessfulExecutionException part of the JobClient.getJobExecutionResult() contract
SteNicholas removed a comment on pull request #14028: URL: https://github.com/apache/flink/pull/14028#issuecomment-737624582 > example @kl0u , IMO, I really want to follow the remaining comments from @tillrohrmann , and continue to update the commit according to Till's above comments. It's better for me to merge this pull request in the end. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] SteNicholas commented on pull request #14028: [FLINK-20020][client] Make UnsuccessfulExecutionException part of the JobClient.getJobExecutionResult() contract
SteNicholas commented on pull request #14028: URL: https://github.com/apache/flink/pull/14028#issuecomment-737624582 > example @kl0u , IMO, I really want to follow the remaining comments from @tillrohrmann , and continue to update the commit according to Till's above comments. It's better for me to merge this pull request in the end. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] JingsongLi commented on pull request #14229: [FLINK-20292][doc] Improve the document about table formats overlap in user fat jar
JingsongLi commented on pull request #14229: URL: https://github.com/apache/flink/pull/14229#issuecomment-737623316 > Links to this section from the specific format pages (like avro) under the Dependencies section so that users jump to specific pages could also see the warnings. I think this is a little redundant, we don't need add same words in every formats, what do you think? @gaoyunhaii @leonardBang This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] JingsongLi commented on a change in pull request #14229: [FLINK-20292][doc] Improve the document about table formats overlap in user fat jar
JingsongLi commented on a change in pull request #14229: URL: https://github.com/apache/flink/pull/14229#discussion_r534622069 ## File path: docs/dev/table/connectors/formats/index.md ## @@ -90,3 +90,62 @@ Flink supports the following formats: + + +Transform Table Format Resources + + +Flink uses Java's [Service Provider Interfaces (SPI)](https://docs.oracle.com/javase/tutorial/sound/SPI-intro.html) to load the table format factories by their identifiers. Since the SPI resource file named `org.apache.flink.table.factories.Factory` for every table format is under the same directory `META-INF/services`, these resource files will override each other when build the uber-jar of the project which uses more than one table format, which will cause Flink to fail to load table format factories. In this situation, a recommended way is transforming these resource files under the directory `META-INF/services` by [ServicesResourceTransformer](https://maven.apache.org/plugins/maven-shade-plugin/examples/resource-transformers.html) of maven shade plugin. Review comment: Can we move these document to connector/index? I think this is common mechanism for all connectors and formats. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (FLINK-20456) Make streaming SQL concepts more approachable
Seth Wiesman created FLINK-20456: Summary: Make streaming SQL concepts more approachable Key: FLINK-20456 URL: https://issues.apache.org/jira/browse/FLINK-20456 Project: Flink Issue Type: Improvement Components: Documentation Reporter: Seth Wiesman Assignee: Seth Wiesman Fix For: 1.12.0 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] xintongsong commented on pull request #8952: [FLINK-10868][flink-yarn] Add failure rater for resource manager
xintongsong commented on pull request #8952: URL: https://github.com/apache/flink/pull/8952#issuecomment-737618805 I'll try to find some time for this next week. Right now I'm a bit underwater. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-19527) Update SQL Pages
[ https://issues.apache.org/jira/browse/FLINK-19527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17242867#comment-17242867 ] Seth Wiesman commented on FLINK-19527: -- merged in master: b31495f41f6646c590af05cdce52bf2f4f624ece > Update SQL Pages > > > Key: FLINK-19527 > URL: https://issues.apache.org/jira/browse/FLINK-19527 > Project: Flink > Issue Type: Sub-task > Components: Documentation >Reporter: Seth Wiesman >Assignee: M Haseeb Asif >Priority: Major > Labels: pull-request-available > > SQL > Goal: Show users the main features early and link to concepts if necessary. > How to use SQL? Intended for users with SQL knowledge. > Overview > Getting started with link to more detailed execution section. > Full Reference > Available operations in SQL as a table. This location allows to further > split the page in the future if we think an operation needs more space > without affecting the top-level structure. > Data Definition > Explain special SQL syntax around DDL. > Pattern Matching > Make pattern matching more visible. > ... more features in the future -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-19527) Update SQL Pages
[ https://issues.apache.org/jira/browse/FLINK-19527?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Seth Wiesman updated FLINK-19527: - Fix Version/s: 1.13.0 > Update SQL Pages > > > Key: FLINK-19527 > URL: https://issues.apache.org/jira/browse/FLINK-19527 > Project: Flink > Issue Type: Sub-task > Components: Documentation >Reporter: Seth Wiesman >Assignee: M Haseeb Asif >Priority: Major > Labels: pull-request-available > Fix For: 1.13.0 > > > SQL > Goal: Show users the main features early and link to concepts if necessary. > How to use SQL? Intended for users with SQL knowledge. > Overview > Getting started with link to more detailed execution section. > Full Reference > Available operations in SQL as a table. This location allows to further > split the page in the future if we think an operation needs more space > without affecting the top-level structure. > Data Definition > Explain special SQL syntax around DDL. > Pattern Matching > Make pattern matching more visible. > ... more features in the future -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] sjwiesman commented on pull request #14003: [FLINK-19527][Doc]Flink SQL Getting Started
sjwiesman commented on pull request #14003: URL: https://github.com/apache/flink/pull/14003#issuecomment-737618317 merged in master: b31495f41f6646c590af05cdce52bf2f4f624ece This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] sjwiesman closed pull request #14003: [FLINK-19527][Doc]Flink SQL Getting Started
sjwiesman closed pull request #14003: URL: https://github.com/apache/flink/pull/14003 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] sjwiesman commented on pull request #14003: [FLINK-19527][Doc]Flink SQL Getting Started
sjwiesman commented on pull request #14003: URL: https://github.com/apache/flink/pull/14003#issuecomment-737616530 this looks much better! there are a number of spelling errors. I'm going to fix them when merging but in the future please use a spell checker before opening a PR. Thanks. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Closed] (FLINK-20436) Simplify type parameter of ExecNode
[
https://issues.apache.org/jira/browse/FLINK-20436?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
godfrey he closed FLINK-20436.
--
Resolution: Done
master: deb9e6e20ff605dd4b4a78fa7e418592746719e0
> Simplify type parameter of ExecNode
> ---
>
> Key: FLINK-20436
> URL: https://issues.apache.org/jira/browse/FLINK-20436
> Project: Flink
> Issue Type: Sub-task
> Components: Table SQL / Planner
>Reporter: godfrey he
>Assignee: godfrey he
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.13.0
>
>
> Currently, {{ExecNode#translateToPlan}} takes {{BatchPlanner}} or
> {{StreamPlanner}} as a parameter, so {{ExecNode}} has a type parameter {{E <:
> Planner}}, which indicates the node is a batch node or a streaming node.
> While in the future, a plan may contain both batch nodes and stream node. The
> type parameter can be removed, the implementation base class can cast the
> planner to expected planner.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-20436) Simplify type parameter of ExecNode
[
https://issues.apache.org/jira/browse/FLINK-20436?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
godfrey he updated FLINK-20436:
---
Fix Version/s: 1.13.0
> Simplify type parameter of ExecNode
> ---
>
> Key: FLINK-20436
> URL: https://issues.apache.org/jira/browse/FLINK-20436
> Project: Flink
> Issue Type: Sub-task
> Components: Table SQL / Planner
>Reporter: godfrey he
>Assignee: godfrey he
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.13.0
>
>
> Currently, {{ExecNode#translateToPlan}} takes {{BatchPlanner}} or
> {{StreamPlanner}} as a parameter, so {{ExecNode}} has a type parameter {{E <:
> Planner}}, which indicates the node is a batch node or a streaming node.
> While in the future, a plan may contain both batch nodes and stream node. The
> type parameter can be removed, the implementation base class can cast the
> planner to expected planner.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] godfreyhe merged pull request #14270: [FLINK-20436][table-planner-blink] Simplify type parameter of ExecNode
godfreyhe merged pull request #14270: URL: https://github.com/apache/flink/pull/14270 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-18633) Download miniconda is instable
[
https://issues.apache.org/jira/browse/FLINK-18633?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dian Fu updated FLINK-18633:
Affects Version/s: 1.13.0
> Download miniconda is instable
> --
>
> Key: FLINK-18633
> URL: https://issues.apache.org/jira/browse/FLINK-18633
> Project: Flink
> Issue Type: Test
> Components: API / Python, Tests
>Affects Versions: 1.13.0
>Reporter: Dian Fu
>Priority: Major
> Labels: test-stability
> Fix For: 1.13.0
>
>
> [https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=4606&view=logs&j=bdd9ea51-4de2-506a-d4d9-f3930e4d2355&t=17a7e096-e650-5b91-858e-3d426f9eeb2f]
> {code}
> RUNNING './flink-python/dev/lint-python.sh'.
> installing environment
> installing wget...
> install wget... [SUCCESS]
> installing miniconda...
> download miniconda...
> Dowload failed.You can try again
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Reopened] (FLINK-18633) Download miniconda is instable
[
https://issues.apache.org/jira/browse/FLINK-18633?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dian Fu reopened FLINK-18633:
-
> Download miniconda is instable
> --
>
> Key: FLINK-18633
> URL: https://issues.apache.org/jira/browse/FLINK-18633
> Project: Flink
> Issue Type: Test
> Components: API / Python, Tests
>Reporter: Dian Fu
>Priority: Major
> Labels: test-stability
>
> [https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=4606&view=logs&j=bdd9ea51-4de2-506a-d4d9-f3930e4d2355&t=17a7e096-e650-5b91-858e-3d426f9eeb2f]
> {code}
> RUNNING './flink-python/dev/lint-python.sh'.
> installing environment
> installing wget...
> install wget... [SUCCESS]
> installing miniconda...
> download miniconda...
> Dowload failed.You can try again
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-18633) Download miniconda is instable
[
https://issues.apache.org/jira/browse/FLINK-18633?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17242849#comment-17242849
]
Dian Fu commented on FLINK-18633:
-
Another instance:
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=10442&view=logs&j=c88eea3b-64a0-564d-0031-9fdcd7b8abee&t=ff888d9b-cd34-53cc-d90f-3e446d355529
> Download miniconda is instable
> --
>
> Key: FLINK-18633
> URL: https://issues.apache.org/jira/browse/FLINK-18633
> Project: Flink
> Issue Type: Test
> Components: API / Python, Tests
>Reporter: Dian Fu
>Priority: Major
> Labels: test-stability
>
> [https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=4606&view=logs&j=bdd9ea51-4de2-506a-d4d9-f3930e4d2355&t=17a7e096-e650-5b91-858e-3d426f9eeb2f]
> {code}
> RUNNING './flink-python/dev/lint-python.sh'.
> installing environment
> installing wget...
> install wget... [SUCCESS]
> installing miniconda...
> download miniconda...
> Dowload failed.You can try again
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-18633) Download miniconda is instable
[
https://issues.apache.org/jira/browse/FLINK-18633?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dian Fu updated FLINK-18633:
Fix Version/s: 1.13.0
> Download miniconda is instable
> --
>
> Key: FLINK-18633
> URL: https://issues.apache.org/jira/browse/FLINK-18633
> Project: Flink
> Issue Type: Test
> Components: API / Python, Tests
>Reporter: Dian Fu
>Priority: Major
> Labels: test-stability
> Fix For: 1.13.0
>
>
> [https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=4606&view=logs&j=bdd9ea51-4de2-506a-d4d9-f3930e4d2355&t=17a7e096-e650-5b91-858e-3d426f9eeb2f]
> {code}
> RUNNING './flink-python/dev/lint-python.sh'.
> installing environment
> installing wget...
> install wget... [SUCCESS]
> installing miniconda...
> download miniconda...
> Dowload failed.You can try again
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-20371) Add docs for outer interval join
[ https://issues.apache.org/jira/browse/FLINK-20371?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17242841#comment-17242841 ] Seth Wiesman commented on FLINK-20371: -- merged in master: e8d6b29f57679b2c5f64d7f614ac0b62d765dc6b release 1.12 28f3aba7961b48c518aa0a67004dd24660437390 > Add docs for outer interval join > > > Key: FLINK-20371 > URL: https://issues.apache.org/jira/browse/FLINK-20371 > Project: Flink > Issue Type: Improvement > Components: Documentation, Table SQL / API >Reporter: Timo Walther >Assignee: Timo Walther >Priority: Major > Labels: pull-request-available > > By looking at the docs, it looks like we only support inner interval joins > but we also support outer joins according to the tests. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (FLINK-20371) Add docs for outer interval join
[ https://issues.apache.org/jira/browse/FLINK-20371?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Seth Wiesman closed FLINK-20371. Resolution: Fixed > Add docs for outer interval join > > > Key: FLINK-20371 > URL: https://issues.apache.org/jira/browse/FLINK-20371 > Project: Flink > Issue Type: Improvement > Components: Documentation, Table SQL / API >Reporter: Timo Walther >Assignee: Timo Walther >Priority: Major > Labels: pull-request-available > Fix For: 1.12.0 > > > By looking at the docs, it looks like we only support inner interval joins > but we also support outer joins according to the tests. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-20371) Add docs for outer interval join
[ https://issues.apache.org/jira/browse/FLINK-20371?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Seth Wiesman updated FLINK-20371: - Fix Version/s: 1.12.0 > Add docs for outer interval join > > > Key: FLINK-20371 > URL: https://issues.apache.org/jira/browse/FLINK-20371 > Project: Flink > Issue Type: Improvement > Components: Documentation, Table SQL / API >Reporter: Timo Walther >Assignee: Timo Walther >Priority: Major > Labels: pull-request-available > Fix For: 1.12.0 > > > By looking at the docs, it looks like we only support inner interval joins > but we also support outer joins according to the tests. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] sjwiesman commented on pull request #14230: [FLINK-20371][docs] Add docs for outer interval joins
sjwiesman commented on pull request #14230: URL: https://github.com/apache/flink/pull/14230#issuecomment-737599221 merging ... master: e8d6b29f57679b2c5f64d7f614ac0b62d765dc6b release 1.12 28f3aba7961b48c518aa0a67004dd24660437390 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] sjwiesman closed pull request #14230: [FLINK-20371][docs] Add docs for outer interval joins
sjwiesman closed pull request #14230: URL: https://github.com/apache/flink/pull/14230 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14291: [FLINK-20431][connector/kafka] Fix the offsets commit in KafkaSourceReader when the SplitFetcher shuts down.
flinkbot edited a comment on pull request #14291: URL: https://github.com/apache/flink/pull/14291#issuecomment-737574351 ## CI report: * 5893b7cc51aae55b09105724aa5be69d92f15266 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10455) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Closed] (FLINK-19989) Add collect operation in Python DataStream API
[ https://issues.apache.org/jira/browse/FLINK-19989?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dian Fu closed FLINK-19989. --- Resolution: Fixed Merged to 1.13.0 via 9a5165ea52f9c26a2722ac8487a99559890af046 > Add collect operation in Python DataStream API > -- > > Key: FLINK-19989 > URL: https://issues.apache.org/jira/browse/FLINK-19989 > Project: Flink > Issue Type: Improvement > Components: API / Python >Reporter: Dian Fu >Assignee: Nicholas Jiang >Priority: Major > Labels: pull-request-available > Fix For: 1.13.0 > > > DataStream.executeAndCollect() has already been supported in FLINK-19508. We > should also support it in the Python DataStream API. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] dianfu closed pull request #13983: [FLINK-19989][python] Add collect operation in Python DataStream API
dianfu closed pull request #13983: URL: https://github.com/apache/flink/pull/13983 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #14291: [FLINK-20431][connector/kafka] Fix the offsets commit in KafkaSourceReader when the SplitFetcher shuts down.
flinkbot commented on pull request #14291: URL: https://github.com/apache/flink/pull/14291#issuecomment-737574351 ## CI report: * 5893b7cc51aae55b09105724aa5be69d92f15266 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #14291: [FLINK-20431][connector/kafka] Fix the offsets commit in KafkaSourceReader when the SplitFetcher shuts down.
flinkbot commented on pull request #14291: URL: https://github.com/apache/flink/pull/14291#issuecomment-737565333 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit 5893b7cc51aae55b09105724aa5be69d92f15266 (Wed Dec 02 23:50:45 UTC 2020) **Warnings:** * No documentation files were touched! Remember to keep the Flink docs up to date! Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-20431) KafkaSourceReaderTest.testCommitOffsetsWithoutAliveFetchers:133->lambda$testCommitOffsetsWithoutAliveFetchers$3:134 expected:<10> but was:<1>
[ https://issues.apache.org/jira/browse/FLINK-20431?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-20431: --- Labels: pull-request-available test-stability (was: test-stability) > KafkaSourceReaderTest.testCommitOffsetsWithoutAliveFetchers:133->lambda$testCommitOffsetsWithoutAliveFetchers$3:134 > expected:<10> but was:<1> > - > > Key: FLINK-20431 > URL: https://issues.apache.org/jira/browse/FLINK-20431 > Project: Flink > Issue Type: Bug > Components: Connectors / Kafka >Affects Versions: 1.13.0 >Reporter: Huang Xingbo >Assignee: Jiangjie Qin >Priority: Critical > Labels: pull-request-available, test-stability > Fix For: 1.12.0 > > > [https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=10351&view=logs&j=c5f0071e-1851-543e-9a45-9ac140befc32&t=1fb1a56f-e8b5-5a82-00a0-a2db7757b4f5] > [ERROR] Failures: > [ERROR] > KafkaSourceReaderTest.testCommitOffsetsWithoutAliveFetchers:133->lambda$testCommitOffsetsWithoutAliveFetchers$3:134 > expected:<10> but was:<1> > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] becketqin opened a new pull request #14291: [FLINK-20431][connector/kafka] Fix the offsets commit in KafkaSourceReader when the SplitFetcher shuts down.
becketqin opened a new pull request #14291: URL: https://github.com/apache/flink/pull/14291 ## What is the purpose of the change The patch fixes a bug that the `KafkaPartitionSplitFetcher` does not finish the pending offsets commit when the contained `KafkaConsumer` exits. This is actually a bug in KafkaConsumer but we need to handle it for now. The previous test tries to play a trick so that the unit test passes. We should have handled this correctly in the `KafkaPartitionSplitFetcher` instead. The patch also fixes the reported instability of the `KafkaSourceReaderTest.testCommitOffsetsWithoutAliveFetchers()`. ## Brief change log The patch tracks all the pending async commit and commit them when the `KafkaPartitionSplitFetcher` exits. ## Verifying this change The following unit tests have been added to verify the change. `KafkaPartitionSplitReaderTest.testFinishPendingOffsetsCommitOnClose()` ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (**no**) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (**no**) - The serializers: (**no**) - The runtime per-record code paths (performance sensitive): (**no**) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn/Mesos, ZooKeeper: (**no**) - The S3 file system connector: (**no**) ## Documentation - Does this pull request introduce a new feature? (**no**) - If yes, how is the feature documented? (**not applicable**) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Closed] (FLINK-20055) Datadog API Key exposed in Flink JobManager logs
[
https://issues.apache.org/jira/browse/FLINK-20055?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chesnay Schepler closed FLINK-20055.
Fix Version/s: 1.12.1
1.13.0
1.11.3
Resolution: Fixed
master: de527fdeec3887f6148ef3dfe7b2a024ec292522
1.12: 23fa6e9798b4d85af4189d8910500fb1ece34ac2
1.11: c430b9e67948ceadf078c37cdfa3285bbbfec886
> Datadog API Key exposed in Flink JobManager logs
>
>
> Key: FLINK-20055
> URL: https://issues.apache.org/jira/browse/FLINK-20055
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Configuration, Runtime / Metrics
>Affects Versions: 1.9.0
>Reporter: Florian Szabo
>Assignee: Florian Szabo
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.11.3, 1.13.0, 1.12.1
>
>
> When Flink is set up to report metrics to Datadog, the JobManager log
> containe the Datadog API key in plain format. In fact it shows up in two
> different places:
> {code:java}
> 2020-08-03 09:03:19,400 INFO
> org.apache.flink.configuration.GlobalConfiguration- Loading
> configuration property: metrics.reporter.dghttp.apikey,
> ...
> 2020-08-03 09:03:20,437 INFO org.apache.flink.runtime.metrics.ReporterSetup
> - Configuring dghttp with {apikey=,
> tags=<...>,profile:<...>,region:<...>,env:<...>,
> class=org.apache.flink.metrics.datadog.DatadogHttpReporter}.
> {code}
> The expected behavior here should be that the API key in both places is
> hidden so that it does not end up in places where it should not be.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on pull request #13964: [FLINK-19314][coordination] Add DeclarativeSlotPoolBridge
flinkbot edited a comment on pull request #13964: URL: https://github.com/apache/flink/pull/13964#issuecomment-723100396 ## CI report: * 002a26e22352ec2a5c79c3cde4ed4b749f87e7e1 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9192) * b2fc4902b25f60f403bf5839ca717b465f09b67a Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10452) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Assigned] (FLINK-20055) Datadog API Key exposed in Flink JobManager logs
[
https://issues.apache.org/jira/browse/FLINK-20055?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chesnay Schepler reassigned FLINK-20055:
Assignee: Florian Szabo
> Datadog API Key exposed in Flink JobManager logs
>
>
> Key: FLINK-20055
> URL: https://issues.apache.org/jira/browse/FLINK-20055
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Configuration
>Affects Versions: 1.9.1, 1.11.2
>Reporter: Florian Szabo
>Assignee: Florian Szabo
>Priority: Major
> Labels: pull-request-available
>
> When Flink is set up to report metrics to Datadog, the JobManager log
> containe the Datadog API key in plain format. In fact it shows up in two
> different places:
> {code:java}
> 2020-08-03 09:03:19,400 INFO
> org.apache.flink.configuration.GlobalConfiguration- Loading
> configuration property: metrics.reporter.dghttp.apikey,
> ...
> 2020-08-03 09:03:20,437 INFO org.apache.flink.runtime.metrics.ReporterSetup
> - Configuring dghttp with {apikey=,
> tags=<...>,profile:<...>,region:<...>,env:<...>,
> class=org.apache.flink.metrics.datadog.DatadogHttpReporter}.
> {code}
> The expected behavior here should be that the API key in both places is
> hidden so that it does not end up in places where it should not be.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-20055) Datadog API Key exposed in Flink JobManager logs
[
https://issues.apache.org/jira/browse/FLINK-20055?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chesnay Schepler updated FLINK-20055:
-
Component/s: Runtime / Metrics
> Datadog API Key exposed in Flink JobManager logs
>
>
> Key: FLINK-20055
> URL: https://issues.apache.org/jira/browse/FLINK-20055
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Configuration, Runtime / Metrics
>Affects Versions: 1.9.1, 1.11.2
>Reporter: Florian Szabo
>Assignee: Florian Szabo
>Priority: Major
> Labels: pull-request-available
>
> When Flink is set up to report metrics to Datadog, the JobManager log
> containe the Datadog API key in plain format. In fact it shows up in two
> different places:
> {code:java}
> 2020-08-03 09:03:19,400 INFO
> org.apache.flink.configuration.GlobalConfiguration- Loading
> configuration property: metrics.reporter.dghttp.apikey,
> ...
> 2020-08-03 09:03:20,437 INFO org.apache.flink.runtime.metrics.ReporterSetup
> - Configuring dghttp with {apikey=,
> tags=<...>,profile:<...>,region:<...>,env:<...>,
> class=org.apache.flink.metrics.datadog.DatadogHttpReporter}.
> {code}
> The expected behavior here should be that the API key in both places is
> hidden so that it does not end up in places where it should not be.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
