[GitHub] [flink] wulei0302 opened a new pull request #14063: [FLINK-19454][docs-zh] Translate page 'Importing Flink into an IDE' into Chinese
wulei0302 opened a new pull request #14063: URL: https://github.com/apache/flink/pull/14063 ## What is the purpose of the change Translate page 'Importing Flink into an IDE' into Chinese The page url is https://ci.apache.org/projects/flink/flink-docs-release-1.11/flinkDev/ide_setup.html The markdown file is located in flink/docs/flinkDev/ide_setup.zh.md ## Brief change log - translate page 'Importing Flink into an IDE' ## Verifying this change This change is a trivial rework / code cleanup without any test coverage. ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): no - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: no - The serializers: no - The runtime per-record code paths (performance sensitive): no - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn/Mesos, ZooKeeper: no - The S3 file system connector: no ## Documentation - Does this pull request introduce a new feature? no - If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-19454) Translate page 'Importing Flink into an IDE' into Chinese
[
https://issues.apache.org/jira/browse/FLINK-19454?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated FLINK-19454:
---
Labels: pull-request-available (was: )
> Translate page 'Importing Flink into an IDE' into Chinese
> -
>
> Key: FLINK-19454
> URL: https://issues.apache.org/jira/browse/FLINK-19454
> Project: Flink
> Issue Type: Improvement
> Components: chinese-translation, Documentation
>Affects Versions: 1.11.2
>Reporter: wulei0302
>Assignee: wulei0302
>Priority: Minor
> Labels: pull-request-available
>
> The page url is
> [https://ci.apache.org/projects/flink/flink-docs-release-1.11/flinkDev/ide_setup.html]
> The markdown file is located in {{flink/docs/flinkDev/ide_setup.zh.md}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (FLINK-19852) Managed memory released check can block IterativeTask
[ https://issues.apache.org/jira/browse/FLINK-19852?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17231231#comment-17231231 ] Andrey Zagrebin edited comment on FLINK-19852 at 11/13/20, 8:01 AM: UnsafeMemory usage has indeed become more safe after 1.11. We do not just expect users of UnsafeMemory to always explicitly release it. MemorySegments are tracked by JVM GC to make sure that they are reused only once no other code refers to them, basically when they are GC'ed but GC takes time, of course, this is the price for safety. It is very similar to JVM direct memory. The problem here is that the limit is relatively small per operator and it is exact (no playground to over-allocate). The usage pattern in TempBarrier is the worst for this safe approach because it tries to (re)-allocate all segments at once. Hence, it has to wait for GC of all segments between iterations (stop the world event). From what I see in SpillingBuffer/ListMemorySegmentSource it does not really need all segments at once, the segments are just pulled on-demand one-by-one. If ListMemorySegmentSource reserved segments also one-by-one then GC would be amortised between segments allocation. Ideally using code should also release segments asap once they are not needed anymore, this would give GC more time. was (Author: azagrebin): UnsafeMemory usage has indeed become more safe after 1.11. We do not just expect users of UnsafeMemory to always explicitly release it. MemorySegments are tracked by JVM GC to make sure that they are reused only once no other code refers to them, basically when they are GC'ed but GC takes time, of course, this is the price for safety. It is very similar to JVM direct memory. The problem here is that the limit is relatively small per operator and it is exact (no playground to over-allocate). The usage pattern in TempBarrier is the worst for this safe approach because it tries to (re)-allocate all segments at once. Hence, it has to wait for GC of all segments between iterations (stop the world event). From what I see in SpillingBuffer/ListMemorySegmentSource it does not really need all segments at once, the segments are just pulled on-demand one-by-one. If ListMemorySegmentSource reserved segments also one-by-one then GC would be amortised between segments allocation. > Managed memory released check can block IterativeTask > - > > Key: FLINK-19852 > URL: https://issues.apache.org/jira/browse/FLINK-19852 > Project: Flink > Issue Type: Bug > Components: Runtime / Task >Affects Versions: 1.11.0, 1.10.2, 1.11.1, 1.11.2 >Reporter: shaomeng.wang >Priority: Critical > Attachments: image-2020-10-28-17-48-28-395.png, > image-2020-10-28-17-48-48-583.png > > > UnsafeMemoryBudget#reserveMemory, called on TempBarrier, needs time to wait > on GC of all allocated/released managed memory at every iteration. > > stack: > !image-2020-10-28-17-48-48-583.png! > new TempBarrier in BatchTask > !image-2020-10-28-17-48-28-395.png! > > These will be very slow than before. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] wulei0302 closed pull request #14063: [FLINK-19454][docs-zh] Translate page 'Importing Flink into an IDE' into Chinese
wulei0302 closed pull request #14063: URL: https://github.com/apache/flink/pull/14063 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-20124) Test pipelined region scheduler
[ https://issues.apache.org/jira/browse/FLINK-20124?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17231233#comment-17231233 ] Robert Metzger commented on FLINK-20124: - ran a job for 2 hours that was failing on average every 30 seconds, in both BATCH and STREAMING mode without any unexpected behavior. > Test pipelined region scheduler > --- > > Key: FLINK-20124 > URL: https://issues.apache.org/jira/browse/FLINK-20124 > Project: Flink > Issue Type: Sub-task > Components: Runtime / Coordination >Affects Versions: 1.12.0 >Reporter: Robert Metzger >Assignee: Robert Metzger >Priority: Critical > Fix For: 1.12.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-20113) Test K8s High Availability Service
[ https://issues.apache.org/jira/browse/FLINK-20113?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17231234#comment-17231234 ] Robert Metzger commented on FLINK-20113: Awesome, thanks a lot! > Test K8s High Availability Service > -- > > Key: FLINK-20113 > URL: https://issues.apache.org/jira/browse/FLINK-20113 > Project: Flink > Issue Type: Sub-task > Components: Deployment / Kubernetes >Affects Versions: 1.12.0 >Reporter: Robert Metzger >Assignee: Guowei Ma >Priority: Critical > Fix For: 1.12.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (FLINK-20113) Test K8s High Availability Service
[ https://issues.apache.org/jira/browse/FLINK-20113?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Metzger reassigned FLINK-20113: -- Assignee: Guowei Ma > Test K8s High Availability Service > -- > > Key: FLINK-20113 > URL: https://issues.apache.org/jira/browse/FLINK-20113 > Project: Flink > Issue Type: Sub-task > Components: Deployment / Kubernetes >Affects Versions: 1.12.0 >Reporter: Robert Metzger >Assignee: Guowei Ma >Priority: Critical > Fix For: 1.12.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-20129) Create a zh download page for all optional sql client components
[ https://issues.apache.org/jira/browse/FLINK-20129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17231239#comment-17231239 ] hailong wang commented on FLINK-20129: -- Hi [~dwysakowicz], can I take this issue ? > Create a zh download page for all optional sql client components > > > Key: FLINK-20129 > URL: https://issues.apache.org/jira/browse/FLINK-20129 > Project: Flink > Issue Type: Improvement > Components: chinese-translation >Reporter: Dawid Wysakowicz >Priority: Major > > The task is to mirror changes introduced in FLINK-20093 in zh docs. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on pull request #14033: [FLINK-19979][e2e] Add sanity check after bash e2e tests for no leftovers
flinkbot edited a comment on pull request #14033: URL: https://github.com/apache/flink/pull/14033#issuecomment-725388043 ## CI report: * 43dee710e0f873adfd5b521183fc1492375d7800 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9467) * 9e5179d99ac3f1ee67388f6236a5ffd087941a94 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-20128) Data loss for over windows with rows unbounded preceding
[
https://issues.apache.org/jira/browse/FLINK-20128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17231243#comment-17231243
]
Shengkai Fang commented on FLINK-20128:
---
I don't get the expected bug by sql-client and java table api. I think it maybe
a pyflink's bug?
> Data loss for over windows with rows unbounded preceding
>
>
> Key: FLINK-20128
> URL: https://issues.apache.org/jira/browse/FLINK-20128
> Project: Flink
> Issue Type: Bug
> Components: API / Python, Table SQL / Planner
>Affects Versions: 1.12.0, 1.11.2
>Reporter: Thilo Schneider
>Priority: Major
>
> When using partitioned, unbounded over windows, all but one partitions are
> dropped in the output dataset:
> {code:python}
> # Setup
> from pyflink.table import EnvironmentSettings, StreamTableEnvironment
> from biafflink import debug_print_table
> env_settings =
> EnvironmentSettings.new_instance().in_streaming_mode().use_blink_planner().build()
> t_env = StreamTableEnvironment.create( environment_settings=env_settings)
> t_env.get_config().get_configuration().set_integer("table.exec.resource.default-parallelism",
> 1)
> t_env.execute_sql("""
> CREATE TABLE datagen (
> foo INT,
> id AS mod(foo, 2),
> message_time AS to_timestamp(from_unixtime(FLOOR(foo/2))),
> WATERMARK FOR message_time AS message_time
> ) WITH (
> 'connector' = 'datagen',
> 'rows-per-second'='2',
> 'fields.foo.kind'='sequence',
> 'fields.foo.start'='0',
> 'fields.foo.end'='19'
> )""")
> t_env.execute_sql("CREATE TABLE output (foo INT, id INT, lagfoo INT) WITH
> ('connector' = 'print')")
> {code}
> Using bounded over windows, everything works as expected:
> {code:python}
> t = t_env.sql_query("""
> SELECT foo, id, avg(foo) OVER w AS lagfoo
> FROM datagen
> WINDOW w AS (PARTITION BY id ORDER BY message_time ROWS BETWEEN 1
> PRECEDING AND CURRENT ROW)""")
> t.execute_insert("output")
> {code}
> yields
> {code:python}
> +I(0,0,0)
> +I(1,1,1)
> +I(2,0,1)
> +I(3,1,2)
> +I(4,0,3)
> ...
> {code}
> If we change the window to unbounded preceding:
> {code:python}
> t = t_env.sql_query("""
> SELECT foo, id, avg(foo) OVER w AS lagfoo
> FROM datagen
> WINDOW w AS (PARTITION BY id ORDER BY message_time ROWS BETWEEN UNBOUNDED
> PRECEDING AND CURRENT ROW)""")
> t.execute_insert("output")
> {code}
> we loose all of id == 1:
> {code:python}
> +I(0,0,0)
> +I(2,0,1)
> +I(4,0,2)
> +I(6,0,3)
> +I(8,0,4)
> ...
> {code}
> I observed this problem with various aggregate functions and both under
> 1.11.2 and 1.12rc1.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on pull request #14061: [FLINK-20141][fs-connector] Add FileSink documentation
flinkbot edited a comment on pull request #14061: URL: https://github.com/apache/flink/pull/14061#issuecomment-726585261 ## CI report: * ffe1daf67c98e39f476f1f15279c77bbc83a1c98 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9551) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Assigned] (FLINK-20129) Create a zh download page for all optional sql client components
[ https://issues.apache.org/jira/browse/FLINK-20129?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dawid Wysakowicz reassigned FLINK-20129: Assignee: hailong wang > Create a zh download page for all optional sql client components > > > Key: FLINK-20129 > URL: https://issues.apache.org/jira/browse/FLINK-20129 > Project: Flink > Issue Type: Improvement > Components: chinese-translation >Reporter: Dawid Wysakowicz >Assignee: hailong wang >Priority: Major > > The task is to mirror changes introduced in FLINK-20093 in zh docs. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-20129) Create a zh download page for all optional sql client components
[ https://issues.apache.org/jira/browse/FLINK-20129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17231242#comment-17231242 ] Dawid Wysakowicz commented on FLINK-20129: -- Fine by me. Just be aware I will not be able to help with the review as I don't know Chinese. > Create a zh download page for all optional sql client components > > > Key: FLINK-20129 > URL: https://issues.apache.org/jira/browse/FLINK-20129 > Project: Flink > Issue Type: Improvement > Components: chinese-translation >Reporter: Dawid Wysakowicz >Priority: Major > > The task is to mirror changes introduced in FLINK-20093 in zh docs. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] flinkbot commented on pull request #14062: [FLINK-17470] Send sigkill to hanging processes in standalone scripts
flinkbot commented on pull request #14062: URL: https://github.com/apache/flink/pull/14062#issuecomment-726595780 ## CI report: * b598725a80261a4f0084bb13704629ca135fa332 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] rkhachatryan commented on pull request #14056: [FLINK-19940][task] Improve naming of multi-input transformation
rkhachatryan commented on pull request #14056: URL: https://github.com/apache/flink/pull/14056#issuecomment-726595759 Blink `ExplainTest.testExplainMultipleInput` failed because of the exact matching of the generated explain plain. Looks like Blink generates a descriptive name and passes it to `MultipleInputTransformation`: ``` MultipleInput(readOrder=[1,0], members=[ HashJoin(joinType=[InnerJoin], where=[=(a, d)], select=[a, EXPR$1, d, EXPR$10], build=[right]) :- HashAggregate(isMerge=[true], groupBy=[a], select=[a, Final_SUM(sum$0) AS EXPR$1]) : +- [#1] Exchange(distribution=[hash[a]]) +- HashAggregate(isMerge=[true], groupBy=[d], select=[d, Final_SUM(sum$0) AS EXPR$1]) +- [#2] Exchange(distribution=[hash[d]]) ]) ``` My change adds (or replaces this with) `[Partition, Partition]`. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (FLINK-20142) Update the document for CREATE TABLE LIKE that source table from different catalog is supported
Danny Chen created FLINK-20142: -- Summary: Update the document for CREATE TABLE LIKE that source table from different catalog is supported Key: FLINK-20142 URL: https://issues.apache.org/jira/browse/FLINK-20142 Project: Flink Issue Type: Improvement Components: Documentation Affects Versions: 1.11.2 Reporter: Danny Chen The confusion from the USER mailing list: Hi, Is it disallowed to refer to a table from different databases or catalogs when someone creates a table? According to [1], there's no way to refer to tables belonging to different databases or catalogs. [1] https://ci.apache.org/projects/flink/flink-docs-master/dev/table/sql/create.html#create-table Best, Dongwon -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-20143) use `yarn.provided.lib.dirs` config deploy job failed in yarn per job mode
zhisheng created FLINK-20143:
Summary: use `yarn.provided.lib.dirs` config deploy job failed in
yarn per job mode
Key: FLINK-20143
URL: https://issues.apache.org/jira/browse/FLINK-20143
Project: Flink
Issue Type: Bug
Components: Client / Job Submission, Deployment / YARN
Affects Versions: 1.12.0
Reporter: zhisheng
use follow command deploy flink job to yarn failed
{code:java}
./bin/flink run -m yarn-cluster -d -ynm flink-1.12-test -ytm 3g -yjm 3g -yD
yarn.provided.lib.dirs=hdfs:///flink/flink-1.12-SNAPSHOT/lib
./examples/streaming/StateMachineExample.jar
{code}
log:
{code:java}
$ ./bin/flink run -m yarn-cluster -d -ynm flink-1.12-test -ytm 3g -yjm 3g -yD
yarn.provided.lib.dirs=hdfs:///flink/flink-1.12-SNAPSHOT/lib
./examples/streaming/StateMachineExample.jar$ ./bin/flink run -m yarn-cluster
-d -ynm flink-1.12-test -ytm 3g -yjm 3g -yD
yarn.provided.lib.dirs=hdfs:///flink/flink-1.12-SNAPSHOT/lib
./examples/streaming/StateMachineExample.jarSLF4J: Class path contains multiple
SLF4J bindings.SLF4J: Found binding in
[jar:file:/data1/app/flink-1.12-SNAPSHOT/lib/log4j-slf4j-impl-2.12.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J:
Found binding in
[jar:file:/data1/app/hadoop-2.7.3-snappy-32core12disk/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J:
Found binding in
[jar:file:/data1/app/hadoop-2.7.3-snappy-32core12disk/share/hadoop/tools/lib/hadoop-aliyun-2.9.2-jar-with-dependencies.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J:
See http://www.slf4j.org/codes.html#multiple_bindings for an
explanation.SLF4J: Actual binding is of type
[org.apache.logging.slf4j.Log4jLoggerFactory]2020-11-13 16:14:30,347 INFO
org.apache.flink.yarn.cli.FlinkYarnSessionCli [] - Dynamic
Property set:
yarn.provided.lib.dirs=hdfs:///flink/flink-1.12-SNAPSHOT/lib2020-11-13
16:14:30,347 INFO org.apache.flink.yarn.cli.FlinkYarnSessionCli
[] - Dynamic Property set:
yarn.provided.lib.dirs=hdfs:///flink/flink-1.12-SNAPSHOT/libUsage with built-in
data generator: StateMachineExample [--error-rate
] [--sleep ]Usage
with Kafka: StateMachineExample --kafka-topic [--brokers
]Options for both the above setups: [--backend ]
[--checkpoint-dir ] [--async-checkpoints ]
[--incremental-checkpoints ] [--output OR null for
stdout]
Using standalone source with error rate 0.00 and sleep delay 1 millis
2020-11-13 16:14:30,706 WARN
org.apache.flink.yarn.configuration.YarnLogConfigUtil [] - The
configuration directory ('/data1/app/flink-1.12-SNAPSHOT/conf') already
contains a LOG4J config file.If you want to use logback, then please delete or
rename the log configuration file.2020-11-13 16:14:30,947 INFO
org.apache.hadoop.yarn.client.AHSProxy [] - Connecting to
Application History server at
FAT-hadoopuat-69117.vm.dc01.hellocloud.tech/10.69.1.17:102002020-11-13
16:14:30,958 INFO org.apache.flink.yarn.YarnClusterDescriptor
[] - No path for the flink jar passed. Using the location of class
org.apache.flink.yarn.YarnClusterDescriptor to locate the jar2020-11-13
16:14:31,065 INFO
org.apache.hadoop.yarn.client.ConfiguredRMFailoverProxyProvider [] - Failing
over to rm22020-11-13 16:14:31,130 INFO
org.apache.flink.yarn.YarnClusterDescriptor [] - The
configured JobManager memory is 3072 MB. YARN will allocate 4096 MB to make up
an integer multiple of its minimum allocation memory (2048 MB, configured via
'yarn.scheduler.minimum-allocation-mb'). The extra 1024 MB may not be used by
Flink.2020-11-13 16:14:31,130 INFO org.apache.flink.yarn.YarnClusterDescriptor
[] - The configured TaskManager memory is 3072 MB. YARN will
allocate 4096 MB to make up an integer multiple of its minimum allocation
memory (2048 MB, configured via 'yarn.scheduler.minimum-allocation-mb'). The
extra 1024 MB may not be used by Flink.2020-11-13 16:14:31,130 INFO
org.apache.flink.yarn.YarnClusterDescriptor [] - Cluster
specification: ClusterSpecification{masterMemoryMB=3072,
taskManagerMemoryMB=3072, slotsPerTaskManager=2}2020-11-13 16:14:31,681 WARN
org.apache.hadoop.hdfs.shortcircuit.DomainSocketFactory [] - The
short-circuit local reads feature cannot be used because libhadoop cannot be
loaded.2020-11-13 16:14:33,417 INFO
org.apache.flink.yarn.YarnClusterDescriptor [] - Submitting
application master application_1599741232083_219902020-11-13 16:14:33,446 INFO
org.apache.hadoop.yarn.client.api.impl.YarnClientImpl [] - Submitted
application application_1599741232083_219902020-11-13 16:14:33,446 INFO
org.apache.flink.yarn.YarnClusterDescriptor [] - Waiting for
the cluster to be allocated2020-11-13 16:14:33,448 INFO
org.apache.flink.yarn.YarnClusterDescript
[jira] [Commented] (FLINK-20128) Data loss for over windows with rows unbounded preceding
[
https://issues.apache.org/jira/browse/FLINK-20128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17231253#comment-17231253
]
Dian Fu commented on FLINK-20128:
-
[~fsk119] I'm still not digging into this issue. However, it seems that this
job doesn't use PyFlink specific functionalities such as Python UDF/UDTF/UDAF,
etc and PyFlink is only a language layer here which just translates the Python
API calls to the Java ones and so I don't think this is a PyFlink's bug.
> Data loss for over windows with rows unbounded preceding
>
>
> Key: FLINK-20128
> URL: https://issues.apache.org/jira/browse/FLINK-20128
> Project: Flink
> Issue Type: Bug
> Components: API / Python, Table SQL / Planner
>Affects Versions: 1.12.0, 1.11.2
>Reporter: Thilo Schneider
>Priority: Major
>
> When using partitioned, unbounded over windows, all but one partitions are
> dropped in the output dataset:
> {code:python}
> # Setup
> from pyflink.table import EnvironmentSettings, StreamTableEnvironment
> from biafflink import debug_print_table
> env_settings =
> EnvironmentSettings.new_instance().in_streaming_mode().use_blink_planner().build()
> t_env = StreamTableEnvironment.create( environment_settings=env_settings)
> t_env.get_config().get_configuration().set_integer("table.exec.resource.default-parallelism",
> 1)
> t_env.execute_sql("""
> CREATE TABLE datagen (
> foo INT,
> id AS mod(foo, 2),
> message_time AS to_timestamp(from_unixtime(FLOOR(foo/2))),
> WATERMARK FOR message_time AS message_time
> ) WITH (
> 'connector' = 'datagen',
> 'rows-per-second'='2',
> 'fields.foo.kind'='sequence',
> 'fields.foo.start'='0',
> 'fields.foo.end'='19'
> )""")
> t_env.execute_sql("CREATE TABLE output (foo INT, id INT, lagfoo INT) WITH
> ('connector' = 'print')")
> {code}
> Using bounded over windows, everything works as expected:
> {code:python}
> t = t_env.sql_query("""
> SELECT foo, id, avg(foo) OVER w AS lagfoo
> FROM datagen
> WINDOW w AS (PARTITION BY id ORDER BY message_time ROWS BETWEEN 1
> PRECEDING AND CURRENT ROW)""")
> t.execute_insert("output")
> {code}
> yields
> {code:python}
> +I(0,0,0)
> +I(1,1,1)
> +I(2,0,1)
> +I(3,1,2)
> +I(4,0,3)
> ...
> {code}
> If we change the window to unbounded preceding:
> {code:python}
> t = t_env.sql_query("""
> SELECT foo, id, avg(foo) OVER w AS lagfoo
> FROM datagen
> WINDOW w AS (PARTITION BY id ORDER BY message_time ROWS BETWEEN UNBOUNDED
> PRECEDING AND CURRENT ROW)""")
> t.execute_insert("output")
> {code}
> we loose all of id == 1:
> {code:python}
> +I(0,0,0)
> +I(2,0,1)
> +I(4,0,2)
> +I(6,0,3)
> +I(8,0,4)
> ...
> {code}
> I observed this problem with various aggregate functions and both under
> 1.11.2 and 1.12rc1.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] rkhachatryan commented on pull request #14051: [FLINK-20107][network] Make InputChannel.statePersister immutable
rkhachatryan commented on pull request #14051: URL: https://github.com/apache/flink/pull/14051#issuecomment-726601295 @flinkbot run azure This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14058: [FLINK-20139][dispatcher] Enrich logs when MiniDispatcher shutting down
flinkbot edited a comment on pull request #14058: URL: https://github.com/apache/flink/pull/14058#issuecomment-726497597 ## CI report: * 7bf1538a0647dcf49515cbae50c28f7fe28918d0 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9544) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14062: [FLINK-17470] Send sigkill to hanging processes in standalone scripts
flinkbot edited a comment on pull request #14062: URL: https://github.com/apache/flink/pull/14062#issuecomment-726595780 ## CI report: * b598725a80261a4f0084bb13704629ca135fa332 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9553) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14051: [FLINK-20107][network] Make InputChannel.statePersister immutable
flinkbot edited a comment on pull request #14051: URL: https://github.com/apache/flink/pull/14051#issuecomment-726175036 ## CI report: * 932afb33fec9d29d35c0b29c3e3b4424c0db9210 Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9534) Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9554) * 9fcb590d47c3e38c04e8fb083108166589c97607 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14033: [FLINK-19979][e2e] Add sanity check after bash e2e tests for no leftovers
flinkbot edited a comment on pull request #14033: URL: https://github.com/apache/flink/pull/14033#issuecomment-725388043 ## CI report: * 43dee710e0f873adfd5b521183fc1492375d7800 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9467) * 9e5179d99ac3f1ee67388f6236a5ffd087941a94 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9552) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (FLINK-20144) Change link format to "{% link ... %}" in docs/ops dir
Xiao Huang created FLINK-20144:
--
Summary: Change link format to "{% link ... %}" in docs/ops dir
Key: FLINK-20144
URL: https://issues.apache.org/jira/browse/FLINK-20144
Project: Flink
Issue Type: Improvement
Components: Documentation
Reporter: Xiao Huang
Some documents' link format in docs/ops is "\{{site.baseurl}/... }". But it is
preferred to use "\{% link ... %}" format.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] wulei0302 opened a new pull request #14064: [FLINK-19454][docs-zh] Translate page 'Importing Flink into an IDE' into Chinese
wulei0302 opened a new pull request #14064: URL: https://github.com/apache/flink/pull/14064 ## What is the purpose of the change Translate page 'Importing Flink into an IDE' into Chinese The page url is https://ci.apache.org/projects/flink/flink-docs-release-1.11/flinkDev/ide_setup.html The markdown file is located in flink/docs/flinkDev/ide_setup.zh.md ## Brief change log - translate page 'Importing Flink into an IDE' ## Verifying this change This change is a trivial rework / code cleanup without any test coverage. ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): no - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: no - The serializers: no - The runtime per-record code paths (performance sensitive): no - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn/Mesos, ZooKeeper: no - The S3 file system connector: no ## Documentation - Does this pull request introduce a new feature? no - If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-20144) Change link format to "{% link ... %}" in docs/ops dir
[
https://issues.apache.org/jira/browse/FLINK-20144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17231258#comment-17231258
]
Xiao Huang commented on FLINK-20144:
Hi, [~sjwiesman].
Could you assign this ticket to me ?
Thanks.
> Change link format to "{% link ... %}" in docs/ops dir
> --
>
> Key: FLINK-20144
> URL: https://issues.apache.org/jira/browse/FLINK-20144
> Project: Flink
> Issue Type: Improvement
> Components: Documentation
>Reporter: Xiao Huang
>Priority: Minor
>
> Some documents' link format in docs/ops is "\{{site.baseurl}/... }". But it
> is preferred to use "\{% link ... %}" format.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-20145) Streaming job fails with IllegalStateException: Should only poll priority events
Robert Metzger created FLINK-20145:
--
Summary: Streaming job fails with IllegalStateException: Should
only poll priority events
Key: FLINK-20145
URL: https://issues.apache.org/jira/browse/FLINK-20145
Project: Flink
Issue Type: Bug
Components: Runtime / Checkpointing, Runtime / Network
Affects Versions: 1.12.0
Reporter: Robert Metzger
Fix For: 1.12.0
While testing the 1.12 release, I came across the following failure cause:
{code}
2020-11-13 09:41:52,110 WARN org.apache.flink.runtime.taskmanager.Task
[] - dynamic filter (3/4)#0 (b977944851531f96e5324e786f055eb7)
switched from RUNNING to FAILED.
java.lang.IllegalStateException: Should only poll priority events
at
org.apache.flink.util.Preconditions.checkState(Preconditions.java:198)
~[flink-dist_2.11-1.12.0.jar:1.12.0]
at
org.apache.flink.streaming.runtime.io.CheckpointedInputGate.processPriorityEvents(CheckpointedInputGate.java:116)
~[flink-dist_2.11-1.12.0.jar:1.12.0]
at
org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$1.runThrowing(StreamTaskActionExecutor.java:47)
~[flink-dist_2.11-1.12.0.jar:1.12.0]
at
org.apache.flink.streaming.runtime.tasks.mailbox.Mail.run(Mail.java:78)
~[flink-dist_2.11-1.12.0.jar:1.12.0]
at
org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.processMail(MailboxProcessor.java:283)
~[flink-dist_2.11-1.12.0.jar:1.12.0]
at
org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:184)
~[flink-dist_2.11-1.12.0.jar:1.12.0]
at
org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:577)
~[flink-dist_2.11-1.12.0.jar:1.12.0]
at
org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:541)
~[flink-dist_2.11-1.12.0.jar:1.12.0]
at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:722)
[flink-dist_2.11-1.12.0.jar:1.12.0]
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:547)
[flink-dist_2.11-1.12.0.jar:1.12.0]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_222]
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink-web] dawidwys opened a new pull request #391: [FLINK-20093] Link to SQL components download page
dawidwys opened a new pull request #391: URL: https://github.com/apache/flink-web/pull/391 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #14064: [FLINK-19454][docs-zh] Translate page 'Importing Flink into an IDE' into Chinese
flinkbot commented on pull request #14064: URL: https://github.com/apache/flink/pull/14064#issuecomment-726624859 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit 41bcdb30f6837f2f22dda55edafe8ba9ef5e98be (Fri Nov 13 08:48:26 UTC 2020) ✅no warnings Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] kl0u commented on pull request #14028: [FLINK-20020][client] Make UnsuccessfulExecutionException part of the JobClient.getJobExecutionResult() contract
kl0u commented on pull request #14028: URL: https://github.com/apache/flink/pull/14028#issuecomment-726624088 I am having a look right now @SteNicholas This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-20145) Streaming job fails with IllegalStateException: Should only poll priority events
[
https://issues.apache.org/jira/browse/FLINK-20145?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Metzger updated FLINK-20145:
---
Description:
While testing the 1.12 release, I came across the following failure cause:
{code}
2020-11-13 09:41:52,110 WARN org.apache.flink.runtime.taskmanager.Task
[] - dynamic filter (3/4)#0 (b977944851531f96e5324e786f055eb7)
switched from RUNNING to FAILED.
java.lang.IllegalStateException: Should only poll priority events
at
org.apache.flink.util.Preconditions.checkState(Preconditions.java:198)
~[flink-dist_2.11-1.12.0.jar:1.12.0]
at
org.apache.flink.streaming.runtime.io.CheckpointedInputGate.processPriorityEvents(CheckpointedInputGate.java:116)
~[flink-dist_2.11-1.12.0.jar:1.12.0]
at
org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$1.runThrowing(StreamTaskActionExecutor.java:47)
~[flink-dist_2.11-1.12.0.jar:1.12.0]
at
org.apache.flink.streaming.runtime.tasks.mailbox.Mail.run(Mail.java:78)
~[flink-dist_2.11-1.12.0.jar:1.12.0]
at
org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.processMail(MailboxProcessor.java:283)
~[flink-dist_2.11-1.12.0.jar:1.12.0]
at
org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:184)
~[flink-dist_2.11-1.12.0.jar:1.12.0]
at
org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:577)
~[flink-dist_2.11-1.12.0.jar:1.12.0]
at
org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:541)
~[flink-dist_2.11-1.12.0.jar:1.12.0]
at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:722)
[flink-dist_2.11-1.12.0.jar:1.12.0]
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:547)
[flink-dist_2.11-1.12.0.jar:1.12.0]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_222]
{code}
I have unaligned checkpointing enabled, the failing operator is a
CoFlatMapFunction. The error happend on all four TaskManagers, very soon after
job submission.
was:
While testing the 1.12 release, I came across the following failure cause:
{code}
2020-11-13 09:41:52,110 WARN org.apache.flink.runtime.taskmanager.Task
[] - dynamic filter (3/4)#0 (b977944851531f96e5324e786f055eb7)
switched from RUNNING to FAILED.
java.lang.IllegalStateException: Should only poll priority events
at
org.apache.flink.util.Preconditions.checkState(Preconditions.java:198)
~[flink-dist_2.11-1.12.0.jar:1.12.0]
at
org.apache.flink.streaming.runtime.io.CheckpointedInputGate.processPriorityEvents(CheckpointedInputGate.java:116)
~[flink-dist_2.11-1.12.0.jar:1.12.0]
at
org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$1.runThrowing(StreamTaskActionExecutor.java:47)
~[flink-dist_2.11-1.12.0.jar:1.12.0]
at
org.apache.flink.streaming.runtime.tasks.mailbox.Mail.run(Mail.java:78)
~[flink-dist_2.11-1.12.0.jar:1.12.0]
at
org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.processMail(MailboxProcessor.java:283)
~[flink-dist_2.11-1.12.0.jar:1.12.0]
at
org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:184)
~[flink-dist_2.11-1.12.0.jar:1.12.0]
at
org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:577)
~[flink-dist_2.11-1.12.0.jar:1.12.0]
at
org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:541)
~[flink-dist_2.11-1.12.0.jar:1.12.0]
at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:722)
[flink-dist_2.11-1.12.0.jar:1.12.0]
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:547)
[flink-dist_2.11-1.12.0.jar:1.12.0]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_222]
{code}
> Streaming job fails with IllegalStateException: Should only poll priority
> events
>
>
> Key: FLINK-20145
> URL: https://issues.apache.org/jira/browse/FLINK-20145
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Checkpointing, Runtime / Network
>Affects Versions: 1.12.0
>Reporter: Robert Metzger
>Priority: Critical
> Fix For: 1.12.0
>
>
> While testing the 1.12 release, I came across the following failure cause:
> {code}
> 2020-11-13 09:41:52,110 WARN org.apache.flink.runtime.taskmanager.Task
> [] - dynamic filter (3/4)#0 (b977944851531f96e5324e786f055eb7)
> switched from RUNNING to FAILED.
> java.lang.IllegalStateException: Should only poll priority events
> at
> org.apache.flink.util.Preconditions.checkState(Preconditions.java:198)
> ~[flink-dist_2.11-1.12.0.jar:1.12.0]
> at
> org.apache.flink.strea
[jira] [Commented] (FLINK-20145) Streaming job fails with IllegalStateException: Should only poll priority events
[
https://issues.apache.org/jira/browse/FLINK-20145?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17231264#comment-17231264
]
Robert Metzger commented on FLINK-20145:
When trying to reproduce the error, I also got this exception:
{code}
2020-11-13 09:54:44,634 INFO
org.apache.flink.runtime.checkpoint.channel.ChannelStateWriteRequestExecutorImpl
[] - dynamic filter (2/4)#0 discarding 0 drained requests
2020-11-13 09:54:44,674 WARN org.apache.flink.runtime.taskmanager.Task
[] - dynamic filter (2/4)#0 (290b30403ae7bc304c40cd5af2ec1b11)
switched from RUNNING to FAILED.
java.lang.NullPointerException: null
at java.util.ArrayDeque.addFirst(ArrayDeque.java:233) ~[?:1.8.0_222]
at
org.apache.flink.runtime.io.network.partition.PrioritizedDeque.addPriorityElement(PrioritizedDeque.java:67)
~[flink-dist_2.11-1.12.0.jar:1.12.0]
at
org.apache.flink.runtime.io.network.partition.PrioritizedDeque.add(PrioritizedDeque.java:101)
~[flink-dist_2.11-1.12.0.jar:1.12.0]
at
org.apache.flink.runtime.io.network.partition.consumer.UnionInputGate.processBufferOrEvent(UnionInputGate.java:219)
~[flink-dist_2.11-1.12.0.jar:1.12.0]
at
org.apache.flink.runtime.io.network.partition.consumer.UnionInputGate.waitAndGetNextData(UnionInputGate.java:207)
~[flink-dist_2.11-1.12.0.jar:1.12.0]
at
org.apache.flink.runtime.io.network.partition.consumer.UnionInputGate.getNextBufferOrEvent(UnionInputGate.java:176)
~[flink-dist_2.11-1.12.0.jar:1.12.0]
at
org.apache.flink.runtime.io.network.partition.consumer.UnionInputGate.pollNext(UnionInputGate.java:168)
~[flink-dist_2.11-1.12.0.jar:1.12.0]
at
org.apache.flink.streaming.runtime.io.CheckpointedInputGate.pollNext(CheckpointedInputGate.java:142)
~[flink-dist_2.11-1.12.0.jar:1.12.0]
at
org.apache.flink.streaming.runtime.io.StreamTaskNetworkInput.emitNext(StreamTaskNetworkInput.java:157)
~[flink-dist_2.11-1.12.0.jar:1.12.0]
at
org.apache.flink.streaming.runtime.io.StreamOneInputProcessor.processInput(StreamOneInputProcessor.java:67)
~[flink-dist_2.11-1.12.0.jar:1.12.0]
at
org.apache.flink.streaming.runtime.io.StreamTwoInputProcessor.processInput(StreamTwoInputProcessor.java:92)
~[flink-dist_2.11-1.12.0.jar:1.12.0]
at
org.apache.flink.streaming.runtime.tasks.StreamTask.processInput(StreamTask.java:372)
~[flink-dist_2.11-1.12.0.jar:1.12.0]
at
org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:186)
~[flink-dist_2.11-1.12.0.jar:1.12.0]
at
org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:577)
~[flink-dist_2.11-1.12.0.jar:1.12.0]
at
org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:541)
~[flink-dist_2.11-1.12.0.jar:1.12.0]
at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:722)
[flink-dist_2.11-1.12.0.jar:1.12.0]
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:547)
[flink-dist_2.11-1.12.0.jar:1.12.0]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_222]
{code}
> Streaming job fails with IllegalStateException: Should only poll priority
> events
>
>
> Key: FLINK-20145
> URL: https://issues.apache.org/jira/browse/FLINK-20145
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Checkpointing, Runtime / Network
>Affects Versions: 1.12.0
>Reporter: Robert Metzger
>Priority: Critical
> Fix For: 1.12.0
>
>
> While testing the 1.12 release, I came across the following failure cause:
> {code}
> 2020-11-13 09:41:52,110 WARN org.apache.flink.runtime.taskmanager.Task
> [] - dynamic filter (3/4)#0 (b977944851531f96e5324e786f055eb7)
> switched from RUNNING to FAILED.
> java.lang.IllegalStateException: Should only poll priority events
> at
> org.apache.flink.util.Preconditions.checkState(Preconditions.java:198)
> ~[flink-dist_2.11-1.12.0.jar:1.12.0]
> at
> org.apache.flink.streaming.runtime.io.CheckpointedInputGate.processPriorityEvents(CheckpointedInputGate.java:116)
> ~[flink-dist_2.11-1.12.0.jar:1.12.0]
> at
> org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$1.runThrowing(StreamTaskActionExecutor.java:47)
> ~[flink-dist_2.11-1.12.0.jar:1.12.0]
> at
> org.apache.flink.streaming.runtime.tasks.mailbox.Mail.run(Mail.java:78)
> ~[flink-dist_2.11-1.12.0.jar:1.12.0]
> at
> org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.processMail(MailboxProcessor.java:283)
> ~[flink-dist_2.11-1.12.0.jar:1.12.0]
> at
> org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:184)
> ~[flink-dist_2
[jira] [Commented] (FLINK-19852) Managed memory released check can block IterativeTask
[ https://issues.apache.org/jira/browse/FLINK-19852?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17231267#comment-17231267 ] Arvid Heise commented on FLINK-19852: - Another option that I discussed with Andrey offline is to not reallocate pages at all during batch iterations. Memory usage on batch-side should be rather safe. [~sewen] WDYT? > Managed memory released check can block IterativeTask > - > > Key: FLINK-19852 > URL: https://issues.apache.org/jira/browse/FLINK-19852 > Project: Flink > Issue Type: Bug > Components: Runtime / Task >Affects Versions: 1.11.0, 1.10.2, 1.11.1, 1.11.2 >Reporter: shaomeng.wang >Priority: Critical > Attachments: image-2020-10-28-17-48-28-395.png, > image-2020-10-28-17-48-48-583.png > > > UnsafeMemoryBudget#reserveMemory, called on TempBarrier, needs time to wait > on GC of all allocated/released managed memory at every iteration. > > stack: > !image-2020-10-28-17-48-48-583.png! > new TempBarrier in BatchTask > !image-2020-10-28-17-48-28-395.png! > > These will be very slow than before. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-20090) Expose SlotId / SlotSharingGroup in Rest API
[
https://issues.apache.org/jira/browse/FLINK-20090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17231268#comment-17231268
]
Maximilian Michels commented on FLINK-20090:
Exactly, the actual SlodId is enough to identify which tasks share the same
slot.
> Expose SlotId / SlotSharingGroup in Rest API
> -
>
> Key: FLINK-20090
> URL: https://issues.apache.org/jira/browse/FLINK-20090
> Project: Flink
> Issue Type: New Feature
> Components: Runtime / REST
>Reporter: Maximilian Michels
>Priority: Minor
>
> There is no information on slot sharing exposed via the Rest API which would
> be useful to monitor how tasks are assigned to task slots.
> We could include the SlotId in {{SubtaskExecutionAttemptDetailsInfo}} and
> provide a list of slots in {{TaskManagersInfo}}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-20145) Streaming job fails with IllegalStateException: Should only poll priority events
[
https://issues.apache.org/jira/browse/FLINK-20145?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Metzger updated FLINK-20145:
---
Description:
While testing the 1.12 release, I came across the following failure cause:
{code}
2020-11-13 09:41:52,110 WARN org.apache.flink.runtime.taskmanager.Task
[] - dynamic filter (3/4)#0 (b977944851531f96e5324e786f055eb7)
switched from RUNNING to FAILED.
java.lang.IllegalStateException: Should only poll priority events
at
org.apache.flink.util.Preconditions.checkState(Preconditions.java:198)
~[flink-dist_2.11-1.12.0.jar:1.12.0]
at
org.apache.flink.streaming.runtime.io.CheckpointedInputGate.processPriorityEvents(CheckpointedInputGate.java:116)
~[flink-dist_2.11-1.12.0.jar:1.12.0]
at
org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$1.runThrowing(StreamTaskActionExecutor.java:47)
~[flink-dist_2.11-1.12.0.jar:1.12.0]
at
org.apache.flink.streaming.runtime.tasks.mailbox.Mail.run(Mail.java:78)
~[flink-dist_2.11-1.12.0.jar:1.12.0]
at
org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.processMail(MailboxProcessor.java:283)
~[flink-dist_2.11-1.12.0.jar:1.12.0]
at
org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:184)
~[flink-dist_2.11-1.12.0.jar:1.12.0]
at
org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:577)
~[flink-dist_2.11-1.12.0.jar:1.12.0]
at
org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:541)
~[flink-dist_2.11-1.12.0.jar:1.12.0]
at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:722)
[flink-dist_2.11-1.12.0.jar:1.12.0]
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:547)
[flink-dist_2.11-1.12.0.jar:1.12.0]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_222]
{code}
I have unaligned checkpointing enabled, the failing operator is a
CoFlatMapFunction. The error happend on all four TaskManagers, very soon after
job submission. The error doesn't happen when unaligned checkpointing is
disabled.
was:
While testing the 1.12 release, I came across the following failure cause:
{code}
2020-11-13 09:41:52,110 WARN org.apache.flink.runtime.taskmanager.Task
[] - dynamic filter (3/4)#0 (b977944851531f96e5324e786f055eb7)
switched from RUNNING to FAILED.
java.lang.IllegalStateException: Should only poll priority events
at
org.apache.flink.util.Preconditions.checkState(Preconditions.java:198)
~[flink-dist_2.11-1.12.0.jar:1.12.0]
at
org.apache.flink.streaming.runtime.io.CheckpointedInputGate.processPriorityEvents(CheckpointedInputGate.java:116)
~[flink-dist_2.11-1.12.0.jar:1.12.0]
at
org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$1.runThrowing(StreamTaskActionExecutor.java:47)
~[flink-dist_2.11-1.12.0.jar:1.12.0]
at
org.apache.flink.streaming.runtime.tasks.mailbox.Mail.run(Mail.java:78)
~[flink-dist_2.11-1.12.0.jar:1.12.0]
at
org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.processMail(MailboxProcessor.java:283)
~[flink-dist_2.11-1.12.0.jar:1.12.0]
at
org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:184)
~[flink-dist_2.11-1.12.0.jar:1.12.0]
at
org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:577)
~[flink-dist_2.11-1.12.0.jar:1.12.0]
at
org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:541)
~[flink-dist_2.11-1.12.0.jar:1.12.0]
at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:722)
[flink-dist_2.11-1.12.0.jar:1.12.0]
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:547)
[flink-dist_2.11-1.12.0.jar:1.12.0]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_222]
{code}
I have unaligned checkpointing enabled, the failing operator is a
CoFlatMapFunction. The error happend on all four TaskManagers, very soon after
job submission.
> Streaming job fails with IllegalStateException: Should only poll priority
> events
>
>
> Key: FLINK-20145
> URL: https://issues.apache.org/jira/browse/FLINK-20145
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Checkpointing, Runtime / Network
>Affects Versions: 1.12.0
>Reporter: Robert Metzger
>Priority: Critical
> Fix For: 1.12.0
>
>
> While testing the 1.12 release, I came across the following failure cause:
> {code}
> 2020-11-13 09:41:52,110 WARN org.apache.flink.runtime.taskmanager.Task
> [] - dynamic filter (3/4)#0 (b977944851531f96e5324e786f055eb7)
> switched from RUNNING to FAILE
[GitHub] [flink] flinkbot edited a comment on pull request #14032: [FLINK-19863][tests] check embedded zookeeper status when start/close hbase resource
flinkbot edited a comment on pull request #14032: URL: https://github.com/apache/flink/pull/14032#issuecomment-725359904 ## CI report: * 3fb3d2ff2f59cb246824d3971d19f2abe87ca060 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9545) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-20092) [Java 11] Multi-thread Flink compilation not working
[
https://issues.apache.org/jira/browse/FLINK-20092?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17231271#comment-17231271
]
Maciej Bryński commented on FLINK-20092:
[~chesnay]
Maybe we can avoid to create dependency reduced pom ? Do we need it ?
{code:java}
false
{code}
> [Java 11] Multi-thread Flink compilation not working
>
>
> Key: FLINK-20092
> URL: https://issues.apache.org/jira/browse/FLINK-20092
> Project: Flink
> Issue Type: Bug
> Components: Build System
>Reporter: Maciej Bryński
>Priority: Major
>
> I'd like to use maven -T option when compiling flink.
> {code:java}
> mvn -T 2C clean install -D"scala-2.12" -DskipTests{code}
> Unfortunately my build is stuck on:
> {code:java}
> [INFO] --- maven-shade-plugin:3.2.1:shade (shade-flink) @
> flink-fs-hadoop-shaded ---
> [INFO] Including org.apache.hadoop:hadoop-common:jar:3.1.0 in the shaded jar.
> [INFO] Including org.apache.hadoop:hadoop-annotations:jar:3.1.0 in the shaded
> jar.
> [INFO] Including com.google.guava:guava:jar:11.0.2 in the shaded jar.
> [INFO] Including commons-io:commons-io:jar:2.7 in the shaded jar.
> [INFO] Including commons-collections:commons-collections:jar:3.2.2 in the
> shaded jar.
> [INFO] Including commons-logging:commons-logging:jar:1.1.3 in the shaded jar.
> [INFO] Including commons-lang:commons-lang:jar:2.6 in the shaded jar.
> [INFO] Including commons-beanutils:commons-beanutils:jar:1.9.3 in the shaded
> jar.
> [INFO] Including org.apache.commons:commons-configuration2:jar:2.1.1 in the
> shaded jar.
> [INFO] Including org.apache.commons:commons-lang3:jar:3.3.2 in the shaded jar.
> [INFO] Including com.google.re2j:re2j:jar:1.1 in the shaded jar.
> [INFO] Including org.apache.hadoop:hadoop-auth:jar:3.1.0 in the shaded jar.
> [INFO] Including org.apache.htrace:htrace-core4:jar:4.1.0-incubating in the
> shaded jar.
> [INFO] Including com.fasterxml.jackson.core:jackson-databind:jar:2.10.1 in
> the shaded jar.
> [INFO] Including com.fasterxml.jackson.core:jackson-annotations:jar:2.10.1 in
> the shaded jar.
> [INFO] Including com.fasterxml.jackson.core:jackson-core:jar:2.10.1 in the
> shaded jar.
> [INFO] Including org.codehaus.woodstox:stax2-api:jar:3.1.4 in the shaded jar.
> [INFO] Including com.fasterxml.woodstox:woodstox-core:jar:5.0.3 in the shaded
> jar.
> [INFO] Including org.apache.flink:force-shading:jar:1.12-SNAPSHOT in the
> shaded jar.
> [INFO] No artifact matching filter io.netty:netty
> [WARNING] Discovered module-info.class. Shading will break its strong
> encapsulation.
> [WARNING] Discovered module-info.class. Shading will break its strong
> encapsulation.
> [WARNING] Discovered module-info.class. Shading will break its strong
> encapsulation.
> [INFO] Replacing original artifact with shaded artifact.
> [INFO] Replacing
> /home/maverick/flink/flink-filesystems/flink-fs-hadoop-shaded/target/flink-fs-hadoop-shaded-1.12-SNAPSHOT.jar
> with
> /home/maverick/flink/flink-filesystems/flink-fs-hadoop-shaded/target/flink-fs-hadoop-shaded-1.12-SNAPSHOT-shaded.jar
> [INFO] Dependency-reduced POM written at:
> /home/maverick/flink/flink-filesystems/flink-fs-hadoop-shaded/target/dependency-reduced-pom.xml
> {code}
> Can we make flink compilation working with multiple maven threads ?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] dannycranmer commented on pull request #14049: [FLINK-20043][kinesis] Add flink-sql-connector-kinesis package
dannycranmer commented on pull request #14049: URL: https://github.com/apache/flink/pull/14049#issuecomment-726641239 Rebased and updated as per @sjwiesman comment. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14034: [FLINK-20088][kinesis] Fix issue encountered when Polling consumer using timestamp with empty shard
flinkbot edited a comment on pull request #14034: URL: https://github.com/apache/flink/pull/14034#issuecomment-725425798 ## CI report: * dc4f57931b735d886d06ae4e302235353367c34a Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9469) * a18a4e6d3f099b269fe0fb5618faf2a2cd53121a UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14051: [FLINK-20107][network] Make InputChannel.statePersister immutable
flinkbot edited a comment on pull request #14051: URL: https://github.com/apache/flink/pull/14051#issuecomment-726175036 ## CI report: * 932afb33fec9d29d35c0b29c3e3b4424c0db9210 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9554) Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9534) * 9fcb590d47c3e38c04e8fb083108166589c97607 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14052: [FLINK-20030][network] Rewrite RemoteInputChannel#getInflightBuffersUnsafe to use sequence numbers
flinkbot edited a comment on pull request #14052: URL: https://github.com/apache/flink/pull/14052#issuecomment-726216034 ## CI report: * 14e0929bf9cebd4db7c60c5928d71e631fb4ad47 UNKNOWN * 0d6c68e97c1c6d23ca2870932db0cdfeddebcafd Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9528) * 9f30c9730919038a0247cd4b7a2929df05ee669f UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #14064: [FLINK-19454][docs-zh] Translate page 'Importing Flink into an IDE' into Chinese
flinkbot commented on pull request #14064: URL: https://github.com/apache/flink/pull/14064#issuecomment-726642209 ## CI report: * 41bcdb30f6837f2f22dda55edafe8ba9ef5e98be UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink-web] dawidwys merged pull request #391: [FLINK-20093] Link to SQL components download page
dawidwys merged pull request #391: URL: https://github.com/apache/flink-web/pull/391 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-20143) use `yarn.provided.lib.dirs` config deploy job failed in yarn per job mode
[
https://issues.apache.org/jira/browse/FLINK-20143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17231276#comment-17231276
]
Kostas Kloudas commented on FLINK-20143:
Can you share with us the job manager and task manager logs [~zhisheng]? This
may help figuring out what is happening.
> use `yarn.provided.lib.dirs` config deploy job failed in yarn per job mode
> --
>
> Key: FLINK-20143
> URL: https://issues.apache.org/jira/browse/FLINK-20143
> Project: Flink
> Issue Type: Bug
> Components: Client / Job Submission, Deployment / YARN
>Affects Versions: 1.12.0
>Reporter: zhisheng
>Priority: Major
>
> use follow command deploy flink job to yarn failed
> {code:java}
> ./bin/flink run -m yarn-cluster -d -ynm flink-1.12-test -ytm 3g -yjm 3g -yD
> yarn.provided.lib.dirs=hdfs:///flink/flink-1.12-SNAPSHOT/lib
> ./examples/streaming/StateMachineExample.jar
> {code}
> log:
> {code:java}
> $ ./bin/flink run -m yarn-cluster -d -ynm flink-1.12-test -ytm 3g -yjm 3g -yD
> yarn.provided.lib.dirs=hdfs:///flink/flink-1.12-SNAPSHOT/lib
> ./examples/streaming/StateMachineExample.jar$ ./bin/flink run -m yarn-cluster
> -d -ynm flink-1.12-test -ytm 3g -yjm 3g -yD
> yarn.provided.lib.dirs=hdfs:///flink/flink-1.12-SNAPSHOT/lib
> ./examples/streaming/StateMachineExample.jarSLF4J: Class path contains
> multiple SLF4J bindings.SLF4J: Found binding in
> [jar:file:/data1/app/flink-1.12-SNAPSHOT/lib/log4j-slf4j-impl-2.12.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J:
> Found binding in
> [jar:file:/data1/app/hadoop-2.7.3-snappy-32core12disk/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J:
> Found binding in
> [jar:file:/data1/app/hadoop-2.7.3-snappy-32core12disk/share/hadoop/tools/lib/hadoop-aliyun-2.9.2-jar-with-dependencies.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J:
> See http://www.slf4j.org/codes.html#multiple_bindings for an
> explanation.SLF4J: Actual binding is of type
> [org.apache.logging.slf4j.Log4jLoggerFactory]2020-11-13 16:14:30,347 INFO
> org.apache.flink.yarn.cli.FlinkYarnSessionCli [] - Dynamic
> Property set:
> yarn.provided.lib.dirs=hdfs:///flink/flink-1.12-SNAPSHOT/lib2020-11-13
> 16:14:30,347 INFO org.apache.flink.yarn.cli.FlinkYarnSessionCli
> [] - Dynamic Property set:
> yarn.provided.lib.dirs=hdfs:///flink/flink-1.12-SNAPSHOT/libUsage with
> built-in data generator: StateMachineExample [--error-rate
> ] [--sleep ]Usage
> with Kafka: StateMachineExample --kafka-topic [--brokers
> ]Options for both the above setups: [--backend ]
> [--checkpoint-dir ] [--async-checkpoints ]
> [--incremental-checkpoints ] [--output OR null for
> stdout]
> Using standalone source with error rate 0.00 and sleep delay 1 millis
> 2020-11-13 16:14:30,706 WARN
> org.apache.flink.yarn.configuration.YarnLogConfigUtil [] - The
> configuration directory ('/data1/app/flink-1.12-SNAPSHOT/conf') already
> contains a LOG4J config file.If you want to use logback, then please delete
> or rename the log configuration file.2020-11-13 16:14:30,947 INFO
> org.apache.hadoop.yarn.client.AHSProxy [] - Connecting
> to Application History server at
> FAT-hadoopuat-69117.vm.dc01.hellocloud.tech/10.69.1.17:102002020-11-13
> 16:14:30,958 INFO org.apache.flink.yarn.YarnClusterDescriptor
> [] - No path for the flink jar passed. Using the location of class
> org.apache.flink.yarn.YarnClusterDescriptor to locate the jar2020-11-13
> 16:14:31,065 INFO
> org.apache.hadoop.yarn.client.ConfiguredRMFailoverProxyProvider [] - Failing
> over to rm22020-11-13 16:14:31,130 INFO
> org.apache.flink.yarn.YarnClusterDescriptor [] - The
> configured JobManager memory is 3072 MB. YARN will allocate 4096 MB to make
> up an integer multiple of its minimum allocation memory (2048 MB, configured
> via 'yarn.scheduler.minimum-allocation-mb'). The extra 1024 MB may not be
> used by Flink.2020-11-13 16:14:31,130 INFO
> org.apache.flink.yarn.YarnClusterDescriptor [] - The
> configured TaskManager memory is 3072 MB. YARN will allocate 4096 MB to make
> up an integer multiple of its minimum allocation memory (2048 MB, configured
> via 'yarn.scheduler.minimum-allocation-mb'). The extra 1024 MB may not be
> used by Flink.2020-11-13 16:14:31,130 INFO
> org.apache.flink.yarn.YarnClusterDescriptor [] - Cluster
> specification: ClusterSpecification{masterMemoryMB=3072,
> taskManagerMemoryMB=3072, slotsPerTaskManager=2}2020-11-13 16:14:31,681 WARN
> org.apache.hadoop.hdfs.shortcircuit.DomainSocketFactory [] - The
> short-circuit local reads feature cannot be used because li
[jira] [Commented] (FLINK-20143) use `yarn.provided.lib.dirs` config deploy job failed in yarn per job mode
[
https://issues.apache.org/jira/browse/FLINK-20143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17231283#comment-17231283
]
Kostas Kloudas commented on FLINK-20143:
Also I think that your second command is not correct. You are using {{-t}}
which activates the {{GenericCLI}} but then you specify parameters using the
{{YarnSessionCLI}} convention of putting a {{-y}} as a prefix. Can you verify
if the memory specifications you put are picked up?
> use `yarn.provided.lib.dirs` config deploy job failed in yarn per job mode
> --
>
> Key: FLINK-20143
> URL: https://issues.apache.org/jira/browse/FLINK-20143
> Project: Flink
> Issue Type: Bug
> Components: Client / Job Submission, Deployment / YARN
>Affects Versions: 1.12.0
>Reporter: zhisheng
>Priority: Major
>
> use follow command deploy flink job to yarn failed
> {code:java}
> ./bin/flink run -m yarn-cluster -d -ynm flink-1.12-test -ytm 3g -yjm 3g -yD
> yarn.provided.lib.dirs=hdfs:///flink/flink-1.12-SNAPSHOT/lib
> ./examples/streaming/StateMachineExample.jar
> {code}
> log:
> {code:java}
> $ ./bin/flink run -m yarn-cluster -d -ynm flink-1.12-test -ytm 3g -yjm 3g -yD
> yarn.provided.lib.dirs=hdfs:///flink/flink-1.12-SNAPSHOT/lib
> ./examples/streaming/StateMachineExample.jar$ ./bin/flink run -m yarn-cluster
> -d -ynm flink-1.12-test -ytm 3g -yjm 3g -yD
> yarn.provided.lib.dirs=hdfs:///flink/flink-1.12-SNAPSHOT/lib
> ./examples/streaming/StateMachineExample.jarSLF4J: Class path contains
> multiple SLF4J bindings.SLF4J: Found binding in
> [jar:file:/data1/app/flink-1.12-SNAPSHOT/lib/log4j-slf4j-impl-2.12.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J:
> Found binding in
> [jar:file:/data1/app/hadoop-2.7.3-snappy-32core12disk/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J:
> Found binding in
> [jar:file:/data1/app/hadoop-2.7.3-snappy-32core12disk/share/hadoop/tools/lib/hadoop-aliyun-2.9.2-jar-with-dependencies.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J:
> See http://www.slf4j.org/codes.html#multiple_bindings for an
> explanation.SLF4J: Actual binding is of type
> [org.apache.logging.slf4j.Log4jLoggerFactory]2020-11-13 16:14:30,347 INFO
> org.apache.flink.yarn.cli.FlinkYarnSessionCli [] - Dynamic
> Property set:
> yarn.provided.lib.dirs=hdfs:///flink/flink-1.12-SNAPSHOT/lib2020-11-13
> 16:14:30,347 INFO org.apache.flink.yarn.cli.FlinkYarnSessionCli
> [] - Dynamic Property set:
> yarn.provided.lib.dirs=hdfs:///flink/flink-1.12-SNAPSHOT/libUsage with
> built-in data generator: StateMachineExample [--error-rate
> ] [--sleep ]Usage
> with Kafka: StateMachineExample --kafka-topic [--brokers
> ]Options for both the above setups: [--backend ]
> [--checkpoint-dir ] [--async-checkpoints ]
> [--incremental-checkpoints ] [--output OR null for
> stdout]
> Using standalone source with error rate 0.00 and sleep delay 1 millis
> 2020-11-13 16:14:30,706 WARN
> org.apache.flink.yarn.configuration.YarnLogConfigUtil [] - The
> configuration directory ('/data1/app/flink-1.12-SNAPSHOT/conf') already
> contains a LOG4J config file.If you want to use logback, then please delete
> or rename the log configuration file.2020-11-13 16:14:30,947 INFO
> org.apache.hadoop.yarn.client.AHSProxy [] - Connecting
> to Application History server at
> FAT-hadoopuat-69117.vm.dc01.hellocloud.tech/10.69.1.17:102002020-11-13
> 16:14:30,958 INFO org.apache.flink.yarn.YarnClusterDescriptor
> [] - No path for the flink jar passed. Using the location of class
> org.apache.flink.yarn.YarnClusterDescriptor to locate the jar2020-11-13
> 16:14:31,065 INFO
> org.apache.hadoop.yarn.client.ConfiguredRMFailoverProxyProvider [] - Failing
> over to rm22020-11-13 16:14:31,130 INFO
> org.apache.flink.yarn.YarnClusterDescriptor [] - The
> configured JobManager memory is 3072 MB. YARN will allocate 4096 MB to make
> up an integer multiple of its minimum allocation memory (2048 MB, configured
> via 'yarn.scheduler.minimum-allocation-mb'). The extra 1024 MB may not be
> used by Flink.2020-11-13 16:14:31,130 INFO
> org.apache.flink.yarn.YarnClusterDescriptor [] - The
> configured TaskManager memory is 3072 MB. YARN will allocate 4096 MB to make
> up an integer multiple of its minimum allocation memory (2048 MB, configured
> via 'yarn.scheduler.minimum-allocation-mb'). The extra 1024 MB may not be
> used by Flink.2020-11-13 16:14:31,130 INFO
> org.apache.flink.yarn.YarnClusterDescriptor [] - Cluster
> specification: ClusterSpecification{masterMemoryMB=3072,
> taskManagerMemoryMB=3072, slotsPerTaskManager=2}2
[jira] [Comment Edited] (FLINK-20143) use `yarn.provided.lib.dirs` config deploy job failed in yarn per job mode
[
https://issues.apache.org/jira/browse/FLINK-20143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17231283#comment-17231283
]
Kostas Kloudas edited comment on FLINK-20143 at 11/13/20, 9:19 AM:
---
Also I think that your second command is not correct. You are using {{-t}}
which activates the {{GenericCLI}} but then you specify parameters using the
{{YarnSessionCLI}} convention of putting a {{-y}} as a prefix. Can you verify
if the memory specifications you put are picked up and the shared.lib.dir is
actually. used?
was (Author: kkl0u):
Also I think that your second command is not correct. You are using {{-t}}
which activates the {{GenericCLI}} but then you specify parameters using the
{{YarnSessionCLI}} convention of putting a {{-y}} as a prefix. Can you verify
if the memory specifications you put are picked up?
> use `yarn.provided.lib.dirs` config deploy job failed in yarn per job mode
> --
>
> Key: FLINK-20143
> URL: https://issues.apache.org/jira/browse/FLINK-20143
> Project: Flink
> Issue Type: Bug
> Components: Client / Job Submission, Deployment / YARN
>Affects Versions: 1.12.0
>Reporter: zhisheng
>Priority: Major
>
> use follow command deploy flink job to yarn failed
> {code:java}
> ./bin/flink run -m yarn-cluster -d -ynm flink-1.12-test -ytm 3g -yjm 3g -yD
> yarn.provided.lib.dirs=hdfs:///flink/flink-1.12-SNAPSHOT/lib
> ./examples/streaming/StateMachineExample.jar
> {code}
> log:
> {code:java}
> $ ./bin/flink run -m yarn-cluster -d -ynm flink-1.12-test -ytm 3g -yjm 3g -yD
> yarn.provided.lib.dirs=hdfs:///flink/flink-1.12-SNAPSHOT/lib
> ./examples/streaming/StateMachineExample.jar$ ./bin/flink run -m yarn-cluster
> -d -ynm flink-1.12-test -ytm 3g -yjm 3g -yD
> yarn.provided.lib.dirs=hdfs:///flink/flink-1.12-SNAPSHOT/lib
> ./examples/streaming/StateMachineExample.jarSLF4J: Class path contains
> multiple SLF4J bindings.SLF4J: Found binding in
> [jar:file:/data1/app/flink-1.12-SNAPSHOT/lib/log4j-slf4j-impl-2.12.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J:
> Found binding in
> [jar:file:/data1/app/hadoop-2.7.3-snappy-32core12disk/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J:
> Found binding in
> [jar:file:/data1/app/hadoop-2.7.3-snappy-32core12disk/share/hadoop/tools/lib/hadoop-aliyun-2.9.2-jar-with-dependencies.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J:
> See http://www.slf4j.org/codes.html#multiple_bindings for an
> explanation.SLF4J: Actual binding is of type
> [org.apache.logging.slf4j.Log4jLoggerFactory]2020-11-13 16:14:30,347 INFO
> org.apache.flink.yarn.cli.FlinkYarnSessionCli [] - Dynamic
> Property set:
> yarn.provided.lib.dirs=hdfs:///flink/flink-1.12-SNAPSHOT/lib2020-11-13
> 16:14:30,347 INFO org.apache.flink.yarn.cli.FlinkYarnSessionCli
> [] - Dynamic Property set:
> yarn.provided.lib.dirs=hdfs:///flink/flink-1.12-SNAPSHOT/libUsage with
> built-in data generator: StateMachineExample [--error-rate
> ] [--sleep ]Usage
> with Kafka: StateMachineExample --kafka-topic [--brokers
> ]Options for both the above setups: [--backend ]
> [--checkpoint-dir ] [--async-checkpoints ]
> [--incremental-checkpoints ] [--output OR null for
> stdout]
> Using standalone source with error rate 0.00 and sleep delay 1 millis
> 2020-11-13 16:14:30,706 WARN
> org.apache.flink.yarn.configuration.YarnLogConfigUtil [] - The
> configuration directory ('/data1/app/flink-1.12-SNAPSHOT/conf') already
> contains a LOG4J config file.If you want to use logback, then please delete
> or rename the log configuration file.2020-11-13 16:14:30,947 INFO
> org.apache.hadoop.yarn.client.AHSProxy [] - Connecting
> to Application History server at
> FAT-hadoopuat-69117.vm.dc01.hellocloud.tech/10.69.1.17:102002020-11-13
> 16:14:30,958 INFO org.apache.flink.yarn.YarnClusterDescriptor
> [] - No path for the flink jar passed. Using the location of class
> org.apache.flink.yarn.YarnClusterDescriptor to locate the jar2020-11-13
> 16:14:31,065 INFO
> org.apache.hadoop.yarn.client.ConfiguredRMFailoverProxyProvider [] - Failing
> over to rm22020-11-13 16:14:31,130 INFO
> org.apache.flink.yarn.YarnClusterDescriptor [] - The
> configured JobManager memory is 3072 MB. YARN will allocate 4096 MB to make
> up an integer multiple of its minimum allocation memory (2048 MB, configured
> via 'yarn.scheduler.minimum-allocation-mb'). The extra 1024 MB may not be
> used by Flink.2020-11-13 16:14:31,130 INFO
> org.apache.flink.yarn.YarnClusterDescriptor [] - The
> configured TaskManager memory is 3072 MB. YARN will allocate 4096 M
[jira] [Commented] (FLINK-20143) use `yarn.provided.lib.dirs` config deploy job failed in yarn per job mode
[
https://issues.apache.org/jira/browse/FLINK-20143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17231284#comment-17231284
]
zhisheng commented on FLINK-20143:
--
[~kkl0u] it does not have jobmanager log and taskmanager log
> use `yarn.provided.lib.dirs` config deploy job failed in yarn per job mode
> --
>
> Key: FLINK-20143
> URL: https://issues.apache.org/jira/browse/FLINK-20143
> Project: Flink
> Issue Type: Bug
> Components: Client / Job Submission, Deployment / YARN
>Affects Versions: 1.12.0
>Reporter: zhisheng
>Priority: Major
>
> use follow command deploy flink job to yarn failed
> {code:java}
> ./bin/flink run -m yarn-cluster -d -ynm flink-1.12-test -ytm 3g -yjm 3g -yD
> yarn.provided.lib.dirs=hdfs:///flink/flink-1.12-SNAPSHOT/lib
> ./examples/streaming/StateMachineExample.jar
> {code}
> log:
> {code:java}
> $ ./bin/flink run -m yarn-cluster -d -ynm flink-1.12-test -ytm 3g -yjm 3g -yD
> yarn.provided.lib.dirs=hdfs:///flink/flink-1.12-SNAPSHOT/lib
> ./examples/streaming/StateMachineExample.jar$ ./bin/flink run -m yarn-cluster
> -d -ynm flink-1.12-test -ytm 3g -yjm 3g -yD
> yarn.provided.lib.dirs=hdfs:///flink/flink-1.12-SNAPSHOT/lib
> ./examples/streaming/StateMachineExample.jarSLF4J: Class path contains
> multiple SLF4J bindings.SLF4J: Found binding in
> [jar:file:/data1/app/flink-1.12-SNAPSHOT/lib/log4j-slf4j-impl-2.12.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J:
> Found binding in
> [jar:file:/data1/app/hadoop-2.7.3-snappy-32core12disk/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J:
> Found binding in
> [jar:file:/data1/app/hadoop-2.7.3-snappy-32core12disk/share/hadoop/tools/lib/hadoop-aliyun-2.9.2-jar-with-dependencies.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J:
> See http://www.slf4j.org/codes.html#multiple_bindings for an
> explanation.SLF4J: Actual binding is of type
> [org.apache.logging.slf4j.Log4jLoggerFactory]2020-11-13 16:14:30,347 INFO
> org.apache.flink.yarn.cli.FlinkYarnSessionCli [] - Dynamic
> Property set:
> yarn.provided.lib.dirs=hdfs:///flink/flink-1.12-SNAPSHOT/lib2020-11-13
> 16:14:30,347 INFO org.apache.flink.yarn.cli.FlinkYarnSessionCli
> [] - Dynamic Property set:
> yarn.provided.lib.dirs=hdfs:///flink/flink-1.12-SNAPSHOT/libUsage with
> built-in data generator: StateMachineExample [--error-rate
> ] [--sleep ]Usage
> with Kafka: StateMachineExample --kafka-topic [--brokers
> ]Options for both the above setups: [--backend ]
> [--checkpoint-dir ] [--async-checkpoints ]
> [--incremental-checkpoints ] [--output OR null for
> stdout]
> Using standalone source with error rate 0.00 and sleep delay 1 millis
> 2020-11-13 16:14:30,706 WARN
> org.apache.flink.yarn.configuration.YarnLogConfigUtil [] - The
> configuration directory ('/data1/app/flink-1.12-SNAPSHOT/conf') already
> contains a LOG4J config file.If you want to use logback, then please delete
> or rename the log configuration file.2020-11-13 16:14:30,947 INFO
> org.apache.hadoop.yarn.client.AHSProxy [] - Connecting
> to Application History server at
> FAT-hadoopuat-69117.vm.dc01.hellocloud.tech/10.69.1.17:102002020-11-13
> 16:14:30,958 INFO org.apache.flink.yarn.YarnClusterDescriptor
> [] - No path for the flink jar passed. Using the location of class
> org.apache.flink.yarn.YarnClusterDescriptor to locate the jar2020-11-13
> 16:14:31,065 INFO
> org.apache.hadoop.yarn.client.ConfiguredRMFailoverProxyProvider [] - Failing
> over to rm22020-11-13 16:14:31,130 INFO
> org.apache.flink.yarn.YarnClusterDescriptor [] - The
> configured JobManager memory is 3072 MB. YARN will allocate 4096 MB to make
> up an integer multiple of its minimum allocation memory (2048 MB, configured
> via 'yarn.scheduler.minimum-allocation-mb'). The extra 1024 MB may not be
> used by Flink.2020-11-13 16:14:31,130 INFO
> org.apache.flink.yarn.YarnClusterDescriptor [] - The
> configured TaskManager memory is 3072 MB. YARN will allocate 4096 MB to make
> up an integer multiple of its minimum allocation memory (2048 MB, configured
> via 'yarn.scheduler.minimum-allocation-mb'). The extra 1024 MB may not be
> used by Flink.2020-11-13 16:14:31,130 INFO
> org.apache.flink.yarn.YarnClusterDescriptor [] - Cluster
> specification: ClusterSpecification{masterMemoryMB=3072,
> taskManagerMemoryMB=3072, slotsPerTaskManager=2}2020-11-13 16:14:31,681 WARN
> org.apache.hadoop.hdfs.shortcircuit.DomainSocketFactory [] - The
> short-circuit local reads feature cannot be used because libhadoop cannot be
> loaded.2020-11-13 16:14:33,417 INFO
> org.apache.
[jira] [Updated] (FLINK-20143) use `yarn.provided.lib.dirs` config deploy job failed in yarn per job mode
[
https://issues.apache.org/jira/browse/FLINK-20143?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
zhisheng updated FLINK-20143:
-
Attachment: image-2020-11-13-17-21-47-751.png
> use `yarn.provided.lib.dirs` config deploy job failed in yarn per job mode
> --
>
> Key: FLINK-20143
> URL: https://issues.apache.org/jira/browse/FLINK-20143
> Project: Flink
> Issue Type: Bug
> Components: Client / Job Submission, Deployment / YARN
>Affects Versions: 1.12.0
>Reporter: zhisheng
>Priority: Major
> Attachments: image-2020-11-13-17-21-47-751.png
>
>
> use follow command deploy flink job to yarn failed
> {code:java}
> ./bin/flink run -m yarn-cluster -d -ynm flink-1.12-test -ytm 3g -yjm 3g -yD
> yarn.provided.lib.dirs=hdfs:///flink/flink-1.12-SNAPSHOT/lib
> ./examples/streaming/StateMachineExample.jar
> {code}
> log:
> {code:java}
> $ ./bin/flink run -m yarn-cluster -d -ynm flink-1.12-test -ytm 3g -yjm 3g -yD
> yarn.provided.lib.dirs=hdfs:///flink/flink-1.12-SNAPSHOT/lib
> ./examples/streaming/StateMachineExample.jar$ ./bin/flink run -m yarn-cluster
> -d -ynm flink-1.12-test -ytm 3g -yjm 3g -yD
> yarn.provided.lib.dirs=hdfs:///flink/flink-1.12-SNAPSHOT/lib
> ./examples/streaming/StateMachineExample.jarSLF4J: Class path contains
> multiple SLF4J bindings.SLF4J: Found binding in
> [jar:file:/data1/app/flink-1.12-SNAPSHOT/lib/log4j-slf4j-impl-2.12.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J:
> Found binding in
> [jar:file:/data1/app/hadoop-2.7.3-snappy-32core12disk/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J:
> Found binding in
> [jar:file:/data1/app/hadoop-2.7.3-snappy-32core12disk/share/hadoop/tools/lib/hadoop-aliyun-2.9.2-jar-with-dependencies.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J:
> See http://www.slf4j.org/codes.html#multiple_bindings for an
> explanation.SLF4J: Actual binding is of type
> [org.apache.logging.slf4j.Log4jLoggerFactory]2020-11-13 16:14:30,347 INFO
> org.apache.flink.yarn.cli.FlinkYarnSessionCli [] - Dynamic
> Property set:
> yarn.provided.lib.dirs=hdfs:///flink/flink-1.12-SNAPSHOT/lib2020-11-13
> 16:14:30,347 INFO org.apache.flink.yarn.cli.FlinkYarnSessionCli
> [] - Dynamic Property set:
> yarn.provided.lib.dirs=hdfs:///flink/flink-1.12-SNAPSHOT/libUsage with
> built-in data generator: StateMachineExample [--error-rate
> ] [--sleep ]Usage
> with Kafka: StateMachineExample --kafka-topic [--brokers
> ]Options for both the above setups: [--backend ]
> [--checkpoint-dir ] [--async-checkpoints ]
> [--incremental-checkpoints ] [--output OR null for
> stdout]
> Using standalone source with error rate 0.00 and sleep delay 1 millis
> 2020-11-13 16:14:30,706 WARN
> org.apache.flink.yarn.configuration.YarnLogConfigUtil [] - The
> configuration directory ('/data1/app/flink-1.12-SNAPSHOT/conf') already
> contains a LOG4J config file.If you want to use logback, then please delete
> or rename the log configuration file.2020-11-13 16:14:30,947 INFO
> org.apache.hadoop.yarn.client.AHSProxy [] - Connecting
> to Application History server at
> FAT-hadoopuat-69117.vm.dc01.hellocloud.tech/10.69.1.17:102002020-11-13
> 16:14:30,958 INFO org.apache.flink.yarn.YarnClusterDescriptor
> [] - No path for the flink jar passed. Using the location of class
> org.apache.flink.yarn.YarnClusterDescriptor to locate the jar2020-11-13
> 16:14:31,065 INFO
> org.apache.hadoop.yarn.client.ConfiguredRMFailoverProxyProvider [] - Failing
> over to rm22020-11-13 16:14:31,130 INFO
> org.apache.flink.yarn.YarnClusterDescriptor [] - The
> configured JobManager memory is 3072 MB. YARN will allocate 4096 MB to make
> up an integer multiple of its minimum allocation memory (2048 MB, configured
> via 'yarn.scheduler.minimum-allocation-mb'). The extra 1024 MB may not be
> used by Flink.2020-11-13 16:14:31,130 INFO
> org.apache.flink.yarn.YarnClusterDescriptor [] - The
> configured TaskManager memory is 3072 MB. YARN will allocate 4096 MB to make
> up an integer multiple of its minimum allocation memory (2048 MB, configured
> via 'yarn.scheduler.minimum-allocation-mb'). The extra 1024 MB may not be
> used by Flink.2020-11-13 16:14:31,130 INFO
> org.apache.flink.yarn.YarnClusterDescriptor [] - Cluster
> specification: ClusterSpecification{masterMemoryMB=3072,
> taskManagerMemoryMB=3072, slotsPerTaskManager=2}2020-11-13 16:14:31,681 WARN
> org.apache.hadoop.hdfs.shortcircuit.DomainSocketFactory [] - The
> short-circuit local reads feature cannot be used because libhadoop cannot be
> loaded.2020-11-13 16:14:33,417 INFO
> org.apache.flink.yarn
[jira] [Commented] (FLINK-20143) use `yarn.provided.lib.dirs` config deploy job failed in yarn per job mode
[
https://issues.apache.org/jira/browse/FLINK-20143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17231287#comment-17231287
]
zhisheng commented on FLINK-20143:
--
!image-2020-11-13-17-21-47-751.png!
!image-2020-11-13-17-22-06-111.png!
> use `yarn.provided.lib.dirs` config deploy job failed in yarn per job mode
> --
>
> Key: FLINK-20143
> URL: https://issues.apache.org/jira/browse/FLINK-20143
> Project: Flink
> Issue Type: Bug
> Components: Client / Job Submission, Deployment / YARN
>Affects Versions: 1.12.0
>Reporter: zhisheng
>Priority: Major
> Attachments: image-2020-11-13-17-21-47-751.png,
> image-2020-11-13-17-22-06-111.png
>
>
> use follow command deploy flink job to yarn failed
> {code:java}
> ./bin/flink run -m yarn-cluster -d -ynm flink-1.12-test -ytm 3g -yjm 3g -yD
> yarn.provided.lib.dirs=hdfs:///flink/flink-1.12-SNAPSHOT/lib
> ./examples/streaming/StateMachineExample.jar
> {code}
> log:
> {code:java}
> $ ./bin/flink run -m yarn-cluster -d -ynm flink-1.12-test -ytm 3g -yjm 3g -yD
> yarn.provided.lib.dirs=hdfs:///flink/flink-1.12-SNAPSHOT/lib
> ./examples/streaming/StateMachineExample.jar$ ./bin/flink run -m yarn-cluster
> -d -ynm flink-1.12-test -ytm 3g -yjm 3g -yD
> yarn.provided.lib.dirs=hdfs:///flink/flink-1.12-SNAPSHOT/lib
> ./examples/streaming/StateMachineExample.jarSLF4J: Class path contains
> multiple SLF4J bindings.SLF4J: Found binding in
> [jar:file:/data1/app/flink-1.12-SNAPSHOT/lib/log4j-slf4j-impl-2.12.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J:
> Found binding in
> [jar:file:/data1/app/hadoop-2.7.3-snappy-32core12disk/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J:
> Found binding in
> [jar:file:/data1/app/hadoop-2.7.3-snappy-32core12disk/share/hadoop/tools/lib/hadoop-aliyun-2.9.2-jar-with-dependencies.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J:
> See http://www.slf4j.org/codes.html#multiple_bindings for an
> explanation.SLF4J: Actual binding is of type
> [org.apache.logging.slf4j.Log4jLoggerFactory]2020-11-13 16:14:30,347 INFO
> org.apache.flink.yarn.cli.FlinkYarnSessionCli [] - Dynamic
> Property set:
> yarn.provided.lib.dirs=hdfs:///flink/flink-1.12-SNAPSHOT/lib2020-11-13
> 16:14:30,347 INFO org.apache.flink.yarn.cli.FlinkYarnSessionCli
> [] - Dynamic Property set:
> yarn.provided.lib.dirs=hdfs:///flink/flink-1.12-SNAPSHOT/libUsage with
> built-in data generator: StateMachineExample [--error-rate
> ] [--sleep ]Usage
> with Kafka: StateMachineExample --kafka-topic [--brokers
> ]Options for both the above setups: [--backend ]
> [--checkpoint-dir ] [--async-checkpoints ]
> [--incremental-checkpoints ] [--output OR null for
> stdout]
> Using standalone source with error rate 0.00 and sleep delay 1 millis
> 2020-11-13 16:14:30,706 WARN
> org.apache.flink.yarn.configuration.YarnLogConfigUtil [] - The
> configuration directory ('/data1/app/flink-1.12-SNAPSHOT/conf') already
> contains a LOG4J config file.If you want to use logback, then please delete
> or rename the log configuration file.2020-11-13 16:14:30,947 INFO
> org.apache.hadoop.yarn.client.AHSProxy [] - Connecting
> to Application History server at
> FAT-hadoopuat-69117.vm.dc01.hellocloud.tech/10.69.1.17:102002020-11-13
> 16:14:30,958 INFO org.apache.flink.yarn.YarnClusterDescriptor
> [] - No path for the flink jar passed. Using the location of class
> org.apache.flink.yarn.YarnClusterDescriptor to locate the jar2020-11-13
> 16:14:31,065 INFO
> org.apache.hadoop.yarn.client.ConfiguredRMFailoverProxyProvider [] - Failing
> over to rm22020-11-13 16:14:31,130 INFO
> org.apache.flink.yarn.YarnClusterDescriptor [] - The
> configured JobManager memory is 3072 MB. YARN will allocate 4096 MB to make
> up an integer multiple of its minimum allocation memory (2048 MB, configured
> via 'yarn.scheduler.minimum-allocation-mb'). The extra 1024 MB may not be
> used by Flink.2020-11-13 16:14:31,130 INFO
> org.apache.flink.yarn.YarnClusterDescriptor [] - The
> configured TaskManager memory is 3072 MB. YARN will allocate 4096 MB to make
> up an integer multiple of its minimum allocation memory (2048 MB, configured
> via 'yarn.scheduler.minimum-allocation-mb'). The extra 1024 MB may not be
> used by Flink.2020-11-13 16:14:31,130 INFO
> org.apache.flink.yarn.YarnClusterDescriptor [] - Cluster
> specification: ClusterSpecification{masterMemoryMB=3072,
> taskManagerMemoryMB=3072, slotsPerTaskManager=2}2020-11-13 16:14:31,681 WARN
> org.apache.hadoop.hdfs.shortcircuit.DomainSocketFactory [] - The
> short-circuit local
[jira] [Updated] (FLINK-20143) use `yarn.provided.lib.dirs` config deploy job failed in yarn per job mode
[
https://issues.apache.org/jira/browse/FLINK-20143?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
zhisheng updated FLINK-20143:
-
Attachment: image-2020-11-13-17-22-06-111.png
> use `yarn.provided.lib.dirs` config deploy job failed in yarn per job mode
> --
>
> Key: FLINK-20143
> URL: https://issues.apache.org/jira/browse/FLINK-20143
> Project: Flink
> Issue Type: Bug
> Components: Client / Job Submission, Deployment / YARN
>Affects Versions: 1.12.0
>Reporter: zhisheng
>Priority: Major
> Attachments: image-2020-11-13-17-21-47-751.png,
> image-2020-11-13-17-22-06-111.png
>
>
> use follow command deploy flink job to yarn failed
> {code:java}
> ./bin/flink run -m yarn-cluster -d -ynm flink-1.12-test -ytm 3g -yjm 3g -yD
> yarn.provided.lib.dirs=hdfs:///flink/flink-1.12-SNAPSHOT/lib
> ./examples/streaming/StateMachineExample.jar
> {code}
> log:
> {code:java}
> $ ./bin/flink run -m yarn-cluster -d -ynm flink-1.12-test -ytm 3g -yjm 3g -yD
> yarn.provided.lib.dirs=hdfs:///flink/flink-1.12-SNAPSHOT/lib
> ./examples/streaming/StateMachineExample.jar$ ./bin/flink run -m yarn-cluster
> -d -ynm flink-1.12-test -ytm 3g -yjm 3g -yD
> yarn.provided.lib.dirs=hdfs:///flink/flink-1.12-SNAPSHOT/lib
> ./examples/streaming/StateMachineExample.jarSLF4J: Class path contains
> multiple SLF4J bindings.SLF4J: Found binding in
> [jar:file:/data1/app/flink-1.12-SNAPSHOT/lib/log4j-slf4j-impl-2.12.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J:
> Found binding in
> [jar:file:/data1/app/hadoop-2.7.3-snappy-32core12disk/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J:
> Found binding in
> [jar:file:/data1/app/hadoop-2.7.3-snappy-32core12disk/share/hadoop/tools/lib/hadoop-aliyun-2.9.2-jar-with-dependencies.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J:
> See http://www.slf4j.org/codes.html#multiple_bindings for an
> explanation.SLF4J: Actual binding is of type
> [org.apache.logging.slf4j.Log4jLoggerFactory]2020-11-13 16:14:30,347 INFO
> org.apache.flink.yarn.cli.FlinkYarnSessionCli [] - Dynamic
> Property set:
> yarn.provided.lib.dirs=hdfs:///flink/flink-1.12-SNAPSHOT/lib2020-11-13
> 16:14:30,347 INFO org.apache.flink.yarn.cli.FlinkYarnSessionCli
> [] - Dynamic Property set:
> yarn.provided.lib.dirs=hdfs:///flink/flink-1.12-SNAPSHOT/libUsage with
> built-in data generator: StateMachineExample [--error-rate
> ] [--sleep ]Usage
> with Kafka: StateMachineExample --kafka-topic [--brokers
> ]Options for both the above setups: [--backend ]
> [--checkpoint-dir ] [--async-checkpoints ]
> [--incremental-checkpoints ] [--output OR null for
> stdout]
> Using standalone source with error rate 0.00 and sleep delay 1 millis
> 2020-11-13 16:14:30,706 WARN
> org.apache.flink.yarn.configuration.YarnLogConfigUtil [] - The
> configuration directory ('/data1/app/flink-1.12-SNAPSHOT/conf') already
> contains a LOG4J config file.If you want to use logback, then please delete
> or rename the log configuration file.2020-11-13 16:14:30,947 INFO
> org.apache.hadoop.yarn.client.AHSProxy [] - Connecting
> to Application History server at
> FAT-hadoopuat-69117.vm.dc01.hellocloud.tech/10.69.1.17:102002020-11-13
> 16:14:30,958 INFO org.apache.flink.yarn.YarnClusterDescriptor
> [] - No path for the flink jar passed. Using the location of class
> org.apache.flink.yarn.YarnClusterDescriptor to locate the jar2020-11-13
> 16:14:31,065 INFO
> org.apache.hadoop.yarn.client.ConfiguredRMFailoverProxyProvider [] - Failing
> over to rm22020-11-13 16:14:31,130 INFO
> org.apache.flink.yarn.YarnClusterDescriptor [] - The
> configured JobManager memory is 3072 MB. YARN will allocate 4096 MB to make
> up an integer multiple of its minimum allocation memory (2048 MB, configured
> via 'yarn.scheduler.minimum-allocation-mb'). The extra 1024 MB may not be
> used by Flink.2020-11-13 16:14:31,130 INFO
> org.apache.flink.yarn.YarnClusterDescriptor [] - The
> configured TaskManager memory is 3072 MB. YARN will allocate 4096 MB to make
> up an integer multiple of its minimum allocation memory (2048 MB, configured
> via 'yarn.scheduler.minimum-allocation-mb'). The extra 1024 MB may not be
> used by Flink.2020-11-13 16:14:31,130 INFO
> org.apache.flink.yarn.YarnClusterDescriptor [] - Cluster
> specification: ClusterSpecification{masterMemoryMB=3072,
> taskManagerMemoryMB=3072, slotsPerTaskManager=2}2020-11-13 16:14:31,681 WARN
> org.apache.hadoop.hdfs.shortcircuit.DomainSocketFactory [] - The
> short-circuit local reads feature cannot be used because libhadoop cannot be
> loaded.2020-11-13 16:14
[GitHub] [flink] flinkbot edited a comment on pull request #14034: [FLINK-20088][kinesis] Fix issue encountered when Polling consumer using timestamp with empty shard
flinkbot edited a comment on pull request #14034: URL: https://github.com/apache/flink/pull/14034#issuecomment-725425798 ## CI report: * dc4f57931b735d886d06ae4e302235353367c34a Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9469) * a18a4e6d3f099b269fe0fb5618faf2a2cd53121a Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9555) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-20143) use `yarn.provided.lib.dirs` config deploy job failed in yarn per job mode
[
https://issues.apache.org/jira/browse/FLINK-20143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17231300#comment-17231300
]
zhisheng commented on FLINK-20143:
--
[~kkl0u] yes, -ytm and -yjm does not take effect，i create a issue some days ago
https://issues.apache.org/jira/browse/FLINK-19973
> use `yarn.provided.lib.dirs` config deploy job failed in yarn per job mode
> --
>
> Key: FLINK-20143
> URL: https://issues.apache.org/jira/browse/FLINK-20143
> Project: Flink
> Issue Type: Bug
> Components: Client / Job Submission, Deployment / YARN
>Affects Versions: 1.12.0
>Reporter: zhisheng
>Priority: Major
> Attachments: image-2020-11-13-17-21-47-751.png,
> image-2020-11-13-17-22-06-111.png
>
>
> use follow command deploy flink job to yarn failed
> {code:java}
> ./bin/flink run -m yarn-cluster -d -ynm flink-1.12-test -ytm 3g -yjm 3g -yD
> yarn.provided.lib.dirs=hdfs:///flink/flink-1.12-SNAPSHOT/lib
> ./examples/streaming/StateMachineExample.jar
> {code}
> log:
> {code:java}
> $ ./bin/flink run -m yarn-cluster -d -ynm flink-1.12-test -ytm 3g -yjm 3g -yD
> yarn.provided.lib.dirs=hdfs:///flink/flink-1.12-SNAPSHOT/lib
> ./examples/streaming/StateMachineExample.jar$ ./bin/flink run -m yarn-cluster
> -d -ynm flink-1.12-test -ytm 3g -yjm 3g -yD
> yarn.provided.lib.dirs=hdfs:///flink/flink-1.12-SNAPSHOT/lib
> ./examples/streaming/StateMachineExample.jarSLF4J: Class path contains
> multiple SLF4J bindings.SLF4J: Found binding in
> [jar:file:/data1/app/flink-1.12-SNAPSHOT/lib/log4j-slf4j-impl-2.12.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J:
> Found binding in
> [jar:file:/data1/app/hadoop-2.7.3-snappy-32core12disk/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J:
> Found binding in
> [jar:file:/data1/app/hadoop-2.7.3-snappy-32core12disk/share/hadoop/tools/lib/hadoop-aliyun-2.9.2-jar-with-dependencies.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J:
> See http://www.slf4j.org/codes.html#multiple_bindings for an
> explanation.SLF4J: Actual binding is of type
> [org.apache.logging.slf4j.Log4jLoggerFactory]2020-11-13 16:14:30,347 INFO
> org.apache.flink.yarn.cli.FlinkYarnSessionCli [] - Dynamic
> Property set:
> yarn.provided.lib.dirs=hdfs:///flink/flink-1.12-SNAPSHOT/lib2020-11-13
> 16:14:30,347 INFO org.apache.flink.yarn.cli.FlinkYarnSessionCli
> [] - Dynamic Property set:
> yarn.provided.lib.dirs=hdfs:///flink/flink-1.12-SNAPSHOT/libUsage with
> built-in data generator: StateMachineExample [--error-rate
> ] [--sleep ]Usage
> with Kafka: StateMachineExample --kafka-topic [--brokers
> ]Options for both the above setups: [--backend ]
> [--checkpoint-dir ] [--async-checkpoints ]
> [--incremental-checkpoints ] [--output OR null for
> stdout]
> Using standalone source with error rate 0.00 and sleep delay 1 millis
> 2020-11-13 16:14:30,706 WARN
> org.apache.flink.yarn.configuration.YarnLogConfigUtil [] - The
> configuration directory ('/data1/app/flink-1.12-SNAPSHOT/conf') already
> contains a LOG4J config file.If you want to use logback, then please delete
> or rename the log configuration file.2020-11-13 16:14:30,947 INFO
> org.apache.hadoop.yarn.client.AHSProxy [] - Connecting
> to Application History server at
> FAT-hadoopuat-69117.vm.dc01.hellocloud.tech/10.69.1.17:102002020-11-13
> 16:14:30,958 INFO org.apache.flink.yarn.YarnClusterDescriptor
> [] - No path for the flink jar passed. Using the location of class
> org.apache.flink.yarn.YarnClusterDescriptor to locate the jar2020-11-13
> 16:14:31,065 INFO
> org.apache.hadoop.yarn.client.ConfiguredRMFailoverProxyProvider [] - Failing
> over to rm22020-11-13 16:14:31,130 INFO
> org.apache.flink.yarn.YarnClusterDescriptor [] - The
> configured JobManager memory is 3072 MB. YARN will allocate 4096 MB to make
> up an integer multiple of its minimum allocation memory (2048 MB, configured
> via 'yarn.scheduler.minimum-allocation-mb'). The extra 1024 MB may not be
> used by Flink.2020-11-13 16:14:31,130 INFO
> org.apache.flink.yarn.YarnClusterDescriptor [] - The
> configured TaskManager memory is 3072 MB. YARN will allocate 4096 MB to make
> up an integer multiple of its minimum allocation memory (2048 MB, configured
> via 'yarn.scheduler.minimum-allocation-mb'). The extra 1024 MB may not be
> used by Flink.2020-11-13 16:14:31,130 INFO
> org.apache.flink.yarn.YarnClusterDescriptor [] - Cluster
> specification: ClusterSpecification{masterMemoryMB=3072,
> taskManagerMemoryMB=3072, slotsPerTaskManager=2}2020-11-13 16:14:31,681 WARN
> org.apache.hadoop.hdfs.shortcircuit.D
[GitHub] [flink] flinkbot edited a comment on pull request #14049: [FLINK-20043][kinesis] Add flink-sql-connector-kinesis package
flinkbot edited a comment on pull request #14049: URL: https://github.com/apache/flink/pull/14049#issuecomment-726094991 ## CI report: * 0a068c0f16865a2c91819d21f52d759151acb97e UNKNOWN * d15f6e11ca8d46da440ed6c2235f73937d70ab3a UNKNOWN * 1a9e4dcb885484aa800a040f1a4674cd5509f7b4 Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9523) Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9525) * 70bd4cfcc42ad54183c03aec2d763a2059fc4b1b UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-19852) Managed memory released check can block IterativeTask
[ https://issues.apache.org/jira/browse/FLINK-19852?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arvid Heise updated FLINK-19852: Affects Version/s: 1.12.0 > Managed memory released check can block IterativeTask > - > > Key: FLINK-19852 > URL: https://issues.apache.org/jira/browse/FLINK-19852 > Project: Flink > Issue Type: Bug > Components: Runtime / Task >Affects Versions: 1.11.0, 1.10.2, 1.12.0, 1.11.1, 1.11.2 >Reporter: shaomeng.wang >Priority: Critical > Attachments: image-2020-10-28-17-48-28-395.png, > image-2020-10-28-17-48-48-583.png > > > UnsafeMemoryBudget#reserveMemory, called on TempBarrier, needs time to wait > on GC of all allocated/released managed memory at every iteration. > > stack: > !image-2020-10-28-17-48-48-583.png! > new TempBarrier in BatchTask > !image-2020-10-28-17-48-28-395.png! > > These will be very slow than before. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on pull request #14064: [FLINK-19454][docs-zh] Translate page 'Importing Flink into an IDE' into Chinese
flinkbot edited a comment on pull request #14064: URL: https://github.com/apache/flink/pull/14064#issuecomment-726642209 ## CI report: * 41bcdb30f6837f2f22dda55edafe8ba9ef5e98be Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9556) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-20143) use `yarn.provided.lib.dirs` config deploy job failed in yarn per job mode
[
https://issues.apache.org/jira/browse/FLINK-20143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17231301#comment-17231301
]
Kostas Kloudas commented on FLINK-20143:
I am not sure if I can figure out what is happening from what is here.
> use `yarn.provided.lib.dirs` config deploy job failed in yarn per job mode
> --
>
> Key: FLINK-20143
> URL: https://issues.apache.org/jira/browse/FLINK-20143
> Project: Flink
> Issue Type: Bug
> Components: Client / Job Submission, Deployment / YARN
>Affects Versions: 1.12.0
>Reporter: zhisheng
>Priority: Major
> Attachments: image-2020-11-13-17-21-47-751.png,
> image-2020-11-13-17-22-06-111.png
>
>
> use follow command deploy flink job to yarn failed
> {code:java}
> ./bin/flink run -m yarn-cluster -d -ynm flink-1.12-test -ytm 3g -yjm 3g -yD
> yarn.provided.lib.dirs=hdfs:///flink/flink-1.12-SNAPSHOT/lib
> ./examples/streaming/StateMachineExample.jar
> {code}
> log:
> {code:java}
> $ ./bin/flink run -m yarn-cluster -d -ynm flink-1.12-test -ytm 3g -yjm 3g -yD
> yarn.provided.lib.dirs=hdfs:///flink/flink-1.12-SNAPSHOT/lib
> ./examples/streaming/StateMachineExample.jar$ ./bin/flink run -m yarn-cluster
> -d -ynm flink-1.12-test -ytm 3g -yjm 3g -yD
> yarn.provided.lib.dirs=hdfs:///flink/flink-1.12-SNAPSHOT/lib
> ./examples/streaming/StateMachineExample.jarSLF4J: Class path contains
> multiple SLF4J bindings.SLF4J: Found binding in
> [jar:file:/data1/app/flink-1.12-SNAPSHOT/lib/log4j-slf4j-impl-2.12.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J:
> Found binding in
> [jar:file:/data1/app/hadoop-2.7.3-snappy-32core12disk/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J:
> Found binding in
> [jar:file:/data1/app/hadoop-2.7.3-snappy-32core12disk/share/hadoop/tools/lib/hadoop-aliyun-2.9.2-jar-with-dependencies.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J:
> See http://www.slf4j.org/codes.html#multiple_bindings for an
> explanation.SLF4J: Actual binding is of type
> [org.apache.logging.slf4j.Log4jLoggerFactory]2020-11-13 16:14:30,347 INFO
> org.apache.flink.yarn.cli.FlinkYarnSessionCli [] - Dynamic
> Property set:
> yarn.provided.lib.dirs=hdfs:///flink/flink-1.12-SNAPSHOT/lib2020-11-13
> 16:14:30,347 INFO org.apache.flink.yarn.cli.FlinkYarnSessionCli
> [] - Dynamic Property set:
> yarn.provided.lib.dirs=hdfs:///flink/flink-1.12-SNAPSHOT/libUsage with
> built-in data generator: StateMachineExample [--error-rate
> ] [--sleep ]Usage
> with Kafka: StateMachineExample --kafka-topic [--brokers
> ]Options for both the above setups: [--backend ]
> [--checkpoint-dir ] [--async-checkpoints ]
> [--incremental-checkpoints ] [--output OR null for
> stdout]
> Using standalone source with error rate 0.00 and sleep delay 1 millis
> 2020-11-13 16:14:30,706 WARN
> org.apache.flink.yarn.configuration.YarnLogConfigUtil [] - The
> configuration directory ('/data1/app/flink-1.12-SNAPSHOT/conf') already
> contains a LOG4J config file.If you want to use logback, then please delete
> or rename the log configuration file.2020-11-13 16:14:30,947 INFO
> org.apache.hadoop.yarn.client.AHSProxy [] - Connecting
> to Application History server at
> FAT-hadoopuat-69117.vm.dc01.hellocloud.tech/10.69.1.17:102002020-11-13
> 16:14:30,958 INFO org.apache.flink.yarn.YarnClusterDescriptor
> [] - No path for the flink jar passed. Using the location of class
> org.apache.flink.yarn.YarnClusterDescriptor to locate the jar2020-11-13
> 16:14:31,065 INFO
> org.apache.hadoop.yarn.client.ConfiguredRMFailoverProxyProvider [] - Failing
> over to rm22020-11-13 16:14:31,130 INFO
> org.apache.flink.yarn.YarnClusterDescriptor [] - The
> configured JobManager memory is 3072 MB. YARN will allocate 4096 MB to make
> up an integer multiple of its minimum allocation memory (2048 MB, configured
> via 'yarn.scheduler.minimum-allocation-mb'). The extra 1024 MB may not be
> used by Flink.2020-11-13 16:14:31,130 INFO
> org.apache.flink.yarn.YarnClusterDescriptor [] - The
> configured TaskManager memory is 3072 MB. YARN will allocate 4096 MB to make
> up an integer multiple of its minimum allocation memory (2048 MB, configured
> via 'yarn.scheduler.minimum-allocation-mb'). The extra 1024 MB may not be
> used by Flink.2020-11-13 16:14:31,130 INFO
> org.apache.flink.yarn.YarnClusterDescriptor [] - Cluster
> specification: ClusterSpecification{masterMemoryMB=3072,
> taskManagerMemoryMB=3072, slotsPerTaskManager=2}2020-11-13 16:14:31,681 WARN
> org.apache.hadoop.hdfs.shortcircuit.DomainSocketFactory [] - The
> short-circu
[jira] [Commented] (FLINK-20143) use `yarn.provided.lib.dirs` config deploy job failed in yarn per job mode
[
https://issues.apache.org/jira/browse/FLINK-20143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17231306#comment-17231306
]
Kostas Kloudas commented on FLINK-20143:
As discussed in the issue, you have to specify the full config option name
prefixed by {{-D}} when using the {{GenericCLI}}. This means for example
{{-Dtaskmanager.memory.process.size=...}}.
> use `yarn.provided.lib.dirs` config deploy job failed in yarn per job mode
> --
>
> Key: FLINK-20143
> URL: https://issues.apache.org/jira/browse/FLINK-20143
> Project: Flink
> Issue Type: Bug
> Components: Client / Job Submission, Deployment / YARN
>Affects Versions: 1.12.0
>Reporter: zhisheng
>Priority: Major
> Attachments: image-2020-11-13-17-21-47-751.png,
> image-2020-11-13-17-22-06-111.png
>
>
> use follow command deploy flink job to yarn failed
> {code:java}
> ./bin/flink run -m yarn-cluster -d -ynm flink-1.12-test -ytm 3g -yjm 3g -yD
> yarn.provided.lib.dirs=hdfs:///flink/flink-1.12-SNAPSHOT/lib
> ./examples/streaming/StateMachineExample.jar
> {code}
> log:
> {code:java}
> $ ./bin/flink run -m yarn-cluster -d -ynm flink-1.12-test -ytm 3g -yjm 3g -yD
> yarn.provided.lib.dirs=hdfs:///flink/flink-1.12-SNAPSHOT/lib
> ./examples/streaming/StateMachineExample.jar$ ./bin/flink run -m yarn-cluster
> -d -ynm flink-1.12-test -ytm 3g -yjm 3g -yD
> yarn.provided.lib.dirs=hdfs:///flink/flink-1.12-SNAPSHOT/lib
> ./examples/streaming/StateMachineExample.jarSLF4J: Class path contains
> multiple SLF4J bindings.SLF4J: Found binding in
> [jar:file:/data1/app/flink-1.12-SNAPSHOT/lib/log4j-slf4j-impl-2.12.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J:
> Found binding in
> [jar:file:/data1/app/hadoop-2.7.3-snappy-32core12disk/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J:
> Found binding in
> [jar:file:/data1/app/hadoop-2.7.3-snappy-32core12disk/share/hadoop/tools/lib/hadoop-aliyun-2.9.2-jar-with-dependencies.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J:
> See http://www.slf4j.org/codes.html#multiple_bindings for an
> explanation.SLF4J: Actual binding is of type
> [org.apache.logging.slf4j.Log4jLoggerFactory]2020-11-13 16:14:30,347 INFO
> org.apache.flink.yarn.cli.FlinkYarnSessionCli [] - Dynamic
> Property set:
> yarn.provided.lib.dirs=hdfs:///flink/flink-1.12-SNAPSHOT/lib2020-11-13
> 16:14:30,347 INFO org.apache.flink.yarn.cli.FlinkYarnSessionCli
> [] - Dynamic Property set:
> yarn.provided.lib.dirs=hdfs:///flink/flink-1.12-SNAPSHOT/libUsage with
> built-in data generator: StateMachineExample [--error-rate
> ] [--sleep ]Usage
> with Kafka: StateMachineExample --kafka-topic [--brokers
> ]Options for both the above setups: [--backend ]
> [--checkpoint-dir ] [--async-checkpoints ]
> [--incremental-checkpoints ] [--output OR null for
> stdout]
> Using standalone source with error rate 0.00 and sleep delay 1 millis
> 2020-11-13 16:14:30,706 WARN
> org.apache.flink.yarn.configuration.YarnLogConfigUtil [] - The
> configuration directory ('/data1/app/flink-1.12-SNAPSHOT/conf') already
> contains a LOG4J config file.If you want to use logback, then please delete
> or rename the log configuration file.2020-11-13 16:14:30,947 INFO
> org.apache.hadoop.yarn.client.AHSProxy [] - Connecting
> to Application History server at
> FAT-hadoopuat-69117.vm.dc01.hellocloud.tech/10.69.1.17:102002020-11-13
> 16:14:30,958 INFO org.apache.flink.yarn.YarnClusterDescriptor
> [] - No path for the flink jar passed. Using the location of class
> org.apache.flink.yarn.YarnClusterDescriptor to locate the jar2020-11-13
> 16:14:31,065 INFO
> org.apache.hadoop.yarn.client.ConfiguredRMFailoverProxyProvider [] - Failing
> over to rm22020-11-13 16:14:31,130 INFO
> org.apache.flink.yarn.YarnClusterDescriptor [] - The
> configured JobManager memory is 3072 MB. YARN will allocate 4096 MB to make
> up an integer multiple of its minimum allocation memory (2048 MB, configured
> via 'yarn.scheduler.minimum-allocation-mb'). The extra 1024 MB may not be
> used by Flink.2020-11-13 16:14:31,130 INFO
> org.apache.flink.yarn.YarnClusterDescriptor [] - The
> configured TaskManager memory is 3072 MB. YARN will allocate 4096 MB to make
> up an integer multiple of its minimum allocation memory (2048 MB, configured
> via 'yarn.scheduler.minimum-allocation-mb'). The extra 1024 MB may not be
> used by Flink.2020-11-13 16:14:31,130 INFO
> org.apache.flink.yarn.YarnClusterDescriptor [] - Cluster
> specification: ClusterSpecification{masterMemoryMB=3072,
> taskManagerMemoryMB=3072, slotsPerTaskManage
[jira] [Comment Edited] (FLINK-20143) use `yarn.provided.lib.dirs` config deploy job failed in yarn per job mode
[
https://issues.apache.org/jira/browse/FLINK-20143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17231306#comment-17231306
]
Kostas Kloudas edited comment on FLINK-20143 at 11/13/20, 9:28 AM:
---
As discussed in the issue, you have to specify the full config option name
prefixed by {{-D}} when using the {{GenericCLI}}. This means for example that
{{-ytm}} should become {{-Dtaskmanager.memory.process.size=...}}.
was (Author: kkl0u):
As discussed in the issue, you have to specify the full config option name
prefixed by {{-D}} when using the {{GenericCLI}}. This means for example
{{-Dtaskmanager.memory.process.size=...}}.
> use `yarn.provided.lib.dirs` config deploy job failed in yarn per job mode
> --
>
> Key: FLINK-20143
> URL: https://issues.apache.org/jira/browse/FLINK-20143
> Project: Flink
> Issue Type: Bug
> Components: Client / Job Submission, Deployment / YARN
>Affects Versions: 1.12.0
>Reporter: zhisheng
>Priority: Major
> Attachments: image-2020-11-13-17-21-47-751.png,
> image-2020-11-13-17-22-06-111.png
>
>
> use follow command deploy flink job to yarn failed
> {code:java}
> ./bin/flink run -m yarn-cluster -d -ynm flink-1.12-test -ytm 3g -yjm 3g -yD
> yarn.provided.lib.dirs=hdfs:///flink/flink-1.12-SNAPSHOT/lib
> ./examples/streaming/StateMachineExample.jar
> {code}
> log:
> {code:java}
> $ ./bin/flink run -m yarn-cluster -d -ynm flink-1.12-test -ytm 3g -yjm 3g -yD
> yarn.provided.lib.dirs=hdfs:///flink/flink-1.12-SNAPSHOT/lib
> ./examples/streaming/StateMachineExample.jar$ ./bin/flink run -m yarn-cluster
> -d -ynm flink-1.12-test -ytm 3g -yjm 3g -yD
> yarn.provided.lib.dirs=hdfs:///flink/flink-1.12-SNAPSHOT/lib
> ./examples/streaming/StateMachineExample.jarSLF4J: Class path contains
> multiple SLF4J bindings.SLF4J: Found binding in
> [jar:file:/data1/app/flink-1.12-SNAPSHOT/lib/log4j-slf4j-impl-2.12.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J:
> Found binding in
> [jar:file:/data1/app/hadoop-2.7.3-snappy-32core12disk/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J:
> Found binding in
> [jar:file:/data1/app/hadoop-2.7.3-snappy-32core12disk/share/hadoop/tools/lib/hadoop-aliyun-2.9.2-jar-with-dependencies.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J:
> See http://www.slf4j.org/codes.html#multiple_bindings for an
> explanation.SLF4J: Actual binding is of type
> [org.apache.logging.slf4j.Log4jLoggerFactory]2020-11-13 16:14:30,347 INFO
> org.apache.flink.yarn.cli.FlinkYarnSessionCli [] - Dynamic
> Property set:
> yarn.provided.lib.dirs=hdfs:///flink/flink-1.12-SNAPSHOT/lib2020-11-13
> 16:14:30,347 INFO org.apache.flink.yarn.cli.FlinkYarnSessionCli
> [] - Dynamic Property set:
> yarn.provided.lib.dirs=hdfs:///flink/flink-1.12-SNAPSHOT/libUsage with
> built-in data generator: StateMachineExample [--error-rate
> ] [--sleep ]Usage
> with Kafka: StateMachineExample --kafka-topic [--brokers
> ]Options for both the above setups: [--backend ]
> [--checkpoint-dir ] [--async-checkpoints ]
> [--incremental-checkpoints ] [--output OR null for
> stdout]
> Using standalone source with error rate 0.00 and sleep delay 1 millis
> 2020-11-13 16:14:30,706 WARN
> org.apache.flink.yarn.configuration.YarnLogConfigUtil [] - The
> configuration directory ('/data1/app/flink-1.12-SNAPSHOT/conf') already
> contains a LOG4J config file.If you want to use logback, then please delete
> or rename the log configuration file.2020-11-13 16:14:30,947 INFO
> org.apache.hadoop.yarn.client.AHSProxy [] - Connecting
> to Application History server at
> FAT-hadoopuat-69117.vm.dc01.hellocloud.tech/10.69.1.17:102002020-11-13
> 16:14:30,958 INFO org.apache.flink.yarn.YarnClusterDescriptor
> [] - No path for the flink jar passed. Using the location of class
> org.apache.flink.yarn.YarnClusterDescriptor to locate the jar2020-11-13
> 16:14:31,065 INFO
> org.apache.hadoop.yarn.client.ConfiguredRMFailoverProxyProvider [] - Failing
> over to rm22020-11-13 16:14:31,130 INFO
> org.apache.flink.yarn.YarnClusterDescriptor [] - The
> configured JobManager memory is 3072 MB. YARN will allocate 4096 MB to make
> up an integer multiple of its minimum allocation memory (2048 MB, configured
> via 'yarn.scheduler.minimum-allocation-mb'). The extra 1024 MB may not be
> used by Flink.2020-11-13 16:14:31,130 INFO
> org.apache.flink.yarn.YarnClusterDescriptor [] - The
> configured TaskManager memory is 3072 MB. YARN will allocate 4096 MB to make
> up an integer multiple of its minimum allocation memory (2048 MB, configured
> via 'yar
[GitHub] [flink] aljoscha commented on pull request #14044: [FLINK-20098] Don't add flink-connector-files to flink-dist
aljoscha commented on pull request #14044: URL: https://github.com/apache/flink/pull/14044#issuecomment-726655329 Thanks for the review! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] aljoscha closed pull request #14044: [FLINK-20098] Don't add flink-connector-files to flink-dist
aljoscha closed pull request #14044: URL: https://github.com/apache/flink/pull/14044 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Comment Edited] (FLINK-15867) LAST_VALUE aggregate function does not support time-related types
[
https://issues.apache.org/jira/browse/FLINK-15867?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17231314#comment-17231314
]
Jannik Schmeier edited comment on FLINK-15867 at 11/13/20, 9:30 AM:
Since we need this functionality for our use cases, I would like to provide a
pull request for this it. Can you please assign the issue to me?
was (Author: shmyer):
Since we need this functionality for our use cases, I would like to provide a
pull request for this it. Can you assign the issue to me?
> LAST_VALUE aggregate function does not support time-related types
> -
>
> Key: FLINK-15867
> URL: https://issues.apache.org/jira/browse/FLINK-15867
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Planner
>Affects Versions: 1.9.2, 1.10.0
>Reporter: Benoît Paris
>Priority: Major
> Attachments: flink-test-lastvalue-timestamp.zip
>
>
> The following fails:
> {code:java}
> LAST_VALUE(TIMESTAMP '2020-02-03 16:17:20')
> LAST_VALUE(DATE '2020-02-03')
> LAST_VALUE(TIME '16:17:20')
> LAST_VALUE(NOW()){code}
> But this works:
>
> {code:java}
> LAST_VALUE(UNIX_TIMESTAMP())
> {code}
> Leading me to say it might be more a type/format issue, rather than an actual
> time processing issue.
> Attached is java + pom + full stacktrace, for reproduction. Stacktrace part
> is below.
>
> The ByteLastValueAggFunction, etc types seem trivial to implement, but the in
> the createLastValueAggFunction only basic types seem to be dealt with. Is
> there a reason more complicated LogicalTypeRoots might not be implemented ?
> (old vs new types?)
>
>
> Caused by: org.apache.flink.table.api.TableException: LAST_VALUE aggregate
> function does not support type: ''TIMESTAMP_WITHOUT_TIME_ZONE''.Caused by:
> org.apache.flink.table.api.TableException: LAST_VALUE aggregate function does
> not support type: ''TIMESTAMP_WITHOUT_TIME_ZONE''.Please re-check the data
> type. at
> org.apache.flink.table.planner.plan.utils.AggFunctionFactory.createLastValueAggFunction(AggFunctionFactory.scala:617)
> at
> org.apache.flink.table.planner.plan.utils.AggFunctionFactory.createAggFunction(AggFunctionFactory.scala:113)
> at
> org.apache.flink.table.planner.plan.utils.AggregateUtil$$anonfun$9.apply(AggregateUtil.scala:285)
> at
> org.apache.flink.table.planner.plan.utils.AggregateUtil$$anonfun$9.apply(AggregateUtil.scala:279)
> at
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
> at
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
> at
> scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
> at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48) at
> scala.collection.TraversableLike$class.map(TraversableLike.scala:234) at
> scala.collection.AbstractTraversable.map(Traversable.scala:104) at
> org.apache.flink.table.planner.plan.utils.AggregateUtil$.transformToAggregateInfoList(AggregateUtil.scala:279)
> at
> org.apache.flink.table.planner.plan.utils.AggregateUtil$.transformToStreamAggregateInfoList(AggregateUtil.scala:228)
> at
> org.apache.flink.table.planner.plan.nodes.physical.stream.StreamExecGroupAggregate.(StreamExecGroupAggregate.scala:72)
> at
> org.apache.flink.table.planner.plan.rules.physical.stream.StreamExecGroupAggregateRule.convert(StreamExecGroupAggregateRule.scala:68)
> at
> org.apache.calcite.rel.convert.ConverterRule.onMatch(ConverterRule.java:139)
> at
> org.apache.calcite.plan.volcano.VolcanoRuleCall.onMatch(VolcanoRuleCall.java:208)
> at
> org.apache.calcite.plan.volcano.VolcanoPlanner.findBestExp(VolcanoPlanner.java:631)
> at org.apache.calcite.tools.Programs$RuleSetProgram.run(Programs.java:328)
> at
> org.apache.flink.table.planner.plan.optimize.program.FlinkVolcanoProgram.optimize(FlinkVolcanoProgram.scala:64)
>
>
>
>
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-15867) LAST_VALUE aggregate function does not support time-related types
[
https://issues.apache.org/jira/browse/FLINK-15867?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17231314#comment-17231314
]
Jannik Schmeier commented on FLINK-15867:
-
Since we need this functionality for our use cases, I would like to provide a
pull request for this it. Can you assign the issue to me?
> LAST_VALUE aggregate function does not support time-related types
> -
>
> Key: FLINK-15867
> URL: https://issues.apache.org/jira/browse/FLINK-15867
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Planner
>Affects Versions: 1.9.2, 1.10.0
>Reporter: Benoît Paris
>Priority: Major
> Attachments: flink-test-lastvalue-timestamp.zip
>
>
> The following fails:
> {code:java}
> LAST_VALUE(TIMESTAMP '2020-02-03 16:17:20')
> LAST_VALUE(DATE '2020-02-03')
> LAST_VALUE(TIME '16:17:20')
> LAST_VALUE(NOW()){code}
> But this works:
>
> {code:java}
> LAST_VALUE(UNIX_TIMESTAMP())
> {code}
> Leading me to say it might be more a type/format issue, rather than an actual
> time processing issue.
> Attached is java + pom + full stacktrace, for reproduction. Stacktrace part
> is below.
>
> The ByteLastValueAggFunction, etc types seem trivial to implement, but the in
> the createLastValueAggFunction only basic types seem to be dealt with. Is
> there a reason more complicated LogicalTypeRoots might not be implemented ?
> (old vs new types?)
>
>
> Caused by: org.apache.flink.table.api.TableException: LAST_VALUE aggregate
> function does not support type: ''TIMESTAMP_WITHOUT_TIME_ZONE''.Caused by:
> org.apache.flink.table.api.TableException: LAST_VALUE aggregate function does
> not support type: ''TIMESTAMP_WITHOUT_TIME_ZONE''.Please re-check the data
> type. at
> org.apache.flink.table.planner.plan.utils.AggFunctionFactory.createLastValueAggFunction(AggFunctionFactory.scala:617)
> at
> org.apache.flink.table.planner.plan.utils.AggFunctionFactory.createAggFunction(AggFunctionFactory.scala:113)
> at
> org.apache.flink.table.planner.plan.utils.AggregateUtil$$anonfun$9.apply(AggregateUtil.scala:285)
> at
> org.apache.flink.table.planner.plan.utils.AggregateUtil$$anonfun$9.apply(AggregateUtil.scala:279)
> at
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
> at
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
> at
> scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
> at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48) at
> scala.collection.TraversableLike$class.map(TraversableLike.scala:234) at
> scala.collection.AbstractTraversable.map(Traversable.scala:104) at
> org.apache.flink.table.planner.plan.utils.AggregateUtil$.transformToAggregateInfoList(AggregateUtil.scala:279)
> at
> org.apache.flink.table.planner.plan.utils.AggregateUtil$.transformToStreamAggregateInfoList(AggregateUtil.scala:228)
> at
> org.apache.flink.table.planner.plan.nodes.physical.stream.StreamExecGroupAggregate.(StreamExecGroupAggregate.scala:72)
> at
> org.apache.flink.table.planner.plan.rules.physical.stream.StreamExecGroupAggregateRule.convert(StreamExecGroupAggregateRule.scala:68)
> at
> org.apache.calcite.rel.convert.ConverterRule.onMatch(ConverterRule.java:139)
> at
> org.apache.calcite.plan.volcano.VolcanoRuleCall.onMatch(VolcanoRuleCall.java:208)
> at
> org.apache.calcite.plan.volcano.VolcanoPlanner.findBestExp(VolcanoPlanner.java:631)
> at org.apache.calcite.tools.Programs$RuleSetProgram.run(Programs.java:328)
> at
> org.apache.flink.table.planner.plan.optimize.program.FlinkVolcanoProgram.optimize(FlinkVolcanoProgram.scala:64)
>
>
>
>
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Closed] (FLINK-20098) Don't add flink-connector-files to flink-dist
[
https://issues.apache.org/jira/browse/FLINK-20098?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aljoscha Krettek closed FLINK-20098.
Resolution: Done
master: 3bd1a8163b6b9d1a1b7f3a293b919ea0af551c54
> Don't add flink-connector-files to flink-dist
> -
>
> Key: FLINK-20098
> URL: https://issues.apache.org/jira/browse/FLINK-20098
> Project: Flink
> Issue Type: Improvement
> Components: API / DataStream
>Reporter: Aljoscha Krettek
>Assignee: Aljoscha Krettek
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.12.0
>
>
> We currently add both {{flink-connector-files}} and {{flink-connector-base}}
> to {{flink-dist}}.
> This implies, that users should use the dependency like this:
> {code}
>
> org.apache.flink
> flink-connector-files
> ${project.version}
> provided
>
> {code}
> which differs from other connectors where users don't need to specify
> {{provided}}.
> Also, {{flink-connector-files}} has {{flink-connector-base}} as a provided
> dependency, which means that examples that use this dependency will not run
> out-of-box in IntelliJ because transitive provided dependencies will not be
> considered.
> I propose to just remove the dependencies from {{flink-dist}} and let users
> use the File Connector like any other connector.
> I believe the initial motivation for "providing" the File Connector in
> {{flink-dist}} was to allow us to use the File Connector under the hood in
> methods such as {{StreamExecutionEnvironment.readFile(...)}}. We could decide
> to deprecate and remove those methods or re-add the File Connector as an
> explicit (non-provided) dependency again in the future.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-16945) Execute CheckpointFailureManager.FailJobCallback directly in main thread executor
[
https://issues.apache.org/jira/browse/FLINK-16945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17231318#comment-17231318
]
Arvid Heise commented on FLINK-16945:
-
I don't expect it to be part of 1.12.0.
> Execute CheckpointFailureManager.FailJobCallback directly in main thread
> executor
> -
>
> Key: FLINK-16945
> URL: https://issues.apache.org/jira/browse/FLINK-16945
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Checkpointing
>Affects Versions: 1.10.0
>Reporter: Biao Liu
>Assignee: Biao Liu
>Priority: Critical
> Labels: pull-request-available
> Fix For: 1.12.0
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> Since we have put all non-IO operations of {{CheckpointCoordinator}} into
> main thread executor, the {{CheckpointFailureManager.FailJobCallback}} could
> be executed directly now. In this way execution graph would fail immediately
> when {{CheckpointFailureManager}} invokes the callback. We could avoid the
> inconsistent scenario of FLINK-13497.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-20143) use `yarn.provided.lib.dirs` config deploy job failed in yarn per job mode
[
https://issues.apache.org/jira/browse/FLINK-20143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17231317#comment-17231317
]
zhisheng commented on FLINK-20143:
--
{code:java}
./bin/flink run -m yarn-cluster -d
-Dyarn.provided.lib.dirs="hdfs:///flink/flink-1.12-SNAPSHOT/lib"
./examples/streaming/StateMachineExample.jar
{code}
i use this command(remove the -ynm flink-1.12-test -ytm 3g -yjm ), it runs ok
> use `yarn.provided.lib.dirs` config deploy job failed in yarn per job mode
> --
>
> Key: FLINK-20143
> URL: https://issues.apache.org/jira/browse/FLINK-20143
> Project: Flink
> Issue Type: Bug
> Components: Client / Job Submission, Deployment / YARN
>Affects Versions: 1.12.0
>Reporter: zhisheng
>Priority: Major
> Attachments: image-2020-11-13-17-21-47-751.png,
> image-2020-11-13-17-22-06-111.png
>
>
> use follow command deploy flink job to yarn failed
> {code:java}
> ./bin/flink run -m yarn-cluster -d -ynm flink-1.12-test -ytm 3g -yjm 3g -yD
> yarn.provided.lib.dirs=hdfs:///flink/flink-1.12-SNAPSHOT/lib
> ./examples/streaming/StateMachineExample.jar
> {code}
> log:
> {code:java}
> $ ./bin/flink run -m yarn-cluster -d -ynm flink-1.12-test -ytm 3g -yjm 3g -yD
> yarn.provided.lib.dirs=hdfs:///flink/flink-1.12-SNAPSHOT/lib
> ./examples/streaming/StateMachineExample.jar$ ./bin/flink run -m yarn-cluster
> -d -ynm flink-1.12-test -ytm 3g -yjm 3g -yD
> yarn.provided.lib.dirs=hdfs:///flink/flink-1.12-SNAPSHOT/lib
> ./examples/streaming/StateMachineExample.jarSLF4J: Class path contains
> multiple SLF4J bindings.SLF4J: Found binding in
> [jar:file:/data1/app/flink-1.12-SNAPSHOT/lib/log4j-slf4j-impl-2.12.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J:
> Found binding in
> [jar:file:/data1/app/hadoop-2.7.3-snappy-32core12disk/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J:
> Found binding in
> [jar:file:/data1/app/hadoop-2.7.3-snappy-32core12disk/share/hadoop/tools/lib/hadoop-aliyun-2.9.2-jar-with-dependencies.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J:
> See http://www.slf4j.org/codes.html#multiple_bindings for an
> explanation.SLF4J: Actual binding is of type
> [org.apache.logging.slf4j.Log4jLoggerFactory]2020-11-13 16:14:30,347 INFO
> org.apache.flink.yarn.cli.FlinkYarnSessionCli [] - Dynamic
> Property set:
> yarn.provided.lib.dirs=hdfs:///flink/flink-1.12-SNAPSHOT/lib2020-11-13
> 16:14:30,347 INFO org.apache.flink.yarn.cli.FlinkYarnSessionCli
> [] - Dynamic Property set:
> yarn.provided.lib.dirs=hdfs:///flink/flink-1.12-SNAPSHOT/libUsage with
> built-in data generator: StateMachineExample [--error-rate
> ] [--sleep ]Usage
> with Kafka: StateMachineExample --kafka-topic [--brokers
> ]Options for both the above setups: [--backend ]
> [--checkpoint-dir ] [--async-checkpoints ]
> [--incremental-checkpoints ] [--output OR null for
> stdout]
> Using standalone source with error rate 0.00 and sleep delay 1 millis
> 2020-11-13 16:14:30,706 WARN
> org.apache.flink.yarn.configuration.YarnLogConfigUtil [] - The
> configuration directory ('/data1/app/flink-1.12-SNAPSHOT/conf') already
> contains a LOG4J config file.If you want to use logback, then please delete
> or rename the log configuration file.2020-11-13 16:14:30,947 INFO
> org.apache.hadoop.yarn.client.AHSProxy [] - Connecting
> to Application History server at
> FAT-hadoopuat-69117.vm.dc01.hellocloud.tech/10.69.1.17:102002020-11-13
> 16:14:30,958 INFO org.apache.flink.yarn.YarnClusterDescriptor
> [] - No path for the flink jar passed. Using the location of class
> org.apache.flink.yarn.YarnClusterDescriptor to locate the jar2020-11-13
> 16:14:31,065 INFO
> org.apache.hadoop.yarn.client.ConfiguredRMFailoverProxyProvider [] - Failing
> over to rm22020-11-13 16:14:31,130 INFO
> org.apache.flink.yarn.YarnClusterDescriptor [] - The
> configured JobManager memory is 3072 MB. YARN will allocate 4096 MB to make
> up an integer multiple of its minimum allocation memory (2048 MB, configured
> via 'yarn.scheduler.minimum-allocation-mb'). The extra 1024 MB may not be
> used by Flink.2020-11-13 16:14:31,130 INFO
> org.apache.flink.yarn.YarnClusterDescriptor [] - The
> configured TaskManager memory is 3072 MB. YARN will allocate 4096 MB to make
> up an integer multiple of its minimum allocation memory (2048 MB, configured
> via 'yarn.scheduler.minimum-allocation-mb'). The extra 1024 MB may not be
> used by Flink.2020-11-13 16:14:31,130 INFO
> org.apache.flink.yarn.YarnClusterDescriptor [] - Cluster
> specification: ClusterSpecification{masterMemoryMB=3072,
[jira] [Commented] (FLINK-20143) use `yarn.provided.lib.dirs` config deploy job failed in yarn per job mode
[
https://issues.apache.org/jira/browse/FLINK-20143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17231319#comment-17231319
]
zhisheng commented on FLINK-20143:
--
in our production environment，has many flink job，every job have the -ytm and
-yjm -ynm config，if we upgrade to 1.12，It could change a lot [~kkl0u]
> use `yarn.provided.lib.dirs` config deploy job failed in yarn per job mode
> --
>
> Key: FLINK-20143
> URL: https://issues.apache.org/jira/browse/FLINK-20143
> Project: Flink
> Issue Type: Bug
> Components: Client / Job Submission, Deployment / YARN
>Affects Versions: 1.12.0
>Reporter: zhisheng
>Priority: Major
> Attachments: image-2020-11-13-17-21-47-751.png,
> image-2020-11-13-17-22-06-111.png
>
>
> use follow command deploy flink job to yarn failed
> {code:java}
> ./bin/flink run -m yarn-cluster -d -ynm flink-1.12-test -ytm 3g -yjm 3g -yD
> yarn.provided.lib.dirs=hdfs:///flink/flink-1.12-SNAPSHOT/lib
> ./examples/streaming/StateMachineExample.jar
> {code}
> log:
> {code:java}
> $ ./bin/flink run -m yarn-cluster -d -ynm flink-1.12-test -ytm 3g -yjm 3g -yD
> yarn.provided.lib.dirs=hdfs:///flink/flink-1.12-SNAPSHOT/lib
> ./examples/streaming/StateMachineExample.jar$ ./bin/flink run -m yarn-cluster
> -d -ynm flink-1.12-test -ytm 3g -yjm 3g -yD
> yarn.provided.lib.dirs=hdfs:///flink/flink-1.12-SNAPSHOT/lib
> ./examples/streaming/StateMachineExample.jarSLF4J: Class path contains
> multiple SLF4J bindings.SLF4J: Found binding in
> [jar:file:/data1/app/flink-1.12-SNAPSHOT/lib/log4j-slf4j-impl-2.12.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J:
> Found binding in
> [jar:file:/data1/app/hadoop-2.7.3-snappy-32core12disk/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J:
> Found binding in
> [jar:file:/data1/app/hadoop-2.7.3-snappy-32core12disk/share/hadoop/tools/lib/hadoop-aliyun-2.9.2-jar-with-dependencies.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J:
> See http://www.slf4j.org/codes.html#multiple_bindings for an
> explanation.SLF4J: Actual binding is of type
> [org.apache.logging.slf4j.Log4jLoggerFactory]2020-11-13 16:14:30,347 INFO
> org.apache.flink.yarn.cli.FlinkYarnSessionCli [] - Dynamic
> Property set:
> yarn.provided.lib.dirs=hdfs:///flink/flink-1.12-SNAPSHOT/lib2020-11-13
> 16:14:30,347 INFO org.apache.flink.yarn.cli.FlinkYarnSessionCli
> [] - Dynamic Property set:
> yarn.provided.lib.dirs=hdfs:///flink/flink-1.12-SNAPSHOT/libUsage with
> built-in data generator: StateMachineExample [--error-rate
> ] [--sleep ]Usage
> with Kafka: StateMachineExample --kafka-topic [--brokers
> ]Options for both the above setups: [--backend ]
> [--checkpoint-dir ] [--async-checkpoints ]
> [--incremental-checkpoints ] [--output OR null for
> stdout]
> Using standalone source with error rate 0.00 and sleep delay 1 millis
> 2020-11-13 16:14:30,706 WARN
> org.apache.flink.yarn.configuration.YarnLogConfigUtil [] - The
> configuration directory ('/data1/app/flink-1.12-SNAPSHOT/conf') already
> contains a LOG4J config file.If you want to use logback, then please delete
> or rename the log configuration file.2020-11-13 16:14:30,947 INFO
> org.apache.hadoop.yarn.client.AHSProxy [] - Connecting
> to Application History server at
> FAT-hadoopuat-69117.vm.dc01.hellocloud.tech/10.69.1.17:102002020-11-13
> 16:14:30,958 INFO org.apache.flink.yarn.YarnClusterDescriptor
> [] - No path for the flink jar passed. Using the location of class
> org.apache.flink.yarn.YarnClusterDescriptor to locate the jar2020-11-13
> 16:14:31,065 INFO
> org.apache.hadoop.yarn.client.ConfiguredRMFailoverProxyProvider [] - Failing
> over to rm22020-11-13 16:14:31,130 INFO
> org.apache.flink.yarn.YarnClusterDescriptor [] - The
> configured JobManager memory is 3072 MB. YARN will allocate 4096 MB to make
> up an integer multiple of its minimum allocation memory (2048 MB, configured
> via 'yarn.scheduler.minimum-allocation-mb'). The extra 1024 MB may not be
> used by Flink.2020-11-13 16:14:31,130 INFO
> org.apache.flink.yarn.YarnClusterDescriptor [] - The
> configured TaskManager memory is 3072 MB. YARN will allocate 4096 MB to make
> up an integer multiple of its minimum allocation memory (2048 MB, configured
> via 'yarn.scheduler.minimum-allocation-mb'). The extra 1024 MB may not be
> used by Flink.2020-11-13 16:14:31,130 INFO
> org.apache.flink.yarn.YarnClusterDescriptor [] - Cluster
> specification: ClusterSpecification{masterMemoryMB=3072,
> taskManagerMemoryMB=3072, slotsPerTaskManager=2}2020-11-13 16:14:31,681 WARN
> org.apache.hadoop.hd
[jira] [Commented] (FLINK-20143) use `yarn.provided.lib.dirs` config deploy job failed in yarn per job mode
[
https://issues.apache.org/jira/browse/FLINK-20143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17231321#comment-17231321
]
zhisheng commented on FLINK-20143:
--
Are there any other methods to make job config compatibility？[~kkl0u]
> use `yarn.provided.lib.dirs` config deploy job failed in yarn per job mode
> --
>
> Key: FLINK-20143
> URL: https://issues.apache.org/jira/browse/FLINK-20143
> Project: Flink
> Issue Type: Bug
> Components: Client / Job Submission, Deployment / YARN
>Affects Versions: 1.12.0
>Reporter: zhisheng
>Priority: Major
> Attachments: image-2020-11-13-17-21-47-751.png,
> image-2020-11-13-17-22-06-111.png
>
>
> use follow command deploy flink job to yarn failed
> {code:java}
> ./bin/flink run -m yarn-cluster -d -ynm flink-1.12-test -ytm 3g -yjm 3g -yD
> yarn.provided.lib.dirs=hdfs:///flink/flink-1.12-SNAPSHOT/lib
> ./examples/streaming/StateMachineExample.jar
> {code}
> log:
> {code:java}
> $ ./bin/flink run -m yarn-cluster -d -ynm flink-1.12-test -ytm 3g -yjm 3g -yD
> yarn.provided.lib.dirs=hdfs:///flink/flink-1.12-SNAPSHOT/lib
> ./examples/streaming/StateMachineExample.jar$ ./bin/flink run -m yarn-cluster
> -d -ynm flink-1.12-test -ytm 3g -yjm 3g -yD
> yarn.provided.lib.dirs=hdfs:///flink/flink-1.12-SNAPSHOT/lib
> ./examples/streaming/StateMachineExample.jarSLF4J: Class path contains
> multiple SLF4J bindings.SLF4J: Found binding in
> [jar:file:/data1/app/flink-1.12-SNAPSHOT/lib/log4j-slf4j-impl-2.12.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J:
> Found binding in
> [jar:file:/data1/app/hadoop-2.7.3-snappy-32core12disk/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J:
> Found binding in
> [jar:file:/data1/app/hadoop-2.7.3-snappy-32core12disk/share/hadoop/tools/lib/hadoop-aliyun-2.9.2-jar-with-dependencies.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J:
> See http://www.slf4j.org/codes.html#multiple_bindings for an
> explanation.SLF4J: Actual binding is of type
> [org.apache.logging.slf4j.Log4jLoggerFactory]2020-11-13 16:14:30,347 INFO
> org.apache.flink.yarn.cli.FlinkYarnSessionCli [] - Dynamic
> Property set:
> yarn.provided.lib.dirs=hdfs:///flink/flink-1.12-SNAPSHOT/lib2020-11-13
> 16:14:30,347 INFO org.apache.flink.yarn.cli.FlinkYarnSessionCli
> [] - Dynamic Property set:
> yarn.provided.lib.dirs=hdfs:///flink/flink-1.12-SNAPSHOT/libUsage with
> built-in data generator: StateMachineExample [--error-rate
> ] [--sleep ]Usage
> with Kafka: StateMachineExample --kafka-topic [--brokers
> ]Options for both the above setups: [--backend ]
> [--checkpoint-dir ] [--async-checkpoints ]
> [--incremental-checkpoints ] [--output OR null for
> stdout]
> Using standalone source with error rate 0.00 and sleep delay 1 millis
> 2020-11-13 16:14:30,706 WARN
> org.apache.flink.yarn.configuration.YarnLogConfigUtil [] - The
> configuration directory ('/data1/app/flink-1.12-SNAPSHOT/conf') already
> contains a LOG4J config file.If you want to use logback, then please delete
> or rename the log configuration file.2020-11-13 16:14:30,947 INFO
> org.apache.hadoop.yarn.client.AHSProxy [] - Connecting
> to Application History server at
> FAT-hadoopuat-69117.vm.dc01.hellocloud.tech/10.69.1.17:102002020-11-13
> 16:14:30,958 INFO org.apache.flink.yarn.YarnClusterDescriptor
> [] - No path for the flink jar passed. Using the location of class
> org.apache.flink.yarn.YarnClusterDescriptor to locate the jar2020-11-13
> 16:14:31,065 INFO
> org.apache.hadoop.yarn.client.ConfiguredRMFailoverProxyProvider [] - Failing
> over to rm22020-11-13 16:14:31,130 INFO
> org.apache.flink.yarn.YarnClusterDescriptor [] - The
> configured JobManager memory is 3072 MB. YARN will allocate 4096 MB to make
> up an integer multiple of its minimum allocation memory (2048 MB, configured
> via 'yarn.scheduler.minimum-allocation-mb'). The extra 1024 MB may not be
> used by Flink.2020-11-13 16:14:31,130 INFO
> org.apache.flink.yarn.YarnClusterDescriptor [] - The
> configured TaskManager memory is 3072 MB. YARN will allocate 4096 MB to make
> up an integer multiple of its minimum allocation memory (2048 MB, configured
> via 'yarn.scheduler.minimum-allocation-mb'). The extra 1024 MB may not be
> used by Flink.2020-11-13 16:14:31,130 INFO
> org.apache.flink.yarn.YarnClusterDescriptor [] - Cluster
> specification: ClusterSpecification{masterMemoryMB=3072,
> taskManagerMemoryMB=3072, slotsPerTaskManager=2}2020-11-13 16:14:31,681 WARN
> org.apache.hadoop.hdfs.shortcircuit.DomainSocketFactory [] - The
> short-circuit local reads
[jira] [Commented] (FLINK-16945) Execute CheckpointFailureManager.FailJobCallback directly in main thread executor
[
https://issues.apache.org/jira/browse/FLINK-16945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17231320#comment-17231320
]
Dian Fu commented on FLINK-16945:
-
[~AHeise] Thanks for the confirmation.
> Execute CheckpointFailureManager.FailJobCallback directly in main thread
> executor
> -
>
> Key: FLINK-16945
> URL: https://issues.apache.org/jira/browse/FLINK-16945
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Checkpointing
>Affects Versions: 1.10.0
>Reporter: Biao Liu
>Assignee: Biao Liu
>Priority: Critical
> Labels: pull-request-available
> Fix For: 1.12.0
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> Since we have put all non-IO operations of {{CheckpointCoordinator}} into
> main thread executor, the {{CheckpointFailureManager.FailJobCallback}} could
> be executed directly now. In this way execution graph would fail immediately
> when {{CheckpointFailureManager}} invokes the callback. We could avoid the
> inconsistent scenario of FLINK-13497.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-16945) Execute CheckpointFailureManager.FailJobCallback directly in main thread executor
[
https://issues.apache.org/jira/browse/FLINK-16945?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dian Fu updated FLINK-16945:
Fix Version/s: (was: 1.12.0)
1.13.0
> Execute CheckpointFailureManager.FailJobCallback directly in main thread
> executor
> -
>
> Key: FLINK-16945
> URL: https://issues.apache.org/jira/browse/FLINK-16945
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Checkpointing
>Affects Versions: 1.10.0
>Reporter: Biao Liu
>Assignee: Biao Liu
>Priority: Critical
> Labels: pull-request-available
> Fix For: 1.13.0
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> Since we have put all non-IO operations of {{CheckpointCoordinator}} into
> main thread executor, the {{CheckpointFailureManager.FailJobCallback}} could
> be executed directly now. In this way execution graph would fail immediately
> when {{CheckpointFailureManager}} invokes the callback. We could avoid the
> inconsistent scenario of FLINK-13497.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (FLINK-20145) Streaming job fails with IllegalStateException: Should only poll priority events
[
https://issues.apache.org/jira/browse/FLINK-20145?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arvid Heise reassigned FLINK-20145:
---
Assignee: Arvid Heise
> Streaming job fails with IllegalStateException: Should only poll priority
> events
>
>
> Key: FLINK-20145
> URL: https://issues.apache.org/jira/browse/FLINK-20145
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Checkpointing, Runtime / Network
>Affects Versions: 1.12.0
>Reporter: Robert Metzger
>Assignee: Arvid Heise
>Priority: Critical
> Fix For: 1.12.0
>
>
> While testing the 1.12 release, I came across the following failure cause:
> {code}
> 2020-11-13 09:41:52,110 WARN org.apache.flink.runtime.taskmanager.Task
> [] - dynamic filter (3/4)#0 (b977944851531f96e5324e786f055eb7)
> switched from RUNNING to FAILED.
> java.lang.IllegalStateException: Should only poll priority events
> at
> org.apache.flink.util.Preconditions.checkState(Preconditions.java:198)
> ~[flink-dist_2.11-1.12.0.jar:1.12.0]
> at
> org.apache.flink.streaming.runtime.io.CheckpointedInputGate.processPriorityEvents(CheckpointedInputGate.java:116)
> ~[flink-dist_2.11-1.12.0.jar:1.12.0]
> at
> org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$1.runThrowing(StreamTaskActionExecutor.java:47)
> ~[flink-dist_2.11-1.12.0.jar:1.12.0]
> at
> org.apache.flink.streaming.runtime.tasks.mailbox.Mail.run(Mail.java:78)
> ~[flink-dist_2.11-1.12.0.jar:1.12.0]
> at
> org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.processMail(MailboxProcessor.java:283)
> ~[flink-dist_2.11-1.12.0.jar:1.12.0]
> at
> org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:184)
> ~[flink-dist_2.11-1.12.0.jar:1.12.0]
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:577)
> ~[flink-dist_2.11-1.12.0.jar:1.12.0]
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:541)
> ~[flink-dist_2.11-1.12.0.jar:1.12.0]
> at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:722)
> [flink-dist_2.11-1.12.0.jar:1.12.0]
> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:547)
> [flink-dist_2.11-1.12.0.jar:1.12.0]
> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_222]
> {code}
> I have unaligned checkpointing enabled, the failing operator is a
> CoFlatMapFunction. The error happend on all four TaskManagers, very soon
> after job submission. The error doesn't happen when unaligned checkpointing
> is disabled.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-20143) use `yarn.provided.lib.dirs` config deploy job failed in yarn per job mode
[
https://issues.apache.org/jira/browse/FLINK-20143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17231326#comment-17231326
]
Kostas Kloudas commented on FLINK-20143:
Also the above command seems to be problematic.
{code:java}
./bin/flink run -m yarn-cluster -d
-Dyarn.provided.lib.dirs="hdfs:///flink/flink-1.12-SNAPSHOT/lib"
./examples/streaming/StateMachineExample.jar
{code}
What if you use?
{code:java}
./bin/flink run -t yarn-per-job -Dexecution.attached=false
-Dyarn.provided.lib.dirs="hdfs:///flink/flink-1.12-SNAPSHOT/lib"
./examples/streaming/StateMachineExample.jar
{code}
and please check the logs to see if the shared dir is picked up or you are
shipping everything from the client.
> use `yarn.provided.lib.dirs` config deploy job failed in yarn per job mode
> --
>
> Key: FLINK-20143
> URL: https://issues.apache.org/jira/browse/FLINK-20143
> Project: Flink
> Issue Type: Bug
> Components: Client / Job Submission, Deployment / YARN
>Affects Versions: 1.12.0
>Reporter: zhisheng
>Priority: Major
> Attachments: image-2020-11-13-17-21-47-751.png,
> image-2020-11-13-17-22-06-111.png
>
>
> use follow command deploy flink job to yarn failed
> {code:java}
> ./bin/flink run -m yarn-cluster -d -ynm flink-1.12-test -ytm 3g -yjm 3g -yD
> yarn.provided.lib.dirs=hdfs:///flink/flink-1.12-SNAPSHOT/lib
> ./examples/streaming/StateMachineExample.jar
> {code}
> log:
> {code:java}
> $ ./bin/flink run -m yarn-cluster -d -ynm flink-1.12-test -ytm 3g -yjm 3g -yD
> yarn.provided.lib.dirs=hdfs:///flink/flink-1.12-SNAPSHOT/lib
> ./examples/streaming/StateMachineExample.jar$ ./bin/flink run -m yarn-cluster
> -d -ynm flink-1.12-test -ytm 3g -yjm 3g -yD
> yarn.provided.lib.dirs=hdfs:///flink/flink-1.12-SNAPSHOT/lib
> ./examples/streaming/StateMachineExample.jarSLF4J: Class path contains
> multiple SLF4J bindings.SLF4J: Found binding in
> [jar:file:/data1/app/flink-1.12-SNAPSHOT/lib/log4j-slf4j-impl-2.12.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J:
> Found binding in
> [jar:file:/data1/app/hadoop-2.7.3-snappy-32core12disk/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J:
> Found binding in
> [jar:file:/data1/app/hadoop-2.7.3-snappy-32core12disk/share/hadoop/tools/lib/hadoop-aliyun-2.9.2-jar-with-dependencies.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J:
> See http://www.slf4j.org/codes.html#multiple_bindings for an
> explanation.SLF4J: Actual binding is of type
> [org.apache.logging.slf4j.Log4jLoggerFactory]2020-11-13 16:14:30,347 INFO
> org.apache.flink.yarn.cli.FlinkYarnSessionCli [] - Dynamic
> Property set:
> yarn.provided.lib.dirs=hdfs:///flink/flink-1.12-SNAPSHOT/lib2020-11-13
> 16:14:30,347 INFO org.apache.flink.yarn.cli.FlinkYarnSessionCli
> [] - Dynamic Property set:
> yarn.provided.lib.dirs=hdfs:///flink/flink-1.12-SNAPSHOT/libUsage with
> built-in data generator: StateMachineExample [--error-rate
> ] [--sleep ]Usage
> with Kafka: StateMachineExample --kafka-topic [--brokers
> ]Options for both the above setups: [--backend ]
> [--checkpoint-dir ] [--async-checkpoints ]
> [--incremental-checkpoints ] [--output OR null for
> stdout]
> Using standalone source with error rate 0.00 and sleep delay 1 millis
> 2020-11-13 16:14:30,706 WARN
> org.apache.flink.yarn.configuration.YarnLogConfigUtil [] - The
> configuration directory ('/data1/app/flink-1.12-SNAPSHOT/conf') already
> contains a LOG4J config file.If you want to use logback, then please delete
> or rename the log configuration file.2020-11-13 16:14:30,947 INFO
> org.apache.hadoop.yarn.client.AHSProxy [] - Connecting
> to Application History server at
> FAT-hadoopuat-69117.vm.dc01.hellocloud.tech/10.69.1.17:102002020-11-13
> 16:14:30,958 INFO org.apache.flink.yarn.YarnClusterDescriptor
> [] - No path for the flink jar passed. Using the location of class
> org.apache.flink.yarn.YarnClusterDescriptor to locate the jar2020-11-13
> 16:14:31,065 INFO
> org.apache.hadoop.yarn.client.ConfiguredRMFailoverProxyProvider [] - Failing
> over to rm22020-11-13 16:14:31,130 INFO
> org.apache.flink.yarn.YarnClusterDescriptor [] - The
> configured JobManager memory is 3072 MB. YARN will allocate 4096 MB to make
> up an integer multiple of its minimum allocation memory (2048 MB, configured
> via 'yarn.scheduler.minimum-allocation-mb'). The extra 1024 MB may not be
> used by Flink.2020-11-13 16:14:31,130 INFO
> org.apache.flink.yarn.YarnClusterDescriptor [] - The
> configured TaskManager memory is 3072 MB. YARN will allocate 4096 MB to make
> up an integer multiple of its minimum a
[GitHub] [flink] kl0u commented on pull request #14061: [FLINK-20141][fs-connector] Add FileSink documentation
kl0u commented on pull request #14061: URL: https://github.com/apache/flink/pull/14061#issuecomment-726660630 Thanks @gaoyunhaii ! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] openinx commented on a change in pull request #14032: [FLINK-19863][tests] check hbase has been ready before use it
openinx commented on a change in pull request #14032:
URL: https://github.com/apache/flink/pull/14032#discussion_r522838516
##
File path:
flink-end-to-end-tests/flink-end-to-end-tests-hbase/src/main/java/org/apache/flink/tests/util/hbase/LocalStandaloneHBaseResource.java
##
@@ -92,62 +95,98 @@ private void setupHBaseDist() throws IOException {
}
private void setupHBaseCluster() throws IOException {
- LOG.info("Starting HBase cluster");
- AutoClosableProcess.runBlocking(
- hbaseDir.resolve(Paths.get("bin",
"start-hbase.sh")).toString());
-
- while (!isHBaseRunning()) {
- try {
- LOG.info("Waiting for HBase to start");
- Thread.sleep(500L);
- } catch (InterruptedException e) {
- Thread.currentThread().interrupt();
- break;
- }
- }
+ LOG.info("Starting HBase cluster...");
+ runHBaseProcessWithRetry("start-hbase.sh", () ->
!isHMasterRunning());
+ LOG.info("Start HBase cluster success");
}
@Override
public void afterTestSuccess() {
+ shutdownResource();
+ downloadCache.afterTestSuccess();
+ tmp.delete();
+ }
+
+ private void shutdownResource() {
+ LOG.info("Stopping HBase Cluster...");
try {
- LOG.info("Stopping HBase Cluster");
- AutoClosableProcess.runBlocking(
- hbaseDir.resolve(Paths.get("bin",
"hbase-daemon.sh")).toString(),
- "stop",
- "master");
+ runHBaseProcessWithRetry("stop-hbase.sh", () ->
isHMasterAlive());
Review comment:
I'd prefer to shutdown the hbase process by killing the process id
because I encountered the hung case before when calling the `stop-hbase.sh`.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14033: [FLINK-19979][e2e] Add sanity check after bash e2e tests for no leftovers
flinkbot edited a comment on pull request #14033: URL: https://github.com/apache/flink/pull/14033#issuecomment-725388043 ## CI report: * 9e5179d99ac3f1ee67388f6236a5ffd087941a94 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9552) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14052: [FLINK-20030][network] Rewrite RemoteInputChannel#getInflightBuffersUnsafe to use sequence numbers
flinkbot edited a comment on pull request #14052: URL: https://github.com/apache/flink/pull/14052#issuecomment-726216034 ## CI report: * 14e0929bf9cebd4db7c60c5928d71e631fb4ad47 UNKNOWN * 0d6c68e97c1c6d23ca2870932db0cdfeddebcafd Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9528) * 9f30c9730919038a0247cd4b7a2929df05ee669f Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9557) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-15992) Incorrect classloader when finding TableFactory
[
https://issues.apache.org/jira/browse/FLINK-15992?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aljoscha Krettek updated FLINK-15992:
-
Labels: pull-request-available (was: pull-request-available test-stability)
> Incorrect classloader when finding TableFactory
> ---
>
> Key: FLINK-15992
> URL: https://issues.apache.org/jira/browse/FLINK-15992
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Kafka, Table SQL / API, Tests
>Reporter: Victor Wong
>Assignee: Victor Wong
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.12.0
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> *Background*
> As a streaming service maintainer in our company, to ensure our users depend
> on the correct version of Kafka and flink-kafka, we add
> "flink-connector-kafka" into "fink-dist/lib" directory.
> *Problem*
> When submitting flink-sql jobs, we encountered below exceptions:
> {code:java}
> Caused by: org.apache.flink.table.api.NoMatchingTableFactoryException: Could
> not find a suitable table factory for
> 'org.apache.flink.table.factories.DeserializationSchemaFactory' in
> the classpath.
> {code}
> But we have add "org.apache.flink.formats.json.JsonRowFormatFactory" in
> "META-INF/services/org.apache.flink.table.factories.TableFactory", which
> implements DeserializationSchemaFactory.
> *Debug*
> We find that it was caused by this:
> {code:java}
> //
> org.apache.flink.streaming.connectors.kafka.KafkaTableSourceSinkFactoryBase#getSerializationSchema
> final SerializationSchemaFactory formatFactory =
> TableFactoryService.find(
> SerializationSchemaFactory.class,
> properties,
> this.getClass().getClassLoader());
> {code}
> It uses `this.getClass().getClassLoader()`, which will be
> BootStrapClassLoader of flink.
> I think we could replace it with
> `Thread.currentThread().getContextClassLoader()` to solve this.
> There is a related issue: https://issues.apache.org/jira/browse/FLINK-15552
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-16945) Execute CheckpointFailureManager.FailJobCallback directly in main thread executor
[
https://issues.apache.org/jira/browse/FLINK-16945?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arvid Heise updated FLINK-16945:
Priority: Major (was: Critical)
> Execute CheckpointFailureManager.FailJobCallback directly in main thread
> executor
> -
>
> Key: FLINK-16945
> URL: https://issues.apache.org/jira/browse/FLINK-16945
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Checkpointing
>Affects Versions: 1.10.0
>Reporter: Biao Liu
>Assignee: Biao Liu
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.13.0
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> Since we have put all non-IO operations of {{CheckpointCoordinator}} into
> main thread executor, the {{CheckpointFailureManager.FailJobCallback}} could
> be executed directly now. In this way execution graph would fail immediately
> when {{CheckpointFailureManager}} invokes the callback. We could avoid the
> inconsistent scenario of FLINK-13497.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-19775) SystemProcessingTimeServiceTest.testImmediateShutdown is instable
[
https://issues.apache.org/jira/browse/FLINK-19775?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17231340#comment-17231340
]
jiawen xiao commented on FLINK-19775:
-
Hi [~dian.fu] , i'm interested in this issus , could you assignee to me to
repair.
> SystemProcessingTimeServiceTest.testImmediateShutdown is instable
> -
>
> Key: FLINK-19775
> URL: https://issues.apache.org/jira/browse/FLINK-19775
> Project: Flink
> Issue Type: Bug
> Components: API / DataStream
>Affects Versions: 1.11.0
>Reporter: Dian Fu
>Priority: Major
> Labels: test-stability
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=8131&view=logs&j=d89de3df-4600-5585-dadc-9bbc9a5e661c&t=66b5c59a-0094-561d-0e44-b149dfdd586d
> {code}
> 2020-10-22T21:12:54.9462382Z [ERROR]
> testImmediateShutdown(org.apache.flink.streaming.runtime.tasks.SystemProcessingTimeServiceTest)
> Time elapsed: 0.009 s <<< ERROR!
> 2020-10-22T21:12:54.9463024Z java.lang.InterruptedException
> 2020-10-22T21:12:54.9463331Z at java.lang.Object.wait(Native Method)
> 2020-10-22T21:12:54.9463766Z at java.lang.Object.wait(Object.java:502)
> 2020-10-22T21:12:54.9464140Z at
> org.apache.flink.core.testutils.OneShotLatch.await(OneShotLatch.java:63)
> 2020-10-22T21:12:54.9466014Z at
> org.apache.flink.streaming.runtime.tasks.SystemProcessingTimeServiceTest.testImmediateShutdown(SystemProcessingTimeServiceTest.java:154)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-20143) use `yarn.provided.lib.dirs` config deploy job failed in yarn per job mode
[
https://issues.apache.org/jira/browse/FLINK-20143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17231346#comment-17231346
]
zhisheng commented on FLINK-20143:
--
{code:java}
$ ./bin/flink run -t yarn-per-job -Dexecution.attached=false
-Dyarn.provided.lib.dirs="hdfs:///flink/flink-1.12-SNAPSHOT/lib"
./examples/streaming/StateMachineExample.jar$ ./bin/flink run -t yarn-per-job
-Dexecution.attached=false
-Dyarn.provided.lib.dirs="hdfs:///flink/flink-1.12-SNAPSHOT/lib"
./examples/streaming/StateMachineExample.jarSLF4J: Class path contains multiple
SLF4J bindings.SLF4J: Found binding in
[jar:file:/data1/app/flink-1.12-SNAPSHOT/lib/log4j-slf4j-impl-2.12.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J:
Found binding in
[jar:file:/data1/app/hadoop-2.7.3-snappy-32core12disk/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J:
Found binding in
[jar:file:/data1/app/hadoop-2.7.3-snappy-32core12disk/share/hadoop/tools/lib/hadoop-aliyun-2.9.2-jar-with-dependencies.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J:
See http://www.slf4j.org/codes.html#multiple_bindings for an
explanation.SLF4J: Actual binding is of type
[org.apache.logging.slf4j.Log4jLoggerFactory]Usage with built-in data
generator: StateMachineExample [--error-rate
] [--sleep ]Usage
with Kafka: StateMachineExample --kafka-topic [--brokers
]Options for both the above setups: [--backend ]
[--checkpoint-dir ] [--async-checkpoints ]
[--incremental-checkpoints ] [--output OR null for
stdout]
Using standalone source with error rate 0.00 and sleep delay 1 millis
2020-11-13 18:05:51,974 WARN
org.apache.flink.yarn.configuration.YarnLogConfigUtil [] - The
configuration directory ('/data1/app/flink-1.12-SNAPSHOT/conf') already
contains a LOG4J config file.If you want to use logback, then please delete or
rename the log configuration file.2020-11-13 18:05:52,202 INFO
org.apache.hadoop.yarn.client.AHSProxy [] - Connecting to
Application History server at
FAT-hadoopuat-69117.vm.dc01.hellocloud.tech/10.69.1.17:102002020-11-13
18:05:52,213 INFO org.apache.flink.yarn.YarnClusterDescriptor
[] - No path for the flink jar passed. Using the location of class
org.apache.flink.yarn.YarnClusterDescriptor to locate the jar2020-11-13
18:05:52,324 INFO
org.apache.hadoop.yarn.client.ConfiguredRMFailoverProxyProvider [] - Failing
over to rm22020-11-13 18:05:52,387 INFO
org.apache.flink.yarn.YarnClusterDescriptor [] - The
configured JobManager memory is 1600 MB. YARN will allocate 2048 MB to make up
an integer multiple of its minimum allocation memory (2048 MB, configured via
'yarn.scheduler.minimum-allocation-mb'). The extra 448 MB may not be used by
Flink.2020-11-13 18:05:52,388 INFO org.apache.flink.yarn.YarnClusterDescriptor
[] - The configured TaskManager memory is 1728 MB. YARN will
allocate 2048 MB to make up an integer multiple of its minimum allocation
memory (2048 MB, configured via 'yarn.scheduler.minimum-allocation-mb'). The
extra 320 MB may not be used by Flink.2020-11-13 18:05:52,388 INFO
org.apache.flink.yarn.YarnClusterDescriptor [] - Cluster
specification: ClusterSpecification{masterMemoryMB=2048,
taskManagerMemoryMB=1728, slotsPerTaskManager=2}2020-11-13 18:05:52,932 WARN
org.apache.hadoop.hdfs.shortcircuit.DomainSocketFactory [] - The
short-circuit local reads feature cannot be used because libhadoop cannot be
loaded.2020-11-13 18:05:55,076 INFO
org.apache.flink.yarn.YarnClusterDescriptor [] - Submitting
application master application_1599741232083_220112020-11-13 18:05:55,307 INFO
org.apache.hadoop.yarn.client.api.impl.YarnClientImpl [] - Submitted
application application_1599741232083_220112020-11-13 18:05:55,308 INFO
org.apache.flink.yarn.YarnClusterDescriptor [] - Waiting for
the cluster to be allocated2020-11-13 18:05:55,310 INFO
org.apache.flink.yarn.YarnClusterDescriptor [] - Deploying
cluster, current state ACCEPTED
The program
finished with the following exception:
org.apache.flink.client.program.ProgramInvocationException: The main method
caused an error: Could not deploy Yarn job cluster. at
org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:330)
at
org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:198)
at org.apache.flink.client.ClientUtils.executeProgram(ClientUtils.java:114) at
org.apache.flink.client.cli.CliFrontend.executeProgram(CliFrontend.java:743) at
org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:242) at
org.apache.flink.client.cli.CliFrontend.parseAndRun(CliFrontend.java:971) at
org.apache.flink.client.cli.CliFro
[jira] [Commented] (FLINK-20128) Data loss for over windows with rows unbounded preceding
[
https://issues.apache.org/jira/browse/FLINK-20128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17231347#comment-17231347
]
Shengkai Fang commented on FLINK-20128:
---
[~dian.fu] In my local enviroment, my java code are
{code:java}
// Some comments here
tEnv.executeSql(
"CREATE TABLE datagen(\n" +
" foo INT,\n" +
" id AS mod(foo, 2),\n" +
" message_time AS
to_timestamp(from_unixtime(FLOOR(foo/2))),\n" +
" WATERMARK FOR message_time
AS message_time\n" +
") WITH (" +
" 'connector' = 'datagen'," +
" 'rows-per-second'='2'," +
"
'fields.foo.kind'='sequence'," +
" 'fields.foo.start'='0'," +
" 'fields.foo.end'='19'" +
")"
);
tEnv.executeSql(
"SELECT foo, id, avg(foo) OVER w AS lagfoo \n" +
"FROM datagen \n" +
"WINDOW w AS (PARTITION BY id
ORDER BY message_time ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW)"
).print();
{code}
The results are
{code}
++-+-+-+
| op | foo | id | lagfoo |
++-+-+-+
| +I | 0 | 0 | 0 |
| +I | 1 | 1 | 1 |
| +I | 3 | 1 | 2 |
| +I | 2 | 0 | 1 |
| +I | 4 | 0 | 2 |
| +I | 5 | 1 | 3 |
| +I | 7 | 1 | 4 |
| +I | 6 | 0 | 3 |
| +I | 8 | 0 | 4 |
| +I | 9 | 1 | 5 |
| +I | 10 | 0 | 5 |
| +I | 12 | 0 | 6 |
| +I | 11 | 1 | 6 |
| +I | 13 | 1 | 7 |
| +I | 14 | 0 | 7 |
| +I | 16 | 0 | 8 |
| +I | 15 | 1 | 8 |
| +I | 17 | 1 | 9 |
| +I | 18 | 0 | 9 |
| +I | 19 | 1 | 10 |
++-+-+-+
{code}
I am not sure the code above is as same as the python code.
> Data loss for over windows with rows unbounded preceding
>
>
> Key: FLINK-20128
> URL: https://issues.apache.org/jira/browse/FLINK-20128
> Project: Flink
> Issue Type: Bug
> Components: API / Python, Table SQL / Planner
>Affects Versions: 1.12.0, 1.11.2
>Reporter: Thilo Schneider
>Priority: Major
>
> When using partitioned, unbounded over windows, all but one partitions are
> dropped in the output dataset:
> {code:python}
> # Setup
> from pyflink.table import EnvironmentSettings, StreamTableEnvironment
> from biafflink import debug_print_table
> env_settings =
> EnvironmentSettings.new_instance().in_streaming_mode().use_blink_planner().build()
> t_env = StreamTableEnvironment.create( environment_settings=env_settings)
> t_env.get_config().get_configuration().set_integer("table.exec.resource.default-parallelism",
> 1)
> t_env.execute_sql("""
> CREATE TABLE datagen (
> foo INT,
> id AS mod(foo, 2),
> message_time AS to_timestamp(from_unixtime(FLOOR(foo/2))),
> WATERMARK FOR message_time AS message_time
> ) WITH (
> 'connector' = 'datagen',
> 'rows-per-second'='2',
> 'fields.foo.kind'='sequence',
> 'fields.foo.start'='0',
> 'fields.foo.end'='19'
> )""")
> t_env.execute_sql("CREATE TABLE output (foo INT, id INT, lagfoo INT) WITH
> ('connector' = 'print')")
> {code}
> Using bounded over windows, everything works as expected:
> {code:python}
> t = t_env.sql_query("""
> SELECT foo, id, avg(foo) OVER w AS lagfoo
> FROM datagen
> WINDOW w AS (PARTITION BY id ORDER BY message_time ROWS BETWEEN 1
> PRECEDING AND CURRENT ROW)""")
> t.execute_insert("output")
> {code}
> yields
> {code:python}
> +I(0,0,0)
> +I(1,1,1)
> +I(2,0,1)
> +I(3,1,2)
> +I(4,0,3)
> ...
> {code}
> If we change the window to unbounded preceding:
> {code:python}
> t = t_env.sql_query("""
> SELECT foo, id, avg(foo) OVER w AS lagfoo
> FROM datagen
> WINDOW w AS (PARTITION BY id
[jira] [Comment Edited] (FLINK-20143) use `yarn.provided.lib.dirs` config deploy job failed in yarn per job mode
[
https://issues.apache.org/jira/browse/FLINK-20143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17231346#comment-17231346
]
zhisheng edited comment on FLINK-20143 at 11/13/20, 10:07 AM:
--
{code:java}
$ ./bin/flink run -t yarn-per-job -Dexecution.attached=false
-Dyarn.provided.lib.dirs="hdfs:///flink/flink-1.12-SNAPSHOT/lib"
./examples/streaming/StateMachineExample.jar$ ./bin/flink run -t yarn-per-job
-Dexecution.attached=false
-Dyarn.provided.lib.dirs="hdfs:///flink/flink-1.12-SNAPSHOT/lib"
./examples/streaming/StateMachineExample.jarSLF4J: Class path contains multiple
SLF4J bindings.SLF4J: Found binding in
[jar:file:/data1/app/flink-1.12-SNAPSHOT/lib/log4j-slf4j-impl-2.12.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J:
Found binding in
[jar:file:/data1/app/hadoop-2.7.3-snappy-32core12disk/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J:
Found binding in
[jar:file:/data1/app/hadoop-2.7.3-snappy-32core12disk/share/hadoop/tools/lib/hadoop-aliyun-2.9.2-jar-with-dependencies.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J:
See http://www.slf4j.org/codes.html#multiple_bindings for an
explanation.SLF4J: Actual binding is of type
[org.apache.logging.slf4j.Log4jLoggerFactory]Usage with built-in data
generator: StateMachineExample [--error-rate
] [--sleep ]Usage
with Kafka: StateMachineExample --kafka-topic [--brokers
]Options for both the above setups: [--backend ]
[--checkpoint-dir ] [--async-checkpoints ]
[--incremental-checkpoints ] [--output OR null for
stdout]
Using standalone source with error rate 0.00 and sleep delay 1 millis
2020-11-13 18:05:51,974 WARN
org.apache.flink.yarn.configuration.YarnLogConfigUtil [] - The
configuration directory ('/data1/app/flink-1.12-SNAPSHOT/conf') already
contains a LOG4J config file.If you want to use logback, then please delete or
rename the log configuration file.2020-11-13 18:05:52,202 INFO
org.apache.hadoop.yarn.client.AHSProxy [] - Connecting to
Application History server at
FAT-hadoopuat-69117.vm.dc01.tech/10.69.1.17:102002020-11-13 18:05:52,213 INFO
org.apache.flink.yarn.YarnClusterDescriptor [] - No path for
the flink jar passed. Using the location of class
org.apache.flink.yarn.YarnClusterDescriptor to locate the jar2020-11-13
18:05:52,324 INFO
org.apache.hadoop.yarn.client.ConfiguredRMFailoverProxyProvider [] - Failing
over to rm22020-11-13 18:05:52,387 INFO
org.apache.flink.yarn.YarnClusterDescriptor [] - The
configured JobManager memory is 1600 MB. YARN will allocate 2048 MB to make up
an integer multiple of its minimum allocation memory (2048 MB, configured via
'yarn.scheduler.minimum-allocation-mb'). The extra 448 MB may not be used by
Flink.2020-11-13 18:05:52,388 INFO org.apache.flink.yarn.YarnClusterDescriptor
[] - The configured TaskManager memory is 1728 MB. YARN will
allocate 2048 MB to make up an integer multiple of its minimum allocation
memory (2048 MB, configured via 'yarn.scheduler.minimum-allocation-mb'). The
extra 320 MB may not be used by Flink.2020-11-13 18:05:52,388 INFO
org.apache.flink.yarn.YarnClusterDescriptor [] - Cluster
specification: ClusterSpecification{masterMemoryMB=2048,
taskManagerMemoryMB=1728, slotsPerTaskManager=2}2020-11-13 18:05:52,932 WARN
org.apache.hadoop.hdfs.shortcircuit.DomainSocketFactory [] - The
short-circuit local reads feature cannot be used because libhadoop cannot be
loaded.2020-11-13 18:05:55,076 INFO
org.apache.flink.yarn.YarnClusterDescriptor [] - Submitting
application master application_1599741232083_220112020-11-13 18:05:55,307 INFO
org.apache.hadoop.yarn.client.api.impl.YarnClientImpl [] - Submitted
application application_1599741232083_220112020-11-13 18:05:55,308 INFO
org.apache.flink.yarn.YarnClusterDescriptor [] - Waiting for
the cluster to be allocated2020-11-13 18:05:55,310 INFO
org.apache.flink.yarn.YarnClusterDescriptor [] - Deploying
cluster, current state ACCEPTED
The program
finished with the following exception:
org.apache.flink.client.program.ProgramInvocationException: The main method
caused an error: Could not deploy Yarn job cluster. at
org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:330)
at
org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:198)
at org.apache.flink.client.ClientUtils.executeProgram(ClientUtils.java:114) at
org.apache.flink.client.cli.CliFrontend.executeProgram(CliFrontend.java:743) at
org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:242) at
org.apache.flink.client.cli.CliFrontend.parseAndRun(CliFrontend.java
[jira] [Updated] (FLINK-20143) use `yarn.provided.lib.dirs` config deploy job failed in yarn per job mode
[
https://issues.apache.org/jira/browse/FLINK-20143?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
zhisheng updated FLINK-20143:
-
Description:
use follow command deploy flink job to yarn failed
{code:java}
./bin/flink run -m yarn-cluster -d -ynm flink-1.12-test -ytm 3g -yjm 3g -yD
yarn.provided.lib.dirs=hdfs:///flink/flink-1.12-SNAPSHOT/lib
./examples/streaming/StateMachineExample.jar
{code}
log:
{code:java}
$ ./bin/flink run -m yarn-cluster -d -ynm flink-1.12-test -ytm 3g -yjm 3g -yD
yarn.provided.lib.dirs=hdfs:///flink/flink-1.12-SNAPSHOT/lib
./examples/streaming/StateMachineExample.jar$ ./bin/flink run -m yarn-cluster
-d -ynm flink-1.12-test -ytm 3g -yjm 3g -yD
yarn.provided.lib.dirs=hdfs:///flink/flink-1.12-SNAPSHOT/lib
./examples/streaming/StateMachineExample.jarSLF4J: Class path contains multiple
SLF4J bindings.SLF4J: Found binding in
[jar:file:/data1/app/flink-1.12-SNAPSHOT/lib/log4j-slf4j-impl-2.12.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J:
Found binding in
[jar:file:/data1/app/hadoop-2.7.3-snappy-32core12disk/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J:
Found binding in
[jar:file:/data1/app/hadoop-2.7.3-snappy-32core12disk/share/hadoop/tools/lib/hadoop-aliyun-2.9.2-jar-with-dependencies.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J:
See http://www.slf4j.org/codes.html#multiple_bindings for an
explanation.SLF4J: Actual binding is of type
[org.apache.logging.slf4j.Log4jLoggerFactory]2020-11-13 16:14:30,347 INFO
org.apache.flink.yarn.cli.FlinkYarnSessionCli [] - Dynamic
Property set:
yarn.provided.lib.dirs=hdfs:///flink/flink-1.12-SNAPSHOT/lib2020-11-13
16:14:30,347 INFO org.apache.flink.yarn.cli.FlinkYarnSessionCli
[] - Dynamic Property set:
yarn.provided.lib.dirs=hdfs:///flink/flink-1.12-SNAPSHOT/libUsage with built-in
data generator: StateMachineExample [--error-rate
] [--sleep ]Usage
with Kafka: StateMachineExample --kafka-topic [--brokers
]Options for both the above setups: [--backend ]
[--checkpoint-dir ] [--async-checkpoints ]
[--incremental-checkpoints ] [--output OR null for
stdout]
Using standalone source with error rate 0.00 and sleep delay 1 millis
2020-11-13 16:14:30,706 WARN
org.apache.flink.yarn.configuration.YarnLogConfigUtil [] - The
configuration directory ('/data1/app/flink-1.12-SNAPSHOT/conf') already
contains a LOG4J config file.If you want to use logback, then please delete or
rename the log configuration file.2020-11-13 16:14:30,947 INFO
org.apache.hadoop.yarn.client.AHSProxy [] - Connecting to
Application History server at
FAT-hadoopuat-69117.vm.dc01.tech/10.69.1.17:102002020-11-13 16:14:30,958 INFO
org.apache.flink.yarn.YarnClusterDescriptor [] - No path for
the flink jar passed. Using the location of class
org.apache.flink.yarn.YarnClusterDescriptor to locate the jar2020-11-13
16:14:31,065 INFO
org.apache.hadoop.yarn.client.ConfiguredRMFailoverProxyProvider [] - Failing
over to rm22020-11-13 16:14:31,130 INFO
org.apache.flink.yarn.YarnClusterDescriptor [] - The
configured JobManager memory is 3072 MB. YARN will allocate 4096 MB to make up
an integer multiple of its minimum allocation memory (2048 MB, configured via
'yarn.scheduler.minimum-allocation-mb'). The extra 1024 MB may not be used by
Flink.2020-11-13 16:14:31,130 INFO org.apache.flink.yarn.YarnClusterDescriptor
[] - The configured TaskManager memory is 3072 MB. YARN will
allocate 4096 MB to make up an integer multiple of its minimum allocation
memory (2048 MB, configured via 'yarn.scheduler.minimum-allocation-mb'). The
extra 1024 MB may not be used by Flink.2020-11-13 16:14:31,130 INFO
org.apache.flink.yarn.YarnClusterDescriptor [] - Cluster
specification: ClusterSpecification{masterMemoryMB=3072,
taskManagerMemoryMB=3072, slotsPerTaskManager=2}2020-11-13 16:14:31,681 WARN
org.apache.hadoop.hdfs.shortcircuit.DomainSocketFactory [] - The
short-circuit local reads feature cannot be used because libhadoop cannot be
loaded.2020-11-13 16:14:33,417 INFO
org.apache.flink.yarn.YarnClusterDescriptor [] - Submitting
application master application_1599741232083_219902020-11-13 16:14:33,446 INFO
org.apache.hadoop.yarn.client.api.impl.YarnClientImpl [] - Submitted
application application_1599741232083_219902020-11-13 16:14:33,446 INFO
org.apache.flink.yarn.YarnClusterDescriptor [] - Waiting for
the cluster to be allocated2020-11-13 16:14:33,448 INFO
org.apache.flink.yarn.YarnClusterDescriptor [] - Deploying
cluster, current state ACCEPTED
The program
finished with the following exception:
org.apache.flink.client.program.ProgramInvocationException: The main me
[jira] [Comment Edited] (FLINK-20143) use `yarn.provided.lib.dirs` config deploy job failed in yarn per job mode
[
https://issues.apache.org/jira/browse/FLINK-20143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17231346#comment-17231346
]
zhisheng edited comment on FLINK-20143 at 11/13/20, 10:10 AM:
--
{code:java}
$ ./bin/flink run -t yarn-per-job -Dexecution.attached=false
-Dyarn.provided.lib.dirs="hdfs:///flink/flink-1.12-SNAPSHOT/lib"
./examples/streaming/StateMachineExample.jar$ ./bin/flink run -t yarn-per-job
-Dexecution.attached=false
-Dyarn.provided.lib.dirs="hdfs:///flink/flink-1.12-SNAPSHOT/lib"
./examples/streaming/StateMachineExample.jar
SLF4J: Class path contains multiple SLF4J bindings.SLF4J: Found binding in
[jar:file:/data1/app/flink-1.12-SNAPSHOT/lib/log4j-slf4j-impl-2.12.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J:
Found binding in
[jar:file:/data1/app/hadoop-2.7.3-snappy-32core12disk/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J:
Found binding in
[jar:file:/data1/app/hadoop-2.7.3-snappy-32core12disk/share/hadoop/tools/lib/hadoop-aliyun-2.9.2-jar-with-dependencies.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J:
See http://www.slf4j.org/codes.html#multiple_bindings for an
explanation.SLF4J: Actual binding is of type
[org.apache.logging.slf4j.Log4jLoggerFactory]Usage with built-in data
generator: StateMachineExample [--error-rate
] [--sleep ]Usage
with Kafka: StateMachineExample --kafka-topic [--brokers
]Options for both the above setups: [--backend ]
[--checkpoint-dir ] [--async-checkpoints ]
[--incremental-checkpoints ] [--output OR null for
stdout]
Using standalone source with error rate 0.00 and sleep delay 1 millis
2020-11-13 18:05:51,974 WARN
org.apache.flink.yarn.configuration.YarnLogConfigUtil [] - The
configuration directory ('/data1/app/flink-1.12-SNAPSHOT/conf') already
contains a LOG4J config file.If you want to use logback, then please delete or
rename the log configuration file.2020-11-13 18:05:52,202 INFO
org.apache.hadoop.yarn.client.AHSProxy [] - Connecting to
Application History server at
FAT-hadoopuat-69117.vm.dc01.hellocloud.tech/10.69.1.17:102002020-11-13
18:05:52,213 INFO org.apache.flink.yarn.YarnClusterDescriptor
[] - No path for the flink jar passed. Using the location of class
org.apache.flink.yarn.YarnClusterDescriptor to locate the jar2020-11-13
18:05:52,324 INFO
org.apache.hadoop.yarn.client.ConfiguredRMFailoverProxyProvider [] - Failing
over to rm22020-11-13 18:05:52,387 INFO
org.apache.flink.yarn.YarnClusterDescriptor [] - The
configured JobManager memory is 1600 MB. YARN will allocate 2048 MB to make up
an integer multiple of its minimum allocation memory (2048 MB, configured via
'yarn.scheduler.minimum-allocation-mb'). The extra 448 MB may not be used by
Flink.2020-11-13 18:05:52,388 INFO org.apache.flink.yarn.YarnClusterDescriptor
[] - The configured TaskManager memory is 1728 MB. YARN will
allocate 2048 MB to make up an integer multiple of its minimum allocation
memory (2048 MB, configured via 'yarn.scheduler.minimum-allocation-mb'). The
extra 320 MB may not be used by Flink.2020-11-13 18:05:52,388 INFO
org.apache.flink.yarn.YarnClusterDescriptor [] - Cluster
specification: ClusterSpecification{masterMemoryMB=2048,
taskManagerMemoryMB=1728, slotsPerTaskManager=2}2020-11-13 18:05:52,932 WARN
org.apache.hadoop.hdfs.shortcircuit.DomainSocketFactory [] - The
short-circuit local reads feature cannot be used because libhadoop cannot be
loaded.2020-11-13 18:05:55,076 INFO
org.apache.flink.yarn.YarnClusterDescriptor [] - Submitting
application master application_1599741232083_220112020-11-13 18:05:55,307 INFO
org.apache.hadoop.yarn.client.api.impl.YarnClientImpl [] - Submitted
application application_1599741232083_220112020-11-13 18:05:55,308 INFO
org.apache.flink.yarn.YarnClusterDescriptor [] - Waiting for
the cluster to be allocated2020-11-13 18:05:55,310 INFO
org.apache.flink.yarn.YarnClusterDescriptor [] - Deploying
cluster, current state ACCEPTED
The program
finished with the following exception:
org.apache.flink.client.program.ProgramInvocationException: The main method
caused an error: Could not deploy Yarn job cluster. at
org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:330)
at
org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:198)
at org.apache.flink.client.ClientUtils.executeProgram(ClientUtils.java:114) at
org.apache.flink.client.cli.CliFrontend.executeProgram(CliFrontend.java:743) at
org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:242) at
org.apache.flink.client.cli.CliFrontend.parseAndRun(CliF
[jira] [Created] (FLINK-20146) Update zh connector and format dependencies link
hailong wang created FLINK-20146:
Summary: Update zh connector and format dependencies link
Key: FLINK-20146
URL: https://issues.apache.org/jira/browse/FLINK-20146
Project: Flink
Issue Type: Improvement
Components: Documentation, Table SQL / Ecosystem
Affects Versions: 1.12.0
Reporter: hailong wang
Fix For: 1.12.0
Based on https://issues.apache.org/jira/browse/FLINK-20093,
We should update zh connector and format dependencies link, such as:
`
{% assign connector = site.data.sql-connectors['xxx'] %}
{% include sql-connector-download-table.html
connector=connector
%}
`
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (FLINK-20143) use `yarn.provided.lib.dirs` config deploy job failed in yarn per job mode
[
https://issues.apache.org/jira/browse/FLINK-20143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17231346#comment-17231346
]
zhisheng edited comment on FLINK-20143 at 11/13/20, 10:12 AM:
--
{code:java}
$ ./bin/flink run -t yarn-per-job -Dexecution.attached=false
-Dyarn.provided.lib.dirs="hdfs:///flink/flink-1.12-SNAPSHOT/lib"
./examples/streaming/StateMachineExample.jar
SLF4J: Class path contains multiple SLF4J bindings.SLF4J: Found binding in
[jar:file:/data1/app/flink-1.12-SNAPSHOT/lib/log4j-slf4j-impl-2.12.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J:
Found binding in
[jar:file:/data1/app/hadoop-2.7.3-snappy-32core12disk/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J:
Found binding in
[jar:file:/data1/app/hadoop-2.7.3-snappy-32core12disk/share/hadoop/tools/lib/hadoop-aliyun-2.9.2-jar-with-dependencies.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J:
See http://www.slf4j.org/codes.html#multiple_bindings for an
explanation.SLF4J: Actual binding is of type
[org.apache.logging.slf4j.Log4jLoggerFactory]Usage with built-in data
generator: StateMachineExample [--error-rate
] [--sleep ]Usage
with Kafka: StateMachineExample --kafka-topic [--brokers
]Options for both the above setups: [--backend ]
[--checkpoint-dir ] [--async-checkpoints ]
[--incremental-checkpoints ] [--output OR null for
stdout]
Using standalone source with error rate 0.00 and sleep delay 1 millis
2020-11-13 18:05:51,974 WARN
org.apache.flink.yarn.configuration.YarnLogConfigUtil [] - The
configuration directory ('/data1/app/flink-1.12-SNAPSHOT/conf') already
contains a LOG4J config file.If you want to use logback, then please delete or
rename the log configuration file.2020-11-13 18:05:52,202 INFO
org.apache.hadoop.yarn.client.AHSProxy [] - Connecting to
Application History server at
FAT-hadoopuat-69117.vm.dc01.tech/10.69.1.17:102002020-11-13 18:05:52,213 INFO
org.apache.flink.yarn.YarnClusterDescriptor [] - No path for
the flink jar passed. Using the location of class
org.apache.flink.yarn.YarnClusterDescriptor to locate the jar2020-11-13
18:05:52,324 INFO
org.apache.hadoop.yarn.client.ConfiguredRMFailoverProxyProvider [] - Failing
over to rm22020-11-13 18:05:52,387 INFO
org.apache.flink.yarn.YarnClusterDescriptor [] - The
configured JobManager memory is 1600 MB. YARN will allocate 2048 MB to make up
an integer multiple of its minimum allocation memory (2048 MB, configured via
'yarn.scheduler.minimum-allocation-mb'). The extra 448 MB may not be used by
Flink.2020-11-13 18:05:52,388 INFO org.apache.flink.yarn.YarnClusterDescriptor
[] - The configured TaskManager memory is 1728 MB. YARN will
allocate 2048 MB to make up an integer multiple of its minimum allocation
memory (2048 MB, configured via 'yarn.scheduler.minimum-allocation-mb'). The
extra 320 MB may not be used by Flink.2020-11-13 18:05:52,388 INFO
org.apache.flink.yarn.YarnClusterDescriptor [] - Cluster
specification: ClusterSpecification{masterMemoryMB=2048,
taskManagerMemoryMB=1728, slotsPerTaskManager=2}2020-11-13 18:05:52,932 WARN
org.apache.hadoop.hdfs.shortcircuit.DomainSocketFactory [] - The
short-circuit local reads feature cannot be used because libhadoop cannot be
loaded.2020-11-13 18:05:55,076 INFO
org.apache.flink.yarn.YarnClusterDescriptor [] - Submitting
application master application_1599741232083_220112020-11-13 18:05:55,307 INFO
org.apache.hadoop.yarn.client.api.impl.YarnClientImpl [] - Submitted
application application_1599741232083_220112020-11-13 18:05:55,308 INFO
org.apache.flink.yarn.YarnClusterDescriptor [] - Waiting for
the cluster to be allocated2020-11-13 18:05:55,310 INFO
org.apache.flink.yarn.YarnClusterDescriptor [] - Deploying
cluster, current state ACCEPTED
The program
finished with the following exception:
org.apache.flink.client.program.ProgramInvocationException: The main method
caused an error: Could not deploy Yarn job cluster. at
org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:330)
at
org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:198)
at org.apache.flink.client.ClientUtils.executeProgram(ClientUtils.java:114) at
org.apache.flink.client.cli.CliFrontend.executeProgram(CliFrontend.java:743) at
org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:242) at
org.apache.flink.client.cli.CliFrontend.parseAndRun(CliFrontend.java:971) at
org.apache.flink.client.cli.CliFrontend.lambda$main$10(CliFrontend.java:1047)
at java.security.AccessController.doPrivileged(Native Method) at
javax.security.au
[jira] [Updated] (FLINK-20145) Streaming job fails with IllegalStateException: Should only poll priority events
[
https://issues.apache.org/jira/browse/FLINK-20145?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Metzger updated FLINK-20145:
---
Priority: Blocker (was: Critical)
> Streaming job fails with IllegalStateException: Should only poll priority
> events
>
>
> Key: FLINK-20145
> URL: https://issues.apache.org/jira/browse/FLINK-20145
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Checkpointing, Runtime / Network
>Affects Versions: 1.12.0
>Reporter: Robert Metzger
>Assignee: Arvid Heise
>Priority: Blocker
> Fix For: 1.12.0
>
>
> While testing the 1.12 release, I came across the following failure cause:
> {code}
> 2020-11-13 09:41:52,110 WARN org.apache.flink.runtime.taskmanager.Task
> [] - dynamic filter (3/4)#0 (b977944851531f96e5324e786f055eb7)
> switched from RUNNING to FAILED.
> java.lang.IllegalStateException: Should only poll priority events
> at
> org.apache.flink.util.Preconditions.checkState(Preconditions.java:198)
> ~[flink-dist_2.11-1.12.0.jar:1.12.0]
> at
> org.apache.flink.streaming.runtime.io.CheckpointedInputGate.processPriorityEvents(CheckpointedInputGate.java:116)
> ~[flink-dist_2.11-1.12.0.jar:1.12.0]
> at
> org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$1.runThrowing(StreamTaskActionExecutor.java:47)
> ~[flink-dist_2.11-1.12.0.jar:1.12.0]
> at
> org.apache.flink.streaming.runtime.tasks.mailbox.Mail.run(Mail.java:78)
> ~[flink-dist_2.11-1.12.0.jar:1.12.0]
> at
> org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.processMail(MailboxProcessor.java:283)
> ~[flink-dist_2.11-1.12.0.jar:1.12.0]
> at
> org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:184)
> ~[flink-dist_2.11-1.12.0.jar:1.12.0]
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:577)
> ~[flink-dist_2.11-1.12.0.jar:1.12.0]
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:541)
> ~[flink-dist_2.11-1.12.0.jar:1.12.0]
> at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:722)
> [flink-dist_2.11-1.12.0.jar:1.12.0]
> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:547)
> [flink-dist_2.11-1.12.0.jar:1.12.0]
> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_222]
> {code}
> I have unaligned checkpointing enabled, the failing operator is a
> CoFlatMapFunction. The error happend on all four TaskManagers, very soon
> after job submission. The error doesn't happen when unaligned checkpointing
> is disabled.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-20109) java.lang.ClassNotFoundException: org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumer
[ https://issues.apache.org/jira/browse/FLINK-20109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17231354#comment-17231354 ] xuhaoran commented on FLINK-20109: -- try to restart the sql client or restart the cluster after add the jar,still Could not execute SQL statement. Reason: java.lang.ClassNotFoundException: org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumer[~zhisheng] > java.lang.ClassNotFoundException: > org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumer > > > Key: FLINK-20109 > URL: https://issues.apache.org/jira/browse/FLINK-20109 > Project: Flink > Issue Type: Bug > Components: Client / Job Submission >Affects Versions: 1.11.2 > Environment: macos 10.14.6 > flink 1.11.2 >Reporter: xuhaoran >Priority: Major > > 按照官网例子 加载Kafka的连接器需要下载flink-sql-connector-kafka_2.11-1.11.2.jar > 然后启动指令为./sql-client.sh embedded -lib xxx/ > 然后创建简单的sql语句 创建 Kafka table > 执行都ok > > 但如果flink-sql-connector-kafka_2.11-1.11.2.jar 放在classpath里面 启动./sql-client.sh > embedded ps -ef|grep java > 也可以看到当前classpath加载了flink-sql-connector-kafka_2.11-1.11.2.jar > 执行之前的创建kafka table 执行select * from table > 报错java.lang.ClassNotFoundException: > org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumer > 怀疑类加载器有问题 > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on pull request #14049: [FLINK-20043][kinesis] Add flink-sql-connector-kinesis package
flinkbot edited a comment on pull request #14049: URL: https://github.com/apache/flink/pull/14049#issuecomment-726094991 ## CI report: * 0a068c0f16865a2c91819d21f52d759151acb97e UNKNOWN * d15f6e11ca8d46da440ed6c2235f73937d70ab3a UNKNOWN * 1a9e4dcb885484aa800a040f1a4674cd5509f7b4 Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9523) Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9525) * 70bd4cfcc42ad54183c03aec2d763a2059fc4b1b Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9559) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14064: [FLINK-19454][docs-zh] Translate page 'Importing Flink into an IDE' into Chinese
flinkbot edited a comment on pull request #14064: URL: https://github.com/apache/flink/pull/14064#issuecomment-726642209 ## CI report: * 41bcdb30f6837f2f22dda55edafe8ba9ef5e98be Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9556) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-20143) use `yarn.provided.lib.dirs` config deploy job failed in yarn per job mode
[
https://issues.apache.org/jira/browse/FLINK-20143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17231360#comment-17231360
]
Kostas Kloudas commented on FLINK-20143:
Can't you use {{yarn logs -applicationId application_1599741232083_22011}} to
get the logs as the message says?
> use `yarn.provided.lib.dirs` config deploy job failed in yarn per job mode
> --
>
> Key: FLINK-20143
> URL: https://issues.apache.org/jira/browse/FLINK-20143
> Project: Flink
> Issue Type: Bug
> Components: Client / Job Submission, Deployment / YARN
>Affects Versions: 1.12.0
>Reporter: zhisheng
>Priority: Major
> Attachments: image-2020-11-13-17-21-47-751.png,
> image-2020-11-13-17-22-06-111.png
>
>
> use follow command deploy flink job to yarn failed
> {code:java}
> ./bin/flink run -m yarn-cluster -d -ynm flink-1.12-test -ytm 3g -yjm 3g -yD
> yarn.provided.lib.dirs=hdfs:///flink/flink-1.12-SNAPSHOT/lib
> ./examples/streaming/StateMachineExample.jar
> {code}
> log:
> {code:java}
> $ ./bin/flink run -m yarn-cluster -d -ynm flink-1.12-test -ytm 3g -yjm 3g -yD
> yarn.provided.lib.dirs=hdfs:///flink/flink-1.12-SNAPSHOT/lib
> ./examples/streaming/StateMachineExample.jar$ ./bin/flink run -m yarn-cluster
> -d -ynm flink-1.12-test -ytm 3g -yjm 3g -yD
> yarn.provided.lib.dirs=hdfs:///flink/flink-1.12-SNAPSHOT/lib
> ./examples/streaming/StateMachineExample.jarSLF4J: Class path contains
> multiple SLF4J bindings.SLF4J: Found binding in
> [jar:file:/data1/app/flink-1.12-SNAPSHOT/lib/log4j-slf4j-impl-2.12.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J:
> Found binding in
> [jar:file:/data1/app/hadoop-2.7.3-snappy-32core12disk/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J:
> Found binding in
> [jar:file:/data1/app/hadoop-2.7.3-snappy-32core12disk/share/hadoop/tools/lib/hadoop-aliyun-2.9.2-jar-with-dependencies.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J:
> See http://www.slf4j.org/codes.html#multiple_bindings for an
> explanation.SLF4J: Actual binding is of type
> [org.apache.logging.slf4j.Log4jLoggerFactory]2020-11-13 16:14:30,347 INFO
> org.apache.flink.yarn.cli.FlinkYarnSessionCli [] - Dynamic
> Property set:
> yarn.provided.lib.dirs=hdfs:///flink/flink-1.12-SNAPSHOT/lib2020-11-13
> 16:14:30,347 INFO org.apache.flink.yarn.cli.FlinkYarnSessionCli
> [] - Dynamic Property set:
> yarn.provided.lib.dirs=hdfs:///flink/flink-1.12-SNAPSHOT/libUsage with
> built-in data generator: StateMachineExample [--error-rate
> ] [--sleep ]Usage
> with Kafka: StateMachineExample --kafka-topic [--brokers
> ]Options for both the above setups: [--backend ]
> [--checkpoint-dir ] [--async-checkpoints ]
> [--incremental-checkpoints ] [--output OR null for
> stdout]
> Using standalone source with error rate 0.00 and sleep delay 1 millis
> 2020-11-13 16:14:30,706 WARN
> org.apache.flink.yarn.configuration.YarnLogConfigUtil [] - The
> configuration directory ('/data1/app/flink-1.12-SNAPSHOT/conf') already
> contains a LOG4J config file.If you want to use logback, then please delete
> or rename the log configuration file.2020-11-13 16:14:30,947 INFO
> org.apache.hadoop.yarn.client.AHSProxy [] - Connecting
> to Application History server at
> FAT-hadoopuat-69117.vm.dc01.tech/10.69.1.17:102002020-11-13 16:14:30,958 INFO
> org.apache.flink.yarn.YarnClusterDescriptor [] - No path
> for the flink jar passed. Using the location of class
> org.apache.flink.yarn.YarnClusterDescriptor to locate the jar2020-11-13
> 16:14:31,065 INFO
> org.apache.hadoop.yarn.client.ConfiguredRMFailoverProxyProvider [] - Failing
> over to rm22020-11-13 16:14:31,130 INFO
> org.apache.flink.yarn.YarnClusterDescriptor [] - The
> configured JobManager memory is 3072 MB. YARN will allocate 4096 MB to make
> up an integer multiple of its minimum allocation memory (2048 MB, configured
> via 'yarn.scheduler.minimum-allocation-mb'). The extra 1024 MB may not be
> used by Flink.2020-11-13 16:14:31,130 INFO
> org.apache.flink.yarn.YarnClusterDescriptor [] - The
> configured TaskManager memory is 3072 MB. YARN will allocate 4096 MB to make
> up an integer multiple of its minimum allocation memory (2048 MB, configured
> via 'yarn.scheduler.minimum-allocation-mb'). The extra 1024 MB may not be
> used by Flink.2020-11-13 16:14:31,130 INFO
> org.apache.flink.yarn.YarnClusterDescriptor [] - Cluster
> specification: ClusterSpecification{masterMemoryMB=3072,
> taskManagerMemoryMB=3072, slotsPerTaskManager=2}2020-11-13 16:14:31,681 WARN
> org.apache.hadoop.hdfs.shortcircuit.DomainSocketFactor
[jira] [Commented] (FLINK-20143) use `yarn.provided.lib.dirs` config deploy job failed in yarn per job mode
[
https://issues.apache.org/jira/browse/FLINK-20143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17231361#comment-17231361
]
Kostas Kloudas commented on FLINK-20143:
Also could you run the same command with DEBUG logging enabled?
> use `yarn.provided.lib.dirs` config deploy job failed in yarn per job mode
> --
>
> Key: FLINK-20143
> URL: https://issues.apache.org/jira/browse/FLINK-20143
> Project: Flink
> Issue Type: Bug
> Components: Client / Job Submission, Deployment / YARN
>Affects Versions: 1.12.0
>Reporter: zhisheng
>Priority: Major
> Attachments: image-2020-11-13-17-21-47-751.png,
> image-2020-11-13-17-22-06-111.png
>
>
> use follow command deploy flink job to yarn failed
> {code:java}
> ./bin/flink run -m yarn-cluster -d -ynm flink-1.12-test -ytm 3g -yjm 3g -yD
> yarn.provided.lib.dirs=hdfs:///flink/flink-1.12-SNAPSHOT/lib
> ./examples/streaming/StateMachineExample.jar
> {code}
> log:
> {code:java}
> $ ./bin/flink run -m yarn-cluster -d -ynm flink-1.12-test -ytm 3g -yjm 3g -yD
> yarn.provided.lib.dirs=hdfs:///flink/flink-1.12-SNAPSHOT/lib
> ./examples/streaming/StateMachineExample.jar$ ./bin/flink run -m yarn-cluster
> -d -ynm flink-1.12-test -ytm 3g -yjm 3g -yD
> yarn.provided.lib.dirs=hdfs:///flink/flink-1.12-SNAPSHOT/lib
> ./examples/streaming/StateMachineExample.jarSLF4J: Class path contains
> multiple SLF4J bindings.SLF4J: Found binding in
> [jar:file:/data1/app/flink-1.12-SNAPSHOT/lib/log4j-slf4j-impl-2.12.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J:
> Found binding in
> [jar:file:/data1/app/hadoop-2.7.3-snappy-32core12disk/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J:
> Found binding in
> [jar:file:/data1/app/hadoop-2.7.3-snappy-32core12disk/share/hadoop/tools/lib/hadoop-aliyun-2.9.2-jar-with-dependencies.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J:
> See http://www.slf4j.org/codes.html#multiple_bindings for an
> explanation.SLF4J: Actual binding is of type
> [org.apache.logging.slf4j.Log4jLoggerFactory]2020-11-13 16:14:30,347 INFO
> org.apache.flink.yarn.cli.FlinkYarnSessionCli [] - Dynamic
> Property set:
> yarn.provided.lib.dirs=hdfs:///flink/flink-1.12-SNAPSHOT/lib2020-11-13
> 16:14:30,347 INFO org.apache.flink.yarn.cli.FlinkYarnSessionCli
> [] - Dynamic Property set:
> yarn.provided.lib.dirs=hdfs:///flink/flink-1.12-SNAPSHOT/libUsage with
> built-in data generator: StateMachineExample [--error-rate
> ] [--sleep ]Usage
> with Kafka: StateMachineExample --kafka-topic [--brokers
> ]Options for both the above setups: [--backend ]
> [--checkpoint-dir ] [--async-checkpoints ]
> [--incremental-checkpoints ] [--output OR null for
> stdout]
> Using standalone source with error rate 0.00 and sleep delay 1 millis
> 2020-11-13 16:14:30,706 WARN
> org.apache.flink.yarn.configuration.YarnLogConfigUtil [] - The
> configuration directory ('/data1/app/flink-1.12-SNAPSHOT/conf') already
> contains a LOG4J config file.If you want to use logback, then please delete
> or rename the log configuration file.2020-11-13 16:14:30,947 INFO
> org.apache.hadoop.yarn.client.AHSProxy [] - Connecting
> to Application History server at
> FAT-hadoopuat-69117.vm.dc01.tech/10.69.1.17:102002020-11-13 16:14:30,958 INFO
> org.apache.flink.yarn.YarnClusterDescriptor [] - No path
> for the flink jar passed. Using the location of class
> org.apache.flink.yarn.YarnClusterDescriptor to locate the jar2020-11-13
> 16:14:31,065 INFO
> org.apache.hadoop.yarn.client.ConfiguredRMFailoverProxyProvider [] - Failing
> over to rm22020-11-13 16:14:31,130 INFO
> org.apache.flink.yarn.YarnClusterDescriptor [] - The
> configured JobManager memory is 3072 MB. YARN will allocate 4096 MB to make
> up an integer multiple of its minimum allocation memory (2048 MB, configured
> via 'yarn.scheduler.minimum-allocation-mb'). The extra 1024 MB may not be
> used by Flink.2020-11-13 16:14:31,130 INFO
> org.apache.flink.yarn.YarnClusterDescriptor [] - The
> configured TaskManager memory is 3072 MB. YARN will allocate 4096 MB to make
> up an integer multiple of its minimum allocation memory (2048 MB, configured
> via 'yarn.scheduler.minimum-allocation-mb'). The extra 1024 MB may not be
> used by Flink.2020-11-13 16:14:31,130 INFO
> org.apache.flink.yarn.YarnClusterDescriptor [] - Cluster
> specification: ClusterSpecification{masterMemoryMB=3072,
> taskManagerMemoryMB=3072, slotsPerTaskManager=2}2020-11-13 16:14:31,681 WARN
> org.apache.hadoop.hdfs.shortcircuit.DomainSocketFactory [] - The
> short-circuit local reads feat
[jira] [Updated] (FLINK-20143) use `yarn.provided.lib.dirs` config deploy job failed in yarn per job mode
[
https://issues.apache.org/jira/browse/FLINK-20143?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
zhisheng updated FLINK-20143:
-
Attachment: image-2020-11-13-18-43-55-188.png
> use `yarn.provided.lib.dirs` config deploy job failed in yarn per job mode
> --
>
> Key: FLINK-20143
> URL: https://issues.apache.org/jira/browse/FLINK-20143
> Project: Flink
> Issue Type: Bug
> Components: Client / Job Submission, Deployment / YARN
>Affects Versions: 1.12.0
>Reporter: zhisheng
>Priority: Major
> Attachments: image-2020-11-13-17-21-47-751.png,
> image-2020-11-13-17-22-06-111.png, image-2020-11-13-18-43-55-188.png
>
>
> use follow command deploy flink job to yarn failed
> {code:java}
> ./bin/flink run -m yarn-cluster -d -ynm flink-1.12-test -ytm 3g -yjm 3g -yD
> yarn.provided.lib.dirs=hdfs:///flink/flink-1.12-SNAPSHOT/lib
> ./examples/streaming/StateMachineExample.jar
> {code}
> log:
> {code:java}
> $ ./bin/flink run -m yarn-cluster -d -ynm flink-1.12-test -ytm 3g -yjm 3g -yD
> yarn.provided.lib.dirs=hdfs:///flink/flink-1.12-SNAPSHOT/lib
> ./examples/streaming/StateMachineExample.jar$ ./bin/flink run -m yarn-cluster
> -d -ynm flink-1.12-test -ytm 3g -yjm 3g -yD
> yarn.provided.lib.dirs=hdfs:///flink/flink-1.12-SNAPSHOT/lib
> ./examples/streaming/StateMachineExample.jarSLF4J: Class path contains
> multiple SLF4J bindings.SLF4J: Found binding in
> [jar:file:/data1/app/flink-1.12-SNAPSHOT/lib/log4j-slf4j-impl-2.12.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J:
> Found binding in
> [jar:file:/data1/app/hadoop-2.7.3-snappy-32core12disk/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J:
> Found binding in
> [jar:file:/data1/app/hadoop-2.7.3-snappy-32core12disk/share/hadoop/tools/lib/hadoop-aliyun-2.9.2-jar-with-dependencies.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J:
> See http://www.slf4j.org/codes.html#multiple_bindings for an
> explanation.SLF4J: Actual binding is of type
> [org.apache.logging.slf4j.Log4jLoggerFactory]2020-11-13 16:14:30,347 INFO
> org.apache.flink.yarn.cli.FlinkYarnSessionCli [] - Dynamic
> Property set:
> yarn.provided.lib.dirs=hdfs:///flink/flink-1.12-SNAPSHOT/lib2020-11-13
> 16:14:30,347 INFO org.apache.flink.yarn.cli.FlinkYarnSessionCli
> [] - Dynamic Property set:
> yarn.provided.lib.dirs=hdfs:///flink/flink-1.12-SNAPSHOT/libUsage with
> built-in data generator: StateMachineExample [--error-rate
> ] [--sleep ]Usage
> with Kafka: StateMachineExample --kafka-topic [--brokers
> ]Options for both the above setups: [--backend ]
> [--checkpoint-dir ] [--async-checkpoints ]
> [--incremental-checkpoints ] [--output OR null for
> stdout]
> Using standalone source with error rate 0.00 and sleep delay 1 millis
> 2020-11-13 16:14:30,706 WARN
> org.apache.flink.yarn.configuration.YarnLogConfigUtil [] - The
> configuration directory ('/data1/app/flink-1.12-SNAPSHOT/conf') already
> contains a LOG4J config file.If you want to use logback, then please delete
> or rename the log configuration file.2020-11-13 16:14:30,947 INFO
> org.apache.hadoop.yarn.client.AHSProxy [] - Connecting
> to Application History server at
> FAT-hadoopuat-69117.vm.dc01.tech/10.69.1.17:102002020-11-13 16:14:30,958 INFO
> org.apache.flink.yarn.YarnClusterDescriptor [] - No path
> for the flink jar passed. Using the location of class
> org.apache.flink.yarn.YarnClusterDescriptor to locate the jar2020-11-13
> 16:14:31,065 INFO
> org.apache.hadoop.yarn.client.ConfiguredRMFailoverProxyProvider [] - Failing
> over to rm22020-11-13 16:14:31,130 INFO
> org.apache.flink.yarn.YarnClusterDescriptor [] - The
> configured JobManager memory is 3072 MB. YARN will allocate 4096 MB to make
> up an integer multiple of its minimum allocation memory (2048 MB, configured
> via 'yarn.scheduler.minimum-allocation-mb'). The extra 1024 MB may not be
> used by Flink.2020-11-13 16:14:31,130 INFO
> org.apache.flink.yarn.YarnClusterDescriptor [] - The
> configured TaskManager memory is 3072 MB. YARN will allocate 4096 MB to make
> up an integer multiple of its minimum allocation memory (2048 MB, configured
> via 'yarn.scheduler.minimum-allocation-mb'). The extra 1024 MB may not be
> used by Flink.2020-11-13 16:14:31,130 INFO
> org.apache.flink.yarn.YarnClusterDescriptor [] - Cluster
> specification: ClusterSpecification{masterMemoryMB=3072,
> taskManagerMemoryMB=3072, slotsPerTaskManager=2}2020-11-13 16:14:31,681 WARN
> org.apache.hadoop.hdfs.shortcircuit.DomainSocketFactory [] - The
> short-circuit local reads feature cannot be used because libhadoop cannot be
>
[jira] [Commented] (FLINK-20143) use `yarn.provided.lib.dirs` config deploy job failed in yarn per job mode
[
https://issues.apache.org/jira/browse/FLINK-20143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17231365#comment-17231365
]
zhisheng commented on FLINK-20143:
--
!image-2020-11-13-18-43-55-188.png!
does not has any log, i had say just now;)
> use `yarn.provided.lib.dirs` config deploy job failed in yarn per job mode
> --
>
> Key: FLINK-20143
> URL: https://issues.apache.org/jira/browse/FLINK-20143
> Project: Flink
> Issue Type: Bug
> Components: Client / Job Submission, Deployment / YARN
>Affects Versions: 1.12.0
>Reporter: zhisheng
>Priority: Major
> Attachments: image-2020-11-13-17-21-47-751.png,
> image-2020-11-13-17-22-06-111.png, image-2020-11-13-18-43-55-188.png
>
>
> use follow command deploy flink job to yarn failed
> {code:java}
> ./bin/flink run -m yarn-cluster -d -ynm flink-1.12-test -ytm 3g -yjm 3g -yD
> yarn.provided.lib.dirs=hdfs:///flink/flink-1.12-SNAPSHOT/lib
> ./examples/streaming/StateMachineExample.jar
> {code}
> log:
> {code:java}
> $ ./bin/flink run -m yarn-cluster -d -ynm flink-1.12-test -ytm 3g -yjm 3g -yD
> yarn.provided.lib.dirs=hdfs:///flink/flink-1.12-SNAPSHOT/lib
> ./examples/streaming/StateMachineExample.jar$ ./bin/flink run -m yarn-cluster
> -d -ynm flink-1.12-test -ytm 3g -yjm 3g -yD
> yarn.provided.lib.dirs=hdfs:///flink/flink-1.12-SNAPSHOT/lib
> ./examples/streaming/StateMachineExample.jarSLF4J: Class path contains
> multiple SLF4J bindings.SLF4J: Found binding in
> [jar:file:/data1/app/flink-1.12-SNAPSHOT/lib/log4j-slf4j-impl-2.12.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J:
> Found binding in
> [jar:file:/data1/app/hadoop-2.7.3-snappy-32core12disk/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J:
> Found binding in
> [jar:file:/data1/app/hadoop-2.7.3-snappy-32core12disk/share/hadoop/tools/lib/hadoop-aliyun-2.9.2-jar-with-dependencies.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J:
> See http://www.slf4j.org/codes.html#multiple_bindings for an
> explanation.SLF4J: Actual binding is of type
> [org.apache.logging.slf4j.Log4jLoggerFactory]2020-11-13 16:14:30,347 INFO
> org.apache.flink.yarn.cli.FlinkYarnSessionCli [] - Dynamic
> Property set:
> yarn.provided.lib.dirs=hdfs:///flink/flink-1.12-SNAPSHOT/lib2020-11-13
> 16:14:30,347 INFO org.apache.flink.yarn.cli.FlinkYarnSessionCli
> [] - Dynamic Property set:
> yarn.provided.lib.dirs=hdfs:///flink/flink-1.12-SNAPSHOT/libUsage with
> built-in data generator: StateMachineExample [--error-rate
> ] [--sleep ]Usage
> with Kafka: StateMachineExample --kafka-topic [--brokers
> ]Options for both the above setups: [--backend ]
> [--checkpoint-dir ] [--async-checkpoints ]
> [--incremental-checkpoints ] [--output OR null for
> stdout]
> Using standalone source with error rate 0.00 and sleep delay 1 millis
> 2020-11-13 16:14:30,706 WARN
> org.apache.flink.yarn.configuration.YarnLogConfigUtil [] - The
> configuration directory ('/data1/app/flink-1.12-SNAPSHOT/conf') already
> contains a LOG4J config file.If you want to use logback, then please delete
> or rename the log configuration file.2020-11-13 16:14:30,947 INFO
> org.apache.hadoop.yarn.client.AHSProxy [] - Connecting
> to Application History server at
> FAT-hadoopuat-69117.vm.dc01.tech/10.69.1.17:102002020-11-13 16:14:30,958 INFO
> org.apache.flink.yarn.YarnClusterDescriptor [] - No path
> for the flink jar passed. Using the location of class
> org.apache.flink.yarn.YarnClusterDescriptor to locate the jar2020-11-13
> 16:14:31,065 INFO
> org.apache.hadoop.yarn.client.ConfiguredRMFailoverProxyProvider [] - Failing
> over to rm22020-11-13 16:14:31,130 INFO
> org.apache.flink.yarn.YarnClusterDescriptor [] - The
> configured JobManager memory is 3072 MB. YARN will allocate 4096 MB to make
> up an integer multiple of its minimum allocation memory (2048 MB, configured
> via 'yarn.scheduler.minimum-allocation-mb'). The extra 1024 MB may not be
> used by Flink.2020-11-13 16:14:31,130 INFO
> org.apache.flink.yarn.YarnClusterDescriptor [] - The
> configured TaskManager memory is 3072 MB. YARN will allocate 4096 MB to make
> up an integer multiple of its minimum allocation memory (2048 MB, configured
> via 'yarn.scheduler.minimum-allocation-mb'). The extra 1024 MB may not be
> used by Flink.2020-11-13 16:14:31,130 INFO
> org.apache.flink.yarn.YarnClusterDescriptor [] - Cluster
> specification: ClusterSpecification{masterMemoryMB=3072,
> taskManagerMemoryMB=3072, slotsPerTaskManager=2}2020-11-13 16:14:31,681 WARN
> org.apache.hadoop.hdfs.shortcircuit.DomainSocketFactory
[jira] [Commented] (FLINK-20109) java.lang.ClassNotFoundException: org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumer
[ https://issues.apache.org/jira/browse/FLINK-20109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17231368#comment-17231368 ] xuhaoran commented on FLINK-20109: -- sorry [~hailong wang] [https://ci.apache.org/projects/flink/flink-docs-release-1.11/dev/table/sqlClient.html] I dont do like this . start client like bin/sql-client.sh embedded, but I put the jar flink-sql-connector-kafka_2.11-1.11.2.jar in the lib,and I try to restart cluster , ps-ef |grep flink I see /Library/Java/JavaVirtualMachines/jdk1.8.0_201.jdk/Contents/Home/bin/java -Xmx1073741824 -Xms1073741824 -XX:MaxMetaspaceSize=268435456 -Dlog.file=/usr/local/Cellar/apache-flink/1.11.2/libexec/log/flink-xuhaoran-standalonesession-0-xuhaorandeMacBook-Pro.local.log -Dlog4j.configuration=file:/usr/local/Cellar/apache-flink/1.11.2/libexec/conf/log4j.properties -Dlog4j.configurationFile=file:/usr/local/Cellar/apache-flink/1.11.2/libexec/conf/log4j.properties -Dlogback.configurationFile=file:/usr/local/Cellar/apache-flink/1.11.2/libexec/conf/logback.xml -classpath /usr/local/Cellar/apache-flink/1.11.2/libexec/lib/flink-connector-jdbc_2.11-1.11.2.jar:/usr/local/Cellar/apache-flink/1.11.2/libexec/lib/flink-csv-1.11.2.jar:/usr/local/Cellar/apache-flink/1.11.2/libexec/lib/flink-json-1.11.2.jar:/usr/local/Cellar/apache-flink/1.11.2/libexec/lib/flink-shaded-zookeeper-3.4.14.jar:/usr/local/Cellar/apache-flink/1.11.2/libexec/lib/flink-sql-connector-kafka_2.11-1.11.2.jar:/usr/local/Cellar/apache-flink/1.11.2/libexec/lib/flink-table-blink_2.12-1.11.2.jar:/usr/local/Cellar/apache-flink/1.11.2/libexec/lib/flink-table_2.12-1.11.2.jar:/usr/local/Cellar/apache-flink/1.11.2/libexec/lib/log4j-1.2-api-2.12.1.jar:/usr/local/Cellar/apache-flink/1.11.2/libexec/lib/log4j-api-2.12.1.jar:/usr/local/Cellar/apache-flink/1.11.2/libexec/lib/log4j-core-2.12.1.jar:/usr/local/Cellar/apache-flink/1.11.2/libexec/lib/log4j-slf4j-impl-2.12.1.jar:/usr/local/Cellar/apache-flink/1.11.2/libexec/lib/flink-dist_2.12-1.11.2.jar::: org.apache.flink.runtime.entrypoint.StandaloneSessionClusterEntrypoint --configDir /usr/local/Cellar/apache-flink/1.11.2/libexec/conf --executionMode cluster I think the jar should be avalible flink-sql-connector-kafka_2.11-1.11.2.jar then I start the cli /Library/Java/JavaVirtualMachines/jdk1.8.0_201.jdk/Contents/Home/bin/java -Dlog.file=/usr/local/Cellar/apache-flink/1.11.2/libexec/log/flink-xuhaoran-sql-client-xuhaorandeMacBook-Pro.local.log -Dlog4j.configuration=file:/usr/local/Cellar/apache-flink/1.11.2/libexec/conf/log4j-cli.properties -Dlog4j.configurationFile=file:/usr/local/Cellar/apache-flink/1.11.2/libexec/conf/log4j-cli.properties -Dlogback.configurationFile=file:/usr/local/Cellar/apache-flink/1.11.2/libexec/conf/logback.xml -classpath /usr/local/Cellar/apache-flink/1.11.2/libexec/lib/flink-connector-jdbc_2.11-1.11.2.jar:/usr/local/Cellar/apache-flink/1.11.2/libexec/lib/flink-csv-1.11.2.jar:/usr/local/Cellar/apache-flink/1.11.2/libexec/lib/flink-json-1.11.2.jar:/usr/local/Cellar/apache-flink/1.11.2/libexec/lib/flink-shaded-zookeeper-3.4.14.jar:/usr/local/Cellar/apache-flink/1.11.2/libexec/lib/flink-sql-connector-kafka_2.11-1.11.2.jar:/usr/local/Cellar/apache-flink/1.11.2/libexec/lib/flink-table-blink_2.12-1.11.2.jar:/usr/local/Cellar/apache-flink/1.11.2/libexec/lib/flink-table_2.12-1.11.2.jar:/usr/local/Cellar/apache-flink/1.11.2/libexec/lib/log4j-1.2-api-2.12.1.jar:/usr/local/Cellar/apache-flink/1.11.2/libexec/lib/log4j-api-2.12.1.jar:/usr/local/Cellar/apache-flink/1.11.2/libexec/lib/log4j-core-2.12.1.jar:/usr/local/Cellar/apache-flink/1.11.2/libexec/lib/log4j-slf4j-impl-2.12.1.jar:/usr/local/Cellar/apache-flink/1.11.2/libexec/lib/flink-dist_2.12-1.11.2.jar:/usr/local/Cellar/apache-flink/1.11.2/libexec/opt/flink-python_2.12-1.11.2.jar/usr/local/Cellar/apache-flink/1.11.2/libexec/opt/flink-sql-client_2.12-1.11.2.jar org.apache.flink.table.client.SqlClient embedded --jar /usr/local/Cellar/apache-flink/1.11.2/libexec/opt/flink-sql-client_2.12-1.11.2.jar you can still see flink-sql-connector-kafka_2.11-1.11.2.jar in the classpath But when I create kafka table ,and just try to select * from table , Could not execute SQL statement. Reason: java.lang.ClassNotFoundException: org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumer I vim the jar ,the class is in the flink-sql-connector-kafka_2.11-1.11.2.jar > java.lang.ClassNotFoundException: > org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumer > > > Key: FLINK-20109 > URL: https://issues.apache.org/jira/browse/FLINK-20109 > Project: Flink > Issue Type: Bug > Components: Client / Job Submission >Affects Versions: 1.11.2 > Environment: macos 10.14.6 > flink 1.11.2 >Reporter: xuhaoran
[jira] [Commented] (FLINK-20143) use `yarn.provided.lib.dirs` config deploy job failed in yarn per job mode
[
https://issues.apache.org/jira/browse/FLINK-20143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17231369#comment-17231369
]
zhisheng commented on FLINK-20143:
--
{code:java}
22020-11-13 18:46:43,014 INFO org.apache.flink.client.cli.CliFrontend
[] -
2020-11-13
18:46:43,014 INFO org.apache.flink.client.cli.CliFrontend
[] -
2020-11-13
18:46:43,019 INFO org.apache.flink.client.cli.CliFrontend
[] - Starting Command Line Client (Version: 1.12-SNAPSHOT, Scala: 2.11,
Rev:c55420b, Date:2020-11-05T05:29:49+01:00)2020-11-13 18:46:43,019 INFO
org.apache.flink.client.cli.CliFrontend [] - OS current
user: deploy2020-11-13 18:46:43,415 INFO
org.apache.flink.client.cli.CliFrontend [] - Current
Hadoop/Kerberos user: deploy2020-11-13 18:46:43,416 INFO
org.apache.flink.client.cli.CliFrontend [] - JVM: Java
HotSpot(TM) 64-Bit Server VM - Oracle Corporation - 1.8/25.92-b142020-11-13
18:46:43,416 INFO org.apache.flink.client.cli.CliFrontend
[] - Maximum heap size: 7136 MiBytes2020-11-13 18:46:43,416 INFO
org.apache.flink.client.cli.CliFrontend [] - JAVA_HOME:
/app/jdk/2020-11-13 18:46:43,418 INFO org.apache.flink.client.cli.CliFrontend
[] - Hadoop version: 2.7.32020-11-13 18:46:43,418 INFO
org.apache.flink.client.cli.CliFrontend [] - JVM
Options:2020-11-13 18:46:43,418 INFO org.apache.flink.client.cli.CliFrontend
[] -
-Dlog.file=/data1/app/flink-1.12-SNAPSHOT/log/flink-deploy-client-FAT-hadoopuat-69120.vm.dc01.
.tech.log2020-11-13 18:46:43,418 INFO org.apache.flink.client.cli.CliFrontend
[] -
-Dlog4j.configuration=file:/data1/app/flink-1.12-SNAPSHOT/conf/log4j-cli.properties2020-11-13
18:46:43,418 INFO org.apache.flink.client.cli.CliFrontend
[] -
-Dlog4j.configurationFile=file:/data1/app/flink-1.12-SNAPSHOT/conf/log4j-cli.properties2020-11-13
18:46:43,418 INFO org.apache.flink.client.cli.CliFrontend
[] -
-Dlogback.configurationFile=file:/data1/app/flink-1.12-SNAPSHOT/conf/logback.xml2020-11-13
18:46:43,419 INFO org.apache.flink.client.cli.CliFrontend
[] - Program Arguments:2020-11-13 18:46:43,420 INFO
org.apache.flink.client.cli.CliFrontend [] -
run2020-11-13 18:46:43,420 INFO org.apache.flink.client.cli.CliFrontend
[] - -t2020-11-13 18:46:43,421 INFO
org.apache.flink.client.cli.CliFrontend [] -
yarn-per-job2020-11-13 18:46:43,421 INFO
org.apache.flink.client.cli.CliFrontend [] -
-Dexecution.attached=false2020-11-13 18:46:43,421 INFO
org.apache.flink.client.cli.CliFrontend [] -
-Dyarn.provided.lib.dirs=hdfs:///flink/flink-1.12-SNAPSHOT/lib2020-11-13
18:46:43,421 INFO org.apache.flink.client.cli.CliFrontend
[] - ./examples/streaming/StateMachineExample.jar2020-11-13 18:46:43,421
INFO org.apache.flink.client.cli.CliFrontend [] -
Classpath:
/data1/app/flink-1.12-SNAPSHOT/lib/flink-connector-jdbc_2.11-1.12-SNAPSHOT.jar:/data1/app/flink-1.12-SNAPSHOT/lib/flink-csv-1.12-SNAPSHOT.jar:/data1/app/flink-1.12-SNAPSHOT/lib/flink-json-1.12-SNAPSHOT.jar:/data1/app/flink-1.12-SNAPSHOT/lib/flink-shaded-zookeeper-3.4.14.jar:/data1/app/flink-1.12-SNAPSHOT/lib/flink-sql-connector-elasticsearch7_2.11-1.12-SNAPSHOT.jar:/data1/app/flink-1.12-SNAPSHOT/lib/flink-sql-connector-kafka_2.11-1.12-SNAPSHOT.jar:/data1/app/flink-1.12-SNAPSHOT/lib/flink-table_2.11-1.12-SNAPSHOT.jar:/data1/app/flink-1.12-SNAPSHOT/lib/flink-table-blink_2.11-1.12-SNAPSHOT.jar:/data1/app/flink-1.12-SNAPSHOT/lib/log4j-1.2-api-2.12.1.jar:/data1/app/flink-1.12-SNAPSHOT/lib/log4j-api-2.12.1.jar:/data1/app/flink-1.12-SNAPSHOT/lib/log4j-core-2.12.1.jar:/data1/app/flink-1.12-SNAPSHOT/lib/log4j-slf4j-impl-2.12.1.jar:/data1/app/flink-1.12-SNAPSHOT/lib/flink-dist_2.11-1.12-SNAPSHOT.jar:/app/hadoop/etc/hadoop:/app/hadoop/share/hadoop/common/lib/log4j-1.2.17.jar:/app/hadoop/share/hadoop/common/lib/jsr305-3.0.0.jar:/app/hadoop/share/hadoop/common/lib/jackson-core-asl-1.9.13.jar:/app/hadoop/share/hadoop/common/lib/commons-httpclient-3.1.jar:/app/hadoop/share/hadoop/common/lib/jackson-xc-1.9.13.jar:/app/hadoop/share/hadoop/common/lib/servlet-api-2.5.jar:/app/hadoop/share/hadoop/common/lib/httpcore-4.2.5.jar:/app/hadoop/share/hadoop/common/lib/jettison-1.1.jar:/app/hadoop/share/hadoop/common/lib/stax-api-1.0-2.jar:/app/hadoop/share/hadoop/common/lib/jsch-0.1.42.jar:/app/hadoop/share/h
[jira] [Comment Edited] (FLINK-20143) use `yarn.provided.lib.dirs` config deploy job failed in yarn per job mode
[
https://issues.apache.org/jira/browse/FLINK-20143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17231369#comment-17231369
]
zhisheng edited comment on FLINK-20143 at 11/13/20, 10:51 AM:
--
{code:java}
22020-11-13 18:46:43,014 INFO org.apache.flink.client.cli.CliFrontend
[] -
2020-11-13
18:46:43,014 INFO org.apache.flink.client.cli.CliFrontend
[] -
2020-11-13
18:46:43,019 INFO org.apache.flink.client.cli.CliFrontend
[] - Starting Command Line Client (Version: 1.12-SNAPSHOT, Scala: 2.11,
Rev:c55420b, Date:2020-11-05T05:29:49+01:00)2020-11-13 18:46:43,019 INFO
org.apache.flink.client.cli.CliFrontend [] - OS current
user: deploy2020-11-13 18:46:43,415 INFO
org.apache.flink.client.cli.CliFrontend [] - Current
Hadoop/Kerberos user: deploy2020-11-13 18:46:43,416 INFO
org.apache.flink.client.cli.CliFrontend [] - JVM: Java
HotSpot(TM) 64-Bit Server VM - Oracle Corporation - 1.8/25.92-b142020-11-13
18:46:43,416 INFO org.apache.flink.client.cli.CliFrontend
[] - Maximum heap size: 7136 MiBytes2020-11-13 18:46:43,416 INFO
org.apache.flink.client.cli.CliFrontend [] - JAVA_HOME:
/app/jdk/2020-11-13 18:46:43,418 INFO org.apache.flink.client.cli.CliFrontend
[] - Hadoop version: 2.7.32020-11-13 18:46:43,418 INFO
org.apache.flink.client.cli.CliFrontend [] - JVM
Options:2020-11-13 18:46:43,418 INFO org.apache.flink.client.cli.CliFrontend
[] -
-Dlog.file=/data1/app/flink-1.12-SNAPSHOT/log/flink-deploy-client-FAT-hadoopuat-69120.vm.dc01.
.tech.log2020-11-13 18:46:43,418 INFO org.apache.flink.client.cli.CliFrontend
[] -
-Dlog4j.configuration=file:/data1/app/flink-1.12-SNAPSHOT/conf/log4j-cli.properties2020-11-13
18:46:43,418 INFO org.apache.flink.client.cli.CliFrontend
[] -
-Dlog4j.configurationFile=file:/data1/app/flink-1.12-SNAPSHOT/conf/log4j-cli.properties2020-11-13
18:46:43,418 INFO org.apache.flink.client.cli.CliFrontend
[] -
-Dlogback.configurationFile=file:/data1/app/flink-1.12-SNAPSHOT/conf/logback.xml2020-11-13
18:46:43,419 INFO org.apache.flink.client.cli.CliFrontend
[] - Program Arguments:2020-11-13 18:46:43,420 INFO
org.apache.flink.client.cli.CliFrontend [] -
run2020-11-13 18:46:43,420 INFO org.apache.flink.client.cli.CliFrontend
[] - -t2020-11-13 18:46:43,421 INFO
org.apache.flink.client.cli.CliFrontend [] -
yarn-per-job2020-11-13 18:46:43,421 INFO
org.apache.flink.client.cli.CliFrontend [] -
-Dexecution.attached=false2020-11-13 18:46:43,421 INFO
org.apache.flink.client.cli.CliFrontend [] -
-Dyarn.provided.lib.dirs=hdfs:///flink/flink-1.12-SNAPSHOT/lib2020-11-13
18:46:43,421 INFO org.apache.flink.client.cli.CliFrontend
[] - ./examples/streaming/StateMachineExample.jar2020-11-13 18:46:43,421
INFO org.apache.flink.client.cli.CliFrontend [] -
Classpath:
/data1/app/flink-1.12-SNAPSHOT/lib/flink-connector-jdbc_2.11-1.12-SNAPSHOT.jar:/data1/app/flink-1.12-SNAPSHOT/lib/flink-csv-1.12-SNAPSHOT.jar:/data1/app/flink-1.12-SNAPSHOT/lib/flink-json-1.12-SNAPSHOT.jar:/data1/app/flink-1.12-SNAPSHOT/lib/flink-shaded-zookeeper-3.4.14.jar:/data1/app/flink-1.12-SNAPSHOT/lib/flink-sql-connector-elasticsearch7_2.11-1.12-SNAPSHOT.jar:/data1/app/flink-1.12-SNAPSHOT/lib/flink-sql-connector-kafka_2.11-1.12-SNAPSHOT.jar:/data1/app/flink-1.12-SNAPSHOT/lib/flink-table_2.11-1.12-SNAPSHOT.jar:/data1/app/flink-1.12-SNAPSHOT/lib/flink-table-blink_2.11-1.12-SNAPSHOT.jar:/data1/app/flink-1.12-SNAPSHOT/lib/log4j-1.2-api-2.12.1.jar:/data1/app/flink-1.12-SNAPSHOT/lib/log4j-api-2.12.1.jar:/data1/app/flink-1.12-SNAPSHOT/lib/log4j-core-2.12.1.jar:/data1/app/flink-1.12-SNAPSHOT/lib/log4j-slf4j-impl-2.12.1.jar:/data1/app/flink-1.12-SNAPSHOT/lib/flink-dist_2.11-1.12-SNAPSHOT.jar:/app/hadoop/etc/hadoop:/app/hadoop/share/hadoop/common/lib/log4j-1.2.17.jar:/app/hadoop/share/hadoop/common/lib/jsr305-3.0.0.jar:/app/hadoop/share/hadoop/common/lib/jackson-core-asl-1.9.13.jar:/app/hadoop/share/hadoop/common/lib/commons-httpclient-3.1.jar:/app/hadoop/share/hadoop/common/lib/jackson-xc-1.9.13.jar:/app/hadoop/share/hadoop/common/lib/servlet-api-2.5.jar:/app/hadoop/share/hadoop/common/lib/httpcore-4.2.5.jar:/app/hadoop/share/hadoop/common/lib/jettison-1.1.jar:/app/hadoop/share/hadoop/common/lib/stax-api-1.0-2.jar:/app/hadoop/shar
[jira] [Commented] (FLINK-20109) java.lang.ClassNotFoundException: org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumer
[ https://issues.apache.org/jira/browse/FLINK-20109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17231377#comment-17231377 ] zhisheng commented on FLINK-20109: -- could you please show you sql and ddl ？ > java.lang.ClassNotFoundException: > org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumer > > > Key: FLINK-20109 > URL: https://issues.apache.org/jira/browse/FLINK-20109 > Project: Flink > Issue Type: Bug > Components: Client / Job Submission >Affects Versions: 1.11.2 > Environment: macos 10.14.6 > flink 1.11.2 >Reporter: xuhaoran >Priority: Major > > 按照官网例子 加载Kafka的连接器需要下载flink-sql-connector-kafka_2.11-1.11.2.jar > 然后启动指令为./sql-client.sh embedded -lib xxx/ > 然后创建简单的sql语句 创建 Kafka table > 执行都ok > > 但如果flink-sql-connector-kafka_2.11-1.11.2.jar 放在classpath里面 启动./sql-client.sh > embedded ps -ef|grep java > 也可以看到当前classpath加载了flink-sql-connector-kafka_2.11-1.11.2.jar > 执行之前的创建kafka table 执行select * from table > 报错java.lang.ClassNotFoundException: > org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumer > 怀疑类加载器有问题 > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-20109) java.lang.ClassNotFoundException: org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumer
[ https://issues.apache.org/jira/browse/FLINK-20109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17231381#comment-17231381 ] xuhaoran commented on FLINK-20109: -- CREATE TABLE kafkaTable ( > user_id BIGINT, > item_id BIGINT, > category_id BIGINT, > behavior STRING, > ts String > ) WITH ( > 'connector' = 'kafka', > 'topic' = 'xx', > 'properties.bootstrap.servers' = ':9092', > 'properties.group.id' = 'testGroup', > 'format' = 'json', > 'scan.startup.mode' = 'earliest-offset' > ); select * from kafkaTable;[~zhisheng] if I do like [https://ci.apache.org/projects/flink/flink-docs-release-1.11/dev/table/sqlClient.html] ,put the jar outside，and start cli like the link told ,the sql select * from xx can show me the data > java.lang.ClassNotFoundException: > org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumer > > > Key: FLINK-20109 > URL: https://issues.apache.org/jira/browse/FLINK-20109 > Project: Flink > Issue Type: Bug > Components: Client / Job Submission >Affects Versions: 1.11.2 > Environment: macos 10.14.6 > flink 1.11.2 >Reporter: xuhaoran >Priority: Major > > 按照官网例子 加载Kafka的连接器需要下载flink-sql-connector-kafka_2.11-1.11.2.jar > 然后启动指令为./sql-client.sh embedded -lib xxx/ > 然后创建简单的sql语句 创建 Kafka table > 执行都ok > > 但如果flink-sql-connector-kafka_2.11-1.11.2.jar 放在classpath里面 启动./sql-client.sh > embedded ps -ef|grep java > 也可以看到当前classpath加载了flink-sql-connector-kafka_2.11-1.11.2.jar > 执行之前的创建kafka table 执行select * from table > 报错java.lang.ClassNotFoundException: > org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumer > 怀疑类加载器有问题 > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-20138) Flink Job can not recover due to timeout of requiring slots when flink jobmanager restarted
[
https://issues.apache.org/jira/browse/FLINK-20138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17231387#comment-17231387
]
Till Rohrmann commented on FLINK-20138:
---
Hi [~1026688210], thanks for creating this ticket. From the logs, it looks as
if the {{JobMaster}} never learns about the running {{ResourceManager}} and,
thus, can never connect to it and ask for the required slots. Could you check
whether the {{ResourceManager}} has written its leader address to
{{/flink/115/2476/application_1576582691395_11322/leader/resource_manager_lock}}.
The address information should be the data payload of this znode.
Some exceptions in your logs are the following:
{code}
2020-11-09 16:01:06,519 ERROR main
(org.apache.flink.shaded.curator.org.apache.curator.framework.imps.CuratorFrameworkImpl.logError:566)
- Background exception was not retry-able or retry gave up
java.net.UnknownHostException: haflink.zk23.lf1.in.autohome.com.cn: unknown
error
at java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method)
at java.net.InetAddress$2.lookupAllHostAddr(InetAddress.java:928)
at
java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1323)
at java.net.InetAddress.getAllByName0(InetAddress.java:1276)
at java.net.InetAddress.getAllByName(InetAddress.java:1192)
at java.net.InetAddress.getAllByName(InetAddress.java:1126)
at
org.apache.flink.shaded.zookeeper.org.apache.zookeeper.client.StaticHostProvider.(StaticHostProvider.java:61)
at
org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper.(ZooKeeper.java:445)
at
org.apache.flink.shaded.curator.org.apache.curator.utils.DefaultZookeeperFactory.newZooKeeper(DefaultZookeeperFactory.java:29)
at
org.apache.flink.shaded.curator.org.apache.curator.framework.imps.CuratorFrameworkImpl$2.newZooKeeper(CuratorFrameworkImpl.java:150)
at
org.apache.flink.shaded.curator.org.apache.curator.HandleHolder$1.getZooKeeper(HandleHolder.java:94)
at
org.apache.flink.shaded.curator.org.apache.curator.HandleHolder.getZooKeeper(HandleHolder.java:55)
at
org.apache.flink.shaded.curator.org.apache.curator.ConnectionState.reset(ConnectionState.java:262)
at
org.apache.flink.shaded.curator.org.apache.curator.ConnectionState.start(ConnectionState.java:109)
at
org.apache.flink.shaded.curator.org.apache.curator.CuratorZookeeperClient.start(CuratorZookeeperClient.java:191)
at
org.apache.flink.shaded.curator.org.apache.curator.framework.imps.CuratorFrameworkImpl.start(CuratorFrameworkImpl.java:259)
at
org.apache.flink.runtime.util.ZooKeeperUtils.startCuratorFramework(ZooKeeperUtils.java:131)
at
org.apache.flink.runtime.highavailability.HighAvailabilityServicesUtils.createHighAvailabilityServices(HighAvailabilityServicesUtils.java:123)
at
org.apache.flink.runtime.entrypoint.ClusterEntrypoint.createHaServices(ClusterEntrypoint.java:292)
at
org.apache.flink.runtime.entrypoint.ClusterEntrypoint.initializeServices(ClusterEntrypoint.java:257)
at
org.apache.flink.runtime.entrypoint.ClusterEntrypoint.runCluster(ClusterEntrypoint.java:202)
at
org.apache.flink.runtime.entrypoint.ClusterEntrypoint.lambda$startCluster$0(ClusterEntrypoint.java:164)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
at
org.apache.flink.runtime.security.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41)
at
org.apache.flink.runtime.entrypoint.ClusterEntrypoint.startCluster(ClusterEntrypoint.java:163)
at
org.apache.flink.runtime.entrypoint.ClusterEntrypoint.runClusterEntrypoint(ClusterEntrypoint.java:501)
at
org.apache.flink.yarn.entrypoint.YarnJobClusterEntrypoint.main(YarnJobClusterEntrypoint.java:102)
{code}
and
{code}
2020-11-09 16:01:22,479 ERROR main
(org.apache.flink.shaded.curator.org.apache.curator.ConnectionState.checkTimeouts:228)
- Connection timed out for connection string
(haflink.zk24.lf1.in.autohome.com.cn:2182,haflink.zk25.lf1.in.autohome.com.cn:2182,haflink.zk23.lf1.in.autohome.com.cn:2182,haflink.zk27.lf1.in.autohome.com.cn:2182,haflink.zk26.lf1.in.autohome.com.cn:2182)
and timeout (15000) / elapsed (20966)
org.apache.flink.shaded.curator.org.apache.curator.CuratorConnectionLossException:
KeeperErrorCode = ConnectionLoss
at
org.apache.flink.shaded.curator.org.apache.curator.ConnectionState.checkTimeouts(ConnectionState.java:225)
at
org.apache.flink.shaded.curator.org.apache.curator.ConnectionState.getZooKeeper(ConnectionState.java:94)
at
org.apache.flink.shaded.curator.org.apache.curator.CuratorZookeeperClient.getZoo
[jira] [Commented] (FLINK-20133) Test Pandas UDAF
[ https://issues.apache.org/jira/browse/FLINK-20133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17231392#comment-17231392 ] Huang Xingbo commented on FLINK-20133: -- I have tested pandas udaf in the following sceneries `Batch Group Agg` `Batch Group Window Agg` `Batch Over Window Agg` `Stream Group Window Agg` `Stream Over Window Agg` > Test Pandas UDAF > > > Key: FLINK-20133 > URL: https://issues.apache.org/jira/browse/FLINK-20133 > Project: Flink > Issue Type: Task > Components: API / Python >Affects Versions: 1.12.0 >Reporter: Dian Fu >Assignee: Huang Xingbo >Priority: Critical > Fix For: 1.12.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
